Coding device and method, image capturing device and storage medium

Abstract

The present disclosure relates to an encoding device and method, an image capture device, and a storage medium. An encoding device includes: a generating unit for generating a plurality of RAW data slices for corresponding exposure times from RAW data obtained from an image sensor, the image sensor being capable of performing shooting at different exposure times for each pixel; and an encoding unit for encoding the plurality of RAW data slices generated by the generating unit.

Description

Technical Field

[0001] The present invention relates to a technique for encoding and recording an image obtained by an image sensor capable of controlling the exposure time of each pixel. Background Art

[0002] In a known image capture device, raw image information (RAW data) obtained by capturing performed by an image sensor is converted into signals consisting of brightness and color difference by applying debayer processing (demosaic processing), and so-called development processing such as noise removal, optical distortion correction, and image optimization is performed on each signal. And, generally, the brightness signal and color difference signal that have been subjected to the development processing are compression-encoded and recorded in a recording medium.

[0003] On the other hand, there is also an image capture device that stores image capture data (RAW data) in a recording medium that is in a state immediately after being output from an image sensor and has not yet undergone development processing. When recording RAW data, data preservation can be performed while maintaining a large number of tones without degrading the color information from the image sensor, and thus editing with a high degree of freedom can be performed. However, there is a problem in that the amount of recorded data of RAW data is huge, and a large amount of free space is required in the recording medium. Therefore, it is desired that RAW data is also subjected to compression encoding and recorded while suppressing the amount of data.

[0004] By the way, as a device for obtaining a high dynamic range image, an image capturing device is known, with which an image with a wide dynamic range can be obtained with one shot due to the arrangement of pixels with different exposure times on the same plane, as disclosed in Japanese Patent Laid-Open No. 2013-21660. Japanese Patent Laid-Open No. 2013-21660 discloses a synthesis method for generating a high dynamic range image at the time of development when using such an image capturing device.

[0005] However, in the known technology disclosed in the above-mentioned Japanese Patent Laid-Open No. 2013-21660, a method of encoding RAW data before being subjected to synthesis is not disclosed.

[0006] In addition, when using an image capture device as described in Japanese Patent Laid-Open No. 2013-21660, if an attempt is made to encode RAW data before undergoing synthesis, a large amount of high-frequency components is generated because the level difference between pixels arranged on the same plane with different exposure times is large, and thus encoding efficiency decreases. Therefore, there is a problem that the amount of data when recording RAW data increases. Summary of the invention

[0007] The present invention has been made in view of the above-mentioned problems, and provides a technique for reducing the amount of data when RAW data in which pixel signals having different exposure times are mixed is encoded and recorded.

[0008] According to a first aspect of the present invention, there is provided an encoding device, comprising: a generating unit for generating a plurality of RAW data pieces for corresponding exposure times from RAW data obtained from an image sensor, wherein the image sensor is capable of performing photographing with different exposure times for each pixel; and an encoding unit for encoding the plurality of RAW data pieces generated by the generating unit.

[0009] According to a second aspect of the present invention, there is provided an image capturing device, the image capturing device comprising: an image sensor capable of controlling the exposure time of each pixel; and the above-mentioned encoding device.

[0010] According to a third aspect of the present invention, there is provided an encoding method, comprising: generating a plurality of RAW data slices for corresponding exposure times from RAW data obtained from an image sensor, the image sensor being capable of performing photographing with different exposure times for each pixel; and encoding the plurality of RAW data slices generated in the generation.

[0011] According to a fourth aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing a program for causing a computer to execute the steps of the above-mentioned encoding method.

[0012] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram illustrating a functional configuration of a digital camera as a first embodiment of the encoding device of the present invention.

[0014] Figure 2 is a diagram illustrating a pixel array of an image capture unit.

[0015] Figure 3 is a diagram illustrating settings of a pixel array and an exposure time of an image capturing unit.

[0016] Figure 4A : is a diagram illustrating a RAW data separation method in the first embodiment.

[0017] Figure 4B : is a diagram illustrating a RAW data separation method in the first embodiment.

[0018] Figure 4C: is a diagram illustrating a RAW data separation method in the first embodiment.

[0019] Figure 4D : is a diagram illustrating a RAW data separation method in the first embodiment.

[0020] Figure 5 is a diagram illustrating RAW data output when the exposure time of pixels is the same.

[0021] Figure 6 is a block diagram illustrating the configuration of a RAW encoding unit.

[0022] Fig. 7A and Figure 7B is a diagram illustrating an example of frequency conversion (sub-band division).

[0023] Figure 8 is a diagram illustrating an example of a unit for generating a quantization parameter.

[0024] Fig. 9A is a diagram illustrating an exemplary generation of quantization parameters.

[0025] Fig. 9B is a diagram illustrating an exemplary generation of quantization parameters.

[0026] Fig. 9C is a diagram illustrating an exemplary generation of quantization parameters.

[0027] Fig. 10A and Fig. 10B : is a diagram illustrating a RAW data separation method in the second embodiment.

[0028] Fig.11 : is a diagram illustrating a RAW data separation method in the third embodiment.

[0029] Fig.12 is a diagram illustrating settings of a pixel array and exposure time in the fourth embodiment.

[0030] Fig.13 is a diagram illustrating rearrangement of a pixel array in the fourth embodiment.

[0031] Fig.14 is a block diagram illustrating the configuration of a RAW encoding unit in the fifth embodiment.

[0032] Fig.15A and Fig. 15B It is a diagram illustrating frequency conversion (sub-band division).

[0033] Fig.16 is a processing block diagram used to describe the HDR synthesis process.

[0034] FIG. 17A to FIG. 17C is a diagram illustrating a synthesis ratio in the HDR synthesis process when a long exposure image has a correct exposure.

[0035] 18A to 18C is a diagram illustrating a synthesis ratio in the HDR synthesis process when a short exposure image has a correct exposure.

[0036] FIG. 19A to FIG. 19C is a diagram illustrating exemplary settings of quantization parameters.

[0037] FIG. 20A to FIG. 20C is a flowchart illustrating a quantization processing procedure of the fifth embodiment.

[0038] FIG. 21A to FIG. 21C is a flowchart illustrating a quantization processing procedure of the sixth embodiment. DETAILED DESCRIPTION

[0039] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but are not limited to the invention requiring all such features, and a plurality of such features may be combined as appropriate. In addition, in the accompanying drawings, the same reference numerals are assigned to the same or similar configurations, and redundant descriptions thereof are omitted.

[0040] (First embodiment)

[0041] Figure 1 1 is a block diagram illustrating a functional configuration of a digital camera 100 as a first embodiment of an encoding device of the present invention. The digital camera 100 includes an image capturing unit 101, a separation unit 102, a RAW encoding unit 103, a recording processing unit 104, a recording medium 105, a memory I / F (memory interface) 106, and a memory 107.

[0042] The image capturing unit 101 includes a lens optical system including an optical lens, an aperture, a focus controller, and a lens driving unit and capable of optical zooming, and an image sensor in which a plurality of pixels each including a photoelectric conversion element are two-dimensionally arranged.

[0043] The image sensor performs photoelectric conversion on the subject image formed by the lens optical system in each pixel, and also performs analog / digital conversion using an A / D conversion circuit, and outputs a digital signal (pixel data, RAW data) in units of pixels. A CCD image sensor, a CMOS image sensor, or the like is used as the image sensor.

[0044] Note that, in this embodiment, each pixel of the image sensor is provided with one of R (red), G1 / G2 (green), and B (blue) color filters, such as Figure 2 Note that the RAW data output from the image capturing unit 101 is stored in the memory 107 via the memory I / F 106 .

[0045] The separation unit 102 is a circuit or module for separating the RAW data obtained by the image capture unit 101 into RAW data pieces for corresponding exposure times. The RAW data stored in the memory 107 is read out via the memory I / F 106 and separated into RAW data pieces for corresponding exposure times, and the RAW data pieces are output to the RAW encoding unit 103.

[0046] The RAW encoding unit 103 is a circuit or module that performs a calculation operation on the RAW data and encodes the RAW data input from the separation unit 102. The RAW encoding unit 103 stores the encoded data generated by the encoding in the memory 107 via the memory I / F 106.

[0047] The recording processing unit 104 reads out various types of data such as encoded data stored in the memory 107 via the memory I / F 106, and records the read data in the recording medium 105. The recording medium 105 is a recording medium constituted by a large-capacity random access memory such as a nonvolatile memory.

[0048] The memory I / F 106 mediates a memory access request from the processing unit and performs read / write control for the memory 107. The memory 107 is a volatile memory such as SDRAM and serves as a storage section. The memory 107 provides a storage area for storing various types of data such as image data and sound data mentioned above or various types of data output from the processing unit constituting the digital camera 100.

[0049] Next, we will refer to Figure 2 The pixel arrangement structure of the image capturing unit 101 is described below. Figure 2 As shown in , the image capture unit 101 is characterized in that R pixels, G1 pixels, G2 pixels, and B pixels are arranged in units of 2×2 pixels, and the same color is arranged in each 2×2 pixel. The image capture unit 101 has a structure in which a total of 4×4 pixels is a minimum unit and the minimum unit is repeatedly arranged.

[0050] Will refer to Figure 3 Description has Figure 2 The pixel arrangement structure shown in FIG. 1 and the exposure time setting in the image sensor in which the exposure time can be controlled for each pixel (photographing is possible when the exposure time is different for each pixel). Figure 3As shown in , the horizontal direction is represented by x, the vertical direction is represented by y, the column number is represented by the x coordinate and the row number is represented by the y coordinate. The bracketed numbers indicate the coordinates indicating the position of each pixel on the image sensor. In addition, white pixels represent short-exposure pixels, and gray pixels represent long-exposure pixels. In this embodiment, short-exposure pixels that perform short exposure and long-exposure pixels that perform long exposure are arranged in a zigzag manner in the column direction, as shown in FIG. Figure 3 as shown in .

[0051] For example, about Figure 3 The exposure time of the four R pixels at the upper left end is set as follows. R(1,1) is a short exposure pixel, R(2,1) is a long exposure pixel, R(1,2) is a long exposure pixel, and R(2,2) is a short exposure pixel. In this way, short exposure pixels and long exposure pixels are alternately set in each column, and short exposure pixels and long exposure pixels are alternately set in each row. When only short exposure pixels are followed in the y direction, in the first column and the second column, in the first row from the top, the first column is a short exposure pixel, in the second row, the second column is a short exposure pixel, in the third row, the first column is a short exposure pixel, and in the fourth row, the second column is a short exposure pixel. Similarly, when only long exposure pixels are followed in the y direction, in the first column and the second column, in the first row from the top, the second column is a long exposure pixel, in the second row, the first column is a long exposure pixel, in the third row, the second column is a long exposure pixel, and in the fourth row, the first column is a long exposure pixel.

[0052] As described above, the pixel arrangement structure and the exposure time are set so that pixels of the same color are set in units of 2×2 pixels, and two short exposure pixels (one of the two exposure times) and two long exposure pixels (the other of the two exposure times) are arranged among these 4 pixels.

[0053] Here, if encoding is attempted in a state where RAW data is obtained by the image capture unit 101 (i.e., in a state where pixels with different exposure times are mixed), a large amount of high-frequency components are generated because the level difference between pixels with different exposure times is large, and the amount of recorded data of the RAW data increases. Therefore, in the present embodiment, the RAW data is separated into RAW data pieces of corresponding exposure times by the separation unit 102, and the generation of high-frequency components is suppressed by matching the levels between pixels, whereby the amount of recorded data of the RAW data is reduced.

[0054] Next, we will refer to FIG. 4A to FIG. 4D Describe the separation method. 4A to 4DAs shown in , the separation unit 102 separates the RAW data input from the image capture unit 101 into Bayer arrangement structure RAW data composed only of short-exposure pixels and Bayer arrangement structure RAW data composed only of long-exposure pixels, and outputs the separated two RAW data pieces to the RAW encoding unit 103.

[0055] Specifically, RAW data consisting only of short-exposure pixels is separated into Figure 4A RAW data 401a and Figure 4B The RAW data 401a is short-exposure RAW data configured by extracting short-exposure pixels each marked by a diamond in odd rows and odd columns, as shown in FIG. Figure 4A In addition, the RAW data 401b is short-exposure RAW data configured by extracting short-exposure pixels each marked by a diamond in an even-numbered row and an even-numbered column, as shown in FIG. Figure 4B as shown in .

[0056] Similarly, RAW data consisting only of long-exposure pixels is separated into Figure 4C RAW data in 401c and Figure 4D The RAW data 401c is long-exposure RAW data configured by extracting long-exposure pixels each marked by a diamond in odd-numbered rows and even-numbered columns, as shown in FIG. Figure 4C In addition, RAW data 401d is long-exposure RAW data configured by extracting long-exposure pixels each marked by a diamond in an even-numbered row and an odd-numbered column, as shown in FIG. Figure 4D The RAW encoding unit 103 encodes the RAW data 401a, 401b, 401c, and 401d input from the separation unit 102 in the Bayer arrangement, respectively.

[0057] Note that the above has been used Figure 2 The pixel array in FIG. 1 describes the separation method of the separation unit 102 when the exposure time is different between pixels arranged on the same plane. Figure 5 The process to be performed by the separation unit 102 when the exposure times of the pixels are all the same is described.

[0058] In this case, for the RAW data obtained by the image capturing unit 101, the separation unit 102 calculates the Figure 5 The RAW data 501 is configured by calculating the pixel average value of every four pixels of the same color component (as shown in ), and the RAW data 501 is output to the RAW encoding unit 103. Specifically, as shown in the following formulas 1 to 4, separation is performed by calculating the addition average of each color component.

[0059]

[0060]

[0061]

[0062]

[0063] Next, we will refer to Figure 6 The block diagram shown in exemplifies a detailed configuration and a processing flow of the RAW encoding unit 103 that performs processing on the short-exposure RAW data 401 a and 401 b and the long-exposure RAW data 401 c and 401 d.

[0064] The RAW encoding unit 103 includes a channel transform unit 601 , a frequency transform unit 602 , a quantization parameter generating unit 603 , a quantization unit 604 , an encoding unit 605 , and a quantization parameter encoding unit 606 .

[0065] The channel conversion unit 601 converts the RAW data configured in the Bayer arrangement input from the separation unit 102 into a plurality of channels. For example, conversion into four channels is performed separately for R, G1, G2, and B in the Bayer arrangement. Alternatively, for R, G1, G2, and B, conversion into four channels is performed by further performing calculations using the following conversion formulas 5 to 8.

[0066] Y=(R+G1+G2+B) / 4 Formula 5

[0067] C0=RB Formula 6

[0068] C1=(G0+G1) / 2=(R+B) / 2 Formula 7

[0069] C2=G0-G1 Formula 8

[0070] Note that an exemplary configuration for transformation to four channels is shown here, but the number of channels and the transformation method are not limited thereto.

[0071] The frequency transform unit 602 performs frequency transform processing by discrete wavelet transform at a predetermined resolution level (hereinafter, represented as “lev”) for each channel, and outputs the generated subband data (transform coefficient) to the quantization parameter generation unit 603 and the quantization unit 604 .

[0072] Fig. 7A1 shows a filter bank configuration for implementing a discrete wavelet transform subband segmentation process with respect to lev=1. When the discrete wavelet transform process is performed in the horizontal direction and the vertical direction, segmentation into one low-frequency subband (LL) and three high-frequency subbands (HL, LH, HH) is performed, as shown in FIG. Figure 7B as shown in .

[0073] Formula 9 and Formula 10 show Fig. 7A . Transfer functions of a low-pass filter (hereinafter, represented as “lpf”) and a high-pass filter (hereinafter, represented as “hpf”) shown in FIG.

[0074] lpf(z)=(-z -2 +2z -1 +6+2z 1 -z 2 ) / 8 Formula 9

[0075] hpf(z)=(-z -1 +2-z 1 ) / 2 Formula 10

[0076] When lev is greater than 1, subband segmentation is performed hierarchically for the low frequency subband (LL). Note that here, as shown in Formula 9 and Formula 10, the discrete wavelet transform is configured by a five-tap lpf and a three-tap hpf, but is not limited thereto, and a filter configuration with different tap numbers and coefficients may be used.

[0077] The quantization parameter generation unit 603 generates, for each specific predetermined subband data unit, a quantization parameter for performing quantization processing on the subband data (transform coefficient) generated by the frequency transform unit 602. The generated quantization parameter is input to the quantization parameter encoding unit 606 and is also supplied to the quantization unit 604.

[0078] The quantization unit 604 performs quantization processing on the sub-band data (transform coefficient) output from the frequency transform unit 602 based on the quantization parameter supplied from the quantization parameter generation unit 603 , and outputs the quantized sub-band data (transform coefficient) to the encoding unit 605 .

[0079] The encoding unit 605 performs predictive differential entropy encoding of the quantized subband data (transform coefficients) output from the quantization unit 604 for each subband in raster scan order, and stores the generated encoded RAW data to the memory 107. Note that other methods may be used as the prediction method and the entropy encoding method.

[0080] The quantization parameter encoding unit 606 is a processing unit for performing encoding on the quantization parameter input from the quantization parameter generating unit 603. The quantization parameter is encoded using the same encoding method as the encoding unit 605, and the generated encoded quantization parameter is stored in the memory 107.

[0081] Next, we will refer to Figure 8 The relationship between sub-band data, channel data, and RAW data when a quantization parameter is generated assuming that the above-mentioned predetermined sub-band unit is 4×4 is described.

[0082] like Figure 8 As shown in , the 4×4 subband corresponds to 8×8 pixels of each channel and also corresponds to a block corresponding to 16×16 pixels of each RAW data. Therefore, in this case, in the short-exposure RAW data 401a and 401b and the long-exposure RAW data 401c and 401d, for each RAW data block corresponding to 16×16 pixels, it is necessary to store the quantization parameter in the memory 107.

[0083] Note that it is effective to apply the same quantization parameter to the short exposure RAW data 401a and 401b and to the long exposure RAW data 401c and 401d in order to reduce the data amount of the quantization parameter. In this case, the data amount can be reduced to half. In addition, in the present embodiment, the quantization parameter generated with an exposure time closer to the correct exposure is used as a reference, and other quantization parameters are calculated to further reduce the data amount. Thus, the data amount of the quantization parameter can be reduced to one-quarter. Here, the reason for using the quantization parameter generated with an exposure time closer to the correct exposure as a reference is because, in the case of an overexposure or underexposure image with a blown out highlight or blacked out, the quantization parameter cannot be generated based on the accurate features of the subject.

[0084] When the short exposure is closer to the correct exposure, as a specific example, a calculation formula for calculating the quantization parameter for the long exposure RAW data with reference to the quantization parameter generated about the short exposure RAW data is shown in Formula 11.

[0085] L_Qp=α×S_Qp+β Formula 11

[0086] here,

[0087] L_Qp: Quantization parameter for long exposure RAW data

[0088] S_Qp: Quantization parameter for short exposure RAW data

[0089] α: Slope

[0090] β: Intercept.

[0091] Note that, in this embodiment, the quantization parameters for long-exposure RAW data are calculated with reference to the quantization parameters generated for short-exposure RAW data. However, the quantization parameters for short-exposure RAW data can be calculated with reference to the quantization parameters generated for long-exposure RAW data. Additionally, the quantization parameters can be calculated by setting α and β for each of the short exposure and the long exposure without using both the short exposure and the long exposure as references.

[0092] Next, the method for determining α and β shown in Formula 11 will be described. Although α and β can be any values, in this embodiment, a detailed parameter determination method will be described. When it is assumed that the short exposure is closer to the correct exposure, as in the above example, in the long exposure, since the exposure time is longer than the short exposure, overexposure is achieved. Therefore, for regions that are medium to bright in brightness during short exposure, in long exposure, the pixel values reach the saturation level and it is highly likely that pixel values according to the subject brightness cannot be output. On the other hand, for dark regions, it is possible to obtain detailed information with respect to the short exposure. Therefore, for regions determined to be medium to bright in brightness in the short-exposure RAW data, the quantization parameters for the long-exposure RAW data are increased relative to the short exposure. Additionally, the same parameters are set for regions determined to be dark, and thus, the data volume of the quantization parameters can be reduced while ensuring image quality.

[0093] A specific description will be given with reference to 9A to 9C the following. Fig. 9A An exemplary setting of the quantization parameters according to the brightness of the short-exposure RAW data in the short-exposure RAW data is shown. Additionally, Fig. 9B an exemplary setting of the quantization parameters according to the brightness of the short-exposure RAW data in the long-exposure RAW data is shown. Note that the 1LL sub-band corresponding to the above quantization parameter generation unit can be used to evaluate the brightness index. The magnitude relationships between the quantization parameters are shown in Formulas 12 to 14.

[0094] Q0 < Q1 < Q2 Formula 12

[0095] Q1 < Q3 Formula 13

[0096] Q2 < Q4 Formula 14

[0097] First, considering the visual characteristics (Q0 < Q1 < Q2), the quantization parameter in the short-exposure RAW data is set such that the quantization parameter decreases as the darkness increases. In contrast, in the long-exposure RAW data, the quantization parameter is set such that Q0 is set to be the same as that in the short-exposure RAW data in the region corresponding to the dark part of the short-exposure RAW data, and the quantization parameter is set to increase relative to the short-exposure RAW data in the region corresponding to the medium-to-bright part (Q1 < Q3, Q2 < Q4).

[0098] Fig. 9C A graph line for calculating the quantization parameter for the long-exposure RAW data with reference to the quantization parameter generated for the short-exposure RAW data is shown. The horizontal axis represents the quantization parameter (S_Qp) for the short-exposure RAW data, and the vertical axis represents the quantization parameter (L_Qp) for the long-exposure RAW data. α and β shown in Formula 11 can be set to achieve the relationships of Formulas 12 to 14.

[0099] Note that α and β are stored in the memory 107 in a similar manner to the encoded data and are recorded on the recording medium 105 via the memory I / F 106 together with the encoded data. In addition, a flag indicating which of the short exposure and the long exposure has the quantization parameter to be used as a reference is stored in the memory 107 and is recorded on the recording medium 105 via the memory I / F 106 together with the encoded data. Note that when α and β are set for each exposure time without using both the short exposure and the long exposure as references, this flag may not be included.

[0100] In addition, when dealing with any of the above cases, the configuration may be such that a flag indicating whether the exposure time will be used as a reference is included, and then, if there is an exposure time to be used as a reference, a flag indicating which of the short exposure and the long exposure is used as a reference is included. In this case as well, each flag information is stored in the memory 107 and is recorded on the recording medium 105 via the memory I / F 106 together with the encoded data.

[0101] As described above, in the present embodiment, the separation unit 102 separates the RAW data into data slices for corresponding exposure times, the level difference between the pixels to be encoded is eliminated, and thereby, high-frequency components are suppressed, and thus, the recording data amount of the RAW data can be reduced. In addition, using the quantization parameter calculated for one type of RAW data as a reference, the quantization parameter for other RAW data with different exposure times is determined, and thus, the recording data amount of the RAW data can be reduced.

[0102] (Second Embodiment)

[0103] Next, a second embodiment of the present invention will be described. In the second embodiment, the separation method of the RAW data in the separation unit 102 is different from the separation method of the RAW data of the first embodiment. Note that the configuration of the digital camera of the second embodiment is the same as that of the digital camera of the first embodiment, and therefore redundant description will be omitted, and the differences will be described.

[0104] In the first embodiment, RAW data pieces obtained by separating pixels into groups of pixels of the same exposure time in the separation unit 102 (i.e., specifically, separating into two planes of RAW data consisting only of short-exposure pixels and two planes of RAW data consisting only of long-exposure pixels) are output to the RAW encoding unit 103.

[0105] In contrast, in a second embodiment, the following method will be described: in order to further reduce the amount of data, in the separation unit 102, the pixel values ​​of pixels with the same exposure time and the same color component that exist nearby are added, and the average pixel value is calculated and output to the RAW encoding unit 103.

[0106] Will refer to Fig. 10A and Fig. 10B The processing of the separation unit 102 in this embodiment is described. For the RAW data input from the image capture unit 101 and in which pixels of different exposure times are mixed, the separation unit 102 calculates the Fig. 10A The pixel values ​​of the pixels enclosed by each rectangle shown in (i.e., pixels that are short-exposure pixels and have the same color component) are averaged and separated into short-exposure RAW data 1001a. Specifically, as shown in the following formulas 15 to 18, separation is performed by calculating the average for each color component.

[0107]

[0108]

[0109]

[0110] . .

[0111] Similarly, by calculating Fig. 10B The long-exposure RAW data 1001b is separated by calculating the addition average of pixels that are long-exposure pixels and have the same color component and are surrounded by each rectangle shown in FIG. Specifically, as shown in the following formulas 19 to 22, the separation is performed by calculating the addition average for each color component.

[0112]

[0113]

[0114]

[0115] . .

[0116] As described above, in the second embodiment, RAW data obtained by the image capturing unit 101 is separated by calculating the addition average in the separation unit 102, and therefore, the amount of data to be output to the RAW encoding unit 103 can be reduced to half relative to the first embodiment.

[0117] (Third Embodiment)

[0118] Next, a third embodiment of the present invention will be described. In the third embodiment, the separation method of RAW data in the separation unit 102 is different from the separation method of RAW data in the first and second embodiments. Note that the configuration of the digital camera of this embodiment is the same as that of the first and second embodiments, and therefore redundant description will be omitted, and the differences will be described.

[0119] In the second embodiment, the addition average of the pixel values ​​of pixels of the same exposure time and the same color component existing nearby is calculated in the separation unit 102, and the addition average is output to the RAW encoding unit 103. In the third embodiment, in order to further reduce the data amount relative to the second embodiment, a gain is applied to the RAW data of one exposure time according to the RAW data of the other exposure time, and the difference therebetween is output to the RAW encoding unit 103. That is, the RAW encoding unit 103 encodes the addition average RAW data for one exposure time, and encodes the RAW data of the difference (differential RAW data) for the other exposure time.

[0120] Will refer to Fig.11 The processing in the separation unit 102 in this embodiment is described. First, the separation unit 102 adds the pixel values ​​of pixels with the same exposure time and the same color component existing nearby, and Fig. 10A and Fig. 10BThe second embodiment shown in is similarly calculated to obtain the RAW data 1001a and 1001b by averaging them. Next, the difference between the first row in the long-exposure RAW data 1001b and the value obtained by multiplying the first row of the short-exposure RAW data 1001a by the gain γ corresponding to the long-exposure RAW data plus the offset ε is obtained. Here, the gain γ and the offset ε can be determined by performing calculations in reverse from the exposure time in advance, or can be determined using the obtained histogram of the pixel values ​​of the short-exposure pixels and the long-exposure pixels.

[0121] Specifically, as shown in the following Formulas 23 to 26, the difference in the first row is calculated.

[0122]

[0123]

[0124]

[0125]

[0126] This operation is similarly performed for the second, third, and fourth rows in addition to the first row, and the calculated difference values ​​are output to the RAW encoding unit 103. Note that in the present embodiment, correction is performed for short-exposure RAW data, but correction may be performed for long-exposure RAW data. However, from the perspective of rounding processing, the accuracy of the difference is better when the gain γ is applied to the short-exposure RAW data.

[0127] As described above, in the third embodiment, instead of outputting the RAW data as it is to the RAW encoding unit 103, the RAW data is output as a difference value, and therefore, the recording data amount of the RAW data can be further reduced relative to the second embodiment.

[0128] (Fourth embodiment)

[0129] Next, a fourth embodiment of the present invention will be described. In the fourth embodiment, a pixel array different from the pixel arrays of the first to third embodiments (ie, specifically, Fig.12 The pixel array shown in ) is applied to the image capturing unit 101.

[0130] In the first to third embodiments described above, the image capturing unit 101 having the structure in which the minimum unit includes 4×4 16 pixels composed of four different pixels R, G1, G2, and B, and the minimum unit is repeatedly arranged as shown in FIG. Figure 2 as shown in .

[0131] In contrast, Fig.12 1 shows the pixel array and exposure time setting of the image capture unit 101 in the fourth embodiment. The horizontal direction is represented by x, the vertical direction is represented by y, the column number is represented by the x coordinate, and the row number is represented by the y coordinate. The bracketed numbers indicate the coordinates indicating the position of each pixel on the image sensor. In addition, white pixels represent short-exposure pixels, and gray pixels represent long-exposure pixels. In this way, Fig.12 , in a pixel array of a Bayer arrangement constituted by an array of R, G1, G2, and B pixels, short-exposure pixels and long-exposure pixels are alternately arranged in units of two columns.

[0132] Likewise, in Fig.12 In the pixel array and exposure time settings, such as Fig.13 As shown in Figure 2 As a result of executing the processing while rearranging the pixel arrangement structure shown in , the processing described in the first to third embodiments can be executed.

[0133] As described above, in the fourth embodiment, even if the pixel array is changed, processing similar to that described in the first to third embodiments can be performed.

[0134] (Fifth Embodiment)

[0135] Next, we will refer to Fig.14 The block diagram shown in FIG. 1 describes a detailed configuration and a processing flow of the RAW encoding unit 103 that performs encoding processing on the short-exposure RAW data 401a and 401b and the long-exposure RAW data 401c and 401d in the fifth embodiment. Note that, Figures 1 to 5 The configuration shown in is similar to that of the first embodiment.

[0136] The RAW encoding unit 103 mainly includes a channel transform unit 1601 , a frequency transform unit 1602 , a quantization parameter generating unit 1603 , a quantization unit 1604 and an encoding unit 1605 .

[0137] The channel conversion unit 1601 converts the RAW data configured in the Bayer arrangement input from the separation unit 102 into a plurality of channels. Here, the conversion into four channels is performed separately for R, G1, G2, and B in the Bayer arrangement.

[0138] The frequency transform unit 1602 performs frequency transform processing by discrete wavelet transform at a predetermined resolution level (hereinafter, represented as “lev”) for each channel, and outputs the generated subband data (transform coefficient) to the quantization parameter generation unit 1603 and the quantization unit 1604 .

[0139] Fig.15A1 shows a filter bank configuration for implementing a discrete wavelet transform subband segmentation process with respect to lev=1. When the discrete wavelet transform process is performed in the horizontal direction and the vertical direction, segmentation into one low-frequency subband (LL) and three high-frequency subbands (HL, LH, HH) is performed, as shown in FIG. Fig. 15B as shown in .

[0140] Formula 27 and Formula 28 show Fig.15A . Transfer functions of a low-pass filter (hereinafter, represented as “lpf”) and a high-pass filter (hereinafter, represented as “hpf”) shown in FIG.

[0141] lpf(Z)=(-Z -2 +2Z -1 +6+2Z 1 -Z 2 ) / 8 Formula 27

[0142] hpf(Z)=(-Z -1 +2-Z 1 ) / 2 Formula 28

[0143] When lev is greater than 1, subband segmentation is performed hierarchically for the low frequency subband (LL). Note that here, as shown in Formula 27 and Formula 28, the discrete wavelet transform is configured by a five-tap lpf and a three-tap hpf, but is not limited thereto, and a filter configuration with different tap numbers and coefficients may be used.

[0144] The quantization parameter generating unit 1603 calculates the brightness feature quantity with a predetermined coefficient (a square block of more than one coefficient, a square area of ​​more than one pixel) for the sub-band data (transformation coefficient) generated by the frequency transform unit 1602, and generates a quantization parameter based on the feature quantity. Similarly, quantization is performed with a predetermined coefficient (a square block of more than one coefficient) as a unit, but considering the controllability of image quality, it is expected to be the same as the unit for calculating the feature quantity. Subsequently, the method for setting the quantization parameter according to the brightness and the process for generating the quantization parameter will be described in detail. Then, the generated quantization parameter is output to the quantization unit 1604.

[0145] The quantization unit 1604 performs quantization processing on the subband data (transform coefficient) input from the frequency transform unit 1602 using the quantization parameter supplied from the quantization parameter generation unit 1603 , and outputs the quantized subband data (transform coefficient) to the encoding unit 1605 .

[0146] The encoding unit 1605 performs predictive differential entropy encoding of the quantized subband data (transform coefficients) input from the quantization unit 1604 for each subband in raster scan order, and stores the generated encoded RAW data to the memory 107. Note that other methods, prediction methods and entropy encoding methods may be used.

[0147] Here, we will use Fig.16 Describes the HDR (High Dynamic Range) synthesis process. Fig.16 1 is a processing block diagram for performing HDR synthesis. The digital camera 100 is configured to record two RAW data tables (sheet of RAW data) with different exposure amounts, and therefore a description is given assuming that the HDR synthesis processing in the present embodiment performs HDR synthesis on the two RAW data tables. Note that one of the exposure RAW data tables is RAW data obtained by capturing performed with correct exposure. The other RAW data table is RAW data obtained with an exposure time that causes overexposure or underexposure, which is auxiliary data for DR expansion.

[0148] The development processing unit 801 performs development processing on the long-exposure RAW data. Then, the generated developed long-exposure image is output to the gain correction unit 803. The development processing unit 802 performs development processing on the short-exposure RAW data. Then, the generated developed short-exposure image is output to the gain correction unit 804.

[0149] The gain correction unit 803 performs gain correction on the long exposure image using a gain value based on a predetermined synthesis ratio. The synthesis ratio will be described later. The gain correction unit 804 performs gain correction on the short exposure image using a gain value based on a predetermined synthesis ratio. The synthesis ratio will be described later. The addition processing unit 805 performs addition processing of pixels at the same coordinate position on the long exposure image and the short exposure image.

[0150] In this way, in the HDR synthesis process, the gain correction process and the addition process are performed on the image generated by performing the development process on the two RAW data tables with different exposure amounts. Note that this HDR synthesis process is similarly performed on each color component (R, G, B) constituting the image data. In addition, the development process includes de-Bayering, brightness color difference conversion, noise removal, optical distortion correction, and the like.

[0151] Next, the synthesis ratio between the short exposure image data and the long exposure image data will be described. The idea of ​​the synthesis ratio is different based on which exposure image data has the correct exposure image data. The case where the long exposure image data has the correct exposure and the case where the short exposure image data has the correct exposure will be described separately.

[0152] First, the synthesis ratio in the case where the long exposure image data has the correct exposure will be described. When the long exposure image data is obtained by capturing performed with the correct exposure, the exposure time of the short exposure image data is relatively shorter than that of the long exposure image data, and thus the short exposure image data has underexposure.

[0153] exist Fig.17A An example of a histogram of image data when capturing is performed under this exposure condition is shown in FIG. Fig.17A The histogram shown in is a histogram of specific color components constituting the image data. The horizontal axis of the histogram represents the pixel value indicating the brightness of the image data, and the vertical axis represents the number of pixels. In addition, Ta and Tb represent pixel thresholds, and Tc represents the pixel upper limit value. The definition is as follows: the area that satisfies the condition of pixel value ≤ Ta is called the dark part, the area that satisfies the condition of Ta<pixel value ≤ Tb is called the middle part, and the area that satisfies the condition of Tb<pixel value is called the bright part. In this histogram, the long exposure image data correctly expresses the tones in the dark part area and the middle part area, but in the bright part area, there are many pixels in the area above Tc, Tc is the pixel upper limit, and therefore the long exposure image data is in a state in which the tonal information is lost due to the occurrence of overexposure highlights. In the HDR synthesis process, in order to expand the tonal range in which overexposure highlights occur, short exposure image data at the same coordinate position is synthesized. In the HDR synthesis processing under this exposure condition, the addition processing is performed by performing gain correction, so that the synthesis ratio of the long exposure image data is large in the dark part area and the medium part area where DR can be ensured when correctly exposed, and the synthesis ratio of the short exposure image data is increased in the bright part area where it is difficult to ensure DR when correctly exposed.

[0154] Fig. 17B An example of the composite ratio is shown in . The horizontal axis represents the pixel value of the long exposure image data (correct exposure), and the vertical axis represents the composite ratio. Fig. 17B The graph in represents the synthesis ratio of the exposure image data pieces according to the pixel value, and the synthesis ratio of the exposure image data pieces changes so that the sum thereof is constantly 100%. Fig.17A As described in , because the bright part includes many pixels that are overexposed and bright, Fig. 17BIn the graph in , the synthesis ratio of the long exposure image data decreases from the pixel value at the threshold value Tb to 0% at the pixel upper limit value Tc, and the synthesis ratio of the short exposure image data increases from the pixel value at the threshold value Tb to 100% at the pixel upper limit value Tc. Since such a synthesis ratio is used, it is possible to extend the DR in the synthesized image while weakening the influence of overexposure highlights. Note that, in order to make the description easier to understand, an example has been described in which the synthesis ratio is changed in the case where the threshold value Tb is a boundary, but the synthesis ratio of the exposure image data piece is not limited to this.

[0155] Based on the above description, Fig. 17C The magnitude relationship between the composite ratios of the long exposure image data and the short exposure image data is shown in the figure. A0 in the figure represents the composite ratio in the dark part of the long exposure pixel, A1 represents the composite ratio in the middle part of the long exposure pixel, and A2 represents the composite ratio in the bright part of the long exposure pixel. In addition, A3 in the figure represents the composite ratio in the dark part of the short exposure pixel, A4 represents the composite ratio in the middle part of the short exposure pixel, and A5 represents the composite ratio in the bright part of the short exposure pixel. The magnitude relationship between the composite ratios for corresponding brightness areas is A0>A3 in the dark part, A1>A4 in the middle part, and A2>A3 in the bright part. <A5。

[0156] Next, the synthesis ratio in the case where the short exposure image data has the correct exposure will be described. When the short exposure image data is obtained by capturing performed with the correct exposure, the exposure time of the long exposure image data is relatively longer than that of the short exposure image data, and thus the long exposure image data is overexposed.

[0157] exist Fig.18A An example of a histogram of image data when capturing is performed under this exposure condition is shown in FIG. Fig.18AThe histogram shown in is a histogram of specific color components constituting image data. The horizontal axis of the histogram represents the pixel value indicating the brightness of the image data, and the vertical axis represents the number of pixels. In addition, Ta and Tb represent pixel thresholds, and Td represents the pixel lower limit value. The definition is as follows: the area that satisfies the condition of pixel value ≤ Ta is called the dark part, the area that satisfies the condition of Ta<pixel value ≤ Tb is called the middle part, and the area that satisfies the condition of Tb<pixel value is called the bright part. In this histogram, the short exposure image data correctly expresses the tones in the middle part area and the bright part area, but in the dark part area, there are many pixels in the area below Td, Td is the pixel lower limit, and therefore the short exposure image data is in a state in which the tones information is lost due to the appearance of blocked up shadows. In the HDR synthesis process, in order to expand the tonal range in which blocked up shadows appear, long exposure image data at the same coordinate position is synthesized. In the HDR synthesis processing under this exposure condition, the addition processing is performed by performing gain correction, so that the synthesis ratio of the short-exposure image data is large in the middle part area and the bright part area where DR can be ensured when correctly exposed, and the synthesis ratio of the long-exposure image data is increased in the dark part area where it is difficult to ensure DR when correctly exposed.

[0158] Next, in Fig.18B An example of the composite ratio is shown in . The horizontal axis represents the pixel value of the short-exposure image data (correct exposure), and the vertical axis represents the composite ratio. Fig.18B The graph in represents the synthesis ratio of the exposure image data pieces according to the pixel value, and the synthesis ratio of the exposure image data pieces changes so that the sum thereof is constantly 100%. Fig.18A As described in , since the dark part includes many pixels where occlusion shadows appear, Fig.18B In the graph in , the synthesis ratio of the long exposure image data is changed to 100% at the pixel lower limit value Td, and the synthesis ratio of the short exposure image data is changed to 0% at the pixel lower limit value Td. Since such a synthesis ratio is used, it is possible to extend the DR in the synthesized image while weakening the influence of the occlusion shadow. Note that, in order to make the description easier to understand, an example has been described in which the synthesis ratio is changed in the case where the threshold value Ta is a boundary, but the synthesis ratio of the exposure image data piece is not limited to this.

[0159] Based on the above description, Fig. 18CThe magnitude relationship between the synthesis ratio of the long exposure image data and the short exposure image data is shown. In the figure, B0 represents the synthesis ratio in the dark part of the short exposure pixels, B1 represents the synthesis ratio in the medium part of the short exposure pixels, and B2 represents the synthesis ratio in the bright part of the short exposure pixels. Additionally, in the figure, B3 represents the synthesis ratio in the dark part of the long exposure pixels, B4 represents the synthesis ratio in the medium part of the long exposure pixels, and B5 represents the synthesis ratio in the bright part of the long exposure pixels. The magnitude relationship between the synthesis ratios for the corresponding brightness regions is B0 < B3 in the dark part, B1 > B4 in the medium part, and B2 > B5 in the bright part.

[0160] As described above, in the HDR synthesis process, the synthesis ratio of the exposure image data slices changes according to whether it has correct exposure and the magnitude of the pixel value (brightness). The magnitude of the synthesis ratio indicates the degree of influence on the image quality. In regions with a large synthesis ratio, the influence on the image quality is large, and in regions with a smaller synthesis ratio, the influence on the image quality is smaller. Therefore, for the RAW data to be compressed and recorded, it is necessary to most appropriately distribute the code amount based on the synthesis ratio in the HDR synthesis process according to the degree of influence on the image quality. That is, it is important to set the quantization parameters such that the image quality is ensured by allocating a larger code amount to regions with a larger synthesis ratio, and the code amount is reduced for regions with a small synthesis ratio and a small influence on the image quality.

[0161] Next, the basic idea in the quantization parameter generation executed by the quantization parameter generation unit 1603 will be described. As described above, it is assumed that the quantization parameters are weighted according to the synthesis ratio obtained through the预想HDR synthesis process. The idea of weighting the quantization parameters according to the brightness considering the visual characteristics of the image is added to it.

[0162] It should be noted that there is an unclear expression "预想HDR synthesis process" in the original text. It might need to be further clarified in the actual context.In the post-processing after development, the RAW data undergoes adjustment of the luminance level such as gamma correction processing and tone curve correction processing. When a dark portion with a small original luminance level is compared with a bright portion with a large original luminance level, the change ratio of the pixel value in the dark portion is large even if the adjustment is performed on the same luminance level. If the quantization process is performed with the same quantization parameter for the dark portion and the bright portion, the change ratio of the pixel value in the dark portion is large, and thus the quantization error caused by the quantization process is amplified, and the image quality degradation becomes obvious. On the other hand, in the bright portion with a small change ratio of the luminance level, the change ratio of the pixel value is also small, and thus the degree of amplification of the quantization error is small, and the image quality degradation is not obvious. In order to ensure the image quality after post-processing, it is necessary to perform quantization of the RAW data in consideration of the quantization error amplified by the post-processing. In addition, in the dark portion, the contrast is small relative to the contrast in the bright portion, and the signal level of the sub-band data is small. Therefore, if coarse quantization is performed on the dark portion, the sub-band data after quantization is likely to be 0. Once the coefficient becomes 0, the signal cannot be restored in the inverse quantization process, and significant image quality degradation occurs.

[0163] For these reasons, control is performed so that the quantization parameter is reduced in the dark part area where the image quality degradation is likely to be noticeable, and the quantization parameter is increased in the bright part area where the image quality degradation is not likely to be noticeable. In the present embodiment, the following configuration will be described: in which a quantization table in which the quantization parameters for the corresponding sub-bands are compiled is prepared in advance, and the quantization table to be referenced is switched according to the synthesis ratio and the brightness feature amount. These quantization tables are composed of quantization parameters for the corresponding sub-band data slices according to lev. The quantization parameter for each sub-band is set so that the quantization parameter is smaller in the lower sub-band where the image quality degradation is likely to be noticeable. If lev=1, the size relationship between the quantization parameters of the corresponding sub-bands is 1LL<1HL=1LH<1HH.

[0164] Based on the idea of ​​weighting the quantization parameter according to the brightness, exemplary settings of the quantization table for the RAW data slice obtained by capturing with the corresponding exposure time will be described with respect to the following three conditions respectively. Note that in this embodiment, an example will be described in which the brightness feature amount is classified into three feature areas of a dark part, a medium part, and a bright part. Note that the definitions of the features to be classified are similar to those in the histograms in FIGS. 17 and 18.

[0165] [Exposure time is the same between short-exposure RAW data and long-exposure RAW data]

[0166] Under this condition, a RAW data slice is generated by calculating the pixel average of every four adjacent pixels of the same color component (see Figure 5). One RAW data piece will be quantized, and because HDR synthesis processing will not be performed, brightness feature classification is performed using the RAW data generated by calculating the pixel average, and quantization is performed using the quantization table based on the classification result. Fig.19A An exemplary setting of the quantization table is shown in FIG. Q0 indicates a quantization table for ensuring image quality in a dark portion, Q1 indicates a quantization table for ensuring image quality in a medium portion, and Q2 indicates a quantization table for ensuring image quality in a bright portion. The size relationship between the quantization tables is as follows.

[0167] Q0 <Q1<Q2

[0168] In this way, a quantization table according to brightness based on visual characteristics is set.

[0169] [When the exposure time is different between the short-exposure RAW data and the long-exposure RAW data and the short-exposure RAW data has the correct exposure]

[0170] Under this condition, the image data is separated into short-exposure RAW data and long-exposure RAW data (see 4A to 4D ).exist Fig.19B An exemplary setting of the quantization table is shown in FIG. Since there is a possibility that occlusion shadows may appear in the short-exposure RAW data obtained by capturing with correct exposure, the long-exposure RAW data obtained by capturing under overexposed conditions is used to extend the DR in the dark portion. The quantization tables indicated by Q1 and Q2 in the figure are the same as those in FIG. Fig.19A Here, two quantization tables indicated by Q3 and Q4 are newly added. Q3 indicates a table that aims to suppress the amount of code generated under the assumption that the area is an area where the synthesis ratio in the HDR synthesis process is small and the influence on the image quality is small. Q4 indicates a quantization table that aims to allocate a large amount of code in order to extend DR in a dark part where an occlusion shadow is likely to appear in the HDR synthesis process. The size relationship between the quantization tables is as follows.

[0171] Q0≤Q4 <Q1<Q2<Q3

[0172] or,

[0173] Q0 <Q4≤Q1<Q2<Q3

[0174] The quantization parameter in Q4 is greater than or equal to the quantization parameter in Q0 and less than the quantization parameter in Q2. In this way, it becomes possible to ensure the image quality after HDR synthesis processing by setting a quantization table in which the quantization parameter is relatively small for a dark portion whose synthesis ratio is large in the long-exposure RAW data, in addition to the quantization table according to brightness based on visual characteristics. On the other hand, since a quantization table in which the quantization parameter is large is set for a dark portion whose synthesis ratio is small in the short-exposure RAW data and for a medium portion and a bright portion in the long-exposure RAW data, the amount of data can be effectively reduced without reducing the image quality after HDR synthesis processing.

[0175] [When the exposure time is different between the short-exposure RAW data and the long-exposure RAW data and the long-exposure RAW data has the correct exposure]

[0176] Also, under this condition, the image data is separated into RAW data consisting of short-exposure pixels and RAW data consisting of long-exposure pixels (see 4A to 4D ).exist Fig.19C An exemplary setting of the quantization table is shown in FIG. As described above, overexposed highlights may appear in long-exposure pixels used for capturing with correct exposure, and short-exposure pixels used for capturing with insufficient exposure are used to extend the DR in bright parts. The quantization table indicated by Q0, Q1, and Q3 is similar to Fig.19A and Fig.19B Here, the quantization table indicated by Q5 is newly added. Q5 indicates a quantization table that aims to allocate a large code amount to extend DR in a bright part where overexposed highlights may appear in HDR synthesis processing. The size relationship between the quantization tables is as follows.

[0177] Q0 <Q1≤Q5<Q2<Q3

[0178] or,

[0179] Q0 <Q1<Q5≤Q2<Q3

[0180] The quantization parameter in Q5 is less than or equal to the quantization parameter in Q2, and greater than the quantization parameter in Q0. In this way, it becomes possible to ensure the image quality after HDR synthesis processing by setting a quantization table in which the quantization parameter is relatively small with respect to a bright portion in which the synthesis ratio is large in the short-exposure RAW data, in addition to the quantization table according to brightness based on visual characteristics. On the other hand, since a quantization table in which the quantization parameter is large is set with respect to a bright portion in which the synthesis ratio is small in the long-exposure RAW data and a dark portion and a medium portion in the short-exposure RAW data, the amount of data can be effectively reduced without reducing the image quality after HDR synthesis processing.

[0181] Next, we will use FIG. 20A to FIG. 20C In the present embodiment, in order to make the description easier to understand, it is assumed that lev=1, and the luminance feature quantity is calculated using sub-band data constituting RAW data obtained by capturing at an exposure time that will be the correct exposure.

[0182] It is assumed that the calculation and quantization processing of the brightness feature quantity is performed in units of one coefficient, and an operation is performed so as to uniquely determine the quantization table to be applied to the corresponding RAW data slices of different exposure times according to the brightness feature quantity of the corresponding coefficient (please refer to Figure 19 for details).

[0183] In the present embodiment, an operation mode in which capturing is performed while changing the exposure time for each pixel is referred to as an HDR mode, and an operation mode in which capturing is performed without changing the exposure time is referred to as a normal mode. As described above, in the HDR mode, the horizontal size and vertical size of the RAW data to be recorded are doubled relative to the normal mode (refer to FIG. 4A to FIG. 5 ), and therefore the amount of data to undergo quantization processing is different between these modes.

[0184] In step S1201, the controller 108 determines whether the operation mode of the digital camera 100 is the HDR mode. If it is determined to be the HDR mode, the process proceeds to step S1202, and if not, the process proceeds to step S1219.

[0185] In step S1202, the controller 108 determines whether the short-exposure RAW data has the correct exposure. If the short-exposure RAW data has the correct exposure, the process advances to step S1203, and if not, the process advances to step S1211.

[0186] In step S1203, the controller 108 calculates the brightness feature quantity using the short exposure sub-band data with correct exposure. The magnitude of the coefficient of the 1LL sub-band of the G1 (green) component is used as the brightness feature quantity. This is because the LL sub-band is a DC component and can therefore represent brightness, and the reason for using the G1 component is because human visual characteristics are sensitive to changes in the G component, and the G1 component is important visual information.

[0187] In step S1204, the controller 108 determines whether the region of interest is a dark portion based on the magnitude relationship between the luminance feature amount calculated in step S1203 and a predetermined threshold. If determined to be a dark portion, the process proceeds to step S1205, and if not, the process proceeds to step S1206.

[0188] In step S1205 , the controller 108 determines the quantization table for the color component subband data pieces constituting the short-exposure RAW data to be Q3 and determines the quantization table for the color component subband data pieces constituting the long-exposure RAW data to be Q4 and performs quantization processing.

[0189] In step S1206, the controller 108 determines whether the region of interest is a medium portion based on the magnitude relationship between the luminance feature amount calculated in step S1203 and a predetermined threshold. If determined to be a medium portion, the process proceeds to step S1207, and if not, the process proceeds to step S1208.

[0190] In step S1207, the quantization table for the color component subband data pieces constituting the short-exposure RAW data is determined to be Q1, and the quantization table for the color component subband data pieces constituting the long-exposure RAW data is determined to be Q3, and quantization processing is performed.

[0191] In step S1208 , the controller 108 determines that the quantization table for the color component subband data pieces constituting the short-exposure RAW data is Q2 , and the quantization table for the color component subband data pieces constituting the long-exposure RAW data is Q3 , and performs quantization processing.

[0192] In step S1209, the controller 108 determines whether the quantization process is completed for all sub-band data slices in the image plane. If the quantization process is completed for all sub-band data slices, the process is ended, and if not, the process proceeds to step S1210.

[0193] In step S1210, the controller 108 updates the quantization process target coefficient. After completing the updating of the coefficient, the controller 108 returns the process to step S1203.

[0194] In step S1211, the controller 108 calculates the luminance feature quantity using the long exposure sub-band data having the correct exposure. The magnitude of the coefficient of the 1LL sub-band of the G1 component is used as the luminance feature quantity, similarly to step S1203.

[0195] In step S1212, the controller 108 determines whether the region of interest is a dark portion based on the magnitude relationship between the luminance feature amount calculated in step S1211 and a predetermined threshold. If it is determined to be a dark portion, the process proceeds to step S1213, and if not, the process proceeds to step S1214.

[0196] In step S1213, the quantization table for the color component subband data pieces constituting the short-exposure RAW data is determined to be Q3, and the quantization table for the color component subband data pieces constituting the long-exposure RAW data is determined to be Q0, and quantization processing is performed.

[0197] In step S1214, the controller 108 determines whether the region of interest is a middle portion based on the magnitude relationship between the luminance feature amount calculated in step S1211 and a predetermined threshold. If determined to be a middle portion, the process proceeds to step S1215, and if not, the process proceeds to step S1216.

[0198] In step S1215 , the controller 108 determines that the quantization table for the color component subband data pieces constituting the short-exposure RAW data is Q3 , and the quantization table for the color component subband data pieces constituting the long-exposure RAW data is Q1 , and performs quantization processing.

[0199] In step S1216, the controller 108 determines that the quantization table for the color component subband data pieces constituting the short-exposure RAW data is Q5, and the quantization table for the color component subband data pieces constituting the long-exposure RAW data is Q3, and performs quantization processing.

[0200] In step S1217, the controller 108 determines whether the quantization process is completed for all sub-band data slices in the image plane. If the quantization process is completed for all sub-band data slices, the process is ended, and if not, the process proceeds to step S1218.

[0201] In step S1218, the controller 108 updates the quantization process target coefficient. After completing the updating of the coefficient, the controller 108 returns the process to step S1211.

[0202] In step S1219, since it is determined that it is the normal mode, the controller 108 calculates the luminance feature quantity using the sub-band data obtained by performing frequency conversion on the RAW data generated by performing the addition averaging. The magnitude of the coefficient of the 1LL sub-band of the G1 component obtained by performing the addition averaging is used as the luminance feature quantity, similar to step S1203.

[0203] In step S1220, the controller 108 determines whether the region of interest is a dark portion based on the magnitude relationship between the luminance feature amount calculated in step S1219 and a predetermined threshold. If it is determined to be a dark portion, the process proceeds to step S1221, and if not, the process proceeds to step S1222.

[0204] In step S1221, the controller 108 determines the quantization table used for the color component sub-band data piece constituting the RAW data to be Q0, and performs quantization processing.

[0205] In step S1222, the controller 108 determines whether the region of interest is a middle portion based on the magnitude relationship between the luminance feature amount calculated in step S1219 and a predetermined threshold. If it is determined to be a middle portion, the process proceeds to step S1223, and if not, the process proceeds to step S1224.

[0206] In step S1223, the controller 108 determines the quantization table used for the color component sub-band data piece constituting the RAW data to be Q1, and performs quantization processing.

[0207] In step S1224, the controller 108 determines the quantization table used for the color component sub-band data piece constituting the RAW data to be Q2, and performs quantization processing.

[0208] In step S1225, the controller 108 determines whether the quantization process is completed for all sub-band data slices in the image plane. If the quantization process is completed for all sub-band data slices, the process ends, and if not, the process proceeds to step S1226.

[0209] In step S1226, the controller 108 updates the quantization process target coefficient. After completing the updating of the coefficient, the controller 108 returns the process to step S1219.

[0210] As described above, in the present embodiment, the separation unit 102 separates the RAW data into data pieces of corresponding exposure times, eliminates the level difference between pixels of different exposure times, and thereby suppresses high-frequency components, and thus, the amount of recorded data of the RAW data can be reduced. In addition, since the weighting of the quantization parameter is performed in consideration of the synthesis ratio while anticipating the HDR synthesis processing after the development processing, the amount of recorded data of the RAW data can be effectively reduced.

[0211] Note that, in the present embodiment, an example has been described in which brightness features are classified into three stages, but the number of stages in which classification is performed is not limited to this, and the number of stages may be further increased. FIG. 20A to FIG. 20C In the flowchart shown in , the following configuration has been described: based on the feature amount calculated using the 1LL subband data of the G1 component, the quantization table for the subband data piece of the other color components is uniquely determined. However, an operation may be performed so that the quantization table is determined by calculating the feature amount independently for each color component.

[0212] In addition, an example has been described in which the calculation unit of the feature amount and the processing unit of quantization are each coefficient, but the processing unit may be a coefficient block (two coefficients or more).

[0213] In addition, FIG. 20A to FIG. 20CThe example of lev=1 is described in the flowchart shown in , but in the case of lev=2 or more, the horizontal size and vertical size of the subband data are different according to the lev. Therefore, the calculation unit of the feature amount cannot be the same as the processing unit of quantization. It is assumed that the feature amount is calculated in units of one coefficient of the 2LL subband data of lev=2. In this case, due to the characteristics of subsampling of the frequency resolution, it is necessary to set a 2×2 block as the processing unit for quantization of the subband data of lev=1.

[0214] In addition, the size of the coefficient of the 1LL subband data is used as a brightness feature, but the feature representing the brightness can be generated using other methods, such as using pixels or average values ​​calculated from the coefficients of the 1LL subband data of multiple color components, and there is no limitation to the above method.

[0215] In addition, the channel transformation unit 1601 has been described using an example in which transformation to four channels is performed for each color element of R, G1, G2, and B in a Bayer arrangement, but the color elements of R, G1, G2, and B can also be transformed to four channels using the following transformation formulas 29 to 32.

[0216] Y=(R+G1+G2+B) / 4 Formula 29

[0217] C0=RB Formula 30

[0218] C1=(G0+G1) / 2-(R+B) / 2 Formula 31

[0219] C2=G0-G1 Formula 32

[0220] The above conversion formula illustrates an exemplary conversion to four channels consisting of brightness and color difference. In this case, if control is performed so that the quantization parameter of the brightness component is reduced and the quantization parameter of the other color difference components is increased, the coding efficiency is improved by utilizing human visual characteristics. It is noted that the number of channels and the conversion method are not limited to this.

[0221] (Sixth embodiment)

[0222] Next, the sixth embodiment will be described. In the sixth embodiment, the method of determining a quantization table for a feature area in which the synthesis ratio is large for RAW data that does not have correct exposure is different from the method of the first embodiment. In the first embodiment, a fixed pattern prepared in advance is set as a quantization table for a feature area in which the synthesis ratio is large for RAW data that does not have correct exposure. Therefore, if each exposure RAW data is obtained by capturing at an exposure time that is extremely different from the exposure time of the correct exposure, the most appropriate quantization table cannot be selected according to the brightness, and there is a possibility that the image quality is degraded, or the amount of code is increased unnecessarily. Therefore, in this embodiment, a method for further improving the encoding efficiency will be described. In this method, for a feature area in which the synthesis ratio is large, brightness feature determination is also performed on the RAW data that does not have correct exposure, and the most appropriate quantization table according to the feature is selected. Note that the configuration of the image capture device of the sixth embodiment is similar to that of the fifth embodiment, and therefore its description is omitted.

[0223] FIG. 21A to FIG. 21C The quantization processing procedure of this embodiment is shown in . The difference from the fifth embodiment is that processing steps S1301 to S1312 are added. Descriptions of processing steps similar to those of the fifth embodiment are omitted, and only the differences will be described.

[0224] In step S1301, the controller 108 calculates a luminance feature quantity using the long exposure sub-band data with overexposure. The magnitude of the coefficient of the 1LL sub-band of the G1 component is used as the luminance feature quantity, similarly to the fifth embodiment.

[0225] In step S1302, the controller 108 determines whether the region of interest is a dark portion based on the magnitude relationship between the luminance feature amount calculated in step S1301 and a predetermined threshold. If it is determined to be a dark portion, the process proceeds to step S1303, and if not, the process proceeds to step S1304.

[0226] In step S1303 , the controller 108 determines that the quantization table for the color component sub-band data piece constituting the long-exposure RAW data is Q0 , and performs quantization processing.

[0227] In step S1304, the controller 108 determines whether the region of interest is a medium portion based on the magnitude relationship between the luminance feature amount calculated in step S1301 and a predetermined threshold. If it is determined to be a medium portion, the process proceeds to step S1305, and if not, the process proceeds to step S1306.

[0228] In step S1305 , the controller 108 determines that the quantization table for the color component sub-band data piece constituting the long-exposure RAW data is Q1 , and performs quantization processing.

[0229] In step S1306 , the controller 108 determines that the quantization table for the color component sub-band data piece constituting the long-exposure RAW data is Q2 , and performs quantization processing.

[0230] In step S1307, the controller 108 calculates the luminance feature quantity using the short exposure sub-band data with underexposure. The magnitude of the coefficient of the 1LL sub-band of the G1 component is used as the luminance feature quantity, similarly to the fifth embodiment.

[0231] In step S1308, the controller 108 determines whether the region of interest is a dark portion based on the magnitude relationship between the luminance feature amount calculated in step S1307 and a predetermined threshold. If determined to be a dark portion, the process proceeds to step S1309, and if not, the process proceeds to step S1310.

[0232] In step S1309, the controller 108 determines that the quantization table for the color component sub-band data piece constituting the short-exposure RAW data is Q0, and performs quantization processing.

[0233] In step S1310, the controller 108 determines whether the region of interest is a middle portion based on the magnitude relationship between the luminance feature amount calculated in step S1307 and a predetermined threshold. If determined to be a middle portion, the process proceeds to step S1311, and if not, the process proceeds to step S1312.

[0234] In step S1311, the controller 108 determines the quantization table for the color component sub-band data piece constituting the short-exposure RAW data to be Q1, and performs quantization processing.

[0235] In step S1312, the controller 108 determines the quantization table used for the color component sub-band data pieces constituting the short-exposure RAW data to be Q2, and performs quantization processing.

[0236] As described above, since the most appropriate quantization table is set according to brightness for RAW data obtained by capturing performed with an exposure time that does not have correct exposure, encoding efficiency can be further improved.

[0237] Other embodiments

[0238] The (one or more) embodiments of the present invention may also be implemented by reading out and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a "non-transitory computer-readable storage medium") to perform one or more functions of the above (one or more) embodiments and / or a system or device including one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions of the above (one or more) embodiments, and by a method performed by a computer of the system or device, for example, reading and executing computer executable instructions from a storage medium to perform one or more functions of the above (one or more) embodiments and / or controlling one or more circuits to perform one or more functions of the above (one or more) embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read out and execute computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, one or more of the following: a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage device of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), a Blu-ray disk (BD) TM ), flash memory devices, memory cards, etc.

[0239] Other embodiments

[0240] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0241] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A coding device, comprising: a generating unit configured to generate a plurality of RAW data slices for corresponding exposure times from RAW data obtained from an image sensor, wherein the image sensor is capable of performing shooting at different exposure times for each pixel; as well as an encoding component, the encoding component being used to encode the plurality of RAW data slices generated by the generating component, The different exposure times for each pixel are composed of two types, namely a first exposure time and a second exposure time longer than the first exposure time, and wherein the generating unit generates differential RAW data from a difference between RAW data obtained by applying a gain to RAW data corresponding to one of the first exposure time and the second exposure time and RAW data corresponding to the other of the first exposure time and the second exposure time, and the encoding unit encodes the differential RAW data and the RAW data corresponding to the other of the first exposure time and the second exposure time.

2. The encoding device according to claim 1, wherein: The generating section generates RAW data in a Bayer arrangement structure for each exposure time.

3. The encoding device according to claim 1, in, The generating means generates two RAW data slices corresponding to the first exposure time and two RAW data slices corresponding to the second exposure time, and The encoding component encodes the two RAW data slices corresponding to the first exposure time and the two RAW data slices corresponding to the second exposure time.

4. The encoding device according to claim 1, in, The generating unit generates a RAW data slice corresponding to the first exposure time by adding a plurality of pixel data slices of the first exposure time, and generates a RAW data slice corresponding to the second exposure time by adding a plurality of pixel data slices of the second exposure time, and The encoding component encodes the one RAW data slice corresponding to the first exposure time and the one RAW data slice corresponding to the second exposure time.

5. The encoding device according to claim 4, wherein: The generating unit generates the one RAW data slice corresponding to the first exposure time and the one RAW data slice corresponding to the second exposure time by calculating an arithmetic average of a plurality of pixel data slices.

6. The encoding device according to claim 1, wherein: The generating section applies a gain to RAW data corresponding to one of the first exposure time and the second exposure time so as to be closer to RAW data corresponding to the other of the first exposure time and the second exposure time.

7. The encoding device according to claim 1, wherein: The generating section generates the plurality of RAW data pieces by calculating an arithmetic average of signals of pixels having the same exposure time and the same color existing nearby.

8. The encoding device according to claim 1, further comprising a control unit for controlling the exposure time of each pixel of the image sensor, in, The generating section generates the plurality of RAW data pieces if the exposure time of each pixel changes, and generates one RAW data piece if the exposure time of each pixel does not change.

9. The encoding device according to claim 8, wherein: If the exposure time of each pixel does not change, the generating section generates one RAW data piece by calculating the average value of image data of pixels of the same color existing nearby.

10. The encoding device according to claim 1, wherein: The encoding section determines a quantization parameter of RAW data corresponding to one of the first exposure time and the second exposure time using a quantization parameter of RAW data corresponding to one of the first exposure time and the second exposure time as a reference.

11. The encoding device according to claim 10, wherein: The encoding section determines a quantization parameter of the RAW data corresponding to the second exposure time using a quantization parameter of the RAW data corresponding to the first exposure time as a reference.

12. The encoding device according to claim 1, in, The generating unit generates first RAW data of a first exposure time and second RAW data of a second exposure time different from the first exposure time, The encoding device further includes a quantization component for quantizing the first RAW data and the second RAW data, The encoding section encodes the first RAW data and the second RAW data which have been quantized by the quantization section, and The quantization section determines a quantization parameter for the first RAW data and a quantization parameter for the second RAW data for corresponding areas classified by brightness of the first RAW data.

13. The encoding device according to claim 12, wherein: The quantization component determining which of the first RAW data and the second RAW data has a correct exposure, If the first RAW data has a correct exposure, determining a quantization parameter for the first RAW data and a quantization parameter for the second RAW data for a corresponding area classified by brightness of the first RAW data, and If the second RAW data has a correct exposure, a quantization parameter for the first RAW data and a quantization parameter for the second RAW data are determined for corresponding areas classified by brightness of the second RAW data.

14. The encoding device according to claim 13, wherein: The first exposure time is shorter than the second exposure time.

15. The encoding device according to claim 14, wherein: The quantization section determines a quantization parameter of a region classified as dark as a larger quantization parameter than a quantization parameter of a region classified as bright for the first RAW data, and determines a quantization parameter of a region classified as bright as a larger quantization parameter than a quantization parameter of a region classified as dark for the second RAW data.

16. The encoding device according to claim 13, in, When capturing is performed by the image sensor with the same exposure time instead of performing capturing with different exposure times for each pixel, The generating means obtains third RAW data obtained by averaging pixel data pieces of pixels of the same color existing nearby, The quantization unit determines a quantization parameter for the third RAW data for a corresponding area classified by brightness of the third RAW data, and quantizes the third RAW data, and The encoding section encodes the quantized third RAW data.

17. The encoding device according to claim 16, in, the quantization means determines that if the first RAW data has a correct exposure, a quantization parameter for the first RAW data of an area classified as bright is a quantization parameter corresponding to a quantization parameter for an area classified as bright in the third RAW data, and The quantization section determines that if the second RAW data has a correct exposure, a quantization parameter for the second RAW data of an area classified as dark is a quantization parameter corresponding to a quantization parameter for an area classified as dark in the third RAW data.

18. The encoding device according to claim 17, in, the quantization means determines that if the first RAW data has a correct exposure, a quantization parameter for the first RAW data of an area classified as bright is a quantization parameter larger than a quantization parameter to be used in the third RAW data, and The quantization section determines that a quantization parameter for the second RAW data of an area classified as dark is a quantization parameter greater than a quantization parameter to be used in the third RAW data if the second RAW data has a correct exposure.

19. The encoding device according to claim 17, wherein: The quantization component determines that if the first RAW data has a correct exposure, a quantization parameter for the second RAW data of an area classified as dark is a quantization parameter that is greater than or equal to a quantization parameter to be used for an area classified as dark in the third RAW data and is less than a quantization parameter to be used for an area classified as bright in the third RAW data.

20. The encoding device according to claim 16, wherein: The quantization component determines that if the second RAW data has a correct exposure, a quantization parameter for the first RAW data of an area classified as bright is a quantization parameter that is smaller than or equal to a quantization parameter to be used for an area classified as bright in the third RAW data and is larger than a quantization parameter to be used for an area classified as dark in the third RAW data.

21. The encoding device according to claim 12, wherein: The quantization section performs classification by brightness for each area of ​​the first RAW data or the second RAW data using a first threshold value for determining whether it is a dark portion and a second threshold value for determining whether it is a bright portion.

22. The encoding device according to claim 12, wherein: A region is a square area larger than one pixel.

23. An image capturing device comprising: Image sensors that can control the exposure time of each pixel; as well as The encoding device according to claim 1.

24. A coding method comprising: generating a plurality of RAW data slices for corresponding exposure times from RAW data obtained from an image sensor capable of performing photographing at different exposure times for each pixel; as well as encoding the plurality of RAW data slices generated in the generating, The different exposure times for each pixel are composed of two types, namely a first exposure time and a second exposure time longer than the first exposure time, and wherein, in the generating, differential RAW data is generated from the difference between RAW data obtained by applying a gain to RAW data corresponding to one of the first exposure time and the second exposure time and RAW data corresponding to the other of the first exposure time and the second exposure time, and wherein, in the encoding, the differential RAW data and the RAW data corresponding to the other of the first exposure time and the second exposure time are encoded.

25. A non-transitory computer-readable storage medium storing a program for causing a computer to execute the steps of the encoding method according to claim 24.

Images