Image processing device, image processing method, imaging device, and control method thereof

Abstract

To provide an image processing device, an image processing method, and an imaging apparatus capable of efficiently generating a composite image while suppressing deterioration in image quality.SOLUTION: Encoding to reduce a data amount of a plurality of pieces of image data different in exposure is applied, and encoded data is transmitted, which reduces an amount of transmission data. After the encoded data is decoded, a composite image is generated. Before the encoding and after the decoding, level conversion of the image data and its inversion are applied. An input / output characteristic to use for the level conversion is set in accordance with a composition ratio of the pieces of image data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and an imaging device, and in particular to a technique for generating a composite image. [Background technology]

[0002] There is known a technique for generating an image with an expanded dynamic range (HDR (High Dynamic Range) image) by synthesizing a plurality of images captured with different exposure amounts. Patent Document 1 discloses a technique for generating an HDR image by synthesizing a plurality of image signals generated by applying different gains to an image signal obtained by one capture, instead of capturing an image a plurality of times with different exposure amounts. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-186910 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 has the advantage of being able to generate an HDR image with a single capture. However, because signals are read out from each pixel of the image sensor multiple times, the amount of data to be processed within one frame transmission period increases, which may strain the memory and data bus bandwidth. On the other hand, simply reducing the amount of data of the image signal may cause a decrease in the image quality of the HDR image.

[0005] An object of the present invention is to provide an image processing device, an image processing method, and an imaging device that can efficiently generate a composite image while suppressing deterioration in image quality. [Means for solving the problem]

[0006] The above-mentioned object can be achieved by an image processing device having an acquisition means for acquiring multiple image data with different exposure amounts, a conversion means for applying input / output characteristics to the multiple image data to perform level conversion, an encoding means for applying coding to reduce the data amount of the level-converted multiple image data, a decoding means for decoding the multiple encoded image data, an inverse conversion means for applying the inverse characteristics of the input / output characteristics to the decoded multiple image data, and a synthesis means for generating synthetic image data from the multiple image data output by the inverse conversion means, wherein the input / output characteristics are set according to the synthesis ratio of the multiple image data in the synthesis means. Effect of the Invention

[0007] According to the present invention, it is possible to provide an image processing device, an image processing method, and an imaging device that can efficiently generate a composite image while suppressing deterioration in image quality. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of a functional configuration of an imaging device 10 according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the functional configuration of an imaging unit 20 according to the first embodiment; [Diagram 3] FIG. 3 is a diagram showing an example of a circuit configuration from a unit pixel 201 to a horizontal transfer circuit 205 in FIG. [Figure 4] FIG. 1 shows a pixel array of a pixel unit 200 according to a first embodiment, exposure time settings, and image output formats in normal mode and HDR mode. [Diagram 5] FIG. 1 is a block diagram showing an example of the functional configuration of a synthesis unit 80 according to an embodiment of the present invention. [Figure 6] FIG. 1 is a diagram showing the operation of gain correction circuits 801 and 802 according to an embodiment of the present invention. [Figure 7] FIG. 13 is a diagram showing an example of a synthesis ratio in a synthesis circuit 804 according to the embodiment; [Figure 8] FIG. 1 is a block diagram showing an example of a functional configuration of an encoding unit 40 according to an embodiment. [Figure 9] A block diagram showing an example of the functional configuration of a decoding unit 60 according to an embodiment. [Figure 10] FIG. 1 is a block diagram showing an example of a functional configuration of a level conversion section 30 according to an embodiment of the present invention. [Figure 11] FIG. 13 is a diagram showing an example of input / output characteristics of a level conversion section 30 according to an embodiment; [Figure 12] FIG. 13 is a diagram showing an example of input / output characteristics of a level conversion section 30 according to an embodiment; [Figure 13] 1 is a flowchart showing the operation of the level conversion section 30 according to the embodiment. [Figure 14] A block diagram of a level inverse conversion unit 70 according to an embodiment of the present invention. [Figure 15] FIG. 13 is a diagram showing an example of input / output characteristics of a level inverse conversion section 70 according to an embodiment; [Figure 16] FIG. 11 is a circuit diagram showing a configuration example of a column amplifier of a column amplifier unit 203 according to a second embodiment. [Figure 17] FIG. 13 is a diagram showing an example of operation timing of the imaging unit 20 according to the third embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0010] In the following, the present invention will be described in terms of a case where the present invention is implemented in an imaging device such as a digital camera. However, an imaging device is one example of an image processing device capable of implementing the present invention. An image processing device capable of implementing the present invention may be any electronic device capable of processing an image signal obtained from an imaging element. Such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0011] ●(First embodiment) <Overall imaging device> FIG. 1 is a block diagram showing an example of a functional configuration of an imaging device 10 capable of implementing the present invention. Each functional block of the imaging device 10 may be realized by software, hardware, or a combination thereof, except for parts that can be realized only by hardware, such as an imaging element, a memory 50, and a recording medium 110. For example, the functional block may be realized by dedicated hardware such as an ASIC. Also, the functional block may be realized by a processor such as a CPU executing a program stored in a memory. Note that a plurality of functional blocks may be realized by a common configuration (for example, one ASIC). Also, hardware that realizes a part of the function of a certain functional block may be included in hardware that realizes another functional block.

[0012] The imaging unit 20 includes an optical system that generates an optical image of a subject, and an imaging element that converts the optical image generated by the optical system into a group of electrical signals. The optical system may be detachable or fixed. The imaging element may be, for example, a CCD or CMOS image sensor, and has a pixel section or pixel array in which pixels having photoelectric conversion regions are arranged in a matrix, and a peripheral circuit that supplies a control signal to the pixel section. In this embodiment, the imaging unit 20 has an A / D conversion function and outputs a group of pixel signals (RAW data) in a digital format. A more detailed configuration of the imaging unit 20 will be described later.

[0013] The level conversion section 30 converts the levels of the individual pixel signals constituting the image data supplied from the imaging section 20 in accordance with set input / output characteristics.

[0014] The encoding unit 40 encodes the image data supplied from the level conversion unit 30, and generates encoded data having a smaller amount of data than the image data. The encoding unit 40 will be described in detail later. The encoded data is stored in the memory 50. The memory 50 functions as a buffer memory for the encoded data.

[0015] The decoding unit 60 reads out and decodes the encoded data stored in the memory 50. The decoding unit 60 outputs image data obtained by decoding to the level inverse conversion unit 70. By reducing the amount of image data through encoding, it is possible to reduce the bandwidth usage of the transmission path between the encoding unit 40 and the decoding unit 60. Therefore, even when it is necessary to transmit image data for multiple frames in one shooting, it is possible to prevent the bandwidth of the transmission path from becoming congested. In addition, by providing the memory 50 between the encoding unit 40 and the decoding unit 60, it is possible to reduce the processing speed required of the decoding unit 60.

[0016] The level inverse conversion unit 70 converts the levels of the individual pixel signals constituting the image data using input / output characteristics that are the inverse of the input / output characteristics applied by the level conversion unit 30. Combining these two input / output characteristics results in a linear input / output characteristic in which the input and output are equal. The level inverse conversion unit 70 outputs the level-converted image data to the synthesis unit 80.

[0017] The synthesis unit 80 synthesizes the image data for a plurality of frames supplied from the level inverse conversion unit 70 based on a synthesis ratio characteristic described later, to generate synthetic image data. The synthesis unit 80 outputs the generated synthetic image data to the development processing unit 90.

[0018] The composite image data output by the composition unit 80 is RAW data. The development processing unit 90 applies a development process to the composite image data to generate image data for display and / or recording. More specifically, the development processing unit 90 applies a color interpolation process also called a DeBayer (demosaic) process, a white balance adjustment process, a color space conversion process, a noise suppression process, a correction of distortion and aberration caused by an optical system, an enlargement / reduction process, and the like. The development processing unit 90 generates image data represented by luminance and color difference, for example, YCbCr4:2:2. The development processing unit 90 may also generate an evaluation value from the generated image data that the control unit 120 uses to adjust exposure and focus, or detect a characteristic area for face recognition, and the like.

[0019] In the present embodiment, the configuration has been described in which image data before development processing is synthesized, but the synthesis unit 80 may be disposed after the development processing unit 90, and image data after development processing may be synthesized.

[0020] The recording processing unit 100 encodes the image data for recording output from the development processing unit 90 as necessary, and generates an image data file including the encoded data and its associated information. The encoding applied by the recording processing unit 100 can be any encoding method adopted as a standard, such as JPEG, JPEG2000, H.264, or HEVC / H.265.

[0021] Moreover, the recording processing unit 100 records the generated image data file on a recording medium 110. The recording medium 110 may be a semiconductor memory card, an SSD, a hard disk drive, etc. The recording medium 110 may be removable, may be built-in, or may be externally connected so as to be capable of communication.

[0022] The display processing unit 130 scales the image data for display output from the development processing unit 90 or the image data for recording to a display size and displays it on the display unit 140. The display processing unit 130 also combines image data representing various information such as setting values ​​of the imaging device 10 with image data obtained by imaging and displays the combined image data on the display unit 140, and displays GUI screens such as menu screens on the display unit 140.

[0023] The display unit 140 is, for example, a liquid crystal display. The display unit 140 may be a touch display. By displaying moving images on the display unit 140 in real time, the display unit 140 can function as an EVF.

[0024] The operation unit 121 is a general term for input devices such as keys, buttons, switches, and dials that allow a user to input instructions to the imaging device 10. When the display unit 140 is a touch display, the touch panel portion constitutes the operation unit 121.

[0025] The control unit 120 has a CPU, a ROM, and a RAM. Although omitted in Fig. 1, the control unit 120 is communicably connected to each functional block of the imaging device 10. The control unit 120 loads a program stored in the ROM into the RAM and executes it with the CPU, thereby controlling the operation of each functional block of the imaging device 10 and realizing the functions of the imaging device 10. The control unit 120 monitors the operation unit 121, and when it detects an operation of the operation unit 121, it executes an operation corresponding to the operation.

[0026] <Imaging unit 20> The imaging element of the imaging unit 20 can control the exposure time (charge accumulation time) on a pixel-by-pixel basis. Therefore, by appropriately arranging pixel groups controlled with different exposure times, it is possible to output image data for a plurality of frames whose exposure periods overlap and whose exposure amounts are different within a given one frame period. For example, by using an imaging element having a color filter with a Quad Bayer array, two adjacent 2×2 pixels of the same color can be controlled with a first exposure time and the remaining two with a second exposure time, thereby obtaining image data for two frames with different exposure times. In this case, if image data is generated by treating 2×2 pixels of the same color as one pixel in normal shooting, the resolution of the image data for two frames with different exposure times will be equal to the resolution of image data obtained in normal shooting.

[0027] Note that the method of controlling the exposure time on a pixel-by-pixel basis to obtain a plurality of frames of images with overlapping exposure periods and different exposure amounts within one frame period is not particularly limited. By synthesizing the image data for the plurality of frames thus obtained for each corresponding pixel between the frames, an HDR image can be obtained with a single exposure.

[0028] The imaging unit 20 has a number of operating modes, including a normal mode in which the exposure time for all pixels is the same, and an HDR mode in which the exposure time differs for each pixel group.

[0029] In normal mode, the imaging unit 20 outputs one frame of image data having a predetermined exposure amount in one shooting. On the other hand, in HDR mode, the imaging unit 20 outputs multiple frames of image data having different exposure amounts in one shooting. The exposure time in each operation mode may be determined by the control unit 120 or may be set by the user via the operation unit 121.

[0030] In this embodiment, in HDR mode, low-exposure image data is obtained from a pixel group controlled by a first exposure time, and high-exposure image data is obtained from a pixel group controlled by a second exposure time (>first exposure time).

[0031] If image data of the same resolution is obtained in both normal mode and HDR mode, the amount of image data output from the imaging unit 20 in one frame period is twice as much in HDR mode as in normal mode. Therefore, one or more downstream functional blocks from the level conversion unit 30 to the development processing unit 90 may be provided. Also, three or more types of exposure times may be used in HDR mode.

[0032] 2 is a block diagram showing an example of the configuration of the imaging unit 20. In the pixel unit 200, a plurality of unit pixels 201 including a photodiode or the like are arranged in a matrix. The operation of the unit pixels 201 arranged in the pixel unit 200 is controlled by a control signal supplied by a timing control unit 208. Therefore, the exposure time of the unit pixels 201 is also controlled by the timing control unit 208.

[0033] The vertical scanning circuit 202 sequentially selects the unit pixels 201 row by row. The vertical scanning circuit 202 controls the rows to be selected and the order in which they are selected so that pixel signals can be read out from all the rows during one frame period.

[0034] The column amplifier unit 203 has a plurality of column amplifiers arranged for each column of the pixel unit 200, and amplifies the voltage of the read-out pixel signal. By amplifying the signal in the column amplifier unit 203, it is possible to increase the pixel signal level relative to noise generated in the downstream column ADC unit 204, thereby improving the S / N ratio.

[0035] The column ADC unit 204 has a plurality of A / D conversion units arranged for each column of the pixel unit 200, and converts pixel signals output from the corresponding column amplifiers into digital pixel signals. The digital pixel signals are sequentially read out by the horizontal transfer circuit 205 and input to the signal processing unit 206.

[0036] The signal processing unit 206 adds an offset value to the pixel signal or applies a shift operation (multiplication and division) to impart a digital gain to the pixel signal. If the pixel unit 200 has a light-shielded pixel (optical black) region, a black level clamp operation may be performed based on a pixel signal obtained from the optical black region. The signal processing unit 206 also has a parallel / serial conversion function, and converts a signal that is handled as a multi-bit parallel signal by the signal processing unit 206 into a serial signal. The signal processing unit 206 converts this serial signal into an LVDS signal or the like, and outputs it to an external device or the level conversion unit 30.

[0037] The memory 207 temporarily holds the digital pixel signals that are read out from the pixel unit 200 and processed by the column amplifier unit 203 , the column ADC unit 204 , and the signal processing unit 206 .

[0038] The timing control unit 208 supplies an operating clock signal and a timing signal to each block in the imaging unit 20 to control the overall operation of the imaging unit 20. In addition, the timing control unit 208 can control the exposure time of the unit pixel 201 by control signals 216 and 217, which will be described later.

[0039] Next, how pixel signals are read from the unit pixels 201 to the horizontal transfer circuit 205 will be described with reference to FIG. FIG. 3 is a diagram showing an example of the circuit configuration of the unit pixel 201 and the configuration existing between the unit pixel 201 and the horizontal transfer circuit 205. As shown in FIG.

[0040] The photoelectric conversion unit 213 is a photodiode, and converts light incident on the unit pixel 201 into an electric charge. The transfer switch 215 is controlled to be turned on or off by a control signal 216, and transfers the electric charge generated in the photoelectric conversion unit 213 to the charge holding unit 218 when in the on state.

[0041] The control signals 216 and 217 are signals that control the on / off of the transfer switch 215 of the unit pixel 201. One of the control signals 216 and 217 is supplied to the unit pixel 201. In the normal mode, the timing control unit 208 controls the exposure times of all the unit pixels 201 with the same timing and period by making the control signals 216 and 217 the same signal. On the other hand, in the HDR mode, the timing control unit 208 differentiates the timing of the control signals 216 and 217 so that the period from the start of the charge accumulation period to the start of charge transfer is different. This makes it possible to differentiate the charge accumulation period, i.e., the exposure time, of the unit pixel 201 to which the control signal 216 is supplied and the unit pixel 201 to which the control signal 217 is supplied.

[0042] The charge holding unit 218 is a floating diffusion that holds the charge transferred from the photoelectric conversion unit 213 through the transfer switch 215 and converts it into a voltage. When a pixel row including the unit pixel 201 is selected by a vertical scanning circuit, a pixel amplifier 219 amplifies the voltage level of the charge holding unit 218 and outputs it to a corresponding column amplifier in the column amplifier unit 203 through a vertical signal line (column signal line) 220. A current control unit 221 controls the current of the vertical signal line 220.

[0043] The column ADC unit 204 is provided with an A / D conversion unit 222, memories 223 and 224, and a subtractor 225 for each column. The A / D conversion unit 222 converts the voltage value output from the column amplifier into a digital signal having a value within a range according to a predetermined bit depth (e.g., 8 bits (0 to 255), 12 bits (0 to 4096), etc.). The memories 223 and 224 temporarily hold the digital signal output from the A / D conversion unit 222.

[0044] The memory 223 holds a digital signal in which a noise signal from a readout circuit unit (a circuit from the charge holding unit 218 to the A / D conversion unit 222) is added to a pixel signal based on the charges read out from the photoelectric conversion unit 213. Meanwhile, the memory 224 holds a noise signal from the readout circuit unit. This noise signal corresponds to the pixel signal read out when the charge holding unit 218 is reset. The subtractor 225 subtracts the output of the memory 224 from the output of the memory 223, and outputs the subtraction result to the horizontal transfer circuit 205 as a pixel signal.

[0045] Next, the pixel array structure of the pixel section 200 will be described with reference to FIG. 4(a). In this embodiment, the pixel section 200 of the image sensor is provided with color filters in a Quad Bayer array. That is, one pixel in a general Bayer array is divided into four pixels, two pixels in the horizontal direction and two pixels in the vertical direction. It can also be said to be a Bayer array with a unit of 2×2 pixels. Hereinafter, a pixel provided with a red (R) color filter is called a red pixel or an R pixel. The same applies to pixels provided with green (Gr or Gb) and blue (B) color filters.

[0046] An example of how to control the exposure time of a unit pixel of the pixel section 200 having the pixel array shown in Fig. 4(a) in HDR mode will be described with reference to Fig. 4(b). In the following description, an orthogonal coordinate system is set with the horizontal direction of the pixel section 200 as the x-axis and the vertical direction as the y-axis, and the position of a pixel is expressed by a combination of an x-coordinate (column number) and a y-coordinate (row number). In addition, the coordinates of the pixel in the upper left corner are (1,1).

[0047] In FIG. 4(b), the gray pixels are controlled to have a first exposure time, and the white pixels are controlled to have a second exposure time. Here, the second exposure time is longer than the first exposure time. Therefore, the gray pixels are called short-exposure pixels, and the white pixels are called long-exposure pixels. In this embodiment, as shown in FIG. 4(b), one of the control signals 216 and 217 is supplied to each unit pixel so that pixels with the same exposure time are not adjacent in the horizontal and vertical directions. As a result, adjacent 2×2 pixels of the same color are controlled to have the same exposure time for every two pixels located diagonally.

[0048] For example, the coordinate settings for the four R pixels at the top left are as follows: R(1,1): Short exposure pixel R(2,1): Long exposure pixel R(1,2): Long exposure pixel R(2,2): Short-exposure pixel

[0049] In this way, short exposure pixels and long exposure pixels are set alternately in each column, and short exposure pixels and long exposure pixels are set alternately in each row. If we trace only the short exposure pixels in the y direction in the first and second columns, we get the following from the top: In the first row, the first column is the short exposure pixel, In the second row, the second column is a short exposure pixel, In the third row, the first column is a short-exposure pixel, In the fourth row, the second column is a short exposure pixel In this way, the short-time exposure pixels are arranged in a zigzag pattern.

[0050] Similarly, if we trace only the long exposure pixels in the y direction in the first and second columns, we get the following from the top: In the first row, the second column is a long exposure pixel, In the second row, the first column is a long exposure pixel, In the third row, the second column is a long exposure pixel, In the 4th row, the 1st column is a long exposure pixel In this way, the long exposure pixels are arranged in a zigzag pattern.

[0051] Next, the arrangement of pixel signals output from the pixel unit 200 having the pixel arrangement shown in Fig. 4(a) in normal mode will be described with reference to Fig. 4(c). When the imaging unit 20 operates in normal mode, the exposure time is controlled so that it is equal for all unit pixels 201. In normal mode, the imaging unit 20 outputs each of the 2x2 same-color pixels shown as gray diamonds in Fig. 4(c)a as one pixel. For example, the pixel signal output by the imaging unit 20 shown in Fig. 4(c)b can be derived as the average value of four pixels of the same color as follows:

number

[0052] As shown in FIG. 4(c)b, the arrangement of R pixels, Gr pixels, Gb pixels, and B pixels in the image data output from the imaging unit 20 is the same as the arrangement of image data output from an imaging element having a typical Bayer array type color filter.

[0053] Next, the arrangement of pixel signals output from the pixel unit 200 having the pixel arrangement shown in FIG. 4(a) in HDR mode will be described with reference to FIG. 4(d). When the imaging unit 20 operates in HDR mode, the exposure time is controlled so that short-exposure pixels and long-exposure pixels are arranged as shown in FIG. 4(b). In HDR mode, the imaging unit outputs two pixels having the same exposure time among 2×2 same-color pixels as one pixel. Specifically, pixel signals read from two short-exposure pixels are output as pixel signals for one pixel of a low-exposure image, and pixel signals read from two long-exposure pixels are output as pixel signals for one pixel of a high- and low-exposure image.

[0054] For example, the low exposure image data shown in FIG. 4(d)b can be derived as the average value of pixel signals of short exposure pixels of the same color as follows:

number

[0055] On the other hand, the high exposure image data shown in FIG. 4(d)b can be derived as the average value of pixel signals of long exposure pixels of the same color as follows.

number

[0056] In this embodiment, in HDR mode, the pixel levels of the low exposure image data and high exposure image data output from the imaging unit 20 are converted using input / output characteristics prepared for each image data by the level conversion unit 30. However, since the input / output characteristics used by the level conversion unit 30 are determined based on the blending ratio in the blending unit 80, a method for generating a blended image (HDR image) in the blending unit 80 will first be described.

[0057] <Synthesizer 80> 5 is a block diagram showing an example of the functional configuration of the synthesis unit 80. The synthesis unit 80 has a gain correction circuit 801 for low exposure images, a gain correction circuit 802 for high exposure images, a selector 803, and a synthesis circuit 804.

[0058] When generating an HDR image by synthesizing images with different exposures, the images to be synthesized are often composed of an image obtained by shooting with a proper exposure and an image obtained by shooting with overexposure and / or underexposure with respect to the proper exposure. In this case, the image obtained by shooting with a proper exposure is used as a reference image, and the remaining images are used to expand the dynamic range by improving the gradation of the reference image. Note that it is not essential to use the image obtained by shooting with a proper exposure. In this embodiment, for ease of explanation and understanding, it is assumed that two frames of images including an image obtained by shooting with a proper exposure (properly exposed image) are synthesized. Below, a synthesis operation will be described for the cases where a high exposure image is a properly exposed image and where a low exposure image is a properly exposed image.

[0059] In the imaging device 10 of the present invention, the control unit 120 performs automatic exposure control in response to a user operation or a predetermined operation mode, and determines the shooting conditions (shutter speed, aperture value, ISO sensitivity, etc.) for obtaining a proper exposure. The synthesis unit 80 can know through the control unit 120 which of the image data to be synthesized is an image obtained by shooting with a proper exposure.

[0060] The gain correction circuits 801 and 802 are used to equalize the exposure amounts of the low-exposure image and the high-exposure image. The operation of the gain correction circuits 801 and 802 will be described in detail with reference to FIG.

[0061] 6(a) shows an example of the operation of the gain correction circuits 801 and 802 when the high exposure image is the appropriate exposure image. Here, an example is shown in which blown-out highlights occur in the appropriate exposure but do not occur in the underexposure. In such a case, the gain correction circuit 801 applies a gain greater than 1 to the low exposure image data (gain up) to estimate pixel values ​​in a part 901 where blown-out highlights occur in the high exposure image.

[0062] Furthermore, by combining the low exposure image data and the high exposure image data with increased gain in the combining circuit 804, it is possible to generate HDR image data having gradation even in the brightness region where whiteout occurs at the appropriate exposure. In this case, it is not necessary to apply gain to the high exposure image data in the gain correction circuit 802 (or a gain of 1 may be applied).

[0063] On the other hand, Fig. 6(b) shows an example of the operation of the gain correction circuits 801 and 802 when the underexposed image is an appropriate exposure image. Here, an example is shown in which blackout occurs with appropriate exposure but does not occur with overexposure. In such a case, the gain correction circuit 802 applies a gain of less than 1 to the high exposure image data (gain down), thereby estimating pixel values ​​in a portion 902 where blackout occurs in the underexposed image.

[0064] Furthermore, by combining the high-exposure image data with the low-exposure image data that has been subjected to gain reduction in the combining circuit 804, it is possible to generate HDR image data having gradation even in the luminance region where blackout occurs in the low-exposure image. In this case, it is not necessary to apply gain to the low-exposure image data in the gain correction circuit 801 (or a gain of 1 may be applied).

[0065] The gains applied by the gain correction circuits 801 and 802 can be determined based on the ratio of the exposure amount or exposure time of the low exposure image and the high exposure image. For example, assume that high exposure image data is acquired by shooting with proper exposure, and the exposure amount ratio of the low exposure image and the high exposure image is 1:4. In this case, the gain applied to the low exposure image data by the gain correction circuit 801 is 4. As a result, the pixel value of the low exposure image data is multiplied by four, and the exposure amount can be made equal to that of the high exposure image data. The ratio of the exposure amount or exposure time can be controlled by a parameter set by the control unit 120 to the timing control unit 208. Therefore, the control unit 120 can set a gain value for the gain correction circuit 801 or 802 to which the gain should be applied, depending on the ratio of the exposure amount or exposure time.

[0066] A selector 803 outputs to a synthesis circuit 804 the pixel values ​​of one of the image data obtained by shooting with the correct exposure, out of the low exposure image data and the high exposure image data output by the gain correction circuits 801 and 802 .

[0067] The synthesis circuit 804 synthesizes the low exposure image data and the high exposure image data output by the gain correction circuits 801 and 802 for each pixel to generate data of a synthesis image (HDR image). Here, an example of a synthesis method in the synthesis circuit 804 will be described with reference to FIG.

[0068] 7, the horizontal axis indicates the range of values ​​that each pixel of the appropriate exposure image can take, and the vertical axis indicates the blending ratio (0 to 1) of low exposure image data or high exposure image data in the blended image. In this embodiment, the bit depth of pixel values ​​is 12 bits (0 to 4095).

[0069] In this embodiment, a blending ratio is set for each image data so that the sum of the blending ratios is always 1. That is, when two frames of image data are blended as in this embodiment, the pixel value hdr_img of the HDR image output from the blending circuit 804 can be derived by the following formula (13). α×main_img+(1-α)×sub_img = hdr_img ...(13) Here, α indicates the blending ratio of image data obtained by shooting with proper exposure, main_img indicates the pixel value of the image with proper exposure, and sub_img indicates the pixel value of image data obtained by shooting with underexposure or overexposure.

[0070] FIG. 7(a) shows an example of the composition ratio of high-exposure image data and low-exposure image data when the appropriate exposure image is a high-exposure image. In the example of FIG. 7(a), as the pixel value of the high-exposure image data becomes higher (brighter), the composition ratio of the high-exposure image data becomes lower and the composition ratio of the low-exposure image data becomes higher in stages. In addition, in low-luminance and medium-luminance areas, the composition ratio of the high-exposure image data obtained by shooting with appropriate exposure is higher than the composition ratio of the low-exposure image data. On the other hand, in high-luminance areas, the composition ratio of the high-exposure image data is lower than the composition ratio of the low-exposure image data. This is because the high-exposure image data, which has an appropriate exposure, is used preferentially in low-luminance and medium-luminance areas, and the gradation is improved by using both low-exposure image data for high-luminance areas where whiteout occurs.

[0071] FIG. 7(b) shows an example of the composition ratio of the high-exposure image data and the low-exposure image data when the appropriate exposure image is the low-exposure image data. In the example of FIG. 7(b), as the pixel value of the low-exposure image data becomes higher (brighter), the composition ratio of the low-exposure image data becomes higher and the composition ratio of the high-exposure image data becomes lower stepwise. In addition, in the medium and high luminance areas, the composition ratio of the low-exposure image data obtained by shooting with the appropriate exposure is higher than the composition ratio of the high-exposure image data. On the other hand, in the low luminance area, the composition ratio of the low-exposure image data is lower than the composition ratio of the high-exposure image data. This is because the low-exposure image data, which has an appropriate exposure, is used preferentially in the medium and high luminance areas, and the gradation is improved by using both high-exposure image data for the low luminance parts where blackout occurs.

[0072] For ease of explanation and understanding, an example in which the blending ratio is controlled in stages is shown in Fig. 7. Basically, the blending ratio can be determined by any method that expands the dynamic range by improving the gradation in the luminance region where whiteout and blackout occur in the reference image data using other image data.

[0073] In this way, the blending ratio in the blending unit 80 is determined according to the exposure amount and pixel value of the image to be blended. Also, image data with a high blending ratio has a large effect on the image quality of the blended image. Therefore, by applying level conversion in consideration of the blending ratio of the image data in the level conversion unit 30, the degradation of image quality due to coding in the coding unit 40 is reduced. Details of the level conversion will be described later.

[0074] <Encoding unit 40> 8 is a block diagram showing an example of the functional configuration of the encoding unit 40. In this embodiment, the encoding unit 40 applies lossy encoding accompanied by quantization to each piece of pixel data constituting the image data.

[0075] The adder 1101 calculates the difference between pixel data to be coded (target pixel data) and data of a predicted pixel generated by a predicted pixel generator 1106. The adder 1101 outputs the calculated difference to a quantizer 1102.

[0076] The quantization unit 1102 quantizes the difference input from the adder 1101 in accordance with a quantization parameter notified by the code amount control unit 1105. More specifically, the quantization unit 1102 truncates the least significant bits of the input difference by the number of bits indicated by the quantization parameter. For example, when the value indicated by the quantization parameter is "4", the quantization unit 1102 quantizes the difference by truncating the least significant 4 bits of the input difference. Then, the quantization unit 1102 outputs the quantized difference to the variable length coding unit 1104 and the inverse quantization unit 1103.

[0077] The variable length coding unit 1104 performs variable length coding on the value (quantized difference) input from the quantization unit 1102. The variable length coding may be an entropy coding method that assigns a shorter code length to a smaller value, thereby improving compression efficiency. The variable length coding unit 1104 outputs the generated variable length code to a code string generation unit 1108. The variable length coding unit 1104 also outputs the code length of the generated variable length code to a code amount control unit 1105.

[0078] The code amount control unit 1105 refers to the code length input from the variable length coding unit 1104, determines a quantization parameter for each specified number of pixels so that the code amount approaches a predetermined target compression rate, and outputs the quantization parameter to the quantization unit 1102.

[0079] The inverse quantization unit 1103 inverse quantizes the quantized difference and outputs the result to an adder 1107. The adder 1107 adds the inverse quantized value and a predicted pixel value for the pixel of interest, and stores the result in a predicted pixel generation unit 1106.

[0080] The predicted pixel generation unit 1106 has a memory capable of holding a plurality of predicted pixel values ​​output by the adder 1107. The predicted pixel generation unit 1106 then selects and outputs the predicted pixel value of a pixel located in the prediction direction relative to the pixel of interest, for example, the pixel adjacent to the left.

[0081] The code string generation unit 1108 outputs to the memory 50 an encoded stream including the variable-length code output from the variable-length encoding unit 1104 and parameters required for decoding the code, such as the quantization parameter output from the code amount control unit 1105.

[0082] The encoded stream stored in the memory 50 is output from the memory 50 to a decoding unit 60 and decoded.

[0083] <Decoding unit 60> 9 is a block diagram showing an example of the functional configuration of the decoding unit 60. In this embodiment, the decoding unit 60 decodes the coded stream stored in the memory 50.

[0084] The demultiplexer 1201 analyzes the encoded stream and separates it into encoded data and a quantization parameter. The demultiplexer 1201 also outputs the encoded data to a variable length decoding unit 1202 and outputs the quantization parameter to an inverse quantization unit 1203.

[0085] The variable length decoding unit 1202 decodes the input encoded data and outputs the decoded data to the inverse quantization unit 1203. The inverse quantization unit 1203 inverse quantizes the decoded value input from the variable length decoding unit 1202 in accordance with the quantization parameter input from the separation unit 1201 , and outputs the result to an adder 1204 .

[0086] The predicted pixel generation unit 1205 has a capacity for holding, for example, a plurality of decoded pixels, and holds the output of the adder 1204. It selects a pixel in the prediction direction from the pixel of interest, for example, the adjacent pixel to the left, and outputs it as a predicted pixel.

[0087] The adder 1204 adds the value input from the inverse quantization unit 1203 and the predicted pixel value input from the predicted pixel generation unit 1205, and outputs the result as decoded pixel data.

[0088] <Level conversion section 30> 10 is a block diagram showing an example of the functional configuration of the level conversion unit 30. It is assumed that high exposure image data and low exposure image data are input to the level conversion unit 30 from the imaging unit 20 in a time-division multiplexed manner, for example, on a pixel row basis.

[0089] The selector 1301 selects whether the input image data is to be input to the first level converter 1303 or the second level converter 1304, based on the signal of the exposure amount selection signal 1302. Note that in a configuration in which processing is applied to the same data by both the first level converter 1303 and the second level converter 1304, the selector 1301 is not necessary. An exposure amount selection signal 1302 is input from the image capturing section 20 and indicates whether high exposure amount image data or low exposure amount image data is being input.

[0090] When the exposure amount selection signal 1302 indicates high exposure amount image data, the selector 1301 outputs the input image data to a first level conversion unit 1303. When the exposure amount selection signal 1302 indicates low exposure amount image data, the selector 1301 outputs the input image data to a second level conversion unit 1304.

[0091] The first level conversion unit 1303 and the second level conversion unit 1304 convert and output values ​​of each pixel data constituting the input image data by applying input / output characteristics that have been set. The input / output characteristics applied by the first level conversion unit 1303 and the second level conversion unit 1304 can be set and changed by the control unit 120. The first level conversion section 1303 applies input / output characteristics for high exposure image data, and the second level conversion section 1304 applies input / output characteristics for low exposure image data.

[0092] Based on the exposure amount selection signal 1302, the selector 1305 selects whether the output of the first level converter 1303 or the second level converter 1304 is to be output to the downstream encoding unit 40. When the exposure amount selection signal 1302 indicates high exposure amount image data, the selector 1305 selects the output of the first level conversion unit 1303. When the exposure amount selection signal 1302 indicates low exposure amount image data, the selector 1305 selects the output of the second level conversion unit 1304.

[0093] 11(a) and 11(b) are diagrams showing examples of input / output characteristics applied by the first level conversion unit 1303 and the second level conversion unit 1304 when the high exposure image is the appropriate exposure image. The horizontal axis indicates the input pixel value (12 bits), and the vertical axis indicates the output pixel value (12 bits).

[0094] As described above, the difference input to the quantization unit 1102 of the encoding unit 40 is truncated by the number of bits indicated by the quantization parameter. Therefore, the pixel data decoded by the decoding unit 60 contains quantization noise equivalent to the information of the lower bits truncated by the quantization unit.

[0095] When the high-exposure image is a proper exposure image, as shown in Fig. 7(a), the smaller the pixel value of the high-exposure image, the higher the synthesis ratio of the high-exposure image data, and therefore the greater the impact on the image quality of the synthesized image. Therefore, when the high-exposure image is a proper exposure image, the image quality of the synthesized HDR image is improved by reducing the quantization noise for pixel data having a relatively small value in the high-exposure image.

[0096] 11(a), the slope of the input / output characteristics of the first level conversion unit 1303 is set to be greater than 1 in an area where the pixel value is small. In this embodiment, input / output characteristics having a slope greater than 1 are used for approximately the lowest 18.75% (3 / 16) of pixel values ​​(0 to 767) within the possible pixel value range (0 to 4095).

[0097] On the other hand, when the high-exposure image is the appropriate exposure image, as shown in Fig. 7(a), the larger the pixel value of the high-exposure image is, the lower the synthesis ratio of the high-exposure image data is, and therefore the effect on the image quality of the synthesized image is smaller. Therefore, when the high-exposure image is the appropriate exposure image, the amount of code for pixel data having a relatively large value of the high-exposure image is reduced, thereby improving the coding efficiency.

[0098] 11(a), the slope of the input / output characteristics of the first level conversion unit 1303 is set to be smaller than 1 in a section corresponding to a range of large pixel values. In this embodiment, input / output characteristics having a slope smaller than 1 in a section corresponding to the top 62.5% (10 / 16) of pixel values ​​(1536 to 4095) of the possible range of pixel values ​​(0 to 4095) are used.

[0099] The input / output characteristics shown in Figure 11(a) are as follows: In the section corresponding to the first range of pixel values ​​(0 to 255), which corresponds to the lowest 6.25% (1 / 16), the slope is 4. In the next section corresponding to the pixel values ​​in the second range (256 to 767), which corresponds to 12.5% ​​(2 / 16), the slope is 2. In the next section corresponding to the third range of pixel values ​​(768 to 1535), which corresponds to 18.75% (3 / 16), the slope is 1. In the last section, which corresponds to the fourth range of pixel values ​​(1536 to 4095), which is 62.5% (10 / 16), the slope is 1 / 2. has.

[0100] Note that the number of divisions into the range of possible input pixel values, the size of each range, and the slopes assigned to the sections of the input / output characteristics corresponding to each range are merely examples and can be changed. However, the input / output characteristics are set so that the section corresponding to the range of pixel values ​​including the minimum pixel value has a slope greater than 1, and the slope does not decrease as the input pixel value increases.

[0101] When an input pixel value is 141, the first level conversion unit 1303 using the input / output characteristics shown in FIG. 11(a) multiplies the input pixel value by a slope of 4 times and outputs a pixel value 564. The level inverse conversion unit 70 applies the inverse characteristic of the input / output characteristics of the level conversion unit 30 to the pixel value after decoding. Therefore, the encoded data of the pixel value 564 is multiplied by 1 / 4 after decoding. By multiplying the pixel value by n (n>1) before encoding and by 1 / n after decoding, the quantization noise is also multiplied by 1 / n. In this way, by converting the pixel level before encoding and after decoding, the image quality of the HDR image after synthesis can be improved.

[0102] Furthermore, when the input pixel value is 2838, the first level conversion unit 1303 outputs a pixel value of 3467 (=(2838-1536) / 2+2816) according to the input / output characteristics of FIG. 11(a). If pixel level conversion is not performed and the pixel value of the predicted pixel is 1536, the difference value between the input pixel and the predicted pixel is 2838-1536=1302, and the difference value "1302" is encoded. On the other hand, when pixel level conversion is performed as in this embodiment, the pixel value "1536" of the predicted pixel is also pixel level converted to 2816 like the input pixel, so the difference value between the input pixel and the predicted pixel is 3467-2816=651, and the difference value "651" is encoded.

[0103] The encoding unit 40 assigns a shorter code length to the input pixel value as the difference between the input pixel value and the predicted pixel value becomes smaller. Therefore, the code length of the difference value "651" when pixel level conversion is performed is shorter than the code length of the difference value "1302" when pixel level conversion is not performed. In this way, for input pixel values ​​in a range where the slope of the input / output characteristics is smaller than 1, the amount of code can be reduced by performing pixel level conversion.

[0104] Note that pixel values ​​to which a slope smaller than 1 is applied in the level conversion unit 30 have a slope larger than 1 applied in the level inverse conversion unit, so that the quantization noise after decoding is increased. However, by applying a slope smaller than 1 to pixel values ​​with a small blending ratio in the level conversion unit 30, the impact of the increase in quantization noise on the image quality of the HDR image after blending is small. Therefore, the effect of reducing the amount of code is prioritized.

[0105] Next, the input / output characteristics of the second level conversion unit 1304 shown in Fig. 11(b) will be described. When the high exposure image is an appropriate exposure image, as shown in Fig. 7(b), the larger the pixel value of the low exposure image, the higher the synthesis ratio of the low exposure image data, and therefore the greater the impact on the image quality of the synthesized image. Therefore, when the high exposure image is an appropriate exposure image, the image quality of the synthesized HDR image is improved by reducing the quantization noise for pixel data having a relatively large value of the low exposure image.

[0106] 11B, the input / output characteristics applied by the second level conversion unit 1304 have a slope greater than 1 in a section corresponding to a range of large pixel values. In this embodiment, input / output characteristics having a slope greater than 1 are used for a section corresponding to the top 31.25% (5 / 16) of pixel values ​​(2816 to 4096) of the possible range of pixel values ​​(0 to 4095).

[0107] Also, the coding efficiency is improved by reducing the amount of code for pixel data having a relatively small value of a low exposure image, which reduces the blending ratio. More specifically, as shown in Fig. 11(b), the slope of the input / output characteristics of the second level conversion unit 1304 is made smaller than 1 in a section corresponding to an area where the pixel value is small. In this embodiment, input / output characteristics having a slope smaller than 1 are used in a section corresponding to the lowest 50% (8 / 16) pixel value range (0 to 2047) of the possible pixel value range (0 to 4095).

[0108] The input / output characteristics shown in Figure 11(b) are as follows: In the section corresponding to the first range of pixel values ​​(0 to 1023), which corresponds to the lowest 25% (4 / 16), the slope is 1 / 4. In the section corresponding to the second range of pixel values ​​(1024 to 2047), which corresponds to the next 25% (4 / 16), the slope is 1 / 2. In the next section, which corresponds to the third range of pixel values ​​(2048 to 2815), which is equivalent to 18.75% (3 / 16), the slope is 1. In the last 31.25% (5 / 16) of the fourth range of pixel values ​​(2816 to 4095), the slope is 2. has.

[0109] By applying such input / output characteristics, the second level conversion unit 1304 reduces quantization noise in the low exposure image in a pixel value range that has a large effect on the image quality of the HDR image after synthesis, improving the image quality of the HDR image. On the other hand, the second level conversion unit 1304 reduces the code amount in a pixel value range that has a small effect on the image quality of the HDR image after synthesis.

[0110] 12(a) and 12(b) are diagrams showing examples of input / output characteristics applied by the first level conversion unit 1303 and the second level conversion unit 1304 when the low exposure image is the appropriate exposure image, similar to Fig. 11. The horizontal axis indicates the input pixel value (12 bits), and the vertical axis indicates the output pixel value (12 bits).

[0111] When the low-exposure image is the appropriate exposure image, as shown in Fig. 7(b), the smaller the pixel value of the high-exposure image is, the higher the synthesis ratio of the high-exposure image data is, and therefore the greater the impact on the image quality of the synthesized image. Therefore, when the low-exposure image is the appropriate exposure image, the image quality of the synthesized HDR image is improved by reducing the quantization noise for pixel data with relatively small values ​​of the high-exposure image.

[0112] This is the same as when the high exposure image is the appropriate exposure image. Therefore, in this embodiment, as shown in FIG. 12(a), the first level conversion unit 1303 applies the same input / output characteristics as when the high exposure image is the appropriate exposure image. However, as shown in FIG. 7(a) and FIG. 7(b), the change in the blending ratio of the high exposure image differs between when the appropriate exposure image is a high exposure image and when the appropriate exposure image is a low exposure image. Therefore, the input / output characteristics applied by the first level conversion unit 1303 may be different between when the appropriate exposure image is a high exposure image and when the appropriate exposure image is a low exposure image.

[0113] For example, the input / output characteristics applied by the first level conversion unit 1303 can be set so that the slope of the section corresponding to the range of pixel values ​​in which the synthesis ratio of the high exposure image is a predetermined value (e.g., a value exceeding 50% or 0.5) exceeds 1. In this way, different input / output characteristics are applied when the appropriate exposure image is a high exposure image and when it is a low exposure image, and the image quality of the HDR image after synthesis can be improved.

[0114] In this way, even when the low exposure image is the appropriate exposure image, the first level conversion unit 1303 performs level conversion on the high exposure image using input / output characteristics in which a section corresponding to a pixel value range in which the blending ratio is equal to or greater than a first predetermined value has a slope greater than 1. This makes it possible to improve the image quality of the combined HDR image. Also, the first level conversion unit 1303 performs level conversion on the high exposure image using input / output characteristics in which a section corresponding to a pixel value range in which the blending ratio is less than a second predetermined value has a slope less than 1. This makes it possible to reduce the amount of code for pixel values ​​that have little effect on the image quality of the combined HDR image.

[0115] Next, the input / output characteristics applied to the low-exposure image by the second level conversion unit 1304 will be described. The main purpose of photographing a low-exposure image at a proper exposure and compositing it with a high-exposure image is to prevent the occurrence of black crush in areas with small pixel values ​​in the low-exposure image. In addition to the fact that blown-out highlights are unlikely to occur in low-exposure images, humans have visual characteristics that make it difficult to notice changes in bright areas. Therefore, in areas with large pixel values, degradation in image quality due to quantization noise is less noticeable. In light of this, it cannot be said that the influence on the image quality of the composite image of a large pixel value range in which the composition ratio of a low-exposure image photographed at a proper exposure is high is necessarily large.

[0116] Therefore, the second level conversion unit 1304 of this embodiment uses input / output characteristics as shown in Fig. 12(b). Specifically, in the section corresponding to the range in which the pixel value is from the minimum value (0) to less than a predetermined value, the input / output characteristics have a slope that decreases as the pixel value decreases, and has a maximum slope of 1. Also, the input / output characteristics have a slope smaller than 1 in the section corresponding to the range from the predetermined pixel value to the maximum pixel value.

[0117] More specifically, the input / output characteristics shown in FIG. In the section corresponding to the first range of pixel values ​​(0 to 1023), which corresponds to the lowest 25% (4 / 16), the slope is 1 / 4. In the section corresponding to the second range of pixel values ​​(1024 to 2047), which corresponds to the next 25% (4 / 16), the slope is 1 / 2. In the next section, which corresponds to the third range of pixel values ​​(2048 to 2815), which is equivalent to 18.75% (3 / 16), the slope is 1. In the last section, which corresponds to the fourth range of pixel values ​​(2816 to 4095), which is 31.25% (5 / 16), the slope is 1 / 2. has.

[0118] In this way, when the low-exposure image is the appropriate exposure image, the low-exposure image data is level-converted by applying input / output characteristics in which the corresponding sections of both the pixel value range with a low blending ratio and the pixel value range with a high blending ratio have a slope smaller than 1. This makes it possible to reduce the amount of code for both pixel values ​​in dark areas where the blending ratio is low and the effect on the image quality of the blended image is small, and pixel values ​​in bright areas where the blending ratio is high and quantization noise is not noticeable.

[0119] In the input / output characteristics shown in FIG. 12, the number of divisions of the range of the input pixel value, the size of each range, and the slope assigned to each range are merely examples and can be changed. However, the input / output characteristics applied to the high exposure image have a slope greater than 1 in a section corresponding to a range of pixel values ​​including the minimum pixel value, and the slope is set so that the slope does not decrease with an increase in the input pixel value. In addition, the input / output characteristics applied to the low exposure image have a slope not exceeding 1 in a section corresponding to a range from the minimum pixel value to a predetermined pixel value, and the slope is set so that the slope does not decrease with an increase in the input pixel value. In addition, a slope smaller than 1 is set in a section corresponding to a range from a predetermined pixel value to a maximum pixel value. However, in the case where image quality is prioritized over code reduction in the range from the predetermined pixel value to the maximum pixel value where the composition ratio is high, a slope of 1 or more may be set in the section of the input / output characteristics corresponding to that range of pixel values.

[0120] Next, the operation of the level conversion unit 30 will be described with reference to the flowchart shown in Fig. 13. In this embodiment, since the input / output characteristics used by the level conversion unit 30 are divided into four sections, the input pixel values ​​and the corresponding output pixel values ​​corresponding to the section points of the sections are defined as follows.

[0121] x_min: The minimum value of the input pixel x_max: maximum value of input pixels x_thn: Input pixel value corresponding to the segment point (n is an integer equal to or greater than 1) The corresponding output values ​​are y_min, y_max, and y_thn.

[0122] For example, in the input / output characteristics shown in Fig. 11(a), x_min = 0 x_max = 4095 x_th1 = 256 x_th2 = 768 x_th3 = 1536 y_min = 0 y_max = 4095 y_th1 = 1024 y_th2 = 2048 y_th3 = 2816 it is as follows.

[0123] Also, the section corresponding to x_min ≤ x < x_th1 is defined as the first section, the section corresponding to x_th1 ≤ x < x_th2 is defined as the second section, the section corresponding to x_th2 ≤ x < x_th3 is defined as the third section, and the section corresponding to x_th3 ≤ x ≤ x_max is defined as the fourth section.

[0124] Also, let the slope in the m-th section of the input / output characteristics be αm (m is an integer from 1 to 4). For example, in the input / output characteristics shown in Fig. 11(a), α1 = 4 α2 = 2 α3 = 1 α4 = 1 / 2 it is as follows.

[0125] In the following description, it is described as being executed by the level conversion unit 30. However, the flowchart shown in Fig. 13 is executed for each input pixel data in each of the first level conversion unit 1303 and the second level conversion unit 1304.

[0126] In S1601, the level conversion unit 30 determines whether the value (pixel value) of the input pixel data is less than x_th1. If the level conversion unit 30 determines that the pixel value is less than x_th1, it executes S1604; otherwise, it executes S1602.

[0127] In S1602, the level conversion unit 30 determines whether or not the value of the input pixel data (pixel value) is less than x_th2. If it is determined that the pixel value is less than x_th2, the level conversion unit 30 executes S1605, and if not, executes S1603.

[0128] In S1603, the level conversion unit 30 determines whether or not the value of the input pixel data (pixel value) is less than x_th3. If it is determined that the pixel value is less than x_th3, the level conversion unit 30 executes S1606, and if not, executes S1607.

[0129] In S1604, the level conversion section 30 calculates the parameters x_th, y_th, and α in the calculation formula for the output level y (described later) as follows: x_th=0 y_th=0 α=α1 After that, the level conversion section 30 executes S1608.

[0130] In step S1605, the level conversion unit 30 x_th=x_th1 y_th=y_th1 α=α2 After that, the level conversion section 30 executes S1608.

[0131] In S1606, the level conversion unit 30 x_th=x_th2 y_th=y_th2 α=α3 After that, the level conversion section 30 executes S1608.

[0132] In S1607, the level conversion unit 30 x_th=x_th3 y_th=y_th3 α=α4 After that, the level conversion section 30 executes S1608.

[0133] In S1608, the level conversion section 30 calculates an output value y for a pixel value x according to the following equation (14). y=(x-x_th)×α+y_th ...(14)

[0134] The level conversion section 30 calculates the output value y, and ends the level conversion process for the target pixel data.

[0135] When the high exposure amount image data and the low exposure amount image data are read in parallel, the respective image data can be directly input to the first level converter 1303 and the second level converter 1304. In this case, the first level converter 1303 and the second level converter 1304 may execute the above-mentioned level conversion operation in parallel. The level-converted pixel data may be output in parallel, or may be multiplexed by the selector 1305 and output as serial data.

[0136] Note that the output value y for the input pixel value x may be obtained by a method different from the method described in Fig. 13. For example, the input / output characteristics may be implemented as a table, and the output value y may be obtained by referring to the table using the input pixel value x. Furthermore, if the input / output characteristics for the entire range of the input pixel value x can be expressed as a function of the input pixel value x, the output value y may be obtained by substituting the input pixel value x into the function.

[0137] <Level inverse conversion section 70> 14 is a block diagram showing an example of the configuration of the level inverse conversion unit 70. It is assumed that the high exposure image data and the low exposure image decoded by the decoding unit 60 are input to the level inverse conversion unit 70 in a time-division multiplexed manner, for example, on a row-by-row basis.

[0138] Based on the signal of the exposure amount selection signal 1702, the selector 1701 selects whether the input image data is to be input to the first level inverse transform unit 1703 or the second level inverse transform unit 1704. Note that in a configuration in which processing is applied to the same data by both the first level inverse transform unit 1703 and the second level inverse transform unit 1704, the selector 1701 is not necessary. The exposure amount selection signal 1702 is input from the decoding unit 60 and indicates whether high exposure amount image data or low exposure amount image data is being input.

[0139] When the exposure amount selection signal 1702 indicates high exposure amount image data, the selector 1701 outputs the input image data to the first level inverse conversion unit 1703. When the exposure amount selection signal 1702 indicates low exposure amount image data, the selector 1701 outputs the input image data to the second level inverse conversion unit 1704.

[0140] The first level inverse conversion unit 1703 and the second level inverse conversion unit 1704 apply the input / output characteristics that have been set for each piece of pixel data that constitutes the input image data, convert the value, and output the converted value. The input / output characteristics that the first level inverse conversion unit 1703 and the second level inverse conversion unit 1704 apply can be set and changed by the control unit 120.

[0141] The first level inverse transform unit 1703 applies input / output characteristics for high exposure image data, and the second level inverse transform unit 1704 applies input / output characteristics for low exposure image data. The input / output characteristics applied by the first level inverse transform unit 1703 are the inverse characteristics of the input / output characteristics applied by the first level transform unit 1303 before encoding. The input / output characteristics applied by the second level inverse transform unit 1704 are the inverse characteristics of the input / output characteristics applied by the second level transform unit 1304 before encoding. It is assumed here that a certain input / output characteristic and its inverse characteristic have a relationship such that when both characteristics are applied to an input value, the output value becomes equal to the input value.

[0142] A selector 1705 selects, based on the exposure amount selection signal 1702, whether the output of the first level inverse conversion section 1703 or the second level inverse conversion section 1704 is to be output to the downstream synthesis section 80. When the exposure amount selection signal 1702 indicates high exposure amount image data, the selector 1705 selects the output of the first level inverse conversion unit 1703. When the exposure amount selection signal 1702 indicates low exposure amount image data, the selector 1705 selects the output of the second level inverse conversion unit 1704.

[0143] As explained above, by using input / output characteristics in the level conversion unit 30 and the level inverse conversion unit 70 that take into account the type of appropriate exposure image and the blending ratio, it is possible to suppress deterioration in image quality of the composite image caused by quantization noise and to achieve efficient coding.

[0144] 15(a) and 15(b) are diagrams showing examples of input / output characteristics applied by the first level inverse transform unit 1703 and the second level inverse transform unit 1704 when the high exposure image is the appropriate exposure image. The horizontal axis indicates the input pixel value (12 bits), and the vertical axis indicates the output pixel value (12 bits).

[0145] The input / output characteristics shown in Fig. 15(a) are input / output characteristics for converting pixel values ​​converted by the input / output characteristics shown in Fig. 11(a) back to the original pixel level. Therefore, the input / output characteristics shown in Fig. 15(a) are equivalent to the input / output characteristics shown in Fig. 11(a) with the input and output interchanged.

[0146] Similarly, the input / output characteristics shown in Fig. 15(b) are input / output characteristics for converting the pixel values ​​converted by the input / output characteristics shown in Fig. 11(b) back to the original pixel level. Therefore, the input / output characteristics shown in Fig. 15(b) are equivalent to the input / output characteristics shown in Fig. 11(b) with the input and output interchanged.

[0147] The range of pixel values ​​of the decoded image data is restored to the range of pixel values ​​before decoding by the level inverse conversion unit 70. Therefore, a composite image with the correct brightness can be obtained in the composite unit 80.

[0148] When the decoding unit 60 outputs high exposure image data and low exposure image data in parallel, each image data can be directly input to the first level inverse conversion unit 1703 and the second level inverse conversion unit 1704. In this case, the first level inverse conversion unit 1703 and the second level inverse conversion unit 1704 may perform level inverse conversion in parallel. The level inverse converted pixel data may be output in parallel, or may be multiplexed by the selector 1705 and output as serial data.

[0149] In this embodiment, the input / output characteristics applied in the level conversion section and the level inverse conversion section are composed of straight lines with different slopes, but nonlinear input / output characteristics in which at least a portion of the slope changes continuously may be used. For example, input / output characteristics in which at least a portion of each of the input / output characteristics shown in Figures 11 and 12 is approximated by a curve may be used. In this case, the slope of the nonlinear portion may be the slope of a tangent line.

[0150] According to this embodiment, in a configuration in which a composite image is generated by synthesizing a plurality of frames of image data output from an image sensor in one shooting, the amount of image data is reduced by encoding and then transmitted, thereby making it possible to effectively use the bandwidth of the transmission path. Also, by level-converting and inversely converting pixel values ​​using appropriate input / output characteristics before encoding and after decoding, it is possible to suppress degradation of the image quality of the composite image due to encoding. Furthermore, by level-converting pixel values ​​using appropriate input / output characteristics, it is possible to reduce the amount of code by utilizing human visual characteristics in addition to reducing the amount of data by encoding. The input / output characteristics applied to the image data are set based on the magnitude of the contribution (combining ratio) to the composite image.

[0151] ●(Second embodiment) Next, a second embodiment of the present invention will be described. In the first embodiment, an image sensor capable of controlling the exposure time for each pixel is used to capture a plurality of frames of images with overlapping exposure periods and different exposure amounts in a single capture.

[0152] In the second embodiment, a configuration will be described in which a single shooting operation is performed to obtain a plurality of frames of images with overlapping exposure periods and different exposure amounts by controlling the gain of an amplifier circuit provided for each column of the image sensor in the column amplifier unit 203. Specifically, different gains are applied to the same image data to obtain image data corresponding to image data with different exposure amounts.

[0153] In this embodiment, there is no need to provide a configuration for controlling the exposure time for each pixel as in the first embodiment. That is, there is only one type of control signal for controlling the transfer switch 215 of the unit pixel 201 per pixel row, and an image sensor having color filters of different colors can be used for adjacent pixels. There is no need to add up the read pixel signals.

[0154] This embodiment can be realized with the same configuration as the first embodiment, except for the configuration for acquiring a plurality of frames of images with overlapping exposure periods and different exposure amounts by one shooting. Therefore, the following description will focus on the configuration and operation of the column amplifier unit 203 in this embodiment, and will omit a description of the same contents as the first embodiment.

[0155] FIG. 16 is a diagram showing an example of the circuit configuration of one column amplifier extracted from the column amplifier section 203 in this embodiment. The column amplifier is an inverting amplifier circuit configured with an operational amplifier 233, input side capacitors 228 and 229, and feedback capacitors 231 and 232. A predetermined reference voltage is input from a reference power supply to a positive input terminal (non-inverting input terminal) of the operational amplifier 233. In addition, the timing control unit 208 can switch whether or not the input side capacitors 228 and 229 and the feedback capacitor 231 are connected by supplying a control signal that controls the operation of the switches 226, 227, and 230.

[0156] Feedback capacitors 231 and 232 are connected in parallel to the inverting input terminal and output terminal of operational amplifier 233. Switch 230 is connected between feedback capacitor 231 and the inverting input terminal of operational amplifier 233, and switches whether or not feedback capacitor 231 is connected. Therefore, the feedback capacitance of operational amplifier 233 can be changed (switched) by switch 230.

[0157] Moreover, the input side capacitors 228, 229 are connected in parallel between the vertical signal line 220 and the inverting input terminal of the operational amplifier 233. Moreover, the switches 226, 227 are connected between the column signal line 220 and the input side capacitors 228, 229, respectively. The switches 226, 227 switch whether or not the input side capacitors 228, 229 are connected to the inverting input terminal of the operational amplifier 233. Therefore, the capacitance on the input side of the operational amplifier 233 can be changed (switched) by the switches 226, 227.

[0158] The amplification factor of the operational amplifier 233 is determined by "input capacitance / feedback capacitance." Therefore, by controlling the switches 226, 227, and 230, the amplification factor of the operational amplifier 233, that is, the gain, can be changed (switched).

[0159] Specifically, when switch 226 is turned ON and switches 227 and 230 are turned OFF, operational amplifier 233 has a gain (first gain) determined by the capacitance ratio between input capacitor 228 and feedback capacitor 232. In addition, when switch 226 is turned OFF and switches 227 and 230 are turned ON, operational amplifier 233 has a gain (second gain) determined by the ratio between the capacitance of input capacitor 229 and the combined capacitance of feedback capacitors 231 and 232. Therefore, by controlling switches 226, 227, and 230, low exposure image data and high exposure image data can be generated from one frame's worth of image data.

[0160] The column amplifier unit 203 outputs low-exposure image data or high-exposure image data to the column ADC unit 204 depending on the states of the switches 226, 227, and 230. The subsequent operations are similar to those in the first embodiment, and therefore will not be described.

[0161] This embodiment can also achieve the same effects as those of the first embodiment. Moreover, this embodiment can use an imaging unit with a general configuration, so that the cost of the imaging unit can be reduced compared to the first embodiment.

[0162] ●(Third embodiment) Next, a third embodiment of the present invention will be described below. In this embodiment, a case will be described in which high exposure image data and low exposure image data are obtained by performing multiple shooting operations with different exposure amounts.

[0163] FIG. 17 is a diagram showing an example of the operation timing of the imaging unit 20 when photographing multiple times with different exposure amounts. In FIG. 17, the i-1th image is photographed by the imaging unit 20 with an exposure time T1 corresponding to a proper exposure. After a certain time T2 has elapsed since the photographing of the i-1th image, the i-th image is photographed with an exposure time T3 shorter than that of the i-1th image. The i-1th image is a high exposure amount image and a proper exposure amount image with a relatively longer exposure time than the i-th image, and the i-th image is a low exposure amount image. The operation timing and exposure time of the imaging unit 20 shown in FIG. 17 can be changed by the control unit 120 controlling the operation of the timing control unit 208 of the imaging unit 20.

[0164] In this case, high exposure image data and low exposure image data are sequentially output from the imaging section 20 to the level conversion section 30. The subsequent operations are similar to those in the first embodiment, and therefore will not be described.

[0165] In this embodiment, the imaging unit 20 outputs one frame of image data per capture to the level conversion unit 30, so the bandwidth usage of the transmission path is smaller than in the first and second embodiments. However, by applying the same level conversion and encoding as in the first embodiment, it is possible to further reduce the bandwidth usage of the transmission path while suppressing image quality degradation due to encoding.

[0166] (Other embodiments) In the above embodiment, the configuration in which image synthesis is performed during shooting has been described. However, the present invention can also be applied to a configuration in which image synthesis is performed using image data prepared in advance. In this case, it is possible to reduce the amount of bandwidth used between the storage device or memory that stores the image data and the synthesis processing circuit.

[0167] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0168] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0169] 20: imaging section, 30: level conversion section, 40: encoding section, 60: decoding section, 70: level inverse conversion section, 80: synthesis section, 120: control section

Claims

1. An acquisition means for acquiring a plurality of image data with different exposure amounts; A conversion means for converting levels of the plurality of image data by applying an input / output characteristic; encoding means for applying encoding to reduce the amount of data of the plurality of level-converted image data; A decoding means for decoding the plurality of pieces of encoded image data; an inverse conversion means for applying an inverse characteristic of the input / output characteristic to the plurality of decoded image data; a synthesis unit that generates a synthetic image data from the plurality of image data output by the inverse conversion unit, 2. An image processing apparatus comprising: a processor for processing an image based on a combination ratio of the plurality of image data;

2. The encoding is lossy encoding, The image processing device according to claim 1 , wherein the input / output characteristics are set in accordance with the blending ratio so as to reduce deterioration in image quality caused by the encoding.

3. 3. The image processing apparatus according to claim 2, wherein the input / output characteristics are set so as to reduce degradation in image quality for pixel values ​​for which the blending ratio is greater than a predetermined value.

4. 4. The image processing device according to claim 2, wherein the input / output characteristics are set so as not to reduce degradation in image quality for pixel values ​​larger than a predetermined value even if the blending ratio is a pixel value larger than a predetermined value.

5. 2 . The image processing apparatus according to claim 1 , wherein the input / output characteristics have at least one of an interval having a slope greater than 1 and an interval having a slope smaller than 1. 3 .

6. 6. The image processing apparatus according to claim 5, wherein the section having a gradient greater than 1 corresponds to a range of pixel values ​​in which the blending ratio is greater than a predetermined value.

7. 6. The image processing apparatus according to claim 5, wherein the section having a gradient smaller than 1 corresponds to a range of pixel values ​​in which the blending ratio is smaller than a predetermined value.

8. the plurality of image data are first image data having an appropriate exposure amount and second image data having an exposure amount lower than the appropriate exposure amount, the combining means combines the first image data and the second image data using a combining ratio according to a pixel value; 2. The image processing device according to claim 1, wherein the input / output characteristics applied to the first image data have a slope greater than 1 in the section from a minimum pixel value to a first pixel value, and a slope less than 1 in the section from a second pixel value greater than the first pixel value to a maximum pixel value.

9. the plurality of image data are first image data having an appropriate exposure amount and second image data having an exposure amount lower than the appropriate exposure amount, the combining means combines the first image data and the second image data using a combining ratio according to a pixel value; 9. The image processing device according to claim 1, wherein the input / output characteristics applied to the second image data have a slope smaller than 1 in the section from a minimum pixel value to a third pixel value, and a slope larger than 1 in the section from a fourth pixel value larger than the third pixel value to a maximum pixel value.

10. the plurality of image data are first image data having an appropriate exposure amount and second image data having an exposure amount higher than the appropriate exposure amount, the combining means combines the first image data and the second image data using a combining ratio according to a pixel value; 2. The image processing device according to claim 1, wherein the input / output characteristics applied to the first image data have a slope smaller than 1 in a section from a minimum pixel value to a fifth pixel value and in a section from a sixth pixel value greater than the fifth pixel value to a maximum pixel value.

11. the plurality of image data are first image data having an appropriate exposure amount and second image data having an exposure amount higher than the appropriate exposure amount, the combining means combines the first image data and the second image data using a combining ratio according to a pixel value; 11. The image processing device according to claim 1, wherein the input / output characteristics applied to the second image data have a slope greater than 1 in the section from the minimum pixel value to the seventh pixel value, and a slope less than 1 in the section from an eighth pixel value greater than the seventh pixel value to the maximum pixel value.

12. 12. The image processing apparatus according to claim 1, wherein the acquisition means acquires the plurality of image data by one shooting operation using an image sensor capable of controlling an exposure time on a pixel-by-pixel basis.

13. 12 . The image processing apparatus according to claim 1 , wherein the acquisition means acquires the plurality of image data by varying a gain applied to image data obtained by one shooting operation.

14. 12. The image processing apparatus according to claim 1, wherein the acquisition means acquires the plurality of pieces of image data by photographing the image a plurality of times with different exposure amounts.

15. An image processing device according to any one of claims 1 to 11, An imaging means for outputting the plurality of image data; An imaging device comprising:

16. An image processing method executed by an image processing device, comprising: An acquisition step of acquiring a plurality of image data with different exposure amounts; a conversion step of converting levels of the plurality of image data by applying input / output characteristics; an encoding step of applying encoding to reduce a data amount of the plurality of level-converted image data; a decoding step of decoding the plurality of encoded image data; an inverse conversion step of applying an inverse characteristic of the input / output characteristic to the plurality of decoded image data; a synthesis step of generating synthetic image data from the plurality of image data output from the inverse conversion step, The image processing method according to claim 1, wherein the input / output characteristics are set in accordance with a synthesis ratio of the plurality of image data in the synthesis step.

17. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 11.

Images