Image encoding device, image decoding device, control method and program thereof, and imaging device

Abstract

To generate encoded data in which a deterioration in image quality is prevented, while generating encoded data with a fixed length.SOLUTION: An image encoder has an acquisition unit that acquires image data of blocks, and an image encoding unit that encodes the acquired image data, generates encoded data with an encoding amount not exceeding a fixed length, and adds and outputs bits required for the fixed length. The image encoding unit 20 includes an encoding unit that quantizes pixels included in an attentional block so that the encoding amount of the attentional block becomes equal to or less than the fixed length to perform variable-length encoding, a determination unit that determines the size of a space area to be the fixed length from the difference between the fixed length and the encoding amount of the attentional block obtained by the encoding unit, an analysis unit that analyzes quantization data of the pixels of the attentional block and calculates an index value indicating the importance of the pixels, and a multiplexing unit that stores, in the space area, the values of more bits lost in the quantization as the importance of the pixels indicated by the index value becomes more significant, and multiplexes the encoded data and the bits lost in the quantization.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to image encoding and decoding techniques. [Background technology]

[0002] In recent years, imaging devices such as digital video cameras have become increasingly sensitive to the need for higher resolution and frame rates, resulting in a dramatic increase in the amount of image data handled per unit time by the system. This has created a demand for faster image memory and bus interface circuits within the device. To address this, it is conceivable to compress and code images before and after the image memory or bus interface. In other words, by suppressing the amount of data per unit time on the bus interface, the demand for faster circuits can be alleviated.

[0003] In this case, it is desirable that the coding method for compression coding has a small circuit scale and coding delay. Therefore, conventional DCT-based coding methods such as JPEG and MPEG2 are not suitable. Therefore, compression coding using a DPCM (Differential Pulse Code Modulation)-based predictive coding method has been proposed (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-213528 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, if a pixel block to be coded contains a steep edge, a large code length must be assigned to the edge component in order to preserve the existence of the edge. However, in order to fix the code length for the pixel block, the quantization step for the entire pixel block must be large. As a result, although the edge component is preserved, the details of the original pixel data are lost, causing a large degradation in image quality.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that generates fixed-length encoded data while suppressing image quality degradation more than ever before when a complex image such as a sharp edge is present within the encoding block. [Means for solving the problem]

[0007] In order to solve this problem, for example, an image coding device of the present invention has the following arrangement. 1. An image coding apparatus that performs coding on a block basis, the block being composed of a predetermined number of pixels, generates coded data of a preset fixed length, and outputs the coded data, acquiring means for acquiring image data of said block from an image generating source; an encoding processing means for encoding the image data acquired by the acquisition means, generating encoded data having a code amount not exceeding the fixed length, and adding a bit shortage to obtain the fixed length and outputting the encoded data; The encoding processing means a variable length coding means for quantizing each pixel included in a target block so that the code amount of the target block is equal to or less than the fixed length, and for variable length coding the quantized data obtained by the quantization; a determining means for determining a size of a free area required for achieving the fixed length from a difference between the amount of code of the block of interest obtained by the variable length coding means and the fixed length; an analysis means for analyzing the quantization data of each pixel of the block of interest and calculating an index value indicating the importance of each pixel; The present invention is characterized in that it includes a multiplexing means for multiplexing the coded data obtained by the variable length coding means and the bits lost in the quantization, thereby storing in the free area a larger number of bits lost in the quantization for pixels having a greater importance indicated by the index value, thereby generating the fixed length data. Effect of the Invention

[0008] According to the present invention, it is possible to generate coded data of fixed length while suppressing degradation in image quality more than ever before, even when a complex image such as a sharp edge is present within the coding block. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of an image processing apparatus according to an embodiment. [Diagram 2] FIG. 2 is a block diagram of an image encoding unit. [Diagram 3] FIG. 2 is a block diagram of an image decoding unit. [Figure 4] FIG. 2 is a block diagram showing details of an encoding unit. [Diagram 5] FIG. 2 is a diagram showing a format of image data to be encoded. [Figure 6] FIG. 1 is a diagram for explaining quantized data and loss bits. [Figure 7] 13 is a flowchart of a process for determining an encoding mode and an applied QP. [Figure 8] 8 is a flowchart showing details of the process of S704 in FIG. 7. [Figure 9] 11A and 11B are diagrams showing specific examples of the relationship between block code length and degradation degree. [Figure 10] 11 is a flowchart showing a feature analysis process. [Figure 11] 11 is a flowchart showing a feature classification method in S1004 of FIG. 10. [Figure 12] FIG. 2 is a diagram showing the format of encoded data in DPCM mode. [Figure 13] FIG. 1 is an explanatory diagram of a method for storing lost bits in DPCM mode. [Figure 14] FIG. 2 is a diagram showing a format of encoded data in PCM mode. [Figure 15] 10 is a flowchart showing a feature analysis process in a feature analysis unit according to the second embodiment. [Figure 16] 13A to 13C are diagrams showing specific examples of subbands generated by feature analysis processing in the second embodiment. [Figure 17] 13 is a flowchart of a process of determining an encoding mode and an application QP according to a third embodiment. [Figure 18] FIG. 13 is a diagram showing a format of encoded data in a PCM mode in the third embodiment. [Figure 19] FIG. 13 is an explanatory diagram of a method for storing lost bits in a PCM mode according to the third embodiment. [Figure 20] FIG. 13 is a block diagram showing the configuration of an image processing device according to a fourth embodiment. [Figure 21] FIG. 13 is a block diagram showing the configuration of an imaging unit according to a fourth embodiment. [Figure 22] FIG. 13 is a diagram showing a flow of signals from a pixel unit to a horizontal transfer circuit in the fourth embodiment. [Figure 23] FIG. 13 is a diagram showing a pixel array of a pixel unit according to a fourth embodiment. [Figure 24] 13 is a diagram showing settings of an exposure time of a pixel unit in the fourth embodiment. [Diagram 25] 13 is a diagram showing an image output format when an imaging unit according to a fourth embodiment operates in a normal mode. [Figure 26] 13A to 13C are diagrams showing an image output format when an imaging unit according to a fourth embodiment operates in HDR mode. [Figure 27] FIG. 13 is a block diagram showing the configuration of an image synthesis unit according to the fourth embodiment. [Figure 28] 13 is a diagram showing details of correction performed by a gain correction unit in the fourth embodiment. [Figure 29] 13A and 13B are diagrams showing an example of a method for synthesizing a low-sensitivity image and a high-sensitivity image performed by a synthesizing unit in the fourth embodiment. [Diagram 30]13 is a flowchart showing an overview of feature analysis processing in a feature analysis unit when an imaging unit according to a fourth embodiment operates in HDR mode. [Figure 31A] 13 is a flowchart showing details of feature analysis processing in a feature analysis unit when an imaging unit according to a fourth embodiment operates in HDR mode. [Figure 31B] 13 is a flowchart showing details of feature analysis processing in a feature analysis unit when an imaging unit according to a fourth embodiment operates in HDR mode. [Figure 31C] 13 is a flowchart showing details of feature analysis processing in a feature analysis unit when an imaging unit according to a fourth embodiment operates in HDR mode. [Figure 31D] 13 is a flowchart showing details of feature analysis processing in a feature analysis unit when an imaging unit according to a fourth embodiment operates in HDR mode. [Diagram 32] FIG. 13 is a diagram showing a circuit configuration of a column amplifier for only a certain column according to a fifth embodiment. [Diagram 33] FIG. 23 is a diagram showing an example of operation timing of an imaging unit according to the sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [First embodiment] <Overall description of image processing device> 1 is a block diagram showing the configuration of an image processing device 100 having the functions of an image encoding device and an image decoding device. The image processing device 100 includes an acquisition unit 10, an image encoding unit 20, a memory 30, and an image decoding unit .

[0012] The functions of each block of the image processing device 100 may be implemented by either software or hardware, except for physical devices such as an imaging element and a storage element of the memory 30. For example, the functions of each block may be implemented by hardware such as a dedicated device, a logic circuit, and a memory. Alternatively, the functions of each block may be implemented by a memory, a processing program stored in the memory, and a computer such as a CPU that executes the processing program. The image processing device 100 can be implemented as, for example, a digital camera, but can also be implemented as other devices. For example, the image processing device 100 can be implemented as any information processing terminal or imaging device, such as a personal computer, a mobile phone, a smartphone, a PDA, a tablet terminal, or a digital video camera.

[0013] The acquisition unit 10 has a function of acquiring image data from an image generation source. The image generation source may be, for example, an imaging unit equipped with an image sensor, or a communication unit that receives image data from an external device via a transmission path. Alternatively, the acquisition unit 10 may acquire the image data via an interface that reads image data from a recording medium or the like. The acquired image data may be still image data or video data. When the image data acquired by the acquisition unit 10 is video data, multiple frames may be acquired continuously. The acquisition unit 10 supplies the acquired image data to the image encoding unit 20.

[0014] The image encoding unit 20 encodes the image data supplied from the acquisition unit 10 according to an encoding method described below, and outputs encoded data with a compressed amount of information. The output encoded data is stored in a memory 30. This memory 30 functions as a buffer memory, and has a storage capacity necessary for storing the encoded data output from the image encoding unit 20.

[0015] The image decoding unit 40 reads out the encoded data held in the memory 30, decodes it into image data by a decoding method described later, and outputs the decoded image data to a downstream processing unit (not shown).

[0016] In addition, in FIG. 1, the acquisition unit 10, the image encoding unit 20, the memory 30, and the image decoding unit 40 are each illustrated independently, but some or all of these blocks may be integrated into one chip.

[0017] <Explanation of the image encoding unit> The image encoding unit 20 of this embodiment encodes each pixel included in a block of interest consisting of a predetermined number of pixels, generates encoded data with a code amount equal to or less than a preset fixed length, and outputs the encoded data by adding bits (lost bits to be described later) that are insufficient to make the data have the fixed length. Fig. 2 is a block diagram of the image encoding unit 20 of this embodiment. The configuration and operation of the unit will be described below with reference to the same figure.

[0018] The image encoding unit 20 includes a provisional encoding system 210 and a final encoding system 220. The provisional encoding system 210 includes encoding units 211A to 211D and a QP determination unit 215, and operates to determine an encoding mode and a quantization parameter (QP) to be used when the final encoding system 220 performs final encoding. The final encoding system 220 includes a delay unit 221, an encoding unit 211E, a multiplexing unit 223, an inverse quantization unit 224, and a feature analysis unit 225. The final encoding system 220 operates to execute final encoding including a quantization process according to the encoding mode and QP determined by the provisional encoding system 210. In the following description, the encoding units 211A to 211E may be collectively referred to as the encoding unit 211 since they have the same internal configuration.

[0019] The image encoding unit 20 may be configured as integrated hardware including a dedicated device, a logic circuit, a memory, etc., or may be configured as a distributed system including a plurality of devices, etc. Alternatively, the image encoding unit 20 may be implemented by a memory, a processing program stored in the memory, and a computer such as a CPU that executes the processing program.

[0020] The image encoding unit 20 inputs image data to be encoded, acquired by the acquisition unit 10, via an input terminal 201. In the following description, the RGB format shown in FIG. 5(a) is used as an example of the format of the image data. The image data is input in raster scan order, and each pixel data of R (red), G (green), and B (blue), which are color elements, is multiplexed in time division and input sequentially. The bit depth of each pixel in this embodiment is 10 bits. The format and bit depth of the image data are not limited to the RGB format shown in FIG. 5(a). The format of the input image data may be the YCbCr4:2:2 format shown in FIG. 5(b) or the Bayer array format shown in FIG. 5(c). The bit depth of the image data may be 8 bits, 12 bits, or the like.

[0021] <Description of coding blocks> In the image encoding process executed by the image encoding unit 20 in this embodiment, image data to be encoded is divided into blocks (encoding blocks) having a predetermined size (number of pixels), and encoding is performed in units of encoding blocks. In the following description, as shown in FIG. 5(a), the encoding block is configured with 24 pieces of data (8 horizontal pixels x 1 vertical pixel x 3 color components). As shown in FIG. 5(a), the encoding block is configured with a total of 8 pixels having R, G, and B color components, and is arranged in the order of R0, G0, B0, R1, G1, B1, ... in raster scan order. Although R0, G0, and B0 represent color components constituting one pixel, in the embodiment, each color component is referred to as a "pixel" for convenience. Therefore, in FIG. 5(a), pixel numbers 0 to 23 are assigned to the 24 pieces of data. Then, encoding is performed in order from the smallest pixel number in the encoding block.

[0022] The structure of the coding block is not limited to the structure shown in Fig. 5(a). For example, in the case of image data in the above-mentioned YCbCr4:2:2 format, as shown in Fig. 5(b), the coding block may be composed of a total of 32 pieces of data, 16 pieces of Y, 8 pieces of Cb, and 8 pieces of Cr. In the case of Fig. 5(b), pixel numbers 0 to 31 are assigned.

[0023] Alternatively, in the case of image data in the above-mentioned Bayer array format, as shown in Fig. 5(c), the coding block may be composed of a total of 32 pixels, including 8 Gr, 8 Gb, 8 R, and 8 B. In Fig. 5(c), pixel numbers 0 to 31 are assigned. Alternatively, in the case of the Bayer array, a two-dimensional structure such as 24 pixels horizontally and 2 pixels vertically may be used.

[0024] In this embodiment, the compression rate for the coding block shown in FIG. 5(a) is 6 / 10. The amount of information of image data before coding is 24 pixels×10 bits=240 bits per coding block, so the target code amount for each coding block is 144 bits. The image coding unit 20 selects a coding mode and QP for each coding block so that the target code amount is not exceeded. The coding modes include a PCM mode that uses quantized image data as code data, and a DPCM mode that quantizes image data and then codes the difference value between the quantized image data and a predicted value. It should be understood that the target code amount of 144 bits is merely an example for ease of understanding.

[0025] <Details of the Encoding Unit 211> 3 is a block diagram showing a detailed configuration of the encoding unit 211 (encoding units 211A to 211E). As shown in FIG. 3, the encoding unit 211 includes a quantization unit 301, a prediction unit 302, a subtractor 303, a variable length encoding unit 304, a code length calculation unit 305, a selector 307, and a selector 308.

[0026] The encoding unit 211 is provided with image data, an encoding mode flag, and a quantization parameter QP for each encoding block. In the case of the provisional encoding system 210, the QP is assigned in advance as a fixed value to each of the encoding units 211A to 211D. In the embodiment, as shown in FIG. 2, 0, 1, 2, and 3 are assigned as QP to the encoding units 211A to 211D in that order. In the case of the provisional encoding system 210, each of the encoding units 211A to 211D may hold a QP value in advance. In addition, in the case of the provisional encoding system 210, the encoding mode flag is input as a fixed value indicating the DPCM mode. The encoding mode flag and QP determined in the provisional encoding system 210 are supplied from the QP determination unit 215 to the encoding unit 211E of the main encoding system 220 via the delay unit 221. Hereinafter, a specific configuration and operation of the encoding unit 211 will be described in detail.

[0027] First, image data of a coding block input to the coding unit 211 (in this embodiment, 24 pixel data in FIG. 5(a)) is input to the quantization unit 301. The quantization unit 301 quantizes each pixel data of the input image data in accordance with a given QP, and generates quantized data and bit data lost by quantization (referred to as lost bits). The quantization unit 301 then outputs the generated quantized data to the prediction unit 302, the subtractor 303, and to an external device outside the coding unit 211. The quantization unit 301 also outputs the generated lost bits to an external device outside the coding unit 211.

[0028] In this embodiment, the QP value is an integer value with 0 as the minimum value, and can be changed in the range from 0 to 3. However, a larger QP value may be set to use a larger quantization step.

[0029] In this embodiment, the quantization unit 301 makes the quantization step smaller (finer) as the QP is smaller, and makes the quantization step larger (coarser) as the QP is larger. Specifically, the quantized data is generated by the calculation expressed by Equation 1, and the loss bits are generated by the calculation expressed by Equation 2. Quant= Data >> QP …(1) Loss = Data - (Quant << QP) …(2) Here, Quant is the quantized data, Loss is the lost bit, Data is the input image data, and QP is the quantization parameter. >> indicates a right shift and << indicates a left shift. In other words, the quantized data Quant is the pixel data value multiplied by 2. QP The quotient when dividing by , indicates that the number of effective bits is reduced from 10 bits to 10-QP bits. Also, the loss bit Loss represents the remainder that is truncated in the division, and indicates that the number of bits is QP.

[0030] Here, the quantized data and loss bits will be described with reference to Fig. 6(a) to (d). Fig. 6(a) to (d) are diagrams showing all bits of an encoded block of image data, and show an arrangement of 24 pixel data in the horizontal direction and a bit depth of 10 bits (bit0 to bit9) in the vertical direction.

[0031] 6(a) shows the quantized data (output of the quantizer 301 of the encoder 211A) corresponding to QP=0, and indicates that the input data is output as is without being quantized. In this case, there are no loss bits.

[0032] Figure 6(b) shows the quantized data and lost bits corresponding to QP=1 (output of the quantization unit 301 of the encoding unit 211B), indicating that the most significant 9 bits are output as quantized data and the least significant bit is output as a lost bit.

[0033] Figure 6(c) shows the quantized data and lost bits corresponding to QP=2 (output of the quantization unit 301 of the encoding unit 211C), indicating that the most significant 8 bits are output as quantized data and the least significant 2 bits are output as lost bits.

[0034] Figure 6(d) shows the quantized data and lost bits corresponding to QP=3 (output of the quantization unit 301 of the encoding unit 211D), indicating that the most significant 7 bits are output as quantized data and the least significant 3 bits are output as lost bits.

[0035] In this embodiment, for example, increasing QP by one from "A" to "A+1" (in this embodiment, A is 0, 1, or 2) is expressed as increasing the quantization step by one step, making it larger by one step, or making it coarser by one step, etc. Conversely, decreasing QP by one from "B" to "B-1" (in this embodiment, B is 1, 2, or 3) is expressed as decreasing the quantization step by one step, making it smaller by one step, or making it finer by one step, etc.

[0036] In the present embodiment, quantization is performed such that the quantized data decreases by 1 bit every time the QP increases by 1, but the present invention is not limited to this method. For example, the quantization unit 301 can also perform nonlinear quantization.

[0037] The code length calculation unit 305 determines the code length of the quantized data per pixel output from the quantization unit 301 from the bit depth of the image data (10 bits in this embodiment) and QP, using the following equation (3). Quantized data length = image data bit depth - QP … (3) In this embodiment, the code length of the quantized data is decreased by one bit each time the value of QP is increased by one, with QP=0 as the initial value, and the code length of the quantized data is shortened by one bit from 10 bits. Here, in the temporary encoding system 210, the value of QP is fixedly assigned to each of the encoding units 211A to 211D, so the quantized data code length is also a fixed value. Therefore, the code length calculation unit 305 may be configured to hold and output a fixed value of the quantized data code length based on the assigned value of QP, rather than calculating the quantized data code length by the operation of Equation 3. The code length calculation unit 305 outputs the determined quantized data code length to the selector 308.

[0038] Next, the operation of the prediction unit 302 will be described. The prediction unit 302 generates prediction data of the current quantized data (prediction value of the quantized value of the current pixel) using the surrounding quantized data. As a simple method, there is a method of using the previous pixel of the same color component as the prediction data as it is. For example, in this embodiment, as shown in FIG. 5(a), image data of each color element of RGB is input in order, and after encoding of image data G0, image data B0 and R1 are encoded first before encoding G1 next. Therefore, the prediction unit 302 delays the quantized data by three pixels and outputs it as prediction data. In addition, the prediction unit 302 may perform calculations such as linear prediction using multiple previous pixels of the same color component. Alternatively, if the coding block has a two-dimensional structure, calculations such as planar prediction using surrounding pixels of the same color component or MED prediction adopted in JPEG-LS may be performed. Note that there is no previous pixel for the first three pixels (R0, G0, B0) of the coding block of each color component. Therefore, the prediction unit 302 outputs a predetermined value (for example, 0) to the subtractor 303 as prediction data for the first three pixels of the coding block.

[0039] The subtractor 303 outputs the difference between the quantized data from the quantizer 301 and the predicted data from the predictor 302 as predicted difference data to the variable-length encoder 304. The predicted difference data has positive and negative values, and has a value close to 0 in flat parts where the image data fluctuates little, and a large value in edge parts where the image data fluctuates greatly. The predicted difference data generally has the characteristics of a Laplace distribution centered on 0.

[0040] The variable length coding unit 304 performs coding on the input prediction difference data by a predetermined variable length coding method, and generates variable length code data and a variable length code data code length for each pixel. Then, the variable length coding unit 304 outputs the generated variable length code data to a selector 307, and outputs the variable length code data code length to a selector 308. For example, Huffman code, Golomb code, etc. can be used as the variable length coding method. In the variable length coding method executed by the variable length coding unit 304, when the input value is 0, code data with the shortest code length is assigned, and the larger the absolute value of the input value, the longer the code length of the code data becomes.

[0041] The selector 307 receives the quantized data and the variable-length code data, selects one of them according to the encoding mode flag, and outputs the selected data as encoded data to the outside of the encoding unit 211. In the embodiment, the selector 307 selects the variable-length code data when the encoding mode flag has a value of 0, and selects the quantized data when the encoding mode flag has a value of 1.

[0042] The selector 308 receives the quantized data code length and the variable-length code data code length, selects one of them according to the encoding mode flag, and outputs the selected code length to the outside of the encoding unit 211. When the encoding mode flag has a value of 0, the selector 308 in the embodiment selects the variable-length code data code length, and when the encoding mode flag has a value of 1, selects the quantized data code length.

[0043] Here, the coding modes will be described. The coding unit 211 operates by switching between DPCM mode and PCM mode according to a coding mode flag indicating the coding mode. In the DPCM mode, a subtractor 303 generates prediction difference data using quantized data quantized by a quantizer 301 and prediction data generated by a predictor 302, and variable-length coding is performed by a variable-length coder 304. In the DPCM mode, variable-length code data is generated based on image data and QP. In the DPCM mode, the coding mode flag is "0". On the other hand, in the PCM mode, input image data is quantized as is by the quantizer 301 and output as quantized data. In the PCM mode, fixed-length code data determined by QP is generated. 。P In the CM mode, the coding mode flag is set to "1".

[0044] In the PCM mode, in multiplexing the coded data described later, it is necessary to multiplex a 1-bit flag indicating that the coded data is generated in the PCM mode. Therefore, a specific pixel among the pixels of the coding block is quantized by increasing the QP by one step, and operates to output coded data and a code length that is 1 bit less than other pixels. The specific pixel is, for example, the pixel located at the left end (first) of the coding block.

[0045] <Description of provisional coding system> Here, referring again to the image encoding unit 20 in Fig. 2, the provisional encoding process in the provisional encoding system 210 will be described. Image data input to the provisional encoding system 210 in Fig. 2 is provisionally encoded by the multiple encoding units 211A to 211D in the encoding mode of DPCM with QPs of 0 to 3. Then, the encoding units 211A to 211D output the encoding results to the QP determination unit 215. As shown in Fig. 3, the signals output from the encoding units 211A to 211D include output signals of code data, code length, quantized data, and loss bits, but only the code length (code length of variable-length code data) is used in this provisional encoding system 210.

[0046] In this embodiment, the range of QP used for encoding is set to 0 to 3, so the provisional encoding system 210 includes four encoding units 211A to 211D. However, this is for the purpose of concretely realizing and simplifying the explanation, and the number of encoding units 211 can be changed depending on the range of QP used for encoding.

[0047] The QP determination unit 215 determines a QP (called an application QP) for a coding block of interest in the coding system 220 based on information on the code length of one coding block for each QP input from the preceding coding units 211A to 211D. Note that the QP determination unit 215 in this embodiment is assumed to have a built-in buffer for temporarily storing data input from each of the coding units 211A to 211D. A method for determining an application QP in the QP determination unit 215 will be described in detail below.

[0048] The process of determining an application QP in QP determination section 215 will be described in detail with reference to FIG.

[0049] First, in S701, the QP determining unit 215 acquires, in pixel units, information on the code length calculated for each QP assigned to each encoding unit from the encoding units 211A to 211D.

[0050] In S702, the QP determination unit 215 calculates the code length of the entire coding block (hereinafter, block code length) for each QP by using the total of the acquired pixel-by-pixel code lengths. In the embodiment, since the provisional coding system 210 has four coding units 211A to 211D, the QP determination unit 215 calculates four block code lengths. When calculating the block code length, it is necessary to take into account the code length of the header information multiplexed on the coded data. The header information is information for each coding block required for decoding. In the present embodiment, the code length of the header information is 3 bits in total, including 2 bits for expressing QP (0 to 3) and 1 bit for the coding mode flag. Therefore, in S702, the QP determination unit 215 calculates the block code length bl_size[qp] by adding the code length hd_size (=3 bits) of the header information and the total of the pixel-by-pixel code lengths (here, qp is a value corresponding to QP). The number of bits for expressing the QP can be changed depending on the range of the QP used for encoding.

[0051] In S703, the QP determination unit 215 calculates the number of free bits free_num[qp] and the number of lost bits loss_num[qp] for each QP using the target code amount target_size (144 bits in the embodiment) and the block code length bl_size[qp] for each QP. The QP determination unit 215 also calculates the degradation level bl_dist[qp] for each QP using the number of free bits free_num[qp] and the number of lost bits loss_num[qp].

[0052] The number of free bits free_num[qp] is calculated by the following equation (4). free_num[qp] = target_size - bl_size[qp] …(4) The number of free bits is a value obtained by subtracting the block code length from the target code amount, as shown in Equation 4. If the number of free bits is a positive value, it indicates that the block code length at the corresponding QP is smaller than the target code amount. If the number of free bits is a negative value, it indicates that the block code length at the corresponding QP is larger than the target code amount.

[0053] The number of lost bits loss_num[qp] is calculated by the following equation (5). loss_num[qp] = qp × pix_num …(5) Here, pix_num is the number of pixels in the coding block ("24" in this embodiment).

[0054] The number of lost bits loss_num[qp] indicates the number of bits lost by quantization, and is a value that is uniquely determined by QP and the number of pixels in the coding block. Here, since the number of pixels in the coding block in this embodiment is pix_num=24, the number of lost bits loss_num[2] when QP=2 is 48.

[0055] The deterioration level bl_dist[qp] is calculated by the following equation (6). bl_dist[qp] = loss_num[qp] - free_num[qp] …(6) The degradation level bl_dist[qp] indicates the degree of coding degradation. In this embodiment, when the block code length is smaller than the target code amount, coding is performed so that bit data lost by quantization (lossy bits) is stored in the free bit area as much as possible. During decoding, the degradation due to quantization is reduced by restoring the lost bits stored in the free bit area. Therefore, the final degradation level is a value obtained by subtracting the number of free bits from the number of lost bits.

[0056] In S704, the QP determination unit 215 determines the coding mode and the QP to be applied, using the bl_size[qp] calculated in S702 and the bl_dist[qp] calculated in S703.

[0057] The process of S704 will now be described in detail with reference to FIG.

[0058] In S801, the QP determination unit 215 initializes several variables. Specifically, the QP determination unit 215 initializes qp (the value of QP) to 0 and initializes min_dist (the value of the minimum degradation degree) to a preset MAX_DIST. Also, the QP determination unit 215 initializes sel_qp (the applied QP) to a value of MAX_QP + 1. Here, MAX_DIST is the maximum value that can be taken as the degradation degree of the encoded block, and it is a value obtained by subtracting the target code amount from the information amount of the image data before encoding. In this embodiment, MAX_DIST = 240 - 144 = 96. Also, MAX_QP is the maximum value of QP used for encoding. Since MAX_QP is 3 in this embodiment, sel_qp is initialized to 4. The initial value of sel_qp is initialized as the QP to be applied when the conditions for selecting the applied QP described below are not satisfied for any QP.

[0059] In S802, the QP determination unit 215 determines whether the condition that the code length bl_size[qp] of the encoded block at the current QP (variable qp) is less than or equal to the target code amount target_size and the degradation degree bl_dist[qp] of the encoded block is less than min_dist is satisfied. If this condition is satisfied, the QP determination unit 215 proceeds to S803, and if not, it proceeds to S804.

[0060] In S803, the QP determination unit 215 substitutes qp into sel_qp as a candidate for the QP to be selected. Also, the QP determination unit 215 substitutes the degradation degree bl_dist[qp] into min_dist. Also, since it is determined at this point to use the DPCM mode as the encoding mode, the QP determination unit 215 sets enc_mode to the DPCM mode.

[0061] In S804, the QP determination unit 215 determines whether the value of the current QP (variable qp) is less than the maximum value MAX_QP. If qp < MAX_QP, the process proceeds to S805, and if not, the process of this flowchart ends.

[0062] In S805, the QP determination unit 215 adds qp by 1. After that, the same process is performed again from S802 for the next QP value.

[0063] If S802 has never returned a "YES" result and S804 returns a "NO" result, this means that in DPCM mode, there is no QP with a block code length smaller than the target code length, and the block code length cannot fit within the target code length. In this case, enc_mode and sel_qp remain in the PCM mode and 4 (=MAX_QP+1) initialized in S801, respectively, and the process in FIG. 8 ends.

[0064] As a result of the above, if any of the block code lengths bl_size[qp] is equal to or less than the target code amount, the DPCM mode is selected, and the QP that minimizes the degradation degree bl_dist[qp] of the coding block is selected as the applicable QP. If any of the block code lengths bl_size[qp] exceeds the target code amount, the PCM mode is selected, and the qp value "4" obtained by adding 1 to the maximum QP value (MAX_QP) is used as the applicable QP.

[0065] FIG. 9 is a diagram showing a specific example of the relationship between bl_size[qp] and bl_dist[qp]. In the example of FIG. 9(a), when QP=3, the block code length bl_size[3]=92 bits is equal to or less than the target code amount of 144 bits, and the degradation degree bl_dist[3]=20 is the minimum. Therefore, in the case of FIG. 9(a), QP=3 is selected as the applied QP, and the DPCM mode is selected as the coding mode. In the example of FIG. 9(b), the block code length exceeds the target code amount for all QPs. Therefore, "4" is selected as the applied QP, and the PCM mode is selected as the coding mode.

[0066] As explained with reference to formula (6), the degradation level bl_dist[qp] is the value obtained by subtracting the number of free bits free_num[qp] from the number of lost bits loss_num[qp]. Here, by substituting the right side of formula (4) for the number of free bits free_num[qp] in formula (6), the following formula (7) is obtained. bl_dist[qp] = loss_num[qp] - (target_size - bl_size[qp]) = loss_num[qp] + bl_size[qp] - target_size …(7) In formula (7), target_size is constant regardless of the value of QP, and the code length corresponding to the header information in the block code length bl_size[qp] is also constant regardless of the value of QP. Therefore, the magnitude relationship of the degradation degree between QPs is determined by the sum of the code lengths (block code length) and the number of lost bits loss_num[qp] output by the encoding unit 211. Therefore, the QP determination unit 215 can determine the applied QP and the coding mode based on the code length and the number of lost bits without calculating the degradation degree. In the example of FIG. 9(a), since the sum of the code length and the number of lost bits is the minimum value when QP=3, QP=3 is determined as the applied QP. Also, in the example of FIG. 9(b), all of the code lengths corresponding to each QP exceed the target code amount. Therefore, QP=4 is determined as the applied QP so that the code amount of the coding block is equal to or less than the target code amount.

[0067] <Encoding process in the encoding system 220> Referring again to FIG. 2, the encoding process in the encoding system 220 will now be described.

[0068] The main encoding system 220 receives the same image data as the image data of the encoding block input to the provisional encoding system 210. However, the encoding process of the main encoding system 220 cannot start until the QP determination unit 215 of the provisional encoding system 210 determines and outputs the encoding mode and the applied QP. Therefore, the input image data of the encoding block is first input to the delay unit 221 and is delayed by the number of processing cycles required for the provisional encoding system 210 to determine the encoding mode and the applied QP. The delayed image data of the encoding block is input from the delay unit 221 to the encoding unit 211E. This allows the encoding unit 211E to perform encoding using the encoding mode and the applied QP determined by the provisional encoding system 210.

[0069] The encoding unit 211E is supplied with image data of the delayed encoding block, the encoding mode flag determined by the QP determination unit 215, and the applied QP. Then, the encoding unit 211E performs main encoding on the delayed image data according to the input encoding mode flag and applied QP. This generates code data having the same code length as the block code length determined by the QP determination unit 215. The encoding unit 211E outputs the generated code data to the multiplexing unit 223 together with the code length and loss bits. The encoding unit 211E also outputs quantized data quantized according to the applied QP to the inverse quantization unit 224.

[0070] The inverse quantization unit 224 generates inverse quantized data by inverse quantizing the quantized data from the encoding unit 211E using the application QP output by the QP determination unit 215. The inverse quantization unit 224 also outputs the inverse quantized data to the feature analysis unit 225.

[0071] Here, the bit depth of the inverse quantized data inversely quantized using the applied QP is always the same as the bit depth of the input image data. Therefore, the feature analysis unit 225 in this embodiment always performs feature analysis on 10-bit data that is the same as the bit depth of the input image data. However, if the feature analysis unit 225 supports processing of all bit depths, it is not necessarily necessary to provide the inverse quantization unit 224 as in this embodiment. In that case, the feature analysis unit 225 may be configured to analyze the quantized data output from the encoding unit 211E as it is.

[0072] The feature analysis unit 225 analyzes the multiple inverse quantization data output from the inverse quantization unit 224, and detects a portion in the coding block where degradation is likely to be visually noticeable to humans. In this embodiment, a dark portion or a flat portion in the image is treated as a portion where degradation is likely to be noticeable.

[0073] To give a specific example, in the embodiment, each pixel is based on the brightness of RGB. Therefore, the feature analysis unit 225 counts the number of pixels that satisfy "data after inverse quantization < threshold value for determining dark area", and determines it as a dark area if this count value is equal to or greater than a predetermined number, and determines it as a non-dark area if the count value is less than the predetermined number.

[0074] In addition, the determination of whether or not it is a flat portion is, for example, performed by calculating the difference between the maximum and minimum values ​​of the inverse quantization data, and if the difference is equal to or less than the threshold value for flatness determination, it is determined to be flat, and if the difference exceeds the threshold value, it is determined to be non-flat. Although it takes some time, it is also possible to determine whether the data is flat or non-flat by calculating the variance of the inverse quantization data and comparing it with the threshold value.

[0075] The above feature analysis method is merely an example, and is not limited to this method. For example, edges in an image (parts where light and dark change), flat parts adjacent to edges, boundary parts of coding blocks, etc. can be treated as parts where degradation is likely to be noticeable. In addition, when the parts where degradation is likely to be noticeable differ depending on the image to be coded or the coding method, it is also possible to adaptively change the feature analysis method according to the situation.

[0076] Hereinafter, a block having a predetermined size (number of pixels) composed of a plurality of dequantized data is called a dequantized block. Therefore, the feature analysis unit 225 performs processing in units of dequantized blocks. Details of the feature analysis method in the feature analysis unit 225 will be described below using a flowchart. Note that in the embodiment, the size of the dequantized block (the number of dequantized data included therein) is the same as the size of the coding block (the number of pixels included therein).

[0077] <Description of Feature Analysis Method in Feature Analysis Unit 225> 10 is a flowchart showing the procedure of feature analysis processing in the feature analysis unit 225 in the embodiment. The processing in the feature analysis unit 225 will be described below with reference to the same drawing.

[0078] In S1001, the feature analysis unit 225 acquires the inverse quantization data from the inverse quantization unit 224 on a pixel-by-pixel basis, and calculates the brightness of the pixel.

[0079] In general, human vision has the property of being more likely to notice changes in dark areas and less likely to notice changes in bright areas. In this embodiment, this property is utilized, and it is assumed that pixels with lower brightness are pixels in which degradation due to encoding is more noticeable, and measures are taken to reduce degradation of such pixels.

[0080] In addition, in this embodiment, the level value of the inverse quantization data is used as an index of brightness (small level value = dark, large level value = bright). Note that the index of brightness in this embodiment is not limited to this method, and for example, the inverse quantization data in RGB format may be further color space converted to YUV format and the level value of the Y component may be used. Alternatively, the inverse quantization block may be divided into a plurality of regions, and the average value of the level values ​​of the inverse quantization data in each divided region may be used.

[0081] In S1002, the feature analysis unit 225 divides the inverse quantized block into a plurality of regions, and calculates the flatness for each of the divided regions.

[0082] In general, human visual perception has the property that changes in flat areas are easily noticed, but changes in complex areas are difficult to notice. In this embodiment, this property is utilized, and a pixel with a higher flatness is considered to be a pixel in which degradation due to encoding is more noticeable, and measures are taken to reduce degradation of the pixel.

[0083] In this embodiment, the variance value (variance of the level values ​​of the inverse quantization data) in the above-mentioned divided region is used as an index of flatness (variance greater than a threshold value = not flat, variance equal to or less than a threshold value = flat). Note that the index of flatness in this embodiment is not limited to this method, and for example, the dynamic range in each divided region (difference between the maximum and minimum values ​​of the level values ​​of the inverse quantization data) may be used.

[0084] In S1003, the feature analysis unit 225 performs a complexity determination process that indicates the features of the entire inverse quantization block. In this embodiment, the number of edges included in the inverse quantization block is used as an index of complexity (number of edges greater than a threshold = complex, number of edges equal to or less than the threshold = not complex). In addition, the feature analysis unit 225 determines that one edge is present when the difference value between adjacent inverse quantization data of the same color component is greater than a predetermined threshold. Note that the index of complexity in this embodiment is not limited to this method, and for example, the variance value of the inverse quantization block may be used.

[0085] In S1004, the feature analysis unit 225 performs feature classification on a pixel-by-pixel basis using the feature amounts acquired in S1001 to S1003. As will be described in detail later, the feature classification result (rank) has a value of 1 to 4, and the higher the value of a pixel, the more noticeable degradation is to the human eye. In other words, the rank obtained from the feature classification result can be said to be an index value indicating the noticeable degree of degradation, or an index value indicating the degree of importance.

[0086] Here, the feature classification method for each pixel in S1004 will be described with reference to FIG.

[0087] In S1101, the feature analysis unit 225 initializes several variables. Specifically, the feature analysis unit 225 assigns the brightness of the pixel of interest acquired in S1001 to the variable brightness, the flatness of the divided region to which the pixel of interest belongs acquired in S1002 to the variable flatness, and the complexity of the inverse quantization block to which the pixel of interest belongs acquired in S1003 to the variable complexity.

[0088] In S1102, the feature analysis unit 225 determines whether the complexity is greater than a preset complexity threshold th_comp. When the feature analysis unit 225 determines that the complexity of the inverse quantization block to which the pixel of interest belongs is greater than the threshold th_comp (if S1102 is Yes), the process proceeds to S1103. In S1103, the feature analysis unit 225 performs feature classification using only the feature amount of "flatness" to which the pixel of interest belongs, in order to place more importance on flat parts present in the inverse quantization block. On the other hand, when the feature analysis unit 225 determines that the complexity of the inverse quantization block is equal to or less than the threshold th_comp (if S1102 is No), in S1106 to S1108, the feature analysis unit 225 performs feature classification using two feature amounts of "brightness" and "flatness".

[0089] In this embodiment, the feature analysis unit 225 is configured to be able to select a feature to be used for the pixel from among multiple feature values, but it may be possible to always use the same feature value (for example, only brightness).

[0090] If the condition of S1102 is met, the feature analysis unit 225 determines in S1103 whether the flatness of the divided region to which the pixel of interest belongs is greater than a preset flatness threshold th_flat. If the condition of S1103 is met, the feature analysis unit 225 assigns "4" to the feature classification result rank of the pixel of interest in S1104. If the condition of S1103 is not met, the feature analysis unit 225 assigns "1" to the feature classification result rank of the pixel of interest in S1105.

[0091] On the other hand, if the condition of S1102 is not satisfied, in S1106, the feature analysis unit 225 determines whether or not the brightness of the pixel of interest is equal to or less than a preset brightness threshold th_bright. Furthermore, in S1107 to S1108, the feature analysis unit 225 determines whether or not the flatness of the divided region to which the pixel of interest belongs is greater than a preset flatness threshold th_flat, respectively, for cases where the condition of S1106 is satisfied and where it is not satisfied. Furthermore, in S1109 to S1112, the feature analysis unit 225 assigns "1" to "4" to the feature classification result rank of the pixel of interest, depending on the result of the determination in S1107 or S1108.

[0092] As mentioned above, the rank of the feature classification result indicates how noticeable degradation is to humans. For example, when the "complexity" of the inverse quantization block is small (when S1102 is No), a pixel with rank=1 is bright (high brightness) and not flat (low flatness), so degradation is less noticeable. On the other hand, a pixel with rank=4 is dark (low brightness) and flat (high flatness), so degradation is more noticeable.

[0093] In this embodiment, the same threshold value (th_flat, th_bright) is used for the pixels of each color component, but these threshold values ​​may be different for each color component. Also, in this embodiment, an example of performing feature classification for each pixel by threshold processing has been shown, but the feature classification method is not necessarily limited to this, and feature classification for each pixel can also be performed using techniques such as pattern matching and machine learning.

[0094] In this manner, the feature analysis unit 225 performs feature classification for each pixel, and outputs the feature classification result to the multiplexing unit 223 .

[0095] <Description of the multiplexing unit in the encoding system 220> 2 again, the multiplexing unit 223 in the encoding system 220 will be described. The multiplexing unit 223 receives the coded data, code length, and loss bits from the encoding unit 211E, and the coding mode flag and applied QP from the QP determination unit 215. Furthermore, the multiplexing unit 223 receives the feature classification result from the feature analysis unit 225. The multiplexing unit 223 then multiplexes these pieces of data in a predetermined format for each coding block.

[0096] Here, the encoding format in the case of the DPCM mode will be described with reference to Fig. 12. As described above, in this embodiment, the compression ratio for the encoding block is set to 6 / 10, so the size of the encoding block 1201 shown in Fig. 12 is equal to the target code amount of 144 bits.

[0097] The coding block 1201 includes a header section 1202 (3 bits) and a pixel data section 1203 (141 bits). The header section 1202 includes a coding mode flag 1204 (1 bit) indicating a coding mode, and a QP value section 1205 (2 bits) indicating a QP. The pixel data section 1203 includes a variable length code data section 1206, a loss bit section 1207, and a stuffing section 1208. Note that the stuffing section 1208 may not be present.

[0098] The coding mode flag and QP output from the QP determination unit 215 are stored in a coding mode flag 1204 (1 bit) and a QP value portion 1205 (2 bits) of the header portion 1202, respectively. In this embodiment, the value of the coding mode flag indicating the DPCM mode is set to 0. The variable-length code data (24 pixels) from the coding unit 211E is packed in order in bit units using the code length for each pixel, and stored in a variable-length code data portion 1206.

[0099] The lost bits output from the encoding unit 211E are stored in order in the lost bit section 1207 of the encoded block 1201. The lost bit section 1207 is an area immediately following the variable length code data section 1206. The method of storing the lost bits will be described later in detail.

[0100] When the amount of variable-length code data in the variable-length code data section 1206 is the same size as the pixel data section 1203, no area remains in the coding block 1201 to store the lost bits. In this case, the lost bits are not stored in the coding block 1201. When the combined area size of the header section 1202, the variable-length code data section 1206, and the lost bit section 1207 does not reach the target amount of codes, the remaining area is filled with a stuffing section 1208 (dummy bits).

[0101] Here, a method of storing lost bits in DPCM mode will be described with reference to Fig. 13(a)-(c). For simplicity, Fig. 13(a)-(c) are diagrams showing all bits of image data of an encoding block when QP=3 is selected, showing an arrangement of 24 pixel data in the horizontal direction and a bit depth of 10 bits in the vertical direction. Fig. 13(a)-(c) also shows the feature classification result of each pixel and the number of stored lost bits in each pixel, showing an example where rank=1 is 7 pixels, rank=2 is 6 pixels, rank=3 is 5 pixels, and rank=4 is 6 pixels.

[0102] FIG. 13(a) shows all bits of the encoded block before the multiplexer 223 stores the loss bits, with the most significant 7 bits of each pixel being quantized data and the least significant 3 bits being loss bits.

[0103] The multiplexing unit 223 determines the number of stored loss bits according to the feature classification result for each pixel, and stores the loss bits in the loss bit area shown in Fig. 13(a). In this embodiment, the number of stored loss bits store_bits[pix] for each pixel is calculated by the following formula (8). store_bits[pix] = free_num[qp] × weight[pix] …(8) Here, pix indicates a pixel number in the coding block (any of 0 to 23 in the embodiment). Also, free_num[qp] indicates the number of free bits in the selected QP. Also, weight[pix] is a decimal number less than 1 (called a weighting coefficient) indicating the degree of noticeable degradation of each pixel. In other words, equation (8) means that the number of free bits free_num[qp] is assigned to each pixel according to the weighting coefficient weight[pix].

[0104] In this embodiment, the weighting coefficient weight[pix] is calculated by the following equation (9). weight[pix] = rank[pix] / SUM(rank(pix)) …(9) Here, rank[pix] is the feature classification result for each pixel, and SUM(rank[pix]) is the sum of rank[pix] for all pixels in the coding block.

[0105] If the value of the number of stored lopsided bits store_bits[pix] calculated by equations (8) and (9) exceeds the value of QP, the multiplexer 223 clips the value of store_bits[pix] to the value of QP. This is because it is not possible to store more lopsided bits in each pixel than the value of QP (3 or more in this specific example).

[0106] In short, the number of free bits when storing encoded data for a fixed length is given by free_num[qp]. This means that the higher the rank of a pixel, the more bits are allocated to hold the lost bits.

[0107] The above-mentioned method of calculating the number of lost bits stored for each pixel and the method of calculating the weighting coefficient are merely examples and are not limited to these methods. For example, the number of lost bits stored according to the feature classification result of each pixel may be determined in advance, such as storing 1 bit for a pixel with rank=1 and storing 3 bits for a pixel with rank=3. In addition, in this embodiment, the weighting coefficient is calculated using only the feature classification result of each pixel, but considering that degradation is easily noticeable at the boundary part of the coding block, it is also possible to adjust the weighting coefficient so that it is larger for pixels closer to both ends of the coding block.

[0108] Fig. 13(b) is a diagram showing all bits of the coded block after the multiplexing unit 223 stores the lost bits. Fig. 13(b) also shows an example in which the number of free bits free_num[qp(=3)] is "52". It can be seen that the more noticeable the degradation, the more lost bits are stored in the pixel, such as 3 bits for the pixel with rank=4, 2 bits for the pixel with rank=3, 1 bit for the pixel with rank=2, and 0 bit for the pixel with rank=1.

[0109] It should be noted here that the total value of the number of stored loss bits store_bits[pix] in all pixels in the coding block shown in Fig. 13(b) is not necessarily equal to the number of free bits free_num[qp]. This is because the value after the decimal point of free_num[qp] × weight[pix] on the left side of equation (8) is truncated because store_bits[pix] on the right side of equation (8) is an integer.

[0110] Therefore, in order to store all the bits of the free bit number free_num[qp] in the coding block without omission, the residual bits (called residual_loss) caused by the calculation error of formula (8) must be further stored in the loss bit area. In this embodiment, the residual bits residual_loss are calculated by the following formula (10). residual_loss = free_num[qp] - SUM(store_bits[pix]) …(10) Here, SUM(rank[pix]) is the total value of store_bits[pix] for all pixels in the coding block.

[0111] The multiplexer 223 further stores the residual bits, residual_loss, calculated by equation (10), in the loss bit area shown in Fig. 13(b). Hereinafter, a method of storing the residual bits in the loss bit area by the multiplexer 223 will be described.

[0112] Fig. 13(c) is a diagram showing all bits of the coded block after storing the residual bits calculated by the multiplexing unit 223 using equation (10). Fig. 13(c) shows an example in which the residual bits are residual_loss=18, and the numbers (0 to 17) written in the loss bit storage areas indicate the storage priority of the residual bits.

[0113] The multiplexing unit 223 stores the remaining bits in the portion of Fig. 13(b) where no lost bits have been stored yet. At this time, the remaining bits are stored in the more significant bits of each pixel from both ends toward the center of the coding block, such as bit 1 of pixel number 0, bit 0 of pixel number 23, bit 0 of pixel number 1, etc. In this way, it is possible to reduce image degradation at the boundaries of the coding blocks.

[0114] The method of storing the surplus bits is not limited to the above-mentioned method. For example, it is also possible to store the surplus bits preferentially from the center of the coding block. Alternatively, it is also possible to store the surplus bits preferentially from pixels in which degradation is easily noticeable according to the feature classification results of each pixel.

[0115] In this way, the multiplexer 223 can reduce degradation perceived by the human eye by preferentially storing lost bits of pixels in which image degradation is likely to be noticeable in the lost bit section 1207 shown in FIG.

[0116] Next, the coding format in the PCM mode will be described with reference to Fig. 14. In this embodiment, the size of a coding block 1401 is 144 bits, which is the target code amount, as in the DPCM mode.

[0117] The coding block 1401 includes a coding mode flag 1402 (1 bit) indicating a coding mode and a quantized data section 1403 (143 bits). The coding mode flag from the QP determination unit 215 is stored in the coding mode flag 1402 (1 bit). In this embodiment, the value of the coding mode flag indicating the PCM mode is assumed to be 1. The quantized data (24 pixels) from the coding unit 211E is stored in the quantized data section 1403, which is packed bit by bit in sequence using the code length for each pixel. In this embodiment, the QP in the PCM mode is 4, but as described above, one pixel among the pixels of the coding block is quantized with QP=5 by increasing the QP by one step. Therefore, 23 pixels out of the 24 pixels are quantized to 6 bits per pixel, and one pixel is quantized to 5 bits. Therefore, a total of 23×6+5=143 bits of quantized data are stored in the quantized data section 1403.

[0118] In this manner, the coded data multiplexed in accordance with the coding mode for each coding block is output as stream data to the output terminal 202 and input to the memory 30.

[0119] As described above, in the image encoding unit 20 according to the embodiment, when the amount of code obtained by encoding one encoding block in pixel units is less than the fixed length, the more noticeable the degradation of a pixel is by encoding, the more the bits lost by quantization are stored in the free space. As a result, degradation of image quality can be suppressed compared to the past.

[0120] <Details of the image decoding unit 40> The configuration and operation of the image decoding unit 40 that decodes the coded data generated by the image coding unit 20 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing a detailed configuration of the image decoding unit 40. The image decoding unit 40 can decode the coded data held in the memory 30.

[0121] The image decoding unit 40 shown in Fig. 4 includes a separation unit 403, a variable length decoding unit 404, an adder 405, a selector 406, an inverse quantization unit 407, a prediction unit 408, a feature analysis unit 409, and a loss bit addition unit 410. The image decoding unit 40 may be configured as integrated hardware including a dedicated device, a logic circuit, a memory, etc., or may be configured in a distributed manner using a plurality of devices, etc. Alternatively, the image decoding unit 40 may be implemented by a memory, a processing program stored in the memory, and a computer such as a CPU that executes the processing program.

[0122] The stream data (encoded data for each encoding block) generated by the image encoding unit 20 and temporarily stored in the memory 30 is input to a separator 403 via an input terminal 401 .

[0123] The separator 403 separates the coding mode flag, QP, encoded data, and loss bits from the beginning of the input stream data, and sequentially outputs them for each processing cycle.

[0124] The separator 403 outputs the coded data (quantized data in the case of PCM) to the variable length decoding unit 404 and the selector 406. The variable length decoding unit 404 performs variable length decoding on the input coded data, outputs the decoded data to the adder 405, and outputs the code length of the code at that time to the separator 403. The separator 403 accumulates the code length until the variable length decoding unit 404 performs variable length decoding for 24 pixels included in one coding block. Then, the separator 403 subtracts the accumulated value from the fixed length to determine the storage position of the loss bit part in the fixed length data (to be precise, the total size of the loss bit part 1207+the stuffing part 1208), and outputs the data including the loss bit part to the loss bit adding unit 410.

[0125] The adder 405 adds predicted data from a prediction unit 408 (described later) and the decoded data to obtain a decoded value, and outputs the decoded value to a selector 406 and the prediction unit 408 .

[0126] The prediction unit 408 generates prediction data of the current image data using the decoded value from the adder 405. The method of generating the prediction data is similar to the method of generating prediction data in the prediction unit 302 of the encoding unit 211 described above.

[0127] Depending on the encoding mode flag, a selector 406 selects the encoded data from the separator 403 in PCM mode, and selects the data output from the adder 405 in DPCM mode, and outputs it to an inverse quantizer 407 as quantized data.

[0128] The inverse quantization unit 407 generates inverse quantized data by inverse quantizing the quantized data from the selector 406 using the QP input from the separation unit 403, and outputs the inverse quantized data to the feature analysis unit 409 and the loss bit addition unit 410. Note that when the coding mode is PCM, the separation unit 403 outputs the QP as "4".

[0129] The feature analysis unit 409 has a buffer memory for storing one coding block's worth of dequantized data. The feature analysis unit 409 performs feature analysis on the dequantized data stored in the buffer in the same manner as the feature analysis unit 225 of the main coding system 220, and classifies the features of each pixel. The feature analysis unit 409 also outputs the feature classification result to the loss bit addition unit 410.

[0130] The lossy bit adding unit 410 also has an internal buffer memory for storing the dequantized data and lossy bits for the encoded blocks. The lossy bit adding unit 410 adds the lossy bits output from the separation unit 403 to the dequantized data to generate decoded image data. The method of adding the lossy bits is the same as that used by the multiplexing unit 223 of the encoding system 220 (in this embodiment, the non-lossy bit portion in FIG. 13(c) is decoded).

[0131] Here, a specific example of the lossy bit addition process of the lossy bit addition unit 410 will be described. Assume that the pixel to be decoded is the first pixel in FIG. 13(c). The most significant 7 bits of the 10 bits are obtained by inverse quantization. Based on the feature classification result from the feature analysis unit 409, the lossy bit addition unit 410 can determine that 2 bits should be added to the corresponding pixel. The lossy bit addition unit 410 reads the corresponding 2 bits from the buffer, and adds the most significant 2 bits (bit2, bi) that are dominant for the number of loss bits "3" of the pixel of interest. t 1). In addition, the loss bit addition unit 410 adds a preset value (for example, "0") to bit 0, the least significant bit that cannot be added. This allows the loss bit addition unit 410 to decode 10 bits of pixel data of the pixel of interest. The loss bit addition unit 410 outputs 10 bits of data representing the pixel of interest after the addition to the output terminal 402.

[0132] As described above, according to this embodiment, even if the image data to be encoded contains sharp edges or the like and it is necessary to assign a large code length to the edge components, the loss bits are preferentially stored in pixels where image degradation is likely to be noticeable, thereby reducing the degradation perceived by humans.

[0133] The image encoding unit 20 may perform main encoding by the main encoding system 220 without performing temporary encoding by the temporary encoding system 210. In this case, the encoding unit 211E of the main encoding system 220 may perform quantization according to, for example, a predetermined quantization parameter. When quantization according to a predetermined quantization parameter is performed, the image decoding unit 40 may also perform inverse quantization according to the same quantization parameter.

[0134] Also, in the above embodiment, the temporary encoding system 210 has four encoding units 211A to 211D, each of which performs encoding in the DPCM mode with QP = 0, 1, 2, 3. And when the target code amount is exceeded even with QP = 3, the encoding system 220 performs encoding in the PCM mode with QP = 4. However, PCM encoding is not necessarily essential. For example, define the bit depth of one pixel as M and the target code amount of one encoding block as Ctarget. Also, assume that the value of QP for which the total code amount per encoding block in the DPCM mode is at most the target code amount even in the worst case is "N" (N < M - 1). And if the encoded data in the DPCM mode with QP = N is to be accepted as the final data, the temporary encoding system 210 is provided with N + 1 encoding units with QP = 0 to N. Then, the QP determination unit 215 determines the code amount of each of the N + 1 encoding units, and determines the maximum QP as the applied QP when the total code amount of one encoding block is less than or equal to the target code amount Ctarget. The encoding system 220 may perform encoding according to the determined applied QP in accordance with the DPCM mode.

[0135] Also, in the above embodiment, the index value (rank) representing the ease of conspicuous degradation is determined from three factors: brightness, flatness, and complexity, but any one of them may be used to simplify the process. For example, the value of rank may be increased as the brightness is lower. Also, the value of rank may be increased as the degree of flatness is higher.

[0136] Also, it is desirable to apply the above embodiment to an imaging device represented by a digital camera or a video camera. For example, a control circuit including an image encoding unit 20 is mounted on the imaging unit side, and a control circuit including an image decoding unit 40 is mounted on the side that performs various image processes in response to the imaging result. As a result, even if the imaging resolution is increased, normal communication can be performed without straining the communication bandwidth between the two control circuits.

[0137] [Second Embodiment] The feature analysis unit 225 according to the second embodiment can obtain indices indicating brightness and flatness in one process. Specifically, the feature analysis unit 225 according to the second embodiment performs frequency conversion on an inverse quantization block composed of a plurality of inverse quantization data, and performs feature classification using the coefficients after the frequency conversion.

[0138] In the second embodiment, similarly to the first embodiment, loss bits are preferentially stored in pixels where image degradation is easily noticeable, thereby reducing degradation of the human visual perception. In addition, the second embodiment will be described in terms of differences from the first embodiment, and descriptions of the same parts as the first embodiment will be omitted.

[0139] The method of feature analysis in the feature analysis unit 225 in the second embodiment will be described with reference to FIG.

[0140] In S1501, the feature analysis unit 225 acquires the inverse quantized data from the inverse quantization unit 224, and generates an inverse quantized block constituted by a plurality of inverse quantized data.

[0141] In S1502, the feature analysis unit 225 performs one wavelet transform in the horizontal direction on the inverse quantized block generated in S1501, and divides the inverse quantized block into an L band (L means low frequency components) and an H band (H means high frequency components). In this embodiment, a Haar wavelet transform that is easy to implement is used, but the number of times, direction, method, etc. of the wavelet transform are not limited to the above-mentioned method. For example, the wavelet transform may be performed multiple times in the horizontal direction to further decompose the inverse quantized block into finer subbands. In addition, when the shape of the coding block is two-dimensional, a DWT using a 5×3 tap filter adopted in JPEG2000, etc. may be used.

[0142] Here, with reference to FIG. 16, a specific example of each subband generated by the feature analysis unit 225 will be described.

[0143] Figure 16(a) shows the image data (encoding block) to be encoded in this embodiment, and similar to the diagram shown in Figure 5(a), pixel numbers 0 to 23 are assigned in raster scan order, R0, G0, B0, R1, G1, B1, etc.

[0144] Fig. 16(b) is a diagram showing the coded block after the feature analysis unit 225 applies wavelet transform to the coded block shown in Fig. 16(a). At this time, the feature analysis unit 225 obtains the coefficients in each subband by the following formulas (11) and (12). In this embodiment, C in formulas (11) and (12) means any one of the color components R, G, or B, and the pixel number pix is ​​an even number from 0 to 6. LC[pix] = (C[pix] + C[pix+1]) / 2 …(11) HC[pix] = (C[pix] - C[pix+1]) / 2 …(12) The feature analysis unit 225 according to the second embodiment uses the L-band coefficient (low-frequency component) shown in Fig. 16(b) as an index indicating the brightness of each pixel, and the H-band coefficient (high-frequency component) as an index indicating the flatness of each pixel. Note that, in equations (11) and (12), one L-band coefficient and one H-band coefficient are calculated for each two adjacent pixels. Therefore, the feature analysis unit 225 according to the second embodiment has feature quantities (brightness and flatness) of the same value for each two adjacent pixels.

[0145] In this manner, in S1502, the feature analysis unit 225 calculates the feature amounts of brightness and flatness of each pixel.

[0146] In S1503, the feature analysis unit 225 performs feature classification using the calculated feature amount. The feature classification method is the same as the method shown in FIG.

[0147] In this embodiment, an example has been described in which the wavelet transform is used as the frequency transform method performed by the feature analysis unit 225, but the present invention is not necessarily limited to this. For example, the feature analysis unit 225 can also perform orthogonal transform such as DCT (discrete cosine transform) or KLT transform on the inverse quantization block, and use the obtained coefficient as one of the feature quantities.

[0148] [Third embodiment] In the first embodiment described above, when the coding mode is DPCM, the coding block of interest (24 pixels) is quantized with QP=0 to 3, and variable-length coding is performed. Then, when the obtained code amount is less than the target code amount (144 bits) and there is a free space for the target code amount, the bits lost by quantization of the pixels whose deterioration is easily noticeable are preferentially stored in the free space. Also, when the coding mode is PCM, quantization is performed with QP=4. Then, fixed-length data of a total of 144 bits is generated, including 1 bit indicating that the coding mode is PCM, 5 bits for one pixel out of the 24 pixels, and 6 bits for the other 23 pixels. In other words, when the coding mode is PCM, the noticeable deterioration of each pixel is not taken into consideration.

[0149] In the third embodiment, an example is described in which fixed-length code data is generated in consideration of how easily degradation is noticeable even when the coding mode is PCM. The device configuration is the same as in Fig. 2, and the difference is the QP determination unit 215 of the provisional coding system 210, the coding unit 213E of the main coding system 220, and the multiplexing unit 223, and the rest is the same as in the first embodiment. In addition, the coding content when the coding mode is DPCM is the same as in the first embodiment, and the data structure of the code data generated in that case is also the same as in Fig. 12, so the description thereof is omitted.

[0150] FIG. 17 shows a flowchart of the encoding mode and QP determination process by the QP determination unit 215 in the provisional encoding system 210 in the third embodiment. The only difference from FIG. 8 in the first embodiment is S801', and the rest is the same. In the first embodiment, MAX_QP+1 is substituted for the variable sel_qp, whereas in the third embodiment, MAX_QP+3 is substituted for the variable sel_qp. Since MAX_QP is "3", the variable sel_qp is initialized to "6". Note that the value for initializing the variable sel_qp should be larger than MAX_QP+1 and smaller than 10, which is the bit depth of the image data. Therefore, it should be understood that "+3" shown here is merely an example.

[0151] Now, from the explanation so far, it can be understood that if the QP determination unit 215 performs the process of Figure 17 without performing the process of S803 even once, the coding mode for the coding block of interest will be PCM, and the applied QP will be determined to be 6.

[0152] Therefore, the encoding unit 211E in the encoding system 220 encodes each pixel in the encoding block in two 6 and outputs the quantized data (4 bits) as code data to the multiplexing unit 223. In addition, the encoding unit 211E outputs the loss bits (6 bits) to the multiplexing unit 223.

[0153] In the third embodiment, when the coding mode for the coding block is PCM mode, the coding data format has a structure as shown in Fig. 18. The coding format in PCM mode will be described below with reference to the same figure. In the third embodiment, the size of the coding block 1401 is 144 bits, which is the target code amount, as in the case of DPCM mode.

[0154] The coding block 1401 is composed of a coding mode flag 1402 (1 bit) indicating the coding mode and a pixel data section 1404 (143 bits). The coding mode flag is stored in the coding mode flag 1402 (1 bit). In this embodiment, the value of the coding mode flag indicating the PCM mode is 1, as in the first embodiment. The quantized data (24 pixels) from the coding unit 211E is stored in the quantized data section 1403, which is packed bit by bit in sequence using the code length for each pixel. In this third embodiment, the QP in the PCM mode is 6, but as described above, one pixel in the pixel block of the coding block is quantized with QP=7, which is one step higher. Therefore, one pixel out of the 24 pixels is quantized to 3 bits, and the remaining 23 pixels are quantized to 4 bits. Therefore, a total of 3+23×4=95 bits of quantized data is stored in the quantized data section 1403.

[0155] As in the case of DPCM mode, the loss bits output from the encoding unit 211E are stored in order in the loss bit section 1405 of the encoding block 1401. As in the case of PCM mode, as in the case of the quantized data section 1403, the size of the loss bit section 1405 is fixedly determined according to QP. In the case of the third embodiment, the size of the loss bit section 1405 is 48 bits obtained by subtracting the quantized data section 1403 (95 bits) from the pixel data section 1404 (143 bits). Also, in the case of PCM mode, the number of free bits free_num[qp] for the target code amount is equal to the size of the loss bit section 1405, which is 48 bits as described above.

[0156] Here, a method for storing lost bits in the PCM mode in the third embodiment will be described with reference to Fig. 19(a) to (c). Fig. 19(a) to (c) are diagrams showing all bits of image data of an encoding block when QP=6 is selected, and as described above, one pixel located at the left end of the encoding block is quantized by QP=7, moving up one step.

[0157] 19(a)-(c), like Fig. 13(a)-(c), show 24 pixel data arranged horizontally and 10-bit bit depth vertically. Fig. 19(a)-(c) also show the feature classification result of each pixel and the number of lost bits stored in each pixel, and show an example where rank=1, which is an index representing the complexity, is 7 pixels, rank=2 is 6 pixels, rank=3 is 5 pixels, and rank=4 is 6 pixels.

[0158] 19(a) shows all bits of the coding block before the multiplexer 223 stores the loss bits, where the most significant 4 bits of each pixel are quantized data and the least significant 6 bits are loss bits. Note that for one pixel located at the left end of the coding block, the most significant 3 bits are quantized data and the least significant 7 bits are loss bits.

[0159] As in the case of the DPCM mode, the multiplexing unit 223 determines the number of loss bits to store for each pixel according to the feature classification result, and stores the loss bits in the loss bit area shown in Fig. 19(a). That is, the multiplexing unit 223 determines the number of loss bits to store for each pixel, store_bits[pix], using equations (8) and (9).

[0160] Fig. 19(b) is a diagram showing all bits of the encoded block after the multiplexing unit 223 stores the lost bits. Fig. 19(b) also shows an example in which the number of free bits free_num[qp(=6)] is 48. It can be seen that the more noticeable the degradation, the more lost bits are stored in pixels, such as 3 bits for pixels with rank=4, 2 bits for pixels with rank=3, 1 bit for pixels with rank=2, and 0 bit for pixels with rank=1.

[0161] As in the case of the DPCM mode, the multiplexer 223 further adds the residual bits residual_loss calculated by the formula (10) to the loss bit area shown in Fig. 19(b). The method of storing the residual bits of the multiplexer 223 in the loss bit area will be described below.

[0162] Fig. 19(c) is a diagram showing all bits of the coded block after storing the residual bits calculated by the multiplexing unit 223 using equation (10). Fig. 19(c) shows an example in which the residual bits are residual_loss=14, and the numbers (0 to 17) written in the loss bit storage areas indicate the storage priority of the residual bits.

[0163] The multiplexing unit 223 stores the remaining bits in the portion of Fig. 19(b) where no lost bits have been stored yet. At this time, the remaining bits are preferentially stored in the upper bit positions of each pixel from both ends of the coding block toward the center, such as bit 5 of pixel number 0, bit 3 of pixel number 23, bit 5 of pixel number 1, etc. In this way, it is possible to reduce image degradation at the boundary portions of the coding blocks.

[0164] The method of storing the surplus bits is not limited to the above-mentioned method. For example, it is also possible to store the surplus bits preferentially from the center of the coding block. Alternatively, it is also possible to store the surplus bits preferentially from pixels in which degradation is easily noticeable according to the feature classification results of each pixel.

[0165] In this way, the multiplexer 223 can reduce degradation perceived by humans' vision by preferentially storing lost bits of pixels in which image degradation is likely to be noticeable in the lost bit section 1405 shown in FIG.

[0166] The processing of the image encoding unit 20 in the third embodiment has been described above. The decoding processing of the image decoding unit 40 when the encoding mode in the third embodiment is DPCM is the same as that in the first embodiment. Also, when the encoding mode is PCM, the image decoding unit 40 only adds a process of adding loss bits to the quantized data in the same way as in the case of DPCM, and therefore the description thereof will be omitted.

[0167] Also, in order to reduce the hardware scale of the provisional coding system, the prediction section, the variable length coding section, and the like of the main coding system, coding may be performed more simply in the PCM mode only.

[0168] As described above, according to the third embodiment, image quality improvement can be expected when the coding mode is PCM compared to the first embodiment. The reason is that according to the third embodiment, even when the coding mode is PCM, coding is performed taking into consideration how easily degradation of each pixel in the coding block of interest is noticeable.

[0169] The second embodiment described above can also be applied to the third embodiment.

[0170] [Fourth embodiment] In recent years, imaging devices have become known that can control the exposure time for each pixel and output images with different sensitivities according to the length of the exposure time. By synthesizing images with different sensitivities obtained from this imaging device, an image with a wider dynamic range than that of conventional imaging devices can be obtained in one shot. In the fourth embodiment, a case where an image obtained from such an imaging device is to be encoded will be described.

[0171] 20 is a block diagram showing the configuration of an image processing device 200 equipped with the above-mentioned imaging device. The image processing device 200 includes an imaging unit 50, an image encoding unit 20, a memory 30, an image decoding unit 40, an image synthesis unit 60, a development processing unit 70, a recording processing unit 80, and a recording medium 90.

[0172] The functions of each block of the image processing device 200 may be implemented by either software or hardware, except for physical devices such as the imaging unit 50, the memory 30, and the recording medium 90. For example, the functions of each block may be implemented by hardware such as a dedicated device, a logic circuit, and a memory. Alternatively, the functions of each block may be implemented by a memory, a processing program stored in the memory, and a computer such as a CPU that executes the processing program. The image processing device 200 can be implemented as, for example, a digital camera, but can also be implemented as other devices. For example, the image processing device 200 can be implemented as any information processing terminal or imaging device, such as a personal computer, a mobile phone, a smartphone, a PDA, a tablet terminal, or a digital video camera.

[0173] The imaging unit 50 is an image sensor such as a CCD or CMOS sensor, which photoelectrically converts an optical image formed on an imaging element via an optical system (not shown), and outputs a digital image signal (i.e., RAW data) by A / D converting the resulting analog signal.

[0174] The imaging unit 50 has two operation modes: an operation mode (normal mode) in which the exposure time of each pixel is the same, and an operation mode (HDR mode) in which the exposure time varies from pixel to pixel. When the imaging unit 50 operates in the normal mode, the imaging unit 50 outputs one type of image having a predetermined sensitivity in one shooting. When the imaging unit 50 operates in the HDR mode, the imaging unit 50 outputs different types of images for each of a plurality of sensitivities in one shooting. These sensitivities and operation modes can be controlled by a CPU (not shown) or the like provided in the image processing device 200. In addition, a user operating the image processing device 200 can change these sensitivities and operation modes via the CPU (not shown) provided in the image processing device 200.

[0175] In this embodiment, when the imaging unit 50 operates in HDR mode, two types of images, a low-sensitivity image exposed for a short time and a high-sensitivity image exposed for a long time, are output from the imaging unit 50. In this case, since the amount of data output from the imaging unit 50 is twice that in normal mode, a downstream processing block (such as the image encoding unit 20) may have a plurality of these processing blocks depending on the processing performance. In addition, the imaging unit 50 may be configured to be capable of outputting three or more types of images, such as a low-sensitivity image, a medium-sensitivity image, and a high-sensitivity image.

[0176] In this embodiment, the imaging unit 50 is configured to be separated from the image encoding unit 20, the memory 30, etc., but, for example, a configuration in which some of the blocks that make up the image processing device 200 are mounted on a logic circuit unit or the like contained within the imaging unit 50 may also be used.

[0177] When the imaging unit 50 operates in the normal mode, the image encoding unit 20 operates in the same manner as in the first to third embodiments. That is, the image encoding unit 20 performs feature analysis on the input image data and reduces loss of pixels contained in dark areas, flat areas, etc. in the image, thereby making visual degradation of the image after compression encoding less noticeable.

[0178] On the other hand, although details will be described later, when the imaging unit 50 operates in HDR mode, the image encoding unit 20 compression-encodes the low-sensitivity image and the high-sensitivity image output from the imaging unit 50. Furthermore, the image encoding unit 20 performs feature analysis according to the sensitivity of the input image data, thereby making visual degradation of the image after compression encoding less noticeable more effectively than in the first to third embodiments.

[0179] When the imaging unit 50 operates in normal mode, the memory 30 stores the encoded data output from the image encoding unit 20 in the same manner as in the first to third embodiments. When the imaging unit 50 operates in HDR mode, the memory 30 stores the encoded data of the low-sensitivity image and the encoded data of the high-sensitivity image output from the image encoding unit 20.

[0180] When the imaging unit 50 operates in normal mode, the image decoding unit 40 reads out the encoded data stored in the memory 30 in the same manner as in the first to third embodiments, decodes the data by a predetermined decoding method, and outputs the decoded image data to a subsequent stage. When the imaging unit 50 operates in HDR mode, the image decoding unit 40 reads out the encoded data of the low-sensitivity image and the encoded data of the high-sensitivity image stored in the memory 30, decodes the data into a low-sensitivity image and a high-sensitivity image, respectively, and outputs the decoded data to a subsequent stage.

[0181] When the imaging unit 50 operates in HDR mode, the image synthesis unit 60 generates a wide dynamic range image (called an HDR image) by synthesizing the low-sensitivity image and the high-sensitivity image output from the image decoding unit 40. When the imaging unit 50 operates in normal mode, the image synthesis unit 60 outputs the image output from the image decoding unit 40 to the subsequent stage as is.

[0182] The development processing unit 70 executes development processing on the image (RAW data) output by the image synthesis unit 60 to generate recording image data. More specifically, the development processing unit 70 executes color interpolation processing called DeBayer (demosaic) processing, color space conversion processing, noise suppression processing, optical distortion correction, enlargement / reduction processing, etc., to generate recording image data represented by luminance and color difference such as YCbCr4:2:2.

[0183] In this embodiment, the image synthesis unit 60 performs image synthesis on the image output from the image decoding unit 40. However, for example, the image synthesis unit 60 may be located downstream of the development processing unit 70 and perform image synthesis on the image after development processing output from the development processing unit 70.

[0184] The recording processing unit 80 performs compression processing on the recording image data output from the development processing unit 70, and stores the compressed recording image data on the recording medium 90. For example, if the recording image data is a still image, the recording processing unit 80 compresses the recording image data using JPEG compression, HEIF compression, etc. If the recording image data is a moving image, the recording processing unit 80 compresses the recording image data using MPEG-2, H.264, H.265, etc.

[0185] FIG. 21 is a diagram showing an example of the configuration of the imaging unit 50 in this embodiment.

[0186] A plurality of unit pixels 501 formed of photoelectric conversion elements such as photodiodes are arranged in a matrix in the pixel section 500. In the pixel section 500 of this embodiment, the exposure time is controlled on a pixel-by-pixel basis by a timing control section 508 described later, and there are two types of pixels: long exposure pixels, which have a long exposure time setting, and short exposure pixels, which have a short exposure time setting.

[0187] A vertical scanning circuit 502 performs timing control for sequentially reading out pixel signals from the pixel section 500 during one frame. In general, pixel signals are sequentially read out row by row from the top row to the bottom row during one frame period.

[0188] The column amplifier 503 is composed of a plurality of column amplifiers configured for each column of the pixel section 500, and is used to electrically amplify signals read out from the unit pixels 501. By amplifying the signal with the column amplifier 503, the signal level of the pixel is amplified against noise generated in the downstream column ADC 504, and the S / N ratio can be equivalently improved.

[0189] The column ADC 504 is made up of a plurality of circuits configured for each column of the pixel unit 500, and converts the signal amplified by the column amplifier 503 into a digital signal. The pixel signals converted into digital signals are sequentially read out by a horizontal transfer circuit 505 and input to a signal processing unit 506.

[0190] The signal processing unit 506 is a circuit that performs signal processing digitally, and in addition to adding a certain offset value in digital processing, it can easily perform gain calculations by performing shift calculations and multiplications. Furthermore, if the pixel unit 500 has a structure having a light-shielded pixel region, a digital black level clamp operation using this may be performed. Furthermore, the signal processing unit 506 has a serializer function, and converts a multi-bit input parallel signal in the signal processing unit 506 into a serial signal. The signal processing unit 506 converts this serial signal into an LVDS signal or the like, and outputs it as pixel information to an external device.

[0191] The memory 507 has a function of temporarily storing signals that are read out from the pixel unit 500 and processed by the column amplifier 503 , the column ADC 504 , and the signal processing unit 506 .

[0192] The timing control unit 508 supplies an operating clock signal and a timing signal to each block in the imaging unit 50, thereby controlling the overall operation of the imaging unit 50. In addition, the timing control unit 508 can control the exposure time of each pixel by controlling control signals 516 and 517, which will be described later.

[0193] Next, the flow of signals from the pixel section 500 to the horizontal transfer circuit 505 will be described with reference to FIG.

[0194] The photoelectric conversion unit 513 converts light into electric charges by photoelectric conversion. The transfer switch 515 transfers the electric charges generated by the photoelectric conversion unit 513 to a downstream circuit.

[0195] Control signals 516 and 517 are control signals for controlling the exposure time of each pixel, and are a control signal for short-term exposure and a control signal for long-term exposure, respectively. These two control signals are wired for each unit pixel 501, and either the control signal 516 or the control signal 517 is connected to the transfer switch 515 of each unit pixel. In the example shown in FIG. 22, the control signal 516 for short-term exposure is connected to the transfer switch 515, so that the unit pixel 501 shown in FIG. 22 is a short-term exposure pixel. The timing control unit 508 can control the exposure time for each pixel by controlling the ON / OFF interval of these two signals.

[0196] The charge holding unit 518 temporarily holds the charge generated by the photoelectric conversion unit 513. The pixel amplifier 519 converts the charge held in the charge holding unit 518 into a voltage signal and outputs it to the column amplifier 503 through a vertical line 520. The current control unit 521 controls the current of the vertical line 520.

[0197] The column ADC 504 includes an A / D conversion unit 522, memories 523 and 524, and a subtractor 525. The A / D conversion unit 522 performs A / D conversion on the output of the column amplifier 503. The memories 523 and 524 temporarily store the digital signal converted by the A / D conversion unit 522. The memory 523 stores the pixel signal read out from the photoelectric conversion unit 513 and the noise signal of the readout circuit unit (the circuit from the charge holding unit 518 to the A / D conversion unit 522). The memory 524 stores the noise signal of the readout circuit unit. The subtractor 525 subtracts the output of the memory 524 from the output of the memory 523, and outputs the subtraction result to the horizontal transfer circuit 505 as a pixel signal.

[0198] Next, the pixel array structure of the pixel section 500 will be described with reference to Fig. 23. The pixel array of the pixel section 500 shown in Fig. 23 is characterized in that R pixels, Gr pixels, Gb pixels, and B pixels are arranged in 2x2 rows, with the same color arranged in a 2x2 array. A total of 4x4 pixels are set as the minimum unit, and this minimum unit is repeatedly arranged.

[0199] Setting of exposure times in the pixel array shown in Fig. 23 will be described with reference to Fig. 24. As shown in Fig. 24, the horizontal direction is x and the vertical direction is y, with column numbers represented by the x coordinate and row numbers represented by the y coordinate. Numbers in parentheses indicate the coordinates indicating the position of each pixel. In Fig. 24, gray pixels represent short exposure pixels and white pixels represent long exposure pixels. In this embodiment, short exposure pixels and long exposure pixels are set in a zigzag pattern in the column direction as shown in Fig. 24.

[0200] For example, the coordinate settings of the four R pixels at the top left corner of FIG. 24 are as follows: R(1,1): Short exposure pixel R(2,1): Long exposure pixel R(1,2): Long exposure pixel R(2,2): Short-exposure pixel In this way, short exposure pixels and long exposure pixels are set alternately in each column, and short exposure pixels and long exposure pixels are set alternately in each row. If we trace only the short exposure pixels in the y direction in the first and second columns, we get the following from the top: In the first row, the first column is the short exposure pixel, In the second row, the second column is a short exposure pixel, In the third row, the first column is a short-exposure pixel, In the fourth row, the second column is a short exposure pixel In this way, the short-exposure pixels are arranged in a zigzag pattern. Similarly, if we trace only the long-exposure pixels in the y direction in the first and second columns, we get the following from the top: In the first row, the second column is a long exposure pixel, In the second row, the first column is a long exposure pixel, In the third row, the second column is a long exposure pixel, In the 4th row, the 1st column is a long exposure pixel In this way, the long exposure pixels are arranged in a zigzag pattern.

[0201] As described above, the pixel arrangement and its exposure time are set in units of 2x2 pixels of the same color, with two short-exposure pixels and one long-exposure pixel being arranged among these four pixels.

[0202] Next, the image output format when the imaging unit 50 operates in the normal mode will be described with reference to FIG.

[0203] When the imaging unit 50 operates in normal mode, the exposure time of each pixel is the same. In this case, the imaging unit 50 outputs the average value of 2x2 pixels of the same color, which are shown by gray diamonds in Fig. 25(a). More specifically, the output of the imaging unit 50 shown in Fig. 25(b) can be derived as follows.

[0204]

number

[0205] 25(b), the format of the image output from the imaging unit 50 is nothing but the format of RAW data output from a general Bayer array image sensor. Therefore, the image encoding unit 20 and the image decoding unit 40 downstream of the imaging unit 50 can reduce visual degradation that occurs when compressing and encoding Bayer array RAW data by performing the same operations as those in the first to third embodiments.

[0206] Next, the output format of an image when the imaging section 50 operates in HDR mode will be described with reference to FIG.

[0207] When the imaging unit 50 operates in HDR mode, the imaging unit 50 outputs the average value of two short-exposure pixels and the average value of the remaining two long-exposure pixels included in the same-color 2x2 pixel area in Fig. 26(a). In this embodiment, the average value of the two short-exposure pixels is set as the pixel of a low-sensitivity image with short-exposure, and the average value of the two long-exposure pixels is set as the pixel of a high-sensitivity image with long-exposure. Here, the output of the low-sensitivity image shown in Fig. 26(b) can be derived as follows.

[0208]

number

[0209] On the other hand, the output of the high-sensitivity image shown in FIG. 26(b) can be derived as follows.

[0210]

number

[0211] The two types of images, the low-sensitivity image and the high-sensitivity image, output from the imaging unit 50 are compression-encoded by the image encoding unit 20 and stored in the memory 30. Furthermore, the two types of encoded data are decoded by the image decoding unit 40, and the decoded low-sensitivity image and high-sensitivity image are synthesized by the image synthesis unit 60 to generate an HDR image. Here, a method of generating an HDR image in the image synthesis unit 60 will be described with reference to FIG. 27.

[0212] 27 is a block diagram of the image synthesis unit 60. The image synthesis unit 60 is made up of a gain correction unit 601 for low-sensitivity images, a gain correction unit 602 for high-sensitivity images, a selector 603, and a synthesis unit 604.

[0213] When generating an HDR image, generally, one of two types of images, a low-sensitivity image and a high-sensitivity image, is taken with the correct exposure, and the other image is used as auxiliary data to expand the dynamic range. In the following explanation, we will explain two cases: when a high-sensitivity image is taken with the correct exposure, and when a low-sensitivity image is taken with the correct exposure.

[0214] It should be noted that a user who operates the image processing device 200 can set whether the low-sensitivity image or the high-sensitivity image is to have the correct exposure via a CPU (not shown) provided in the image processing device 200. Also, the CPU (not shown) provided in the image processing device 200 may be configured to automatically determine which image is to have the correct exposure.

[0215] The gain correction unit 601 and the gain correction unit 602 are used to align the exposure of the low-sensitivity image and the high-sensitivity image. Here, the details of the correction performed by the gain correction unit 601 and the gain correction unit 602 will be described with reference to FIG.

[0216] FIG. 28(a) shows an example of gain correction when a high-sensitivity image is shot with a proper exposure. When the brightness (input light amount) of the subject exceeds a certain threshold th_white, the output of the high-sensitivity image is overexposed and loses gradation. On the other hand, in a low-sensitivity image, even at a brightness exceeding the threshold th_white, overexposed areas do not occur and gradation remains. By increasing the gain of this low-sensitivity image in the gain correction unit 601, pixel values ​​of the overexposed areas of the high-sensitivity image are estimated. Furthermore, by combining the gain-up-increased low-sensitivity image and the high-sensitivity image in the subsequent synthesis unit 604, an HDR image in which overexposed areas are compensated can be generated. Note that when a high-sensitivity image is shot with a proper exposure, the gain correction unit 602 does not need to perform gain correction on the high-sensitivity image.

[0217] FIG. 28(b) shows an example of gain correction when a low-sensitivity image is shot with proper exposure. When the brightness (input light amount) of the subject falls below a certain threshold th_dark, the output of the low-sensitivity image is subject to blackout and loss of gradation. On the other hand, in a high-sensitivity image, even when the brightness falls below the threshold th_dark, there is an area where blackout does not occur and gradation remains. By performing gain down on this high-sensitivity image in the gain correction unit 602, pixel values ​​of the blackout-occurring part of the low-sensitivity image are estimated. Furthermore, by synthesizing the gain-down-reduced high-sensitivity image and the low-sensitivity image in the downstream synthesis unit 604, an HDR image in which the blackout part is compensated can be generated. Note that when a low-sensitivity image is shot with proper exposure, the gain correction unit 601 does not need to perform gain correction on the low-sensitivity image.

[0218] The degree of gain correction (gain value) performed by the gain correction units 601 and 602 may be calculated from the sensitivity ratio between the low-sensitivity image and the high-sensitivity image, the ratio of exposure times between the low-sensitivity image and the high-sensitivity image, etc. For example, if the high-sensitivity image is captured with proper exposure and the sensitivity ratio between the low-sensitivity image and the high-sensitivity image is 1:4, the pixel value of the low-sensitivity image can be multiplied by four to estimate the pixel value of the highlight-occurring portion of the high-sensitivity image.

[0219] The selector 603 outputs to a synthesis unit 604 the pixel values ​​of the image captured at the correct exposure, either the low-sensitivity image or the high-sensitivity image.

[0220] The synthesis unit 604 synthesizes the gain-corrected low-sensitivity image and high-sensitivity image to generate an HDR image. Here, an example of a method of synthesizing the low-sensitivity image and the high-sensitivity image performed by the synthesis unit 604 will be described with reference to FIG.

[0221] 29, the horizontal axis indicates the brightness (pixel value) of the properly exposed image, which is the pixel value obtained from the selector 603. The vertical axis indicates the composition ratio of the low-sensitivity image or the high-sensitivity image.

[0222] In this embodiment, the sum of the synthesis ratios of the images is set to always be 100% when synthesizing the images. In other words, the pixel value hdr_img[pix] of the HDR image output from the synthesis unit 604 can be derived by the following formula. hdr_img[pix]= α × main_img[pix] + (1-α) × sub_img[pix] …(25) Here, α indicates the blending ratio of the properly exposed image, main_img[pix] indicates the pixel value of the properly exposed image, and sub_img[pix] indicates the pixel value of the not properly exposed image.

[0223] Fig. 29(a) shows an example of the composition ratio of each image when a high-sensitivity image is photographed at a proper exposure. As shown in Fig. 28, at a brightness exceeding the threshold th_white, whiteout of the high-sensitivity image occurs, so in this embodiment, the composition ratio of the high-sensitivity image is changed so as to decrease to 0% at the threshold th_white, with the threshold th_pre_white as a boundary. Also, the composition ratio of the low-sensitivity image is changed so as to increase to 100% at the threshold th_white, with the threshold th_pre_white as a boundary. By setting the composition ratio in this way, it is possible to compose a wide dynamic range image in which whiteout parts are compensated.

[0224] Fig. 29(b) shows an example of the composition ratio of each image when a low-sensitivity image is photographed with a proper exposure. As shown in Fig. 28, at a brightness below the threshold th_dark, blackout of the low-sensitivity image occurs, so in this embodiment, the composition ratio of the low-sensitivity image is changed so as to decrease to 0% at the threshold th_dark, with the threshold th_pre_dark as a boundary. Also, the composition ratio of the high-sensitivity image is changed so as to increase to 100% at the threshold th_dark, with the threshold th_pre_dark as a boundary. By setting the composition ratio in this way, a wide dynamic range image in which blackout portions are compensated can be composed.

[0225] In the present embodiment, for the sake of simplicity, an example has been described in which the blending ratio changes from a certain threshold value, but the algorithm for the blending process is not limited to this.

[0226] In this way, when the image synthesis unit 60 generates an HDR image, the synthesis ratio of each image changes depending on whether the image is properly exposed or not, or the magnitude (brightness) of the pixel value. The magnitude of the synthesis ratio is the degree of influence on the image quality, and an area with a large synthesis ratio has a large influence on the image quality, and an area with a small synthesis ratio has a small influence on the image quality. In consideration of this, in the image encoding unit 20 and the image decoding unit 40, the larger the synthesis ratio in the image synthesis unit 60 at the latter stage is assigned to an area, and the code amount is reduced in the other areas, so that the visual deterioration of the image can be more effectively reduced. In this embodiment, such flexible allocation of the code amount is realized by the feature analysis unit 225 in the image encoding unit 20. Hereinafter, the details of the feature analysis method in the feature analysis unit 225 of this embodiment will be described using a flowchart.

[0227] 30 is a flowchart showing an outline of the feature analysis process in the feature analysis unit 225 when the imaging unit 50 operates in HDR mode in this embodiment. The process of the feature analysis unit 225 will be described below with reference to the same drawing.

[0228] In S3001, the feature analysis unit 225 acquires the inverse quantization data from the inverse quantization unit 224 shown in Fig. 2 on a pixel-by-pixel basis, and calculates the brightness of the pixel. In this embodiment, as in the first embodiment, the level value of the inverse quantization data is used as an index indicating the brightness (small level value = dark, large level value = bright). The index indicating the brightness in this embodiment is not limited to this method, and for example, the inverse quantization data in RGB format may be further color space converted to YUV format, and the level value of the Y component may be used. Alternatively, the inverse quantization block may be divided into a plurality of regions, and the average value of the level values ​​of the inverse quantization data in each divided region may be used.

[0229] In S3002 to S3004, the feature analysis unit 225 switches between the processes S3005 to S3008 to be executed based on whether the properly exposed image is a high-sensitivity image and whether the image to be encoded is a high-sensitivity image. For example, if the properly exposed image is a high-sensitivity image and the image to be encoded is a high-sensitivity image, the feature analysis unit 225 executes S3005. On the other hand, if the properly exposed image is a low-sensitivity image and the image to be encoded is a low-sensitivity image, the feature analysis unit 225 executes S3007. Here, the details of S3005 to S3008 executed by the feature analysis unit 225 will be described with reference to Figs. 31A to 31D.

[0230] 31A to 31D are flowcharts showing details of the feature analysis process executed by the feature analysis unit 225 when the imaging unit 50 operates in HDR mode in this embodiment. Hereinafter, the process of the feature analysis unit 225 will be described with reference to the drawings.

[0231] FIG. 31A is a flowchart showing the process (S3005) of the feature analysis unit 225 when the correct exposure is a high-sensitivity image and the encoding target image is a high-sensitivity image.

[0232] In the processes of S3101 to S3105, the feature analysis unit 225 calculates a feature classification result rank for the pixel of interest. As in the first embodiment, the rank has a value of 1 to 4, and the higher the value of a pixel, the more noticeable the degradation is to the human eye, or the more important the pixel is.

[0233] In S3101, the feature analysis unit 225 determines whether the brightness of the pixel of interest is equal to or less than the threshold value th_pre_white. As shown in FIG. 29(a), when the brightness of the pixel of interest is equal to or less than the threshold value th_pre_white, the composition ratio of the pixel to the HDR image is 100%. Therefore, the pixel of interest has a large influence on the image quality of the HDR image, and it can be said that the pixel is a highly important pixel. In such a case, in S3102, the feature analysis unit 225 assigns "4" to the feature classification result rank of the pixel of interest.

[0234] When the brightness of the pixel of interest is greater than the threshold th_pre_white, the feature analysis unit 225 determines whether the brightness of the pixel of interest is equal to or less than the threshold th_white. As shown in FIG. 29(a), when the brightness of the pixel of interest is greater than the threshold th_pre_white and equal to or less than the threshold th_white, the blending ratio of the pixel to the HDR image is between 100% and 0%. Therefore, although the influence of the pixel on the image quality of the HDR image is not necessarily large, the pixel is treated as having medium importance in this embodiment. In such a case, in S3104, the feature analysis unit 225 assigns "3" to the feature classification result rank of the pixel of interest.

[0235] On the other hand, when the brightness of the pixel of interest is greater than the threshold value th_white, the blending ratio of the pixel to the HDR image is 0%, as shown in Fig. 29(a). Therefore, the pixel of interest has little effect on the image quality of the HDR image, and it can be said to be a pixel of low importance. In such a case, in S3105, the feature analysis unit 225 assigns "1" to the feature classification result rank of the pixel of interest.

[0236] Fig. 31B is a flowchart showing the process (S3006) of the feature analysis unit 225 when the correct exposure is a high-sensitivity image and the encoding target image is a low-sensitivity image. As in the example shown in Fig. 28(a), the feature analysis unit 225 determines the rank for each pixel according to the composition ratio of the pixel of interest to the HDR image.

[0237] Unlike FIG. 31A, in FIG. 31B, the feature analysis unit 225 uses the brightness of the low-sensitivity image in determining the rank of the pixel of interest in S3106 and S3108. Here, since the image encoding unit 20 is located before the gain correction unit 601, the gain correction of the low-sensitivity image is not performed at this point, and the exposure is not aligned with that of the high-sensitivity image with the correct exposure. Therefore, in S3106 and S3108, the thresholds th_pre_white' and th_white' for the low-sensitivity image are used to determine the position (brightness) at which the synthesis ratio in the downstream image synthesis unit 60 changes. Here, the thresholds th_pre_white' and th_white' may be obtained from the sensitivity ratio between the low-sensitivity image and the high-sensitivity image, the ratio of the exposure times between the low-sensitivity image and the high-sensitivity image, etc., based on the thresholds th_pre_white and th_white. For example, if a high-sensitivity image is captured with proper exposure and the sensitivity ratio between the low-sensitivity image and the high-sensitivity image is 1:4, the threshold for the low-sensitivity image can be multiplied by 1 / 4 to estimate the threshold th_pre_white in the downstream image synthesis unit 60 and the position (brightness) equivalent to th_white.

[0238] FIG. 31C is a flowchart showing the process (S3007) of the feature analysis unit 225 when the appropriate exposure is a low-sensitivity image and the encoding target image is a low-sensitivity image.

[0239] In S3111, the feature analysis unit 225 determines whether the brightness of the pixel of interest is greater than the threshold th_pre_dark. As shown in FIG. 29(b), if the brightness of the pixel of interest is greater than the threshold th_pre_dark, the composition ratio of the pixel to the HDR image is 100%. Therefore, the pixel of interest has a large influence on the image quality in the HDR image, and it can be said that the pixel is a highly important pixel. In such a case, in S3112, the feature analysis unit 225 assigns "4" to the feature classification result rank of the pixel of interest.

[0240] When the brightness of the pixel of interest is equal to or less than the threshold th_pre_dark, the feature analysis unit 225 determines whether or not the brightness of the pixel of interest is greater than the threshold th_dark. As shown in FIG. 29(b), when the brightness of the pixel of interest is equal to or less than the threshold th_pre_dark and greater than the threshold th_dark, the composition ratio of the pixel to the HDR image is between 100% and 0%. Therefore, although the influence of the pixel on the image quality of the HDR image is not necessarily large, in this embodiment, the pixel is treated as having medium importance. In such a case, in S3114, the feature analysis unit 225 assigns "3" to the feature classification result rank of the pixel of interest.

[0241] On the other hand, the brightness of the pixel of interest is the threshold th_white below In this case, as shown in Fig. 29(b), the composition ratio of the pixel to the HDR image is 0%. Therefore, the effect of the pixel on the image quality of the HDR image is small, and it can be said that the pixel is a pixel of low importance. In this case, in S3115, the feature analysis unit 225 assigns "1" to the feature classification result rank of the pixel of interest.

[0242] Fig. 31D is a flowchart showing the process (S3008) of the feature analysis unit 225 when the correct exposure is a low-sensitivity image and the encoding target image is a high-sensitivity image. As in the example shown in Fig. 31C, the feature analysis unit 225 determines the rank for each pixel according to the composition ratio of the pixel of interest to the HDR image.

[0243] Unlike Fig. 31C, in Fig. 31D, the brightness of the high-sensitivity image is used in determining the rank of the pixel of interest in S3116 and S3118. As in Fig. 31B, gain correction of the high-sensitivity image has not been performed at this point, and the exposure is not consistent with that of the low-sensitivity image with the correct exposure. Therefore, in S3116 and S3118, the threshold values ​​th_pre_dark' and th_dark' for the high-sensitivity image are used to determine the position (brightness) where the composition ratio changes in the image composition unit 60 at the downstream stage.

[0244] The method of setting the rank is not limited to the above-mentioned method. For example, it is also possible to set the rank of the low-sensitivity image and the high-sensitivity image taken with the appropriate exposure by the same algorithm as the method described in the first to third embodiments, and to set the rank of the remaining images taken with underexposure or overexposure by the algorithm described in Figs. 30 to 31D.

[0245] As described above, the feature analysis unit 225 sets a large rank to pixels with a large synthesis ratio when generating an HDR image, and sets a small rank to pixels with a small synthesis ratio. Also, as already described in the first embodiment, the feature classification result rank of the feature analysis unit 225 is output to the multiplexing unit 223. Based on the feature classification result output from the feature analysis unit 225, the multiplexing unit 223 preferentially stores lost bits caused by quantization in the encoded data for pixels with higher importance. In this way, it is possible to allocate a large amount of code to areas that affect image quality when generating an HDR image, and reduce the amount of code for other areas. Therefore, it is possible to more effectively reduce visual deterioration of an HDR image than when this embodiment is not implemented, that is, when the amount of code is evenly allocated to all areas of a block of interest.

[0246] [Fifth embodiment] In recent years, imaging devices have been known that have a function of switching the input capacitance of an amplifier circuit provided for each column of an imaging element, and can output images with multiple different sensitivities by switching the gain according to the signal level. As with the imaging device described in the fourth embodiment, by synthesizing multiple images with different sensitivities obtained from this imaging device, an image with a wider dynamic range than that of a conventional imaging device can be obtained in one shot. In the fifth embodiment, a description will be given of a case where an image obtained from such an imaging device is to be coded.

[0247] In the fifth embodiment, although only the imaging device is different from the fourth embodiment, it can be realized with a configuration basically similar to that of the fourth embodiment shown in Fig. 20, and similar effects can be obtained. Therefore, the fifth embodiment will be mainly described with respect to the imaging device, which is the difference from the fourth embodiment, and a description of the same parts as the fourth embodiment will be omitted.

[0248] Although details will be described later, the column amplifier 503 in this embodiment can amplify signals using multiple gains, and the dynamic range of an image can be expanded by combining signals amplified with different gains. Also, the timing control unit 508 in this embodiment can change the gain of the column amplifier 503 by controlling each switch provided in the column amplifier 503.

[0249] In this embodiment, since it is possible to output images having a plurality of different sensitivities by switching the gain of the column amplifier 503, the pixel section 500 in this embodiment does not need to have two types of pixels, pixels for short exposure and pixels for long exposure, as in the fourth embodiment. That is, in this embodiment, it is not necessary to wire two types of control signals 516 and 517 for controlling the exposure time to the unit pixel 501, and it is sufficient that one type of control signal for controlling the transfer switch 515 of the unit pixel 501 is wired.

[0250] 32 is a diagram showing the configuration of a circuit in which only one column is extracted from the column amplifier 503 in this embodiment. This circuit is an inverting amplifier circuit composed of an operational amplifier 531, input side capacitors 536 and 537, and feedback capacitors 539 and 530. A predetermined reference voltage is applied to the positive input terminal (non-inverting input terminal) of the operational amplifier 531 from a reference power supply. In addition, the timing control unit 508 can switch the connections of the capacitors 536, 537, and 539 by operating the switches 534, 535, and 538.

[0251] Feedback capacitors 539 and 530 are connected in parallel to the inverting input terminal and output terminal, respectively, of an operational amplifier 531. A switch 538 is connected in series with the feedback capacitor 539 and controls the connection of the feedback capacitor 539. The amplification factor of the operational amplifier 531 is determined by "input capacitance / feedback capacitance".

[0252] When switch 534 is turned ON and switches 535 and 538 are turned OFF, operational amplifier 531 outputs a signal amplified by a first gain corresponding to the capacitance ratio between capacitor 536 and capacitor 530 to column ADC 504. When switch 534 is turned OFF and switches 535 and 538 are turned ON, operational amplifier 531 outputs a signal amplified by a second gain corresponding to the ratio between capacitor 537 and the combined capacitance of capacitors 539 and 530 to column ADC 504. In this way, by switching the capacitance of the capacitors with each switch, two types of image signals with different sensitivities can be output.

[0253] As described above, the imaging unit 50 of the fifth embodiment can output a low-sensitivity image and a high-sensitivity image, similar to the imaging unit 50 of the fourth embodiment, by devising the circuit structure of the column amplifier 503. In addition, these images are compression-coded in the same manner as in the fourth embodiment, and an HDR image is generated in the image synthesis unit 60. In the compression-coding of this embodiment, as in the fourth embodiment, a large amount of code is allocated to an area that affects the image quality when generating an HDR image, and the amount of code is reduced for other areas. Therefore, visual deterioration of the HDR image can be reduced more effectively than when this embodiment is not implemented, that is, when the amount of code is evenly allocated to all areas of the block of interest.

[0254] [Sixth embodiment] In the fourth and fifth embodiments, a compression encoding method for generating an HDR image using an imaging device capable of outputting a plurality of images with different sensitivities in one shooting has been described. On the other hand, when an image sensor such as a general CCD or CMOS sensor is used, multiple images are captured while changing the exposure amount. Pieces The technique of taking multiple images in a time-division manner and synthesizing the images to generate an HDR image is already implemented in digital cameras, smartphones, and the like. In the sixth embodiment, a method of synthesizing multiple images taken in a time-division manner while changing the amount of exposure in this way is described. Pieces The following description will be given of the case where the image to be coded is the image shown in FIG.

[0255] FIG. 33 is a diagram showing an example of the operation timing of the imaging section 50 when capturing an image of a subject in a time-division manner while changing the amount of exposure.

[0256] In FIG. 33, the (i-1)th image is captured by the imaging unit 50 with a proper exposure time T1. After a certain time T2 has elapsed since the (i-1)th image was captured, the i-th image is captured with a shorter exposure time T2 (underexposure) than the (i-1)th image. Here, the (i-1)th image is a high-sensitivity image with a relatively longer exposure time than the i-th image, and conversely, the i-th image is a low-sensitivity image with a shorter exposure time. In this way, the imaging unit 50 captures multiple images while changing the amount of exposure. Pieces It is possible to capture and output images having different sensitivities in a time-division manner.

[0257] The operation timing and exposure time of the imaging section 50 shown in FIG. 33 can be changed by a CPU (not shown) included in the image processing device 200 operating the timing control section 508 of the imaging section 50 and the like.

[0258] In this way, the imaging unit 50 of the sixth embodiment can output a low-sensitivity image and a high-sensitivity image, similar to the imaging unit 50 of the fourth and fifth embodiments, by performing time-division shooting while changing the exposure amount. Moreover, these images are compression-coded in the same manner as the fourth and fifth embodiments, and an HDR image is generated in the image synthesis unit 60. In the compression-coding of this embodiment, similar to the fourth and fifth embodiments, a large code amount is allocated to an area that affects the image quality when generating an HDR image, and the code amount is reduced for other areas. Therefore, it is possible to more effectively reduce visual deterioration of an HDR image compared to a case where this embodiment is not implemented, that is, a case where the code amount is evenly allocated to all areas of a block of interest.

[0259] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0260] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0261] 10...acquisition unit, 20...image encoding unit, 30...memory, 40...image decoding unit, 210...temporary encoding system, 220...main encoding system, 211A to 211E...encoding unit, 215...QP determination unit, 221...delay unit, 223...multiplexing unit, 224...inverse quantization unit, 225...feature analysis unit

Claims

1. 1. An image coding apparatus that performs coding on a block basis, the block being composed of a predetermined number of pixels, generates coded data of a preset fixed length, and outputs the coded data, acquiring means for acquiring image data of said block from an image generating source; an encoding processing means for encoding the image data acquired by the acquisition means, generating encoded data having a code amount not exceeding the fixed length, and adding a bit shortage to obtain the fixed length and outputting the encoded data; The encoding processing means a variable length coding means for quantizing each pixel included in a target block so that the code amount of the target block is equal to or less than the fixed length, and for variable length coding the quantized data obtained by the quantization; a determining means for determining a size of a free area required for achieving the fixed length from a difference between the amount of code of the block of interest obtained by the variable length coding means and the fixed length; an analysis means for analyzing the quantization data of each pixel of the block of interest and calculating an index value indicating the importance of each pixel; a multiplexing means for multiplexing the coded data obtained by the variable-length coding means and the bits lost in the quantization by storing a larger number of bits lost in the quantization in the free area for pixels with a higher importance indicated by the index value, thereby generating the fixed-length data; 13. An image encoding device comprising:

2. 2. The image encoding device according to claim 1, wherein the analysis means calculates the index value based on the quantized data of each pixel of the target block, so that the index value has a larger value the darker the area to which the target pixel belongs.

3. 2. The image encoding device according to claim 1, wherein the analysis means calculates the index value based on the quantized data of each pixel of the target block, so that the flatter the region to which the target pixel belongs, the larger the index value becomes.

4. The analysis means a brightness determination means for determining whether or not the brightness of the pixel of interest is equal to or less than a preset threshold value for determining a dark area, based on the quantized data of each pixel of the block of interest; a flatness determination means for determining whether or not the pixel of interest belongs to a flat portion set in advance based on the quantized data of each pixel of the block of interest; a complexity determining means for determining a complexity of an area to which the pixel of interest belongs by determining a number of edges based on quantized data in the block of interest and comparing the number of edges thus determined with a preset threshold value; a determination means for determining an index value of the importance of the pixel of interest based on the determinations of the brightness determination means, the flatness determination means, and the complexity determination means; The image encoding device according to claim 1 , further comprising:

5. The encoding processing means generating means for quantizing each pixel of the block of interest using a quantization step that is preset so that the length of the pixel is equal to or less than the fixed length when the variable length encoding means is unable to generate a code amount equal to or less than the fixed length, thereby generating quantized data; 2. The image encoding device according to claim 1 .

6. the generating means performs quantization using a quantization step that is set so as to generate a preset free space for the fixed length; The analysis means calculates the index value for the quantized data obtained from the block of interest and generated by the generation means, The multiplexing means comprises: The more important a pixel is indicated by the index value, the more bits lost in the quantization by the generating means are stored in the free area, and the quantized data generated by the generating means and the bits lost in the quantization are multiplexed to generate the fixed-length data.

6. The image encoding device according to claim 5.

7. The image source outputs image data having a plurality of different sensitivities; The acquisition means acquires image data having a plurality of different sensitivities output from the image generation source, The encoding processing means encodes each of the image data having a plurality of sensitivities acquired by the acquisition means, The analysis means changing a calculation method of the index value according to the sensitivity of the image data to which the pixel of interest belongs and the brightness of the pixel of interest based on the quantized data of the pixel of interest; 2. The image encoding device according to claim 1 .

8. The image generation source includes: First image data having a correct exposure; outputting second image data having a lower or higher sensitivity compared to the first image data; 8. The image encoding device according to claim 7.

9. If the second image data has low sensitivity, The analysis means In analyzing the second image data, The brighter the pixel of interest is, the larger the index value is calculated.

9. The image encoding device according to claim 8.

10. If the second image data has low sensitivity, The analysis means In analyzing the first image data, The index value is calculated so that the darker the pixel of interest is, the larger the index value is.

9. The image encoding device according to claim 8.

11. When the second image data has high sensitivity, The analysis means In analyzing the second image data, The index value is calculated so that the darker the pixel of interest is, the larger the index value is.

9. The image encoding device according to claim 8.

12. When the second image data has high sensitivity, The analysis means In analyzing the first image data, The brighter the pixel of interest is, the larger the index value is calculated.

9. The image encoding device according to claim 8.

13. The image generation source includes: A pixel section including a plurality of pixels each including a photoelectric conversion element; A control means for controlling an exposure time on a pixel-by-pixel basis, An image sensor capable of outputting pixel information having a plurality of different exposure times under the control of the control means, The sensitivity of the image data is based on the length of exposure time by the control means.

13. The image encoding device according to claim 7,

14. The image generation source includes: A pixel section including a plurality of pixels each including a photoelectric conversion element; an amplifier for amplifying a signal output from the pixel unit at a plurality of different amplification factors; An image sensor capable of outputting pixel information amplified by the amplifier at a plurality of different amplification factors, The sensitivity of the image data is based on the magnitude of the amplification factor by the amplifying means.

13. The image encoding device according to claim 7,

15. the image source is an image sensor, The sensitivity of the image data is Based on the exposure amount at the time of shooting set for each of the image data, 13. The image encoding device according to claim 7,

16. 2. An image decoding device that decodes encoded data generated by the image encoding device according to claim 1, comprising: An acquisition means for acquiring fixed-length encoded data of a block composed of a predetermined number of pixels; a decoding processing means for decoding the encoded data acquired by the acquisition means, generating an image of the block, and outputting the image; The decoding processing means a decoding means for decoding coded data of quantized data in the coded data of the block of interest to generate quantized data of each pixel included in the block of interest; an inverse quantization means for generating inverse quantized data by inverse quantizing the quantized data of the block of interest obtained by the decoding means; a determination means for determining a storage position of a lost bit in the encoded data of the block of interest based on a result of the decoding by the decoding means; an analysis means for analyzing the inverse quantization data of each pixel of the block of interest and calculating an index value indicating the importance of each pixel; a loss bit adding means for adding loss bits based on the index value of each pixel to the inverse quantization data of each pixel of the target block based on the storage position determined by the determining means, and outputting the inverse quantization data after the addition as pixel data.

2. An image decoding device comprising:

17. 17. An imaging apparatus comprising: an imaging means functioning as an image generation source; an image encoding apparatus according to claim 1; and an image decoding apparatus according to claim 16.

18. The imaging device includes: an acquisition means for acquiring image data having a plurality of different sensitivities from the imaging means; A synthesis unit that adds up a plurality of image data having different sensitivities for each pixel, The image encoding device comprises: encoding each of the plurality of image data having different sensitivities obtained from the acquisition means; the image decoding device generates decoded image data having a plurality of different sensitivities by decoding each of the encoded data of the image data having a plurality of different sensitivities generated by the image encoding device; The synthesizing means adds the decoded image data having the plurality of different sensitivities for each pixel, the analysis means in the image encoding device changes a calculation method of the index value in accordance with a blending ratio of the pixel of interest in the blending means.

20. The imaging device according to claim 17,

19. The analysis means The index value calculated by the combining means is larger as the combining ratio of the pixel of interest is larger.

20. The imaging device according to claim 18.

20. The imaging device includes: A pixel section including a plurality of pixels each including a photoelectric conversion element; A control means for controlling exposure time on a pixel-by-pixel basis; An image sensor capable of outputting pixel information having a plurality of different exposure times under the control of the control means, The sensitivity of the image data is based on the length of exposure time by the control means.

20. The imaging device according to claim 18 or 19.

21. The imaging device includes: A pixel section including a plurality of pixels each including a photoelectric conversion element; an amplifier for amplifying a signal output from the pixel unit at a plurality of different amplification factors; An image sensor capable of outputting pixel information amplified by the amplifier at a plurality of different amplification factors, The sensitivity of the image data is based on the magnitude of the amplification factor by the amplifying means.

20. The imaging device according to claim 18 or 19.

22. The imaging device is an image sensor, The sensitivity of the image data is Based on the exposure amount at the time of capturing the image data, 20. The imaging device according to claim 18 or 19.

23. A method for controlling an image coding device that performs coding on a block basis, the block being made up of a predetermined number of pixels, generates coded data of a preset fixed length, and outputs the coded data, the method comprising the steps of: acquiring image data for said block from an image source; an encoding process step of encoding the image data acquired in the acquisition step, generating encoded data having a code amount not exceeding the fixed length, and adding a missing bit to obtain the fixed length and outputting the encoded data; The encoding process includes: a variable-length coding step of quantizing each pixel included in the block of interest so that the code amount of the block of interest is equal to or less than the fixed length, and variable-length coding the quantized data obtained by the quantization; a determining step of determining a size of a free space required for achieving the fixed length from a difference between the amount of code of the block of interest obtained in the variable length coding step and the fixed length; an analysis step of analyzing the quantized data of each pixel of the block of interest and calculating an index value indicating the importance of each pixel; a multiplexing step of multiplexing the coded data obtained in the variable-length coding step and the bits lost in the quantization by storing a larger number of bits lost in the quantization in the free area for pixels with a higher importance indicated by the index value, thereby generating the fixed-length data; 23. A method for controlling an image encoding device, comprising:

24. 2. A method for controlling an image decoding device that decodes encoded data generated by the image encoding device according to claim 1, comprising the steps of: An acquisition step of acquiring fixed-length encoded data of a block composed of a predetermined number of pixels; a decoding process step of decoding the encoded data acquired in the acquisition step, generating an image of the block, and outputting the image; The decoding process includes: a decoding step of decoding coded data of quantized data in the coded data of the block of interest to generate quantized data of each pixel included in the block of interest; an inverse quantization step of generating inverse quantized data by inverse quantizing the quantized data of the block of interest obtained in the decoding step; a determination step of determining a storage position of a loss bit in the encoded data of the block of interest based on a result of the decoding step; a feature analysis step of analyzing the inverse quantization data of each pixel of the block of interest and calculating an index value indicating the importance of each pixel; a loss bit addition step of supplementing the inverse quantization data of each pixel of the target block with loss bits based on the index value of each pixel based on the storage position determined in the determination step, and outputting the supplemented inverse quantization data as pixel data.

23. A method for controlling an image decoding device comprising the steps of:

25. A program for causing a computer to execute each step of the method according to claim 23 or 24 when the program is read and executed by the computer.

Images