Image encoding apparatus, control method thereof, and computer readable storage medium

Abstract

Provided are an image encoding device, a control method thereof, and a computer-readable storage medium. The device includes an acquisition unit that acquires information indicating an optical compression ratio in a horizontal direction and a vertical direction when an imaging unit is imaging; a transformation unit that performs wavelet transformation on image data to generate a plurality of subband data; a determination unit that determines a quantization parameter for a transform coefficient in the plurality of subbands obtained by the transformation unit; and an encoding unit that quantizes the transform coefficient in the subband data obtained by the transformation unit according to the quantization parameter determined by the determination unit, and encodes the quantized transform coefficient, wherein the determination unit determines the quantization parameter based on the information indicating the compression ratio acquired by the acquisition unit, and weights each subband.

Description

Technical Field

[0001] This invention relates to image encoding devices and control methods thereof, as well as non-transitory computer-readable storage media. Background Technology

[0002] Currently, digital camera devices, such as digital video cameras, for recording moving images have become popular. In recent years, the method of recording RAW (raw) images has been applied not only to still images but also to moving images. This is believed to be because, although RAW images require a large amount of data for recording, the correction and degradation of the initial image can be kept to a minimum, and there is a high degree of freedom in image editing even after recording.

[0003] When recording RAW moving images on a recording medium, it is desirable to record a sufficiently long length of moving images. Therefore, it is desirable to compress and encode the RAW moving images before recording. Typically, RAW images are Bayer pattern images, in which pixels (each a color of R, G, or B) are arranged in a mosaic pattern. Adjacent pixels in a Bayer pattern have different color components, resulting in low correlation between adjacent pixels. Therefore, it is difficult to achieve high compression efficiency if the pixels are encoded as is. Therefore, pixels with similar color components are extracted from the RAW image to generate multiple planes, each with a single component. These planes are then encoded, which increases the correlation between pixels within a plane, thereby improving compression efficiency; this plane transformation technique is commonly used as one of the compression encoding techniques.

[0004] Furthermore, H.264 (H.264 / MPEG-4 Part 10: Advanced Video Coding) is known as a traditional and typical compression coding scheme. In this scheme, for each block consisting of a predetermined number of pixels in a frame, the amount of data is compressed by utilizing the temporal and spatial redundancy inherent in moving images. In H.264, compression coding is achieved by combining techniques such as motion detection and motion compensation for temporal redundancy, discrete cosine transform (DCT) as a frequency transform for spatial redundancy, quantization, and entropy coding. However, if the compression ratio increases beyond a certain point, the block distortion characteristic of the DCT transform becomes significant, and subjective image degradation becomes apparent.

[0005] Therefore, subband coding techniques using Discrete Wavelet Transform (DWT) have been adopted in schemes such as JPEG2000. These techniques apply low-pass and high-pass filters in the horizontal and vertical directions respectively, thereby decomposing the frequency bands into subbands. Subband coding produces less block distortion than coding techniques using DCT and is characterized by better compression characteristics at high compression levels.

[0006] For example, the technique described in Patent Document 1 (Japanese Patent Application Laid-Open No. 2019-4428) efficiently compresses RAW data by performing planar transformation on the RAW image, using DWT transform to separate frequency components to generate multiple sub-bands, and then quantizing and encoding each sub-band. In Patent Document 1, considering the characteristics of human vision, the relationship between the quantization parameters to be set for each sub-band is as follows: the further the sub-band is from the low-frequency component side, the smaller the quantization parameter; conversely, the further the sub-band is from the high-frequency component side, the larger the quantization parameter. Therefore, since the HL (high-low) sub-bands and LH (low-high) sub-bands at the same decomposition level are bands at the same frequency level, the relationship between the quantization parameters will be set to be the same, resulting in the same level of horizontal and vertical quantization distortion.

[0007] However, when RAW images with different optical compression ratios in the horizontal and vertical directions are encoded and subsequently displayed using the techniques described in the aforementioned literature, quantization distortion increases in these directions due to stretching. In this case, if the optical compression direction is in one direction (horizontal or vertical), the associated stretching direction will also be in one direction, increasing quantization distortion in that direction. This results in the distortion being visually perceived as image quality degradation during display. Summary of the Invention

[0008] In view of the above problems, the present invention provides an image encoding device for suppressing the increase of quantization distortion in the horizontal or vertical direction associated with images displaying different optical compression ratios in the horizontal and vertical directions.

[0009] In a first aspect, the present invention provides an image encoding apparatus operable to encode image data acquired by a camera unit, the image encoding apparatus comprising: an acquisition unit configured to acquire information representing optical compression ratios in the horizontal and vertical directions during imaging; a transform unit configured to perform wavelet transform on the image data to generate a plurality of sub-band data; a determination unit configured to determine quantization parameters for transform coefficients in the plurality of sub-bands acquired by the transform unit; and an encoding unit configured to quantize the transform coefficients in the sub-band data acquired by the transform unit according to the quantization parameters determined by the determination unit, and to encode the quantized transform coefficients, wherein the determination unit is operable to determine the quantization parameters by weighting each sub-band based on the compression ratio information acquired by the acquisition unit.

[0010] In a second aspect, the present invention provides a control method for an image encoding device operable to encode image data acquired by a camera unit, the control method comprising the following steps: (a) acquiring information representing optical compression ratios in the horizontal and vertical directions during imaging; (b) performing wavelet transform on the image data to generate multiple sub-band data; (c) determining quantization parameters for transform coefficients in the multiple sub-bands obtained in step (b); and (d) quantizing the transform coefficients in the sub-band data obtained in step (b) according to the quantization parameters determined in step (c), and encoding the quantized transform coefficients, wherein in step (c), the quantization parameters are determined by weighting each sub-band based on the information representing the compression ratio acquired in step (a).

[0011] In a third aspect, the present invention provides a non-transitory computer-readable storage medium storing a program, which, when read and executed by a computer, causes the computer to perform steps of a control method for an image encoding device, the image encoding device being operable to encode image data acquired by a camera unit, the control method comprising the steps of: (a) acquiring information representing optical compression ratios in the horizontal and vertical directions during imaging; (b) performing wavelet transform on the image data to generate a plurality of sub-band data; (c) determining quantization parameters for the transform coefficients in the plurality of sub-bands obtained in step (b); and (d) quantizing the transform coefficients in the sub-band data obtained in step (b) according to the quantization parameters determined in step (c), and encoding the quantized transform coefficients, wherein, in step (c), the quantization parameters are determined by weighting each sub-band based on the information representing the compression ratio acquired in step (a).

[0012] According to the present invention, encoded data suitable for images with different optical compression ratios in the horizontal and vertical directions can be generated.

[0013] Other features of the invention will become apparent from the following description of exemplary embodiments (with reference to the accompanying drawings). Attached Figure Description

[0014] Figure 1 This is a block configuration diagram of an image encoding device according to the first embodiment.

[0015] Figure 2 This is a diagram used to illustrate the planar transformation in the first embodiment.

[0016] Figure 3 This is a subband formation diagram when the vertical and horizontal filtering of the Discrete Wavelet Transform (DWT) are each performed three times.

[0017] Figure 4This is a diagram illustrating the relationship between sub-blocks and block rows in the first embodiment.

[0018] Figures 5A to 5C This is a diagram illustrating a lens with horizontal and vertical optical compression ratios α according to a first embodiment, and the relationship between the lens and the image captured by the lens.

[0019] Figures 6A to 6D This is a diagram showing the direction of the signal components of the planar data according to the second embodiment.

[0020] Figures 7A to 7C This is a diagram illustrating the decomposition of the diagonal signal components in the U, V, and GH plane data in the horizontal and vertical directions according to the second embodiment.

[0021] Figure 8 This is a block configuration diagram of an image decoding device according to the first embodiment.

[0022] Figure 9 This is a program configuration diagram of an information processing device according to a variation of the first embodiment.

[0023] Figure 10 It is a flowchart used to illustrate the image encoding process.

[0024] Figure 11 It is a flowchart used to illustrate the image decoding process. Detailed Implementation

[0025] In the following, embodiments will be described in detail with reference to the accompanying drawings. Several features are described in the embodiments, but the invention is not limited to requiring all such features; rather, multiple such features can be suitably combined. Furthermore, in the drawings, the same reference numerals denote the same or similar configurations, and redundant descriptions thereof are omitted.

[0026] [First Embodiment]

[0027] Figure 1 This is a block diagram of the main parts related to encoding of the camera device 100 according to the first embodiment.

[0028] Camera device 100 includes controls Figure 1 The control unit 150 of each processing unit shown. The control unit 150 is configured with a CPU, a ROM for holding the program to be executed by the CPU, and RAM to be used as the working area.

[0029] The camera device 100 can be implemented as, for example, a digital camera or a digital video camera. Alternatively, the camera device 100 can be implemented as any information processing terminal or image processing device such as a personal computer, cellular phone, smartphone, PDA, tablet terminal, or portable media player. Figure 1 The illustration shows a configuration including the camera unit 101, considering its use as a digital camera or similar imaging device. However, the source of the image to be encoded is not limited to the camera unit and can be a storage medium for storing the image to be encoded. It should be understood that the embodiments applicable to the imaging device are merely illustrative and easily understood examples.

[0030] [Camera Unit]

[0031] The imaging unit 101 includes: an optical lens; an aperture; a lens optical system capable of optical zoom and including a focus control and lens drive unit; and an imaging element, such as a CCD image sensor or a CMOS image sensor for converting optical information from the lens optical system into electrical signals. The imaging unit 101 outputs RAW image data obtained by converting the electrical signals obtained by the imaging element into digital signals to the planar conversion unit 102. Furthermore, the imaging unit 101 includes a non-volatile memory (not shown) for maintaining information related to the optical compression ratio in the horizontal and vertical directions as metadata, and supplies this metadata to a quantization parameter weight setting unit 112. For example, it is assumed that the imaging unit 101 can capture images at a rate of 30 frames per second. For the optical lens, the lens unit can be attached to and detached from the imaging device, and can be configured to allow the installation of different types of lenses. If the optical lens can be installed and detached, it can be configured such that not only a typical lens unit with an optical compression ratio of 1 in the horizontal and vertical directions can be installed, but also anamorphic lenses with different optical compression ratios in the horizontal and vertical directions can be installed.

[0032] [Plane Transformation Unit]

[0033] The planar conversion unit 102 receives RAW image data of the Bayer mode frame of interest captured by the camera unit 101 as input. Then, the planar conversion unit 102 separates a RAW image data into multiple planes, each configured with individual components. Figure 2This is a planar formation diagram when RAW image data in Bayer pattern, used as input image data, is separated into four planes. In the Bayer pattern, adjacent 2×2 pixels are configured with one red component (R), one blue component (B), and two green components (G1, G2). The planar conversion unit 102 separates the RAW image data into an R plane configured only by the R component, a G1 plane configured only by the G1 component, a G2 plane configured only by the G2 component, and a B plane configured only by the B component. When the number of horizontal pixels in the RAW image data is represented by W and the number of vertical pixels in the RAW image data is represented by H, the size of these four planes is W / 2 pixels horizontally and H / 2 pixels vertically. As a result, when a plane representing a color component is of interest, the correlation between adjacent pixels is high, making it easier to improve compression efficiency.

[0034] Discrete Wavelet Transform (DWT) Unit

[0035] DWT unit 103 performs frequency transformation on the planes sequentially output from plane transformation unit 102 and generates transformation coefficients. DWT transformation is a transformation that filters the entire image. Vertical and horizontal filtering can be performed when pixel data corresponding to the number of taps of the filter to be used has accumulated in the buffer memory. Therefore, by performing DWT on a row-by-row basis for plane data and recursively applying DWT to the generated low-frequency subband LL, individual subbands can be processed in parallel.

[0036] Figure 3This is a subband formation diagram obtained when three vertical and horizontal filtering processes are performed as a group in a DWT. In the diagram, "L" and "H" indicate low frequency and high frequency, respectively, and regarding their order, the first part indicates the frequency band resulting from horizontal filtering, and the second part indicates the frequency band resulting from vertical filtering. The number after "Lv" indicates the decomposition level of the DWT. When DWT is performed two or more times, the transform object is the subband LL, which is the low-frequency band obtained by the immediately preceding transform. Therefore, each time DWT is performed, the size will be half of the horizontal and vertical direction of the subband of the immediately preceding transform. Furthermore, for this reason, the subband LL is retained after the last DWT; therefore, as shown, no label indicating the decomposition level is given. In this embodiment, the DWT unit 103 performs DWT sequentially on four planes generated from the frame of interest (RAW image) to be encoded; however, to shorten the processing time, multiple DWT units 103 can be provided. For example, when two DWT units 103 are set up in parallel, the burden associated with DWT becomes half that of when only one DWT unit 103 is present, and the time spent performing the transformation can also be halved. Furthermore, when four DWT units 103 are set up in parallel, the time spent performing DWT can be reduced to one-quarter of the time spent when only one DWT unit 103 is present.

[0037] In this embodiment, the following assumptions are made for the description: each time a row of transform coefficients is generated in each subband, the DWT unit 103 outputs the transform coefficients of a row of each subband to the quantization unit 104 in sequence.

[0038] [Quantization Unit]

[0039] Quantization unit 104 uses the quantization parameter Qp generated by quantization control unit 106 to quantize the transform coefficients input from DWT unit 103 in coefficient units. The quantization parameter Qp is a parameter with a value that, when larger, results in a smaller quantization value. This reduces the amount of encoded data but significantly degrades image quality. Furthermore, the quantization of the transform coefficients in the four planes can be performed plane-by-plane or in parallel for all planes. However, it is assumed that in this embodiment, quantization unit 104 quantizes the transform coefficients at the same sub-band and position in each plane using a common quantization parameter Qp and supplies this quantization result to encoder 105.

[0040] [Encoder]

[0041] The encoder 105 generates and outputs encoded data by entropy encoding the transform coefficients of each plane after quantization in the quantization unit 104.

[0042] [Recording Processing Unit]

[0043] The recording processing unit 107 formats the encoded data output from the encoder 105 into a predetermined recording format and records it as a file in the recording medium 108, including information required for decoding in the header. The recording processing unit 107 stores the metadata (compression ratio) obtained from the imaging unit 101 in this header. This is to enable the image decoding device to generate a normal image by horizontally or vertically interpolating the image obtained through decoding. The recording processing unit 107 stores information related to the initial values ​​set by the initial target encoded data amount setting unit 114 and the initial quantization value setting unit 115 in this header. However, if the initial values ​​set by the initial target encoded data amount setting unit 114 and the initial quantization value setting unit 115 are pre-unified between the encoding and decoding devices, it is not necessary to include this information in the file header.

[0044] [Recording medium]

[0045] Recording medium 108 is, for example, a recording medium configured as a non-volatile memory and is configured to be attached to and detached from camera device 100.

[0046] [Initial Settings]

[0047] The initial target coding data volume setting unit 114 sets the target coding data volume at the start of coding for the frame of interest (the RAW image of interest). The initial quantization value setting unit 115 sets the quantization parameter Qp at the start of coding for the frame of interest. Typically, various settings at the start of coding are calculated through feedback control based on the coding information of the previous plane.

[0048] [Quantization Control Unit]

[0049] Next, the quantization control unit 106 will be described. The quantization control unit 106 controls the quantization parameter Qp such that the amount of coded data generated for the frame of interest converges to the target amount of coded data for the frame of interest. In this embodiment, the RAW image of the frame of interest is separated into four planes, each of which undergoes discrete wavelet transform, quantization, and encoding. If the subbands are of the same type and the location for updating the quantization parameter Qp is also the same, then the quantization parameter Qp used when quantizing the four planes is common. This is because the common quantization parameter Qp is updated based on the amount of coded data for the frame of interest (4 planes).

[0050] Figure 4 This is a diagram showing the units of quantization control. (Refer to...) Figure 4The control unit describes the quantization parameter Qp. As mentioned above, the input image to be encoded is a Bayer-mode RAW image and is separated into four planes by the plane transformation unit 102. Furthermore, DWT is applied to each plane. As mentioned above, the encoding unit is the row of the corresponding sub-band; however, the unit of quantization control is the set of encoding results for each sub-band at the same pixel location. That is, as... Figure 4 As shown, one row of subbands {Lv3HL, Lv3LH, Lv3HH} and subband {LL} at decomposition level 3, two rows of subbands {Lv2HL, Lv2LH, Lv2HH} at decomposition level 2, and four rows of subbands {Lv1HL, Lv1LH, Lv1HH} at decomposition level 1 constitute a control unit of Qp. That is, this unit of quantization control corresponds to the data of one color component in a row of a RAW image obtained by capturing real space. Subsequently, the set of corresponding transform coefficients in each subband, which serves as the unit of control, is called a "block row".

[0051] The generated encoded data volume retention unit 109 takes the encoded data volume of each block line notified by the encoder 105 as input and retains it. The target encoded data volume calculation unit 113 calculates the target encoded data volume of a block line based on the target encoded data volume of the frame of interest and the total number of block lines.

[0052] The difference calculation unit 110 calculates the difference between the generated encoded data amount and the target encoded data amount for each block row, and further calculates the integral difference amount as the integral value of the difference.

[0053] The quantization calculation unit 111 calculates (updates) the quantization parameter Qp of the i-th block row of interest based on the integral difference D(i-1) notified by the difference calculation unit 110. Details will be described later.

[0054] [Quantization Value Calculation]

[0055] One method for calculating quantization parameters is the known technique described in MPEG2 Test Model 5. According to Test Model 5, when the initial quantization parameter is Qini and the pixel block of interest is the i-th pixel block, the quantization parameter Qp[i] of the pixel block of interest is calculated from ΣE(i-1) using the following equation (1), where ΣE(i-1) represents the integral of the difference between the amount of encoded data in each pixel block from the first pixel block to the (i-1)-th pixel block immediately preceding it and the amount of target encoded data in each pixel block.

[0056] Qp[i]=Qini+r×ΣE[i-1]...(1)

[0057] Here, r indicates the control sensitivity of the quantization parameter. The larger the control sensitivity r, the greater the fluctuation of Qp[i] according to ∑E[i-1], which improves the controllability of the encoded data volume, but also increases the variation in image quality. On the other hand, the smaller the control sensitivity r, the smaller the dependence of Qp[i] on ∑E[i-1], which reduces the fluctuation of Qp[i] and can reduce the variation in image quality; however, the controllability of the encoded data volume decreases.

[0058] In this embodiment, a frame of a Bayer-mode RAW image is divided into R, G1, G2, and B planes. Values ​​0, 1, 2, and 3, used to identify the corresponding planes, are assigned to each plane, and these values ​​are represented as variables p1. Furthermore, the i-th block row of the corresponding plane p1 is represented as BL(p1, i). Then, the amount of encoded data generated when block row BL(p1, i) is encoded is defined as C(BL(p1, i)), and the target amount of encoded data for one block row is represented as TC.

[0059] At this time, the difference calculation unit 110 calculates the integral value ΣE[i-1] of the difference between the generated coded data and the target coded data from the first block line up to the (i-1)th block line immediately preceding the i-th block line of interest according to the following equation (2).

[0060] ΣE[i-1]=ΣΣ{TC-C(BL(pl,k))}...(2)

[0061] Here, ΣΣ represents the total value of pl = 0, 1, 2, 3 and k = 0, 1, 2, ..., i-1.

[0062] The quantization calculation unit 111 calculates the quantization parameter Qp of the i-th block row of interest by applying the integral value ΣE[i-1] obtained using equation (2) above to equation (1) described earlier. Then, after the quantization calculation unit 111 converts the quantization parameter Qp into the actual quantization parameters Qp[pl][sb] of each subband, the vectorization unit 104 notifies the quantization parameter Qp. pl and sb indicate the type and decomposition level of the corresponding plane and the corresponding subband, respectively.

[0063] The method for calculating the quantization parameters of each plane and sub-band by the quantization value calculation unit 111 will be described below. As shown in equation (3), the quantization value calculation unit 111 calculates Qp[pl][sb] by multiplying the matrix mtx held for each plane and sub-band and the optical compression ratios α in the horizontal and vertical directions to be set by the quantization parameter weight setting unit 112 by the quantization parameter Qp calculated by equation (1). The optical compression ratios α in the horizontal and vertical directions to be set by the quantization parameter weight setting unit 112 are set using the metadata recorded by the camera unit 101.

[0064] Qp[pl][sb]=Qp[i]×mtx[pl][sb]×α...(3)

[0065] A configuration can be adopted where Qp[pl][sb] is provided as a preset value instead of the calculation using equation (3), and the preset value is switched according to the optical compression ratio in the horizontal and vertical directions. Typically, mtx is set to control the amount of encoded data, such that the higher the frequency range of the subband, the larger Qp is set, and the lower the frequency range of the subband, the smaller Qp is set. This makes the frequency components of image data that are more difficult to perceive visually due to the characteristics of human vision higher, and the amount of encoded data generated will be compressed more, thereby improving coding efficiency. Therefore, the matrix mtx is set such that the higher the frequency of the subband, the larger the quantization parameter Qp, and the lower the frequency of the subband, the smaller the quantization parameter Qp. In addition, the matrix mtx is set such that the quantization parameter is the same in HL subbands and LH subbands at the same decomposition level. In this embodiment, it is assumed that the matrix mtx is set such that 3LL:3HL:3LH:3HH:2HL:2LH:2HH:1HL:1LH:1HH=1:2:2:4:4:4:8:8:8:16.

[0066] Here, we will refer to Figures 5A to 5C Describes the optical compression ratio α in the horizontal and vertical directions.

[0067] Figure 5A The image shown illustrates a subject captured using a 1:1 lens with equal horizontal and vertical compression ratios in camera unit 101. When the subject is circular, its horizontal and vertical lengths are in a 1:1 ratio, and similarly, the horizontal and vertical lengths of the subject in the captured image are also in a 1:1 ratio. Therefore, in... Figure 5A In the example, the horizontal and vertical optical compression ratio α is 1.

[0068] Next, the description Figure 5B . Figure 5B The image shown is captured when a 2:1 lens is used to capture the subject in the camera unit 101.

[0069] Despite Figure 5A Similarly, the horizontal and vertical lengths of the subject are in a 1:1 ratio, but because a lens with a 1 / 2x optical compression in the horizontal direction is used to capture the image, the horizontal and vertical lengths in the captured image are in a 1 / 2:1 ratio. Therefore, since the horizontal length is 1 / 2 when the vertical length is 1 (reference), the horizontal optical compression ratio α becomes 1 / 2, and the vertical optical compression ratio α becomes 1.

[0070] Finally, the description Figure 5C . Figure 5C The image shown is captured when a 1:2 lens is used to capture the subject in the camera unit 101. Although with... Figure 5A and 5B Similarly, the relationship between the horizontal and vertical lengths of the subject is 1:1, but because a lens that optically compresses the image in the vertical direction is used to capture the image, the relationship between the horizontal and vertical lengths in the captured image is 1:1 / 2. Therefore, since the vertical length is 1 / 2 when the horizontal length is 1 (reference), the vertical optical compression ratio α becomes 1 / 2, and the horizontal optical compression ratio α becomes 1.

[0071] Here, in this embodiment, as a method for obtaining α, it is assumed that the quantization parameter weight setting unit 112 obtains α from metadata acquired through communication with the lens unit mounted on the camera unit 101; however, α can be obtained according to user settings. For example, a configuration can be adopted in which the user selects information related to the horizontal and vertical compression ratios of the lens via an operation unit and a display unit (not shown), and the control unit 150 obtains the compression ratio of the lens selected by the user. Furthermore, it can be obtained from information such as whether a lens with equal optical compression ratios in the horizontal and vertical directions is installed or a lens with different optical compression ratios in the horizontal and vertical directions is installed.

[0072] Next, we will refer to again Figures 5A to 5C This section describes a specific example of the process of calculating the quantization parameters calculated by the quantization value calculation unit 111 in equation (3) based on the optical compression ratio α in the horizontal and vertical directions set by the quantization parameter weight setting unit 112 for each sub-band.

[0073] First, the description Figure 5A The case shown illustrates the situation where the horizontal and vertical optical compression ratios are equal. When... Figure 5A When the captured image is encoded and the encoded image is displayed, the image is not stretched in the horizontal or vertical direction, so there is no increase in quantization distortion in one direction associated with the display. Therefore, when calculating the quantization parameter Qp of each subband using equation (3), the quantization value calculation unit 111 performs a calculation of α = 1 in all subbands including the HL and LH subbands (i.e., the calculation is performed without changing the weight settings of each subband). Therefore, as the value of α when calculating the quantization parameter Qp in the HL and LH subbands, the relationship of the following equation (4) is applied.

[0074] HL:LH=1:1....(4)

[0075] Therefore, the ratio of quantization parameters for each subband is 3LL:3HL:3LH:3HH:2HL:2LH:2HH:1HL:1LH:1HH = 1:2:2:4:4:4:8:8:8:16.

[0076] Next, the description will use Figure 5B The example shown is a 2:1 lens. In... Figure 5B In this image, relative to the vertical direction, the horizontal direction is compressed to half. Therefore, if we utilize... Figure 5A If the same subband's quantization parameter weights are used for encoding, the quantization distortion will be the same in both the horizontal and vertical directions during encoding. However, since only the horizontal direction is stretched by a factor of 2 during display, the quantization distortion in the horizontal direction increases, resulting in the distortion being visually perceived as a degradation of image quality. Therefore, it is desirable to suppress the increase in quantization distortion in the horizontal direction by making the quantization parameter Qp of the HL subband, which is related to image quality in the horizontal direction, smaller than the quantization parameter Qp of other subbands. Therefore, when calculating the quantization parameter Qp of each subband using equation (3), when sb is an HL subband, the quantization value calculation unit 111 calculates the quantization parameter with a horizontal optical compression ratio α = 1 / 2. When sb is an LH subband, the quantization value calculation unit 111 calculates the quantization parameter Qp with a vertical optical compression ratio α = 1. Therefore, the following equation (5) is applied as the value of α when calculating the quantization parameter Qp in the HL and LH subbands.

[0077] HL:LH=1 / 2:1...(5)

[0078] As described above, in the case of a lens used to compress images horizontally rather than vertically, assuming the compression ratio in the horizontal direction relative to the vertical direction is A (A<1), the ratio of the quantization parameters of each sub-band is 3LL:3HL:3LH:3HH:2HL:2LH:2HH:1HL:1LH:1HH=1:2×A:2:4:4×A:4:8:8×A:8:16. Figure 5BIn the case of a 2:1 lens, since the image is compressed to 1 / 2 in the horizontal direction relative to the vertical direction, the compression ratio A in the horizontal direction relative to the vertical direction is 1 / 2. Therefore, the ratio of the quantization parameters of each sub-band is 3LL:3HL:3LH:3HH:2HL:2LH:2HH:1HL:1LH:1HH = 1:2×1 / 2:2:4:4×1 / 2:4:8:8×1 / 2:8:16. Although a 2:1 lens is described, in the case of a 1.33:1 lens (a lens that compresses the image to 1 / 1.33 in the horizontal direction relative to the vertical direction), A = 1 / 1.33, and the quantization parameters are determined using a weighting coefficient of A. A configuration can be adopted whereby, instead of obtaining the compression ratio information from the lens, the user selects the horizontal and vertical compression ratios of the lens (the captured image), and A is determined from the user-selected compression ratios as the compression ratio in the horizontal direction relative to the vertical direction.

[0079] Finally, the description will use Figure 5C The example shown is a 1:2 lens. In... Figure 5C In the case of using and Figure 5A If the same subband's quantization parameter weights are used for encoding, the quantization distortion will be the same in both the horizontal and vertical directions during encoding. However, since only the vertical direction is stretched by a factor of 2 during display, the quantization distortion in the vertical direction increases, resulting in the distortion being visually perceived as a degradation of image quality. Therefore, it is desirable to suppress the increase in quantization distortion in the vertical direction by making the quantization parameter Qp of the LH subband, which is related to image quality in the vertical direction, smaller than the quantization parameter Qp of other subbands. Therefore, when calculating the quantization parameter Qp of each subband using equation (3), when sb indicates the LH subband, the quantization value calculation unit 111 calculates the quantization parameter Qp with a vertical optical compression ratio α = 1 / 2. When sb is the HL subband, the quantization value calculation unit 111 calculates the quantization parameter Qp with a horizontal optical compression ratio α = 1. Therefore, the following equation (6) is applied as the value of α when calculating the quantization parameter Qp in the HL and LH subbands.

[0080] HL:LH=1:1 / 2...(6)

[0081] In the case of a lens used to compress images vertically rather than horizontally, assuming the compression ratio of the vertical direction relative to the horizontal direction is B (B<1), the ratio of the quantization parameters of each sub-band is 3LL:3HL:3LH:3HH:2HL:2LH:2HH:1HL:1LH:1HH=1:2:2×B:4:4:4×B:8:8:8×B:16. Figure 5CIn the case of a 1:2 lens, since the image is formed by compressing it to 1 / 2 in the vertical direction relative to the horizontal direction, the ratio of the vertical direction to the horizontal direction, B, is 1 / 2. Therefore, the ratio of the quantization parameters of each sub-band is 3LL:3HL:3LH:3HH:2HL:2LH:2HH:1HL:1LH:1HH = 1:2:2×1 / 2:4:4:4×1 / 2:8:8:8×1 / 2:16. A 1:2 lens has already been described; however, in the case of a 1:1.5 lens (a lens that forms an image compressed to 1 / 1.5 in the vertical direction relative to the horizontal direction), the quantization parameters are determined using the compression ratio of the vertical direction relative to the horizontal direction, B, = 1 / 1.5. A configuration can be adopted whereby, instead of obtaining the compression ratio information from the lens, the user selects the horizontal and vertical compression ratios of the lens (the captured image), and B is determined from the user-selected compression ratios as the compression ratio of the vertical direction relative to the horizontal direction.

[0082] Therefore, according to the principle that compression is horizontal ( Figure 5B () or vertical () Figure 5C This is used to switch the subbands to be weighted in the quantization parameters of each subband. Then, if the compression is horizontal ( Figure 5B If the compression is vertical, the HL subband will be weighted by a horizontal compression ratio A relative to the vertical direction, making the quantization parameter of the HL subband smaller. Then, if the compression is vertical ( Figure 5C If the compression ratio B between the vertical and horizontal directions is used, the LH subband will be weighted, making the quantization parameter of the LH subband smaller.

[0083] As described above, by changing the value of α to be set for each sub-band by the quantization parameter weight setting unit 112 according to the optical compression ratio in the horizontal and vertical directions, it becomes possible to suppress the increase in quantization distortion accompanying the display in one direction. If sb is an HH sub-band, unlike HL and LH sub-bands, the quantization parameter Qp is calculated without changing the weight of the quantization parameter according to the optical compression ratio in the horizontal and vertical directions (α = 1). So far, the method of setting the weight of the quantization parameter for each sub-band has been described using examples of 2:1 and 1:2 lenses; however, the horizontal and vertical optical compression ratios are not limited to this. Furthermore, values ​​close to the horizontal and vertical optical ratios can be applied to the optical compression ratio α instead of the horizontal and vertical ratios.

[0084] Image decoding equipment

[0085] Next, an image decoding device according to an embodiment will be described.

[0086] Figure 8 It is used with the image encoding device mentioned above ( Figure 1This is a block diagram showing the main components of the image decoding device 800 that decodes the generated encoded image data. Since encoding and decoding are closely related, the configuration and processing details related to decoding of the image decoding device will be briefly described with reference to the accompanying drawings.

[0087] The image decoding device 800 includes a control unit 850 for controlling the entire device. The control unit 850 is configured with a CPU, a ROM for holding the program to be executed by the CPU, and RAM to be used as the working area.

[0088] The header analysis unit 801 analyzes the file header of the encoded image data to be decoded and obtains the information required for decoding. This information includes information needed for decoding, such as the initial quantization value (and...). Figure 1 (corresponding to the initial quantization value setting unit 115 in the text), initial target encoded data volume (and) Figure 1 This includes information such as the initial target encoded data volume setting unit 114 (corresponding to the initial target encoded data volume setting unit 114) and the horizontal and vertical optical compression ratios. If the initial quantization value and initial target encoded data volume are pre-unified between the encoding and decoding devices, this information does not need to be included in the file header.

[0089] It should be understood that the quantization parameter weight setting unit 811, target encoded data volume calculation unit 812, quantization value calculation unit 813, encoded data volume holding unit 814, and difference calculation unit 815 constituting the quantization control unit 810 include components related to... Figure 1 Components with the same name in the same language have essentially the same functionality.

[0090] The head analysis unit 801 supplies the initial quantization value obtained through analysis to the quantization value calculation unit 813 of the quantization control unit, supplies the initial target encoded data volume to the target encoded data volume calculation unit 812, and supplies information representing the horizontal and vertical optical compression ratios to the quantization parameter weight setting unit 811 and the output unit 807.

[0091] Decoder 802 obtains the quantization coefficients of block rows in each subband of each plane by decoding the encoded data after the header. Decoder 802 supplies the quantization coefficients of all planes obtained in the decoding process to inverse quantization unit 803. In addition, since decoder 802 calculates the amount of encoded data for each block row of each plane in the decoding process, decoder 802 supplies information indicating the amount of encoded data to encoded data holding unit 814 each time the amount of encoded data for a block row is calculated.

[0092] As a result, during the decoding of block lines, the quantization value calculation unit 813 can determine the quantization parameters of the next block line according to the equations (1) to (3) used on the encoding side.

[0093] The inverse quantization unit 803 performs inverse quantization on each plane in blocks according to the quantization parameters set by the quantization control unit 810, and obtains the DWT transform coefficients. Naturally, the quantization parameters to be set in the inverse quantization unit 803 are the same as the quantization parameters used in the image encoding device. The inverse quantization unit 803 supplies the inverse-quantized transform coefficients to the inverse DWT unit 804.

[0094] The inverse DWT unit 804 performs the inverse DWT (inverse wavelet transform) based on the transform coefficients input from the inverse quantization unit 803 and outputs the result to the memory 805. As a result, in response to the progress of the decoding process, four planes will be constructed in the memory 805.

[0095] The planar integration unit 806 integrates the four planes to generate a Bayer-patterned RAW image, and supplies the RAW image to the output unit 807.

[0096] For example, output unit 807 processes the input RAW image and generates a normal color image where each pixel is configured using the three RGB components. Then, output unit 807 interpolates the generated color image in both the horizontal and vertical directions based on information indicating the horizontal and vertical compression ratios supplied from header analysis unit 801, and outputs it to an external display device. However, when the horizontal and vertical compression ratio is 1:1, output unit 807 does not perform interpolation. When the output destination of output unit 807 is a printing device, an image with color components such as YMCK is generated, and then horizontal and vertical interpolation is performed.

[0097] In the above decoding process, while decoding is being performed block by block, the quantization value calculation unit 813 of the quantization control unit 810 updates the quantization parameters used for inverse quantization of the block rows of each plane. Because the processing of updating the quantization parameters is related to... Figure 1 The quantization calculation unit 111 in the camera device 100 is the same, so its detailed description will be omitted.

[0098] [Modifications of the First Embodiment]

[0099] Examples of processing corresponding to the first embodiment described above, implemented by an application executed by an information processing device, such as a personal computer, will be described as variations of the first embodiment.

[0100] Figure 9This is a block diagram of an information processing device. When the device is powered on, the CPU 901 executes the boot program stored in ROM 902 to initialize the hardware, loads the operating system (OS) stored in hard disk (HDD) 904 into RAM 903, and transfers control to the OS (control is executed on the OS), enabling the device to function as an information processing device. That is, the display unit 906, keyboard 907, and mouse 908 serve as interfaces with the user. I / F 905 is an interface for communicating with external devices (typically a network interface or Universal Serial Bus (USB), etc.). In this embodiment, the RAW image data to be encoded is also input from an external unit via I / F 905. In the above configuration, when the user inputs a predetermined command by operating the keyboard 907 or mouse 908, the CPU 901 loads the encoding or decoding application from HDD 904 into RAM 903 and executes it, enabling the device to function as an image encoding or decoding device.

[0101] In the following text, reference will be made to Figure 10 The flowchart describes the processing when the device is used as an image encoding device. Basically, when the CPU 901 is used as an image encoding device by executing a program, the CPU 901 performs... Figure 1 The processing corresponding to each processing unit in the process; for details, refer to the first embodiment.

[0102] Furthermore, in the following description, it is assumed that the Bayer pattern RAW image data to be encoded is already stored as a file on the HDD 904. Additionally, it is assumed that information indicating the horizontal and vertical optical compression ratios at the time of recording is stored in the file header.

[0103] In step S101, the CPU 901 obtains information indicating the horizontal and vertical optical compression ratios from the RAW image file to be encoded. This information can be input by the user from the operation unit.

[0104] Then, in step S102, CPU 901 inputs RAW image data from the file and expands the RAW image data in RAM 903.

[0105] In step S103, CPU 901 generates four planes R, G1, G2 and B based on the RAW image data extended in RAM 903 and stores them in RAM 903.

[0106] In step S104, CPU 901 initializes various parameters before encoding. This initialization process includes setting the initial quantization value and the initial target encoded data size, etc. The initialization process also includes clearing the region used to store the accumulated sum of the encoded data size for the block line and the target encoded data size for the block line.

[0107] In step S105, CPU 901 performs DWT on each of the four planes. CPU 901 stores the sub-bands obtained through this process in RAM 903. Figure 10 Step S105 in the flowchart shows four boxes to indicate DWT for the four planes.

[0108] In step S106, CPU 901 sets the variable i, which defines the block line order, to the initial value "1". Then, in step S107, CPU 901 reads and quantizes the DWT coefficients of the i-th block line to be encoded in each plane from the subband obtained by DWT and stored in RAM 903.

[0109] In step S108, CPU 901 encodes the quantization coefficients of the i-th block row of each plane and temporarily stores the generated encoded data in RAM 903. At this time, CPU 901 obtains the difference between the encoded data amount of the i-th block row of each plane and the target encoded data amount, and accumulates the difference (the processing corresponding to equation (2)).

[0110] In step S109, CPU 901 updates the quantization parameters of the next block line to be encoded (the processing corresponding to equation (3)).

[0111] Then, in step S110, CPU 901 determines whether all block lines have been encoded based on the value of variable i. If there are unencoded block lines, CPU 901 advances the process to step S111 and increments variable i by 1. Then, CPU 901 returns the process to step S107 to encode the next block line.

[0112] On the other hand, once all block lines have been encoded, CPU 901 will proceed from step S110 to step S112. In step S112, CPU 901 creates a file header in HDD 904 that includes various information required for decoding (including information indicating the horizontal and vertical compression ratios). CPU 901 then formats the encoded data stored in RAM 903 after the file header into a preset format structure and outputs it, thereby creating an encoded image data file.

[0113] Next, refer to Figure 11 The flowchart describes the processing when the device is used as an image decoding device. Basically, when the CPU 901 is used as an image decoding device by executing a program, the CPU 901 performs... Figure 8 The processing corresponding to each processing unit in the process; for details, refer to the first embodiment.

[0114] Furthermore, in the following description, the encoded image file is stored in HDD 904, and a description will be given from the start of the decoding process until the image is displayed on display unit 906.

[0115] In step S201, the CPU 901 obtains information indicating the optical compression ratio in the horizontal and vertical directions by analyzing the file header of the encoded image file to be decoded. This information can be input by the user from the operation unit.

[0116] In step S202, CPU 901 initializes various parameters before decoding. This initialization process includes setting the initial quantization value and the initial target encoded data amount, etc. The initialization process also includes clearing the region in RAM 903 used to store the accumulated sum of the encoded data amount and the target encoded data amount of the block line. Next, in step S203, CPU 901 sets the variable i, which defines the block line order, to the initial value "1".

[0117] In step S204, CPU 901 decodes the i-th block row of each plane and obtains the quantized transform coefficients. At this time, since the amount of encoded data of the i-th block row is calculated, CPU 901 obtains the difference between the amount of encoded data of the block row and the target amount of encoded data, and updates it by adding the difference to the value of the above region (the processing corresponding to equation (2)).

[0118] In step S205, CPU 901 performs inverse quantization on the quantization coefficients of the i-th block row of each plane. When variable i is "1", quantization is performed according to the quantization parameters obtained through the initialization process. Furthermore, when variable i is not "1", the immediately preceding block row is inverse quantized using the updated quantization parameters obtained through the decoding process.

[0119] In step S206, CPU 901 performs inverse DWT on the transform coefficients obtained through inverse quantization and obtains a block-row image. CPU 901 stores the obtained block-row image data in RAM 903. When performing inverse DWT, since the number of taps of the filter to be used must be obtained for the transform coefficients, inverse DWT is not performed unless the transform coefficients are satisfied.

[0120] In step S207, when preparing to decode the next block, the CPU 901 updates the quantization parameters based on the sum of the differences between the encoded data amount of the block and the target encoded data amount of the block (the processing corresponding to equation (3)). That is, the quantization parameters corresponding to each sub-band are determined according to the horizontal and vertical optical compression ratios.

[0121] In step S208, CPU 901 determines whether all block lines have been decoded. If there are undecoded block lines, CPU 901 advances the process to step S209 and increments the variable i by "1". Then, CPU 901 returns the process to step S204 to decode the next block line.

[0122] On the other hand, when all block lines have been decoded, CPU 901 proceeds processing from step S208 to step S211. In step S210, CPU 901 performs a process of integrating the four planes reproduced on RAM 903 and generates a RAW image in Bayer mode. Then, CPU 901 advances processing to step S211.

[0123] In step S211, CPU 901 performs image processing on the generated RAW image and generates a color image in which each pixel is configured by R, G, and B components. Then, CPU 901 generates an image with a horizontal and vertical ratio of 1:1 by performing processing to enlarge the generated color image in the horizontal or vertical direction according to the horizontal and vertical compression ratio obtained in step S201, and outputs the image to display unit 906.

[0124] As described above, the same process as in the first embodiment can also be implemented by a computer program.

[0125] [Second Embodiment]

[0126] Next, the second embodiment will now be described. The configuration of the device in the second embodiment is the same as that in the first embodiment. Figure 1 same.

[0127] However, the plane conversion unit 102 according to the second embodiment converts the RAW image data into one brightness plane (Y plane) and three non-brightness planes. In the first embodiment, the quantization parameter weights of each sub-band in each plane are determined using only the optical compression ratio in the horizontal and vertical directions, while in the second embodiment, an example will be described that considers the optical compression ratio and the characteristics of the signal component orientation of each plane when determining the quantization parameters.

[0128] According to the second embodiment, the plane conversion unit 102 converts RAW image data into three non-luminance planes (U, V, GH planes) and a luminance plane (Y plane) as shown in equation (10) according to the following equations (7) to (9).

[0129] U=B-(G1+G2) / 2...(7)

[0130] V=R-(G1+G2) / 2...(8)

[0131] GH = G1 - G2...(9)

[0132] Y=(R+B+G1+G2) / 4...(10)

[0133] In the second embodiment, the planes are transformed as described above as an example of plane transformation; however, the transformation method is not limited to this.

[0134] Figures 6A to 6D This shows the direction representation of the signal components in each plane. Figures 6A to 6D It is focused on Figure 2 The image shows a diagram of adjacent 2×2 pixels (R, G1, G2, B) of the Bayer pattern.

[0135] First, according to equation (7), the U-plane is the difference between the arithmetic mean of the G1 and G2 components and the B component. Therefore, the direction represented by the signal component U is through... Figure 6A The solid line shown connects points P' and Q' along their diagonal direction. Point P' is the centroid of the B component, and point Q' is the centroid of the signal obtained by averaging the G1 and G2 components.

[0136] According to equation (8), the V-plane is the difference between the arithmetic mean of the G1 and G2 components and the R component. The direction represented by the signal component V is through... Figure 6B The solid line shown connects the diagonal direction of points S' and T'. Point S' is the centroid of the R component, and point T' is the centroid of the signal obtained by averaging the G1 and G2 components.

[0137] According to equation (9), the GH plane is the difference between the G1 and G2 components. Therefore, the direction represented by the signal component GH is as follows: Figure 6C The solid line shown is the diagonal direction connecting the centroid positions V' of component G1 and W' of component G2.

[0138] Finally, according to equation (10), the Y-plane is the arithmetic mean of the R, G1, G2, and B components. Therefore, the centroid of the signal component Y is point Z', which is as follows: Figure 6D The diagram shows the center positions of each component, and since there is no direction for the signal component Y, it is represented as a point.

[0139] As a method for setting the weights to be set by the quantization parameter weight setting unit 112 in the second embodiment, the weights can be calculated based on the optical compression ratio α in the horizontal and vertical directions as described in the first embodiment. However, considering the fact that some signal components after planar conversion are in the diagonal direction, setting the weights after decomposing these signal components into horizontal and vertical directions allows for encoding that further considers the influence of quantization distortion in the horizontal and vertical directions accompanying the stretching during display.

[0140] like Figures 7A to 7C As shown, the diagonal signal components in the U, V, and GH planes are decomposed into horizontal and vertical components. Figure 7A In China, when targeting Figure 6A The intersection of the horizontal axis and the perpendicular line drawn from point P' in the horizontal direction is point R'. The line segment connecting two points p1 and p2 is denoted as "L(p1, p2)". Furthermore, the angle ∠P'Q'R' is set to θ, and the length L(P'Q') is set to 1 as a reference. In this case, the lengths of line segments L(Q'R'), L(R'P'), and L(P'Q') are as shown in equation (11) below.

[0141] L(Q'R'):L(R'P'):L(P'Q')=cosθ:sinθ:1....(11)

[0142] then, Figure 7B Corresponding to Figure 6B When a perpendicular line is drawn from point S' in the direction of the horizontal axis, the intersection of the horizontal axis and the point is point U'. When the angle ∠S'T'U' is set to θ, and the length of line segment L(S'T') is set to 1 as the reference, the lengths of line segments L(T'U'), L(U'S'), and L(S'T') are as shown in the following equation (12).

[0143] L(T'U'):L(U'S'):L(S'T')=cosθ:sinθ:1...(12)

[0144] at last, Figure 7C Corresponding to Figure 6C The intersection of a perpendicular line drawn from point V' in the horizontal direction and a perpendicular line drawn from point W' in the vertical direction is point X'. When the angle ∠V'W'X' is set to θ and the length of line segment L(V'W') is set to 1 as a reference, the lengths of line segments L(W'X'), L(X'V'), and L(V'W') are as shown in equation (13) below.

[0145] L(W'X'):L(X'V'):L(V'W')=cosθ:sinθ:1...(13)

[0146] Here, we will refer to Figures 5A to 5C This describes the method for setting α in the quantization parameter weight setting unit 112 for each plane and sub-band. First, in Figure 5A In this context, since the horizontal and vertical lengths of the image to be encoded are in a 1:1 ratio, cosθ and sinθ are respectively... and Furthermore, equations (11) to (13) are expressed by the relations in equation (14) below.

[0147] L(Q'R'):L(R'P'):L(P'Q')=L(T'U'):L(U'S'):L(S'T')=L(W'X'):

[0148]

[0149] Since the length ratio of the line segments in equation (14) is synonymous with the α to be set in each sub-band, the following equation (15) is used as the value of α for the HL, LH and HH sub-bands in the U, V and GH planes.

[0150]

[0151] Since the Y plane does not have a specific direction for the signal components, the following equation (16) is applied as the value of α for the HL, LH and HH subbands.

[0152] HL:LH:HH=1:1:1...(16)

[0153] Next, we will target Figure 5B The following description is given; in this case, since the relationship between the horizontal and vertical lengths is 1 / 2 to 1, cosθ and sinθ are respectively... and Therefore, equation (12) will become the relation of the following equation (17).

[0154] L(Q'R'):L(R'P'):L(P'Q')=L(T'U'):L(U'S'):L(S'T')=L(W'X'):

[0155]

[0156] Therefore, the following equation (18) is applied as the value of α for the HL, LH and HH subbands in the U, V and GH planes.

[0157]

[0158] Apply the following equation (19) as the value of α for the HL, LH and HH subbands in the Y plane.

[0159] HL:LH:HH=1 / 2:1:1...(19)

[0160] Finally, the description Figure 5C In this case, since the relationship between the horizontal and vertical lengths is 1:1 / 2, cosθ and sinθ are respectively... and Therefore, equation (13) will become the relation of the following equation (20).

[0161] L(Q'R'):L(R'P'):L(P'Q')=L(T'U'):L(U'S'):L(S'T')=L(W'X'):

[0162]

[0163] Therefore, the following equation (21) is applied as the value of α for the HL, LH and HH subbands in the U, V and GH planes.

[0164]

[0165] Apply the following equation (22) as the value of α for the HL, LH and HH subbands in the Y plane.

[0166] HL:LH:HH=1:1 / 2:1...(22)

[0167] As described above, according to the second embodiment, by setting the weights of each plane and sub-band not only considering the horizontal and vertical optical compression ratios but also the direction of the signal components of each plane, it becomes possible to perform encoding that further considers the influence of quantization distortion in the horizontal or vertical directions accompanying stretching during display.

[0168] In the first embodiment, regarding the value of the quantization parameter, if the subbands have the same type and the block rows have the same position, it is assumed that the quantization parameter is the same across the planes. This is because the only difference between the four planes in the first embodiment is the color component, and these planes have the same weight. In contrast, the second embodiment encodes the luminance Y-plane and the three chromaticity planes, and for the luminance plane, the quantization parameter is set to be sufficiently smaller than the quantization parameters of the other chromaticity planes.

[0169] Furthermore, as in the variations of the first embodiment, the processing corresponding to the second embodiment described above can be implemented by a computer-executed program.

[0170] Furthermore, in the above embodiments, examples of compression ratios of 2:1 or 1:2 in the horizontal and vertical directions were described; however, this ratio can be other ratios. In this case, it is sufficient to set the quantization parameters to be set for the HL and LH subbands of the same decomposition level according to the ratio.

[0171] In the first, modified, and second embodiments described above, the encoding object is a Bayer-mode RAW image; however, the image can be in other formats. For example, when encoding a monochrome image, since the image is initially configured with only a single luminance component, a plane conversion unit 102 is not required. Furthermore, the present invention can be applied to color images represented by luminance and chrominance components such as YCbCr. In this case, it is sufficient to perform the same processing as described above on the Y plane, Cb plane, and Cr plane.

[0172] In the above embodiments, a Bayer pattern in which the R, G1, G2, and B pixels are arranged in a 2×2 pixel arrangement according to the order of raster scanning is given as an example; however, in other Bayer patterns, the quantization parameters can be determined based on their arrangement.

[0173] Other embodiments

[0174] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0175] While the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be accorded the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An image encoding device operable to encode image data acquired by a camera unit, the image encoding device comprising: The acquisition unit is configured to acquire information representing the optical compression ratio in the horizontal and vertical directions during image capture; A transformation unit is configured to perform wavelet transform on the image data to generate multiple sub-band data; A determining unit is configured to determine quantization parameters for the transform coefficients in a plurality of subbands obtained for the transform unit; as well as An encoding unit is configured to quantize the transform coefficients in the sub-band data obtained by the transform unit according to the quantization parameters determined by the determining unit, and to encode the quantized transform coefficients. The determining unit is operable to weight each sub-band based on the information representing the compression ratio acquired by the acquiring unit to determine the quantization parameters.

2. The image encoding device according to claim 1, wherein, The determining unit is operable to determine the quantization parameter by weighting it using weighting coefficients corresponding to the information representing the compression ratio acquired by the acquiring unit.

3. The image encoding device according to claim 2, wherein, The determining unit is operable to determine the subband to be weighted among the plurality of subbands based on the information representing the compression ratio acquired by the acquiring unit.

4. The image encoding device according to claim 1, wherein, The determining unit is operable to switch between weighted LH subband and weighted HL subband based on the compression ratio obtained by the acquiring unit.

5. The image encoding device according to claim 1, wherein, With regard to the optical compression ratio in the horizontal and vertical directions, if the compression ratio in the horizontal direction is relative to the compression ratio in the vertical direction is a first ratio, the determining unit is operable to determine quantization parameters such that the ratio of the quantization parameters to be applied to the HL subband to the quantization parameters to be applied to the LH subband at the same decomposition level is the first ratio, wherein the first ratio is less than 1.

6. The image encoding device according to claim 1, wherein, With an optical compression ratio of 1 in both the horizontal and vertical directions, the determining unit is operable to determine the quantization parameters to be the same for HL and LH subbands of the same resolution level. With respect to the optical compression ratio in the horizontal and vertical directions, where the compression ratio in the horizontal direction is relative to the vertical direction as a first ratio, the determining unit is operable to determine quantization parameters by weighting the HL subband using the first ratio, such that the quantization parameters to be applied to the HL subband are set to be less than the quantization parameters to be applied to the LH subband at the same decomposition level, wherein the first ratio is less than 1.

7. The image encoding device according to claim 1, wherein, With respect to the optical compression ratio in the horizontal and vertical directions, if the compression ratio in the vertical direction is a second ratio relative to the horizontal direction, the determining unit is operable to determine quantization parameters such that the ratio of the quantization parameters to be applied to the LH subband to the quantization parameters to be applied to the HL subband at the same decomposition level is the second ratio, wherein the second ratio is less than 1.

8. The image encoding device according to claim 1, wherein, With an optical compression ratio of 1 in both the horizontal and vertical directions, the determining unit is operable to determine the quantization parameters to be the same for HL and LH subbands of the same resolution level. In the case where the compression ratio of the vertical direction relative to the horizontal direction is a second ratio, the determining unit is operable to determine the quantization parameter by weighting the LH subband using the second ratio, such that the quantization parameter to be applied to the LH subband is set to be less than the quantization parameter to be applied to the HL subband at the same decomposition level, wherein the second ratio is less than 1.

9. The image encoding device according to claim 1, wherein, The optical lens unit can be attached to or detached from the image encoding device. The acquisition unit is operable to acquire information representing the optical compression ratio in the horizontal and vertical directions by communicating with the lens unit, and The determining unit is operable to determine the quantization parameters based on information obtained from the lens unit representing the optical compression ratio in the horizontal and vertical directions.

10. The image encoding device according to claim 1, wherein, The acquisition unit is operable to acquire information representing the compression ratio selected by the user via the operation unit of the image encoding device, and The determining unit is operable to determine the quantization parameter based on information representing the compression ratio selected by the user, acquired by the acquiring unit.

11. The image encoding device according to claim 10, wherein, When the information indicating the compression ratio selected by the user indicates a compression ratio in which more compression is performed in the horizontal direction than in the vertical direction, the determining unit is operable to determine the quantization parameter to be applied to the HL subband as less than the quantization parameter to be applied to the LH subband at the same decomposition level.

12. The image encoding apparatus according to claim 1, further comprising a plane conversion unit, the plane conversion unit being configured to convert the image obtained by the imaging unit into a plurality of planes, each composed of individual components, wherein... The transformation unit is operable to perform wavelet transform on each plane obtained by the plane transformation unit.

13. The image encoding device according to claim 12, wherein, The camera unit is operable to output RAW images in Bayer mode, and The plane conversion unit is operable to convert the RAW image into the R plane, G1 plane, G2 plane, and B plane.

14. The image encoding device according to claim 13, wherein, When the optical compression ratio in the horizontal and vertical directions indicates that the horizontal compression ratio in the image acquired by the imaging unit is 1 / 2 times that in the vertical direction, the determining unit is operable to determine the quantization parameter to be applied to the HL subband as 1 / 2 of the quantization parameter to be applied to the LH subband at the same decomposition level.

15. The image encoding device according to claim 13, wherein, When the optical compression ratio in the horizontal and vertical directions indicates that the compression ratio in the vertical direction relative to the horizontal direction in the image obtained by the imaging unit is 1 / 2, the determining unit is operable to determine the quantization parameter to be applied to the LH subband as 1 / 2 of the quantization parameter to be applied to the HL subband at the same decomposition level.

16. The image encoding device according to claim 12, wherein, The camera unit is operable to output a RAW image in Bayer mode, in which R, G1, G2, and B pixels are arranged in 2×2 pixels according to the raster scan order. The plane conversion unit is operable to convert the RAW image into the following: chromatic aberration U, V, and GH planes, and luminance Y plane. U = B - (G1 + G2) / 2 V = R - (G1 + G2) / 2 GH = G1 - G2 Y = (R + B + G1 + G2) / 4.

17. The image encoding device according to claim 16, wherein, The determining unit, where the optical compression ratio in the horizontal and vertical directions indicates a compression ratio of 1 / 2 in the horizontal direction relative to the vertical direction in the image acquired by the imaging unit: It can be operated to set the ratio of the values ​​of the quantization parameters to be set for the same decomposition level of the HL, LH, and HH subbands in the Y plane to 1 / 2:1:1, and Capable of operating to set the ratio of the values ​​of the quantization parameters to be set for the same decomposition level of the HL, LH, and HH subbands for the U, V, and GH planes.

18. The image encoding device according to claim 16, wherein, The determining unit, where the optical compression ratio in the horizontal and vertical directions indicates that the compression ratio in the vertical direction relative to the horizontal direction in the image acquired by the imaging unit is 1 / 2, is as follows: It can be operated to set the ratio of the values ​​of the quantization parameters to be set for the same decomposition level of the HL, LH, and HH subbands in the Y plane to 1:1 / 2:1, and Capable of operating to set the ratio of the values ​​of the quantization parameters to be set for the same decomposition level of the HL, LH, and HH subbands for the U, V, and GH planes.

19. A control method for an image encoding device, the image encoding device being operable to encode image data acquired by a camera unit, the control method comprising the following steps: (a) Obtain information representing the optical compression ratio in the horizontal and vertical directions during image capture; (b) Perform wavelet transform on the image data to generate multiple sub-band data; (c) Determine the quantization parameters for the transform coefficients in the multiple subbands obtained in step (b); and (d) Quantize the transform coefficients in the subband data obtained in step (b) according to the quantization parameters determined in step (c), and encode the quantized transform coefficients. In step (c), the quantization parameters are determined by weighting each subband based on the information representing the compression ratio obtained in step (a).

20. A non-transitory computer-readable storage medium storing a program, which, when read and executed by a computer, causes the computer to perform steps of a control method for an image encoding device, the image encoding device being operable to encode image data acquired by a camera unit, the control method comprising the following steps: (a) Obtain information representing the optical compression ratio in the horizontal and vertical directions during image capture; (b) Perform wavelet transform on the image data to generate multiple sub-band data; (c) Determine the quantization parameters for the transform coefficients in the multiple subbands obtained in step (b); and (d) Quantize the transform coefficients in the subband data obtained in step (b) according to the quantization parameters determined in step (c), and encode the quantized transform coefficients. In step (c), the quantization parameters are determined by weighting each subband based on the information representing the compression ratio obtained in step (a).

Images