Decoding device, encoding device, decoding method, and encoding method

Abstract

To provide an efficient decoding device.SOLUTION: The decoding device selects a filter for the boundary between the first block and the second block from among a plurality of candidates, and in the deblocking filtering process in a case where the first filter is selected, performs a clipping process of setting a pixel change amount of each of a first pixel and a second pixel adjacent to each other with the boundary interposed therebetween to a negative first clipping value or more and a positive first clipping value or less, and in the deblocking filtering process in a case where the second filter is selected, performs a clipping process of setting the pixel change amount of each of the first pixel and the second pixel to a negative second clipping value or more and a positive second clipping value or less. The first clipping value is different from the second clipping value, the first block and the second block are vertically adjacent, and the boundary is a horizontal boundary.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to a method for encoding and decoding an image, and more particularly to deblocking filtering during image decoding and a rounding process used in the deblocking filtering process. [Background technology]

[0002] Currently, most standard video coding algorithms are based on hybrid video coding. Hybrid video coding methods typically combine several different lossless and lossy compression techniques to achieve a desired compression gain. Hybrid video coding is the basis for ITU-T standards (H.26x standards, such as H.261 and H.263) as well as ISO / IEC standards (MPEG-1, MPEG-2, and MPEG-4). The latest video coding standard, called H.264 / MPEG-4 Advanced Video Coding (AVC), is the result of standardization efforts by the Joint Video Team (JVT), a joint team between the ITU-T and ISO / IEC MPEG groups. This codec is being further developed by the Joint Collaborative Team on Video Coding (JCT-VC) under the name High-Efficiency Video Coding (HEVC), with the aim of improving the efficiency of video coding, especially for high-resolution video.

[0003] The video signal input to the encoder is a sequence of images called frames, each consisting of a two-dimensional matrix of pixels. All of the above-mentioned standards based on hybrid video coding involve dividing each video frame into smaller blocks of pixels. The size of the blocks varies, for example, depending on the image content. The coding method may typically vary from block to block. For example, in HEVC, the maximum size allowed for such a block is 64x64 pixels. This maximum size is called the maximum coding unit (LCU). In H.264 / MPEG-4 AVC, a macroblock (usually a block of 16x16 pixels) is the basic image element on which coding is performed and which may be further divided into smaller sub-blocks to which some encoding / decoding steps are applied.

[0004] Typically, the encoding step in hybrid video coding involves spatial and / or temporal prediction. Thus, each block to be coded is first predicted using spatially or temporally neighboring blocks, i.e., blocks from a previously coded video frame. A difference block between the block to be coded and the prediction, also called a prediction residual block, is then calculated. In a next coding step, the residual block is transformed from the spatial (pixel) domain to the frequency domain. The purpose of the transformation is to reduce the correlation of the input blocks. In a next coding step, the transform coefficients are quantized. This step results in a substantially lossy (irreversible) compression. Typically, the compressed transform coefficient values ​​are further compressed (losslessly) by entropy coding. Furthermore, side information necessary to reconstruct the coded video signal is coded and provided together with the coded video signal. This information relates, for example, to the spatial and / or temporal prediction, the amount of quantization, etc.

[0005] 1 shows an example of a typical video encoder 100 compliant with H.264 / MPEG-4 AVC and / or HEVC. A subtractor 105 first subtracts a current block of an input video (input signal s) from a corresponding predicted block

number

[0006] To obtain a decoded (reconstructed) video signal s', a decoding unit is incorporated into the video encoder 100. In line with the encoding step, the decoding step includes inverse quantization and inverse transform (130). The prediction error signal e' thus obtained differs from the original prediction error signal due to quantization errors, also referred to as quantization noise. The decoded prediction error signal e' is then converted into a prediction signal

number

number

[0007] As a result of quantization, quantization noise is superimposed on the reconstructed video signal. Because coding is performed on a block-by-block basis, the superimposed noise often has blocking characteristics, making block boundaries in the decoded image more noticeable, especially if strong quantization is performed. Blocking artifacts have a negative effect on human visual perception. To reduce these artifacts, a deblocking filter 150 is applied to all reconstructed image blocks. The deblocking filter is applied to the reconstructed signal s'. For example, the deblocking filter in H.264 / MPEG-4 AVC has local adaptive capabilities. When the degree of blocking noise is high, a strong (narrow-bandwidth) low-pass filter is used, and when the degree of blocking noise is low, a weak (wide-bandwidth) low-pass filter is used. The strength of the low-pass filter is determined by the prediction signal s^ and the quantized prediction error signal e'. A deblocking filter generally smooths block edges to improve the subjective quality of the decoded image. Furthermore, because the filtered portion of the image is used for motion-compensated prediction of the next image, filtering also reduces the prediction error, thereby improving coding efficiency.

[0008] After the deblocking filter, a sample adaptive offset 155 and / or an adaptive loop filter 160 may be applied to the image containing the deblocked signal s''. The deblocking filter improves subjective quality, while the sample adaptive offset (SAO) and ALF aim to improve pixel-by-pixel reliability ("objective" quality). In particular, the SAO adds an offset according to a pixel's nearest neighbors. Specifically, the adaptive loop filter (ALF) is used to compensate for image distortion caused by compression. Typically, the adaptive loop filter is a Wiener filter with filter coefficients determined to minimize the mean squared error (MSE) between the reconstructed image s' and the source image s. The ALF coefficients may be calculated and transmitted on a frame-by-frame basis. The ALF can be applied to the entire frame (image of a video sequence) or to local regions (blocks). Additional side information indicating the region to filter may be transmitted (block-by-block, frame-by-frame, or quadtree-by-quadtree).

[0009] Decoding an inter-coded image also requires storing a portion of the coded and decoded image in the reference frame buffer 170. Inter-coded blocks are predicted (180) using motion-compensated prediction. First, a motion detector finds the best match for the current block in the coded and decoded video frame. This best match becomes the prediction signal, and the relative displacement (motion) between the current block and the best match is signaled as motion data in the form of a three-dimensional motion vector. This motion data is included in the side information provided with the coded video data. The three dimensions consist of two spatial dimensions and one spatial dimension. To optimize prediction accuracy, motion vectors may be calculated at a spatial sub-pixel resolution, such as half-pixel or quarter-pixel resolution. Spatial sub-pixel resolution motion vectors may point to spatial locations in the decoded frame where no real pixel values ​​exist, i.e., sub-pixel locations. Therefore, spatial interpolation of such pixel values ​​is required to perform motion-compensated prediction. This may be achieved by an interpolation filter (integrated with the prediction block 180 in FIG. 1).

[0010] In both intra- and inter-coding modes, the difference e between the target input signal and the predicted signal is transformed (110) and quantized (120) to produce quantized coefficients. Typically, an orthogonal transform, such as the two-dimensional discrete cosine transform (DCT) or its integer versions, is used because it effectively reduces correlation in natural video. After the transform, more bits are spent coding low-frequency components than high-frequency components, since low-frequency components are typically more important to image quality than high-frequency components. In an entropy coder, the two-dimensional array of quantized coefficients is converted to a one-dimensional array, typically using a so-called zigzag scan. In a zigzag scan, the array is scanned in a predetermined order from the DC coefficient in the upper left corner to the AC coefficient in the lower right corner. Because energy is typically concentrated in the upper left part of the coefficients in a two-dimensional array, which corresponds to low frequencies, zigzag scanning typically results in an array with a final zero. This allows for efficient encoding using run-length coding, either as part of the actual entropy coding or as a preprocessing step.

[0011] H.264 / MPEG-4 H.264 / MPEG-4 AVC and HEVC have two layers: the Video Coding Layer (VCL) and the Network Abstraction Layer (NAL). The Video Coding Layer provides coding functionality, as briefly described above. The NAL encapsulates information elements into standard units called NAL units for further use, such as transmission over a channel or storage in a storage device. Information elements are, for example, coded prediction error signals or other information required for decoding a video signal, such as prediction types, quantization parameters, and motion vectors. There are VCL NAL units that contain compressed video data and related information, as well as non-VCL units that encapsulate additional data, such as parameter sets related to an entire video sequence, or supplemental enhancement information (SEI), which provides additional information used to improve decoding accuracy.

[0012] FIG. 2 shows an exemplary decoder 200 compliant with the H.264 / MPEG-4 AVC or HEVC video coding standard. The coded video signal (input signal to the decoder) is first sent to an entropy decoder 290, which decodes the quantized coefficients and other information elements required for decoding, such as motion data and prediction modes. The quantized coefficients are inverse scanned to obtain a two-dimensional array, which is then input to an inverse quantization and inverse transform unit 230. After inverse quantization and inverse transform (230), a decoded (quantized) prediction error signal e′ is obtained. In the absence of quantization noise and error, the decoded (quantized) prediction error signal e′ corresponds to the difference obtained by subtracting the predicted signal from the signal input to the encoder.

[0013] The prediction signal is obtained by either temporal or spatial prediction (280). The decoded information typically also includes information necessary for prediction, such as the prediction type in the case of intra prediction, or motion data in the case of motion-compensated prediction. The quantized prediction error signal in the spatial domain is then added by adder 240 to the prediction signal obtained from either motion-compensated or intra-frame prediction (280). The reconstructed image s′ may be sent through a deblocking filter 250, a sample adaptive offset processor 255, and an adaptive loop filter 260, and the resulting decoded signal is stored in memory 270 and used for the temporal or spatial prediction of subsequent blocks / images.

[0014] When compressing and decompressing an image, blocking artifacts are usually the most annoying to the user. By smoothing the edges between blocks in the reconstructed image, deblocking filtering improves the user's perceptual experience. One of the challenges of deblocking filtering is accurately distinguishing between edges that are caused by blocking due to the application of a quantizer and edges that are part of the encoded signal. It is desirable to apply a deblocking filter only when edges occur at block boundaries due to compression artifacts. In other cases, applying a deblocking filter may distort the reconstructed signal. Another challenge is selecting an appropriate filter for deblocking filtering. Typically, a decision is made between multiple low-pass filters with different frequency responses, resulting in strong or weak low-pass filtering. To determine whether to apply deblocking filtering and to select an appropriate filter, the image data present near the boundary between two blocks is taken into account.

[0015] For example, H.264 / MPEG-4 AVC evaluates the absolute value of the first derivative of each of two adjacent blocks whose boundaries are to be deblocked, and also evaluates the absolute value of the first derivative at the edge between the two blocks, as specified in, for example, Section 8.7.2.2 of the H.264 / MPEG-4 AVC standard. HEVC uses a similar mechanism, but also uses the second derivative.

[0016] The deblocking filter may determine for each sample at a block boundary whether to filter it and with which filter or type of filter. If a filter is selected, a low-pass filter is applied to smooth the block boundary. The purpose of this filtering decision is to filter only samples that experience large signal changes at the block boundary as a result of the quantization applied to the block-based processing described in the Background section above. This deblocking filtering results in a smoothed signal at the block boundary. The smoothed signal is less annoying to the viewer than blocking artifacts. Samples that experience large signal changes at the block boundary, belonging to the original signal to be coded, should not be filtered to preserve high frequencies and therefore visual sharpness. If an incorrect decision is made, the image will be unnecessarily smoothed or the block boundaries will remain distorted. Deblocking filtering is performed on both vertical edges of the blocks (horizontal filtering) and horizontal edges of the blocks (vertical filtering).

[0017] 4A illustrates a decision regarding a vertical boundary (whether to filter with a horizontal deblocking filter), and FIG. 4B illustrates a decision regarding a horizontal boundary (whether to filter with a vertical deblocking filter). Specifically, FIG. 4A illustrates a current block 440 and its decoded neighboring blocks 410, 420, and 430. A decision is made for pixel 460 within a line. Similarly, FIG. 4B illustrates the same current block 440, with a decision being made for pixel 470 within a column.

[0018] Similar to H.264 / MPEG-4 AVC, the decision on whether to apply a deblocking filter may be performed as follows. As also shown in FIG. 4 , take a line of six pixels 460 as an example, among which the first three pixels p2, p1, and p0 belong to the left neighboring block A 430, and the next three pixels q0, q1, and q2 belong to the current block B 440. A line 510 indicates the boundary between blocks A and B. Pixels p0 and q0 are pixels of the left neighboring block A and the current block B, respectively, and are adjacent to each other. Pixels p0 and q0 are filtered by the deblocking filter when, for example, the following condition is satisfied:

[0019]

number

number

number

[0020] Here, the following usually holds:

[0021]

number

[0022] These conditions aim to detect whether the difference between p0 and q0 is due to blocking artifacts. They correspond to the evaluation of the first derivative within block A, within block B, and between blocks A and B, respectively.

[0023] For example, pixel p1 is filtered if it satisfies the following conditions in addition to the above three conditions.

[0024]

number

[0025] A pixel q1 is filtered if, for example, in addition to the first three conditions above, it also satisfies the following conditions:

[0026]

number

[0027] These conditions correspond to the first differential in the first block and the first differential in the second block, respectively. In the above conditions, QP denotes the quantization parameter that indicates the amount of quantization applied. β and α are scalar constants. In particular, QP NEW are the quantization parameters QP applied to the first block A and the second block B, respectively, as follows: A and QP B The quantization parameter is derived based on

[0028]

number

[0029] Here, ">>1" means to shift one bit to the right.

[0030] The decision may be made only for a selected line or lines within the block, but pixel filtering is optionally performed for all lines 460. Figure 5 shows an example 520, comprised in part by line 530, where an HEVC-compliant decision is made. Based on line 530, a decision is made whether to filter the entire block.

[0031] Another example of deblocking filtering in HEVC is described in Section 8.6.1 of the JCTVC-E603 document of JTC-VC of ITU-T SG16 WP3 and ISO / IEC JTC1 / SC29 / WG11, which is freely available from http: / / wftp3.itu.int / av-arch / jctvc-site / 2011_01_D_Daegu / .

[0032] Two lines 530 are used to determine whether and how to apply deblocking filtering. In example 520, the third line (index 2) and the sixth line (index 5) are evaluated for horizontal deblocking filtering. In particular, evaluating the second derivative within each block results in the following measurements d2 and d5:

[0033]

number

number

[0034] Pixel p belongs to block A, and pixel q belongs to block B. The number immediately after p or q indicates the column index, and the next subscript number indicates the row number within the block. Deblocking is possible for all eight lines shown in example 520 if the following conditions are met:

[0035]

number

[0036] If the above conditions are not met, no deblocking is applied. If deblocking is possible, a filter to be used for deblocking is determined. This decision is based on evaluating the first derivative between blocks A and B. In particular, for each line i (i is an integer between 0 and 7), a decision is made as to whether to apply a strong or weak low-pass filter. A strong filter is selected if the following conditions are met:

[0037]

number

number

number

[0038] According to the HEVC model, the "strong filter" is i , p2 i , p1 i , p0 i , q0 i , q1 i , q2 i , q3 i Using sample p2 i , p1 i , p0 i , q0 i , q1 i , q2 i On the other hand, the "weak filter" filters the sample p2 i , p1 i , p0 i , q0 i , q1 i , q2 i Using sample p1 i , p0 i , q0 i , q1 i In the above conditions, the parameters β and t c Both are quantization parameters QP Frame and may be set for slices of an image. Usually, β and t c The value of QP is calculated using a lookup table. Frame It is derived based on

[0039] Note that strong filtering only works for very flat signals, otherwise weak low-pass filtering is more advantageous.

[0040] FIG. 6A shows pixels that are subject to strong low-pass filtering in conventional hybrid coding. In particular, FIG. 6A shows samples used for filtering. These samples correspond to four neighboring pixels on the left and right sides of the boundary between blocks A and B. These samples are used for filtering, i.e., their values ​​are input to the filtering process. FIG. 6A also shows samples that are modified by the filter. These are the pixel values ​​of the three neighboring pixels that are closest to the boundary between blocks A and B, respectively, on the left and right sides of the boundary. These values ​​are modified by the filter, i.e., smoothed. In particular, modified sample p0' in line with index i i , p1' i , p2' i , q0' i , q1' i , q2' i The values ​​are shown below.

[0041]

number

number

[0042] The function Clip(x) is defined as follows:

[0043]

number

[0044] where max_allowed_value is the maximum value that x can take. For PCM coding using k-bit samples, the maximum value is max_allowed_value=2 k For example, for PCM coding with 8-bit samples, the maximum value is max_allowed_value=255. For PCM coding with 10-bit samples, the maximum value is max_allowed_value=1023.

[0045] Thus, the above equation shows the process of strong filtering that is applied. As can be seen from the above equation, pixel p3 of row i i and q3 i is used in the above equation, i.e., filtering, but is not modified, i.e., not filtered.

[0046] 6B illustrates the application of a weak deblocking filter. In particular, the samples used for filtering are shown on the left side of the drawing, and the samples modified by filtering are shown on the right side. In the weak filtering process, three neighboring pixels close to the boundary between block A and block B are used, and only two neighboring pixels close to the boundary that belong to block A and block B are filtered. To perform the weak filtering, two decisions are made. The first decision is whether the weak filter should be applied to all lines or not to certain lines. This decision is based on a value Δ, which is calculated as follows:

[0047]

number

[0048] Based on the calculated Δ,

number

[0049] If filtering is applied, it is done as follows:

[0050]

number

number

number

[0051] The function Clip(x) is defined as above. The function Clip3(x) is defined as follows:

[0052]

number

[0053] The application of filtering is determined, and pixel p0' i and p0' i If it is determined that p1' has been filtered, i and p1' i A further decision is made as to whether to filter or not.

[0054] d p Pixel p1' if and only if <(β / 6) i is filtered. Similarly, d q Pixel q1' only if <(β / 6) i are filtered. The filtering of these pixels is done as follows:

[0055]

number

[0056] In this case, the following is used:

[0057]

number

number

number

[0058] The above discussion regarding the decision and selection of a deblocking filter generally applies to luma samples.

[0059] The following method is used for chrominance deblocking filtering: The delta values ​​are calculated as follows:

[0060]

number

[0061] The value Delta1 is calculated based on the Delta value as follows and is used directly for deblocking filtering:

[0062]

number

number

number

[0063] As can be seen from the above sequence of steps in the weak filtering process, the value Δ may be a value that is obtained by performing several determination and filtering steps. In particular, calculating Δ involves: Decide whether to apply a weak filter to a specific line (luminance) p0' i and q0' i (Weak) filtering against p1' i and q1' i The value Δ1 is also used when filtering

[0064] As can be seen, it is advantageous to calculate the value of Δ as accurately as possible. Summary of the Invention

[0065] In view of these problems of the prior art, it would be advantageous to provide an efficient deblocking filtering approach in which the filtering parameters are controlled separately. In a specific approach of the present invention, strong filtering and weak filtering are controlled separately. This is achieved by the features of the independent claims. Preferred embodiments are the subject of the dependent claims. [Brief explanation of the drawings]

[0066] The accompanying drawings are incorporated into and form a part of the specification, illustrating several embodiments of the present invention. The drawings, together with the description of the embodiments, serve to explain the principles of the invention. The purpose of the drawings is only to illustrate preferred and alternative examples of how the invention can be made and used, and should not be construed as limiting the invention to only the embodiments shown and described. Further features and advantages of the present invention will become apparent from the following detailed description of various embodiments of the invention, as illustrated in the accompanying drawings, in which like reference numerals refer to like elements. In the drawings, [Figure 1] FIG. 1 is a block diagram illustrating an example of a video encoder. [Figure 2] FIG. 2 is a block diagram illustrating an example of a video decoder. [Figure 3] FIG. 3 is a block diagram illustrating an example of a video encoder that performs vertical filtering and horizontal filtering separately. [Figure 4A] FIG. 4A is a schematic diagram illustrating the application of horizontal deblocking filtering. [Figure 4B] FIG. 4B is a schematic diagram illustrating the application of vertical deblocking filtering. [Figure 5] FIG. 5 is a schematic diagram illustrating the decision on whether to apply a deblocking filter and the selection of a deblocking filter. [Figure 6A]FIG. 6A is a schematic diagram illustrating an example of pixels near a common boundary of two blocks that are subject to strong and weak deblocking filtering. [Figure 6B] FIG. 6B is a schematic diagram illustrating an example of pixels near a common boundary of two blocks that are subject to strong and weak deblocking filtering. [Figure 7] FIG. 7 is a table showing the results of the rounding process. [Figure 8] FIG. 8 is a schematic diagram illustrating an asymmetric delta quantizer. [Figure 9A] FIG. 9A is a schematic diagram illustrating quantization of delta according to an example of the present invention. [Figure 9B] FIG. 9B is a schematic diagram illustrating quantization of delta according to an example of the present invention. [Figure 10] FIG. 10 is a flow chart illustrating a method according to one embodiment of the present invention. [Figure 11] FIG. 11 illustrates the results achieved by one embodiment of the present invention, compared with the prior art. [Figure 12] FIG. 12 is a diagram showing the overall configuration of a content supply system that implements a content distribution service. [Figure 13] FIG. 13 is a diagram showing the overall configuration of a digital broadcasting system. [Figure 14] FIG. 14 is a block diagram showing an example of the configuration of a television. [Figure 15] FIG. 15 is a block diagram showing an example of the configuration of an information reproducing / recording unit that reads and writes information from and to a recording medium that is an optical disc. [Figure 16] FIG. 16 is a diagram showing an example of the structure of a recording medium that is an optical disc. [Figure 17A] FIG. 17A is a diagram showing an example of a mobile phone. [Figure 17B] FIG. 17B is a block diagram showing an example of the configuration of a mobile phone. [Figure 18] FIG. 18 is a diagram showing the structure of multiplexed data. [Figure 19]FIG. 19 is a diagram showing a schematic diagram of how each stream is multiplexed in multiplexed data. [Figure 20] FIG. 20 shows in more detail how a video stream is stored in a sequence of PES packets. [Figure 21] FIG. 21 is a diagram showing the structure of TS packets and source packets in multiplexed data. [Figure 22] FIG. 22 shows the data structure of a PMT. [Figure 23] FIG. 23 is a diagram showing the internal structure of the multiplexed data information. [Figure 24] FIG. 24 shows the internal structure of the stream attribute information. [Figure 25] FIG. 25 shows the steps for identifying video data. [Figure 26] FIG. 26 is a diagram showing an example of the configuration of an integrated circuit that realizes the video encoding method and the video decoding method according to each embodiment. [Figure 27] FIG. 27 is a diagram showing a configuration for switching the drive frequency. [Figure 28] FIG. 28 is a diagram showing steps for identifying video data and switching the drive frequency. [Figure 29] FIG. 29 is a schematic diagram showing an example of a lookup table in which video data standards and drive frequencies are associated with each other. [Figure 30A] FIG. 30A is a diagram showing an example of a configuration in which modules of a signal processing unit are shared. [Figure 30B] FIG. 30B is a diagram showing another example of a configuration in which modules of a signal processing unit are shared. DETAILED DESCRIPTION OF THE INVENTION

[0067] As described above, the delta (Δ) value is calculated by the following shift operation:

[0068]

number

[0069] The shift operation can be interpreted as a quantization of the delta value (to perform integer division), and corresponds to the following unquantized delta value:

[0070]

number

[0071] So, shifting four bits to the right (">>") is equivalent to dividing by 16. The shift operation not only introduces quantization error, but also causes the resulting delta value to have an offset relative to zero mean after quantization.

[0072] Figure 7 illustrates partial results of a rounding operation using a shift of 4 bits. The first column of the table in Figure 7 illustrates variables i ranging from -20 to +20. The second column shows the corresponding value of i after adding an offset value of 8. The third column shows a shift of 4 bits corresponding to an integer division of the term (i + 8) by 16. As can be seen from the table, the distribution of the resulting values ​​in the third column is asymmetric and does not have zero mean, even though i does have zero mean.

[0073] This results in an asymmetric quantizer characteristic as shown in Figure 8. Even though the unquantized delta values ​​have a general symmetric probability distribution, after quantization as described above, an offset value is generated, leading to the following equation:

[0074]

number

[0075] A delta value is calculated to perform the deblocking filtering process. It is generated as a weighted sum of sample values ​​at the boundary between two adjacent blocks, to which a constant offset value is added. In this example, the constant is 8. The result thus obtained is then clipped by shifting it to the right by one or more bits. For a weighted sum of symmetrically distributed sample values, the resulting delta value produces an offset value, i.e., its expected value is not zero.

[0076] To overcome this problem, we add a different constant to the weighted sum of sample values ​​before shifting them to the right. The different constant is chosen so that the expected value of the delta is zero. In particular, the quantization of the delta values ​​may be performed as follows:

[0077]

number

[0078] The values ​​a and b are chosen to result in a zero-mean distribution of delta values.

[0079] For example, in the first example, a is 7 (a=7) and b is 8 (b=8). Figure 9A shows the resulting symmetric quantizer characteristics when a=7 and b=8. If the probability distribution of the unquantized delta values ​​is symmetric, the quantizer will not produce additional offset values.

[0080]

number

[0081] However, the present invention is not limited to specific values ​​of the parameters a and b. The constants a and b may be arbitrarily selected so that the distribution of quantized delta values ​​has zero mean. For example, in the second example, the constant a is 6 (a=6) and the constant b is 9 (b=9). A corresponding quantizer is shown in FIG. 9B. The dead band width of the quantizer in FIG. 9B is wider than the dead band width of the quantizer in FIG. 9A. In this way, the dead band of the quantizer may be controlled by selecting the parameters a and b. As in the example of FIG. 9A, the average value of the quantized delta values ​​has no offset with respect to the average value of the delta values ​​before quantization.

[0082] When using a shift of 4 bits (">>4"), which is usually applied to luminance samples, it is advantageous to choose a=6 and b=9 or a=7 and b=8. On the other hand, when applying a shift of 3 bits (">>3"), values ​​of a=2 and b=5 are advantageous. A shift of 3 bits to the right is usually performed when deblocking chrominance samples. When shifting by 1 bit (">>1"), values ​​of a=0 and b=1 may be advantageously applied.

[0083] As can be seen from the above equation, symmetric quantization of delta values ​​requires an additional "if" operation to appropriately quantize the delta values. k This may increase the computational load. To efficiently perform such symmetric quantization of delta values, this quantization may be combined with a subsequent clipping operation. In the following, the calculation of the delta values ​​and the subsequent operation of comparing the calculated delta values ​​with a threshold t c The following pseudo code illustrates a comparison between

[0084] delta1 = (9*(q0-p0) -3*(q1-p1)+8)>>4 if (delta1>tc) { delta1=tc; } else if (delta1<-tc) { delta1=-tc; }

[0085] This operation involves two comparisons to non-zero values. Two comparisons are performed in each execution of the code. Alternatively, the following pseudocode may accomplish the same function:

[0086] delta1 = (9*(q0-p0) -3*(q1-p1)+8)>>4 if (delta1>0) { if (delta1>tc) { delta1=tc; } } else { if (delta1<-tc) { delta1=-tc; } }

[0087] In this case, one comparison to zero (the sign comparison) is made, and only one of the two comparisons to non-zero digits is made (because the two comparisons to non-zero digits are alternatives).

[0088] According to one embodiment of the present invention, the threshold t for calculating the delta value and for performing the subsequent clipping process is c The comparison with may be achieved as follows:

[0089] delta = 9*(q0-p0) -3*(q1-p1) if (delta>0) { delta1=(delta+a)>>4; if (delta1>tc) { delta1=tc; } } else { delta1=(delta+b)>>4; if (delta1<-tc) { delta1=-tc; } }

[0090] As can be seen from the pseudocode above, no additional processing is required when comparing with the code above: only one comparison with zero is required, and one of two further comparisons with non-zero values ​​will be performed during code execution.

[0091] Figure 10 is a flow chart illustrating a method according to an embodiment of the present invention. For each line of a block 1010, a decision parameter is calculated. This is done by calculating 1020 a weighted sum of adjacent samples, the samples forming a sample line at the boundary between the current block and its neighboring blocks. A sample line may be a column or a row of a block. The samples may be, for example, pixels p0 as described above. i , q0 i , p1 i , q1 i In this case, for deblocking vertical boundaries, the samples form the rows of the block. However, the invention is equally applicable to deblocking filtering of horizontal boundaries. In this case, the pixels to be filtered or used for filtering form the columns. After calculating the weighted sum (1020), an offset value is determined (1030). The offset value depends on the value of the weighted sum. The offset value is added to the weighted sum (1040). The weighted sum thus obtained after adding the offset value is shifted to the right by a predetermined number of bits (1050). This shift results in an integer division. This may be performed separately for luminance and chrominance samples.

[0092] The decision parameter corresponds to the weighted sum shifted after adding the offset. The decision parameter may further be used to determine (1060) whether a deblocking filter is applied to samples of the current block at the boundary. If it is determined in 1070 that deblocking filtering is applied ("yes" in step 1270), the line is deblocked (1280). That is, the samples of the current block at the boundary (e.g., p0' i ) are filtered. If a different decision is made in 1070 (step 1270 "no"), deblocking filtering is not applied to the same sample and / or the entire line. However, the decision process does not necessarily have to be performed. It may be advantageous to perform the decision process based on decision parameters calculated only for luminance samples. For chrominance samples, the decision on whether to apply a deblocking filter or not may be omitted and they may be filtered directly. However, the present invention is also not limited to the above-mentioned different handling of luminance and chrominance. In general, no decision process needs to be performed at all, and filter selection and / or filtering may be performed based on decision parameters. Therefore, the decision parameters may be used to modify (filter) samples, and may therefore be called "sample modification parameters."

[0093] According to one embodiment of the present invention, if the weighted sum is positive, the offset value takes a first value. If the weighted sum is negative, the offset value takes a second value different from the first value. If the weighted sum has a value of 0, the offset value may further be defined as the first value, the second value, or the first value. The first and second values ​​are advantageously determined such that the average of the weighted sum coincides with the average of the determination parameters. In this way, the integer division operation performed as a right shift does not artificially shift the average value.

[0094] Advantageously, if the predetermined number of bits is 4, the first value is 6 and the second value is 9, or the first value is 7 and the second value is 8. Advantageously, if the predetermined number of bits is 3, the first value is 2 and the second value is 5. In yet another case, if the predetermined number of bits is 1, the first value may be 0 and the second value may be 1. Note that the values ​​may be swapped, i.e. the first value may become the second value and the second value may become the first value. For deblocking filtering of luminance samples, a predetermined value (number of bits to shift) of size 4 is generally used. For filtering luminance, a shift value of 1 bit is generally used. On the other hand, a predetermined value with a shift of 3 bits is generally used for filtering the chrominance part of the image.

[0095] According to a preferred embodiment of the present invention, the method further comprises the step of calculating a filtering parameter by clipping the absolute value of the decision parameter to a clipping value. The filtering parameter is used to filter samples at the boundary of the current block. In particular, the filtering is performed by applying the filtering parameter to the samples, and the result may be clipped. To simplify the process and reduce the computational load, the decision parameter and the filtering parameter may be calculated by the following steps:

[0096] First, calculate the weighted sum of the pixel values ​​of the line. If the weighted sum is greater than 0, offset the weighted sum by a first value and shift it by a predetermined number of bits to obtain the decision parameter. If the decision parameter is greater than the clipping value, clip the decision parameter to the clipping value to obtain the filtering parameter. Otherwise, the filtering parameter is equal to the decision parameter.

[0097] On the other hand, if the weighted sum is less than or equal to 0, the weighted sum is offset by a second value and shifted by a predetermined number of bits to obtain the decision parameter. If the decision parameter is less than the negative clipping value, the decision parameter is clipped to the negative clipping value to obtain the filtering parameter. Otherwise, the filtering parameter is equal to the decision parameter. In the above, it is assumed that the clipping value is a positive integer.

[0098] As mentioned above, integer processing can result in efficiency and / or quality losses when performing deblocking filtering, and the deblocking process can introduce additional quantization errors.

[0099] According to another aspect of the present invention, the maximum correction by the deblocking filter is controlled in addition to adjusting the subjective quality by controlling the ratio of the three selections of strong filtering, weak filtering, and no filtering. This allows the adjustment of the objective quality by limiting the quantization error caused by deblocking. Therefore, the objective quality can be adjusted independently of the subjective quality and adjusted in an optimal way. In particular, in HM4.0, the threshold t c is determined as follows:

[0100]

number

number

number

number

[0101] Parameter QP max indicates the maximum possible QP value. This value is usually 51, and the upper clipping threshold is 55. max+4. The parameter "bitdepthscale" is a scaling factor that depends on the bit depth of the samples to be deblocked. For example, the input signal may have a bit depth of 8, while the internal bit depth may be 10. The parameter BS indicates the "boundary strength" and its value usually differs between intra-prediction and inter-prediction. For example, for intra-prediction, it may take the value 2 or 3, and for inter-prediction, it may take the value 0, 1, or 2.

[0102] tctable indicates the range of possible values, for example: Const UChar tctable_8X8

[56] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 6, 6, 7, 8, 9, 9, 10, 10, 11, 11, 12, 12, 13, 13, 14, 14 } ;

[0103] Similarly, the threshold β is determined as follows:

[0104]

number

number

[0105] "Betatable" is defined, for example, as follows: Const UChar betatable_8X8

[52] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64 };

[0106] In "CE12: Slice-level deblocking filter parameter adjustment" of JCTVC-F143, an additional offset value coded in the slice header is used to adjust the threshold t c It is proposed to seek

[0107]

number

number

[0108] where tctable is the same as above.

[0109] Similarly, JCTVC-F143 proposes adding an additional offset value coded in the slice header to the calculation of the threshold β as follows:

[0110]

number

number

[0111] The additional offset value in JCTVC-F143 affects the effectiveness or ineffectiveness of deblocking for the entire block. In particular, a block is filtered only if, as explained in the background section, d = d q + d p < β

[0112] If deblocking filtering is enabled, the parameter β is further used to adjust the ratio between strong and weak filtering. For each line or column, it determines whether to apply a weak or strong filter, and whether to apply a strong filter to the pixel next closest to the boundary (p1 i , q1 i ) is used to decide whether to deblock or not. However, it does not affect the strong filtering process itself.

[0113] When making a decision about whether to apply a weak filter, a threshold t c Also, the parameter t c further includes the calculation of the parameter Delta1 and the sample p0' as described above. i and q0' i The parameter t is used for the actual filtering. c further adjusts the maximum correction by the deblocking filter (see clip3 values ​​a and b of the filtering process described in the Background section).

[0114] According to one embodiment of the present invention, the deblocking filtering is further controlled by advanced parameters that may be transmitted at the picture level, slice level, or sequence level. In particular, the deblocking filtering is controlled by the parameter t c and parameter β (as shown in the Background section). These parameters are used to determine the strength of filtering and / or whether to apply filtering at all. They also have a direct impact on the clipping process during filtering. The decision to apply weak / strong filtering primarily affects the subjective image quality, whereas clipping is considered to primarily affect the objective quality. The present invention makes it possible to separately control the processes related to objective and subjective quality in deblocking filtering.

[0115] This allows for different deblocking decisions and filtering operations with different offset values ​​that may be indicated in (or embedded in) the slice header, depending on the threshold t c In particular, the first threshold t c1 may be calculated as follows:

[0116]

number

number

[0117] The second threshold t c2 may be defined as follows:

[0118]

number

number

[0119] That is, the calculated offset (tc offset ) and the first value indicated in the bitstream (tc offset、coded ) based on the quantization parameter (QP), the first parameter t c1 is obtained. The calculated offset (tc offset ) and a second value (tc offset2、coded ) based on the quantization parameter (QP), the second parameter t c2 is obtained.

[0120] Two different thresholds t c1 and t c2is used in a series of steps for determining and selecting a deblocking filter as follows: As shown in the prior art, the decision as to whether to apply a deblocking filter to an entire block of 8x8 pixels may be made. The decision as to whether to apply a strong or weak deblocking filter to a particular line or column of pixels is made by a first threshold t c1 In particular, a strong filter is applied in the following cases:

[0121]

number

[0122] In all cases, a weak deblocking filter is applied. Therefore, the first parameter (threshold) t c1 is used to control the subjective quality, i.e., to control the application of weak or strong filtering.

[0123] The delta value is calculated in a similar manner to that shown in the background of the invention, namely as follows:

[0124]

number

[0125] The decision as to whether to perform weak filtering or not is also determined by a first threshold t c1 In particular, filtering is only applied if:

[0126]

number

[0127] Otherwise, no filtering is applied to line i. If it is decided to apply a weak deblocking filter, then pixel p0' at the block boundary is i and q0'i is filtered. According to this embodiment of the present invention, a threshold t c2 The second value of is used to perform the filtering. The filtering may be performed as follows:

[0128]

number

number

[0129] After filtering the pixels on the boundary between two blocks A and B, pixel p1' of block A that is the second closest to the boundary is i and pixel q1' of block B i In particular, these two pixels are each filtered if and only if:

[0130]

number

number

[0131] Filter pixel p1' i If we decide to filter, we set a second threshold t c2 Using pixel p1' i This is because the second threshold controls the filtering process and influences the clipping process, thereby affecting the objective quality. The filtering may be performed as follows:

[0132]

number

number

[0133] Similarly, pixel q1' i If it is decided to filter, the filtering may be done as follows:

[0134]

number

number

[0135] Therefore, the second parameter (threshold) t c2 are used in the filtering process. In particular, the second parameter controls the maximum (upper) and minimum (lower) clipping values ​​(a and b in the clip3 process, as described above). The parameter t c1 and t c2 are determined separately, and as mentioned above, their values ​​may be different.

[0136] Advantageously, as shown below, the strong filtering is performed by a threshold t c3 Also based on.

[0137]

number

number

number

number

number

number

[0138] The third parameter, t c3controls the filtering of a "strong filter", i.e., a filter applied to the three samples (pixels) closest to the block boundary. In particular, it controls the filtering by setting the upper and lower clipping thresholds (-tc3, tc3) to calculate an offset value added to the filtered sample. This offset value is calculated as a weighted sum of the pixels around the boundary of blocks A and B, offset by a given value and shifted by a given number of bits.

[0139] The third threshold t c3 is the second threshold t c2 The third threshold t c3 and the second threshold t c2 However, there is a threshold t that is coded and embedded in the bitstream at the slice level or at another level, such as the frame level, block level, or each level of several frames. c1 and t c2 Apart from this, there is a third value t c3 may be determined.

[0140] Different thresholds t c1 and t c2 The use of these thresholds improves coding efficiency. offset、coded and tc offset2、coded may be rate-distortion optimized by using

[0141] FIG. 11 shows the coding efficiency improvement when applying two thresholds according to an embodiment of the present invention, compared to the coding efficiency of HM4.0 and JCTVC-F143. FIG. 11 shows the peak signal to noise ratio (dB) as a function of data rate. To obtain these results, the video sequence "basketball-pass" was compressed using a quantization parameter with a value of QP=37 and a low-delay, highly efficient prediction structure. The coding efficiency of the present invention is evaluated for an intra-only structure, i.e., spatial prediction, based on the coding conditions typically used in ISO and ITU standardization activities (see document JCTVC-E700: http: / / phenix.int-evry.fr / jct / doc_end_user / current_document.php?id=2454). As can be seen from FIG. 11, tc offset , coded=-5 and β offet , coded It is proposed in JCTVC-F143 to reduce coding efficiency by adjusting subjective quality using t = -2. In the above embodiment of the present invention, it is possible to maintain subjective quality because the first parameter t c1 is the second parameter t c2 At the same time, the objective image quality can be determined by the independent parameter t c2 As mentioned above, the parameter t c1 and t c2 By using two different offset values ​​obtained by rate-distortion optimization, this embodiment of the present invention improves coding efficiency.

[0142] The present invention is not limited to the examples given above, and in general, for each decision step (whether to apply a filter or not, or which filter to apply to a particular sample, line, or block) and for each filtering step, individual parameters may be used.

[0143] The individual parameter values ​​allow for more flexible adaptation to image content and characteristics, potentially achieving higher coding efficiency. To reduce the overhead of signaling the individual parameters used to determine and apply deblocking filtering, the parameters may be further coded by differential coding or by utilizing advantageous entropy coding. For example, the first offset value tc offset、coded explicitly encode the remaining offset values ​​(threshold t c3 may be used to calculate tc offset、coded2 or tc offset、coded3 , or other additional offset values) may be coded as differences or mutual differences with the first offset value.

[0144] Alternatively, the threshold t c may be determined as follows:

[0145]

number

number

[0146] Therefore, an additional offset value is simply added to the value of tctable. This offset value may be coded and signaled in the slice header. As in the above example, a table offset value may be determined for each filter decision (strong or weak filter) and filtering operation (e.g., clipping control). The additional offset value reduces the value t c This allows for finer adjustment of , since it allows for values ​​greater than 14, assuming the example values ​​of tctable above.

[0147] Similarly, a more sophisticated adaptation to the threshold β may be performed as follows:

[0148]

number

number

[0149] Assuming the example value of betatable above, additional values ​​such as β=55 are possible, so the additional offset value β offset , coded may be used to achieve finer adjustments.

[0150] It should be noted that the above examples do not limit the present invention to this particular filtering process. For example, the above examples have been described primarily with respect to general filtering of luminance samples. However, the present invention is also applicable to filtering applied to chrominance samples, for example, filtering performed without determining whether to filter specific lines within a block. The idea behind the above embodiments is to enable separate control of subjective quality and objective quality. In particular, advanced parameters are used to separately control the filtering process by determining whether to apply a deblocking filter and controlling processes included in the filtering process, such as clipping. This idea may apply to only a portion of the above processes. It should be noted that the above examples do not limit the present invention to the specific filtering of HEVC.

[0151] A further embodiment of the present invention is summarized as follows: A method for deblocking filtering samples of a current block of an image, the method comprising the steps of: calculating a weighted sum of adjacent samples forming a sample line at a boundary between the current block and its adjacent blocks; determining sample modification parameters by adding an offset value to the weighted sum, the offset value being determined by the value of the weighted sum, and shifting the weighted sum after adding the offset value to the right by a predetermined number of bits; and applying a deblocking filter to the samples including the offset value using the sample modification parameters.

[0152] Advantageously, the offset value has a first value when the weighted sum is positive, and a second value different from the first value when the weighted sum is negative. The first and second values ​​are determined so that the average of the sample correction parameters shifted left by a predetermined number of bits matches the average of the weighted sum. Specifically, the predetermined number of bits is 4, the first value is 6, and the second value is 9, or the first value is 7 and the second value is 8. Alternatively, the predetermined number of bits is 3, the first value is 2, and the second value is 5. Alternatively, the predetermined number of bits is 1, the first value is 0, and the second value is 1.

[0153] The method may further include calculating a filtering parameter by clipping the absolute value of the sample modification parameter to a clipping value, and filtering the sample by adding the filtering parameter to the sample. Here, the sample modification parameter and the filtering parameter are calculated by the following steps: calculating a weighted sum; if the weighted sum is greater than zero, offsetting the weighted sum by a first value and shifting it by a predetermined number of bits to obtain the sample modification parameter; if the sample modification parameter is greater than the clipping value, clipping the sample modification parameter to the clipping value to obtain the filtering parameter; otherwise, the filtering parameter matches the sample modification parameter; on the other hand, if the weighted sum is less than or equal to zero, offsetting the weighted sum by a second value and shifting it by a predetermined number of bits to obtain the sample modification parameter; if the sample modification parameter is less than the negative clipping value, clipping the sample modification parameter to the negative clipping value to obtain the filtering parameter; otherwise, the filtering parameter matches the sample modification parameter.

[0154] The method may include determining whether a deblocking filter is to be applied to the samples of the current block at the boundary based on the sample modification parameters, and / or applying or not applying the deblocking filter to the samples depending on the result of the determining step.

[0155] Another embodiment provides a method for deblocking filtering samples of a current block of an image, the method comprising: determining whether to apply a deblocking filter based on a comparison between a decision value and a predetermined threshold; filtering the samples, the filtering step comprising offsetting the samples with sample correction values; and calculating the predetermined threshold and the sample correction values, the offset values ​​being determined separately for calculating the predetermined threshold and the sample correction values.

[0156] Advantageously, both the offset values ​​for calculating the predetermined thresholds and the individual offset values ​​for calculating the sample correction values ​​are included in the bitstream of the coded image data.

[0157] Another embodiment presents an apparatus for deblocking filtering samples of a current block of an image, the apparatus comprising: a sum calculation unit for calculating a weighted sum of adjacent samples forming a sample line at the boundary between the current block and its adjacent blocks; an adder unit for adding an offset value determined by the value of the weighted sum to the weighted sum; and a shifter unit for shifting the weighted sum after adding the offset value to the right by a predetermined number of bits; and a filtering unit for applying a deblocking filter, including an offset operation, to the samples according to the sample modification parameters.

[0158] Advantageously, the offset value has a first value when the weighted sum is positive and a second value different from the first value when the weighted sum is negative, the first and second values ​​being determined such that the average of the sample modification parameters shifted left by a predetermined number of bits matches the average of the weighted sum.

[0159] Specifically, the predetermined number of bits is 4, the first value is 6, and the second value is 9, or the first value is 7 and the second value is 8. Alternatively, the predetermined number of bits is 3, the first value is 2, and the second value is 5. Alternatively, the predetermined number of bits is 1, the first value is 0, and the second value is 1.

[0160] Advantageously, the calculation unit calculates the filtering parameters by clipping the absolute values ​​of the sample modification parameters to a clipping value, and filtering of the samples includes adding the filtering parameters to the samples. Here, the sum calculation unit calculates a weighted sum, and the calculation unit calculates the sample modification parameters and the filtering parameters as follows: If the weighted sum is greater than zero, the adder offsets the weighted sum by a first value and shifts it by a predetermined number of bits to obtain the sample modification parameters. If the sample modification parameters are greater than the clipping value, the calculation unit clips the sample modification parameters to the clipping value to obtain the filtering parameters. Otherwise, the filtering parameters match the sample modification parameters. On the other hand, if the weighted sum is less than or equal to zero, the adder offsets the weighted sum by a second value and shifts it by a predetermined number of bits to obtain the sample modification parameters. If the sample modification parameters are less than the negative clipping value, the calculation unit clips the sample modification parameters to the negative clipping value to obtain the filtering parameters. Otherwise, the filtering parameters match the sample modification parameters.

[0161] The device may further include a judgment unit that judges whether a deblocking filter is to be applied to the samples of the target block at the boundary based on the sample modification parameters, and / or a filtering unit that applies or does not apply the deblocking filter to the samples depending on the judgment result.

[0162] Another embodiment provides an apparatus for deblocking filtering samples of a current block of an image, the apparatus comprising: a decision unit for determining whether to apply a deblocking filter based on a comparison between a decision value and a predetermined threshold, a filtering unit for filtering the samples, the filtering unit comprising a process of offsetting the samples with sample modification values, and a calculation unit for calculating the predetermined threshold and the sample modification values, the calculation unit comprising a process of adding an offset value to a quantization parameter associated with the samples, wherein the offset value is determined separately in calculating the predetermined threshold and in calculating the sample modification values.

[0163] Advantageously, both the offset values ​​for calculating the predetermined thresholds and the individual offset values ​​for calculating the sample correction values ​​are included in the bitstream of the encoded image data.

[0164] By recording a program for implementing the video coding method (image coding method) or video decoding method (image decoding method) shown in each of the above embodiments on a storage medium, it becomes possible to easily perform the processes shown in each of the above embodiments on an independent computer system. The storage medium may be a magnetic disk, optical disk, magneto-optical disk, IC card, semiconductor memory, or any other medium capable of recording a program.

[0165] Furthermore, here, we will explain application examples of the video coding method (image coding method) and video decoding method (image decoding method) shown in each of the above embodiments, and a system using the same. The system is characterized by having an image coding / decoding device consisting of an image coding device using the image coding method and an image decoding device using the image decoding method. Other components of the system can be appropriately changed depending on the situation.

[0166] (Embodiment A) 12 is a diagram showing the overall configuration of a content supply system ex100 that provides a content distribution service. The area where communication services are provided is divided into cells of a desired size, and base stations ex106, ex107, ex108, ex109, and ex110, which are fixed wireless stations, are installed in each cell.

[0167] This content supply system ex100 is connected to the Internet ex101 via an Internet service provider ex102, a telephone network ex104, and base stations ex106 to ex110, and devices such as a computer ex111, a PDA (Personal Digital Assistant) ex112, a camera ex113, a mobile phone ex114, and a game console ex115.

[0168] However, the content supply system ex100 is not limited to the configuration shown in Fig. 12, and any combination of elements may be connected. Also, each device may be directly connected to the telephone network ex104 without going through base stations ex106 to ex110, which are fixed wireless stations. Also, each device may be directly connected to each other via short-range wireless or the like.

[0169] The camera ex113 is a device capable of shooting moving images, such as a digital video camera, and the camera ex116 is a device capable of shooting still images and moving images, such as a digital camera. The mobile phone ex114 may be any of a GSM (registered trademark) (Global System for Mobile Communications) system, a CDMA (Code Division Multiple Access) system, a W-CDMA (Wideband-Code Division Multiple Access) system, an LTE (Long Term Evolution) system, an HSPA (High Speed ​​Packet Access) mobile phone, or a PHS (Personal Handyphone System) system.

[0170] In the content supply system ex100, a camera ex113 and the like are connected to a streaming server ex103 via a base station ex109 and a telephone network ex104, thereby enabling live streaming and the like. In live streaming, a user shoots content (e.g., video of a live music concert) using the camera ex113, and encodes the content as described in the above embodiments (i.e., the camera functions as an image encoding device according to an aspect of the present invention) and transmits the content to the streaming server ex103. The streaming server ex103 then streams the transmitted content data to a client that has made a request. Examples of clients include a computer ex111, a PDA ex112, a camera ex113, a mobile phone ex114, a game console ex115, and the like that are capable of decoding the encoded data. Each device that receives the distributed data decodes and plays back the received data (i.e., functions as an image decoding device according to an aspect of the present invention).

[0171] The encoding process of the captured data may be performed by the camera ex113, by the streaming server ex103 that processes the data transmission, or by a mutually shared responsibility. Similarly, the decoding process of the distributed data may be performed by the client, by the streaming server ex103, or by a mutually shared responsibility. Furthermore, still images and / or video data captured by camera ex116, not limited to camera ex113, may be transmitted to the streaming server ex103 via computer ex111. In this case, the encoding process may be performed by the camera ex116, the computer ex111, or the streaming server ex103, or by a mutually shared responsibility.

[0172] Furthermore, these encoding and decoding processes are generally performed by the computer ex111 or an LSIex500 possessed by each device. The LSIex500 may be a single chip or may be configured with multiple chips. It is also possible to embed video encoding and decoding software on some kind of recording medium (CD-ROM, flexible disk, hard disk, etc.) that can be read by the computer ex111, etc., and perform the encoding and decoding processes using that software. Furthermore, if the mobile phone ex114 is equipped with a camera, video data captured by the camera may be transmitted. This video data is data that has been encoded and processed by the LSIex500 possessed by the mobile phone ex114.

[0173] The streaming server ex103 may also be a plurality of servers or computers that process, record, and distribute data in a distributed manner.

[0174] In this way, the content delivery system ex100 allows a client to receive and play back encoded data. In this way, the content delivery system ex100 allows a client to receive, decode, and play back information sent by a user in real time, enabling even users without special rights or equipment to realize personal broadcasting.

[0175] In addition to the example of the content supply system ex100, as shown in FIG. 13, at least one of the video encoding device (image encoding device) or video decoding device (image decoding device) according to each of the above embodiments can also be incorporated into a digital broadcasting system ex200. Specifically, a broadcasting station ex201 transmits multiplexed data, in which music data and the like are multiplexed onto video data, via radio waves to a satellite ex202. This video data is data encoded using the video encoding method described in each of the above embodiments (i.e., data encoded by an image encoding device according to one aspect of the present invention). Receiving this, the broadcasting satellite ex202 transmits broadcasting radio waves, which are received by a home antenna ex204 capable of receiving satellite broadcasts. The received multiplexed data is decoded and played back by a device such as a television (receiver) ex300 or a set-top box (STB) ex217 (i.e., functions as an image decoding device according to one aspect of the present invention).

[0176] The video decoding device or video encoding device described in each of the above embodiments can also be implemented in a reader / recorder ex218 that reads and decodes multiplexed data recorded on a recording medium ex215 such as a DVD or Blu-ray, or encodes a video signal onto the recording medium ex215 and, in some cases, multiplexes it with an audio signal before writing it. In this case, the reproduced video signal is displayed on a monitor ex219, and the video signal can be reproduced in another device or system using the recording medium ex215 on which the multiplexed data is recorded. Alternatively, a video decoding device may be implemented in a set-top box ex217 connected to a cable television cable ex203 or a satellite / terrestrial broadcast antenna ex204, and the video may be displayed on the television monitor ex219. In this case, the video decoding device may be incorporated into the television rather than the set-top box.

[0177] 14 is a diagram showing a television (receiver) ex300 that uses the video decoding method and video encoding method described in each of the above embodiments. The television ex300 includes a tuner ex301 that acquires or outputs multiplexed data in which audio data is multiplexed onto video data via an antenna ex204 that receives the broadcasts or a cable ex203, a modulation / demodulation unit ex302 that demodulates the received multiplexed data or modulates it into multiplexed data to be transmitted externally, and a multiplexing / demultiplexing unit ex303 that separates the demodulated multiplexed data into video data and audio data or multiplexes the video data and audio data encoded by a signal processing unit ex306.

[0178] The television ex300 also has a signal processing unit ex306 having an audio signal processing unit ex304 and a video signal processing unit ex305 (which function as an image encoding device or an image decoding device according to an embodiment of the present invention) that decode the audio data and the video data, respectively, or encode the respective information, and an output unit ex309 having a speaker ex307 that outputs the decoded audio signal and a display unit ex308 such as a display that displays the decoded video signal.The television ex300 also has an interface unit ex317 that has an operation input unit ex312 that accepts user operation input, etc.The television ex300 also has a control unit ex310 that controls each unit overall, and a power supply circuit unit ex311 that supplies power to each unit. In addition to the operation input unit ex312, the interface unit ex317 may have a bridge ex313 connected to an external device such as a reader / recorder ex218, a slot unit ex314 for allowing a recording medium ex216 such as an SD card to be attached, a driver ex315 for connecting to an external recording medium such as a hard disk, a modem ex316 for connecting to a telephone network, etc. The recording medium ex216 is a non-volatile / volatile semiconductor memory element that stores information and allows it to be electrically recorded. The various units of the television ex300 are connected to each other via a synchronous bus.

[0179] First, a configuration in which the television ex300 decodes and plays back multiplexed data acquired from an external source via an antenna ex204 or the like will be described. The television ex300 receives user operation via a remote controller ex220 or the like, and, under the control of a control unit ex310 having a CPU or the like, separates the multiplexed data demodulated by a modulation / demodulation unit ex302 in a multiplexing / separation unit ex303. The television ex300 then decodes the separated audio data in an audio signal processing unit ex304 and decodes the separated video data in a video signal processing unit ex305 using the decoding method described in each of the above embodiments. The decoded audio and video signals are output to the outside from an output unit ex309. When outputting, it is preferable to temporarily store these signals in buffers ex318, ex319, or the like so that the audio and video signals are played back in sync. The television ex300 may also read the multiplexed data from recording media ex215, ex216, such as magnetic / optical discs or SD cards, rather than from broadcasts or the like. Next, a configuration in which the television ex300 encodes audio and video signals and transmits them externally or writes them to a recording medium or the like will be described. The television ex300 receives user operation from a remote controller ex220 or the like, and, under the control of the control unit ex310, encodes the audio signal in the audio signal processing unit ex304 and encodes the video signal in the video signal processing unit ex305 using the encoding method described in each of the above embodiments. The encoded audio and video signals are multiplexed by the multiplexing / demultiplexing unit ex303 and output externally. When multiplexing, these signals may be temporarily stored in buffers ex320, ex321, etc., so that the audio and video signals are synchronized. Note that multiple buffers ex318, ex319, ex320, and ex321 may be provided as shown, or one or more buffers may be shared. Furthermore, data may be stored in buffers other than those shown in the figure, for example, between the modulation / demodulation unit ex302 and the multiplexing / demultiplexing unit ex303, as a buffer to prevent system overflow and underflow.

[0180] Furthermore, in addition to acquiring audio data and video data from broadcasts, recording media, etc., the television ex300 may also be configured to accept AV input from a microphone or camera and perform encoding processing on the data acquired from them. Note that while the television ex300 has been described here as being configured to be capable of the above encoding processing, multiplexing, and external output, it may also be configured not to be able to perform these processes and only be capable of the above reception, decoding processing, and external output.

[0181] Furthermore, when multiplexed data is read from or written to a recording medium using the reader / recorder ex218, the above-mentioned decoding or encoding process may be performed by either the television ex300 or the reader / recorder ex218, or the television ex300 and the reader / recorder ex218 may share the process.

[0182] As an example, Figure 15 shows the configuration of the information reproducing / recording unit ex400 when reading or writing data from an optical disc. The information reproducing / recording unit ex400 includes the following elements: ex401, ex402, ex403, ex404, ex405, ex406, and ex407. The optical head ex401 writes information by irradiating a laser spot onto the recording surface of the recording medium ex215, which is an optical disc, and reads the information by detecting the light reflected from the recording surface of the recording medium ex215. The modulation / recording unit ex402 electrically drives the semiconductor laser built into the optical head ex401 and modulates the laser light according to the recorded data. The reproduction / demodulation unit ex403 amplifies the reproduction signal obtained by electrically detecting the light reflected from the recording surface using a photodetector built into the optical head ex401, separates and demodulates the signal components recorded on the recording medium ex215, and reproduces the required information. The buffer ex404 temporarily stores information to be recorded on the recording medium ex215 and information reproduced from the recording medium ex215. The disk motor ex405 rotates the recording medium ex215. The servo control unit ex406 controls the rotation of the disk motor ex405, moves the optical head ex401 to a specified information track, and performs laser spot tracking. The system control unit ex407 controls the entire information reproduction / recording unit ex400. The system control unit ex407 performs the above read and write processes by using various information stored in the buffer ex404, generating and adding new information as needed, and recording and reproducing information through the optical head ex401 while coordinating the modulation recording unit ex402, reproduction demodulation unit ex403, and servo control unit ex406. The system control unit ex407 is composed of, for example, a microprocessor and performs these processes by executing read and write programs.

[0183] In the above description, the optical head ex401 is described as irradiating a laser spot, but it may be configured to perform higher density recording using near-field light.

[0184] FIG. 16 shows a schematic diagram of recording medium ex215, an optical disc. A spiral guide groove is formed on the recording surface of recording medium ex215, and address information indicating absolute positions on the disc is recorded in advance on information track ex230 by varying the shape of the groove. This address information includes information for identifying the position of recording block ex231, which is the unit of data recording. A recording or playback device can identify a recording block by reproducing information track ex230 and reading the address information. Recording medium ex215 also includes a data recording area ex233, an inner peripheral area ex232, and an outer peripheral area ex234. The data recording area ex233 is the area used to record user data, while the inner peripheral area ex232 and outer peripheral area ex234, which are located either inner or outer than data recording area ex233, are used for specific purposes other than recording user data. The information reproducing / recording unit ex400 reads and writes encoded audio data, video data, or multiplexed data obtained by multiplexing these data, from the data recording area ex233 of such recording medium ex215.

[0185] The above explanation has been given using examples of optical discs such as single-layer DVDs and BDs, but the present invention is not limited to these and may be an optical disc with a multi-layer structure that allows recording on areas other than the surface. It may also be an optical disc with a structure that allows multidimensional recording / playback, such as recording information using light of various different wavelengths in the same location on the disc or recording different layers of information from various angles.

[0186] In addition, in the digital broadcasting system ex200, a car ex210 equipped with an antenna ex205 can receive data from a satellite ex202 or the like, and the video can be played on a display device such as a car navigation system ex211 installed in the car ex210. The car navigation system ex211 can be configured, for example, as shown in Fig. 14, with a GPS receiver added, and similar configurations can be considered for a computer ex111, a mobile phone ex114, and the like.

[0187] 17A is a diagram showing a mobile phone ex114 that uses the video decoding method and video encoding method described in the above embodiment. The mobile phone ex114 includes an antenna ex350 for transmitting and receiving radio waves to and from base station ex110, a camera unit ex365 capable of capturing video and still images, and a display unit ex358 such as an LCD display that displays decoded data of video captured by the camera unit ex365 and video received by the antenna ex350. The mobile phone ex114 further includes a main body unit having an operation key unit ex366, an audio output unit ex357 such as a speaker for outputting audio, an audio input unit ex356 such as a microphone for inputting audio, a memory unit ex367 for storing captured video, still images, recorded audio, or encoded or decoded data of received video, still images, email, etc., or a slot unit ex364 that serves as an interface with a recording medium for similarly storing data.

[0188] 17B, ​​an example of the configuration of mobile phone ex114 will be described. Mobile phone ex114 has a main control unit ex360 that comprehensively controls each unit of a main body unit including a display unit ex358 and an operation key unit ex366, and a power supply circuit unit ex361, an operation input control unit ex362, a video signal processing unit ex355, a camera interface unit ex363, an LCD (Liquid Crystal Display) control unit ex359, a modulation / demodulation unit ex352, a multiplexing / demultiplexing unit ex353, an audio signal processing unit ex354, a slot unit ex364, and a memory unit ex367, which are all connected to each other via a bus ex370.

[0189] When the end call and power key is turned on by the user, the power supply circuit unit ex361 starts up the mobile phone ex114 into an operable state by supplying power to each unit from the battery pack.

[0190] Based on the control of a main control unit ex360 having a CPU, ROM, RAM, etc., the mobile phone ex114 converts an audio signal collected by an audio input unit ex356 into a digital audio signal by an audio signal processing unit ex354 in a voice call mode, which undergoes spectrum spread processing by a modulation / demodulation unit ex352, digital-to-analog conversion processing and frequency conversion processing by a transmission / reception unit ex351, and then transmits the digital audio signal via an antenna ex350. Furthermore, the mobile phone ex114 amplifies received data received via the antenna ex350 in a voice call mode, performs frequency conversion processing and analog-to-digital conversion processing, performs spectrum despread processing by the modulation / demodulation unit ex352, converts the data into an analog audio signal by the audio signal processing unit ex354, and then outputs the data from an audio output unit ex357.

[0191] Furthermore, when sending an e-mail in data communication mode, the text data of the e-mail entered by operating the operation key unit ex366 or the like of the main unit is sent to the main control unit ex360 via the operation input control unit ex362. The main control unit ex360 performs spectrum spread processing on the text data in the modulation / demodulation unit ex352, performs digital-to-analog conversion processing and frequency conversion processing in the transmission / reception unit ex351, and then transmits the data to the base station ex110 via the antenna ex350. When receiving an e-mail, the received data is subjected to roughly the reverse processing and output to the display unit ex358.

[0192] When transmitting video, still images, or video and audio in the data communication mode, the video signal processing unit ex355 compresses and encodes the video signal supplied from the camera unit ex365 using the video encoding method described in each of the above embodiments (i.e., functions as an image encoding device according to one aspect of the present invention), and sends the encoded video data to the multiplexing / separating unit ex353. In addition, the audio signal processing unit ex354 encodes the audio signal collected by the audio input unit ex356 while the camera unit ex365 is capturing video, still images, etc., and sends the encoded audio data to the multiplexing / separating unit ex353.

[0193] The multiplexing / separation unit ex353 multiplexes the encoded video data supplied from the video signal processing unit ex355 and the encoded audio data supplied from the audio signal processing unit ex354 using a predetermined method, and the resulting multiplexed data is subjected to spectrum spreading processing in the modulation / demodulation unit (modulation / demodulation circuit unit) ex352, digital-to-analog conversion processing and frequency conversion processing in the transmission / reception unit ex351, and then transmitted via the antenna ex350.

[0194] When receiving video file data linked to a website or the like in data communication mode, or when receiving an email with video and / or audio attachments, the multiplexer / demultiplexer ex353 decodes the multiplexed data received via the antenna ex350 into a video data bitstream and an audio data bitstream. The multiplexer / demultiplexer ex353 then decodes the multiplexed data into a video data bitstream and an audio data bitstream via a synchronization bus ex370. The video signal processor ex355 decodes the video signal using a video decoding method corresponding to the video encoding method described in each of the above embodiments (i.e., functions as an image decoding device according to one aspect of the present invention). The display unit ex358 displays, via an LCD controller ex359, video and still images included in the video file linked to a website, for example. The audio signal processor ex354 decodes the audio signal, and audio is output from an audio output unit ex357.

[0195] Furthermore, like the television ex300, terminals such as the mobile phone ex114 can be implemented in three ways: a transmitting / receiving terminal with both an encoder and a decoder, a transmitting terminal with only an encoder, and a receiving terminal with only a decoder. Furthermore, in the digital broadcasting system ex200, it has been explained that multiplexed data in which music data and the like are multiplexed onto video data is received and transmitted, but the data may also be multiplexed with text data related to the video in addition to audio data, or it may be video data itself rather than multiplexed data.

[0196] In this way, it is possible to use the video encoding method or video decoding method shown in each of the above embodiments in any of the above-mentioned devices and systems, and by doing so, it is possible to obtain the effects described in each of the above embodiments.

[0197] Furthermore, the present invention is not limited to the above-described embodiment, and various modifications and alterations are possible without departing from the scope of the present invention.

[0198] (Embodiment B) It is also possible to generate video data by switching between the video encoding method or device shown in each of the above embodiments and a video encoding method or device conforming to a different standard, such as MPEG-2, MPEG4-AVC, or VC-1, as needed.

[0199] When multiple pieces of video data conforming to different standards are generated, it is necessary to select a decoding method corresponding to each standard when decoding. However, since it is not possible to identify which standard the video data to be decoded conforms to, a problem arises in that it is not possible to select an appropriate decoding method.

[0200] To solve this problem, multiplexed data, which is video data multiplexed with audio data, etc., is configured to include identification information that indicates which standard the video data conforms to. A specific configuration of multiplexed data including video data generated by the video encoding methods or devices described in the above embodiments is described below. The multiplexed data is a digital stream in MPEG-2 transport stream format.

[0201] FIG. 18 shows the structure of multiplexed data. As shown in FIG. 18, the multiplexed data is obtained by multiplexing one or more of a video stream, an audio stream, a presentation graphics stream (PG), and an interactive graphics stream. The video stream represents the main video and secondary video of a movie, the audio stream (IG) represents the main audio portion of the movie and the secondary audio mixed with the main audio, and the presentation graphics stream represents the subtitles of the movie. Here, the main video refers to the normal video displayed on the screen, and the secondary video refers to the video displayed on a small screen within the main video. The interactive graphics stream represents an interactive screen created by arranging GUI components on the screen. The video stream is encoded using the video encoding method or device described in each of the above embodiments or a video encoding method or device conforming to conventional standards such as MPEG-2, MPEG4-AVC, or VC-1. The audio stream is encoded using a format such as Dolby AC-3, Dolby Digital Plus, MLP, DTS, DTS-HD, or Linear PCM.

[0202] Each stream included in the multiplexed data is identified by a PID. For example, 0x1011 is assigned to the video stream used for movie images, 0x1100 to 0x111F to the audio stream, 0x1200 to 0x121F to the presentation graphics, 0x1400 to 0x141F to the interactive graphics stream, 0x1B00 to 0x1B1F to the video stream used for movie secondary video, and 0x1A00 to 0x1A1F to the audio stream used for secondary audio to be mixed with the main audio.

[0203] 19 is a diagram showing how multiplexed data is multiplexed. First, a video stream ex235 consisting of multiple video frames and an audio stream ex238 consisting of multiple audio frames are converted into PES packet sequences ex236 and ex239, respectively, and then converted into TS packets ex237 and ex240. Similarly, presentation graphics stream ex241 and interactive graphics data ex244 are converted into PES packet sequences ex242 and ex245, respectively, and then converted into TS packets ex243 and ex246. Multiplexed data ex247 is constructed by multiplexing these TS packets into a single stream.

[0204] FIG. 20 shows in more detail how a video stream is stored in a PES packet sequence. The first row in FIG. 20 shows a video frame sequence of the video stream. The second row shows a PES packet sequence. As indicated by arrows yy1, yy2, yy3, and yy4 in FIG. 20, I-pictures, B-pictures, and P-pictures, which are multiple Video Presentation Units in the video stream, are divided into individual pictures and stored in the payload of a PES packet. Each PES packet has a PES header, which stores a Presentation Time-Stamp (PTS), which is the display time of the picture, and a Decoding Time-Stamp (DTS), which is the decoding time of the picture.

[0205] Figure 21 shows the format of the TS packet that is ultimately written to the multiplexed data. TS packets are 188-byte fixed-length packets consisting of a 4-byte TS header containing information such as a PID that identifies the stream, and a 184-byte TS payload that stores the data. The PES packets are divided and stored in the TS payload. In the case of BD-ROM, a 4-byte TP_Extra_Header is added to the TS packet, forming a 192-byte source packet that is written to the multiplexed data. The TP_Extra_Header contains information such as an ATS (Arrival Time Stamp). The ATS indicates the start time of the TS packet's transfer to the PID filter of the decoder. As shown in the lower part of Figure 21, source packets are arranged in the multiplexed data, and the number that increments from the beginning of the multiplexed data is called the SPN (Source Packet Number).

[0206] In addition to the individual streams (video, audio, subtitles, etc.), the TS packets contained in the multiplexed data also contain a Program Association Table (PAT), Program Map Table (PMT), and Program Clock Reference (PCR). The PAT indicates the PID of the PMT used in the multiplexed data, and the PAT's own PID is registered as 0. The PMT contains the PIDs of each stream (video, audio, subtitles, etc.) contained in the multiplexed data, as well as attribute information for the streams corresponding to each PID. It also contains various descriptors related to the multiplexed data. The descriptors include copy control information that indicates whether copying of the multiplexed data is permitted or prohibited. The PCR contains information about the Arrival Time Clock (ATC), which is the time axis of the ATS, and the System Time Clock (STC), which is the time axis of the PTS and DTS, and contains information about the STC time corresponding to the ATS at which the PCR packet is transferred to the decoder.

[0207] Figure 22 is a diagram explaining the data structure of a PMT in detail. At the beginning of a PMT is a PMT header that describes the length of the data contained in the PMT, among other things. This is followed by multiple descriptors related to the multiplexed data. The above-mentioned copy control information and other information are written as descriptors. After the descriptors are multiple stream information items related to each stream included in the multiplexed data. The stream information consists of stream descriptors that describe the stream type to identify the stream compression codec, the stream PID, and stream attribute information (frame rate, aspect ratio, etc.). There are as many stream descriptors as there are streams in the multiplexed data.

[0208] When recording on a recording medium, the multiplexed data is recorded together with a multiplexed data information file.

[0209] As shown in FIG. 23, the multiplexed data information file is management information for multiplexed data, has one-to-one correspondence with the multiplexed data, and is made up of multiplexed data information, stream attribute information, and an entry map.

[0210] As shown in Fig. 23, the multiplexed data information consists of a system rate, a playback start time, and a playback end time. The system rate indicates the maximum transfer rate of the multiplexed data to the PID filter of the system target decoder, which will be described later. The interval between ATSs contained in the multiplexed data is set to be equal to or less than the system rate. The playback start time is set to the PTS of the first video frame of the multiplexed data, and the playback end time is set to the PTS of the last video frame of the multiplexed data plus the playback interval of one frame.

[0211] As shown in Figure 24, the stream attribute information for each stream included in the multiplexed data is registered for each PID. The attribute information has different information for each video stream, audio stream, presentation graphics stream, and interactive graphics stream. The video stream attribute information includes information such as the compression codec used to compress the video stream, the resolution of the individual picture data that make up the video stream, the aspect ratio, and the frame rate. The audio stream attribute information includes information such as the compression codec used to compress the audio stream, the number of channels included in the audio stream, the language it supports, and the sampling frequency. This information is used to initialize the decoder before playback on the player.

[0212] In this embodiment, the stream type included in the PMT of the multiplexed data is used. Furthermore, if multiplexed data is recorded on a recording medium, the video stream attribute information included in the multiplexed data information is used. Specifically, the video coding method or device shown in each of the above embodiments includes a step or means for setting, in the stream type included in the PMT or the video stream attribute information, unique information indicating that the video data is generated by the video coding method or device shown in each of the above embodiments. This configuration makes it possible to distinguish between video data generated by the video coding method or device shown in each of the above embodiments and video data that conforms to other standards.

[0213] FIG. 25 shows the steps of the video decoding method according to this embodiment. In step exS100, the stream type included in the PMT or the video stream attribute information included in the multiplexed data information is obtained from the multiplexed data. Next, in step exS101, it is determined whether the stream type or the video stream attribute information indicates that the multiplexed data was generated by the video coding method or device described in the above embodiments. If it is determined that the stream type or the video stream attribute information was generated by the video coding method or device described in the above embodiments, in step exS102, decoding is performed using the video decoding method described in the above embodiments. If the stream type or the video stream attribute information indicates that the data complies with a conventional standard such as MPEG-2, MPEG4-AVC, or VC-1, decoding is performed using the video decoding method according to the conventional standard in step exS103.

[0214] In this way, by setting a new unique value in the stream type or video stream attribute information, it is possible to determine whether the video decoding method or device shown in each of the above embodiments can decode the data when decoding. Therefore, even when multiplexed data conforming to a different standard is input, an appropriate decoding method or device can be selected, enabling decoding without errors. Furthermore, the video encoding method or device or video decoding method or device shown in this embodiment can be used in any of the above-mentioned devices and systems.

[0215] (Embodiment C) The video encoding method and device, and video decoding method and device described in each of the above embodiments are typically realized by an LSI, which is an integrated circuit. As an example, FIG. 26 shows the configuration of a single-chip LSI ex500. LSI ex500 includes elements ex501, ex502, ex503, ex504, ex505, ex506, ex507, ex508, and ex509, which are described below, and each element is connected via a bus ex510. When the power supply is on, a power supply circuit unit ex505 supplies power to each unit, thereby activating them into an operable state.

[0216] For example, when performing encoding processing, the LSI ex500 inputs AV signals from the microphone ex117, camera ex113, etc. via the AV I / O ex509 under the control of a control unit ex501 including a CPU ex502, a memory controller ex503, a stream controller ex504, a drive frequency control unit ex512, etc. The input AV signals are temporarily stored in an external memory ex511 such as an SDRAM. Under the control of the control unit ex501, the stored data is divided into multiple batches as appropriate depending on the processing volume and processing speed and sent to the signal processing unit ex507, where the audio signal and / or video signal is encoded. Here, the video signal encoding processing is the encoding processing described in each of the above embodiments. The signal processing unit ex507 may further perform processing such as multiplexing the encoded audio data and the encoded video data, and output the resulting data to the outside from the stream I / O ex506. This output multiplexed data is transmitted to the base station ex107 or written to a recording medium ex215. When multiplexing, it is advisable to temporarily store the data in a buffer ex508 to ensure synchronization.

[0217] Although the memory ex511 has been described above as being external to the LSIex500, it may be included within the LSIex500. The buffer ex508 is not limited to one, and multiple buffers may be provided. Furthermore, the LSIex500 may be formed as a single chip or multiple chips.

[0218] Furthermore, in the above description, the control unit ex501 is described as having a CPU ex502, a memory controller ex503, a stream controller ex504, a drive frequency control unit ex512, etc., but the configuration of the control unit ex501 is not limited to this configuration. For example, the signal processing unit ex507 may further include a CPU. By providing a CPU inside the signal processing unit ex507, it is possible to further improve processing speed. As another example, the CPU ex502 may include the signal processing unit ex507, or a part of the signal processing unit ex507, such as an audio signal processing unit. In such a case, the control unit ex501 is configured to include a CPU ex502 that includes the signal processing unit ex507, or a part of it.

[0219] Although we have referred to it as an LSI here, it may also be called an IC, system LSI, super LSI, or ultra LSI depending on the level of integration.

[0220] Furthermore, the method of integration is not limited to LSI, but may be realized by dedicated circuits or general-purpose processors. It is also possible to use FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI manufacturing, or reconfigurable processors, which allow the connections and settings of circuit cells within LSIs to be reconfigured.

[0221] Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, it is natural that such technology could be used to integrate functional blocks. The application of biotechnology is also a possibility.

[0222] (Embodiment D) When decoding video data generated by the video encoding method or device described in each of the above embodiments, the amount of processing is likely to increase compared to when decoding video data conforming to conventional standards such as MPEG-2, MPEG4-AVC, or VC-1. Therefore, it is necessary to set the drive frequency of the LSIex500 to a higher frequency than the drive frequency of the CPUex502 when decoding video data conforming to conventional standards. However, increasing the drive frequency raises the problem of increased power consumption.

[0223] To solve this problem, video decoding devices such as televisions ex300 and LSIs ex500 are configured to identify the standard to which video data conforms and switch the drive frequency according to the standard. FIG. 27 shows a configuration ex800 in this embodiment. If the video data was generated using the video encoding method or device described in each of the above embodiments, a drive frequency switching unit ex803 sets a high drive frequency. The unit then instructs a decoding processing unit ex801, which executes the video decoding method described in each of the above embodiments, to decode the video data. On the other hand, if the video data conforms to a conventional standard, the unit sets a low drive frequency compared to when the video data was generated using the video encoding method or device described in each of the above embodiments. The unit then instructs a decoding processing unit ex802, which conforms to the conventional standard, to decode the video data.

[0224] More specifically, the drive frequency switching unit ex803 is composed of the CPU ex502 and drive frequency control unit ex512 in FIG. 26. The decoding processing unit ex801 that executes the video decoding method described in each of the above embodiments and the decoding processing unit ex802 that complies with the conventional standard correspond to the signal processing unit ex507 in FIG. 26. The CPU ex502 identifies the standard to which the video data conforms. The drive frequency control unit ex512 sets the drive frequency based on the signal from the CPU ex502. The signal processing unit ex507 decodes the video data based on the signal from the CPU ex502. Here, the video data can be identified using, for example, the identification information described in embodiment B. The identification information is not limited to that described in embodiment B, and can be any information that can identify the standard to which the video data conforms. For example, if it is possible to identify the standard to which the video data conforms based on an external signal that identifies whether the video data is for use on a television or a disc, then the identification can be based on such an external signal. Furthermore, the selection of the drive frequency in the CPUex502 can be performed based on a lookup table that associates the video data standard with the drive frequency, as shown in Fig. 29. The lookup table is stored in the buffer ex508 or the internal memory of the LSI, and the CPUex502 can select the drive frequency by referring to this lookup table.

[0225] FIG. 28 shows steps for implementing the method of this embodiment. First, in step exS200, the signal processing unit ex507 acquires identification information from the multiplexed data. Next, in step exS201, the CPU ex502 identifies, based on the identification information, whether the video data was generated by the encoding method or device described in any of the above embodiments. If the video data was generated by the encoding method or device described in any of the above embodiments, in step exS202, the CPU ex502 sends a signal to the driving frequency control unit ex512 to set the driving frequency to a high level. The driving frequency control unit ex512 then sets the driving frequency to a high level. On the other hand, if the video data indicates that it complies with a conventional standard such as MPEG-2, MPEG4-AVC, or VC-1, in step exS203, the CPU ex502 sends a signal to the driving frequency control unit ex512 to set the driving frequency to a low level. The driving frequency control unit ex512 then sets the driving frequency to a lower level than when the video data was generated by the encoding method or device described in any of the above embodiments.

[0226] Furthermore, by changing the voltage applied to the LSIex500 or a device including the LSIex500 in conjunction with switching the drive frequency, it is possible to further enhance the power saving effect. For example, when the drive frequency is set low, it is conceivable to set the voltage applied to the LSIex500 or a device including the LSIex500 lower in response to this change than when the drive frequency is set high.

[0227] Furthermore, the method of setting the drive frequency is not limited to the above-described setting method, and may be such that a high drive frequency is set when the decoding processing volume is large, and a low drive frequency is set when the decoding processing volume is small. For example, if the processing volume required to decode video data conforming to the MPEG4-AVC standard is larger than the processing volume required to decode video data generated by the video encoding method or device described in each of the above-described embodiments, the drive frequency may be set in the opposite way to the above-described setting method.

[0228] Furthermore, the method of setting the drive frequency is not limited to a configuration that lowers the drive frequency. For example, if the identification information indicates that the video data is generated by the video encoding method or device described in each of the above embodiments, the voltage applied to the LSIex500 or a device including the LSIex500 can be set high. If the identification information indicates that the video data complies with conventional standards such as MPEG-2, MPEG4-AVC, or VC-1, the voltage applied to the LSIex500 or a device including the LSIex500 can be set low. As another example, if the identification information indicates that the video data is generated by the video encoding method or device described in each of the above embodiments, the drive of the CPUex502 can be suspended without stopping. If the identification information indicates that the video data complies with conventional standards such as MPEG-2, MPEG4-AVC, or VC-1, the drive of the CPUex502 can be suspended temporarily because there is sufficient processing capacity. Even if the identification information indicates that the video data is generated by the video encoding method or device described in each of the above embodiments, the drive of the CPUex502 can be suspended temporarily if there is sufficient processing capacity. In this case, it is conceivable to set the stop time shorter than when the video data indicates that it is video data that complies with conventional standards such as MPEG-2, MPEG4-AVC, and VC-1.

[0229] In this way, by switching the drive frequency depending on the standard to which the video data conforms, it is possible to achieve power savings. Furthermore, if the LSIex500 or a device including the LSIex500 is driven by a battery, the power savings can also extend the battery life.

[0230] (Embodiment E) The above-mentioned devices and systems, such as televisions and mobile phones, may receive multiple inputs of video data conforming to different standards. To ensure that the signal processing unit ex507 of the LSIex500 can decode such inputs, the signal processing unit ex507 must support multiple standards. However, using separate signal processing units ex507 for each standard increases the circuit size of the LSIex500 and increases costs.

[0231] To solve this problem, a configuration is provided in which a decoding processing unit for executing the video decoding method described in each of the above embodiments is partially shared with a decoding processing unit conforming to conventional standards such as MPEG-2, MPEG4-AVC, and VC-1. An example of this configuration is shown in ex900 of FIG. 30A . For example, the video decoding method described in each of the above embodiments and a video decoding method conforming to the MPEG4-AVC standard share some of the processing content, such as entropy coding, inverse quantization, deblocking filtering, and motion compensation. For the common processing content, a decoding processing unit ex902 conforming to the MPEG4-AVC standard is shared, and a dedicated decoding processing unit ex901 is used for other processing content unique to one aspect of the present invention that does not conform to the MPEG4-AVC standard. In particular, since one aspect of the present invention is characterized by inverse quantization, for example, a dedicated decoding processing unit ex901 is used for inverse quantization, and a decoding processing unit is shared for any or all of the other processing, such as entropy decoding, deblocking filtering, and motion compensation. Regarding the sharing of the decoding processing unit, for common processing content, the decoding processing unit for executing the video decoding method shown in each of the above embodiments may be shared, and for processing content specific to the MPEG4-AVC standard, a dedicated decoding processing unit may be used.

[0232] Another example of partially sharing processing is shown in ex1000 in FIG. 30B. In this example, a dedicated decoding processing unit ex1001 corresponding to processing content specific to one aspect of the present invention, a dedicated decoding processing unit ex1002 corresponding to processing content specific to another conventional standard, and a shared decoding processing unit ex1003 corresponding to processing content common to the video decoding method according to one aspect of the present invention and the video decoding method of another conventional standard are used. Here, the dedicated decoding processing units ex1001 and ex1002 are not necessarily specialized for processing content specific to one aspect of the present invention or another conventional standard, and may be capable of performing other general-purpose processing. The configuration of this embodiment can also be implemented using an LSI ex500.

[0233] In this way, by sharing a decoding processing unit for processing content that is common between a video decoding method according to one embodiment of the present invention and a video decoding method of a conventional standard, it is possible to reduce the circuit size of the LSI and reduce costs.

[0234] In summary, the present invention relates to deblocking filtering applicable to smoothing block boundaries in image or video encoding and decoding. In particular, deblocking filtering is applied to samples at block boundaries according to predetermined parameters. The predetermined parameters are used to cancel samples as part of the filtering. According to the present invention, the parameters are determined by calculating a weighted sum of samples around the boundary, canceling the weighted sum by a value depending on the value of the weighted sum, and shifting the result to the right by a predetermined number of bits.

[0235] The present invention relates to deblocking filtering applicable to smoothing block boundaries in image or video encoding and decoding. Specifically, deblocking filtering can be weak or strong, and clipping performed in strong filtering and weak filtering is different.

Claims

1. a processing circuit; a memory connected to the processing circuitry; The processing circuitry uses the memory to: selecting a filter for the boundary between the first block and the second block from a plurality of candidates; performing a deblocking filter process using the selected filter to change the values ​​of a plurality of pixels aligned on a line that horizontally intersects the boundary; the plurality of pixels include a first pixel and a second pixel adjacent to each other across the boundary, the plurality of candidates includes a first filter that modifies values ​​of a first number of the plurality of pixels, and a second filter that modifies values ​​of a second number of the plurality of pixels that is greater than the first number; In the deblocking filtering process when the first filter is selected, a clipping process is performed to set pixel change amounts of the first pixel and the second pixel to be equal to or greater than a negative first clipping value and equal to or less than a positive first clipping value; In the deblocking filtering process when the second filter is selected, a clipping process is performed to set pixel change amounts of the first pixel and the second pixel to be equal to or greater than a negative second clipping value and equal to or less than a positive second clipping value; the first clipping value is different from the second clipping value; the first block and the second block are adjacent to each other in the vertical direction, and the boundary is a horizontal boundary. Decryption device.

2. a processing circuit; a memory connected to the processing circuitry; The processing circuitry uses the memory to: selecting a filter for the boundary between the first block and the second block from a plurality of candidates; performing a deblocking filter process using the selected filter to change the values ​​of a plurality of pixels aligned on a line that horizontally intersects the boundary; the plurality of pixels include a first pixel and a second pixel adjacent to each other across the boundary, the plurality of candidates includes a first filter that modifies values ​​of a first number of the plurality of pixels, and a second filter that modifies values ​​of a second number of the plurality of pixels that is greater than the first number; In the deblocking filtering process when the first filter is selected, a clipping process is performed to set pixel change amounts of the first pixel and the second pixel to be equal to or greater than a negative first clipping value and equal to or less than a positive first clipping value; In the deblocking filtering process when the second filter is selected, a clipping process is performed to set pixel change amounts of the first pixel and the second pixel to be equal to or greater than a negative second clipping value and equal to or less than a positive second clipping value; the first clipping value is different from the second clipping value; the first block and the second block are adjacent to each other in the vertical direction, and the boundary is a horizontal boundary. Encoding device.

3. selecting a filter for the boundary between the first block and the second block from a plurality of candidates; performing a deblocking filter process using the selected filter to change the values ​​of a plurality of pixels aligned on a line that horizontally intersects the boundary; the plurality of pixels include a first pixel and a second pixel adjacent to each other across the boundary, the plurality of candidates includes a first filter that modifies values ​​of a first number of the plurality of pixels, and a second filter that modifies values ​​of a second number of the plurality of pixels that is greater than the first number; In the deblocking filtering process when the first filter is selected, a clipping process is performed to set pixel change amounts of the first pixel and the second pixel to be equal to or greater than a negative first clipping value and equal to or less than a positive first clipping value; In the deblocking filtering process when the second filter is selected, a clipping process is performed to set pixel change amounts of the first pixel and the second pixel to be equal to or greater than a negative second clipping value and equal to or less than a positive second clipping value; the first clipping value is different from the second clipping value; the first block and the second block are adjacent to each other in the vertical direction, and the boundary is a horizontal boundary. Decryption method.

4. selecting a filter for the boundary between the first block and the second block from a plurality of candidates; performing a deblocking filter process using the selected filter to change the values ​​of a plurality of pixels aligned on a line that horizontally intersects the boundary; the plurality of pixels include a first pixel and a second pixel adjacent to each other across the boundary, the plurality of candidates includes a first filter that modifies values ​​of a first number of the plurality of pixels, and a second filter that modifies values ​​of a second number of the plurality of pixels that is greater than the first number; In the deblocking filtering process when the first filter is selected, a clipping process is performed to set pixel change amounts of the first pixel and the second pixel to be equal to or greater than a negative first clipping value and equal to or less than a positive first clipping value; In the deblocking filtering process when the second filter is selected, a clipping process is performed to set pixel change amounts of the first pixel and the second pixel to be equal to or greater than a negative second clipping value and equal to or less than a positive second clipping value; the first clipping value is different from the second clipping value; the first block and the second block are adjacent to each other in the vertical direction, and the boundary is a horizontal boundary. Encoding method.

Images