Method and apparatus for allocating bits to portions of a frame

Abstract

A multimedia system [100] allocates bits to portions of a frame [102] based on quality metrics and bit usage for different quantization parameters (QPs) during encoding of a multimedia stream. An encoder [110] of the multimedia system encodes a frame with a first QP in a first pass and with a second QP in a second pass. A comparator [420] of the multimedia system measures a quality metric (such as a mean square error) for each portion of the frame encoded with the first QP and the second QP and compares them. The comparator compares the difference between the quality metrics for each portion encoded with each QP with a threshold. If the difference in the quality metrics of a portion exceeds the threshold, the comparator selects the portion to be included in a subset of portions to be encoded with the second QP.

Description

Background Art

[0001] The multimedia server generates data representing pictures in a multimedia stream (e.g., a multimedia stream that a user has requested). The encoder of the multimedia server encodes the data for each picture to form a bit stream that is transmitted over a network to a decoder, which decodes the bit stream and provides the decoded video information to a multimedia application or any other application for display to a user. Such multimedia encoders and decoders are used in a variety of applications to facilitate the storage and delivery of multimedia streams in a compressed manner.

[0002] In order to compress multimedia streams, conventional encoders implement video compression algorithms in which the degree of compression depends in part on a quantization parameter. A higher quantization parameter generally results in a lower bit usage of the picture, while a lower quantization parameter generally results in a higher bit usage of the picture. The compression algorithm uses different quantization parameters that allocate bits to the title, frame, slice, and block of the picture. However, due to the quantization parameter being too low or too high for the application, bits are often inefficiently allocated within the picture by the compression algorithm. In the case of being too low, the quantization parameter results in unnecessary consumption of computing resources and bandwidth when encoding, transmitting, and decoding the picture, without any corresponding benefit. If the quantization parameter is too high, the result is an unnecessary reduction in the quality of the encoded picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] By referring to the accompanying drawings, those skilled in the art may better understand the present disclosure and appreciate its numerous features and advantages. The use of the same reference numerals in different drawings indicates similar or identical items.

[0004] Figure 1 is a block diagram of a multimedia system configured to allocate encoding bits to macroblocks of a picture based on a quality metric and a bit usage rate, according to some embodiments.

[0005] Figure 2 is a diagram showing quality metrics for macroblocks of a frame encoded with a first quantization parameter.

[0006] Figure 3 is a diagram showing quality metrics for macroblocks of a frame encoded with a second quantization parameter.

[0007] Figure 4 According to some embodiments Figure 1 A block diagram of a comparator of a multimedia system of the present invention, the comparator being configured to select a subset of macroblocks of a frame for encoding with a second quantization parameter based on a quality metric and a bit usage.

[0008] Figure 5 is a diagram showing a subset of macroblocks of a frame encoded with a second quantization parameter and the remainder of the macroblocks of the frame encoded with a first quantization parameter according to some embodiments.

[0009] Figure 6 According to some embodiments Figure 1 A block diagram of a comparator of a multimedia system of the present invention, the comparator being configured to select a subset of macroblocks of a frame for encoding with a second quantization parameter based on pixel activities of the macroblocks.

[0010] Figure 7 is a flow chart illustrating a method for selecting a subset of macroblocks of a frame to be encoded with a second quantization parameter based on a quality metric and bit usage, according to some embodiments.

[0011] Figure 8 is a flow chart illustrating a method for selecting a subset of macroblocks of a frame to be encoded with a second quantization parameter based on pixel activity of the macroblocks, according to some embodiments. DETAILED DESCRIPTION

[0012] Figures 1 to 8 Systems and techniques are shown for allocating bits to portions of a frame based on quality metrics and bit usage for different quantization parameters (QPs) during encoding of a multimedia stream. The portions of the frame to which bits are allocated include any portion or all of a frame, such as a macroblock, a coding tree block (CTB), a header, and a slice. For ease of reference, portions of a frame are referred to herein as portions or macroblocks. An encoder of a multimedia system encodes a frame with a first QP in a first pass and with a second QP in a second pass. For purposes of the following description, the second QP is considered to produce more bits and better quality. A comparator of the multimedia system measures a quality metric (such as a mean square error) for each portion of the frame encoded with the first QP and the second QP and compares them. The comparator compares the difference (improvement) between the quality metrics of each portion encoded with each QP to a threshold. If the difference in the quality metrics of the portions exceeds the threshold, the comparator selects the portion to be included in a subset of portions to be encoded with the second QP.

[0013] In some implementations, a comparator measures a bit usage rate for each portion encoded with the first QP and the second QP to determine how many additional bits to use to encode the portion with the second QP compared to the first QP. The comparator selects the portion to include in the subset of portions to be encoded with the second QP based on at least one of a difference in the quality metric and a difference in the bit usage rate.

[0014] In some embodiments, the multimedia system allocates bits to portions of the image based on the difference in pixel activity between the portions. Many video processing algorithms (such as those used for texture analysis) calculate metrics (such as pixel activity within portions of a frame). For example, pixel activity (such as changes in pixel intensity within a portion) can be used to detect edges, repeating patterns, and other structures or objects in an image. Pixel activity can be measured using various metrics (such as, for example, using a gray level co-occurrence matrix, a two-dimensional spatial average gradient, a wavelet or other transform, a discrete cosine function, or an average within a portion). Based on the pixel activity within the portion, a comparator predicts whether the portion will benefit from being encoded with a second QP. In some embodiments, the comparator selects a portion to be included in a subset of the portion to be encoded with a second QP based at least in part on the pixel activity, as explained in more detail below.

[0015] The comparator provides an encoder with an identification of the portions included in the subset of portions to be encoded with the second QP. Based on the identification, the encoder encodes each portion of the frame with the first QP (if the portion was not selected for inclusion in the subset) or the second QP (if the portion was selected for inclusion in the subset). The encoder transmits the encoded frame over a network to a decoder, which decodes the bitstream and provides the decoded video information to a multimedia application for display to a user.

[0016] Figure 1 A block diagram of a multimedia system 100 including a comparator 120 according to some embodiments is shown, wherein the comparator 120 is configured to allocate coded bits to blocks of pictures based on one or more of a quality metric and a bit usage rate. The multimedia system 100 includes an encoder 110, a comparator 120, a decoder 150, and a display 160. In some embodiments, the multimedia system 100 is distributed across a variety of electronic devices, such as servers, personal computers, tablet computers, set-top boxes, game systems, mobile phones, etc. The comparator 120 is implemented as, for example, a processor executing software, programmable logic, hard-coded logic, or a combination thereof. The comparator 120 is implemented or otherwise associated with a source device (not shown) communicating with a target device (not shown) to provide an image to be displayed at a display device 160.

[0017] Encoder 110 is configured to receive digital information, and the digital information represents a stream or sequence of image frames (such as frame 102) with a multimedia stream. The term "multimedia" refers to a combination of video only or video and audio. Encoder 110 encodes digital information to transmit through a communication channel (such as a wide area network (WAN), an intranet, the Internet, a wireless network, etc.). For example, in some embodiments, encoder 110 is used to encode digital information according to coding standards (such as Moving Picture Experts Group ("MPEG")-2, MPEG-4, Advanced Video Coding ("AVC"), etc.). In some embodiments, encoder 110 is a hardware encoder or software executed by one or more processors.

[0018] The encoder 110 is configured to apply one or more QPs to encode a frame or a portion of a frame. The QP is an index used to derive a scaling matrix and adjust how much spatial detail of the image is preserved. For example, in an H.264 encoder, the QP ranges from 0 to 51. The lower the QP, the more details are retained during the encoding process. As the QP increases, some details aggregate, causing the encoding bit rate to decrease, resulting in quality loss.

[0019] In the depicted example, the encoder 110 encodes the frame 102 in multiple passes, encoding the frame 102 using the first QP 112 during a first pass to generate a frame 122 encoded with the first QP 112, and using a second QP 114 lower than the first QP 112 during a second pass to generate a frame 124 encoded with the second QP 114. Therefore, the frame 124 encoded with the second QP 114 uses more bits than the frame 122 encoded with the first QP 112, and preserves more spatial details of the image.

[0020] The comparator 120 includes a quality metric calculator 125, a threshold 130, a bit usage rate calculator 135, and a macroblock subset identifier 140. The comparator 120 is configured to receive a frame 122 encoded with a first QP 112 and a corresponding frame 124 encoded with a second QP 114. The quality metric calculator 125 is configured to measure a quality metric for each macroblock of the frame 122 encoded with the first QP 112 and the frame 124 encoded with the second QP 114. In some embodiments, the method used by the quality metric calculator 125 to measure the quality metric is based on a mean square error (MSE) for each macroblock of the frame 122 encoded with the first QP 112 and the frame 124 encoded with the second QP 114. The MSE is a measure of the average of the squares of the differences between the encoded frames 122, 124 and the image frame 102. A higher MSE indicates that the encoded frame contains more errors relative to the image frame 102. In some embodiments, quality metric calculator 125 uses the following method: Measure quality metric based on how a viewer would perceive video quality.Quality metric calculator 125 applies the same method of measuring quality metric to each of frames 122 and 124 (eg, based on MSE or some other quality metric).

[0021] The quality metric calculator 125 is configured to divide the encoded frames 122, 124 into portions (e.g., blocks of pixels, such as macroblocks, slices, or headers) and measure, for example, a quality metric for each macroblock of a frame 122 encoded with a first QP 112 and each macroblock of a frame 124 encoded with a second QP 114. Thus, the quality metric calculator 125 measures the difference in quality of each portion of the frame 102 when encoded with a first QP 112 versus when encoded with a second (lower) QP 114 with a programmable granularity (i.e., per frame, per slice, per header, or per macroblock).

[0022] The macroblock subset selector 140 is configured to compare the measured quality metric of each macroblock of the frame 122 with each corresponding macroblock of the frame 124 to determine the quality difference of each macroblock when encoded with the first QP 112 and when encoded with the second QP 114. The macroblock subset selector 140 compares the quality difference of each macroblock for each QP to a threshold 130. In some embodiments, the threshold 130 is preset. In some embodiments, the threshold 130 is dynamically adjusted. The dynamic adjustment is based on, for example, a local region spatiotemporal measurement or model. An example is a metric or model that predicts the importance of a region to human perception or the human visual system, etc. In some embodiments, current or predicted bit usage is a factor in the adjustment. If the macroblock subset selector 140 determines that the quality difference of a given macroblock when encoded with the second QP 114 and when encoded with the first QP 112 exceeds the threshold 130, the macroblock subset selector 140 selects the macroblock to be included in the subset of macroblocks to be encoded with the second QP 114. The subset includes some (but not all) of the macroblocks of frame 122 .

[0023] The comparator 120 includes a bit usage rate calculator 135 configured to determine a difference in bit usage per macroblock of the frame 122 encoded with the first QP 112 and the frame 124 encoded with the second QP 114. Assigning a lower QP (i.e., more bits) to some macroblocks may result in a greater improvement in MSE than other macroblocks. The bit usage rate calculator 135 identifies the macroblocks that show the greatest improvement in MSE compared to the increase in bit usage when encoded with the second QP 114 compared to the first QP 112. The macroblock subset selector 140 bases the selection of macroblocks to be included in the subset of macroblocks to be encoded with the second QP 114 in part on the identification by the bit usage rate calculator 135 of the macroblocks that show the greatest improvement in MSE (or other quality metric) compared to the increase in bit usage when encoded with the second QP 114 compared to the first QP 112. Macroblock subset selector 140 compares the difference in bit usage of each macroblock of frame 122 and the corresponding macroblock of frame 124 to a threshold (not shown) to determine which macroblocks to include in the subset of macroblocks to be encoded with second QP 114 .

[0024] The comparator 120 provides the encoder 110 with an identification of a subset 142 of macroblocks to be encoded with the second QP 114. The encoder 110 encodes the subset 142 of macroblocks with the second QP 114 and encodes the remainder of the macroblocks of the frame with the first QP 112 to generate an encoded frame 140. The encoded frame 140 includes the subset 144 of macroblocks encoded with the second QP. The encoder 110 transmits the encoded frame 140 to the decoder 150 over a network (not shown). In some embodiments, the decoder 150 is implemented as a hardware decoder or a software decoder, for example, as a set of executable instructions. The decoder 150 is configured to decode the encoded frame 140 and provide the decoded video information to a multimedia application or any other application for display to a user at a display screen 160. It should be noted that any number of QPs may be used; that is, the number of QPs available for selection may be greater than two. Even though there are only two choices, the two QPs that will be considered vary within the picture based on metrics, models, and / or current / predicted bit usage.

[0025] To illustrate, in operation, the encoder 110 receives a frame 102 from a multimedia application (not shown). The encoder 110 encodes the frame 102 with a first QP 112 in a first pass and encodes the frame 102 with a second QP 114 in a second pass to generate an encoded frame 122 encoded with the first QP 112 and an encoded frame 124 encoded with the second QP 114. The encoder 110 provides the encoded frame 122 and the encoded frame 124 to the comparator 120.

[0026] The quality metric calculator 125 of the comparator 120 measures the quality metric of each macroblock (or slice or other portion) of the coded frame 122 and the coded frame 124. The quality metric calculator 125 compares the quality metric of each macroblock of the coded frame 122 and the coded frame 124 to determine the difference in the quality metric of each macroblock when encoded with the first QP 112 and encoded with the second QP 114. In some embodiments, the bit usage rate calculator 135 calculates the bit usage rate of each macroblock of the coded frame 122 and the coded frame 124 to determine the difference in the bit usage rate of each macroblock when encoded with the first QP 112 and encoded with the second QP 114.

[0027] The macroblock subset selector 140 compares the difference in the quality metric of each macroblock when encoded with the first QP 112 and when encoded with the second QP 114 to the threshold 130. The macroblock subset selector 140 selects macroblocks to include in the subset 142 of macroblocks to be encoded with the second QP based on whether the difference in the quality metric of each macroblock when encoded with the first QP 112 and when encoded with the second QP 114 exceeds the threshold 130. In some embodiments, the macroblock subset selector 140 selects macroblocks to include in the subset 142 of macroblocks to be encoded with the second QP based at least in part on whether the difference in the bit usage of each macroblock when encoded with the first QP 112 and when encoded with the second QP 114 exceeds a second threshold (not shown).

[0028] The macroblock subset selector 140 identifies to the encoder 110 a subset 142 of macroblocks to be encoded with the second QP 114. The encoder 110 encodes the frame 102 based on the identification of the subset 142 by encoding the subset 142 of macroblocks with the second QP 114 to generate the subset 144 of macroblocks encoded with the second QP 114 and encoding the remainder of the macroblocks of the frame 102 with the first QP 112 to generate the encoded frame 140. The encoder 110 transmits the encoded frame 140 to the decoder 150 over a network (not shown). The encoder 150 decodes the encoded frame 140 and provides the decoded video information to a multimedia application or any other application for display to a user at the display 160.

[0029] Although the above example is described with respect to two QPs, it should be appreciated that in some embodiments, the encoder 110 decodes the frame 102 using more than two QPs, each QP being different from the other QPs. Furthermore, in some embodiments, the first QP 112 is selected based on a history of QPs used to encode one or more frames prior to the frame 102. In some embodiments, the comparator 120 uses a mathematical, empirical, or machine learning model to avoid using the two QPs under consideration to calculate the model. The model or machine learning system uses input data to predict the most likely more effective QP. The input data for training / regression includes, for example, pixel data, calculated pixel activity, other calculated metrics about local pixel samples (including summary statistics, mathematical transformations, etc.), quality calculated at multiple QPs, motion information, quality of motion information, etc.

[0030] Figure 2is a diagram showing quality metrics of macroblocks of a frame 200 encoded with a first QP 112 according to some embodiments. Frame 200 includes macroblocks 201-216, where coordinates of the macroblocks are indicated as MB(x,y), where x indicates the row of the macroblock in frame 200, and y indicates the column of the macroblock in frame 200. In the example shown, macroblocks 201, 207, 208, 210, and 215 have low quality metrics, macroblocks 202, 209, 212, and 214 have medium quality metrics, and macroblocks 203, 204, 206, 211, 213, and 216 have high quality metrics when encoded with the first QP 112.

[0031] Figure 3 is a diagram showing quality metrics for blocks of frame 300 encoded with the second QP 114. Similar to Figure 2 , frame 300 includes macroblocks 301-316, where the coordinates of the macroblocks are indicated as MB(x,y), where x indicates the row of the macroblock in frame 300, and y indicates the column of the macroblock in frame 300. In the example shown, macroblocks 302, 304, 309, 311, and 314 have medium quality metrics, and macroblocks 301, 303, 305, 306, 307, 308, 310, 312, 313, 315, and 316 have high quality metrics when encoded with the second QP 114.

[0032] Figure 4 According to some embodiments Figure 1 Block diagram of a comparator 420 of the multimedia system 100, the comparator 420 being configured to select a subset of macroblocks of a frame to be encoded with a second QP 114 based on a quality metric and a bit usage. The comparator 420 is configured to analyze a macroblock 402 encoded with a first QP 112 and a corresponding macroblock 404 encoded with a second QP 114 belonging to the same frame and having the same coordinates as the macroblock 402. A quality metric calculator 425 measures a quality metric (such as an MSE) for each of the macroblocks 402 and 404 to determine a quality metric 412 for the macroblock 402 and a quality metric 414 for the macroblock 404. The quality metric calculator 425 compares the quality metric 412 with the quality metric 414 to determine a difference in the quality metrics of the macroblocks 402 and 404. Bit usage calculator 435 is configured to determine how many bits to use to encode macroblock 402 with first QP 112 and how many bits to use to encode macroblock 404 with second QP 114. Bit usage calculator 435 compares the bit usage to determine the difference in bit usage for encoding macroblock 402 and encoding macroblock 404.

[0033] The macroblock subset selector 440 compares the difference between the quality metric 412 of the macroblock 402 and the quality metric 414 of the macroblock 404 to the threshold 430 to identify whether the difference in the quality metric exceeds the threshold 430. If the difference in the quality metric exceeds the threshold 430, the macroblock subset selector 440 identifies the macroblock 404 as included in the subset of macroblocks to be encoded with the second QP 114. The macroblock subset selector 440 further determines whether the difference in bit usage for encoding the macroblock 402 and encoding the macroblock 404 exceeds a second threshold (not shown). The identification by the macroblock subset selector 440 that the macroblock 404 is included in the subset of macroblocks to be encoded with the second QP 114 is based in part on whether the difference in bit usage for encoding the macroblock 402 and encoding the macroblock 404 exceeds the second threshold.

[0034] For example, if macroblock subset selector 440 determines that the difference between quality metrics 412 and 414 exceeds threshold 430, but also determines that the difference between the bit usage for encoding macroblock 402 and encoding macroblock 404 exceeds a second threshold, then in some embodiments, macroblock subset selector 440 does not identify macroblock 404 as included in the macroblock subset to be encoded with the second QP 114 because the cost is too high. Conversely, if macroblock subset selector 440 determines that the difference between quality metrics 412 and 414 does not exceed threshold 430, but also determines that the difference between the bit usage for encoding macroblock 402 and encoding macroblock 404 does not exceed the second threshold, then in some embodiments, macroblock subset selector 440 identifies macroblock 404 as included in the macroblock subset to be encoded with the second QP 114 because the cost is lower.

[0035] Figure 5 is a diagram showing a subset of the blocks of the frame 500 selected to be encoded with the second QP 114 and the remainder of the blocks of the frame selected to be encoded with the first QP 112 according to some embodiments. Figure 5 The macroblock corresponds to Figure 2 and Figure 3 In the example shown, macroblocks 501, 507, 508, 510, and 515 have been selected by macroblock subset selector 440 to be included in the subset of macroblocks to be encoded with the second QP 114. Figure 2 As shown, each of the macroblocks 201, 207, 208, 210, and 215 of the frame 200 has a low quality as measured by the quality metric when encoded with the first QP 112. Figure 3As shown, each of the corresponding macroblocks 301, 307, 308, 310, and 315 of the frame 300 has high quality per quality metric measurement when encoded with the second QP 114. Therefore, each of these macroblocks shows a relatively high improvement in quality metric when encoded with the second QP 114 as compared to when encoded with the first QP 112. Therefore, the macroblock subset selector 440 selects these macroblocks to be encoded with the second QP 114.

[0036] Figure 6 According to some embodiments Figure 1 1 is a block diagram of a comparator 620 of the multimedia system 100, the comparator 620 being configured to select a subset of macroblocks of a frame to be encoded with a second QP 114 based on the pixel activity of the macroblocks. Figures 1 to 5 In each of the examples shown, encoder 110 encodes the frame in multiple passes using a different QP per pass. In contrast, in some embodiments, comparator 620 identifies macroblocks to include in the subset of macroblocks to be encoded with a higher QP based on a prediction of which macroblocks show the greatest improvement in quality when encoded with a higher QP, without requiring encoder 110 to first encode the frame in multiple passes using different QPs.

[0037] exist Figure 6 In the illustrated embodiment, comparator 620 predicts which macroblocks will have the most improved quality when encoded with a higher QP based on the pixel activity of each macroblock. Many video processing algorithms (such as those used for texture analysis) calculate metrics (such as pixel activity within a block). For example, pixel activity (such as changes in pixel intensity within a block) can be used to detect edges, repeating patterns, and other structures or objects in an image. Pixel activity can be measured using various metrics (such as, for example, using a gray level co-occurrence matrix, a two-dimensional spatial average gradient, a wavelet or other transform, a discrete cosine function, or an average value within a block).

[0038] The comparator 620 is configured to analyze the macroblock 602 encoded with the first QP 112. The comparator 620 includes a pixel activity calculator 645, a threshold 630, and a macroblock subset selector 640. The pixel activity calculator 645 calculates a pixel activity level 612 of the macroblock 602. The macroblock subset selector 640 compares the pixel activity level 612 to the threshold 630. If the pixel activity level 612 exceeds the threshold 630, the macroblock subset selector 640 identifies the macroblock 602 as included in the subset of macroblocks to be encoded with the second QP 114. If the pixel activity level 612 is below the threshold 630, the macroblock subset selector 640 does not include the macroblock 602 in the subset of macroblocks to be encoded with the second QP 114. Therefore, macroblocks with lower pixel activity will be encoded with a lower QP. In some embodiments, the macroblock subset selector 640 identifies the macroblock as included in the subset of macroblocks to be encoded with the second QP 114 even if the pixel activity level 612 does not exceed the threshold 630 (i.e., even for quiet or flat areas). In some embodiments, the determination of whether to include a macroblock in the subset of macroblocks to be encoded with the second QP 114 based on the pixel activity level 612 is based on a default setting of the macroblock subset selector 640, while in other embodiments, the determination is adaptive and is made on a per-macroblock basis.

[0039] Figure 7 is a diagram showing the Figure 1 1. A flowchart of a method 700 implemented by the multimedia system 100 of FIG. 1 for selecting a subset of blocks of a frame for encoding with a second QP 114 based on a quality metric and a bit usage. At block 704, the encoder 110 encodes the frame 102 using the first QP 112 to generate the encoded frame 122. At block 706, the encoder 110 encodes the frame 102 using the second QP 114 to generate the encoded frame 124. At block 708, the quality metric calculator 125 measures a quality metric (such as MSE) for each macroblock of the encoded frame 122 and the encoded frame 124, and compares the measured quality metrics to determine a difference in the quality metric for each macroblock of the encoded frames 122 and 124. At block 710, the macroblock subset selector 140 compares the difference in the quality metric for each macroblock of the encoded frames 122 and 124 with the threshold 130 to determine whether the difference in the quality metric exceeds the threshold 130. If the quality metric difference for a given macroblock does not exceed the threshold 130, the method flow proceeds to block 716. At block 716 , the macroblock subset selector 140 selects macroblocks to include in the subset of macroblocks to be encoded with the first QP 112 .

[0040] If at block 710, the macroblock subset selector 140 determines that the difference in the quality metric for the given macroblock exceeds the threshold 130, then method flow proceeds to block 712. At block 712, the bit usage calculator 135 calculates the difference in bit usage for the macroblock when encoded with the second QP 114 versus the first QP 112. At block 714, the macroblock subset selector 140 selects the macroblock to include in the subset of macroblocks to be encoded with the second QP 114 based on the increase in the quality metric and the difference in bit usage for the macroblock when encoded with the second QP 114 versus the first QP 112.

[0041] Figure 8 is a diagram showing the Figure 1 The multimedia system 100 Figure 6 1. A flowchart of a method 800 implemented by the comparator 620 of the encoder 110 for selecting a subset of macroblocks of a frame for encoding with a second QP 114 based on the pixel activity of the macroblocks. At block 802, the encoder 110 encodes the frame 102 using the first QP 112. At block 804, the pixel activity calculator 645 calculates the pixel activity level of each macroblock of the frame 102. At block 806, the macroblock subset selector 640 determines whether the pixel activity level of each macroblock exceeds the threshold 630. If at block 806, the macroblock subset selector 640 determines that the pixel activity level of the given macroblock exceeds the threshold 630, then the method flow proceeds to block 808. At block 808, the macroblock subset selector 640 selects the macroblock to be included in the subset of macroblocks to be encoded with the second QP 114. If at block 806, the macroblock subset selector 640 determines that the pixel activity level of the given macroblock does not exceed the threshold 630, then the method flow proceeds to block 810. At block 810 , the macroblock subset selector 640 selects macroblocks to include in the subset of macroblocks to be encoded with the first QP indicated by block 112 .

[0042] As disclosed herein, a method includes: calculating a first quality metric using a first measurement method for each portion of a first frame encoded with a first quantization parameter; calculating a second quality metric using the first measurement method for each portion of the first frame encoded with a second quantization parameter lower than the first quantization parameter; selecting a first subset of portions of the first frame to be encoded with the second quantization parameter based on a comparison of the first quality metric and the second quality metric; and encoding the selected first subset of portions of the first frame with the second quantization parameter for transmission over a network. In one aspect, selecting the first subset includes selecting portions of the first frame having a difference between the first quality metric and the second quality metric greater than a threshold. In another aspect, the method includes: encoding a second subset of portions of the first frame with the first quantization parameter based on the comparison, the second subset being different from the first subset. In yet another aspect, selecting the first subset of portions is further based on a comparison of the number of bits used to encode each portion of the first frame using the second quantization parameter and the number of bits used to encode each portion of the first frame using the first quantization parameter.

[0043] In one aspect, the first measurement method is based on an average of squares of differences between the first frame and the first frame encoded with one of the first quantization parameter or the second quantization parameter. In another aspect. In yet another aspect, the first quantization parameter is selected based on a second frame preceding the first frame. In yet another aspect, the method includes: calculating a third quality metric using the first measurement method for each portion of the first frame encoded with a third quantization parameter higher than the first quantization parameter; selecting a second subset of portions of the first frame to be encoded with the third quantization parameter based on a comparison of the first quality metric and the third quality metric; and encoding the second subset of portions of the first frame with the third quantization parameter.

[0044] As disclosed herein, a method includes: encoding a first subset of portions of a first frame with a first quantization parameter; in response to predicting that a quality metric of the second subset of portions of the first frame when encoded with a second quantization parameter lower than the first quantization parameter will be greater than a threshold value than the quality metric of the second subset of portions when encoded with the first quantization parameter, selecting a second subset of the portions to be encoded with the second quantization parameter, the second subset being different from the first subset; and encoding the second subset of portions of the first frame with the second quantization parameter. In one aspect, selecting the second subset of portions is further based on a comparison of a number of bits used to encode each portion of the first frame using the second quantization parameter and a number of bits used to encode each portion of the first frame using the first quantization parameter. In another aspect, the quality metric is based on an average of squares of differences between the first frame and the first frame encoded with one of the first quantization parameter or the second quantization parameter.

[0045] In one aspect, the first quantization parameter is selected based on a second frame preceding the first frame. In another aspect, the method includes: calculating a quality metric for each portion of the first frame encoded with a third quantization parameter higher than the first quantization parameter; for each portion, comparing the quality metric of the portion encoded with the first quantization parameter with the quality metric of the portion encoded with the third quantization parameter; selecting a third subset of portions of the first frame to be encoded with the third quantization parameter based on the comparison; and encoding the third subset of portions of the first frame with the third quantization parameter. In yet another aspect, the method includes: predicting, based on a pixel activity level of the second subset of the portions, whether a quality metric of the second subset of the portions will be greater than a threshold when encoded with the second quantization parameter than a quality metric of the second subset of the portions when encoded with the first quantization parameter. In yet another aspect, the pixel activity level is based on one of: a gray level co-occurrence matrix, a spatial gradient, a wavelet transform, a discrete cosine function, or an average within a portion.

[0046] As disclosed herein, an apparatus includes a comparator configured to: identify a first quality metric based on a first measurement method for a portion of a first frame when encoded with a first quantization parameter; identify a second quality metric based on the first measurement method for a portion of the first frame when encoded with a second quantization parameter lower than the first quantization parameter; and select a first subset of portions of the first frame for which a difference between the first quality metric and the second quality metric exceeds a specified threshold; and an encoder configured to: encode the selected first subset of portions with the second quantization parameter. In one aspect, the comparator is further configured to: calculate a number of bits for encoding each portion of the first frame using the first quantization parameter; calculate a number of bits for encoding each portion of the first frame using the second quantization parameter; and further select the first subset of portions based on a comparison of the number of bits for encoding each portion of the first frame using the second quantization parameter and the number of bits for encoding each portion of the first frame using the first quantization parameter.

[0047] In one aspect, the first measurement method is based on an average of the squares of the differences between the first frame and the first frame encoded with one of the first quantization parameter or the second quantization parameter. In another aspect, the first quantization parameter is selected based on a second frame preceding the first frame. In yet another aspect, the comparator is further configured to: predict based on a pixel activity level of a first subset of the portion whether a quality metric of the first subset of the portion when encoded with the second quantization parameter will be greater than a threshold value than the quality metric of the first subset of the portion when encoded with the first quantization parameter; and further select the first subset of the portion based on the prediction. In yet another aspect, the pixel activity level is based on one of: a gray level co-occurrence matrix, a spatial gradient, a wavelet transform, a discrete cosine function, or an average value within a portion.

[0048] In some embodiments, the apparatus and techniques described above are implemented in a system (such as the one described above with reference to) having one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips). Figures 1 to 8The electronic design automation (EDA) and computer-aided design (CAD) software tools can be used to design and manufacture these IC devices. These design tools are usually represented as one or more software programs. The one or more software programs include codes that can be executed by a computer system to manipulate the computer system to operate the code representing the circuit of one or more IC devices so as to execute at least a part of the process for designing or adapting the manufacturing system to manufacture the circuit. This code may include instructions, data, or a combination of instructions and data. Software instructions representing design tools or manufacturing tools are usually stored in a computer-readable storage medium that can be accessed by a computing system. Similarly, codes representing one or more stages of the design or manufacture of an IC device can be stored in the same computer-readable storage medium or different computer-readable storage media, and accessed from the same computer-readable storage medium or different computer-readable storage media.

[0049] Computer-readable storage media include any non-transitory storage media or combination of non-transitory storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical system (MEMS)-based storage media. Computer-readable storage media may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disc or flash memory based on a universal serial bus (USB)), or coupled to a computer system via a wired or wireless network (e.g., a network accessible storage device (NAS)).

[0050] In some embodiments, certain aspects of the technology described above are implemented by one or more processors of a processing system that executes software. The software includes one or more executable instruction sets stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software includes instructions and certain data that manipulate one or more processors to execute one or more aspects of the technology described above when executed by one or more processors. Non-transitory computer-readable storage media include, for example, magnetic or optical disk storage devices, solid-state storage devices (such as flash memory, cache, random access memory (RAM) or one or more other non-volatile memory devices), etc. The executable instructions stored on the non-transitory computer-readable storage medium are in the form of source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executed by one or more processors.

[0051] It should be noted that not all activities or elements described above in the general description are required, a portion of a particular activity or device is not necessarily required, and one or more other activities may be performed, or elements other than those described may be included. In addition, the order in which the activities are listed is not necessarily the order in which the activities are performed. In addition, the concepts have been described with reference to specific embodiments. However, it will be appreciated by those of ordinary skill in the art that various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the appended claims. Therefore, this specification and the drawings will be regarded as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the present disclosure.

[0052] The benefits, other advantages, and solutions to problems have been described above with reference to specific embodiments. However, the benefits, advantages, solutions to problems, and any features that may make any benefit, advantage, or solution to the problem appear or become more prominent should not be interpreted as being key, essential, or necessary features of any or all claims. In addition, the specific embodiments disclosed above are merely illustrative, as the disclosed subject matter may be modified and practiced in different but equivalent ways that are obvious to those skilled in the art who benefit from the teachings of this article. Except as described in the appended claims, it is not intended to limit the details of the construction or design shown herein. Therefore, it is apparent that the specific embodiments disclosed above may be changed or modified, and all such variations are considered to be within the scope of the disclosed subject matter. Therefore, the protection sought herein is as stated in the appended claims.

Claims

1. A method for allocating bits to portions of a frame, comprising: calculating a first quality metric using a measurement method for each portion of a first frame encoded with a first quantization parameter; calculating a second quality metric using the measurement method for each portion of the first frame encoded with a second quantization parameter lower than the first quantization parameter; selecting a first subset of portions of the first frame to be encoded with the second quantization parameter based on a difference between the first quality metric and the second quality metric exceeding a threshold; as well as A selected first subset of a portion of the first frame is encoded with the second quantization parameter for transmission over a network.

2. The method of claim 1, further comprising: A second subset of portions of the first frame is encoded with the first quantization parameter based on the difference, the second subset being different from the first subset.

3. The method of claim 1, wherein selecting the first subset of the portions is further based on a comparison of a number of bits used to encode each portion of the first frame using the second quantization parameter and a number of bits used to encode each portion of the first frame using the first quantization parameter.

4. The method of claim 1, wherein the measurement method is based on an average of squares of differences between the first frame and the first frame encoded with one of the first quantization parameter or the second quantization parameter.

5. The method of claim 1, wherein the first quantization parameter is selected based on a second frame preceding the first frame.

6. The method of claim 1, further comprising: calculating a third quality metric using the measurement method for each portion of the first frame encoded with a third quantization parameter higher than the first quantization parameter; selecting a second subset of portions of the first frame to be encoded with the third quantization parameter based on a comparison of the first quality metric and the third quality metric; as well as A second subset of the portion of the first frame is encoded using the third quantization parameter.

7. A method for allocating bits to portions of a frame, comprising: encoding a first subset of the portion of the first frame using a first quantization parameter; selecting a second subset of portions of the first frame to be encoded with a second quantization parameter lower than the first quantization parameter, the second subset being different from the first subset, in response to predicting that a difference between a quality metric of a second subset of portions of the first frame when encoded with a second quantization parameter and a quality metric of the second subset of portions when encoded with the first quantization parameter will exceed a threshold; as well as A second subset of the portion of the first frame is encoded using the second quantization parameter.

8. The method of claim 7, wherein selecting the second subset of the portions is further based on a comparison of a number of bits used to encode each portion of the first frame using the second quantization parameter and a number of bits used to encode each portion of the first frame using the first quantization parameter.

9. An apparatus for allocating bits to portions of a frame, comprising: A comparator, the comparator being used to: identifying a first quality metric based on a measurement method for a portion of the first frame when encoded with a first quantization parameter; identifying a second quality metric based on the measurement method for a portion of the first frame when encoded with a second quantization parameter lower than the first quantization parameter; and selecting a first subset of portions of the first frame for which a difference between the first quality metric and the second quality metric exceeds a specified threshold; as well as An encoder is provided for encoding the selected first subset of portions using the second quantization parameter.

10. The apparatus of claim 9, wherein the comparator is further configured to: calculating a number of bits for encoding each portion of the first frame using the first quantization parameter; calculating a number of bits for encoding each portion of the first frame using the second quantization parameter; and A first subset of the portions is selected further based on a comparison of a number of bits used to encode each portion of the first frame using the second quantization parameter and a number of bits used to encode each portion of the first frame using the first quantization parameter.

11. The apparatus of claim 9, wherein the measurement method is based on an average of squares of differences between the first frame and the first frame encoded with one of the first quantization parameter or the second quantization parameter.

12. The apparatus of any one of claims 9-11, wherein the first quantization parameter is selected based on a second frame preceding the first frame.

13. The computing device of any one of claims 9 to 11, wherein the comparator is further configured to: predicting, based on the pixel activity levels of the first subset of the portions, whether a difference between a quality metric of the first subset of the portions when encoded with the second quantization parameter and a quality metric of the first subset of the portions when encoded with the first quantization parameter will exceed a threshold; and A first subset of the portions is selected based further on the prediction.

14. The device of claim 13, wherein the pixel activity level is based on one of: a gray level co-occurrence matrix, a spatial gradient, a wavelet transform, a discrete cosine function, or an average within a section.

Images