Fixed rate intraframe compression and decompression of video based on visual quality

Abstract

To provide methods, programs and systems for fixed bitrate intraframe compression of video based on 2D discrete cosine transform (DCT) to control a target code amount, on the basis of the visual quality of final decompressed imagery.SOLUTION: A method involves assigning an initial bit budget per macroblock of a given video frame, resulting in degrees of compression, i.e., quantization scale factors, that vary among the macroblocks according to the complexity of the macroblocks. The scale factors are then adjusted while maintaining the overall frame bit budget to reduce the visibility of artifacts in the decompressed frame. The adjustments may include increasing scale factors for simple macroblocks, and reducing scale factors for complex blocks. This can reduce the visibility of compression-related artifacts both in complex and simple portions of the frame.SELECTED DRAWING: Figure 2

Description

Prior art

[0001] The digital data representing moving images is enormous and grows as video image resolution increases from SD to HD, 4K / UHD, and then 8K / UHD2. At each stage, the number of pixels increases fourfold. However, when editing video, editors do not need to view images at their full native resolution or the full resolution of the final rendered imagery. In most situations, HD images provide sufficient visual quality to guide editors through the editing process. Using such reduced-resolution imagery, called proxies, for editing purposes avoids unnecessary consumption of computing resources and bandwidth because video sources are often shared from central server farms and not copied locally. Reduced bandwidth requirements mean that more editing clients can be supported by a given infrastructure, increasing cost efficiency. Therefore, as increasing levels of compression are required to generate proxies from increasingly large amounts of original media while still meeting available bandwidth (bitrate) constraints, there is an increasing need for methods to generate visually acceptable proxies.

[0002]

[0002] Typically, maximum visual quality per bitrate is achieved using interframe compression techniques, such as long Group of Pictures (GOP) compression / decompression (codecs). These typically achieve the same image quality as comparable single-frame (intraframe) compression methods at a bitrate cost of about 10%. However, interframe codecs incur significant processing overhead, especially when seeking between individual frames and quickly moving between different locations, two key requirements during the editing process. GOP codecs cannot match the performance of intraframe codecs with respect to these requirements.

[0003]

[0003] Various intraframe compression techniques have been used to generate proxies. A commonly used technique employs variations of the (still image) JPEG algorithm, which is based on a combination of a 2D Discrete Cosine Transform (DCT), a tunable quantization step (lossy), and a final Variable Length Coding (VLC) stage.

[0004]

[0004] Still image codecs such as JPEG only need to accommodate a single image. One of their goals is to conserve storage space, but image quality (quantization error) is considered the primary factor in driving compression control, and the resulting (reduced) frame size is taken for granted.

[0005] In video, the complexity and detail of image content frequently vary widely over time from frame to frame. Applying image-focused JPEG techniques to video without modification maintains the same quality level across subsequent frames, resulting in a bitstream that varies widely, e.g., in bytes per second, during transmission. However, because video relies on the regular transmission of a fixed number of images over a transmission channel with limited bandwidth (maximum bit-rate capacity), one of two possible outcomes can occur: (1) if the compressed frame size is too large, transmission will not be completed within the time slice available for that frame's transmission to maintain a regular image sequence, resulting in a wobbly display; or (2) the quality of the video may have to be significantly degraded in order for one or more frames to remain within the bit budget available for the time slice.

[0006] To avoid these problems, it is necessary to devise a rate controller that controls the compressed size of frames over time to achieve either a constant bit rate (CBR), where each frame has exactly the same size, or a variable bit rate (VBR), avoiding a long-term constant average bit rate. The latter is only possible if the receiver is willing to tolerate buffering and time delays in decoding to cope with excess frame sizes.

[0007] One widely used constant bitrate intraframe video codec is the Avid DNx codec, developed by Avid Technology, Inc. of Burlington, Massachusetts. It is described in U.S. Patent Nos. 7,403,561, 7,433,519, 7,729,423, 7,916,363, and 8,154,776, the contents of which are incorporated herein by reference in their entirety. Avid DNx supports both CBR and VBR encoding and uses a rate controller that emphasizes minimizing mathematical errors introduced by compression at relatively moderate compression ratios (medium or archival quality). As discussed below, this leads to several unreasonable assumptions that, when applied to high compression ratios, adversely affect visual image quality and reduce its usefulness for post-production low-bitrate proxy workflows. Summary of the Invention [Problem to be solved by the invention]

[0008]

[0008] Therefore, there is a need for a compression scheme for intra-frame compression of video that can optimize the visual quality of highly compressed video proxies to support the video editing process while satisfying bandwidth requirements and computational constraints. This need is further emphasized by the fact that the typical bandwidth available for distribution of video proxies does not increase proportionally with the size and dimensions (4K, 8K) of the original video images. [Means for solving the problem]

[0009]

[0009] Typically, constant bit rate, intraframe compression methods for video emphasize final visual quality over evaluating the degree of compression.

[0010]

[0010] Generally, in one aspect, a method for constant bit rate, intraframe compression of a video comprising a sequence of images includes, for each image, the steps of: segmenting the image into n portions, for each of the n portions, compression parameters being individually adjustable; transforming each of the image portions to generate frequency domain coefficients for each of the portions; selecting a stored quantization matrix for the image based on a desired bit rate for the compressed video; determining an initial bit budget for each transformed portion of the image, the initial bit budget equal to the bit budget for the image within the compressed video divided by n; and The method includes determining, for each transform portion, an initial scaling factor that is the smallest scaling factor, which, when applied to the frequency-domain coefficients along with a quantization matrix, results in a bit-size of the transform portion that is less than or equal to an initial bit budget for the transform portion; and generating a bitstream representing the compressed image, wherein, for each transform portion, adjusting the frequency-domain coefficients of that portion by applying the initial scaling factor determined for that portion along with the quantization matrix to that portion, and arranging the adjusted frequency-domain coefficients in a predetermined sequence and applying variable-length coding to the sequence of adjusted frequency-domain coefficients to generate the bitstream.

[0011] Various embodiments include one or more of the following features: Evaluating the transform portions in order of their initial magnifications and determining an average of the initial magnifications; Proceeding in ascending order of initial magnifications, downwardly adjusting the initial magnifications of a plurality of transform portions having an initial magnification greater than the average initial magnification, where the amount of adjustment is a decremental fraction of the difference between the initial magnification of a given transform portion and the average initial magnification; Terminating the adjustment of the initial magnifications if, as a result of adjusting the additional transform portions, the sum of the bit sizes of the adjusted transform portions of the image exceeds the bit budget for the image within the compressed video; Adjusting the initial magnification of each portion having an initial magnification less than a threshold magnification upward before downwardly adjusting the initial magnifications of a plurality of transform portions having an initial magnification greater than the average initial magnification, where the threshold magnification is based on the average initial magnification and the standard deviation of the initial magnifications.

[0012] In general, in another aspect, a computer program product comprises a non-transitory computer-readable medium having computer-readable instructions encoded thereon that, when processed by a processing device, direct the processing device to perform a constant bit rate, intra-frame compression method for a video including a sequence of images. The method includes, for each image, the steps of segmenting the image into n portions, for each of the n portions, compression parameters being individually adjustable; transforming each of the portions of the image to generate frequency-domain coefficients for each of the portions; selecting a stored quantization matrix for the image based on a desired bit rate for the compressed video; determining an initial bit budget for each transformed portion of the image, the initial bit budget equal to the bit budget for the image in the compressed video divided by n; determining, for each transformed portion, an initial scaling factor that is the smallest scaling factor, where applying the initial scaling factor together with the quantization matrix to the frequency-domain coefficients results in a bit-size of the transformed portion less than or equal to the initial bit budget for the transformed portion; and generating a bitstream representing the compressed image, where for each transformed portion, the steps of adjusting the frequency-domain coefficients for the portion by applying the determined initial scaling factor to the portion together with the quantization matrix; and arranging the adjusted frequency-domain coefficients into a predetermined sequence and applying variable length coding to the sequence of adjusted frequency-domain coefficients to generate the bitstream.

[0013]

[0013] Generally, in yet another aspect, a system includes a memory storing computer-readable instructions and a processor connected to the memory, which, when executed by the processor, causes the system to perform a constant bit rate, intra-frame compression method for video including a sequence of images. The method includes, for each image, the steps of segmenting the image into n portions, for each of the n portions, compression parameters being individually adjustable; transforming each of the portions of the image to generate frequency-domain coefficients for each of the portions; selecting a stored quantization matrix for the image based on a desired bit rate for the compressed video; determining an initial bit budget for each transformed portion of the image, the initial bit budget equal to the bit budget for the image in the compressed video divided by n; determining, for each transformed portion, an initial scaling factor that is the smallest scaling factor, where applying the initial scaling factor together with the quantization matrix to the frequency-domain coefficients results in a bit-size of the transformed portion less than or equal to the initial bit budget for the transformed portion; and generating a bitstream representing the compressed image, where for each transformed portion, the steps of adjusting the frequency-domain coefficients for the portion by applying the determined initial scaling factor to the portion together with the quantization matrix; and arranging the adjusted frequency-domain coefficients into a predetermined sequence and applying variable length coding to the sequence of adjusted frequency-domain coefficients to generate the bitstream. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a high-level flow diagram of the major steps involved in digital image compression using the discrete cosine transform. [Figure 2] 1 is a high-level flow diagram of the steps involved in describing a digital image compression method based on the visual quality of the results. [Figure 3]1 is a graph showing a typical distribution of initial scaling factors determined for macroblocks of an image, approximating a constant compression bit budget per macroblock. [Figure 4] 1 is a graph comparing the error actually introduced by compression with the predicted error used in traditional rate controllers as a function of the compression factor q. [Figure 5] FIG. 1 is a high-level block diagram of a system and use case implementing the described rate controller. DETAILED DESCRIPTION OF THE INVENTION

[0015]

[0019] This paper describes a video rate controller designed primarily for low bit rates. It approaches the rate control problem from a fundamentally different starting point than previous approaches. It accepts the typical presence of coding artifacts in decoded frames and emphasizes a human-centered approach that attempts to minimize the visibility of the artifacts instead of trying to minimize the overall mathematical error of the encoded frames. Below, we describe how previous rate controllers work, point out the pitfalls of using them for high compression ratios, and then describe a human-centered video rate controller that addresses these pitfalls.

[0016]

[0020] Figure 1 shows a high-level flow diagram of the major steps involved in digital image compression using the Discrete Cosine Transform. An uncompressed image input 102 is accepted by a DCT 104. The DCT used in the codecs discussed herein operates on blocks of 8x8 pixels. The DCT converts spatial components into an equivalent set of 8x8 floating-point frequency coefficients. This process is lossless, so that when subsequently accepted intact by an inverse DCT, the original 8x8 pixels are recovered. For original pixel values ​​with a bit depth of b, the resulting floating-point frequency components span a floating-point range equal to the integer values ​​of bit depth b+3. In practice, this integer representation is effectively lossless, since the floating-point components can therefore be rounded to their nearest integer value, and this rounding does not significantly affect the perfect reproduction. The frequency components can then be accepted by a VLC encoding scheme 108 without any further processing. The VLC encoding method 108 exploits common properties associated with coefficient data to achieve somewhat lighter compression compared to the original. Bitrate or quality adjustments associated with DCT-based compression (discussed below) serve to enhance these properties and reduce VLC-encoded size.

[0017]

[0021] The VLC-encoded size of a DCT block cannot be predicted mathematically with any precision. To determine the final bit cost of a DCT block, VLC encoding must be performed at least up to the level of determining the final total code length. This efficient size determination is a central problem in existing rate controllers, as it must be performed very frequently during the rate control process.

[0018]

[0022] The quantization step 106 performs compression adjustment. The parameters that control this adjustment are the frequency coefficients

number

number

[0019]

[0023] where [...] indicates rounding down to the next lower integer.

number

number

[0020]

[0024] Next range

number

number

number

number

number

[0021]

[0025] The original JPEG compression algorithm uses a single quantization factor q for the entire image. The rationale for this is that it is an error

number

number

[0022]

[0026] DCT-based video codecs typically follow the same paradigm, since they are primarily designed for storing media in mezzanine formats. For them, mathematically defined quality is the primary objective, subject to the extra size constraints of CBR or VBR. The simplest way to control bitrate in I-frame codecs is to increase the quantization parameter q until the final size is reduced below the target compressed frame size. Applying a single q across an entire frame, coupled with a constant (CBR) compressed frame size, results in large, clearly visible quality fluctuations across successive frames. Using an average (VBR) compressed frame size across multiple frames avoids these (fast) quality fluctuations (known as "flutter"), but significantly degrades image quality over longer time periods. This also makes rate control more difficult and incurs significant encoding delays for multiple frames, since the encoder may have to encode a frame multiple times in succession to achieve the desired temporal average.

[0027] To improve bitrate control, video codecs require an additional control lever for adjusting the bitrate per frame. This allows them to overcome the problem of flutter while still controlling the overall compressed frame size. In doing so, they segment the entire image into smaller sections, called slices, and then adjust the quantization parameter q for these slices individually. Avid DNx takes this to the extreme by allowing adjustment of q per 16x16 pixel block, known as a macroblock (MB). A macroblock represents the smallest entity into which all components of a subsampled video signal (4:2:2 or 4:2:0) can be organized into 8x8 DCT blocks.

[0023]

[0028] At a high level, traditional rate controllers like Avid DNx perform the following steps: (1) Choose a uniform q, so that the frame size is a rough approximation of the target frame size. (2) Fine-tune the q value for each individual macroblock (up or down from the initially assigned uniform value) to more closely approximate the compressed target frame size while simultaneously reducing the overall quantization error.

number

[0024]

[0029] To simplify the calculation of the total error E, the relationship

number

number

number

[0025]

[0030] As mentioned above, the final VLC-encoded size of a MB cannot be accurately predicted. This is because the dependence of frame size on q is determined by the actual coefficient values ​​in the MB. Varying q results in a highly nonlinear and generally abrupt change in MB size. This is because, depending on how q is adjusted, multiple (integer) coefficient values ​​may become zero or nonzero. However, MB size maintains the following qualitative relationship with q: (A) MB size decreases as q increases. (B) Blocks with many significantly nonzero coefficients, i.e., visually complex blocks, respond only slowly in terms of quality to changes in q but may experience large size changes. (C) Blocks with relatively few significantly nonzero coefficients, i.e., simple blocks, typically respond very quickly to changes in q (both quality and size) at small values ​​of q and very slowly as q is further increased. For these blocks, the overall size change is always small, regardless of how q is varied.

[0026]

[0031] Because traditional rate controllers attempt to minimize the overall error, and because the error contribution of an MB depends only on q and not on the MB's complexity, rate controllers often focus on adjusting the q parameter that has the greatest effect on the compressed MB size. This applies to complex blocks, leaving simple blocks alone. Once the bitrate begins to approach, the rate controller rebalances the q distribution between simple and complex blocks to reach a minimum value for Σq. However, this later stage does not affect the quantization of simple blocks, because a change in q → q-1 in a complex block must be compensated for either by a change in q → q+1 in many simple blocks or by a change in q → q+x in a few blocks. In either case, the consequences for Σq are very negative, since this always leads to an effective increase in the compressed size. As a result, the rate controller must redistribute the quantization only among blocks with roughly the same complexity. Compensating for this by moving one block from q to q+1 and the other from q to q-1 only results in a slight approximation to the target bitrate. This is consistent with the observation that complex blocks contain a lot of detail, while simple blocks are relatively flat. If the goal is to stay as close to the original quality as possible, the rate controller will naturally prioritize complex blocks, since they contribute significantly to the compressed size of the frame.

[0027]

[0032] Traditional methods have two fundamental pitfalls that become apparent at high compression ratios. The first pitfall has to do with the fact that traditional controllers, as discussed above, emphasize complex blocks at the expense of simple blocks. At low compression ratios, it is not too difficult to reach the target bit rate, and the number of q-induced zeros in the coefficients is low. Compression is obtained primarily through a numerical reduction of amplitudes, and values

number

number

[0028]

[0033] The second pitfall stems from the way previous controllers approximate the error. The control algorithm assumes that the error E is always proportional to q, as shown in 402 of FIG. 4. While this is true for small values ​​of q used for low compression ratios, the actual error is much less proportional for larger values ​​of q, as shown in 404 of FIG. 4. This is because once a coefficient becomes zero as a result of division by q, the error for this coefficient no longer increases. However, the error estimate approximated by the value of q implicitly assumes that all coefficients continue to contribute to the error as q increases. This discrepancy primarily affects low- to medium-complexity blocks. Because they start with only a few nonzero coefficient values ​​even before quantization, the contribution of these blocks to the overall error is grossly overestimated, especially for the large values ​​of q required for high compression ratios. This makes adjustments to improve these blocks seem much less attractive to the rate controller than they should be. The first pitfall described above leads to much lower quality on simple blocks than would otherwise be achievable, but produces virtually no discernible change in visual quality on high-complexity blocks. The net result of these pitfalls is that it effectively "blinds" the rate controller to adjusting for high compression ratios on low-complexity blocks.

[0029]

[0034] To overcome these pitfalls, the rate controller described herein extends the basic control framework to

number

[0030]

[0035] There are various ways to segment an image into relevant and irrelevant sections. One method involves assigning the same compression size (bit cost) to each macroblock and then determining the q factor required to achieve this goal, thereby determining the complexity of the image portion. Visually simple blocks will have a small q factor, while visually complex blocks will have a large q factor. The final q factor can then be used as a basis for classifying the complexity of the macroblock, which then drives the rest of the rate controller based solely on this initial classification.

[0031]

[0036] This offers several advantages over the RDO approach. In the RDO approach, the rate controller effectively attempts to control a simple, 100% predictable entity (q) by indirectly adjusting the unpredictable compressed size to minimize Σq. In this case, the controller makes tradeoff decisions on a large number of complex microblocks in order to predict the entity and achieve a better overall result for Σq. The new approach ignores Σq and focuses on improving the accuracy of the predictable parameter q from a starting point determined by the initial complexity classification until it reaches the total bitrate budget. To achieve this, the new approach utilizes only very general, generic properties (A) through (C) to manipulate the q coefficient and drive the compressed size approximately toward the target bitrate. This transforms the RDO approach's consideration of a large number of complex microblocks into a simple macroblock-by-macroblock (i.e., parallelizable) adjustment decision.

[0032]

[0037] The main steps of a fixed-rate encoder using the rate controller described herein are as follows: The encoder receives uncompressed image data (202), segments the image into macroblocks (204), and applies a DCT to convert the macroblock pixels into frequency coefficients (206). The rate controller then determines an initial bit budget for each macroblock (208) and allocates this constant compression size to each MB in an allocation step (210). The rate controller then determines (210) a value of q that results in a compression size that corresponds exactly to the allocated size. This allows low-complexity blocks to maintain a low compression ratio (low q), while forcing high-complexity blocks to a high compression ratio (high q). In an analysis step (212), the macroblocks are sorted according to the determined q factor, from simple blocks with low q to complex blocks with high q. Then, in an adjustment step (214), the q coefficients are adjusted to rebalance the q distribution, maintaining this ordering by complexity until the target bit rate is met, as described in more detail below. Once one or more adjustments have been performed, as described below, the adjusted scaling factors are used to quantize the macroblock frequency coefficients in a quantization step 218. VLC encoding is then applied to the quantized frequency coefficients (220), outputting a constant bit rate compressed bitstream (222).

[0033]

[0038] Figure 3 shows a typical distribution of the initial q coefficient. s This roughly indicates the area where most of the blocks in the image will initially fall, based on the available bitrate. The distribution does not usually follow a Gaussian normal distribution at this stage, but instead follows a Gaussian normal distribution, q s (towards the simple blocks) while extending significantly to the right (towards the more complex blocks and finally to the very complex blocks). As a result, the average q coefficient q determined from this distribution will have a peak q sDuring the adjustment step, the rate controller moves this distribution towards a normal distribution, one macroblock at a time.

[0039] The blocks below the q-bar are already considered to have high visual quality, but further compression would generally degrade them to the point where artifacts become visible. The overall goal is not to adjust these blocks, but to focus on significantly reducing the q-factors above the q-bar to improve their visual quality, while using the extra bitrate created by the underperformance encountered during the analysis step.

[0034]

[0040] Since all blocks have the same size, q-bar is not only the average q-factor of the distribution, but also the "center of weight" of the block size distribution. After the analysis step, exactly half of the target size is allocated to blocks below q-bar, and half to blocks above q-bar. Adjusting the q-factor shifts the q-distribution towards a more balanced q-distribution, i.e., q-bar → q s Decreasing the value of q shifts the size distribution from q to q s This results in a shift in the size of the blocks, but increases the size of the blocks, i.e., redistributing the (compressed) sizes from low-complexity blocks to high-complexity blocks. This also has the desired effect of reducing the overall mathematical error E, but achieves this as an implicit result of the adjustment stage, rather than as an explicit goal or control parameter. At low compression ratios, the results of a visual quality-based rate controller are therefore similar to those of the RDO approach.

[0035]

[0041] q sIf is not too large, i.e., less than a value of about 16, increasing the q factor for simple blocks by only a small amount (e.g., by 1) usually does not cause any noticeable quality degradation. This can be used by the rate controller to free up extra bitrate for downward adjustments to complex MBs. However, the rate controller remains cautious in these adjustments, because the general property (C) indicates that there is little margin to be gained without causing significant damage to the visual quality of these blocks.

[0036]

[0042] When adjusting the q-factor of a block, the rate controller preserves the ranking established during the analysis step, since it thus determines the relative priority for the block to be adjusted if free bitrate is still available.

[0037]

[0043] The exact strategy used by the rate controller to peruse the available bitrate is open to variation, provided it maintains the relative ranking of macroblocks. The initial analysis will often provide a reasonable starting point, and subsequent adjustment steps will generally result in relatively small incremental improvements in the visual quality of the encoded result.

[0038]

[0044] The way the rate controller prioritizes block adjustment is influenced by two considerations. First, at high compression ratios, some blocks are so overly complex that a correspondingly large q is required to bring them within the macroblock bit cost range of the analysis range, resulting in these blocks being significantly more prominent at the high end of this q range. These very high q values ​​result in effectively only the DC component (X(0,0)) remaining after quantization, resulting in the entire 16x16 block being a monochromatic block embedded in many neighboring high-complexity blocks, yet still exhibiting a lot of intra-macroblock variation. If left uncorrected, such blocks would be noticeable in the decompressed image. To address this, the rate controller's first adjustment step reduces the top 5% of q values ​​(based on a two-sigma approach when using a normal distribution), i.e., the upper end of the distribution, to the value at the 95th percentile. This adjustment preserves the block complexity ranking.

[0039]

[0045] A second consideration arises from hard edges in the image. These are characterized by large jumps in luminance or saturation and can cause blocks to be classified as complex despite their normally much lower q level (typically at or just below the q + σ level in the initial q distribution). Excessively strong distortion in these areas draws the viewer's attention because visible edges typically segment the environment into large-scale objects, and the human brain focuses attention on these edges. To address this issue, the rate controller reduces the q value of all macroblocks at or above this level. In most cases, making relatively conservative adjustments per macroblock is preferable because it allows a larger number of macroblocks to be improved with the available bit budget.

[0040]

[0046] These two considerations are often so important to overall quality that if the free bitrate is used up before they can be completed, the rate controller may slightly degrade some of the blocks around the q-bar in order to gain the necessary bit-budget headroom. If free bitrate is still available after these two adjustments, the rate controller can begin to successively decrease q by small increments for all blocks above the q-bar, thus equally sharing the available free bitrate, and can repeat this process again and again whenever free bitrate budget becomes available.

[0041]

[0047] All of the individual adjustment steps are performed individually on a macroblock without any interaction with any other macroblock, except for the final acceptance of the adjustment based on the available bitrate, so all bit cost evaluation (which accounts for 98% of the rate controller's work) can be fully parallelized.

[0042]

[0048] The rate controller described herein results in a reduction of the mean q for the distribution because it simplifies the downward distribution of q towards lower values ​​and emphasizes larger q values. This minimizes or reasonably approaches a minimum value for the resulting Σq (respectively, the total error E) of the final q distribution. This means that even for low compression ratios, the results meet the expectation of "near-minimum E" that is the objective of the RDO controller, and therefore produce very similar results.

[0043]

[0049] Images compressed with the rate controller described above are fully compatible with existing decoders for compressed images based on rate-distortion optimization. In particular, it is compatible with all variants of the Avid DNx codec, including Avid DNx HD (High Definition) and Avid DNx HR (High Resolution). The described controller does not require any changes to the structure of previously compressed bitstreams. Other than macroblock orientation and a constant target bitrate, the algorithm does not require any further codec-specific assumptions, and as a result, it is generally applicable to all codecs that allow for adjustment of the scaling factor for macroblocks on a macroblock-by-macroblock basis.

[0044]

[0050] FIG. 5 illustrates a high-level block diagram of a system and use case for implementing the described rate controller. An editor 502 includes a software, non-linear video editing application, such as Media Composer®, a product of Avid Technology, Inc. This video editing application may be hosted on a computer system local to the editor, a remote system, or the cloud. A video encoder 504 includes software, hardware, firmware, or a combination that hosts a codec 506 that performs the video compression methods described herein. The video encoder receives uncompressed video from storage 508, which may be local or remote to the editor. In various use cases, uncompressed video is streamed directly to the editor from a live source, such as a video camera. The editor may initially receive the uncompressed video before transferring it to the video encoder. Alternatively, the video encoder may receive the uncompressed video directly from the source. The codec includes an input formatter module 510, a DCT module 512 that converts blocks of image data into frequency coefficients, a rate controller module 514, and an entropy encoder 516. The rate controller module coordinates the compression, as previously described. It receives frequency coefficients from the DCT module and generates a quantization factor. The quantization factor is applied to the frequency coefficients before they undergo VLC encoding by the entropy encoder module. Essentially, the rate controller instructs the entropy coder how to encode values ​​by providing a q factor per MB. The entropy-encoded data is sent to a bitstream writer module 518, from which the compressed video bitstream is sent either back to the editor and then to storage 520 or directly to storage.When proxy video is needed for video editing purposes, the compressed video is pulled from storage 520 and decompressed by a decoder (not shown). The decoder can be implemented as a module in a system either inside the editor or external to the editor. The video editing application can then use the proxy video as part of its editing workflow.

[0045]

[0051] The various components of the systems described herein can be implemented as computer programs using a general-purpose computer system. Such a computer system typically includes a main unit connected to both output devices that display information to an operator and input devices that receive input from the operator. The main unit typically includes a processor connected to a memory system via an interconnection mechanism. The input and output devices are also connected to the processor and memory system via an interconnection mechanism.

[0046]

[0052] One or more output devices may also be connected to the computer system. Examples of output devices include, but are not limited to, liquid crystal displays (LCDs), plasma displays, OLED displays, various stereoscopic displays, including glasses- and glasses-free displays, cathode ray tubes, video projection systems and other video output devices, loudspeakers, headphones and other audio output devices, printers, devices for communicating over low- or high-bandwidth networks, including network interface devices, cable modems, and storage devices such as disks, tapes, and solid-state media, including flash memory. One or more input devices may also be connected to the computer system. Examples of input devices include, but are not limited to, keyboards, keypads, trackballs, mice, pens / styluses and tablets, touchscreens, cameras, communication devices, and data input devices. The present invention is not limited to the particular input or output devices used in conjunction with the computer system, nor to the devices described herein.

[0047]

[0053] A computer system may be a general-purpose computer system, programmable using a computer programming language, scripting language, or even assembly language. A computer system may also be specially programmed, special-purpose hardware. In a general-purpose computer system, the processor is typically a commercially available processor. A general-purpose computer typically has an operating system. The operating system controls the execution of other computer programs and provides scheduling, debugging, input / output control, accounting, compilation, storage allocation, data and memory management, and communication control and related services. A computer system may be connected to a local network and / or a wide-area network such as the Internet. The connected network may transfer program instructions executing on the computer, media data such as video data, still image data, or audio data, metadata, reporting and authorization information about media compositions, media annotations, and other data to and from the computer system.

[0048]

[0054] A memory system typically includes a computer-readable medium. This medium may be volatile or nonvolatile, writable or non-writable, and / or rewritable or non-rewritable. A memory system typically stores data in binary form. Such data may define an application program executed by a microprocessor or information stored on a disk and processed by the application program. The present invention is not limited to a particular memory system. Time-based media may be stored on and read from magnetic, optical, or solid-state drives. These drives may also include arrays of local or network-attached disks.

[0049]

[0055] A system as described herein may be implemented in software, hardware, firmware, or a combination of the three. Various elements of the system, either individually or in combination, may be implemented as one or more computer program products. Within a computer program product, computer program instructions are stored on a non-transitory computer-readable medium or transmitted to a computer system over a connected local or wide area network for execution by a computer. Various steps of a process may be performed by a computer executing such computer program instructions. The computer system may be a multiprocessor computer system, may include multiple computers connected over a computer network, or may be implemented in a cloud. The components described herein may be separate modules of a computer program or may be separate computer programs. These computer programs may be operable on separate computers. Data generated by these components may be stored in a memory system or transmitted between computer systems via various communication media, such as carrier wave signals.

[0050]

[0056] While exemplary embodiments have been described above, it should be apparent to those skilled in the art that the foregoing description is merely illustrative and not limiting, and has been presented by way of example only. Numerous modifications and other embodiments will be within the purview of those skilled in the art and are contemplated as falling within the scope of the invention.

Claims

1. A method for constant bit rate, intraframe compression of video containing a sequence of images, comprising, for each image: Segmenting the image into n portions, for each of the n portions compression parameters are individually adjustable; transforming each of the portions of the image to generate frequency domain coefficients for each of the portions; selecting a stored quantization matrix for the image based on a desired bit rate for the compressed video; determining an initial bit budget for each transformed portion of the image, the initial bit budget equal to the bit budget for the image within the compressed video divided by n; determining, for each transform portion, an initial scaling factor that is the smallest scaling factor, which, when applied to the frequency-domain coefficients together with the quantization matrix, results in a bit size for the transform portion that is less than or equal to the initial bit budget for the transform portion; generating a bitstream representing the compressed image, for each transform portion, adjusting the frequency domain coefficients of the portion by applying to the portion an initial scaling factor determined for that portion together with the quantization matrix; arranging the adjusted frequency domain coefficients in a predetermined sequence; and applying variable length coding to the sequence of adjusted frequency domain coefficients; generating the bitstream by A method comprising:

2. 10. The method of claim 1, further comprising: evaluating the transformed portions in order of their initial scaling factors and determining an average of the initial scaling factors; Proceeding in ascending order of initial magnifications, adjusting downward the initial magnifications of a plurality of transformed portions having initial magnifications greater than the average initial magnification, the amount of adjustment being a decreasing fraction of the difference between the initial magnification of a given transformed portion and the average initial magnification; terminating the adjustment of the initial magnification factor when adjusting the additional transform portions causes the sum of the bit sizes of the adjusted transform portions of the image to exceed the bit budget for the image within the compressed video; A method comprising:

3. 3. The method of claim 2, further comprising the step of adjusting upward the initial magnification of each portion having an initial magnification less than a threshold magnification before adjusting downward the initial magnifications of the plurality of transformed portions having initial magnifications greater than the average initial magnification, wherein the threshold magnification is based on the average initial magnification and a standard deviation of the initial magnifications.

4. 1. A computer program product comprising: a non-transitory computer readable medium having computer readable instructions encoded thereon, the computer readable instructions, when processed by a processing device, directing the processing device to perform a constant bit rate, intraframe compression method for a video comprising a sequence of images, the method comprising, for each image: Segmenting the image into n portions, for each of the n portions compression parameters are individually adjustable; transforming each of the portions of the image to generate frequency domain coefficients for each of the portions; selecting a stored quantization matrix for the image based on a desired bit rate for the compressed video; determining an initial bit budget for each transformed portion of the image, the initial bit budget equal to the bit budget for the image within the compressed video divided by n; determining, for each transform portion, an initial scaling factor that is the smallest scaling factor, which, when applied to the frequency-domain coefficients together with the quantization matrix, results in a bit size for the transform portion that is less than or equal to the initial bit budget for the transform portion; generating a bitstream representing the compressed image, for each transform portion, adjusting the frequency domain coefficients of the portion by applying to the portion the initial scaling factor determined for that portion together with the quantization matrix; arranging the adjusted frequency domain coefficients in a predetermined sequence and applying variable length coding to the sequence of adjusted frequency domain coefficients; generating the bitstream by 1. A computer program product comprising:

5. 1. A system comprising: a memory storing computer readable instructions; a processor connected to the memory; wherein the processor, when executing the computer readable instructions, causes the system to perform a constant bit rate, intraframe compression method for video comprising a sequence of images, the method comprising, for each image: Segmenting the image into n portions, for each of the n portions compression parameters are individually adjustable; transforming each of the portions of the image to generate frequency domain coefficients for each of the portions; selecting a stored quantization matrix for the image based on a desired bit rate for the compressed video; determining an initial bit budget for each transformed portion of the image, the initial bit budget equal to the bit budget for the image within the compressed video divided by n; determining, for each transform portion, an initial scaling factor that is the smallest scaling factor, which, when applied to the frequency-domain coefficients together with the quantization matrix, results in a bit size for the transform portion that is less than or equal to the initial bit budget for the transform portion; generating a bitstream representing the compressed image, for each transform portion, adjusting the frequency domain coefficients of the portion by applying to the portion the initial scaling factor determined for that portion together with the quantization matrix; arranging the adjusted frequency domain coefficients in a predetermined sequence and applying variable length coding to the sequence of adjusted frequency domain coefficients; generating the bitstream by Including, the system.

Images