Efficient transforms and transposes for rate-distortion optimization and reconstruction in video encoders

Abstract

The disclosed methods can provide a non-blocking video processing pipeline with the ability to effectively share transform hardware resources. Transform hardware resources can be shared across processing parameters (e.g., pixel block size, transform type, video stream bit depth, and / or multiple encoding formats) and for inter and intra frame encoding. The video processing pipeline can be divided into phases, each phase having a half butterfly circuit to perform a respective portion of the computation of a transform. The phases can be selectable and configurable to perform transforms for a plurality of different combinations of the processing parameters. In each configuration, a phase can be able to perform a transform by sequentially passing through at least some of the phases, resulting in a high throughput. Methods are also described that relate to improving the performance and efficiency of the transpose operation of a transform.

Description

Background Technology

[0001] The key to hardware video encoders lies in the implementation of efficient transformations between the time and frequency domains. For video codecs such as AV1 and VP9, ​​both forward and inverse transformations are essential for both Rate-Distortion Optimization (RDO) and Reconstruction (REC), where the transformation operations are separated by memory-intensive transpose operations. RDO may involve optimizing rate-distortion to improve quality by testing different potential combinations of transform types, shapes, and / or sizes used to encode superblocks. REC may involve using information from the RDO to fine-tune the selection and execution of encoding. The different transform types, shapes, and sizes that can be supported by the video codec can result in a large search space for RDO. Due to the size of the search space, it may be impossible to evaluate all possibilities using hardware throughput, which specifies how many possibilities can be evaluated for a given superblock. However, achieving high throughput while supporting many different configurations, as well as implementing area- and power-efficient circuitry, is extremely challenging.

[0002] Conventional methods for video coding can employ iterative approaches to perform transforms, where further pixel input is blocked from entering the pipeline as iterates over multiple passes of data. Real multipliers are used for transforms that are expensive in terms of area, power, and timing, thus limiting performance. Another conventional approach iterates over several small functions to compose a larger function that results in the overall transform. This can require a considerable number of iterations, and for larger transform sizes, the resulting blockage can be hundreds of cycles. Transpose can be performed using resource-intensive physical transpose circuitry. For these and other reasons, conventional methods can potentially require large and inefficient circuitry, as they do not scale well for high throughput. Summary of the Invention

[0003] Embodiments of this disclosure relate to efficient transforms and transposes for rate-distortion optimization and reconstruction in video encoders. Systems and methods are disclosed that can be used to provide efficient use of area and power for high-throughput video encoders or decoders.

[0004] In other aspects of the invention, the disclosed method provides a non-blocking video processing pipeline with the ability to efficiently share transform hardware resources. In at least one embodiment, the transform hardware resources of the video processing pipeline can be shared across processing parameters (e.g., pixel block size, transform type, video stream bit depth, and / or multiple encoding formats) and used for inter-frame and intra-frame coding. The video processing pipeline can be divided into multiple phases, each phase having multiple half-butterfly (HB) circuits, adders, and / or subtractors operable to perform corresponding portions of transform computation. Phases can be selected and configured to perform multiple transforms of different pixel block sizes, transform types, video stream bit depths, and / or multiple encoding formats, as well as for inter-frame and intra-frame coding, thereby allowing high reusability of transform hardware resources. In each configuration, phases may be able to perform transforms by sequentially traversing at least some phases, resulting in high throughput. Attached Figure Description

[0005] The following describes in detail, with reference to the accompanying drawings, the system and method for efficient transformation and transpose in a video encoder for rate-distortion optimization and reconstruction, wherein:

[0006] Figure 1 Example data streams of quantization and transform units suitable for rate-distortion optimization and reconstruction of video are shown according to some embodiments of this disclosure;

[0007] Figure 2 Examples of circuits for sharing transformation hardware resources according to some embodiments of the present disclosure are shown;

[0008] Figure 3 Examples of tables according to some embodiments of the present disclosure are shown, the tables reflecting how transform hardware resources can be mapped to the phase of the transform;

[0009] Figure 4A Examples of half-butterfly circuits that can be used to implement data flow according to some embodiments of the present disclosure are shown;

[0010] Figure 4B Example circuits for bypassing the inlet pipeline stage are shown according to some embodiments of the present disclosure;

[0011] Figure 4C Example circuits for bypassing the outlet pipeline stage are shown according to some embodiments of the present disclosure;

[0012] Figure 5 Examples of aspects of the transpose operation of an 8x8 pixel block according to some embodiments of the present invention are described;

[0013] Figure 6 Examples of spiral memory access modes for transposed pixel blocks according to at least some embodiments of the present disclosure are shown;

[0014] Figure 7 It is possible to implement at least some embodiments of this disclosure. Figure 6 A flowchart illustrating an example of the spiral memory access mode process;

[0015] Figure 8A Including at least some embodiments according to this disclosure that can be used to implement Figure 6 A flowchart illustrating an example of the spiral memory access mode process;

[0016] Figure 8B Including at least some embodiments according to this disclosure that can be used to implement Figure 6 A flowchart illustrating an example of the spiral memory access mode process;

[0017] Figure 9 The table, which includes at least some embodiments of the present disclosure, illustrates examples of how pixel blocks can be stored in the same buffer using offsets;

[0018] Figure 10 Examples are shown of how an address space according to at least some embodiments of the present disclosure can be divided into a write range, a read range, and a free range;

[0019] Figure 11 This is a flowchart illustrating a method for operating a video processing pipeline using shared hardware resources according to some embodiments of the present disclosure;

[0020] Figure 12 It is a block diagram of one or more example computing devices suitable for implementing some embodiments of this disclosure; and

[0021] Figure 13 This is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0022] Embodiments of this disclosure relate to efficient transforms and transposes for rate-distortion optimization and reconstruction in video encoders. Systems and methods are disclosed that can be used to provide efficient use of area and power for high-throughput video encoders or decoders.

[0023] Compared to conventional methods, this disclosure provides a transform pipeline that applies transforms to pixel blocks (which may also be referred to as tiles, block partitions, or sub-blocks) without blocking by using iterations on pixels or by computational resources. The throughput PPC can be equal to the input PPC. These properties can exist for various processing parameters (e.g., pixel block size, transform type, video stream bit depth, and / or multiple coding formats) and for inter-frame and intra-frame coding. Providing a non-blocking design can lead to high throughput and significantly simplify scheduling, thereby allowing entities such as inter-frame and intra-frame coding units to more easily share the same data stream 100.

[0024] In other aspects of the invention, the disclosed method provides a non-blocking video processing pipeline with the ability to efficiently share transform hardware resources. In at least one embodiment, the transform hardware resources of the video processing pipeline can be shared across processing parameters (e.g., pixel block size, transform type, video stream bit depth, and / or multiple encoding formats) and used for inter-frame and intra-frame coding. The video processing pipeline can be divided into multiple phases, each phase having multiple half-butterfly (HB) circuits, adders, and / or subtractors operable to perform corresponding portions of transform computation. Phases can be selected and configured to perform multiple transforms of different pixel block sizes, transform types, video stream bit depths, and / or multiple encoding formats, as well as for inter-frame and intra-frame coding, thereby allowing high reusability of transform hardware resources. In each configuration, phases may be able to perform transforms by sequentially traversing at least some phases, resulting in high throughput.

[0025] The disclosed pipeline can be implemented using a half-butterfly network (including multiple HBs and multiple adders), followed by a transpose and another half-butterfly network (including multiple HBs) to implement a transformation unit (e.g., a forward transform circuit, an inverse transform circuit, etc.). As used herein, a half-butterfly network may refer to a circuit that includes at least a portion of a half-butterfly circuit for performing a transformation on the coefficients. A half-butterfly or half-butterfly circuit may refer to a circuit configured to scale a linear combination of inputs (e.g., two inputs) to a single output (e.g., using a trigonometric function value) as described above. Figure 4A A circuit that combines multiples of inputs (and adds them together). As described in this paper, a semi-butterfly circuit can form smaller operations on input pairs that can be networked to produce larger transformations.

[0026] In at least one embodiment, a semi-butterfly may correspond to one of the outputs of a correlated pair in a butterfly processing flow (e.g., derived from a Fast Fourier Transform (FFT)), where the input pairs are linearly combined to produce two outputs. For example, consider the extraction of stage 5 of the FDCT16 transform in AV1, where the values ​​of cospi[8] and cospi

[40] are known trigonometric constants:

[0027]

[0028] As shown above, the outputs 4 to 7 are linear combinations of the inputs 4 to 7.

[0029] These terms can be rearranged as follows:

[0030]

[0031] In this FFT, in[4] and in[7] are linearly combined to produce outputs out[4] and out[7] independently of all other inputs. Similarly, in[5] and in[6] are combined to produce out[5] and out[6]. One of the outputs of a related pair, for example, out[5] from the pair out[5] and out[6], can refer to half of the butterfly processing flow and can be implemented as a half-butterfly circuit. The FFT in which input pairs are mixed to produce output pairs can be considered to be of the radix-2 type. Radix-2 can represent the algorithmic factor by which a larger FFT is divided and conquered into smaller FFTs. This disclosure also relates to radix formats, such as radix-4, which, in contrast, represent the range of each digit in a digital representation.

[0032] In one or more embodiments, the four high-efficiency conversion units can operate at different throughput scalings across 64 pixels-per-clock (PPC), 32 PPC, and 16 PPC. For example, the REC can operate at 32 PPC, while the RDO operates at 64 PPC for the forward conversion and 16 PPC for the inverse conversion. The varying PPC across the conversion units can be tailored to the performance needs of each unit in exchange for circuit area and efficiency.

[0033] The RDO throughput mismatch between the forward and inverse transforms can be caused by calculating the relative cost of the transform for the RDO in the frequency domain rather than the spatial domain. Therefore, the inverse transform can handle only a fraction of the transform performed in the forward domain, and the average trial latency can be significantly reduced, while the forward transform circuitry delivers higher performance. Compared to uniform throughput, the PPC mismatch allows for a considerable reduction in area and power because the inverse transform can support fewer pixel block types, such as only one transform type and size.

[0034] All transform trigonometric coefficients can be pre-encoded in sequential radix-4 format (or another numerical format). Pre-encoding allows transform circuits to be implemented without any multipliers. For example, a transform circuit can rely solely on minimum-width multiphase addition (or multi-adder) operations, where each half-butterfly uses multiphase addition. (For example, the rectangular scaling operation of AV1) is also multiplier-free, making the entire transform circuit multiplier-free.

[0035] The transform pipeline can share arithmetic logic across all supported transform types and sizes. ADST4 and IDT transforms can be mapped to HB operations while provably preserving bit precision. HB networks can phase-aggregate across the pipeline to minimize the total number of HB operations required to perform a particular transform. Resources can be shared for the same maximum throughput across all transform types and sizes. For example, four 4x4 transforms can be processed in the same clock cycles (4x4 / 64 = 1 / 4) of a 64PPC implementation on the pipeline, while a 32x32 transform can use 16 cycles per dimension (32x32 / 64 = 16).

[0036] Rounding and truncation operations for different encoding formats (such as AV1 and VP9) can be combined into a high-speed multi-adder circuit without separate processing circuitry, even when there are fractional precision differences between codecs. This provides a uniform pipeline where shared HBs, adders, and subtractors can be used to support multiple codecs for their respective bit precisions. Aspects of the invention also allow resource sharing between VP9 full-precision ADST arithmetic and the corresponding AV1 truncation arithmetic. The multi-adder circuitry may also include a variable rounding position capability that can be used to reduce the output bit width of certain transform operations. For example, the identity transform HB operation can exceed the bit width of HB operations found in other transforms without corresponding rounding.

[0037] In at least one embodiment, the inverse transform circuitry for REC can be bit-accurate for each supported encoding format transform precision (e.g., for both AV1 and VP9). Other transform circuitry (e.g., for RDO or for REC) can approximate the encoding format without preserving bit accuracy, thus allowing for savings in area, power, and complexity. The total bit precision required at each phase can also be minimized by propagating a bit-width wavefront across all codecs, transform types, and / or sizes of shared resources, resulting in the smallest bit-width hardware across all computations—leading to area and power savings.

[0038] Range clamping can be performed efficiently by sharing configurable circuitry across various possible ranges per clamp instance, thus adapting to different bit depths and / or codecs. To reduce latency, one or more pipeline stages can be skipped. One or more configuration circuitry can be configured and selected to allow resource sharing of low-power, one-hot muxes with a low number of selectable sources per mux. Sharing can be performed across a unified pipeline for multiple coding formats (e.g., both VP9 and AV1), all bit depths, all transform types, and / or all transform sizes. In this way, the disclosed methods can be easily scaled to larger transform sizes that may appear in future codecs (128x128, 256x256, and above).

[0039] Incorrect transform size sequences can lead to input blocking, albeit with less complexity using the disclosed methods. In at least one embodiment, blocking can be avoided by tuning the depth of the transpose buffer within a pipeline of sufficient throughput while processing transforms in the order of primarily increasing sizes. In at least one embodiment, a FIFO-based approach that allows fine-grained adjustment of the transpose buffer size between transforms can be used to perform the transpose operation. The transpose buffer size can be adjusted, for example, based on a factor of the maximum supported transform size. Variable spiral memory read and write patterns can be used to transpose transforms of all sizes using a mixture of interleaving, deinterleaving, and bucket shifting of transform data. All transform sizes can be processed at maximum throughput, thus ensuring efficiency matching the computational portion of the transform.

[0040] The disclosed method further allows NxN transformations to be implemented using fewer than N independent memory cells (such as logical RAM). This reduction can be achieved, for example, using time multiplexing, write-enabled, and excess bandwidth relative to the transformation size. Accordingly, the disclosed method can scale to future codecs with larger transformation sizes. In one or more embodiments, an alternation scheme of read / write modes can be used to save buffer space.

[0041] Some codecs (such as VP9) lack fault safety for heavyweight streams, which can cause transform arithmetic to overflow during decoding. The disclosed method provides protection by clamping and reducing any drift errors caused by out-of-range streams. In at least one embodiment, a hardware-based error reporting mechanism can be provided, allowing firmware to quickly retry encoding with less aggressive parameters. Overflow detection can be implemented in the transform unit without further propagating the detection problem to downstream units.

[0042] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aerial systems, intermediate systems, rowing systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems implemented using edge devices, systems incorporating one or more virtual machines (VMs), systems implemented at least partially in a data center, systems implemented at least partially using cloud computing resources, and / or other types of systems. While specific examples are provided, these examples can be generalized beyond implementation details. For example, while the disclosed methods are compatible with VP9 and AV1, other encoding formats can be implemented. Furthermore, while video encoding is primarily described, aspects of the invention can be integrated into other types of pipelines, such as video decoders and / or other pipelines employing transformations and / or transposes.

[0043] refer to Figure 1 , Figure 1 Example data stream 100 of a quantization and transformation unit suitable for rate-distortion optimization and reconstruction of video according to some embodiments of this disclosure is shown. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, circuits, functional groups, etc.) may be used in addition to or instead of the arrangements and elements shown, and some elements may be omitted entirely. Furthermore, some elements described herein are functional entities or units that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities or units can be performed by hardware, firmware, and / or software. For example, the various functions can be implemented by a processor that executes instructions stored in memory and / or one or more circuits.

[0044] Data stream 100 includes forward transform 102, quantization 104, inverse quantization 106, and inverse transform 108. An example using RDO illustrates data stream 100, but as described herein, similar data streams can be used for reconstruction. Figure 1 As shown, forward transform 102 can be performed on a block sequence or pixel block (e.g., a partition of a superblock) as residual pixels or residuals, followed by quantization 104, then inverse quantization 106, and finally inverse transform 108 to produce residual pixels for the pixel blocks. As described herein, not every pixel block can pass through all stages of data stream 100 (e.g., for RDO). However, when data stream 100 is used for intra-frame prediction in RDO, the residual pixels from inverse transform 108 can be used as feedback for some pixel blocks. Also as shown, for RDO, the outputs from quantization 104 and inverse quantization 106 can be used to calculate the bit rate and distortion of the pixel blocks used to compute RDO metrics. Similar outputs can also be used to facilitate reconstruction.

[0045] As shown in the figure, the forward transform 102 includes a forward column transform 102A, a forward transpose 102B, and a forward row transform 102C. The inverse transform 108 includes an inverse row transform 108A, an inverse transpose 108B, and an inverse column transform 108C. Similar components may be included when the data stream 100 is used for reconstruction. However, those components can be compared with... Figure 1 The PPC shown is a different PPC operation (e.g., with a lower maximum PPC, such as a constant 32 PPC throughout data stream 100). Additionally... Figure 1 The inverse transform funnel 114 is shown, which may not be used in the quantization and transform units provided for reconstruction.

[0046] Resource sharing pipeline

[0047] Data stream 100 can be implemented without blocking by using iterative rounds on pixels or iterative rounds on computational resources. Therefore, the throughput PPC can be equivalent to the input PPC. Providing a non-blocking design greatly simplifies scheduling, allowing inter-frame and intra-frame coding units to more easily share the same data stream 100, among other benefits. However, providing this non-blocking design is extremely challenging in terms of effectively sharing transform hardware resources for various processing parameters (e.g., pixel block size, transform type, video stream bit depth, and / or multiple coding formats) and for inter-frame and intra-frame coding.

[0048] According to one or more embodiments, scheduler 116 may provide data stream 100 with pixel blocks having a variety of different potential sizes, thereby producing residual pixel blocks of different shapes and sizes (e.g., square and non-square, such as rectangles of different sizes). Examples include 64x64, 64x32, 32x64, 32x32, 32x16, 16x32, 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, or 4x4, any of which can be processed by data stream 100. In at least one embodiment, scheduler 116 may represent any combination of circuitry implementing a scheduling strategy, such as typically providing data stream 100 with pixel blocks that transition from smaller transform sizes to larger transform sizes. As described herein, some exceptions to the scheduling strategy may involve slightly inserting a 16x16 transform (the size of which may vary depending on the pipeline implementation) into the RDO, such that feedback transforms make data ready on time to prevent future gaps in scheduling. As a further example, scheduler 116 can be implemented using a linear integer program.

[0049] Data stream 100 can also support various transform types, such as, by way of example rather than limitation, Discrete Cosine Transform (DCT), Identity Transform (IDT), Asymmetric Discrete Sine Transform (ADST), and / or Flipped AST. Furthermore, data stream 100 can support different potential video stream bit depths, such as, by way of example rather than limitation, 8, 10, and / or 12. Data stream 100 can also support various encoding formats, such as, by way of example rather than limitation, AO Media Video 1 (AV1) and / or Video Predictor 9 (VP9).

[0050] In at least one embodiment, each process in data stream 100 may correspond to a corresponding clock cycle. Thus, for RDO forward column transformation, data stream 100 may be nine clock cycles deep; for row transformation, data stream 100 may be nine clock cycles deep (excluding the latency of transpose units, which may vary depending on the transform size); and for reconstruction, data stream 100 may be nine clock cycles deep. As shown, data stream 110 is used to perform forward column transformation 102A, forward transpose 102B, forward row transformation 102C, quantization 104, and inverse quantization 106 with the same PPC. In this example, the PPC is 64 PPC, which may not be fast enough for each step to fully process certain pixel blocks in a single cycle, but allows for smaller and more efficient hardware designs. For example, a 4x4 pixel block may only need to be processed together with 16 PPCs in one cycle per step, but a 32x32 pixel block may need to be processed with 1024 PPCs in one cycle per step. Therefore, the processing of pixel blocks (e.g., pixel blocks corresponding to more pixels than the PPC of data stream 100) may be broken down into multiple cycles.

[0051] The forward column transformation 102A of the forward transformation 102 can perform a forward column transformation on one or more columns of a pixel block. Processing can be performed on one or more columns of a pixel block. For example, for the forward column transformation 102A, all four columns of a 4x4 pixel block can be operated on in a period. For a 32x32 pixel block, the forward column transformation 102A can be operated on a certain number of columns in a period, such that the total number of elements is less than or equal to the PPC. For example, two columns of a 32x32 pixel block can fit into a 64PPC example. In one or more embodiments, the data stream 100 is configured to operate on up to a number of columns and / or elements of a pixel block in a period that conforms to the PPC configuration. Furthermore, portions of a pixel block can be processed serially through the steps of the data stream 100 (parallel processing can be used within the steps). For example, after the first two columns of a 32x32 pixel block, the forward column transformation 102A can operate on the next two columns of the 32x32 pixel block, while the forward transpose 102B operates on the first two columns of the 32x32 pixel block, where the step of processing the entire pixel block in 64PPC uses 16 cycles.

[0052] The forward transpose 102B of forward transform 102 can perform a forward transpose on one or more columns from forward column transform 102A. For example, forward transpose 102B can use matrix transpose operations to shift one or more columns into one or more rows. Forward row transform 102C can then perform a forward transform on said one or more rows. Inverse transform 108 can be operated similarly to forward transform 102, but may utilize a different PPC (e.g., for RDO as described herein).

[0053] As described herein, data stream 100 can support a variety of different processing parameters, such as pixel block size, transform type, video stream bit depth, and / or multiple encoding formats. In at least one embodiment, one or more signals can be provided for the pixel block to configure data stream 100 for processing the pixel block. Specifically, data stream 100 can use shared hardware to support any of these different combinations of parameters, and signals can be provided to configure the hardware, such as to specify one or more pixel block dimensions, transform type, bit depth, input format (e.g., whether the input is row or column), pixel block shape, and encoding format. For example, the same transform network can be used to process the pixel block according to any combination of these different configurations.

[0054] See now Figure 2 , Figure 2 Examples of circuitry 200 for sharing transform hardware resources 204 according to some embodiments of the present disclosure are shown. Circuitry 200 includes one or more configuration circuits 202 operable to configure transform hardware resources 204 according to one or more processing parameters 206. As shown, transform hardware resources 204 include multiple phases, such as phases 208A, 208B, 208C, 208D, 208E, 208F, 208G, 208H, and 208I (which may be collectively referred to as "phase 208" or individually as "phase 208"). In at least one embodiment, a corresponding transform hardware resource 204 may be provided for each transform. For example, although only one set of phases 208 is shown, transform hardware resources 204 may be provided for both the forward transform 102 and the inverse transform 108 of both the RDO and the reconfiguration (each of which may be configured by configuration circuitry 202).

[0055] Transforms used to represent a finite sequence of data points based on wavefunctions can be represented as a sequence of stages. For example, the specification of a DCT uses ten computational stages. In at least one embodiment, each stage can be implemented as one or more half-butterfly operations and two (or more) operand additions or subtractions, even if the specification of the transform does not provide such a representation. Specifically, each half-butterfly operation may include two multiplications and an addition found in a butterfly DCT (a*b+c*d). In at least one embodiment, as described herein, half-butterflies can be implemented in multiple addition operations (comprising two multipliers and an adder) by pre-coding the trigonometric multipliers.

[0056] In at least one embodiment, configuration circuitry 202 can configure which one or more phases 208 process which one or more stages and / or portions of a stage, based on processing parameters 206. For transformations that do not require every phase 208, configuration circuitry 202 can configure transformation hardware resources 204 to pass through and / or bypass one or more phases 208. By mapping transformation stages to one or more corresponding phases of transformation hardware resources 204 based on processing parameters 206, configuration circuitry 202 can determine which portions of transformation hardware resources 204 perform corresponding computation sequences on pixel blocks, thereby allowing efficient resource sharing.

[0057] In at least one embodiment, each phase may include a semi-butterfly network that operates in parallel and is configured to perform at least a portion of a transformation on the coefficients of a pixel block (e.g., using the composition of a function to achieve an equivalent of matrix multiplication). A phase may use one or more multi-adder circuits and not a multiplier to implement at least a portion of one or more semi-butterfly operations of a phase. The multi-adder circuit may include a binary adder at the end of the semi-butterfly that propagates bits from least significant bit to most significant bit when adding a pair of numbers to produce the output of the multi-addition operation. Any suitable adder, such as a ripple adder or a look-ahead carry adder, may be used. As described herein, each multi-addition operation (in embodiments using multi-adders) may require only a single binary adder, which may occur after multiple items are recombinated into two operands. Furthermore, the semi-butterflies of phases 208 may not know where they are placed in the data stream 100, thus allowing the configuration circuitry 202 great flexibility in the order and / or sequence of assignments to a particular phase 208 when processing transformations.

[0058] In at least one embodiment, resources can be shared for the same maximum throughput across all supported transform types and sizes, all bit depths, and all encoding formats. For example, four 4x4 transforms can be processed in the same clock cycle (4x4 / 64 = 1 / 4) of a 64PPC pipeline, while a 32x32 transform might require 16 cycles per dimension (32x32 / 64 = 16). In a transform instance, the number of instantiated half-butterflies can be 2^64, with 4224 multi-adds performed at 64PPC for a full 2DDCT3232x32 (e.g., using 2^64 multi-add circuits revisited as each column or row passes through the pipeline). This is minimal for 64PPC throughput because there is at least one transform type that might require many half-butterflies, namely DCT32x32.

[0059] Configuration circuitry 202 can generally refer to any combination of circuitry capable of configuring data stream 100 and / or phase 208 (e.g., centralized and / or distributed circuitry encompassing the entire system, which may or may not be integrated with other circuitry). In at least one embodiment, data for implementing one or more aspects of the configuration can travel along with data from the top of the pipeline (e.g., as in-band packets). For example, the configuration can be published upstream of phase 208 and re-registered in each phase 208. This approach simplifies operating two different phases 208 on different transform types. Having a local copy of the configuration for each phase 208 also reduces the load that one or more configuration circuitry 202 must drive, which in turn makes timing objectives easier to meet. For example, transform size and type can be emitted along the pipeline along with coefficient data. This allows two or more different transform types to flow through the pipeline simultaneously. In at least one embodiment, one or more configuration aspects can be broadcast from a central configuration unit. For example, codec or encoding format selection (e.g., codec or encoding format selection for AV1 or VP9) can be sent directly to the transpose circuitry without tracing the data through the pipeline. This method can be used to avoid passing configuration data across buffers (e.g., FIFO buffers) found in the transpose.

[0060] See now Figure 3 , Figure 3 Examples of Table 300 according to some embodiments of the present disclosure are shown, the table reflecting how transform hardware resources can be mapped to the phase of the transform. In at least one embodiment, configuration circuitry 202 may be configured to map transform hardware resources 204 to phase 208 according to Table 300. By way of example and not limitation, Table 300 is used for 1-D DCT transforms. However, similar mappings may be provided for other transforms.

[0061] exist Figure 3 In this context, N can refer to one or more dimensions of the pixel block to be processed using transformation hardware resource 204. Therefore, column 302 reflects the mapping information of the 64x64 pixel block. Column 304 provides different aspects of the phase decomposition information for different pixel block dimensions. For example, phase breakdown information for phase and pixel block size includes the level mapped to the phase. Thus, stage 2 of the 1-D DCT is mapped to phase 208A of the 64x64 pixel block. For other pixel block sizes, phase 208A can be used as a path, such as... Figure 3 As shown. Furthermore, for some phases, the phase can be divided into multiple phases 208. For example, as shown in column 306, phase 4 can be provided on phases 208F and 208G of the 1-D DCT 16x16.

[0062] The phase breakdown information for both phase and pixel block size also includes the calculated r / c, which indicates the number of rows or columns calculated by phase 208. For example, all phases 208 for a 64x64 pixel block are indicated in column 302 as processing one row or one column. In the phase breakdown information (±, hb) for each r / c, ± represents the number of adders for each phase, and hb represents the number of multipliers implemented in half-butterfly per phase per row or column. Thus, phase 208D for a 64x64 pixel block may include 44 adders / subtractors (e.g., two-variable) and 10 half-butterflies (e.g., every two multipliers and adders or multiple adders), as indicated in column 302. Similarly, in the phase breakdown information (±, hb) for all r / c, ± represents the total number of adders / subtractors for each phase, and hb represents the total number of multipliers implemented in half-butterfly per phase for all rows or columns. All additions within the semi-butterfly can be additions, where any negative is handled by a negative constant coefficient. In at least one embodiment, the combination of addition and subtraction is not performed dynamically. Addition operations can be shared only with other addition operations and subtraction operations can be shared with other subtraction operations without dynamically selecting resources for addition versus subtraction. Because adders and subtractors are dedicated to one type of operation, this approach can lead to smaller, faster, and more power-efficient hardware. This can also be achieved without additional adder or subtractor resources due to possible mismatches on a given phase 208.

[0063] Column 308 indicates the maximum number of adders and multipliers required for each phase, and column 310 indicates the increments relative to the maximum value of N=64 for DCT64. As indicated, phase 208F may require a limited number of additional multipliers, thus indicating efficient resource sharing. Although mappings are shown for different dimensions, not all transform sizes need to be implemented in data flow 100. For example, in at least one embodiment, N=64 may not be used, where the maximum transform size is N=32. In such an example, the increments relative to the maximum value of N=32 would be 0.

[0064] In at least one embodiment, configuration circuitry 202 can configure the inputs and outputs of phase 208 to process a specific transformation using a multiplexer between the inputs and outputs of a half-butterfly. Phase breakdown information also includes column 312 indicating the number of multiplexer inputs required for configuration circuitry 202 to implement the phase. As indicated, the multiplexed input count is minimal, indicating the number of transformations HB performs in the worst-case scenario. However, other half-butterflies can be attributed to having fewer transformations and requiring less multiplexing. For different phases, no multiplexer is needed, advantageously providing unmultiplexed adder terms for the adder.

[0065] In at least one embodiment, the multiplexer configured by the configuration circuit 202 for configuring the transformation hardware resource 204 can be implemented using hot selection, thereby providing a low-power design. All multiplexing can be implemented using direct AND-OR expressions for high-speed multiplexing with single hot selection. In addition to speed, this multiplexer can produce 0 as a default value when conditions are not met, and 0 can be a subtractor operation for multiplication and an identity operator for addition, making it an ideal default value for this embodiment. This multiplexing structure also has low dependence on the selection of the total two natural powers (2x1, 4x1, etc.), and is effective even when the natural powers are not numbered correctly.

[0066] ADST transformation hardware resource sharing

[0067] The AV1 and VP9 codec standards do not specify ADST4 and IDT for half-butterfly operations, making their hardware unshareable with other transforms. The disclosed method provides a mapping of ADST4 and IDT to half-butterfly or two-operand addition / subtraction operations while maintaining bit accuracy, thus allowing those transforms to directly share hardware with other transforms, thereby avoiding separate circuit paths used only for those transforms.

[0068] Mapping ADST presents several challenges, illustrated by ADST4. One issue is that ADST4 has more than four outputs at certain computational stages. Other transformations of size N, such as DCT-N and ADST-N, have exactly N outputs. This complicates flip-flop management and makes hardware sharing more difficult. Another problem is that all half-butterfly operations can be rounded and truncated to preserve the bit width of the result under inspection. However, the ADST4 transformation preserves the full width of its multiplied result until the final output stage (similar to VP9 ADST transformations of other sizes, but not for AV1). Furthermore, in ADST4, addition and subtraction operations can have three operands. All other transformations can have two operands for addition and subtraction, and the adder has a much wider bit width due to the previous issues. Additionally, the difference between the forward and inverse transformations is often a stage reversal. This is not true for ADST4, and the early stages are multiplied for ADST4, requiring a different design for ADST4. Moreover, line rates are difficult to achieve due to the high multiplier cost that cannot be shared with hardware used for other transformations.

[0069] To avoid the various problems associated with ADST, the general concept of ADST can be reformulated and restructured so that it can be expressed in a manner similar to other transformations, thus allowing resource sharing. For example, forward ADST4 (FADST4) can be represented using a semi-butterfly and two operand adders / subtractors.

[0070] Table 1 below highlights the steps involved in refactoring FADST4:

[0071]

[0072]

[0073] The table above expresses FADST4 in the form of a half-butterfly and two operand adders. The original VP9 / AV1 code carries full precision throughout the computation and only rounds and truncates after the final addition. However, a close approximation is sufficient by carrying rounding, truncating the half-butterfly, and clamping the adder at the end, since the precision difference may be small.

[0074] Table 2 below highlights the FADST implementation with a half-butterfly and two operand adders:

[0075]

[0076]

[0077] In at least one embodiment, all eight half-butterflies can be in one stage, rather than two half-butterflies in two stages, to reduce the number of additional outputs as quickly as possible, assuming the second stage has eight outputs instead of four. This also reduces the number of additional adders required for the entire process.

[0078] Representing the inverse ADST4 with a half-butterfly and two operand adders / subtractors may be more complex in order to match the bit precision in the reconstruction.

[0079] Table 3 below highlights the steps involved in refactoring IADST4:

[0080]

[0081]

[0082] Table 4 below highlights the IADST implementation with a half-butterfly and two operand adders:

[0083]

[0084]

[0085] Full-precision half-butterflies can be shared with rounding and truncation half-butterfly operations. The full-precision output can be computed before truncation and rounding. Special versions of truncated and rounded half-butterflies can be implemented, producing a full-precision output as a secondary output. Furthermore, a single input rounding mode selection bit can be provided at low cost. Utilizing these added features, half-butterflies can be implemented if IADST4 can be shared with those of other transforms.

[0086] Pre-calculated trigonometric coefficients

[0087] Implementing multiplication using multipliers is typically hardware-intensive. The disclosed method allows for a 100% data flow multiplier-free implementation. For example, multipliers may not be needed to perform both rectangular scaling and half-butterfly transformations. For rectangular scaling, the operation can be reduced to multiple additions. Half-butterfly can also be implemented as multiple addition operations (e.g., one per half-butterfly), and two multiplications and additions can be simulated without using multipliers. For half-butterfly, conventionally, to apply a specific transformation, a real multiplier is needed to encode the trigonometric coefficients of the multiplicands in a base-4, base-8, or other numeric format to tabulate the addition coefficients of the transformation. In other problems, this process requires slow and large XOR gates.

[0088] like Figure 1 As indicated in the diagram, and according to aspects of the invention, the pre-coded trigonometric coefficients 120 of the transformation can be provided, for example, in radix-4 format. While radix-4 format is described, other transformation formats may be suitable, such as radix-8 or Booth format. For example, the signed radix-4 format can represent the numbers {0, 1, 2, 3} as {-2, -1, 0, 1, 2}. Instead of performing full-scale multiplication within the half-butterfly, pre-coded trigonometric coefficients 120 (e.g., pre-coded signed radix-4 trigonometric coefficients 120) can be used to generate the multiplication terms of the half-butterfly operation, so that only high-speed addition is required to complete the operation (e.g., in a single cycle). Accordingly, the half-butterfly operation may require only a single binary adder at the end to produce the output. Furthermore, in contrast to parallel encoding, serial radix-4 encoding can be used to save one bit width in the operation. In particular, since cosine / sine has an even number of bits in AV1 / VP9, one bit width may be lost. In other cases, an additional item may be lost (e.g., while still using serial encoding).

[0089] In at least one embodiment, the pre-encoded trigonometric coefficients 120 can be stored on-chip using any suitable method (on-chip or externally computed), and the appropriate set of coefficients for the transformation can be selected by the configuration circuitry 202 according to the processing parameters 206. As an example, and not a limitation, for AV1 and VP9 implementations, the 12-bit AV1 and 14-bit VP9 cosine and sine tables can be pre-converted to base-4. When the one-dimensional transformation supports both AV1 and VP9, ​​a 14-bit representation of AV1 can be used instead of a 12-bit representation, allowing AV1 and VP9 to share the same half of the butterfly logic depending on the codec in use.

[0090] Excluding constants, 12-bit precision trigonometric transformation results in 20 bits of 7-bit base-4 (six 3-bit values ​​(-2, -1, 0, +1, +2) and one 2-bit leading number (-1, 0, +1)), while 14-bit precision requires 23 bits of 8-bit base-4 (six 3-bit values ​​and one 2-bit leading number). Table 5 below provides some examples of cosine expressed in base-4:

[0091]

[0092]

[0093] Several transformation types can share the same half of the butterfly circuit. This can be achieved, for example, by configuring circuit 202 to multiplex which coefficients are currently in use, since only one type of transformation can be performed on a specific clock cycle.

[0094] Ignoring constants and rounding terms, each half-butterfly (X*cosA + Y*cos B) in the AV112-bit triangulation requires up to 14 adder terms: 7 for the left operand (X*cosA) and 7 for the right operand (Y*cos B). For the 14-bit VP9 triangulation, the number of adder terms is 16 (not including constants and rounding terms). Each radix-4 bit represents an adder term, but it can be observed that each radix-4 coefficient term that is 0 represents a smaller term to be added. Therefore, in at least one embodiment, the number of zero radix-4 coefficients can be maximized in the half-butterfly to improve timing while reducing area and power. In embodiments where half-butterfly operations are shared across several transformation types, adder terms can be eliminated by configuring circuitry 202 to multiplex triangulation coefficients that all have 0 coefficients at similar locations.

[0095] Table 6 shows examples of additional items with matching zero:

[0096]

[0097] In the example in Table 6, the 4^5 coefficients have matching 0s of 4017, 3996, and 3973. Therefore, all three inputs can be multiplexed together in a shared half-butterfly, resulting in one less adder term in the shared multiplier.

[0098] Reduced PPC for intra-frame prediction feedback

[0099] Conventionally, after quantization and inverse quantization, the quantization and transform unit can convert the data from the frequency domain back to the spatial or time domain. The data transformed to the spatial domain can then be used to perform RDO calculations, which can also be used as feedback for intra-frame prediction. Thus, an inverse transform can be performed on the data to compute the RDO. To maintain the pipeline's PPC and simplify the design, the same hardware design can be used to perform the inverse transform with the same PPC as the forward transform.

[0100] The disclosed method can perform RDO calculations (e.g., calculating one or more RDO costs for a pixel block) using data still in the frequency domain. Therefore, it is not necessary to perform an inverse transform on the data to calculate the RDO. Furthermore, only a subset of the pixel blocks configured for operation in the forward transform 102 can be used to provide feedback for intra-frame prediction (e.g., for calculating approximate neighbors). Moreover, the pixel blocks used to provide feedback for intra-frame prediction can be smaller than the largest pixel block configured for operation in the forward transform 102. For example, the inverse transform funnel 114 can filter out pixel blocks not used to provide feedback. Therefore, the inverse transform 108 can have lower PPC operations than the forward transform 102, quantization 104, and inverse quantization 106, while other data can be fed into the data stream 100. Thus, the inverse transform 108 can be implemented using a smaller and lower-power design with reduced average latency.

[0101] In at least one embodiment, scheduler 116 may be configured to provide data stream 100 with pixel blocks that will not be blocked or stalled by the slower circuitry of inverse transform 108. This may include scheduler 116 selecting pixel blocks that do not require operation with inverse transform 108 (e.g., not for feedback), and / or the interval between pixel blocks that require operation with inverse transform 108 should at least take into account multiple cycles of the difference in PPC as inverse transform funnel 114 (e.g., having a gap corresponding to the processing speed ratio between portions of data stream 100, in this example, the gap may be approximately 1 / 4).

[0102] The inverse transform funnel 114 can use various possible criteria to select a subset of pixel blocks. In at least one embodiment, the selection criteria can be based on at least one or more dimensions of the pixel blocks. For example, the inverse transform funnel 114 can be configured to select pixel blocks smaller than the largest pixel block configured for operation by the forward transform 102. In at least one embodiment, the inverse transform funnel 114 can allow operation only on pixel blocks of a specific size and / or dimension using the inverse transform 108. By way of example and not limitation, the inverse transform funnel 114 can allow operation only on 16x16 pixel blocks using the inverse transform 108 (e.g., supporting only one transform type and size). Using the example of 16x16 pixel blocks, the boundary pixels of the pixel block may have sufficient quality to provide feedback for intra-frame prediction for subsequent transforms (which may be for different transform sizes) without requiring the inverse transform 108 to be performed on the largest pixel block.

[0103] Furthermore, scheduler 116 can maintain flexibility in scheduling remaining pixel blocks by only spacing the 16×16 pixel blocks to be operated on using inverse transform 108 at least a threshold number of clock cycles to avoid stopping. For example, as long as scheduler 116 of intra-frame coding units ensures that pixel blocks to be operated on using inverse transform 108 (e.g., corresponding to subsets used to determine the upper and left boundaries of another pixel block) are sufficiently spaced to avoid blocking, scheduler 116 of inter-frame coding units can provide pixel blocks to data stream 100 when available without considering blocking. In at least one embodiment, scheduler 116 for each unit can provide blocks to the pipeline at least based on a small-to-large pixel block size order (e.g., all 4x4 followed by all 8x4, 4x8, 8x8, etc.), where some out-of-order in the 16×16s is used for the application of inverse transform 108. Thus, these units can share the pipeline without requiring a separate pipeline for each unit.

[0104] The inverse transformation funnel 114 can be implemented using circuitry that converts a large PPC input into a smaller PPC output. In at least one embodiment, a first-in-first-out (FIFO) buffer is used to implement the inverse transformation funnel 114. Figure 1 In the example, the FIFO buffer may include three 64PPC entries, each with four rows, and output flip-flops for a single 16PPC row. When the circuit's output is toggled, there may be a first-in, first-out cycle wait time. When the inverted converter 108A is implemented using a half-butterfly on the first cycle, the toggled output may be desirable.

[0105] Variable clamping and precision of adder output

[0106] Transformations can be implemented using advancements in continuous networks, such as semi-butterfly networks. Each successive network can result in a bit gain due to the inclusion of addition and multiplication operations. For example, each time a number is added, bits are gained, and multiplication can produce even more bit gains. While it might be desirable to drive a data stream 100 steps with infinite precision for practical purposes (such as limiting the size of circuit designs), the number of bits used to represent the inputs and outputs of the computation can be reduced. Therefore, clamping and rounding can be used to limit precision and reduce bit growth.

[0107] To produce a compliant video stream from a data stream, the video stream must be decoded identically at any destination. To achieve this, the inverse transform design must be computationally bit-accurate to the codec specification employed. Support for bit depths of 8, 10, or 12 may be desirable, each requiring different clamping ranges across computation. These ranges may differ between different coding formats (e.g., AV1 and VP9), thus allowing up to six clamping cases in a given operation. The clamping range itself can also vary at different points in the process. Precision requirements can also differ between coding formats. For example, some VP9 computations require up to 37 bits of full precision. Furthermore, for the ADST transform, full VP9 precision needs to be carried across multiple steps.

[0108] To accommodate different bit depths and encoding formats, conventional methods perform hardware arithmetic on the largest possible representation across all transforms throughout the transform pipeline. Using separate shift and round operations requiring additional computational steps and supplementary binary additions, the different precision used between VP9 and AV1 can be inefficiently interpreted. Bit depth clamping to different ranges can be performed directly on a case-by-case basis. Consequently, the resulting design may be detrimental to scaling, high-speed timing, region friendliness, and short latency.

[0109] Compared to conventional methods, the disclosed method provides variable bit width, precision, and rounding throughout the data stream 100, which can significantly reduce circuit size and power consumption. While conventional methods use the maximum possible operator size throughout the pipeline, the disclosed method grows and propagates with each bit pair for each transformation, requiring only the maximum size of a particular transformation shared at its location in the pipeline (i.e., the minimum bit width required for the supported transformation). The encoding format is typically specified using integer arithmetic. According to aspects of the invention, clamping can be used after performing multiplication on integer inputs using fixed-point precision arithmetic, thereby allowing a reduction of at least some fractional precision. Clamping may be accompanied by truncation to discard bits, and width rounding, to maintain possible precision when discarding bits.

[0110] Range clamping can be efficiently performed using configurable clamping circuitry (e.g., pervasive bounding), where each clamping instance shares multiple possible ranges (e.g., all required ranges, such as using AV1 and VP9 for a 3-bit depth of 6), thus accommodating different bit depths and codecs. Using the disclosed method, the total bit precision required at each stage can be minimized by propagating the wavefront of the bit width across all codecs, transform types, and sizes of shared resources, thereby resulting in hardware with the minimum bit width in each computation.

[0111] Furthermore, rounding and truncation for the supported encoding formats can be combined into a high-speed, multi-addition operation in a half-butterfly shape without separate steps, while taking into account the fractional precision differences between codecs. This provides a uniform pipeline where shared half-butterflies, adders, and subtractors can be used to support each codec for its corresponding bit precision. Resources can be shared between VP9 full-precision ADST arithmetic and the corresponding AV1 truncation arithmetic. The multi-addition circuitry can include variable rounding positions, which can be used to reduce the output bit width of the unit transform operation. Otherwise, the unit transform half-butterfly operation may exceed the bit width of half-butterfly operations found in other transforms. In at least one embodiment, only one of the four transform units is bit-accurate for both AV1 and VP9 transform precision (reconstructed inverse transform 108). Any of the three other transforms can approximate VP9 and AV1, thus allowing for savings in area, power, and complexity.

[0112] During the transformation, clamps can be applied at the output of the adder to maintain bit growth for checking as the transformation proceeds. These clamps can be called "adder clamps," which, in addition to clamping, can also round and truncate. Furthermore, at the end of each ID transformation, there can be circuitry that removes precision by rounding, shifting, and then clamping. The clamps on these circuits can be called "round-shift clamps."

[0113] Rectangular and isotopic transformations involve similar scaling operations to √2, and the codec specifies them as separate scaling operations, each with its own rounding and truncation steps. Due to the nature of rounding and truncation, each operation can be implemented separately for bit accuracy, which would equate to a very large number of scaling factors, and the area cost incurred in supporting these features would be high. The disclosed method can reuse existing resources, even if only for cases where bit accuracy is not required.

[0114] Where bit precision is required, scaling can be performed in multiple explicit steps. Using the disclosed method, for cases where bit precision is not required, the steps can be combined into a single multiplication operation, thereby reducing hardware requirements. A constant on the input side of the inverse transform, passing through (1) and the output side of the forward transform. or Multiplexers may exist between multiplications in rectangular scaling (when the multiplier is...). (There may be no bit growth at that time). Because Therefore, constant multipliers may not have bit growth. Finally, the output stage, scaled from powers of two, is a simple variable-length left shift operation with five possible choices.

[0115] At a throughput of 64 PPC, the number of additional multipliers required for rectangular scaling can be on the order of 64. All other multiplication resources (semi-butterflies) are available for use. Consider the case where both 1-D transformations during a 32x16 rectangular transformation are DCTs. Most existing semi-butterflies can be used for DCT computations, leaving very few multiplication resources for rectangular scaling, and any such resources would be very difficult to share. Therefore, in other published approaches, the pipeline stage could be supplemented with one of 64 multipliers for 64 PPC or 32 multipliers for 32 PPC. Alternatively, these scalars can be incorporated into existing pipeline stages if the clock timing has sufficient slack. To improve area and timing, these additional multipliers can have constant values. or One variable. Therefore, for the purpose of rectangular scaling, an additional constant multiplier can be added. In at least one embodiment, the constant multiplier can be implemented as multiple additions and can use a different stage than that used for the semi-butterfly.

[0116] For an IDT, using variable rounding positions allows the transformation to be performed to be identified in the fractional part of the circuit to limit the number of integer bits on the output. For example, a quarter rounding position can be used to reduce the need for a wider, half-butterfly multi-adder circuit. In at least one embodiment, instead of rounding at the first point after the binary point, an eighth point can be used to save 2 bits of multiplier width at the end of the IDT. This will be sufficient to avoid even the integer bit growth of IDT64, which can be the largest such multiplier, meaning that IDT64 can be sufficient to handle all cases.

[0117] See now Figure 4A , Figure 4A Examples of half-butterfly circuits 400A and 400B, according to some embodiments of the present disclosure, that can be used to implement data stream 100 are shown. Half-butterfly circuit 400A can be used where bit precision is not required. By way of example, and not limitation, half-butterfly circuit 400A can be used to implement forward transform 102 and inverse transform 108 for RDO, and to implement forward transform 102 for reconstruction. Half-butterfly circuit 400B can be used where bit precision is required. By way of example, and not limitation, half-butterfly circuit 400B can be used to implement inverse transform 108 for reconstruction. Figure 4AAs shown, the bit precision of IADST4 can imply that full precision must be performed from the half-butterfly circuit to accommodate the additional outputs that hold all fractional bits for multiplication-addition (multi-addition) operations. Full precision can also imply no rounding on the output. The bit precision of IDCT16 can imply that the half-butterfly circuit must use a different subset of output bits and different rounding positions than a regular half-butterfly circuit.

[0118] Overflow handling and error resilience

[0119] Some codecs (such as VP9) lack failsafe for heavyweight streams, which can cause transform arithmetic to overflow during decoding (violating bitwidth rules) and resulting in highly visible image corruption. Without intervention, some hardware overflows cause large errors due to the wraparound from positive to negative values, and vice versa. Conventionally, some overflows can be detected by designing circuitry to produce large errors on the final output signal, where this is attributed to the corruption of corresponding pixels. The disclosed method provides overflow protection by clamping and reducing any drift errors caused by out-of-range streams, thereby minimizing errors on the signal and resulting in a cleaner image. An efficient hardware error reporting mechanism can be implemented using interrupts, allowing firmware to quickly retry the codec's encoding with less aggressive parameters. Alternatively, formulas can be implemented to limit overflow detection to the transform unit without propagating the detection problem further to downstream units where it would otherwise occur.

[0120] In at least one embodiment, when any adder clamp of the transform unit applies clamping, a reported adder clamp overflow signal may exist. Furthermore, at the end of each ID transform, there may be circuitry that removes precision by rounding, shifting, and then clamping. The clamps on these circuits may be referred to as “round-shift clamps.” When a round-shift clamp is applied, a round-shift clamp overflow signal may be reported. Each ID transform may contribute both adder clamps and round-shift clamp overflow signals. For example, an RDO may include four 1D transforms, and thus may generate four adder clamp overflows and four round-shift clamp overflow signals, for a total of eight overflow signals across the RDO. Some of these overflow signals may be expected, and therefore these signals may only be used to report negligible or masked states. However, there are cases where overflows are never expected to occur on a legitimate video bitstream, and in such cases, the circuitry may be configured to generate an interrupt.

[0121] In each 1D transform, there may be as many adder clamp overflow sources as there are adders with clamps, so when PPC is large, it can be hundreds. For round-shift clamp overflow signals, there may be as many PPCs as there are PPCs (e.g., 64, 32, or 16, depending on the transform). In at least one embodiment, an interrupt can only be declared if the hardware behaves unexpectedly and is therefore undefined. For AV1, the clamp is directly specified in the protocol, and therefore clamps that occur may not cause an interrupt. For VP9, ​​there is no clamp directly specified in the protocol; only unexpected behavior in the marking design is checked. Except for IDCT reconstruction, AV1 can be used to calculate approximate all VP9 1D transforms. Accordingly, only the inverse transform 108A and inverse transform 108C are used to reconstruct a real VP9 1D transform that can be implemented. Accordingly, these units can generate interrupts on the adder clamp circuit and the round-shift clamp circuit while VP9 is being processed. Other overflows can be filtered out.

[0122] The VP9 codec specifies that for a pixel stream with a bit depth of 8, the addition stage of the inverse transform (reconstructed IDCT) should not exceed 16 bits; for a pixel stream with a bit depth of 10, it should not exceed 18 bits; and for a pixel stream with a bit depth of 12, it should not exceed 20 bits. This also applies to the inverse quantization output (IQ). The inverse transform unit can detect such bit width out-of-bounds errors and report them via an interrupt. In at least one embodiment, overflow detection can be implemented based on detecting predefined conditions. For example, at the end of each 1D transform pipeline, there is a round-shift operation for both VP9 and AV1 to reduce precision to a more manageable number of bits. The round-shift at the end of the reconstructed inverse transform can be used to trigger an interrupt. In at least one embodiment, overflow can be detected and an interrupt can be triggered for a specific combination of the overflow check bit width C, the number of bits S to which the round-shift is applied, and / or the input pixel value before the round-shift. In at least one embodiment, the specific C, S combination can be defined by one or more ranges of incoming pixel values ​​that trigger an interrupt (if any). Status bits can be used to determine which ID conversion circuit is the source of an unexpected interruption and to apply appropriate remedial action. The same checks on C and S limit overflow checks to the conversion block itself, without imposing the burden of additional overflow checks that VP9 might require on downstream blocks.

[0123] Skip Pipeline Stage

[0124] The latency of the transform and quantization units can play a significant role in reconstruction performance. In at least one embodiment, to improve transform performance, latency can be improved by skipping transform phases when the data stream 100 processes smaller transforms. For example, one or more ingress pipeline phases can be bypassed or traversed. In at least one embodiment, the scheduler 116 and / or the circuit configuration indicating individual ready and valid signals according to the data stream can cause the circuitry to wait until one or more initial phases are empty, and then directly provide the pixel block to the processing that accommodates the pixel block without blocking the required remaining one or more phases (e.g., the minimum number of phases). Furthermore, in at least one embodiment, one or more exit pipeline stages can be bypassed or traversed. In at least one embodiment, after the pixel block has been fully processed, the scheduler 116 and / or the circuit configuration can cause the circuitry to wait until one or more remaining phases are empty, and then bypass any remaining phases to clear the current phase.

[0125] Figure 4B An example circuit 410 for bypassing the inlet pipeline stage is shown according to some embodiments of the present disclosure. Figure 4C An exemplary circuit 412 for bypassing the outlet pipeline stage is shown according to some embodiments of the present disclosure. Figure 4B and 4C In this configuration, multiplexers 414, 416, and 418 may each be multi-bit in the data path. In at least one embodiment, bypassing can be used to save power (e.g., reduce trigger power consumption).

[0126] like Figure 4B As shown, if the incoming 1D DAT size is 4, circuit 410 can wait until phases A to D are empty, then proceed to phase E while bypassing phases A to D. If the incoming 1D DAT size is 8, 16, or 32, circuit 410 can wait until phases A and B are empty, then proceed to phase C while bypassing phases A and B. If the input 1D DAT size is 64, circuit 410 can push data into phase A without bypassing any phases.

[0127] like Figure 4C As shown, if the input 1D DAT is of magnitude 4 at phase G, circuit 410 can wait until phases H and J are empty, and then bypass them. Otherwise, circuit 410 can allow the data to pass through phases G, H, and J in sequence.

[0128] Transpose unit buffer

[0129] A transpose unit can be provided to perform the transpose operation on data stream 100 to account for variable latency with respect to the transform size. For example, an inappropriate sequence of transform sizes can cause input blocking. However, blocking can be avoided if the depth of the transpose buffer placed in the pipeline has sufficient throughput. Moreover, the scheduler 116 provides transforms for processing in a generally increasing order of size to mitigate blocking.

[0130] A forward transpose can take a set of column inputs and produce row outputs (and vice versa for a reverse transpose), thus allowing recovery from one 1d operation to the next. Figure 5 An example illustrating aspects of the transpose operation 500 of an 8x8 pixel block according to some embodiments of the present invention is provided. These columns are shown arriving over eight periods: times t0, t1, t2, ..., t7, where pixels P00 to P07 represent the first column arriving at time t0. They are then read over eight periods: times u0, u1, ..., u7, where the first row from P00 to P70 is read at u0. For simplicity, column and row coordinates (Pxy) can be used in the discussion instead of the number of pixels in a linear order of 0, 1, ..., 63.

[0131] When sending columns in each cycle, the first row may only be processed after the last column has arrived. Assume an NxN matrix Q follows P (an NxN pixel block transposed from column to row) into the transpose operation unit. Assume there is no memory duplication, only N... 2 A number of storage pixels are used to accommodate the transpose operation, and it is assumed that Q follows the same fixed memory access pattern maintained by P. Then, the first column of Q must wait until the last row of P is read before data from Q can enter the transpose buffer. The aforementioned indication allows at least N cycles to be spent after the first column arrives and before a row can be published, since the column arrival rate can be at most one column per cycle, corresponding to the minimum waiting time from the first column to the first row output. Furthermore, unless memory duplication exists, the transpose design can provide at least N pause cycles for the next block while the first block is being read.

[0132] One way to remedy this type of stopping is to copy the storage in a ping-pong buffer. In a traditional ping-pong buffer, half of the buffer is written to while the other half is read from, and vice versa in the next pass. Ping-pong buffers can mitigate stopping when managing the order of pixel block sizes. However, for NxN blocks (N columns written, then N rows read), the latency can remain 2N, which may be sufficient but can be improved, as described in this paper.

[0133] The primary factor indicative of the size of the transpose buffer can be the maximum pixel block size tolerated in the pipeline. When given an example, 64x64 will be discussed as the maximum pixel block size because it is the hardest case for AV1. However, any suitable maximum size can be used, such as 32x32, which can be inferred from the 64x64 case described herein.

[0134] The goal could be to process the transpose of a 64x64 pixel block in column-by-column order per clock cycle, with the output of one row per cycle giving a throughput of 1. This can be achieved using large arrays of flip-flops and multiplexers. However, RAM (e.g., based on latches and flip-flops) can also be used for the transpose operation. For example, if each pixel is 20 bits, then a ping-pong buffer would imply 2 x 64 x 64 x 20 = 163,840 bits. This would represent a significant flip-flop count, and therefore RAM can be used in some embodiments. In at least one embodiment, at least N RAMs can be used for an N x N pixel block. While RAM is used in the examples below, it can be generalized to memory or memory cells.

[0135] See now Figure 6 , Figure 6 An example of a spiral memory access pattern for transposed pixel blocks according to at least some embodiments of the present disclosure is shown. This example is provided for an 8x8 transpose and is applicable to RAM access patterns with 8 RAMs, but can be generalized to different pixel block sizes and memory types. An array of 8 RAMs is shown, totaling 8x8 = 64 entries, each RAM having the same address depth or space. Furthermore, this example attempts to organize write data such that reading all RAMs at address 0 at time u0 provides row 0, reading all RAMs at address 1 at time u1 provides row 1, and so on, until reading all RAMs at address 7 at time u7 provides row 7. This property can be used in flexible storage methods as described herein.

[0136] Figure 600 shows the arrival times of pixel columns p00-p77 to memory from time t0 to time t7, which can correspond to the use of Figure 6 The memory access modes are shown. Figure 602 shows the write mode for pixel p00-p07 at time t0. Figure 6 The diagram shows memory addresses A0-A7 and memory cells or RAMs, RAM0-RAM7. At time t0, P00 is written to address A0 of RAM0 because it is in row 0. Since P01 is in row 1, it is written to address A1, but RAM1 is used as the write port, and RAM0 may already be busy at address A0, and so on, until P07 is written to address A7 of RAM7 because it is in row 7. Figure 6The write pattern for pixels p10-p17 at time t1 is shown. Here, pixel P10 is stored at address A0 because it is in row 0, but in RAM1 because RAM0 might not be available at address A0 (since it can store P00). After this write pattern, the same pixel and address order is used, but RAM writes increment for each column, resulting in a significant reduction in the total memory usage after time t7. Figure 6 . Figure 6 The time intervals u0-u7 are also shown. Reading all RAM at address A0 will give row 0 at time u0, reading all RAM at address A1 will give row 1 at time u1, but the periodic shift is 1, and so on, up to address A7.

[0137] In at least one embodiment, a bucket shifter can be used to achieve this. Figure 6 The memory signpost or spiral filling pattern is used to adjust for circular shifts. See now. Figure 7 , Figure 7 It is possible to implement at least some embodiments of this disclosure. Figure 6 A flowchart illustrating an example of a spiral memory access mode process 700 is provided. Process 700 may receive column COL at time t and may perform a bucket shift 702 on that column. In an 8x8 pixel block example, a bucket shift 8 may be performed. Write data WD may be performed at time t to provide concatenated data to RAM 704 (e.g., eight dual-port RAMs in this example). WA(m, t) represents the RAM m write address provided to RAM 704 at time t, and RA(m, u) represents the RAM m read address provided to RAM 704 at time u. Read data RD may be performed at time u to provide concatenated data to RAM 704. Process 700 may use a bucket shift 706 (e.g., a bucket shift 8) to provide row ROW at time u.

[0138] Here, all eight RAMs have an address depth of 8 to transpose an 8x8. In at least one embodiment, the buffer can be at least doubled to remedy the stop for the ping-pong buffer. However, an alternative is to at least double the address space (from 8 to 16 in this example). For dual-port RAM, read and write operations can be issued in the same cycle. Using regular ping-pong, the bottom of the address space (0 to 7) can then be used for writing, while the top portion of the address space can be used for reading (8 to 15), and vice versa as the next 8x8 pixel block arrives. This ensures that there are no read-write conflicts at the same address, and that the most significant bits of the write and read addresses are always opposite to each other. Thus, even after adding duplicate buffers, the number of RAMs can remain the same. While in this example, conflicts between the upper and lower halves of the address space can be avoided by using a first MSB for the read address and a second MSB for the write address, this is not intended to be limiting, and many other methods and configurations can be used to avoid such conflicts between portions of the address space.

[0139] Reduce RAM count and area

[0140] The method discussed using the 8x8 pixel block example would likely require at least 64 separate RAM blocks for a 64x64 pixel block. To reduce the RAM requirement, the operational constraints / assumptions of this method can be relaxed. First, potential solutions utilizing additional memory are examined by using a shift register of depth S on the input columns. See now. Figure 8A , Figure 8A Including at least some embodiments according to this disclosure that can be used to implement Figure 6 A flowchart illustrating an example of a spiral memory access mode process 800 is provided. Process 800 includes a shift register 802 that receives 64 items column-wise in each cycle. Shift register 802 may include an S-1 stage over the 64 items (partial rows), where S is a power of 2, which allows accumulation of several columns such that multiple entries from the same row are always available when writing to RAM. Interleaver 804 performs interleaver permutation on 64*S items, which may depend on the pixel block size. In particular, interleaver permutation may be used for different processing for different block sizes, where each different pixel block size may have a different interleaver pattern. Bucket shifter 806 receives the output of interleaver 804 and has a size of 64 / S over the 64*S items in the group of S. RAM 808 (which may include 64 / S RAM) receives 64*S items every S cycles. The address depth of the RAM 808 can be 128 / s with ping-pong or 64 / s without ping-pong. Each address has one word and the word width is the item width * S. 2 .

[0141] A read operation can be performed every S cycles, reading 64*S items and providing them to a multiplexer 810. The multiplexer 810 can be an Sx1 multiplexer with 64*S input items and 64 output items. Time-division multiplexing (TDM) can be provided: 0, 1, ..., S-1 selection. The multiplexer 810 can provide 64 items per cycle to a bucket shifter 812. The bucket shifter 812 can have a size of 64 / S, exceeding the 64 items in the S groups. The deinterleaving unit 814 can receive the output from the bucket shifter 812 and perform a deinterleaving permutation on the 64 items, which can be pixel block size-dependent. Specifically, the deinterleaving permutation can be used for different processing for different block sizes, where each different pixel block size can have a different deinterleaving mode.

[0142] use Figure 8A This method reduces the amount of RAM by a factor of S. Each RAM can be written to once every S cycles and read every S cycles. As a benefit, shift registers can be repurposed to transpose smaller pixel blocks (e.g., 16x8 or 8x16) while bypassing RAM 808. Figure 8A The process 800 can be used with RAM that does not have write enable. However, write enable can be used to offload the burden of write multiplexing to RAM. Using this method, RAM 808 can be written to every cycle, while the interleaver 804 and bucket shifter 806 preceding RAM 800 are not amplified by S times. Furthermore, the use of write enable eliminates the need for shift registers, relative to... Figure 8B An example of a shift register is described. If write-enabled RAM is available, using write-enabled RAM is better than reducing RAM count compared to using a shift register.

[0143] In addition to using shift registers or write-enabled RAM, or instead of using shift registers or write-enabled RAM, another way to store data on RAM 808 is to have a higher PPC pipeline (e.g., 64 PPC) when the maximum 1D dimension is smaller (e.g., DTC32 (32 < 64)). This configuration can be constructed such that it is equivalent to having a shift register, but the shifted data is all directly available from the pipeline within a cycle. Here, writes can be performed every cycle and reads can be performed every cycle to fully occupy the pipeline with multiple columns and rows of data. In at least one embodiment, this method can be used to further reduce memory count.

[0144] As described herein, write-enabled use can eliminate the need for a shift register. For example, shift register 802 can be removed because a portion of the RAM cells in RAM 808 can be written to in each cycle; RAM 808 is used as a physical shift register, such as... Figure 8B As shown. Now see Figure 8B , Figure 8B Including at least some embodiments according to this disclosure that can be used to implement Figure 6 A flowchart illustrating an example of process 820 for spiral memory access mode is provided. Process 820 can be implemented using a buffer or a hyper-elastic buffer, and the same bucket shifter can be reused across all pixel block types. Process 820 can use a write mode where two out of every four words written to memory are always a repetition of the other two. However, the alternating odd and even cycle mode on the write-enabled input of RMA 808 ensures that only the necessary subset of data words is written to the addressed memory entry. Process 820 can use a read mode that allows RAM 808 to output data bits to regular pixel blocks with simple multiplexing on odd and even cycles. To make the buffer hyper-elastic, the same read address can be shared across all RAMs, and the same read address increments every second as the pixel block occupies more than one cycle.

[0145] Figure 8B For an example of S=2, interleaver 804 can perform interleaver permutation on 64 items, which can be dependent on the pixel block size. The 64 interleaved items can be provided to bucket shifter 806, which can have a size of 32 (on both the read and write sides) on the 64 items in pairs. Using an address depth of 64, 4 words per address with write enabled, and a word width of 20 (which can be a placeholder width), 64 items can be written to RAM 808 per cycle. 128 items can be read from RAM 808 every 2 cycles to provide 128 inputs to multiplexer 810. Multiplexer 810 can include a 2x1 multiplexer for 32 outputs and a 3x1 multiplexer for 32 outputs, providing 64 items per cycle. Bucket shifter 812 can have a size of 32 (on both the read and write sides) on the 64 items in pairs and provide 64 items for deinterleaving. The deinterleaving unit 814 can perform deinterleaving permutations on 64 items, which can be related to the pixel block size, and provides 64 items in a row per cycle.

[0146] In at least one embodiment, a single pixel block can be small enough that the pixel elements of the pixel block do not fill a single address in RAM, resulting in sparsely filled RAM cells. Using the examples above, an 8x8 pixel block can be a sparse pixel block when S=2, and 8x8, 8x16, and 16x8 pixel blocks can be sparse pixel blocks when S=4. To address this issue, the read-side multiplexer 810 can include slight overhead to handle these cases. For example, for S=2, a 3x1 multiplexer can be used. The 3x1 multiplexer can be decomposed into a 2x1 multiplexer stage, followed by another 2x1. The first 2x1 multiplexer stage can select between odd and even RAM entries for non-sparse blocks and create an output N. The second 2x1 multiplexer can bypass the odd-numbered cycles of the first multiplexer. For example, a bypass can be used (e.g., for sparse pixel blocks) when it is not necessary to use two cycles to send the read data (because it fits into one cycle). In other embodiments, a write-side multiplexer can be added to the RAM input data. Bucket-shifted sparse pixel blocks may not be necessary, as they can be small enough to fit perfectly across a single address in all RAM. As another example, in at least one embodiment, the transpose buffer can be bypassed using sparse blocks. Since all data is available in a single cycle, transposition can be achieved by mixing the data within a single cycle (although scheduling can become more difficult).

[0147] Elastic Ping-Pong Buffer

[0148] When the buffer still has a high occupancy rate from previously sent pixel blocks, the transpose unit buffer can be improved for situations where different pixel block sizes are mixed into the pipeline by avoiding stopping the current pixel block at the transpose input. In at least one embodiment, a flexible ping-pong buffer implementing a FIFO-based method can be used to perform the transpose operation, thereby allowing fine-grained tuning of the buffer size between 1D transforms. In at least one embodiment, the buffer size can be adjusted in a fine-grained factor of the maximum supported transform size (e.g., 1.375). By way of example and not limitation, buffer size factors of 1.0 (for a 16PPC / 32PPC pipeline) and 2.0 (for a 64PPC pipeline) can be used to match the throughput and occupancy required for the computation pipeline of a 32x32 transform.

[0149] Using a flexible ping-pong buffer, RAM can be divided into address ranges for reading stored pixel blocks that have been fully written to but not fully read, write address ranges consisting of the data stored in the portion of the current pixel block being written to, and empty memory available for upcoming pixel blocks. When a pixel block is fully read, it can be returned to the free address range of empty memory. Using mutex ranges prevents simultaneous read and write conflicts to the same address. As described in this paper, such a method can utilize a dual address space for storing a single pixel block of the largest size to keep RAM count low.

[0150] In at least one embodiment, the transpose buffer can be made resilient using read and write pointers similar to a synchronous FIFO method; this can be referred to as a hyper-resilient buffer or storage. The latency of the transpose buffer can vary depending on the pixel block size. In a conventional ping-pong buffer, there can be a penalty when the size of pixel blocks in the pixel block sequence is changed. Smaller blocks must wait for larger blocks to be cleared before they can be processed, and only one block can be stored on each side of the ping-pong buffer. To reduce the oscillation effect between smaller and larger sizes, the disclosed method can utilize the write and read address organization described herein, which restricts all pixel blocks to be written to a confined address space starting from 0. In at least one embodiment, data for additional pixel blocks can be written until the write-side buffer is fully filled, provided that previous pixel block data in the same buffer has not been overwritten and sufficient space remains. This can be achieved by maintaining a table of pixel blocks to be written to the read side. The first pixel block in the write buffer can begin at offset 0. The second pixel block can begin at an offset equal to the address depth (AD) of the first pixel block. Any further pixel blocks can begin at an offset equal to the sum of the address depths of the previous pixel blocks.

[0151] See now Figure 9 , Figure 9 Table 900 illustrates an example of how pixel blocks can be stored in the same buffer using an offset. This offset can be implemented using a count maintained by hardware. This count may correspond to the sum of the address depths of previous pixel blocks. Whenever a pixel block is fully written, a check can be performed to see if the read-side buffer is still busy. If the check indicates that the read side is not busy, the buffer can be turned off for writing and transfer to the read side. As can be seen in this example, approximately 30 cycles can be saved on a single transpose unit. Once the IDCT is processed, the same amount can be saved again. On the read side, each entry in the table can be completely cleared to report that it is not busy. If there is not enough space in the buffer to leave the address depth for input pixel blocks, the buffer may still stop input on the write side.

[0152] When using an offset, the buffer acts like a FIFO because 4x4, 4x8, 8x4, and 8x8 blocks can pass through the ping-pong buffer rather than bypass it. Otherwise, these blocks may stop until the ping-pong is released.

[0153] Table 7 provides a generalized offset formula that can be employed:

[0154] Pixel block size Offset C1xR1 0 C2xR2 AD(C1xR1) C3xR3 AD(C1xR1)+AD(C2xR2) C4xR4 AD(C1xR1)+AD(C2xR2)+AD(C3xR3) … … CnxRn AD(C1xR1)+AD(C2xR2)+…+AD(C(n-1)xR(n-1))

[0155] It can be provided that AD(C1xR1) + AD(C2xR2) + … + AD(C(n - 1)xR(n - 1)) + AD(CnxRn) <= AD(ExE).

[0156] The concepts of read and write offsets can be extended to read and write pointers. According to at least one embodiment, instead of resetting the address to 0 at each pixel block, a pointer addressing scheme for the address space AS of the wrap-around buffer can be used. Doing so eliminates the need for a hard partitioning between the two sides of the RAM. As in the previous example, the transpose buffer can still be considered ping-pong because pixel blocks are fully written (ping) before being read in transposed format (pong). The available space in the buffer can be referred to as elastic because the buffer may not be partitioned into fixed-size sections where reads or writes must occur. Since there is a progressive release of space for write access during the process of pixel blocks being read out, this can result in super-elasticity. It is not necessary to fully read out a pixel block before making the pixel block storage idle.

[0157] In at least one embodiment, the pointer divides the address space into a write range (where pixel blocks are written), an empty or idle range (where there are entries available for filling), and a read range (where there are filled entries to be read and sent). Now referring to Figure 10 , Figure 10 illustrates an example of how the address space can be divided into a write range, a read range, and a free range according to at least some embodiments of the present disclosure. In Figure 10 , “%” is the modulo operator and AS represents the address space of the RAM. When A < B implies [A, AS - 1] U [0, B], the range symbol “[]” around AS means [A, B].

[0158] Figure 10Examples of pointers are also shown, which may include a write range pointer wr_ptr, a read range pointer rd_ptr, and a free range pointer wr_ptr_nxt. When writing the last pixel / coefficient of a pixel block, the write pointer wr_ptr can advance to the free range pointer wr_ptr_nxt. Each time physical RAM is read, the read pointer rd_ptr can be advanced. The super-elastic storage can fill over time, especially if there is backpressure that stops read operations. Conversely, if no data is presented to the transpose buffer, then the data becomes empty. Full and empty conditions can be used to control reads and writes using the buffer, and can be tracked using bits or flags during RAM operations to control write and read enablement. In at least one embodiment, if the write range of an incoming pixel block overlaps with the read range and the storage is not empty, then the elastic ping-pong buffer or storage can report full, meaning the read pointer address falls within the write range and implying there is not enough free space for the entire pixel block to be written. In at least one embodiment, the read pointer rd_ptr, which is equal to the write pointer wr_ptr, can indicate whether the buffer is full or empty. If this state is caused by a change to the read pointer rd_ptr, it indicates that the buffer is full because a write is being caught up with a read. Otherwise, if this state is caused by a change to the write pointer wr_ptr, it indicates that the buffer is empty because the write is a folded read.

[0159] Although Figure 6 In the example, write operations for 8x8 pixel blocks follow an addressing pattern with a spiral pattern, and read addresses are consistent across all eight RAM blocks. However, this doesn't need to be true for each individual 8x8 block being written to memory. For example, suppose a sequence of 8x8 pixel blocks enters a buffer. For even 8x8 pixel blocks in a sequence, a spiral addressing pattern can be used to perform writes, and a uniform addressing pattern can be used to perform reads, such as those concerning... Figure 6 As described. However, for odd-numbered 8x8 pixel blocks entering the buffer, a uniform addressing mode can be used for writing, and a spiral addressing mode can be used for reading. Thus, the read and write modes can be toggled for every other pixel block entering the buffer. This scheme requires only 1x the buffer size while avoiding blocking. Once a column or row of data is read, the next write can fill the just-read entry using the same addressing scheme used for reading without waiting.

[0160] Optimize area, power, and timing by minimizing additional features.

[0161] As described herein, the number of half-butterflies can be reduced by appropriately distributing them across phase 208 to achieve data flow 100. The number of additional items within phase 208 can also be reduced using the methods described herein. The parameters of phase 208 may include H, where H refers to the number of half-butterflies used in phase 208; T, the number of transformations using half-butterflies within phase 208 (AV1 and VP9 transformations of the same type can be counted separately if both are implemented); and h. t , where t is the index of the transformation type supported in a given phase 208, where 1 ≤ t ≤ T, and h t This is the number of semi-butterflies used by transformation t within a phase of 208. Therefore, 1 ≤ t ≤ T, 1 ≤ h t ≤H, and there must exist at least one transformation s such that h s =H, otherwise, phase 208 would require fewer half-butterflies.

[0162] Given the parameters of phase 208, the size of the space for minimizing the additive terms can be estimated. Each shared half-butterfly circuit can include two multiplexers. One multiplexer selects the left cosA of (X*cos A + Y*cos B), and the other selects the right cos B. Because the additions are interchangeable, these left and right terms can be swapped, resulting in many possible arrangements. If the transformation requires h... t If there is a half-butterfly shape, then the left and right combinations of its terms are... If we consider all T transformations, then the possible combinations are expanded according to equation (1):

[0163]

[0164] Given H semi-butterflies, for a specific transformation type index t, h can be selected. t They are distributed using a semi-butterfly-shaped subset. The number of possible subsets i can be represented by equation (2):

[0165]

[0166] The subset i chosen by the half-butterfly can be replaced by h. t !, leaving many combinations of transformations corresponding to equation (3):

[0167]

[0168] To reduce the size of this space, any transformation can be chosen, and the semi-butterfly shape to be fixed can be selected according to the order of its semi-butterfly shape and the left-right distribution of its independent variable. For example, the transformation using the maximum number of semi-butterfly shapes H can be chosen, allowing the above combination to be divided by 2. HH! This can produce multiple combinations corresponding to a given phase 208 in equation (4):

[0169]

[0170] Given these combinations, the number of non-zero adder terms can be minimized by performing a search across combinations. In at least one embodiment, a computer-aided design (CAD) tool for circuit design can therefore implement the aforementioned methods or heuristics to determine and / or reduce the number of adders.

[0171] See now Figure 11 Each block of method 1100, and other methods described herein, includes computational processes that can be performed using any combination of hardware, firmware, and / or software. For example, different functions can be implemented by a processor executing instructions stored in memory and / or one or more circuits. The methods can also be embodied, at least in part, as computer-usable instructions stored on a computer storage medium. To name just a few, the methods can be provided by a standalone application, service, or managed service (standalone or in combination with other managed services) or a plug-in to another product. These methods can be additionally or alternatively performed by any system or any combination of systems, including but not limited to those described herein.

[0172] Figure 11 This is a flowchart illustrating a method 1100 for a video processing pipeline operating in a phase using shared hardware resources, according to some embodiments of the present disclosure. At block B1102, method 1100 includes providing pixel blocks to the video processing pipeline. For example, scheduler 116 may provide pixel blocks to an implementation... Figure 1 The data stream 100 is the input to the quantization and transformation unit. In block B1104, method 1100 includes selecting and configuring a set of phases to perform a transformation on a pixel block based on one or more processing parameters. For example, configuration circuitry 202 may select and configure a set of phases 208 based on one or more of the processing parameters 206 to perform a transformation on the pixel block. In block B1106, method 1100 includes using this set of phases to perform a transformation. For example, the quantization and transformation unit may use a set of phases 208 selected and configured using one or more configuration circuitry 202 to perform a transformation.

[0173] Example computing device

[0174] Figure 12This is a block diagram of an example computing device 1200 suitable for implementing some embodiments of the present disclosure. The computing device 1200 may include an interconnect system 1202 directly or indirectly coupled to the following devices: a memory 1204, one or more central processing units (CPUs) 1206, one or more graphics processing units (GPUs) 1208, a communication interface 1210, input / output (I / O) ports 1212, input / output components 1214, a power supply 1216, one or more presentation components 1218 (e.g., one or more displays), and one or more logic units 1220. In at least one embodiment, one or more computing devices 1200 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 1208 may include one or more vGPUs, one or more CPUs 1206 may include one or more vCPUs, and / or one or more logic units 1220 may include one or more virtual logic units. Thus, one or more computing devices 1200 may include discrete components (e.g., a full GPU dedicated to computing device 1200), virtual components (e.g., a portion of the GPU dedicated to computing device 1200), or combinations thereof.

[0175] although Figure 12 The various blocks are shown as connected via interconnect system 1202 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 1218 (such as a display device) may be considered I / O component 1214 (e.g., if the display is a touchscreen). As another example, CPU 1206 and / or GPU 1208 may include memory (e.g., memory 1204 may represent a storage device other than the memory of GPU 1208, CPU 1206, and / or other components). In other words, Figure 12 The computing devices described are for illustrative purposes only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of… Figure 12 Within the scope of computing devices.

[0176] Interconnect system 1202 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 1202 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 1206 may be directly connected to memory 1204. Further, CPU 1206 may be directly connected to GPU 1208. In cases where there is a direct or point-to-point connection between components, interconnect system 1202 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 1200.

[0177] The memory 1204 may include any computer-readable medium from a variety of computer-readable media. The computer-readable medium may be any available medium accessible by the computing device 1200. The computer-readable medium may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable medium may include computer storage media and communication media.

[0178] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1204 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 1200. As used herein, computer storage media does not include the signal itself.

[0179] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0180] CPU 1206 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1200 to perform one or more of the methods and / or processes described herein. Each CPU 1206 may contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 1206 may contain any type of processor and may contain different types of processors depending on the type of computing device 1200 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1200, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 1200 may also include one or more CPUs 1206.

[0181] In addition to or in lieu of one or more CPUs 1206, one or more GPUs 1208 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1200 to perform one or more of the methods and / or processes described herein. One or more GPUs 1208 may be integrated GPUs (e.g., with one or more CPUs 1206) and / or one or more GPUs 1208 may be discrete GPUs. In embodiments, one or more GPUs 1208 may be coprocessors of one or more CPUs 1206. GPUs 1208 may be used by computing device 1200 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 1208 may be used for general-purpose computing on a GPU (GPGPU). GPUs 1208 may include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPUs 1208 may produce pixel data of an output image in response to rendering commands (e.g., rendering commands received from CPUs 1206 via a host interface). GPU 1208 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 1204. GPU 1208 may include two or more GPUs operating in parallel (e.g., via links). The links may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 1208 may produce pixel data or GPGPU data for different portions of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0182] In addition to or in lieu of CPU 1206 and / or GPU 1208, logic unit 1220 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1200 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 1206, one or more GPUs 1208, and / or one or more logic units 1220 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 1220 may be a portion of one or more CPUs 1206 and / or GPUs 1208 and / or integrated into one or more CPUs 1206 and / or GPUs 1208, and / or one or more logic units 1220 may be discrete components or otherwise external to CPUs 1206 and / or GPUs 1208. In an embodiment, one or more of the logic units 1220 may be coprocessors of one or more of the CPU 1206 and / or one or more of the GPU 1208.

[0183] Examples of logic unit 1220 include one or more processing cores and / or components thereof, such as data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or fast peripheral component interconnect (PCIe) element, etc.

[0184] Communication interface 1210 may include one or more receivers, transmitters, and / or transceivers enabling computing device 1200 to communicate with other computing devices via electronic communication networks (including wired and / or wireless communications). Communication interface 1210 may include components and functions for enabling communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, one or more logic units 1220 and / or communication interface 1210 may include one or more data processing units (DPUs) for directly transmitting data received via a network and / or via interconnect system 1202 to one or more GPUs 1208 (e.g., their memory).

[0185] I / O port 1212 enables computing device 1200 to be logically coupled to other devices including I / O component 1214, one or more presentation components 1218, and / or other components, some of which may be built into (e.g., integrated into) computing device 1200. Illustrative I / O component 1214 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 1214 can provide a natural user interface (NUI) for processing aerial gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of speech recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, aerial gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 1200. Computing device 1200 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 1200 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 1200 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0186] Power supply 1216 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 1216 may provide power to computing device 1200 to enable the components of computing device 1200 to operate.

[0187] The presentation component 1218 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 1218 may receive data from other components (e.g., GPU 1208, CPU 1206, DPU, etc.) and output the data (e.g., as images, videos, sounds, etc.).

[0188] Example Data Center

[0189] Figure 13 An example data center 1300 that may be used in at least one embodiment of this disclosure is shown. The data center 1300 may include a data center infrastructure layer 1310, a framework layer 1320, a software layer 1330, and / or an application layer 1340.

[0190] like Figure 13 As shown, the data center infrastructure layer 1310 may include a resource coordinator 1312, grouped computing resources 1314, and node computing resources (“nodes CR”) 1316(1)-1316(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CRs 1316(1)-1316(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and / or cooling modules, etc. In some embodiments, one or more node CRs from nodes CRs 1316(1)-1316(N) may correspond to servers having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR1316(1)-13161(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more of nodes CR1316(1)-1316(N) may correspond to virtual machines (VMs).

[0191] In at least one embodiment, the grouped computing resources 1314 may include individual groups of nodes CR1316 housed within one or more racks (not shown), or multiple racks housed within a data center at different geographical locations (also not shown). Individual groups of nodes CR1316 within the grouped computing resources 1314 may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several nodes CR1316, including CPUs, GPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0192] Resource coordinator 1312 may be configured or otherwise control one or more nodes CR1316(1)-1316(N) and / or grouped computing resources 1314. In at least one embodiment, resource coordinator 1312 may include a Software Design Infrastructure (SDI) management entity for data center 1300. Resource coordinator 1312 may include hardware, software, or some combination thereof.

[0193] In at least one embodiment, such as Figure 13 As shown, framework layer 1320 may include a job scheduler 1344, a configuration manager 1334, a resource manager 1336, and / or a distributed file system 1338. Framework layer 1320 may include a framework for software 1332 supporting software layer 1330 and / or one or more applications 1342 supporting application layer 1340. Software 1332 or application 1342 may respectively contain web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1320 may be, but is not limited to, free and open-source software web application frameworks (such as Apache Spark) that can utilize distributed file system 1338 for large-scale data processing (e.g., "big data"). TM(Hereinafter referred to as "Spark") is a type of resource. In at least one embodiment, the job scheduler 1344 may include Spark drivers to facilitate the scheduling of workloads supported by different layers of data center 1300. The configuration manager 1334 may be able to configure different layers, such as software layer 1330 and framework layer 1320 (which includes Spark and distributed file system 1338 for supporting large-scale data processing). The resource manager 1336 may be able to manage clusters or groups of computing resources mapped to distributed file system 1338 and job scheduler 1344 or allocated to support clusters or groups of distributed file system 1338 and job scheduler 1332. In at least one embodiment, clusters or groups of computing resources may include grouped computing resources 1314 in data center infrastructure layer 1310. The resource manager 1336 may coordinate with resource coordinator 1312 to manage these mapped or allocated computing resources.

[0194] In at least one embodiment, the software 1332 included in software layer 1330 may include software used in at least a portion of nodes CR1316(1)-1316(N), grouped computing resources 1314, and / or the distributed file system 1338 of framework layer 1320. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0195] In at least one embodiment, the application 1342 included in the application layer 1340 may include one or more types of applications used at least in part by nodes CR1316(1)-1316(N), grouped computing resources 1314, and / or the distributed file system 1338 of the framework layer 1320. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in combination with one or more embodiments.

[0196] In at least one embodiment, any of the configuration manager 1334, resource manager 1336, and resource coordinator 1312 can implement any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. Self-modification actions can free data center operators of data center 1300 from making potentially poor configuration decisions and may prevent underutilization and / or poor performance of the data center.

[0197] According to one or more embodiments described herein, data center 1300 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, one or more machine learning models may be trained by using the software and / or computing resources described above with respect to data center 1300 to compute weight parameters according to a neural network architecture. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1300 by using weight parameters computed through one or more training techniques, such as, but not limited to, those described herein.

[0198] In at least one embodiment, the data center 1300 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform services that infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0199] Example network environment

[0200] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 12 This is implemented on one or more instances of computing devices 1200—for example, each device may include similar components, features, and / or functions of one or more computing devices 1200. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of a data center 1300, examples of which are described herein. Figure 13 To describe in more detail.

[0201] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0202] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0203] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0204] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). The core server may assign at least a portion of the functionality to the edge server if the connection to the user (e.g., a client device) is relatively close to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0205] One or more client devices may include the information described in this article. Figure 12At least some of the components, features, and functions of one or more example computing devices 1200 described. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0206] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, which are executed by a computer or other machine such as a personal digital assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific extracted data type. This disclosure can be practiced in a wide variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a communication network.

[0207] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0208] The subject matter of this disclosure is described in detail herein to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms “step” and / or “block” may be used herein to imply different elements of the method employed, these terms should not be construed as suggesting any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. A processing circuit, comprising: A phased video processing pipeline, each phase having multiple half-butterfly HB circuits, wherein the phases are selectable and configurable to perform multiple transform types on pixel blocks of multiple sizes using the HB circuits, and each transform is performed by sequentially passing through at least some of the phases; as well as A configuration circuit is configured to select and configure a set of phases to perform the transformation on the pixel block based at least on one or more dimensions of the pixel block and the transformation type of one of a plurality of transformations.

2. The processing circuit according to claim 1, wherein, Each transformation type defines a computational phase of the transformation type, and one phase of the phase implements at least a portion of one phase of the multiple transformation types.

3. The processing circuit according to claim 1, wherein, The configuration circuitry is further configured to configure the set of phases to perform the transformation, based at least on the bit depth associated with the pixel block and the encoding format associated with the pixel block.

4. The processing circuit according to claim 1, wherein, The configuration circuitry is further configured to configure one or more clamping ranges of the output of one or more phases to perform the transformation, based at least on the encoding format and bit depth associated with the pixel block.

5. The processing circuit according to claim 1, wherein, The configuration circuitry is further configured to configure the rounding positions of one or more phase outputs to perform the transformation, based at least on the bit depth associated with the pixel block.

6. The processing circuit according to claim 1, wherein, The video processing pipeline includes: A transformation circuit is used to perform a forward transformation on the pixel block, wherein one or more rate-distortion optimization costs are calculated from data in the frequency domain, the data corresponding to the output of the forward transformation; A funnel circuit is used to filter out one or more of the pixel blocks before the inverse transform circuit of the video processing pipeline, based on at least one or more dimensions of the pixel blocks. The inverse transform circuit is configured to perform an inverse transform on a subset of the pixel blocks based on filtering by the funnel circuit, the inverse transform being performed with a lower throughput than the forward transform; and Intra-frame prediction circuitry is used to perform intra-frame prediction using feedback corresponding to the output of the inverse transform.

7. The processing circuit according to claim 1, wherein, The transformation is performed using pre-calculated trigonometric coefficients.

8. The processing circuit according to claim 1, wherein, The transformation is an asymmetric discrete sine transformation or an identity transformation of the plurality of HBs mapped to one or more of the phases.

9. The processing circuit according to claim 1, wherein, The video processing pipeline includes: An adder clamp for generating a first one or more overflow signals based at least on clamping one or more calculations of the transform; and A rounding shift clamp is used to generate a second one or more overflow signals based at least on rounding the output of the transform, wherein the video processing pipeline is used to generate an interrupt based at least on the first one or more overflow signals and the second one or more overflow signals.

10. The processing circuit according to claim 1, wherein, The configuration circuit includes multiple independent thermal multiplexers, each having a selection source corresponding to one or more dimensions and the transformation type to perform the selection and configuration of the set of phases.

11. The processing circuit according to claim 1, wherein, The configuration circuit is used to bypass one or more inlet or outlet phases of the phase, at least based on the size of the pixel block.

12. The processing circuit of claim 1, further comprising a scheduler configured to provide the pixel blocks to the video processing pipeline in an order corresponding to the size of the pixel blocks, wherein, The dimensions increase primarily along the stated order.

13. The processing circuit according to claim 1, wherein, The transformation is applied without using a real multiplier.

14. A method for performing a transformation on a block of pixels, comprising: A pixel block is provided as input to the video processing pipeline of a phase, each phase having a plurality of half-butterfly HB circuits, wherein the phase is selectable and configurable to perform multiple transformation types on the pixel block of multiple sizes using the HB circuits, and each transformation is performed by sequentially passing through at least some of the phases; Using configuration circuitry, a set of phases is selected and configured to perform the transformation on the pixel block, based at least on one or more dimensions of the pixel block and the transformation type of one of a plurality of transformations. as well as The transformation is performed using the set of phases, wherein the set of phases is selected and configured using the configuration circuitry.

15. The method of claim 14, wherein the video processing pipeline includes a transpose circuit having a resilient ping-pong buffer for simultaneously storing at least a portion of the pixel block with at least a portion of different pixel blocks among a plurality of pixel blocks.

16. The method of claim 14, wherein the video processing pipeline includes a transpose circuit for performing spiral memory read and write modes to perform the transpose operation of the transform using interleaving, deinterleaving, and bucket shifting.

17. The method of claim 14, wherein, The number of random access memory (RAM) used to process the transpose of the transformation is less than one or more dimensions of the pixel block, and the transpose includes: writing a portion of the RAM at a first cycle rate using a shift register; and reading from the RAM using time-division multiplexing at a second cycle rate corresponding to the depth of the shift register.

18. The method according to claim 14, wherein, The PPC of the video processing pipeline is higher than the maximum 1D dimension of the transformation type supported by the video processing pipeline, so that writing to the transposed random access memory RAM for processing the transformation can be performed every cycle, and reading from the RAM can be performed every cycle to occupy the RAM with multiple columns and multiple rows of one or more of the pixel blocks.

19. The method of claim 14, wherein, An alternating mode of write-enabled input is provided for the transposed random access memory (RAM) used to process the transformation, so that a portion of the RAM is written at a first cycle rate and read from the RAM using time-division multiplexing at a second cycle rate.

20. The method of claim 14, comprising alternating between a first read / write mode that uses a spiral addressing mode to write to a transposed random access memory RAM for processing the transformation and reads from the RAM using a uniform addressing mode, and a second read / write mode that uses a uniform addressing mode to write to the RAM and reads from the RAM using a spiral addressing mode.

21. The method of claim 14, comprising: A forward transform is performed on the pixel block, wherein one or more rate-distortion optimization costs are calculated from data in the frequency domain, the data corresponding to the output of the forward transform; One or more of the pixel blocks are filtered out before the inverse transform circuit of the video processing pipeline, based on at least one or more dimensions of the pixel blocks; Based on the inverse transformation of a group of pixel blocks performed by filtering by a funnel circuit, the inverse transformation is performed with a lower throughput than the forward transformation; as well as Intra-frame prediction is performed using feedback corresponding to the output of the inverse transform.

22. The method according to claim 14, wherein, The number of addends included in at least one phase of the phase is estimated based at least on minimizing the non-zero addends of the transform type, the minimization of the non-zero addends of the transform type being based at least on the left-right combination of the addends of the transform type, wherein the transform type has the maximum number of HBs among the plurality of transform types supported by the phase.

23. A system for performing transformations on a block of pixels, comprising: A positive transform circuit having a first phase, each first phase having a first plurality of half-butterfly HB circuits, wherein the first phase is selectable and configurable to perform a plurality of transform types of positive transforms on pixel blocks of a plurality of sizes using the first HB circuits, and each positive transform is performed by sequentially passing through at least some of the first phases; An inverse transform circuit having a second phase, each second phase having a second plurality of HB circuits, wherein the second phase is selectable and configurable to perform an inverse transform on at least some of the pixel blocks using the second HB circuits, and each inverse transform is performed by sequentially passing through at least some of the second phases; as well as One or more configuration circuits are used to select and configure a first group of the first phase and a second group of the second phase based at least on one or more dimensions of the pixel blocks to perform forward and inverse transformations on the pixel blocks.

24. The system according to claim 23, wherein, The inverse transform circuit is used to perform the inverse transform on a subset of the pixel blocks based at least on the size of the pixel blocks.

25. The system according to claim 23, wherein, The inverse transform circuit performs the inverse transform with a lower throughput than the forward transform circuit.

26. The system according to claim 23, wherein, At least each of the plurality of HB circuits in the second phase has a corresponding minimum bit width required to support the bit precision specified by one or more encoding formats for each of the plurality of transform types.

Images