Low-delay consumption of encoded video bitstreams

Abstract

To provide systems, apparatuses, and methods for reducing latency when consuming an encoded video bitstream in real-time.SOLUTION: In a computing system having a video encoder coupled to a consumer module, the consumer module consumes encoded chunks of a bitstream before a video encoder finishes encoding an entire frame or an entire slice of the frame. To enable pipelining of the consumption with the encoding, the encoder updates a buffer write pointer BWP with an indication of the amount of data that has been written to the bitstream buffer. The consumer module retrieves encoded data from the bitstream buffer up to the location indicated by a bitstream write pointer.SELECTED DRAWING: Figure 5

Description

[Background technology]

[0001] The bandwidth requirements of digital video streaming continue to increase over time. Various applications benefit from video compression, which requires less storage space for archived video information and / or less bandwidth for transmission of video information. Thus, various techniques have been developed to improve the quality and availability of digital video. An example of such a technique is described in H.264, a video compression standard proposed by the Joint Video Team (JVT). Many current multimedia-enabled digital devices incorporate digital video codecs (hardware and / or software configured to encode and / or decode data) that comply with the H.264 standard. The High Efficiency Video Coding (HEVC) standard is another video compression standard that follows H.264.

[0002] To prepare video for streaming, storage, or further processing, a hardware accelerator (e.g., a video encoder) outputs an encoded bitstream for each video frame. The encoded bitstream is typically written to memory before being consumed by another logical unit (e.g., for network transmission). Video consumption (e.g., streaming) of each encoded video frame typically begins after each frame is fully encoded. In other words, the video encoder typically waits until the entire frame is encoded to ensure that all bitstream memory write operations are complete, to verify the number of bits that have been written, and to provide a way for the bitstream consumer to determine the number of bits that have been written and the location of the bitstream on the storage device. A disadvantage of this approach is that it means that consumption of an encoded frame can only begin when the entire frame is encoded. This prevents frame encoding from being pipelined with other actions, such as network transmission, resulting in increased latency.

[0003] The advantages of the methods and mechanisms described herein may be best understood by reference to the following description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0004] [Figure 1] FIG. 1 is a block diagram of one embodiment of a computing system. [Diagram 2] FIG. 13 is a timing diagram of one embodiment of operations for an encoder and a bitstream consumer. [Diagram 3] FIG. 13 is a timing diagram of another embodiment of the operations for an encoder and a bitstream consumer. [Figure 4] FIG. 2 is a block diagram of another embodiment of a computing system. [Diagram 5] FIG. 2 is a block diagram of one embodiment of a video encoder coupled to a consumer module. [Figure 6] FIG. 1 is a general flow diagram illustrating one embodiment of a method for enabling on-the-fly access to an encoded video bitstream of a given video frame while the given video frame is being encoded. [Figure 7] FIG. 2 is a general flow diagram illustrating one embodiment of a method for a consumer module to access an encoded video bitstream in real-time as the bitstream is being encoded. [Figure 8] FIG. 1 is a general flow diagram illustrating one embodiment of a method for controlling the granularity of bitstream write pointer updates. [Figure 9] 1 is a general flow diagram illustrating one embodiment of a method for a video encoder that performs prediction across consumed chunks. [Figure 10] 2 shows a timing diagram of one embodiment of real-time consumption of chunks of an encoded video bitstream. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, those skilled in the art should recognize that various embodiments may be practiced without those specific details. In some cases, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. It will be understood that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements.

[0006] Various systems, apparatus, methods, and computer-readable media are disclosed herein for enabling low-latency consumption of an encoded video bitstream before an entire frame or slice is encoded. The systems include at least one or more processors coupled to a memory device. The one or more processors include circuitry for executing at least a video encoder and a consumer module. The consumer module is any of various types of logical units that access and consume the encoded video bitstream in real-time as the video bitstream is encoded by the video encoder (e.g., transmit the video bitstream over a network). Rather than waiting until an entire frame or slice is encoded by the video encoder, the consumer module starts processing intermediate frames and / or intermediate slices of the encoded bitstream. It should be noted that slice encoding is typically codec-specific, and any of various types of slice encoding may be used in various embodiments. The techniques described herein for consuming an encoded video bitstream on the fly are applicable to full-frame encoding or any codec-specific sub-frame encoding (e.g., H-264 slice encoding).

[0007] In one embodiment, to facilitate consumption of the encoded bitstream before completing encoding of an entire frame, the video encoder periodically updates a buffer write pointer to indicate the amount of encoded data written to the video buffer. In one embodiment, the granularity at which the video encoder updates the buffer write pointer is programmable. In one embodiment, the consumer module programs the granularity at which the video encoder updates the buffer write pointer. In other words, the video encoder can be configured to update the buffer write pointer after writing a certain number of bits (e.g., 512 bits, 1024 bits, etc.) of encoded data to the buffer. If the last part of the encoded frame results in less than the granularity, the video encoder updates the buffer write pointer and sets a frame done flag to indicate that encoding of the frame is complete. In this case, the chunks have a size different from the above-mentioned granularity (i.e., the size of the chunks is variable).

[0008] In parallel with the video encoder encoding the video bitstream, the consumer module monitors updates to the buffer write pointer. After detecting an update to the buffer write pointer, the consumer module reads from the bitstream buffer to the location indicated by the buffer write pointer. In addition to the bit count, the buffer write pointer also includes a frame done flag that indicates whether frame encoding is complete for the current frame. In one implementation, the frame done flag is set to true (e.g., 1) by the video encoder when frame encoding is complete for the current frame and the buffer write pointer has the final frame bit count for the current frame. In this implementation, the frame done flag is reset to false (e.g., 0) by the consumer module after the consumer module retrieves all the encoded data from the video buffer for the current frame. The frame done flag provides a way for the consumer module to determine that the buffer write pointer will not be updated for the current frame, allowing the consumer module to read the remaining encoded bitstream data even if the amount of data is smaller than the specified granularity. In one implementation, the frame done synchronization is performed separately from the buffer write pointer updates.

[0009] 1, a block diagram of one embodiment of a computing system 100 is shown. In one embodiment, computing system 100 includes at least processors 105A-N, input / output (I / O) interface 120, bus 125, memory controller(s) 130, network interface 135, and memory device(s) 140. In other embodiments, computing system 100 includes other components and / or is arranged differently. Processors 105A-N represent any number of processors that may be included in system 100.

[0010] In one embodiment, processor 105A is a general-purpose processor such as a central processing unit (CPU). In one embodiment, processor 105N is a data parallel processor with a highly parallel architecture. Data parallel processors include graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc. One or more of processors 105A-N include circuitry implementing a video encoder and a consumer module. The consumer module accesses and consumes the encoded bitstream of video frames in real time while the video encoder is encoding the video frames.

[0011] Memory controller(s) 130 represent any number and type of memory controller accessible by processors 105A-N and input / output devices (not shown) coupled to I / O interface 120. Memory controller(s) 130 are coupled to any number and type of memory device(s) 140. Memory device(s) 140 represent any number and type of memory devices. For example, memory types in memory device(s) 140 may include dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, or ferroelectric random access memory (FeRAM), etc.

[0012] I / O interface 120 represents any number and type of I / O interfaces (e.g., Peripheral Component Interconnect (PCI) bus, PCI-Extended (PCI-X), PCI Express (PCIE) bus, Gigabit Ethernet (GBE) bus, Universal Serial Bus (USB)). Various types of peripheral devices (not shown) are coupled to I / O interface 120. Such peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and the like. Network interface 135 is used to send and receive network messages across a network.

[0013] In various implementations, computing system 100 is a computer, a laptop, a mobile device, a gaming console, a server, a streaming device, a wearable device, or any of a variety of other types of computing systems or computing devices. It should be noted that the number of components of computing system 100 varies from implementation to implementation. For example, there may be more or fewer components than those shown in FIG. 1 . It should also be noted that computing system 100 may include other components not shown in FIG. 1 . Additionally, in other implementations, computing system 100 may be structured in a manner other than that shown in FIG. 1 .

[0014] Referring now to FIG. 2, a timing diagram 200 of one embodiment of the operations for the encoder 205 and the bitstream consumer 210 is shown. In one embodiment, the encoder 205 encodes individual frames 0-2 of the video stream and makes each frame available to the bitstream consumer 210 at the end of each frame. For example, the encoder 205 verifies that a bitstream memory write operation is complete, verifies the number of bits that have been written to the bitstream buffer, and then provides a way for the bitstream consumer to determine the size and location of the encoded bitstream in memory. It should be noted that the bitstream consumer 210 may also be referred to as a "consumer module." The encoder 205 encodes the video stream according to any of a variety of coding standards. For example, in one embodiment, the encoder 205 encodes the bitstream to comply with the H.264 video compression standard. In another embodiment, the encoder 205 encodes the bitstream to comply with the High Efficiency Video Coding (HEVC) standard. In other embodiments, the encoder 205 encodes the bitstream to comply with other standards.

[0015] Thus, as shown in timing diagram 200, bitstream consumer 210 can start consuming a given frame only after encoder 205 has encoded the entire frame. For example, bitstream consumer 210 can start consuming the encoded bitstream generated for frame 0 only after frame 0 has been completely encoded by encoder 205. The same sequence of events occurs for frames 1-2. This results in a delay from when encoder 205 starts encoding the frame to when bitstream consumer 210 can start consuming the encoded frame.

[0016] Referring now to FIG. 3, a timing diagram 300 of another embodiment of the operation of the encoder 305 and the bitstream consumer 310 is shown. In the embodiment shown in the timing diagram 300, the encoder 305 encodes slices of a frame, and then the bitstream consumer 310 can consume the individual slices after the encoder 305 finishes encoding a given slice. This helps to reduce the latency of the operation compared to the embodiment shown in the timing diagram 200 (of FIG. 2). For example, the encoder 305 encodes slice 0 of frame 0 and makes the location and size of encoded slice 0 available to the bitstream consumer 310. The bitstream consumer 310 can then consume encoded slice 0 while the encoder 305 starts encoding slice 1 of frame 0. This pattern continues with slices 1-2 of frame 0 and subsequent slices of frame 0. Since the content and structure of a slice is typically defined by a compression standard, the codec used to generate such slices is compliant with the corresponding standard. A slice as used herein is also referred to herein as a "codec-level slice."

[0017] Although the embodiment shown in timing diagram 300 provides reduced latency compared to the embodiment shown in timing diagram 200, a drawback of the slice-based approach is slice boundary effects that potentially introduce visual artifacts that degrade the user experience. Furthermore, the slice-based approach reduces compression efficiency because some coding standards (such as H.264) stipulate that prediction across slices cannot be performed. In such cases, each slice of a frame is coded independently of other slices of the frame. In addition, the slice-based approach results in increased slice header overhead because each slice includes a header at the beginning of the slice. The negative impact of the slice-based approach increases as the number of slices per frame increases.

[0018] 4, a block diagram of another embodiment of a computing system 400 is shown. In one embodiment, the computer system 400 includes processors 405A-N, an input / output interface 420, a bus 425, a memory controller(s) 430, a memory device(s) 435, and a network interface 440. The processors 405A-N represent any type and number of processors (e.g., CPU, GPU, DSP, FPGA, ASIC, etc.). In one embodiment, the processor 405A includes circuitry for implementing a video encoder 410 for encoding a video bitstream. In one embodiment, the circuitry of the processor 405A executes software instructions to implement the functionality of the video encoder 410. In another embodiment, the circuitry of the processor 405A includes control logic for implementing the functionality of the video encoder 410. In other embodiments, any combination of hardware and / or software of the processor 405A implements the functionality of the video encoder 410.

[0019] In one embodiment, the processor 405N includes circuitry for executing the consumer module 415 for accessing and processing the encoded video bitstream. In one embodiment, the circuitry of the processor 405N executes software instructions to implement the functionality of the consumer module 415. In another embodiment, the circuitry of the processor 405N includes control logic for implementing the functionality of the consumer module 415. In other embodiments, any combination of hardware and / or software of the processor 405N implements the functionality of the consumer module 415. In another embodiment, a single processor 405 includes hardware and / or software for implementing both the video encoder 410 and the consumer module 415. In further embodiments, the multiple processors 405A-N include hardware and / or software for implementing the video encoder 410 and / or the multiple processors 405A-N include hardware and / or software for implementing the consumer module 415. It will be appreciated that in other embodiments, the system 400 includes other components and / or is organized in other suitable manners.

[0020] The video encoder 410 includes circuitry for encoding frames of a video sequence into an encoded video bitstream. In one implementation, the video encoder 410 stores the encoded bitstream in a bitstream buffer 445 in memory device(s) 435. The bitstream buffer 445 represents any number and size of buffers, and the memory device(s) 435 represents any number and type of memory devices. The memory device(s) 435 are located in any suitable location in the system 400. For example, in various embodiments, the memory device(s) 435 are implemented as one or more cache(s) of the processor 405A, external to the processor 405A, internal to the processor 405A, or otherwise. When the video encoder 410 writes a portion of the encoded data to the bitstream buffer 445, the video encoder 410 updates the value of the buffer write pointer 450 to indicate the amount of data (e.g., number of bits) that has been written to the bitstream buffer 445. In some implementations, the data ready flag 460 is used to indicate that the encoded data has been written to the buffer 445 and is ready for consumption. In other implementations, updates to the buffer write pointer 450 are detected and serve to indicate that the encoded data has been written to the buffer 445 and is ready for consumption.

[0021] In one embodiment, the frequency with which the video encoder 410 updates the buffer write pointer 450 is programmable. For example, in one embodiment, the consumer module 415 programs the frequency with which the video encoder 410 updates the buffer write pointer 450. In other words, the consumer module 415 specifies an update granularity, which is defined as the number of bits that triggers an update to the buffer write pointer 450. When the video encoder 410 writes a number of bits to the bitstream buffer 445 and the number of bits is equal to or greater than the update granularity, the video encoder 410 updates the buffer write pointer 450. Otherwise, if the number of bits written to the bitstream buffer 445 is less than the update granularity, the video encoder 410 does not update the buffer write pointer 450. This helps to avoid unnecessary memory transactions to update the buffer write pointer 450 when the consumer module 415 only consumes a certain size of bitstream chunk at a time. As used herein, a "chunk" refers to one or more bits. In some implementations, a chunk refers to a specific number of bytes of data, while in other implementations, a chunk refers to a number of bits that is not evenly divisible into bytes. Such implementations are possible and contemplated. For example, in one implementation, the consumer module 415 consumes a chunk size that matches the maximum size of a network transmission packet payload. In one implementation, the maximum size of a network transmission packet payload is 1024 bytes, and therefore the consumer module 415 programs the update granularity to be equal to 1024 bytes (i.e., the chunk size is 8192 bits). In other implementations, the update granularity is set equal to another number of bits.

[0022] When an entire frame is being encoded, the video encoder 410 ensures that the bitstream write pointer 450 contains the final bit count of the entire frame bitstream, even if this final bit count is smaller than the granularity programmed by the consumer module 415. To address the final bit count issue, the video encoder 410 sets a frame done flag when an entire frame is encoded. Setting the frame done flag 455 notifies the consumer module 415 that the frame encoding is complete and that the bitstream write pointer 450 has the final frame bit count. In various implementations, the consumer module 415 polls the frame done flag 455 to determine if the frame encoding is complete. In other implementations, a signal or other indication is generated to notify the consumer module 415 that the data is ready for consumption. In this way, the consumer module 415 is notified that the bitstream write pointer 450 will no longer be updated for the frame, so that the consumer module 415 is able to read any of the remaining bitstream bits, even if the number of remaining bitstream bits is smaller than the specified granularity. The frame done flag 455 is reset by the consumer module 415 after the consumer module 415 has retrieved all the bits in the bitstream buffer 445 for the current frame.

[0023] In one implementation, the location of the bitstream write pointer 450 is determined by the consumer module 415 (e.g., a specific location in GPU memory). In one implementation, a separate bitstream write pointer 450 location is specified for each frame. This allows the consumer module 415 and the video encoder 410 to synchronize their actions and avoid the need to maintain the state of a single bitstream write pointer 450 when a single bitstream write pointer 450 is used for multiple frames. In some implementations, updates to the bitstream write pointer 450 are made and communicated to the consumer module 415 by other entities other than the video encoder 410.

[0024] Referring now to FIG. 5, a block diagram of one embodiment of a computing system 500 having a video encoder coupled to a consumer module is shown. The system 500 includes at least a video encoder 510, a bitstream buffer 515, a bitstream write pointer (BWP) and a frame done flag 520, and a consumer module 525. The bitstream buffer 515 represents any type of buffer or collection of storage elements for storing encoded video bitstreams. The bitstream buffer 515 may also be located within any of a variety of memory devices or memory subsystems. For example, in one embodiment, the bitstream buffer 515 is stored in a GPU memory. In other embodiments, the bitstream buffer 515 is stored elsewhere. An exemplary routine implemented by the video encoder 510 and the consumer module 525 is shown in FIG. 5. It should be noted that this routine is merely illustrative of one embodiment. In other embodiments, other routines different from the example shown in FIG. 5 are possible and contemplated.

[0025] In one implementation, the exemplary routine begins at step 505A, where the consumer module 525 resets the BWP and frame done flag 520 at the end of a given frame and before the start of the next frame. When the video encoder 510 starts encoding the next frame, the video encoder 510 sends a write request to the bitstream buffer 515 with the encoded data at step 505B. The amount of data in each write request varies depending on the implementation. In response to the video encoder 510 sending a write request to the bitstream buffer 515, the bitstream buffer 515 sends write request completion acknowledgements ("Acks") to the video encoder 510 when the data for the write request has been written to the bitstream buffer 515. In various implementations, the data for a given frame is globally encoded in the sense that each portion of the frame data that is encoded depends on one or more other portions of the frame data. This is in contrast to prior art approaches in which slices of a frame are encoded independently of each other. As noted above, coding slices independently of one another results in boundary artifacts due to poor prediction across slice boundaries, and by globally coding the entire frame data, such boundary artifacts are eliminated, even though portions of the coded frame data are prepared for consumption during the coding process.

[0026] In response to receiving the write request complete Ack in step 505C, the video encoder 510 updates the value of the BWP in step 505D to indicate the amount of encoded data written to the bitstream buffer 515. Alternatively, in some implementations, circuitry associated with the buffer 515 updates the BWP. Also, the video encoder 510 updates the frame complete flag when the video encoder 510 completes encoding an entire frame. In one implementation, the frame complete flag 455 is a single bit, and the video encoder 510 sets the frame complete flag to a value of 1 when it reaches the end of the frame. The consumer module 525 reads the BWP and the frame complete flag 520 in step 505E to determine whether encoded data is ready for consumption, whether encoding of the entire frame is complete, and to what extent the consumer module 525 is able to read from the bitstream buffer 515 (the amount of data ready for consumption). Then, based on the value of the BWP, the consumer module 525 reads a portion of the encoded bitstream from the bitstream buffer 515 in step 505F. In some implementations, a separate data ready flag is used to indicate that the encoded data has been written to the buffer 515 and is ready for consumption. For example, by polling the data ready flag, the consumer module 525 determines that the data is ready for consumption. For example, the video encoder 510 sets the data ready flag when it receives a write request Ack. Upon reading the data from the buffer, the consumer module 525 resets the data ready flag. In such a case, the frame done flag serves to indicate whether the data that is ready for consumption represents the last part of a frame. It should be noted that the consumer module 525 can read the encoded data from the bitstream buffer 515 while the video encoder 510 is still encoding the given frame to obtain the encoded data for the given frame.Also, consumer module 525 does not need to wait until an entire frame or slice is encoded by video encoder 510 before reading the encoded data from bitstream buffer 515. This helps reduce the latency experienced by consumer module 525 when consuming encoded data from bitstream buffer 515, and also reduces the overall latency of the consumer application.

[0027]

[0033] Referring now to Figure 6, one embodiment of a method 600 for enabling on-the-fly access to an encoded video bitstream of a given video frame while the given video frame is being encoded is shown. For purposes of discussion, the steps of this embodiment and those of Figures 7-8 are shown in sequence. However, it should be noted that in various implementations of the method described, one or more of the elements described may occur simultaneously, in a different order than shown, or be omitted entirely. Other additional elements may also occur as desired. Any of the various systems or devices described herein may be configured to perform the method 600.

[0028] The video encoder begins encoding a given frame of the video sequence (block 605). Once the video encoder has encoded a portion of the given frame, the video encoder sends a write request to the bitstream buffer with the encoded data (block 610). In addition, the video encoder tracks the completion of the write request of the encoded data to the bitstream buffer (block 615). If the completion of the write request is confirmed (conditional block 620, "yes" leg), the video encoder updates the bitstream write pointer to indicate the amount of data written to the bitstream buffer (block 625). Depending on the implementation, the indication may specify the amount of data, a memory address containing the last portion of data written to the bitstream buffer, etc. Note that in one implementation, the update to the bitstream write pointer is performed as an atomic operation to prevent simultaneous access to the bitstream write pointer by both the video encoder and the consumer module. If an acknowledgement has not been received for the pending write request (conditional block 620, "no" leg), the method 600 returns to block 610.

[0029] After block 625, the video encoder determines whether the entire given frame has been encoded and written to the bitstream buffer (conditional block 630). If the entire frame has been encoded and written to the bitstream buffer (conditional block 630, "yes" leg), the video encoder sets a frame done flag (block 635). After block 635, the video encoder waits for the frame done flag to be cleared (block 640), after which the method 600 returns to block 605. In one embodiment, the consumer module clears the frame done flag. In another embodiment, another entity different from the consumer module clears the frame done flag. If the entire frame has not been encoded and written to the bitstream buffer (conditional block 630, "no" leg), the method 600 returns to block 610. Note that after the video encoder updates the bitstream write pointer following completion of the write request, the video encoder does not insert a header (e.g., a slice header) into the next portion of data to be encoded and written to the bitstream write buffer. In addition, the video encoder may perform predictions (e.g., intraframe coding predictions) across the boundary between the last portion of data that was coded and written to the bitstream write buffer and the next portion of data being coded. For example, various coding techniques use information (coded and / or uncoded) from other pixels (e.g., neighboring pixels or other pixels) within a frame to code pixel data. In this manner, coding data across boundaries (slices, chunks, etc.) reduces artifacts and provides more consistent results. Various such approaches are known to those skilled in the art, and all such approaches are contemplated.

[0030]

[0046] Referring now to FIG. 7, one embodiment of a method 700 for a consumer module to access an encoded video bitstream in real-time as the bitstream is being encoded is shown. Note that in one embodiment, method 700 is performed by the consumer module in parallel with the video encoder performing method 600. The consumer module monitors the bitstream write pointer (block 705). For example, in some implementations, the consumer module polls the bitstream write pointer to determine if an update has been received. In other implementations, a flag (e.g., a data ready flag) is used to indicate that data has been received and is ready for consumption. Note that in one implementation, the consumer module performs an atomic read at block 705 to prevent simultaneous access to the bitstream write pointer by the video encoder and the consumer module. If the consumer module detects an update to the bitstream write pointer (or if the data ready flag is set) (conditional block 710, "yes" leg), the consumer module retrieves the encoded data from the bitstream buffer up to the location indicated by the bitstream write pointer (block 715). Note that the consumer module retrieves the encoded data from the bitstream buffer for a given frame while the video encoder is still in the process of encoding the given frame. The consumer module then processes the encoded data that has been retrieved from the bitstream buffer (block 720). The consumer module performs any type of processing (e.g., streaming the encoded data over a network), depending on the implementation. If the bitstream write pointer has not been updated (conditional block 710, "no" leg), method 700 jumps to block 725.

[0031] After block 720, the consumer module monitors the frame done flag (block 725). If the frame done flag is set (conditional block 730, "yes" leg), the consumer module retrieves the final amount of encoded data from the bitstream buffer (block 735), even if the final amount is less than the specified granularity. The consumer module then clears the frame done flag (block 740). Alternatively, in another embodiment, another hardware or software module clears the frame done flag in block 740. After block 740, method 700 returns to block 705. If the frame done flag is not set (conditional block 730, "no" leg), method 700 returns to block 705.

[0032]

[0031] Referring now to FIG. 8, one embodiment of a method 800 for controlling the granularity of bitstream write pointer updates is shown. The consumer module programs a granularity setting at which updates to the bitstream write pointer are made (block 805). For example, in one embodiment, the consumer module programs the granularity setting to be equal to the maximum size of a network transmission packet payload. In other embodiments, the consumer module programs the value of the granularity setting based on other factors. The video encoder then reads the granularity setting programmed by the consumer module (block 810). The video encoder then tracks completion of write requests to the bitstream buffer during the encoding of a video frame (block 815). The video encoder then determines whether the amount of encoded data (e.g., number of bits) written to the bitstream buffer is equal to or greater than the granularity setting based on the completed write requests (conditional block 820).

[0033] If the amount of encoded data written to the bitstream buffer is equal to or greater than the granularity setting (conditional block 820, "yes" leg), the video encoder updates the bitstream write pointer to indicate the amount of encoded data written to the bitstream buffer (block 825). The video encoder also resets a count used to track the amount of encoded data written to the bitstream buffer (block 830). After block 830, the method 800 returns to block 815. Otherwise, if the amount of encoded data written to the bitstream buffer is less than the granularity setting (conditional block 820, "no" leg), the video encoder waits for more data to be written to the bitstream buffer (block 835). After block 835, the method 800 returns to block 815.

[0034]

[0046] Referring now to Figure 9, one embodiment of a method 900 for a video encoder that performs prediction across chunks being consumed is shown. The video encoder encodes a given chunk of a video frame (block 905). The consumer module then starts consuming the given chunk of the video frame (block 910). In one embodiment, the video encoder updates a buffer write pointer to inform the consumer module that there is an encoded chunk in the buffer that the consumer module can start consuming. The video encoder then starts encoding the subsequent chunk using intraframe prediction (block 915). Note that blocks 910 and 915, in one embodiment, occur simultaneously. The video encoder then finishes encoding the subsequent chunk (block 920). The consumer module then starts consuming the subsequent chunk (block 925). Note that blocks 925 and 915, in one embodiment, occur simultaneously. If there are more chunks to encode (conditional block 930, "yes" leg), method 900 returns to block 915. If there are no more chunks to encode (conditional block 930, "no" leg), method 900 ends.

[0035] Referring now to FIG. 10, a timing diagram 1000 of one embodiment of real-time consumption of chunks of an encoded video bitstream is shown. In the embodiment shown in timing diagram 1000, while encoder 1005 encodes frames 0-2, bitstream consumer 1010 can consume a given chunk of a frame while encoder 1005 encodes the next chunk of the frame. This helps reduce the latency of the operation as compared to the embodiment shown in timing diagram 200 (FIG. 2) and timing diagram 300 (FIG. 3). For example, in one embodiment, encoder 1005 encodes a chunk of frame 0 and then updates a write pointer after each chunk to indicate the amount of encoded data that has been written to the buffer. Consumer 1010 can then access and consume each chunk of frame 0 from the buffer while encoder 1005 encodes the subsequent chunk. This pattern continues for frames 1-2 and subsequent frames. The implementation shown in timing diagram 1000 helps minimize latency by allowing the consumption of encoded frames 0 to 2 to be pipelined with the encoding of frames 0 to 2. In this implementation, the bitstream consumer 1010 is able to consume a chunk of a frame at the same time that the encoder 1005 is encoding a subsequent chunk of the frame.

[0036] In various embodiments, program instructions of a software application are used to implement the methods and / or mechanisms described herein. For example, program instructions executable by a general-purpose processor or a special-purpose processor are contemplated. In various embodiments, such program instructions are expressed by a high-level programming language. In other embodiments, the program instructions are compiled from the high-level programming language into a binary language, an intermediate language, or other format. Alternatively, the program instructions are written to describe the behavior or design of hardware. Such program instructions are expressed by a high-level programming language such as C. Alternatively, a hardware design language (HDL) such as Verilog is used. In various embodiments, the program instructions are stored in any of a variety of non-transitory computer-readable storage media. The storage medium is accessible by the computing system while being used to provide the program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute the program instructions.

[0037] It should be emphasized that the above-described embodiments are merely non-limiting examples of embodiments. Once the above disclosure is fully understood, many variations and modifications will become apparent to those skilled in the art. It is intended that the following claims be interpreted to embrace all such variations and modifications.

Claims

1. 1. A system comprising: A buffer; a consumer module including a circuit; The circuit comprises: programming a granularity specifying a condition for providing an indication to notify the consumer module that one or more chunks of encoded data, less than an entire encoded video frame, are ready to be consumed by the consumer module from the buffer; responsive to detecting the indicator, obtaining one or more chunks of encoded data from the buffer; 4. The method of claim 3, system.

2. the circuitry of the consumer module is configured to receive the indication before an entire given video frame that includes a portion corresponding to the one or more chunks is encoded. The system of claim 1.

3. the granularity specifying a size of one or more chunks of encoded data to be stored in the buffer; The system of claim 2.

4. the circuitry of the consumer module is configured to receive an indication of an amount of data contained in the one or more chunks. The system of claim 1.

5. The granularity specifies a size equal to a maximum size of a network transmission packet payload. The system of claim 3.

6. the consumer module circuitry is configured to, in response to detecting that a frame done flag is set indicating that encoding of a given video frame is complete, retrieve the final amount of encoded data from the buffer if the final amount of encoded data is less than the granularity; The system of claim 3.

7. the consumer module circuitry is configured to clear the frame done flag in response to retrieving a final amount of the encoded data from the buffer. The system of claim 6.

8. programming a granularity specifying conditions for circuitry of a consumer module to provide an indication to notify said consumer module that one or more chunks of encoded data, less than an entire encoded video frame, are ready to be consumed by said consumer module from a buffer; and retrieving one or more chunks of encoded data from the buffer in response to detecting the indicator, the consumer module circuitry. method.

9. a circuit in the consumer module receiving the indication before an entire given video frame including a portion corresponding to the one or more chunks is encoded; 9. The method of claim 8.

10. the granularity specifying a size of one or more chunks of encoded data to be stored in the buffer; 10. The method of claim 9.

11. the consumer module circuitry receiving an indication of an amount of data contained in the one or more chunks; 9. The method of claim 8.

12. The granularity specifies a size equal to a maximum size of a network transmission packet payload. The method of claim 10.

13. and in response to circuitry of the consumer module detecting that a frame done flag is set indicating that encoding of a given video frame is completed, if the final amount of encoded data is less than the granularity, obtaining the final amount of encoded data from the buffer. The method of claim 10.

14. circuitry of the consumer module clearing the frame done flag in response to retrieving a final amount of the encoded data from the buffer.

14. The method of claim 13.

15. An apparatus comprising a circuit, The circuit comprises: programming a granularity that specifies a condition for providing an indication to inform the device that one or more chunks of encoded data, less than an entire encoded video frame, are ready to be consumed by the device from a buffer; responsive to detecting the indicator, obtaining one or more chunks of encoded data from the buffer; 4. The method of claim 3, Device.

16. the circuitry is configured to receive the indication before an entire given video frame that includes a portion corresponding to the one or more chunks is encoded.

16. The apparatus of claim 15.

17. the granularity specifying a size of one or more chunks of encoded data to be stored in the buffer; 17. The apparatus of claim 16.

18. the circuitry being configured to receive an indication of an amount of data contained in the one or more chunks; 16. The apparatus of claim 15.

19. the circuitry is configured to retrieve a final amount of encoded data from the buffer if the final amount of encoded data is less than the granularity in response to a frame done flag being set indicating that encoding of a given video frame is completed; 20. The apparatus of claim 17.

20. the circuitry is configured to clear the frame done flag in response to retrieving a final amount of the encoded data from the buffer.

20. The apparatus of claim 19.

Images