Low latency coding using bypass substream and entropy coded substream

Abstract

The invention relates to low latency coding using a bypass substream and an entropy coded substream. A system includes an encoder configured to entropy encode a bitstream including both compressible symbols and non-compressible symbols. The encoder parses the bitstream into a compressible symbol substream and a non-compressible substream. When entropy encoding the compressible symbol substream, the non-compressible symbol substream bypasses an entropy encoding component of the encoder. When accumulating a certain amount of bytes of entropy-encoded symbols and bypass symbols, the accumulated entropy-encoded symbol bytes and bypass bytes are used to form blocks of fixed or known size without waiting for the encoder to process the complete bitstream. In a complementary manner, a decoder reconstructs the bitstream according to packets or blocks.

Description

[0001] This application is a divisional application of the invention patent application with application number 202080070069.5, filed on October 2, 2020, and invention name “Low-latency coding using bypass sub-stream and entropy coded sub-stream”. Technical Field

[0002] The present disclosure generally relates to compression and decompression of bitstreams using entropy-based encoders. Background Art

[0003] Various devices, including but not limited to personal computer systems, desktop computer systems, laptops and notebook computers, tablet computers or tablet devices, digital cameras, digital video recorders, and mobile phones or smart phones, may include software and / or hardware that implements encoding or decoding processes using entropy coding techniques such as arithmetic coding, Huffman decoding, etc. For example, a device may include a device (e.g., an integrated circuit (IC), such as a system on a chip (SOC) or a subsystem of an IC) that can receive and process digital inputs from one or more sources according to one or more processing methods involving entropy-based coding and output processed information (e.g., pictures / frames or picture / frame regions) or other information. For another example, a software program that can receive and process digital inputs from one or more sources according to one or more processing methods involving entropy-based coding and output processed information can be implemented on the device. Summary of the invention

[0004] In some embodiments, a system for encoding a bitstream includes a computing device configured to separate a symbol stream to be encoded into: (a) a compressible symbol substream and (b) a non-compressible symbol substream. In order to encode the bitstream, the computing device is also configured to entropy encode the symbols of the compressible symbol substream and allow the incompressible symbol substream to bypass the encoder, which entropy encodes the compressible symbol substream. In addition, the computing device is configured to form a block including entropy encoded symbols from the compressible symbol substream and non-compressible symbols from the non-compressible symbol bypass substream, and in response to the accumulated amount of entropy encoded symbols and non-compressible symbols satisfying the payload size of the payload of the block, provide the block for transmission or storage. In some embodiments, the block can be formed into a packet for transmission over a network. In addition, in some embodiments, the system may also include a bypass buffer and an entropy encoded symbol buffer, wherein the bypass byte and the entropy encoded symbol byte are written to the corresponding buffer and concatenated to form a block or packet in response to the accumulated amount of bytes satisfying the payload size of the block or packet. In other embodiments, the bypass bytes and entropy encoded symbol bytes may be written directly into a block or packet, with one substream written in forward order from one end of the block or packet and the other substream written in reverse order from the other end of the block or packet.

[0005] In some embodiments, a system for decoding a bitstream includes a computing device configured to receive a packet including a payload, the payload including an entropy coded symbol portion and a non-entropy coded symbol portion. The packet also includes a header indicating the length of the payload filled with entropy coded symbols or the length of the payload filled with non-entropy coded symbols. In order to decode the packet, the computing device is configured to determine the starting point of the entropy coded symbols in the payload and the starting point of the non-entropy coded symbols in the payload based on the header, read the entropy coded symbols from the determined starting point of the entropy coded symbols and decode the read entropy coded symbols, read the non-entropy coded symbols from the determined starting point of the non-entropy coded symbols, and combine the decoded entropy coded symbols and non-entropy coded symbols into a decoded symbol stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1A An exemplary encoding loop of an encoder according to some embodiments is shown.

[0007] Figure 1B Example blocks or packets encoded by an encoder are shown according to some embodiments.

[0008] Figure 1C An exemplary decoding loop of a decoder according to some embodiments is shown.

[0009] Figure 1D An exemplary bitstream including compressible symbols and non-compressible symbols before being encoded and a set of blocks or packets produced by encoding the bitstream are shown according to some embodiments.

[0010] Figure 2 An exemplary encoding loop of an encoder that writes directly to packets or blocks is shown according to some embodiments.

[0011] Figure 3 An exemplary encoding loop of an encoder that writes bypass bits to a reserved bypass byte is shown according to some embodiments.

[0012] Figure 4 An example packet or block including reserved bypass bytes is shown according to some embodiments.

[0013] Figure 5 An exemplary process for encoding a symbol stream including compressible symbols and non-compressible symbols is shown according to some embodiments.

[0014] Figure 6An exemplary process for decoding a payload of a packet including an entropy coded symbol portion and a non-entropy coded symbol portion is shown in accordance with some embodiments.

[0015] Figure 7 An exemplary computer system is shown that may implement an encoder or decoder according to some embodiments.

[0016] This specification includes references to "one embodiment" or "an embodiment." The appearance of the phrase "in one embodiment" or "in an embodiment" does not necessarily refer to the same embodiment. The particular features, structures or characteristics may be combined in any suitable manner consistent with the present disclosure.

[0017] The term "comprising" is open ended. As used in the appended claims, the term does not exclude additional structures or steps. Consider the following cited claim: "an apparatus comprising one or more processor units..." Such a claim does not exclude the apparatus from including additional components (e.g., a network interface unit, a graphics circuit, etc.).

[0018] "Configured to", various units, circuits or other components may be described or described as "configured to" perform one or more tasks. In such contexts, "configured to" is used to imply a structure (e.g., a circuit) that performs the one or more tasks during operation by indicating that the unit / circuit / component includes the structure. In this way, the unit / circuit / component is said to be configured to perform the task even when the specified unit / circuit / component is currently inoperable (e.g., not turned on). The units / circuits / components used with the "configured to" language include hardware-such as circuits, memories storing executable program instructions to implement operations, etc. Reference to a unit / circuit / component "configured to" perform one or more tasks is explicitly intended not to invoke 35 U.S.C. §112 (f) for the unit / circuit / component. In addition, "configured to" may include a general structure (e.g., a general circuit) manipulated by software and / or firmware (e.g., an FPGA or a general processor executing software) to operate in a manner capable of performing one or more tasks to be solved. "Configured to" may also include adapting a manufacturing process (eg, a semiconductor fabrication facility) to produce a device (eg, an integrated circuit) suitable for implementing or performing one or more tasks.

[0019] "First," "second," etc. As used herein, these terms act as labels for the nouns that precede them, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing a write operation of a "first" value and a "second" value. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.

[0020] "Based on". As used herein, this term is used to describe one or more factors that influence a determination. This term does not exclude additional factors that influence a determination. That is, a determination may be based solely on these factors or at least in part on these factors. Consider the phrase "A is determined based on B". In this case, B is a factor that influences the determination of A, and such a phrase does not exclude that the determination of A may also be based on C. In other examples, A may be determined based solely on B. DETAILED DESCRIPTION

[0021] Entropy coding, such as arithmetic coding, Huffman coding, etc., can be a bottleneck for processing images, videos, point clouds, and / or other types of data. For example, decoding entropy-coded data can be a bottleneck for applications that utilize image, video, or point cloud data. Moreover, entropy encoding and decoding can be highly serializable and computationally expensive processes.

[0022] In some embodiments, the entropy encoder may parse the incoming symbol bitstream to be compressed / encoded into the compressible symbol substream and the non-compressible symbol substream. Because arithmetic coding uses symbol occurrence probabilities to compress symbols, some symbols in the bitstream that have even or no occurrence probabilities may not benefit from entropy encoding in terms of compression efficiency. However, such non-compressible symbols may still require computational resources and time to be processed by the entropy encoding component of the encoder. Therefore, compression efficiency and processing time can be reduced by excluding non-compressible symbols from the symbol substream to be entropy encoded by the entropy encoding component of the encoder, and instead directing the non-compressible symbols to the bypass substream.

[0023] In some embodiments, further improvements in latency can be achieved by forming blocks or groups from the encoded symbols of the compressible symbol bitstream that have been encoded by the entropy encoding component and from the non-compressible symbols of the bypass bitstream. For example, instead of waiting to process the entire bitstream, the already processed portions of the bitstream can be formed into blocks or groups, thereby reducing encoding and decoding latency. The decoding latency can be reduced because the decoder does not need to buffer the entire bitstream before decoding the entropy encoded compressible symbols and combining the decoded compressible symbols with the non-compressible symbols to reconstruct the original bitstream.

[0024] In some embodiments, a packet or block may have a fixed size or may have a known size that is signaled to a decoder. For example, in some embodiments, a packet or block may have a fixed size of 256 bytes.

[0025] In some embodiments, a packet or block may include a header. The header may indicate the length of the payload portion of the packet or block filled with entropy coded symbol bytes, or may indicate the portion of the packet or block filled with bypass bytes. Because the packet has a fixed or known payload size, the decoder may deduct the indicated portion length from the fixed or known payload size to determine the length of another portion of these portions. In addition, the decoder may use this information to determine the demarcation point or segmentation point between these portions. Therefore, the decoder may be able to determine the starting point in the payload of two portions (e.g., a compressible symbol portion and a non-compressible symbol bypass portion of entropy coding). Because the decoder knows where the two portions start, the decoder can read from the two portions at the same time without having to buffer one of the two portions first. For example, an alternative strategy is to encode one portion in forward order and encode another portion in reverse order. However, in such embodiments, the decoder may need to buffer the full length of the reverse sorted portion to reach the starting point from which it is read. However, in such embodiments, the corresponding size of the portion in the header of the block or packet may not need to be signaled.

[0026] In some embodiments, the block or packet size (S_p) controls the overall system latency. In some embodiments, the block or packet size (S_p) may be fixed or may be adaptively selected. Additionally, the block or packet size (S_p) may be implicitly defined or explicitly signaled in the bitstream header.

[0027] In some embodiments using a 256-byte packet or block size, the payload size may be 255 bytes and the header size may be 1 byte. Even if a portion of the payload of the packet or block is entropy encoded, the header information may not be entropy encoded. In some embodiments, the encoding order (forward or reverse) of the two parts (e.g., the entropy encoded symbol part and the bypassed non-compressible symbol part) may be fixed or adaptively switched based on implicit rules or explicitly signaled in the bitstream.

[0028] In some embodiments, the header may be placed at the beginning or end of a packet or block. This is possible because the packet or block has a known size so that the decoder can locate the header as the last byte of a known size packet or block.

[0029] In some embodiments, the behavior of the encoder when constructing a sequence of packets or blocks has a significant impact on the resource requirements of the decoder and the end-to-end system delay. In particular, there is a delay between encoding a symbol using an entropy encoder and the representation of the symbol appearing in its bit-by-bit output as part of the internal renormalization process of the entropy encoder. Similarly, there is a delay between generating a bit (entropy encoded or bypassed) and combining eight bits to form a byte. In addition, there may be data dependencies between symbols decoded by the entropy encoder and bypass bits (e.g., bits corresponding to non-compressible symbols included in the bypass substream).

[0030] Figure 1A An exemplary encoding loop of an encoder according to some embodiments is shown.

[0031] In some embodiments, the encoding process may include separating the bit stream into compressible symbols and non-compressible symbols. The compressible symbols may be provided to an entropy encoder (such as an arithmetic encoder), which outputs bytes to an entropy encoded symbol buffer. For example, the encoder 100 receives symbols 120 to be encoded, and the symbol parser 102 parses these symbols into a first substream 122 including compressible symbols and a second substream 124 including non-compressible symbols. The compressible substream 122 is entropy encoded via the entropy encoder 108 and output to the encoded symbol buffer 110. In addition, a bypass substream 124 may be added to the bypass buffer 104. When enough bytes have been accumulated in the encoded symbol buffer 110 or the bypass buffer 104 to add these bytes to form a packet or block, the multiplexer 106 combines the encoded symbols and non-compressible symbols from the corresponding buffer into a fixed or known length (e.g., 256 bytes) of the block or packet. Although not shown, the encoder 100 may further include a means for including a header in the formed packet or block, and indicating the size of at least one of the parts included in the packet or block (e.g., the length of the entropy-encoded symbol part or the length of the non-compressible symbol part) in the header. For example, Figure 1B An exemplary grouping is shown in .

[0032] In some embodiments, the bytes output to the encoded symbol buffer 110 may be labeled (S_ac), and the bytes output to the bypass buffer 104 may be labeled (S_bp). When the sum of the sizes of bytes used for the two buffers (e.g., S_ac+S_bp) is equal to the packet size S_p (e.g., S_p=255), a new packet may be created by concatenating the bytes stored in the encoded symbol buffer 110 or the bypass buffer 104. A fixed-size header may also be added at the beginning of the packet to specify the number of entropy-encoded bytes (e.g., arithmetic-encoded bytes) S_ac. The generated packets may be streamed directly to the decoder side without any additional encoding delay.

[0033] Figure 1B Example blocks or packets encoded by an encoder are shown according to some embodiments.

[0034] The packet 130 includes a header 132 and a payload 134. The payload includes an entropy coded symbol portion 136 and a bypassed non-compressible symbol portion 138. The header indicates the length of the entropy coded portion 136 of the payload 134 of the packet 130 or the length of the bypassed non-compressible symbol portion 138 of the payload 134 of the packet 130. In some embodiments, the packet may have a fixed or known overall length 140, which may be 256 bytes, and the header 132 may have a known length, such as one byte.

[0035] Figure 1C An exemplary decoding loop of a decoder according to some embodiments is shown.

[0036] In some embodiments, a decoder such as decoder 150 decodes packets or blocks in the order in which they were received as soon as they are received. Packet parser 152 of decoder 150 may read a fixed size header of an incoming packet or block to determine the corresponding sizes of portions of the payload of the packet or block, such as S_ac and S_bp, where the length of one portion may be determined by subtracting the length of another portion from a known or fixed packet length of the payload of the packet. For example, S_bp may be equal to S_p-S_ac.

[0037] In some implementations, packet parser 152 may output entropy encoded byte stream 160 to entropy decoder 158 via entropy encoded buffer 168, and may output bypass byte stream 162 to bypass buffer 154. In some implementations, entropy encoded bytes decoded by entropy encoder 158 may have decoding dependencies on bits included in bypass bytes in bypass buffer 154. For example, Figure 1C Dependencies 164 are shown. In some implementations, entropy coded buffer 168 may buffer entropy coded byte stream 160 while waiting for bypass bytes that include bits that entropy coded bytes depend on.

[0038] The entropy decoded symbols from entropy decoder 158 may be combined with bypass bytes from bypass buffer 154 at multiplexer 156 to form a reconstructed stream of decoded symbols 166 .

[0039] Figure 1D An exemplary bitstream including compressible symbols and non-compressible symbols before being encoded and a set of blocks or packets produced by encoding the bitstream are shown according to some embodiments.

[0040] In some embodiments, an incoming bitstream such as bitstream 120 may include both compressible symbols and non-compressible symbols, as shown in bitstream 170. For example, bitstream 170 includes compressible symbols 172 and non-compressible symbols 174. An encoder such as encoder 100 may output a sequence of packets to represent bitstream 170. For example, the encoder may output a sequence of packets 180, where each packet includes header 132, entropy encoded compressible symbols 136, and bypassed non-compressible symbols 138.

[0041] Figure 2 An exemplary encoding loop of an encoder that writes directly to packets or blocks is shown according to some embodiments.

[0042] In some implementations, encoded symbols from the compressible symbol substream 122 that have been entropy encoded by the entropy encoder 108 may be written directly to a packet such as the packet 130. For example, the entropy encoded symbols may be written in forward order in the entropy encoded bytes 136. Additionally, bypass symbols from the non-compressible symbol substream 124 may be written directly to the packet 130 in reverse order in the bypass bytes 138.

[0043] In some implementations, the symbol parser 102 may further parse or receive the contexts 202 and provide these contexts to the entropy encoder 108 for use in encoding the compressible symbols of the compressible symbol substream 122 .

[0044] Figure 3 An exemplary encoding loop of an encoder that writes bypass bits to a reserved bypass byte is shown according to some embodiments.

[0045] In some scenarios, a decoder such as decoder 150 may be required to buffer the entropy coded stream 160 or bypass stream 162 to resolve the dependency. Alternatively, the decoder may buffer the packet until the required packet is received to resolve the dependency. However, in some cases, this may increase the decoding delay of the decoder. For example, imagine the extreme case of a symbol sequence {x_0, x_1, ...}, where x_0 is a bypass bit and x_1 is an arithmetic coded bit with a decoding dependency on x_0 (e.g., the context of x_1 is determined by the value of x_0). If there are no more bypass bits in the formed packet, the bypass byte will not be written in the first packet. This is because the bypass buffer 104 may need to accumulate 8 bypass bits to form a bypass byte that can be written to the packet. In fact, the first bypass byte may not be written until seven more bypass bits are generated. This delay is theoretically unbounded, which means that the decoder may have to buffer a significant number of packets (or worse, all packets) before decoding can be resumed.

[0046] In some embodiments, such concerns are addressed by reserving at least one bypass byte in the formed packet. In one embodiment, decoding delay / resources are bounded by reserving bytes in the current packet and back-propagating "future" bypass bits.

[0047] For example, whenever a bypass symbol is sent to the encoder, the following process can be performed:

[0048] o If the encoder's byte-by-byte bypass bit accumulator is empty, reserve one byte in the current packet for use as a bypass byte (eg, increment S_bp). Record the position of the reserved bypass byte as P_bp.

[0049] ○ Append bits to the accumulator byte

[0050] ○ If the accumulator byte is full, the completion byte is written to the reserved position P_bp

[0051] Only when the packet is fully allocated (S_bp+S_ac==S_p) and any reserved bypass bytes have been completed, the packet can be transmitted. (Additional entropy coded bytes are written to subsequent packets.) Note that P_bp can be a pointer to a buffer location. Also note that storage for the bypass bin accumulator can occupy reserved bytes.

[0052] For example, Figure 3 A packet 130 is shown with a reserved bypass byte 138 comprising eight bits 302. Two of these bits have been written with non-compressible symbols, while the other six bits have not yet been filled.

[0053] While the above defines the decoding resources, it imposes a burden on the encoder (albeit slightly less than that of an equivalent decoder). An alternative embodiment that allows both encoder and decoder resources and system latency to be constrained is as follows. Each packet containing at least one bypass byte (e.g., S_bp>0) is composed of an additional header indicating the number of valid bits in the final byte of the bypass sub-buffer. The header may, for example, include the first three bits of the first bypass byte of each packet, or alternatively, the last three bits of the last byte. This mechanism allows the encoder to instead use partial bytes to refresh packets without waiting for any reserved bytes to complete.

[0054] More generally, the encoder may select any combination of entropy coded bytes and bypass bytes to form packets according to certain design constraints.To ensure that the decoder has the resources to decode such a stream, the maximum number of buffers that the decoder may need for buffering may be signaled in a parameter set.

[0055] Figure 4Shows an exemplary packet or block including reserved bypass bytes according to some embodiments.

[0056] For example, 402 of packet 130 may indicate that bits 1, 2, and 3 of reserved bypass byte 302 are valid. In such a scenario, the encoder may not have to buffer packet 130 until all 8 bits of the reserved bypass byte 302 are filled, but may instead provide packet 130 for transmission or storage before the reserved bypass byte is filled and, conversely, before it is indicated in header 402 which bits in the reserved bypass byte 302 are valid bits. For example, bit 404 may be valid while bit 406 may be an invalid empty or padding bit.

[0057] In some embodiments, additional enhancements may include signaling both S_ac and S_bp for the last packet of a packet sequence. For example, the last packet may have a number of useful bytes less than S_p.

[0058] In some embodiments, the encoder may pad the last packet with additional bytes so that the last packet has the size of S_p. The padding bytes may be discarded on the decoder side. The information needed to know when to stop decoding without using the padding bytes may be implicitly derived or explicitly signaled in the bitstream or in another layer. Another alternative would include padding the packet with zero bytes or one byte.

[0059] In some embodiments, the encoder may explicitly write the size of the packet (for packets smaller than normal) in the header. Such information may also be handled by the file format / network transport layer.

[0060] In some embodiments, the encoder may explicitly write the size S_bp in the header. Note that if two headers are supported, the headers may be placed together (at the start), but the second header (for S_bp) may also be placed at the position (header_size_ap + 1 + S_ap) (of course if that position < S_p). Also note that if only one byte remains, a 0 (empty bypass) may be entered in it.

[0061] The method selected to handle the last packet may be implicitly specified or explicitly signaled in the high-level syntax header / parameter set.

[0062] In some embodiments, signaling both S_ac and S_bp allows data to be signaled without having to wait for a complete packet to be filled (e.g., sending available data with a predefined delay). Although this method may waste some bytes, it may be desirable for some low-latency applications. Additionally, signaling both S_ac and S_bp allows packets corresponding to a particular frame to be sent without having to interleave the frame with the next frame. Whether to signal a single size or two sizes can be specified in the high-level syntax header / parameter set.

[0063] Alternative constructions that do not require signaling of both S_ac and S_bp are possible. For example, one embodiment assumes a transmission system that can identify the length of the entire data unit containing the packet. In this case, it can be clearly identified that the last packet exceeds the data unit length, and the value S_p is trimmed so that the packet adapts to the data unit. This process works with a dual forward sub-buffer structure or a forward plus reverse sub-buffer method. In another embodiment, the embodiment relies on the self-termination characteristics of the depacketization process, but it is necessary to write two sub-streams in the forward direction. In this construction, the termination of the parsing process causes all valid data to be consumed. This construction allows additional data to be after the truncated packets without using the method outlined in the present disclosure.

[0064] As mentioned above, in some cases, there may be dependencies between packets where decoding of entropy coded symbols requires some symbols included in the bypass stream. To resolve these dependencies, some of the following techniques may be used:

[0065] o The packet header may signal whether the probability of a context is "restarted" (ie, reset to a default value). Allowing the probability to be restarted allows the associated packets to be decoded independently,

[0066] It is called independent grouping (as opposed to dependent grouping).

[0067] ○ For independent packets, the encoder ensures that a symbol is not encoded on two packets.

[0068] o For fault tolerance reasons, this previous requirement can also be extended to dependent groups.

[0069] o The header information specifying whether the probabilities should be reset can be extended to explicitly specify the initial probability values.

[0070] For example, instead of using the default probability state (i.e., uniform probability for all symbols),

[0071] Many presets favoring different symbol statistics can be defined.

[0072] o This feature can be turned on and off via higher level syntax structures (eg, sequence parameter sets, slice / tile / block headers).

[0073] In some embodiments, the dependent / independent packet mechanism may enable the implementation of a wavefront process. In this case, another header information specifying the location of the independent packets may be introduced.

[0074] In some embodiments, the packet ordering information may be processed in at least the following ways:

[0075] o The packet order is explicitly specified in the bitstream as header information. For example, a certain number N bytes (e.g., 1 byte) may be used to indicate the order, which specifies the packet index modulo the number 2^N. This may be similar to the way picture order is counted for a video codec, which splits the information into a signaled LSB element and an implicitly derived MSB component.

[0076] o Obtain information from another system level (e.g., the network layer in the case of TCP packets).

[0077] o The order is implicitly derived from the order in which the packets were received.

[0078] In some embodiments, the packet ordering information may be combined with the probability resetting information described in the previous section to achieve a more compact header. For example, 1 to 2 bits are used for the probability information, and the remaining bits (e.g., 7 to 6 bits) are used for the ordering information.

[0079] In some implementations, an exemplary bitstream syntax may be as follows:

[0080] Bitstream Syntax

[0081]

[0082] chunk_padding_enabled_flag equal to 1 indicates that the last byte of a chunk containing fewer than 254 chunk_ae_byte elements contains a variable number of padding bits. chunk_flush_enabled_flag equal to 0 indicates that chunk_bypass_5bits and chunk_bypass_pad are not present in the bitstream.

[0083] chunk_num_ae_bytes indicates the number of chunk_ae_byte and chunk_bypass_byte elements present in the chunk.

[0084] The variable NumChunkBypassBytes is derived as follows:

[0085] Max(0,(chunk_padding_enabled_flag?253:254)-chunk_num_ae_bytes)

[0086] hunk_ae_byte[i] specifies the i-th byte of the arithmetic coded symbol substream of the current chunk. Each chunk_ae_byte[i] is appended to the AeByteStream array as follows:

[0087] for(i=0;i <chunk_num_ae_bytes;i++)

[0088] AeByteStream[AeStreamLen++]=chunk_ae_byte[i]

[0089] chunk_bypass_byte[j] specifies the jth byte of the bypass symbol substream of the current chunk. Append each chunk_bypass_byte to the BypassBitStream array as follows:

[0090] for(j=0; j <NumChunkBypassBytes;j++)

[0091] for(b=7;b>=0;b--)

[0092] BypassBitStream[BypassBitStreamLen++]=(chunk_bypass_byte[j]>>b)&1

[0093] chunk_bypass_5bits specifies the value of up to five bypass bits at the end of the bypass symbol substream for the current chunk. Append each bit to the BypassBitStream array as follows:

[0094] for(b=4;b>=0;b--)

[0095] BypassBitStream[BypassBitStreamLen++]=(chunk_bypass_5bits>>b)&1

[0096] chunk_bypass_num_flushed_bits specifies the number of bypass bits to be discarded from the end of the BypassBitStream:

[0097] BypassBitstreamLen-=chunk_bypass_num_flushed_bits

[0098] In some implementations with reverse order bypass bytes, an exemplary bitstream syntax may be as follows:

[0099] Bitstream Syntax

[0100]

[0101] chunk_num_ae_bytes is the number of chunk_ae_byte elements present in the chunk. The value of chunk_num_ae_bytes indicates the number of chunk_bypass_byte elements present in the chunk.

[0102] The variable NumChunkBypassBytes is derived as follows:

[0103] NumChunkBypassBytes=Max(0,256-2-chunk_num_ae_bytes)

[0104] hunk_ae_byte[i] specifies the i-th byte of the arithmetic coded symbol substream of the current chunk. Each chunk_ae_byte[i] is appended to the AeByteStream array as follows:

[0105] for(i=0;i <chunk_num_ae_bytes;i++)

[0106] AeByteStream[AeStreamLen++]=chunk_ae_byte[i]

[0107] chunk_bypass_byte[j] specifies the jth byte of the bypass symbol substream of the current chunk. Append each chunk_bypass_byte to the BypassBitStream array as follows:

[0108] for(j=NumChunkBypassBytes-1; j>=0; j--)

[0109] for(b=7;b>=0;b--)

[0110] BypassBitStream[BypassBitStreamLen++]=(chunk_bypass_byte[j]>>

[0111] b)&1

[0112] chunk_bypass_5bits specifies the value of up to five bypass bits at the end of the bypass symbol substream for the current chunk. Each bit is appended to the BypassBitStream array (after chunk_bypass_bytes is appended) as follows:

[0113] for(b=4;b>=0;b--)

[0114] BypassBitStream[BypassBitStreamLen++]=(chunk_bypass_5bits>>b)

[0115] &1

[0116] chunk_bypass_num_flushed_bits specifies the number of bypass bits to be discarded from the end of the BypassBitStream:

[0117] BypassBitstreamLen-=chunk_bypass_num_flushed_bits

[0118] Exemplary Encoding Process

[0119] Figure 5 An exemplary process for encoding a symbol stream including compressible symbols and non-compressible symbols according to some embodiments is shown. At box 502, an encoder receives a symbol stream for entropy encoding. The encoder may be implemented in hardware or may be implemented via program instructions of one or more processors executing a computing device. At box 504, a parser of the encoder separates the symbol stream into a first substream including compressible symbols and a second substream including non-compressible symbols. At box 506, entropy encoding is performed on the symbols of the compressible symbol substream. It should be noted that the non-compressible symbol substream bypasses the entropy encoding step of box 506. At box 508, the entropy encoded symbols encoded at 506 and the non-compressible symbols of the second substream separated into non-compressible symbols at box 504 are then combined in a block of the payload of the packet. At box 510, additional entropy encoded symbols and additional non-compressible symbols that have not yet been entropy encoded are accumulated in the block formed at 508. For example, entropy coded symbols may be added to a block from the front of the block in a direction toward the back of the block, and non-compressible symbols may be added to a block from the back of the block in a direction toward the front of the block, such as Figure 1B At block 512, in response to the accumulated entropy coded symbols and the non-compressible symbols accumulated in the block reaching a payload size of the block (such as 256 bytes), the block is provided for transmission, such as in a payload of a packet, or the block is provided for storage.

[0120] Exemplary Decoding Process

[0121] Figure 6 An exemplary process for decoding a payload of a packet including an entropy coded symbol portion and a non-entropy coded symbol portion according to some embodiments is shown. At block 602, a decoder receives a packet of a stream to be decoded. The decoder may be implemented in hardware or may be implemented via program instructions executed by one or more processors. The packet includes a payload, which includes an entropy coded symbol portion and a non-entropy coded symbol portion. The packet also includes a header indicating the length of the entropy coded symbol portion or the length of the non-entropy coded symbol portion. At block 604, the decoder determines the corresponding starting points of the entropy coded symbol portion and the non-entropy coded symbol portion based on the length indicated in the header. For example, if the payload has a known block size, and the length of the entropy coded symbol portion or the length of the non-entropy coded symbol portion is known (e.g., via information included in the header), the starting point of one of these portions will be the start or end of the block, and the starting point of another of these portions may be determined based on the known block size of the payload minus the known portion length.

[0122] At block 606, the decoder reads the entropy encoded symbols from the determined starting point of the entropy encoded symbol portion.At block 608, the decoder entropy decodes the entropy encoded symbols read at block 606 from the entropy encoded symbol portion.

[0123] At block 610 , non-entropy coded symbols are read from the determined starting point of the non-entropy coded symbol portion of the payload. Note that the non-entropy coded symbols bypass the entropy decoding step at block 608 .

[0124] At block 612, the entropy decoded symbols and the non-entropy encoded symbols are combined into a decoded symbol stream that is provided as an output of the decoder.

[0125] Exemplary Computer System

[0126] Figure 7 An exemplary computer system 700 is shown that can implement an encoder or decoder or any other of the components described herein (e.g., as described above with reference to FIGS. 1 to 4). Figure 6700 may be configured to perform any or all of the above embodiments. In various embodiments, the computer system 700 may be any of various types of devices, including, but not limited to, a personal computer system, a desktop computer, a laptop computer, a notebook computer, a tablet computer, an all-in-one computer, a tablet or netbook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a camera, a set-top box, a mobile device, a consumer device, a video game console, a handheld video game device, an application server, a storage device, a television, a video recording device, a peripheral device (such as a switch, a modem, a router), or generally any type of computing or electronic device.

[0127] Various embodiments of the encoder or decoder as described herein may be executed in one or more computer systems 700, which may interact with various other devices. Figure 6 Any component, action or function described may be configured as Figure 7 700. In the illustrated embodiment, the computer system 700 includes one or more processors 710 coupled to a system memory 720 via an input / output (I / O) interface 730. The computer system 700 also includes a network interface 740 coupled to the I / O interface 730 and one or more input / output devices 750 such as a cursor control device 760, a keyboard 770, and a display 780. In some cases, it is contemplated that the embodiments may be implemented using a single instance of the computer system 700, while in other embodiments, multiple such systems or multiple nodes constituting the computer system 700 may be configured to host different parts or instances of the embodiments. For example, in one embodiment, some elements may be implemented via one or more nodes of the computer system 700 that are different from those nodes that implement other elements.

[0128] In various embodiments, computer system 700 may be a uniprocessor system including one processor 710, or a multiprocessor system including a number of processors 710 (e.g., two, four, eight, or another suitable number). Processor 710 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 710 may be a general-purpose or embedded processor that implements any of a variety of instruction set architectures (ISAs), such as x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISAs. In a multiprocessor system, each of processors 710 may typically, but not necessarily, implement the same ISA.

[0129] The system memory 720 may be configured to store point cloud compression or point cloud decompression program instructions 722 and / or sensor data accessible by the processor 710. In various embodiments, the system memory 720 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory. In the illustrated embodiment, the program instructions 722 may be configured to implement an image sensor control application that incorporates any of the above-described functionalities. In some embodiments, the program instructions and / or data may be received, sent, or stored on different types of computer accessible media or similar media separate from the system memory 720 or the computer system 700. Although the computer system 700 is described as implementing the functionality of the functional blocks of the preceding figures, any functionality described herein may be implemented via such a computer system.

[0130] In one embodiment, the I / O interface 730 may be configured to coordinate I / O communications between the processor 710, the system memory 720, and any peripheral devices (including the network interface 740 or other peripheral device interfaces, such as the input / output device 750) in the device. In some embodiments, the I / O interface 730 may perform any necessary protocol, timing, or other data conversion to convert data signals from one component (e.g., the system memory 720) into a format suitable for use by another component (e.g., the processor 710). In some embodiments, the I / O interface 730 may include support for devices attached, for example, via various types of peripheral buses (such as, variations of the peripheral component interconnect (PCI) bus standard or the universal serial bus (USB) standard). In some embodiments, the functionality of the I / O interface 730 may be split into two or more separate components, such as a north bridge and a south bridge, for example. In addition, in some embodiments, some or all of the functionality of the I / O interface 730 (such as an interface to the system memory 720) may be directly incorporated into the processor 710.

[0131] The network interface 740 may be configured to allow data to be exchanged between the computer system 700 and other devices (e.g., carriers or proxy devices) attached to the network 785 or between nodes of the computer system 700. In various embodiments, the network 785 may include one or more networks, including, but not limited to, a local area network (LAN) (e.g., an Ethernet or an enterprise network), a wide area network (WAN) (e.g., the Internet), a wireless data network, some other electronic data network, or some combination thereof. In various embodiments, the network interface 740 may support communication via a wired or wireless general data network (such as any suitable type of Ethernet network), for example; communication via a telecommunications / telephone network (such as an analog voice network or a digital fiber optic communication network); communication via a storage area network (such as a Fiber Channel SAN), or communication via any other suitable type of network and / or protocol.

[0132] In some embodiments, input / output devices 750 may include one or more display terminals, keyboards, keypads, touch pads, scanning devices, voice or optical recognition devices, or any other device suitable for inputting or accessing data by one or more computer systems 700. Multiple input / output devices 750 may be present in computer system 700, or may be distributed across various nodes of computer system 700. In some embodiments, similar input / output devices may be separate from computer system 700 and may interact with one or more nodes of computer system 700 via a wired or wireless connection (such as via network interface 740).

[0133] like Figure 7 As shown, the memory 720 may include program instructions 722, which may be executed by the processor to implement any of the above elements or actions. In one embodiment, the program instructions may execute the method described above. In other embodiments, different elements and data may be included. It should be noted that the data may include any data or information described above.

[0134] Those skilled in the art will appreciate that computer system 700 is merely illustrative, and is not intended to limit the scope of the embodiments. Specifically, computer systems and devices may include any combination of hardware or software that can perform the indicated functions, including computers, network equipment, Internet equipment, PDAs, wireless telephones, pagers, etc. Computer system 700 may also be connected to other devices not shown, or may otherwise be operated as an independent system. In addition, the functions provided by the components shown may be combined in fewer components or distributed in additional components in some embodiments. Similarly, in some embodiments, the functions of some components in the components shown may not be provided, and / or other additional functions may be available.

[0135] Those skilled in the art will also recognize that, although various items are shown as being stored in memory or on storage devices during use, for the purpose of memory management and data integrity, these items or parts thereof may be transmitted between memory and other storage devices. Alternatively, in other embodiments, some or all of these software components may be executed in a memory on another device, and communicate with the illustrated computer system via inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer accessible medium or portable article to be read by a suitable driver, and its various examples are described above. In some embodiments, instructions stored on a computer accessible medium separated from the computer system 700 may be transmitted to the computer system 700 via a transmission medium or signal (such as an electrical signal, an electromagnetic signal, or a digital signal transmitted via a communication medium such as a network and / or a wireless link). Various embodiments may also include receiving, sending, or storing instructions and / or data implemented according to the above description on a computer accessible medium. Generally speaking, a computer-accessible medium may include a non-transitory computer-readable storage medium or memory medium, such as a magnetic medium or an optical medium, for example a disk or a DVD / CD-ROM, a volatile or non-volatile medium, such as a RAM (e.g., SDRAM, DDR, RDRAM, SRAM, etc.), a ROM, etc. In some embodiments, a computer-accessible medium may include a transmission medium or signal, such as an electrical signal, an electromagnetic signal, or a digital signal transmitted via a communication medium such as a network and / or a wireless link.

[0136] In different embodiments, the methods described herein can be implemented in software, hardware or a combination thereof. In addition, the order of the frames of the method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc. For those skilled in the art who benefit from the present disclosure, various modifications and changes can obviously be made. The various embodiments described herein are intended to be illustrative and not restrictive. Many variations, modifications, additions and improvements are possible. Therefore, multiple examples can be provided for the components described as a single example in this article. The boundaries between various components, operations and data repositories are arbitrary to a certain extent, and specific operations are shown in the context of a specific exemplary configuration. Other allocations of functions are expected, and they may fall within the scope of the appended claims. Finally, the structure and function presented as discrete components in the exemplary configuration may be implemented as a combined structure or component. These and other variations, modifications, additions and improvements may fall within the scope of the embodiments defined in the following claims.

Claims

1. A non-transitory computer-readable medium storing program instructions that, when executed by one or more processors, cause the one or more processors to: Separating a symbol stream to be entropy encoded into: (a) a compressible symbol substream and (b) a non-compressible symbol bypass substream; performing entropy encoding on the symbols of the compressible symbol substream; as well as forming a block comprising entropy coded symbols from the compressible symbol substream and non-compressible symbols from the non-compressible symbol bypass substream; A syntax element that populates the block, the syntax element indicating: a length of the block padded with the entropy coded symbols from the compressible substream; or the length of the block filled with the non-compressible symbols from the non-compressible substream; as well as In response to accumulating an amount of entropy coded symbols and non-compressible symbols that satisfies a payload size of a payload of the block, the block is provided for transmission or storage.

2. The non-transitory computer-readable medium of claim 1 , wherein the payload size of the block is less than a length of a stream of symbols to be encoded, and wherein the program instructions cause the one or more processors to: The blocks are provided for transmission or storage prior to processing the full length of the symbol stream to be entropy encoded.

3. The non-transitory computer-readable medium of claim 1, wherein the payload has a defined total length that is known to or signaled to a decoder.

4. The non-transitory computer-readable medium of claim 1 , wherein the syntax element is included in a header of the block, and wherein the header is further populated with an indicator as to whether entropy encoding probabilities are to be reset prior to entropy decoding of entropy encoded symbols of the block.

5. The non-transitory computer readable medium of claim 1 , wherein to form the block, the program instructions cause the one or more processors to: writing a first of the substreams to a first end of a packet in forward order; and A second substream of the substreams is written to a second end of the packet in reverse order.

6. The non-transitory computer readable medium of claim 1 , wherein to form the blocks, the program instructions cause the one or more processors to: writing the entropy encoded symbols of the compressible symbol substream to a compressible symbol substream buffer; writing the non-compressible symbols of the bypass sub-stream into a bypass sub-stream buffer; and In response to the sum of the entropy encoded symbols written to the compressible symbol sub-stream buffer and the non-compressible symbols written to the bypass symbol sub-stream buffer accumulating to satisfy the payload size of the block, the compressible sub-stream buffer is concatenated with the bypass sub-stream buffer to form the payload of the block.

7. The non-transitory computer readable medium of claim 1 , wherein to form the block, the program instructions cause the one or more processors to: reserving bytes of the payload of the block for writing a non-compressible sign bit; and Avoid providing the block for transmission until at least the reserved bytes are filled with non-compressible sign bits.

8. The non-transitory computer readable medium of claim 1, wherein the program instructions cause the one or more processors to: identifying one or more entropy coded bytes having a decoding dependency on one or more non-compressible symbols in the bypass substream; estimating a number of blocks buffered at a decoder to enable the decoder to decode the one or more entropy encoded bytes using the one or more non-compressible symbols, wherein the one or more non-compressible symbols are included in separate blocks from the one or more entropy encoded bytes having the decoding dependency on the one or more non-compressible symbols; as well as responsive to determining that the estimated number of blocks buffered at the decoder is less than a threshold number of blocks buffered, continuing to provide blocks for transmission or storage, the blocks including entropy encoded bytes having decoding dependencies on one or more non-compressible symbols not already included in the provided blocks; or In response to determining that the estimated number of blocks buffered at the decoder is equal to or greater than the threshold number of blocks, avoiding providing a block for transmission or storage, the block including entropy encoded bytes having decoding dependencies on one or more non-compressible symbols not already included in the provided block.

9. The non-transitory computer-readable medium of claim 1 , wherein the syntax element is included in a header of the block, and wherein to form the block, the program instructions cause the one or more processors to: reserving bytes of the payload of the block for writing a non-compressible sign bit; and In the header of the block it is indicated that: In the case where all bits of the reserved byte are not filled with the non-compressible sign bit, the valid bits of the reserved byte are filled with the non-compressible sign bit.

10. A non-transitory computer readable medium storing program instructions which, when executed by the one or more processors, cause the one or more processors to: Receive a packet consisting of: A payload, the payload comprising an entropy coded symbol portion and a non-entropy coded symbol portion; and A header indicating: The length of the payload filled with entropy encoded symbols; or The length of the payload filled with non-entropy encoded symbols; determining, based on the header, a starting point of the entropy coded symbols in the payload and a starting point of the non-entropy coded symbols in the payload; reading the entropy coded symbols from the determined starting point of the entropy coded symbols and decoding the read entropy coded symbols; reading the non-entropy coded symbols from the determined starting point of the non-entropy coded symbols; as well as The decoded entropy coded symbols and the non-entropy coded symbols are combined into a decoded symbol stream.

11. The non-transitory computer readable medium of claim 10, wherein the header further comprises: Data indicating invalid bits of non-entropy encoded bytes included in the payload.

12. The non-transitory computer-readable medium of claim 10, wherein the header further comprises an indicator indicating that entropy encoding probabilities are to be reset prior to entropy decoding of the entropy encoded symbols of the packet, wherein the program instructions cause the one or more processors to: Probabilities for the entropy decoding of the entropy encoded symbols are reset.

13. The non-transitory computer-readable medium of claim 10 , wherein a sequence header of a sequence of frames compressed into a set of packets including the packet and one or more additional packets comprises an indicator indicating that entropy encoding probabilities are to be reset prior to entropy decoding of entropy encoded symbols of the sequence of frames, wherein the program instructions cause the one or more processors to: The probabilities of the entropy decoding of the entropy encoded symbols for the packet and the one or more additional packets in the set are reset.

14. The non-transitory computer-readable medium of claim 10, wherein a frame header of a frame compressed into a set of one or more packets including the packet comprises an indicator indicating that entropy encoding probabilities are to be reset prior to entropy decoding of entropy encoded symbols of the frame, and wherein the program instructions cause the one or more processors to: A probability of the entropy decoding of the entropy encoded symbols for the set of one or more packets is reset.

15. The non-transitory computer-readable medium of claim 10, wherein a frame header of a frame or a sequence header of a sequence of frames comprises an indicator indicating that entropy coding probabilities are to be reset prior to entropy decoding of entropy coded symbols of the frame or prior to decoding of entropy coded symbols of the sequence of frames, and wherein the packet is included in the sequence of frames or the frame, and wherein the header of the packet includes an indicator indicating that entropy encoding probabilities are to be reset prior to entropy decoding of entropy encoded symbols of the packet, wherein the program instructions cause the one or more processors to: resetting the probabilities before decoding the sequence of frames or the frame; and The probabilities are additionally reset before decoding said packets comprised in said sequence of frames or comprised in said frame.

16. A system, comprising: one or more memories storing program instructions; as well as one or more processors, wherein the program instructions, when executed on the one or more processors, cause the one or more processors to: Separating a symbol stream to be entropy encoded into: (a) a compressible symbol substream and (b) a non-compressible symbol substream; performing entropy encoding on the symbols of the compressible symbol substream; as well as forming a packet comprising entropy coded symbols from the compressible symbol substream and non-compressible symbols from the bypass substream; A syntax element that populates the block, the syntax element indicating: a length of the block padded with the entropy coded symbols from the compressible substream; or the length of the block filled with the non-compressible symbols from the non-compressible substream; as well as In response to accumulating an amount of entropy coded symbols and non-compressible symbols that satisfies a payload size of a payload of the packet, the packet is provided for transmission or storage.

17. The system of claim 16, wherein the program instructions, when executed by the one or more processors, cause the one or more processors to: reserving a byte of the payload of the packet for writing a non-compressible sign bit; and Avoid providing the packet for transmission until at least the reserved bytes are filled with non-compressible sign bits.

18. The system of claim 17, wherein the program instructions, when executed by the one or more processors, cause the one or more processors to: In response to refraining from providing the packet for transmission or storage for more than a threshold amount of time: The packet is provided for transmission or storage with at least some bits of the reserved bytes not filled with non-compressible sign bits.

19. The system of claim 18, wherein the program instructions, when executed by the one or more processors, cause the one or more processors to: Data indicating valid bits of the reserved bytes padded with non-compressible sign bits is included in a header of the packet.

20. The system of claim 16, wherein to form the blocks, the program instructions cause the one or more processors to: writing a first of the substreams to a first end of the packet in forward order; and A second substream of the substreams is written to a second end of the packet in reverse order.

Images