Compression of bitstream indexes for parallel entropy coding

Abstract

This paper describes techniques for video transcoding, including compression of bitstream indices for neural network-based video transcoding and / or parallel entropy transcoding. An example includes: obtaining a video data sequence; identifying positions in the video data sequence associated with entry points of individually entropy transcoded packets for parallel entropy transcoding of the video data sequence; and generating a parallel entropy transcoded video data sequence. Then, an index is generated for the parallel entropy transcoded video data sequence, identifying individually entropy transcoded packets within the parallel entropy transcoded video data sequence.

Description

[Technical Field]

[0001] In summary, the subject matter of this case relates to image and video coding, including encoding (or compression) and decoding (decompression) of images and / or videos. Various aspects of the subject matter involve techniques for parallel entropy transcoding of images and / or videos. [Previous Technology]

[0002] Many devices and systems allow video data to be processed and output for consumption. Digital video data comprises a large amount of data to meet the needs of consumers and video providers. For example, consumers of video data expect high-quality video, including high fidelity, high resolution, and high playback rate. As a result, the large amount of video data required to meet these needs places a burden on the communication networks and equipment that process and store the video data.

[0003] Various video transcoding technologies can be used to compress video data. The goal of video transcoding is to compress video data into a form using a lower bit rate while avoiding or minimizing degradation of video quality. As evolving video services become available, there is a need for encoding technologies with better transcoding efficiency. [Summary of the Invention]

[0004] A system and technique for transcoding (e.g., encoding and / or decoding) image and / or video content using parallel entropy transcoding are described. A conventional method of data and media transcoding (also known as media compression) transcodes all information in a single bitstream processed sequentially. For many demanding applications (such as high-definition video transcoding), a single bitstream can become a data bottleneck. Standards using entropy transcoding (such as High Efficiency Video Transcoding (HEVC) and Universal Video Transcoding (VVC)) use a single bitstream structure to perform multiple processing paths on data used for other parts of the bitstream, thus breaking the transcoding system's dependence on previously processed parts of the single bitstream. Therefore, conventional modifications to such transcoding systems are not feasible because packets of such a bitstream cannot be decoded independently due to the interdependencies between different parts of the bitstream (e.g., predictive decoding).

[0005] The examples described herein use compression techniques to extensively parallelize entropy transcoding using novel transcoding methods. Furthermore, the examples achieve massive parallelization by segmenting the bitstream into small blocks with independent entry points (e.g., points in the datastream that do not depend on data from other parts of the stream outside the block), allowing these blocks to be encoded and decoded concurrently. The example implementations described herein employ this parallelization with the system to limit the encoding index management burden and computational complexity associated with transcoding additional index information.

[0006] In one illustrative example, a method for processing video data is provided. The method includes: obtaining a sequence of video data; identifying a position in the video data sequence associated with an entry point of an individually entropy-transcoded packet for parallel entropy transcoding of the video data sequence; generating the parallel entropy-transcoded video data sequence; and generating an index for the parallel entropy-transcoded video data sequence, the index identifying the individually entropy-transcoded packet within the parallel entropy-transcoded video data sequence.

[0007] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by one or more processors, cause the one or more processors to: obtain a video data sequence; identify a location in the video data sequence associated with an entry point of an individually entropy-transcoded packet for parallel entropy-transcoded video data sequence; generate the parallel entropy-transcoded video data sequence; and generate an index for the parallel entropy-transcoded video data sequence, the index identifying the individually entropy-transcoded packet within the parallel entropy-transcoded video data sequence.

[0008] In another example, an apparatus for encoding video data is provided. The apparatus includes: means for obtaining a sequence of video data; means for identifying a position in the video data sequence associated with an entry point of an individually entropy-transcoded packet for parallel entropy transcoding of the video data sequence; means for generating the parallel entropy-transcoded video data sequence; and means for generating an index for the parallel entropy-transcoded video data sequence, the index identifying the individually entropy-transcoded packet within the parallel entropy-transcoded video data sequence.

[0009] In another example, an apparatus for encoding video data is provided. The apparatus includes: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: obtain a sequence of video data; identify a location in the video data sequence associated with an entry point of an individually entropy-transcoded packet for parallel entropy transcoding of the video data sequence; generate the parallel entropy-transcoded video data sequence; and generate an index for the parallel entropy-transcoded video data sequence, the index identifying the individually entropy-transcoded packet within the parallel entropy-transcoded video data sequence.

[0010] In some cases, the above-described method, apparatus, and computer can read the media for further operation, wherein the entry point is identified at least in part by performing a bipartite enumeration of the video data sequence.

[0011] In some forms, the above-described method, apparatus and computer can read the media for further operation, wherein the parallel entropy transcoding video data sequence is generated at least in part by performing a bipartite search, which encodes information associated with the individually entropy transcoding packet with position-determined data from the video data sequence to generate the individually entropy transcoding packet.

[0012] In some forms, the above-described method, apparatus, and computer can read the media for further operation, wherein, in order to perform the bisection enumeration, the one or more processors are configured to: bisection the video data sequence to identify an initial position among the positions, wherein the initial position is associated with a first entry point and a subsequence divided by the initial position; and bisection the subsequence hierarchy to identify the remaining portions of the positions.

[0013] In some cases, the above-described method, apparatus and computer can read the media for further operation, wherein the individually entropy-transcoded packet includes corresponding data from the video data sequence determined by the positions, and information about the individually entropy-transcoded packet.

[0014] In some cases, the above-described method, apparatus, and computer can read the media for further operation, wherein the information about the individually entropy-transcoded packet is information about the scale of the binary representation of the corresponding data from the video data sequence.

[0015] In some cases, the above-described method, apparatus and computer can read the medium for further operation, wherein the difference between the largest and smallest packets in the individually entropy-transcoded packet is less than a threshold difference.

[0016] In some forms, the above-described method, apparatus, and computer-readable medium further operate to generate the parallelizable entropy transcoding sequence, wherein the one or more processors are configured to: identify multiple packets to define a set of individually entropy transcoding packets from a first packet having a size between a minimum packet size and a maximum packet size to a last packet, wherein the sum of the packet sizes of the multiple packets is the size of the parallelizable entropy transcoding sequence.

[0017] In some forms, the above-described method, apparatus, and computer-readable medium further operate, wherein, in order to generate the parallelizable entropy-transcoded video data sequence, the one or more processors are configured to: initialize a stack in the memory comprising a plurality of packets and a scale for a binary representation of the corresponding data for the plurality of packets; and generate the parallelizable entropy-transcoded video data sequence at least in part by performing the following operations until the stack is empty: analyzing the stack to retrieve the current interval set; selecting the packet by defining a new index for the packet as the average of the current interval set; encoding the bits of the packet using the average of the current interval set using a bisection; updating the stack using a set from the first packet to the packet when the packet is not the first packet; and updating the stack using a set from the packet to the last packet when the packet used is not the last packet.

[0018] In some cases, the above-described methods, apparatus and computer can read the media for further operation, wherein the stack is initialized using a common encoding character set.

[0019] In some forms, the above-described method, apparatus, and computer-readable medium can be further operated, wherein, in order to encode the bit of the packet using a bitwise division, the one or more processors are configured to: initialize a first variable to the minimum packet size, initialize a second variable to the maximum packet size, and initialize a third variable to the average of the minimum packet size and the maximum packet size; access the value of the scale of the packet in binary representation; encode the first bit when the third variable is less than or equal to the value; encode the second bit when the third variable is greater than the value; and update the third variable to the new average of the first variable and the second variable.

[0020] In some forms, the above-described method, apparatus, and computer can read the medium for further operation, wherein the one or more processors are further configured to: define packets in the individually entropy-transcoded packet as forward packets or reverse packets; wherein the corresponding packet pair includes the corresponding forward packet and the corresponding reverse packet defined using a single shared entry point in the index.

[0021] In some cases, the above-described method, apparatus and computer can read the medium for further operation, wherein when the number of individually entropy-transcoded packets is even, the index includes N / 2 entry points, and when the number of individually entropy-transcoded packets is odd, the index includes [(N-1) / 2]+1 entry points.

[0022] In some forms, the above-described method, apparatus, and computer-readable medium are further operated, wherein the one or more processors are further configured to: store the individually entropy-transcoded packet in the memory, the individually entropy-transcoded packet being constructed as a file having a file structure including the index and a plurality of parts, each part including a plurality of blocks, each block including a plurality of segments, and each segment including a forward packet structure and a reverse packet structure for the corresponding forward packet and the corresponding reverse packet of the corresponding packet pair.

[0023] In some forms, the above-described method, apparatus and computer can read the medium for further operation, wherein the one or more processors are further configured to: send the file including the index and the individually entropy-transcoded packet.

[0024] In some cases, the above-described methods, apparatus, and computer can read media for further operation, wherein the index is used to identify the individually entropy-transcoded packet by an entry point identified after encoding the video data sequence in the encoded bitstream.

[0025] In some forms, the above-described methods, apparatus and computer can read the media for further operation, wherein the index identifies the individually entropy-transcoded packet, including identifying the bit length and position of each packet of the encoded video bitstream generated from the video data sequence.

[0026] In some embodiments, the above-described method, apparatus and computer-readable medium further include: a transmitter coupled to the one or more processors and configured to transmit the video data sequence entropy encoded using the index and the individually entropy transcoding packet.

[0027] In some embodiments, the above-described method, apparatus and computer-readable medium further include: a display coupled to the one or more processors and configured to display an image of the video data sequence.

[0028] In some embodiments, the device includes: a mobile device (e.g., a mobile phone or so-called "smartphone," a tablet computer, or other types of mobile devices), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television, a vehicle (or a computing device of a vehicle), or other devices. In some embodiments, the device also includes at least one camera for capturing one or more images or video frames. For example, the device may include a camera (e.g., an RGB camera) or multiple cameras for capturing one or more images including video frames and / or one or more videos. In some embodiments, the device includes a display for displaying one or more images, videos, notifications, or other displayable data. In some embodiments, the device includes a transmitter configured to transmit one or more video frames and / or syntax data to at least one device over a transmission medium (e.g., an antenna for transmitting over a wireless medium). In some cases, the processor includes a neural processing unit (NPU), a central processing unit (CPU), a graphics processing unit (GPU), or other processing devices or components.

[0029] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0030] The foregoing and other features and embodiments will become more apparent after referring to the following description, requests and drawings.

Implementation Method

[0054] Certain forms and embodiments of the present invention are provided below. As will be apparent to those skilled in the art, some of these forms and embodiments can be applied independently, and some can be applied in combination. In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of the embodiments of the present invention. However, it will be apparent that various embodiments can be practiced without these specific details. The drawings and description are not intended to be limiting.

[0055] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present invention. Rather, the subsequent description of the exemplary embodiments will provide those skilled in the art with feasible descriptions for implementing the exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the spirit and scope of the present invention as set forth in the appended claims.

[0056] Digital video data can include a large amount of data, especially with the continued growth in demand for high-quality video data. For example, video data consumers typically expect increasingly higher quality video, with high fidelity, high resolution, and high playback speed. However, the large amount of video data required to meet such demands can place a significant burden on communication networks and the equipment used to process and store video data.

[0057] Video transcoding devices implement video compression technology to efficiently encode and decode video data. Video compression technology may include applying different prediction modes (including spatial prediction (e.g., intra-frame prediction or intra-prediction), temporal prediction (e.g., inter-frame prediction or inter-prediction), inter-layer prediction (across different layers of video data), and / or other prediction techniques) to reduce or remove redundancy inherent in the video sequence. The video transcoder can divide each image of the original video sequence into rectangular regions, which are called video blocks or transcoding units (described in more detail below). Specific prediction modes can be used to encode these video blocks.

[0058] A video block may be divided into one or more smaller blocks in one or more ways. A block may include a transcoding tree block, a prediction block, a transform block, and / or other suitable blocks. Unless otherwise specified, the reference to "block" generally refers to such a video block (e.g., a transcoding tree block, a transcoding block, a prediction block, a transform block, or other suitable block or sub-block, as will be understood by those skilled in the art). Furthermore, each of these blocks may also be interchangeably referred to herein as a "unit" (e.g., a decoding tree unit (CTU), a transcoding unit, a prediction unit (PU), a transform unit (TU), etc.). In some cases, a unit may refer to a decoding logic unit encoded in a bitstream, while a block may refer to a portion of the video frame buffer targeted by the program.

[0059] For inter-frame prediction mode, the video transcoder can search for a block in a frame (or image) that is similar to the block being encoded at another time position; this is called a reference frame or reference image. For intra-frame prediction mode, the video transcoder can use spatial prediction techniques to form predicted blocks based on data from previously encoded adjacent blocks within the same image.

[0060] The video transcoder can determine the prediction error (also known as the residual). For example, the prediction error can be determined as the difference between the image sample or primitive value in the block being encoded and the predicted block. The video transcoder can also apply a transform to the prediction error (e.g., Discrete Cosine Transform (DCT) or other suitable transform) to produce transform coefficients. After the transform, the video transcoder can quantize the transform coefficients. The quantized transform coefficients and motion vectors can be represented using syntax elements and, together with control information, form a decoded representation of the video sequence. The video transcoder can then perform entropy transcoding on the syntax elements, further reducing the number of bits required for their representation. In many video transcoding systems, entropy transcoding is the final step in data compression, where information is converted into bits, which form a compressed bitstream to be stored or transmitted.

[0061] The video decoder can use the syntax elements and control information discussed above to construct prediction data (e.g., prediction blocks) for decoding the current frame. For example, the video decoder can add the predicted block to the compressed prediction error. The video decoder can determine the compressed prediction error by weighting the transform basis function using quantized coefficients. The difference between the reconstructed frame and the original frame is called the reconstruction error.

[0062] Various technologies can be used to transcode video data. For example, video transcoding can be performed according to specific video transcoding standards or transcoding programs. Example video transcoding standards include High Efficiency Video Transcoding (HEVC), Advanced Video Transcoding (AVC), Moving Picture Experts Group (MPEG) transcoding, and Universal Video Transcoding (VVC). Video transcoding typically uses prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.), which utilize redundancy present in video images or sequences.

[0063] A common goal of video transcoding technologies is to compress video data into a form using a lower bit rate while avoiding or minimizing video quality degradation. However, HEVC, AVC, VVC, and many other such decoding standards rely on a single thread as part of their prediction methods. Simply breaking the stream into multiple packets makes it impossible to decode these packets using such decoding standards. This undecodeability may be due to the prediction methods used for decoding from a portion of the bitstream (e.g., potentially in different packets if the bitstream is broken down) and the effects of other such transcoding techniques.

[0064] Therefore, massive parallelization (e.g., using many decoding elements, such as dozens, hundreds, or more encoding or decoding processors) is only possible when video data is segmented into blocks that can be encoded and decoded concurrently. Such parallelization introduces two problems related to compression and computational efficiency. One problem relates to the administrative burden of limiting the encoding of indexes associated with independently decodeable data segments (e.g., blocks, packets, etc.). The other problem is the computational complexity of handling the decoding of additional index information. The example described herein provides an index structure with limited computational complexity and low administrative burden for indexing. This example leverages massive parallelization to achieve improved device operation for video transcoding, increasing decoding speed and avoiding unit stream bottlenecks, while limiting the administrative burden associated with improvements from parallel transcoding.

[0065] This document describes systems, apparatuses, methods (also referred to herein as "programs"), and computer-readable media (collectively referred to herein as "systems and techniques") for parallel entropy transcoding of video data. These systems and techniques provide parallel entropy transcoding of video data at least in part by segmenting the video data and adding additional information (e.g., managing burden data) to the video data when encoding the segmented video data into individually entropy transcoding packets. These systems and techniques can use two nested applications of partitioning, referred herein as nested partitioning decoding. Starting with the encoded video data to be entropy transcoded, a first partitioning application divides the encoded video data into two subsequences. In some examples, based on the characteristics of the data, the partitioning point of the two subsequences can be pseudo-random, such that when encoding is complete, the partitioning point becomes the entry point of the encoded video stream. The two subsequences can then be further divided into additional subsequences in a hierarchy, each additional subsequence having a position associated with the entry point that will become part of the encoded video data.

[0066] The second pairwise application adds additional information to each subsequence of video data determined by the position from the first pairwise application described above. This additional information may be information about the scale of values ​​in the subsequences associated with the video data (e.g., a portion of a video data subsequence between two positions, or a portion of a video data subsequence included in a separately entropy-transcoded packet). Management burden data is encoded using data from the original video data sequence to produce separately entropy-transcoded packets. These packets together form a parallelizable entropy-transcoded video data sequence, where the entry point is associated with a position where the decoder knows the appropriate state to perform parallel decoding.

[0067] In some examples, the system and techniques can generate an index to store information (e.g., entry point location, packet location, and length) that the decoder can use to identify individually entropy-transcoded packets. The index can be stored or transmitted along with a sequence of parallel entropy-transcoded video data (e.g., composed of many individually entropy-transcoded packets). The index can be the first item to be decoded, allowing the decoder unit to perform parallel decoding of the video data. Additional details are provided below with reference to the various figures.

[0068] Figure 1 is a block diagram illustrating an example of a system 100 including an encoding device 104 and a decoding device 112, which generally describes an entropy transcoding procedure. The example system 100 can use the parallel structures described below (e.g., in Figures 3 and 6) to perform parallel entropy encoding or decoding according to the examples described herein. The parallel transcoding structures and indexes can improve the operation of system 100 compared to a single-threaded system as described above.

[0069] In system 100, encoding device 104 may be part of a source device, and decoding device 112 may be part of a receiving device. The source device and / or receiving device may include electronic devices such as mobile or landline phones (e.g., smartphones, cellular phones, etc.), desktop computers, laptops or notebook computers, tablets, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some examples, the source device and receiving device may include one or more wireless transceivers for wireless communication. The transcoding techniques described herein are applicable to video transcoding in a variety of multimedia applications, including streaming video transmission (e.g., via the Internet), television broadcasting or transmission, encoding of digital video for storage on data storage media, decoding of digital video stored on data storage media, or other applications. In some examples, system 100 may support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video replay, video broadcasting, gaming, and / or video telephony.

[0070] The video data from video source 102 may include one or more input images. Images may also be referred to as "frames". An image or frame is a still image that is part of the video in some cases. In some examples, the data from video source 102 may be a still image that is not part of the video. An image may include three sampling arrays, denoted as SL, SCb, and SCr. SL is a two-dimensional array of luminance samples, SCb is a two-dimensional array of Cb chrominance samples, and SCr is a two-dimensional array of Cr chrominance samples. Chrominance sampling may also be referred to herein as "chroma" sampling. A primitive may refer to a location within the image that includes the luminance component, Cb component, and Cr component. In other cases, the image may be monochrome and may consist only of an array of luminance samples.

[0071] The encoder engine 106 (or encoder) of the encoding device 104 encodes the video data to produce an encoded video bitstream. In some examples, the encoded video bitstream (or "video bitstream" or "bitstream") is a series of one or more decoded video sequences. The encoder engine 106 produces a decoded representation of the image by segmenting each image. Different segments may contain luma and chroma sampling arrays. In some examples, the luma and chroma arrays may be further subdivided into prediction blocks (PBs). A PB is a sampling block of the luma component or chroma component that uses the same motion parameters for inter-frame prediction or intra-block copy prediction (when available or enabled for use). The luma PB and one or more chroma PBs, together with the associated syntax, form a prediction unit (PU). For inter-frame prediction, a set of motion parameters (e.g., one or more motion vectors, reference indices, etc.) is signaled in the bitstream for each PU and used for inter-frame prediction of the luma PB and one or more chroma PBs. Motion parameters can also be referred to as motion information.

[0072] As part of entropy transcoding, encoder engine 106 uses prediction modes to predict each PU. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (described below). The prediction mode can be signaled within the bitstream using syntax data. The prediction mode can include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (inter-picture prediction). For example, a decision can be made at the CU level whether to use inter-picture prediction or intra-picture prediction to decode a picture region.

[0073] The encoding device 104 can then perform transformation and quantization. For example, after prediction, the encoder engine 106 can calculate a residual value corresponding to the prediction. The residual value can include the primitive difference between the current block of primitives being decoded and the prediction block used to predict the current block (e.g., a predicted version of the current block). For example, after generating a prediction block (e.g., using inter-frame prediction or intra-frame prediction), the encoder engine 106 can generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of primitive differences that quantize the difference between the primitive values ​​of the current block and the primitive values ​​of the prediction block. In some examples, the residual block can be represented in a two-dimensional block format (e.g., a two-dimensional matrix or array of primitive values). In such examples, the residual block is a two-dimensional representation of the primitive values.

[0074] The encoder engine 106 can perform quantization of the transform coefficients. Quantization provides further compression by reducing the amount of data used to represent the coefficients. For example, quantization can reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient with an n-bit value can be rounded down to an m-bit value during quantization, where n is greater than m.

[0075] During quantization, the decoded video bitstream includes quantized transform coefficients, prediction information (e.g., prediction patterns, motion vectors, block vectors, etc.), segmentation information, and any other suitable data (such as other grammatical data). The different elements of the decoded video bitstream can then be entropy-encoded by encoder engine 106. In some examples, encoder engine 106 can scan the quantized transform coefficients using a predefined scanning order to produce a serialized vector that can be entropy-encoded. In some examples, encoder engine 106 can perform adaptive scanning. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), encoder engine 106 can entropy-encode that vector. For example, encoder engine 106 can use context-adaptive variable-length decoding, context-adaptive binary arithmetic decoding, grammar-based context-adaptive binary arithmetic decoding, probabilistic interval segmentation entropy transcoding, or another suitable entropy coding technique.

[0076] The output 110 of the encoding device 104 can transmit units constituting the encoded video bitstream data to the decoding device 112 of the receiving device over the communication link 120. The input 114 of the decoding device 112 can receive units. The communication link 120 may include a channel provided by a wireless network, a wired network, or a combination of wired and wireless networks. The wireless network may include any wireless interface or combination of wireless interfaces and may include any suitable wireless network (e.g., the Internet or other wide area networks, packet-based networks, WiFi™, radio frequency (RF), UWB, WiFi Direct, cellular, LTE, WiMax™, etc.). The wired network may include any wired interface (e.g., fiber optic, Ethernet, powerline Ethernet, coaxial cable Ethernet, digital signal line (DSL), etc.). Various devices can be used to implement wired and / or wireless networks, such as base stations, routers, access points, bridges, gateways, switches, etc. Encoded video bitstream data can be modulated according to communication standards such as wireless communication protocols and then sent to the receiving device.

[0077] In some examples, encoding device 104 may store encoded video bitstream data in storage unit 108. Output 110 may retrieve encoded video bitstream data from encoder engine 106 or from storage unit 108. Storage unit 108 may include any of a variety of distributed or locally accessed data storage media. For example, storage unit 108 may include hard disks, storage disks, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data. Storage unit 108 may also include a decoded picture buffer (DPB) for storing reference pictures for use in inter-frame prediction.

[0078] The encoder engine 106 and the decoder engine 116 (described in more detail below) can be configured to operate according to the parallel transcoding operation described herein, wherein each independently entropy transcoding packet is encoded and decoded independently using packet details in the index.

[0079] The input 114 of the decoding device 112 receives encoded video bitstream data and can provide the video bitstream data to the decoder engine 116 or to the storage unit 118 for later use by the decoder engine 116. For example, the storage unit 118 may include a DPB for storing reference images used in inter-frame prediction. A receiving device including the decoding device 112 can receive the encoded video data to be decoded via the storage unit 108.

[0080] Encoded video data can be modulated according to communication standards such as wireless communication protocols and transmitted to a receiving device. Decoder engine 116 can decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more encoded video sequences that make up the encoded video data. Decoder engine 116 can then rescale the encoded video bitstream data and perform an inverse transform on it. The residual data is then passed to the prediction stage of decoder engine 116. Decoder engine 116 then predicts blocks of primitives based on the decoding structure of the packets used for the data, as described below.

[0081] Decoding device 112 can output the decoded video to video destination device 122, which may include a display or other output device for displaying the decoded video data to a consumer of the content. In some embodiments, video destination device 122 may be part of a receiving device that includes decoding device 112. In some embodiments, video destination device 122 may be part of a separate device different from the receiving device.

[0082] Figure 2 is a diagram illustrating an example implementation of the filter unit 200, including placing reference images in a reference image storage unit 208 that can be identified by a reference image table. The filter unit 200 can be part of any encoding element (e.g., encoding elements 310A-N described below) or decoding element (e.g., decoding elements 340A-N described below), and can therefore be used for parallel entropy transcoding according to the examples described herein.

[0083] The filter unit 200 shown in the figure includes a deblocking filter 202, a sampling adaptive offset (SAO) filter 204, and a general filter 206, but may include fewer filters than those shown in Figure 2 and / or may include additional filters. Furthermore, the specific filters shown in Figure 2 may be implemented in a different order. Other loop filters (in or after the decoding loop) may also be used to smooth primitive transitions or otherwise improve video quality. When in the decoding loop, the decoded video block in a given frame or picture is then stored in a decoded picture buffer (DPB), which stores reference data as part of a reference picture storage unit 208. The DPB (e.g., reference picture storage unit 208) may be part of or separate from additional memory storing the decoded video for later presentation on a display device (such as the display of the video destination device 122 of Figure 1).

[0084] If system 100 and filter unit 200 are single-threaded processing systems, they can use significant prediction to improve compression and decoding performance. However, the system described herein using multiple filter units can greatly improve decoding speed at the cost of index management burden and parallel transcoding hardware (e.g., multiple filter units 200 in encoding device 104 or decoding device 112). For parallel encoding, the decoder is configured to achieve the same state as the encoder when decoding a particular element. Thus, random access (e.g., parallel processing of different parts of a video stream) is achieved to some extent by splitting the bitstream and establishing access locations (e.g., entry points) corresponding to the positions where the encoder and decoder states are reset to known values. Such entry points identify the positions in the bitstream where subsequent portions of the bitstream become independently entropy-transcoded. By configuring such entry points and somewhat regular positions within the bitstream, the data between entry points becomes such independently entropy-transcoded portions of the bitstream, referred to herein as packets, bags, or independently entropy-transcoded packets. The following section describes in detail the additional details related to the structure and indexing of such packages.

[0085] Figure 3 is a diagram illustrating various forms of parallel transcoding for independently entropy-transcoded packets according to the examples described herein. Figure 3 illustrates a structure 300 having multiple encoder units 310A, 310B, 310C, and 310N, each encoder unit being configured to perform parallel encoding of independently entropy-transcoded packets. Buffers 320A, 320B, 320C, and 320N correspond to the encoder units and can buffer data for storage, transmission, or any processing used to generate index 330. Each encoder unit and buffer pair outputs a corresponding packet during a single parallel processing operation, wherein the packets are shown as packets 332A, 332B, 332C, and 332N. Each encoder unit can process any number of packets for a given video data sequence. For example, if the video data sequence is divided into 10,000 packets and the parallel encoding system uses 50 encoder units, then each encoder unit can produce an average of 200 independently entropy-transcoded packets. During operation, if some encoder units encode longer or shorter packets, a unit may encode more or fewer packets compared to other encoding units, depending on the system architecture.

[0086] Structure 300 supports parallel entropy encoding and decoding, wherein the compressed data bitstream is divided into packets (e.g., including packets 332A-N). The number of bytes in each packet is random or pseudo-random to accommodate the internal dependencies or decodeability of each packet. As described above for single-threaded decoding, where single-threaded partitioning may compromise decodeability, packets are not simply a fixed number of bytes because some partitions may render video data segments undecodeable (e.g., if the partition is not an entry point as described above). Utilizing the random or pseudo-random number of encoded bytes in each packet, the encoded bytes are stored in separate memory buffers, and when the encoding of individually entropy-transcoded packets is complete, the data of the packets can be continuously copied to a single compressed data byte array for storage or transmission. In some examples, different structures other than a single compressed data byte array can be used, as long as the integrity of the individually entropy-transcoded packets and the relationship between packets identified by index 330 are maintained.

[0087] Packet index 330 is used to describe the relationships between packets and maintain the integrity of compressed data, even if the data is segmented into packets that can be independently transcoded in parallel. When data is decoded from a storage unit or received as part of a transmission, the information in index 330 is used to manage the decoding of packets using decoder units (such as decoder units 340A, 340B, 340C, and 340N). As shown, an encoder unit sends data to a specific decoder unit (e.g., encoder unit 310A sends packet 332A to decoder unit 340A). However, in various examples, packets from a single encoder unit can be decoded by any number of different decoder units (e.g., three packets encoded by encoder unit 310B can be transcoded in parallel by decoder units 340A, 340C, and 340N).

[0088] Figure 4 is a diagram illustrating various forms of an encoded bitstream 400 with index 430 according to the example described herein, depicting entry points for separating independently entropy-transcoded packets for parallel entropy transcoding. The encoded bitstream 400 of Figure 4 may, for example, include packets 332A-N of Figure 3 arranged for storage or transmission as described above, or may be any such video material decoded for parallel transcoding according to the example described herein. The encoded bitstream 400 includes packets 440, 441, 442, and 449. Other examples of encoded bitstreams constructed from a sequence of video materials may have any number of packets. Each packet has an associated entry point. As shown, entry point 460 corresponds to packet 440, entry point 461 corresponds to packet 441, entry point 462 corresponds to packet 442, and entry point 469 corresponds to packet 449. Entry point 463 corresponds to a packet (not shown) following packet 442. Each entry point is associated with a position in the encoded bitstream where the decoding state is known to the decoder configured according to the examples described herein, thus allowing packets to be decoded independently and in parallel by the decoder units. Index 430 is a data structure that, in some examples, contains multiple bytes per packet or equivalently at each entry point array position. The packet index 430 data can then be used to identify packets from the stored or transmitted encoded bitstream 400, and to reassemble the video data sequence used to generate the encoded bitstream 400.

[0089] Since packet index 430 is required to decode the encoded bitstream 400, index 430 is stored together with packets of the encoded bitstream 400. Any decoding begins with decoding index 430 to identify information associated with individual decodeable packets. Additionally, to efficiently identify packets, the number of bits used to encode packet entry points is added to the total number of bits required to encode that packet. Therefore, packet identification in index 430 considers the entry points in the encoded bitstream for encoding, or similarly accommodates any bits used for encoding by considering the number and position of bits in the bitstream for each packet. In some examples, for efficient parallelization, the video bitstream may generate many packets with a relatively small number of bytes. In such examples, in addition to the limited ability to predict with small packet sizes, the administrative burden of decoding the index may also degrade compression. For example, to simplify hardware design and minimize costs, some examples may limit packet sizes to the range of 16-64 bytes. In other examples, different threshold limits or ranges can be used. For packets of this size, the administrative burden of managing several bytes per packet translates into significantly worse compression.

[0090] The examples described herein provide a technical solution to the technical problem of how to limit the increased administrative burden by limiting the number of bits used to encode the entire packet index. Assume there are N packets, and the information to be decoded is a sequence having a random number of bytes used by each packet, expressed as follows:

[0091] Since packet data decoding depends on these numerical values, decoding cannot begin until the numerical values ​​are decoded, and using a low-complexity method to transcode the numerical values ​​provides improved device performance during decoding. Furthermore, the examples described herein can use a single decoding method that can be used for any type of statistical distribution and any range of scales, allowing for the benefit of compatibility across many different systems.

[0092] The universal codeword set is applicable to integers of any size. Within the universal codeword set, Elias gamma code (also known as "exp-Golomb" code) is used in several video compression standards because it is easy to implement and allows for acceptable device performance. The number of bits used in Elias gamma code to transcode a positive integer n is...

[0093] Elias delta codes can decode positive integers using progressively smaller bit counts, equal to

[0094] However, in practice, this code is not usually used for decoding large numbers of integers because it uses two logarithms for each value.

[0095] Other decoding methods (such as Huffman decoding) are more complex to implement because they can only be applied to a finite set of integers, thus requiring special strategies for handling values ​​outside the set (e.g., additional decoding definitions to handle overflows), and can only be optimized to a known probability distribution.

[0096] The examples described herein use nested bipartite decoding operations that are also described as generally applicable. Such operations can improve device operation when compared to the general use of the above-described general-purpose codeword sets, without relying on any assumptions about the statistics of the numerical values ​​or their scale range (e.g., it works equally well for small or very large numerical values).

[0097] For example, a fundamental limitation of general-purpose codeword sets is that they are designed for use without any prior information about the integers to be decoded. Therefore, when using a general-purpose codeword structure, some additional bits are required to encode information about the scale of the numerical value, followed by bits for the binary representation of the numerical value. Compared to general-purpose codeword usage, the examples described herein further improve device operation by utilizing the fact that video packets, as mentioned above, are datasets with many non-negative integers. More specifically, the examples described herein provide benefits based on the use of scale information shared by all the numerical values ​​to be decoded and some similarity between scales (e.g., the expectation that similar extreme cases are uncommon). To implement the conditions leading to the benefits from the examples described herein, the encoding system can apply threshold constraints to packets constructed for parallel encoding. For example, some implementations may include threshold constraints on the size differences between packets (e.g., a minimum size of 32 bits and a maximum size of 128 bits, or a minimum size of 16 bits and a maximum size of 64 bits, etc.). Additional information used to provide the benefits of the examples described herein includes: the example decoding operations are implicitly defined by a simple and efficient algorithm rather than by a predefined set of binary encoded words, and the operations involve the accumulation of information during the encoding process.

[0098] The above features work together to improve the functionality of the decoding device by utilizing parallel transcoding with limited management overhead. To enable the use of all the above features, the following monotonic sequence is defined in an illustrative example:

[0099] Among them,

[0100] and define the sequence of byte positions at the beginning of the packet as (6).

[0101] Encoding the integers in a sequence is equivalent to encoding the quantity in the sequence.

[0102] Integers in a sequence can be decoded sequentially, and information can be collected after each value is decoded to improve the decoding of the next value. Information can be collected efficiently in a hierarchical manner. The example decoding operation described above, known as nested bipartite decoding, uses two nested applications of bipartite decoding.

[0103] In the first application of bisection (referred to as outer bisection enumeration), the order in which the elements of sequence P are encoded is determined, and the decoded information is aggregated according to the tree data structure.

[0104] Figure 5A illustrates the hierarchy of the tree structure used for the bisection operation in conjunction with the examples described herein. Figure 5A specifically illustrates the various forms of the first application of the above bisection (external bisection enumeration). The bisection search (including external bisection enumeration) is performed on two types of integer sets. An integer set is an open interval of indices and a closed interval of positions, which can be represented as:

[0105] Among them

[0106]

[0107]

[0108] and (9),

[0109] The tree structure diagram in Figure 5A illustrates the accumulation of information during the sub-period. Each block 512, 522, 524, 542, 544, 546, 548, 552, 554, 556, and 558 in Figure 5A is associated with an entry point in the decoded bitstream, which is associated with a position in the encoded bitstream. When decoding each value of a block according to the pre-sorted tree traversal sequence, the value can use previously decoded values ​​to reduce uncertainty about the range of values ​​in the packet defined between two values ​​when deciding all values, and thus make decoding more efficient. Blocks at the top level of the tree require more bits to decode, but the administrative burden is relatively small because the number of blocks grows exponentially with the depth of the tree. Value 510 shows the number of values ​​at each level, with block 512 at the first level starting with a single value, doubling with each level.

[0110] The external bisection enumeration operation can also be considered as follows. Starting with the video data sequence, the sequence is split into two subsequences by selecting a position in the video data sequence (e.g., bisecting the video data sequence). The selected position will be associated with the entry point in the encoded video bitstream. The first selected position is associated with the first block 512 at the first position in the tree structure of FIG5A. These two subsequences are then bisected to select two values ​​associated with blocks 522 and 524 at the second level. The second-level bisection produces four subsequences, which can then be bisected similarly to identify the values ​​of blocks 542, 544, 546, and 548. As mentioned above, each value is associated with what will become the entry point in the encoded video bitstream. This procedure is repeated by hierarchically bisecting the subsequences of the video data sequence to produce additional subsequences defined by additional entry points between additional sequences and additional entry points between two packets of the video data sequence, resulting in individual decodable packets of the video data sequence, each identified by an entry point of the video data sequence. At the final level, each point identified during the bisecting of the subsequences is associated with an entry point, and the data between points is the data of a separate decodable packet. The bisecting operation can use rules to identify appropriate points in the data used as entry points. As mentioned earlier, because the segments are split not uniformly but based on the characteristics of the data (e.g., along decodable boundaries in the data), the bisecting positions end up being random or pseudo-random. For example, pseudo-randomness might cause a 1000-bit sequence to be bisected into a 20-bit subsequence and a 980-bit sequence. If the minimum sequence length of a packet is 16 bits, then a 20-bit subsequence is not further bisected, and the branch of the tree stops at that level, while a 980-bit sequence is further bisected. Each branch may terminate further bisecting when the associated sequence does not contain a suitable location for the entry point, or when further bisecting would result in a sequence shorter than an acceptable bit length. In other examples, other such criteria for ending the outer bisecting enumeration may be used. When the outer bisecting enumeration is complete, the data of all packets, along with the data boundaries between packets, are known.

[0111] Each square in Figure 5A can be viewed as associated with a bisection operation, and the position is associated with the entry point of the data encoded into a parallelizable entropy-transcoded video data sequence. Square 512 is associated with the bisection of the initial video data sequence, and the position associated with square 512 becomes the entry point after additional data is added to the final subsequence of the data (e.g., after selecting all boxes or bisection positions). The additional squares below square 512 are associated with the bisection of data subsequences in additional levels. As each position is selected and the additional levels of the subsequence are bisected, the data subsequences become smaller until the final subsequence of the data is selected by the final bisection position. The final complete set of positions (e.g., positions from each of the squares in Figure 5A) defines the data subsequences from the complete set of video data that will become separate packets. Before the data used for the final subsequence is individually decodable, additional information (e.g., information about the number of bits in the final subsequence) is added to convert the final subsequence data from the video data into individually entropy-transcoded packets.

[0112] Then, the next operation decodes the data and takes into account the decoding management burden when identifying packet information for the index (e.g., the boundaries in the decoded video bitstream that include the decoding management burden, or the bit length of the decoded packet that includes all bits of the decoded bitstream).

[0113] Using the above, the operation can continue to process values ​​from the external bipartite enumeration (e.g., the identified locations of unencoded data to be associated with entry points and packet boundaries), and the data of each subsequence between values, along with any administrative burden data, can be encoded into packets. In one example, a stacked data structure is used with values ​​from (e.g., pre-sorted tree traversals) the external bipartite enumeration, where the same notation as described above is used to describe the following operation.

[0114] Use a general code set (e.g., Elias delta code or any other such code set) to encode the number of packets N and the total number of bytes S determined by an external bipartite enumeration. Initialize the stack in memory by pushing a set: (10);

[0115] As mentioned above. Then, as long as the stack is not empty, use the following operation:

[0116] Pop up the stack to retrieve the current interval set: (11)

[0117] The new index is defined as: (12);

[0118] And the fraction is used to encode the following items: (13)

[0119] If, then the set is pushed onto the stack, and if, then the set is pushed onto the stack.

[0120] Then repeat the operation after Equation 10 above until the sum is reached. At this point, the entire video data sequence has been decoded into individual decodeable packets, which can be decoded in parallel into an encoded video bitstream, and all the necessary information is available for indexing.

[0121] In one example of the operation described above, the bisection for encoding is performed as follows: Given and , initialize the variables. Then, repeat the following operations: If , then encode and set bit "1". If , then encode and set bit "0". Update c.

[0122] The decoding algorithm is similar to the encoding operation described above. For decoding, given and , initialize the variables. When , repeat the following operation. If the decoded bit is "1", set , otherwise set . Update, and return the decoded value.

[0123] The above examples illustrate various forms of decoding operations according to the embodiments described herein, but it will be apparent that other examples and implementations are possible. The following further illustrates the various forms of the example encoding implementation: / / "Push" initial interval set (i_(min ),i_max )=(0,N),[p_(min ),p_max ]= [0,S] stack.push(0, num_parcels, 0, byte_sum); / / Encoding loop while (!stack.empty()) { / / "Push" current set stack.pop(idx_min, idx_max, pos_min, pos_max); / / (i_(min ),i_max ),[p_(min ),p_max ] i = (idx_min + idx_max) >> 1; / / New index defined by the external enumeration a = pos_min; / / Internal bipartite search and decoding b = pos_max + 1; c = (a + b) >> 1; while (c != a) { if (c <= position[i]) { / / The code bits defined by the search result are encode_bit(1); a = c;} else { encode_bit(0); b = c;} c = (a + b) >> 1;} / / Push the new subset onto the stack if (i - idx_min > 1) stack.push(idx_min, i, pos_min, position[i]); if (idx_max - i > 1) stack.push(i, idx_max, position[i], pos_max);} Similarly, the following diagram illustrates the various ways of implementing the decoding method: / / Push the initial interval set (i_(min ),i_max )=(0,N),[p_(min ),p_max ]= [0,S] stack.push(0, num_parcels, 0, byte_sum); / / Decoding loop while (!stack.empty()) { / / Push the current set stack.pop(idx_min, idx_max, pos_min, pos_max); / / (i_(min ),i_max ),[p_(min ),p_max ] i = (idx_min + idx_max) >> 1; / / New index defined by the outer enumeration a = pos_min; / / Internal bipartite search and encoding b = pos_max + 1; c = (a + b) >> 1; while (c != a) { if (decode_bit() == 1) / / Code bits defined by the search result a = c; else b = c;} c = (a + b) >> 1;} position[i] = a; / / Push the new subset onto the stack if (i - idx_min > 1) stack.push(idx_min, i, pos_min, position[i]); if (idx_max - i > 1) stack.push(i, idx_max, position[i], pos_max);}.

[0124] The above-described operations, or similar operations according to the examples described herein, can be used for data compression, wherein the parallelization of entropy transcoding is used to improve the device performance of video processing by utilizing parallel encoding or decoding. Compared to similar devices, the examples described herein allow for parallel entropy transcoding with reduced processing resources and power consumption due to the management burden and packet identification structure of the above-described bipartite operations. Examples can be combined with neural network-based video processing transcoders to achieve low-cost parallel encoding and decoding to improve the processing of high-resolution video data, high frame rate data, improved color resolution, and improved dynamic range video data. Since such improvements in video quality increase the amount of video data, the bottleneck of the above-described single-threaded entropy transcoding can significantly impair performance compared to parallel transcoding. Parallel transcoding using the operations described herein can be performed with low management burden and efficient processing circuitry with acceptable power consumption and improved data throughput associated with parallelization.

[0125] The binary-based decoding example described herein requires approximately log2(li) execution steps to decode the index element. For systems that can efficiently compute the function |log2(x)| (e.g., within one clock cycle of the device or processing system, or within a few clock cycles), a class of prefix codes can be used. Prefix codes (or unprefixed codes) (such as Huffman codes or Golomb codes) are typically optimized for some probability distribution of the data symbols. However, as mentioned above, some systems do not assume knowledge of the probability distribution. When the value n to be encoded is such that the device can efficiently encode that number of bits using a fixed number equal to B. In general, when , for some values, one bit can be used to decode the quantity n, and for the remaining values, one bit can be used to decode the quantity n.

[0126] Some examples use the following methods to specify how to decode values ​​for simple and efficient systems instead of binary-based decoding examples. Some such examples increase the number of bits used to transcode the value sequentially from one bit to the last value in the set of possible values. Table 1 below illustrates examples of prefix codes for integer values ​​of Np from 4 to 8, with Figures 5B-5F illustrating tree representations of these codes. n N p = 4 N p = 5 N p = 6 N p = 7 N p = 8 0 00 000 000 000 000 1 01 01 010 010 001 2 10 10 10 100 010 3 11 11 11 11 011 4 – 001 001 001 100 5 – – 011 011 101 6 – – – 101 110 7 – – – – 111

[0127] Table 1 - Binary Encoded Words

[0128] Figure 5B illustrates various forms of prefix codes that can be used with the examples described herein. Figure 5B is a tree representation 560 of the code values ​​for Np = 4. The paths from the starting position 561 to the n-value positions 562-565 illustrate how each code value is generated. For the n = 0 code, the path from the starting position 561 to the n = 0 position 561 provides the code 00. Similarly, the paths from the starting position 561 to the n = 1 position 562, the n = 2 position 563, and the n = 3 position 565 are 01, 10, and 11, respectively.

[0129] Figure 5C illustrates various forms of prefix codes that can be used with the examples described herein. Figure 5C is a tree representation 568 of the code value Np=5. As shown in tree representation 570 and the paths from the starting position 560 to positions 563, 564, and 565, the codes for values ​​n=1, n=2, and n=3 are the same for Np=5 and Np=4. For the new n=4 position 572 where Np=4 does not exist, it is 001, and the n=0 position 571 has an additional bit with a value of 000.

[0130] Figure 5D illustrates various forms of prefix codes that can be used with the examples described herein. Figure 5E illustrates various forms of prefix codes that can be used with the examples described herein. Figure 5F illustrates various forms of prefix codes that can be used with the examples described herein. Each of these figures illustrates the corresponding tree representations 570, 580, and 590, which illustrate code values ​​Np=6-8. At each increment of the Np value tree representation, the lowest value position in the lowest complete branch level is replaced with two values ​​from the level below. From Np=5 to Np=6, position 563 is replaced by two positions 573 and 574 branching from that position. From Np=6 to Np=7, position 564 is replaced by two positions 575 and 576 branching from that position. From Np=6 to Np=7, position 565 is replaced by two positions 577 and 578 branching from that position. For each of the trees shown, the path from the initial position 561 branch to the lowest level position of each branch provides the corresponding code value for the value of n.

[0131] The code shown above differs from the well-known canonical Huffman code, where the number of bits increases with the decoded value. Canonical Huffman codes are easier to encode and decode without requiring a list of codewords and their bit count. However, the code shown allows for the description of a similarly simple set of prefix codes by the following encoding and decoding algorithms. Example encoding algorithm: 1. Input: Values ​​of n and Np such that; 2. Calculate and; 3. Encode the pair using B bits; 4. If, then: a. Encode the pair using one bit. Example corresponding decoding algorithm: 1. Input: Value of Np; 2. Calculate and; 3. Decode k using B bits, and set n = k; 4. If, then: a. Decode the pair using one bit, and if its value is 1, set n = k + M; 5. Return value n.

[0132] Then, some examples described herein can be used in place of the above-described decoding and decoding index elements instead of the partition-based decoding described in other parts of this description.

[0133] Figure 6 is a diagram illustrating various forms of parallel transcoding for independently entropy transcoding packets according to the examples described herein. Figure 6 illustrates a diagram similar to that of Figure 3, with additional improvements to reduce the decoding management burden for parallelization of entropy transcoding. As mentioned earlier, indices identify packets and entry points, using either the number of bits and position for each packet, or entry points (e.g., each of which can be used to identify other data, where entry points can be identified from packet bits and positions, and packet bits and positions can be identified from entry points). Since adjacent packets share a common boundary, the positions of two packets can be defined using a single entry point. In such examples, packets can be defined as forward packets or backward packets. Encoder unit 610 and buffer 620 can operate like the encoder unit and buffer of Figure 3, instead of identifying entry points at the beginning or end of each packet as described in Figure 4, they can identify each pair of packets in association with the shared boundary between packets, and the number of entry points in index 630 can be halved.

[0134] For example, as shown in Figure 6, forward packets 632, 633, and 634 and backward packets 642, 643, and 644 are included. These packets are formed as packets joined at entry points 652, 653, and 654, such that the first pair of packets 632 and 642 are joined by entry point 652, the second pair of packets 633 and 643 are joined by entry point 653, and the third pair of packets 634 and 644 are joined by entry point 654. As shown, forward and backward packets are defined as bidirectional byte packings with shared entry points that allow for parallel entropy transcoding, wherein the number of entry points in index 630 is half that of index 430 without bidirectional byte packing. The forward packets are then decoded by forward decoder units 662, 663, and 664, and the backward packets are decoded by backward decoder units 672, 673, and 674. The decoder can operate using the implementation described above, but the forward decoder increments the byte counter during decoding, and the backward decoder decrements the byte counter to decode the corresponding packet. As mentioned earlier, packet pairs remain independently decodeable and do not require decoding by a specific decoder unit pair. In some examples, the decoder can be configured to decode either forward or backward packets, while in other units, the decoder is customized to efficiently decode either forward or backward packets instead of both, as shown in Figure 6. Such examples further improve the operation of the decoding system and the devices within it by reducing administrative overhead, increasing throughput, and reducing power consumption (e.g., associated with the reduced administrative overhead).

[0135] Figure 7 illustrates a file structure 710 for parallel transcoding of independently entropy transcoding packets of video data sequences, according to an example described herein. File structure 710 includes a P section and a file index 702, similar to index 630 as described above. A section of file structure 720 in the example of Figure 7 includes B blocks, and each block of block structure 730 contains S sections. The example structure for parallel entropy transcoding provides a unified hierarchical structure that can be used with the aforementioned packets and has minimal administrative burden. This structure allows the user to control file opening and closing operations, and to start and stop operations at any partial element of file structure 710 (e.g., section 0, section 1, section P-1, etc.). For example, in some implementations, partial elements can be used for video frames, and blocks of partial structure 720 (e.g., blocks 0, 1, B-2, etc., using any partial element of partial structure 720 in file structure 710) can be associated with video transcoder function calls. The main block can include scalar data for a portion, while separate non-main blocks contain vector data. In some implementations, neural network decoding combined with the aforementioned parallel entropy transcoding operation can be used to automate segment and packet usage.

[0136] Segment structure 740 includes structures for packet pairs, which include forward packets and backward packets as described above. In some cases, there may be an odd number of packets, such that a segment may contain only forward or backward packets, but the pair structures of these segments allow byte packing as described above with respect to FIG6. Each forward packet may then have Nf bytes as defined by forward packet structure 750, and each backward packet may have Nb bytes as defined by backward packet structure 760. As mentioned above, a threshold may be applied to the number of bytes, and in some examples, the threshold may be the same or different for forward and backward packets. In either case, the threshold may include an upper and lower limit on the number of bytes allowed for a given packet to allow ordered bipartitioning of video data and to simplify decoding and improve the operation of the decoding device by limiting the variation in the size of the packets to be decoded.

[0137] Figure 8A is a flowchart illustrating an example of a program 800 that encodes video using operations for parallel entropy transcoding according to the examples described herein. In some examples, program 800 is executed by a device including memory and one or more processors (such as a SOC 900 or a device according to architecture 1300). In other examples, program 800 is implemented as computer-readable instructions stored in a storage medium that, when executed by processing circuitry, cause a decoding operation to be performed. In some examples, the device uses an encoding device such as encoding device 104 to execute program 800. In some examples, neural networks may be utilized to perform various forms of entropy encoding in order to achieve complex massive parallelization with the decoding system, as further described below.

[0138] At block 802, program 800 includes an operation to obtain a sequence of video data. As previously mentioned, the sequence of video data can be any video data to be encoded using the parallel entropy coding described herein for storage or transmission. Such parallel transcoding can be particularly beneficial when single-threaded decoding results in increased data bottlenecks, such as ultra-high resolution video (e.g., 8K video or higher), high frame rate video (e.g., 60 frames per second, 120 frames per second or higher), or data streams with wider color resolution or color gamut, high dynamic range, or high quality. Parallel transcoding of video data sequences can also be used for artificial intelligence analytics or machine learning, where machines analyze video data in a manner unconstrained by standard time display (e.g., where a computer can "watch" and / or process two hours of film in one second, compared to humans displaying and watching video frames over two hours). Such parallel entropy transcoding can allow machine learning or artificial intelligence systems to decode and process video data at a much higher rate compared to single-threaded transcoding systems.

[0139] At block 804, program 800 includes operations to identify positions in the video data sequence associated with entry points of individually entropy-encoded packets used for parallel entropy-encoded video data sequences. In some examples, the entry point is identified at least in part by performing a bipartition enumeration of the video data sequence. In some examples, performing a bipartition enumeration includes: bipartitioning the video data sequence to identify an initial position in the positions. The initial position is associated with a first entry point and a subsequence divided by the initial position. Performing a bipartition enumeration may also include: grading the subsequence to identify the remaining portion of the positions. Other examples may use prefix codes, such as those described above with respect to Figures 5B-5F.

[0140] In some forms, the operation associated with block 804 may include determining decoding information, which includes information associated with an entry point for a separately entropy-transcoded packet. Some examples may include performing an external bisection enumeration on the video data sequence to determine the decoding information. Details of an example of such an external bisection enumeration are described above with respect to Figure 5A. In some examples, block 804 may include the operation of bisecting the video data sequence to identify a first entry point among the entry points, the first entry point defining a first boundary between two subsequences of the video data sequence and between two packets in a separately entropy-transcoded packet. Block 804 may also include repeatedly bisecting the hierarchy of subsequences of the video data sequence to produce additional subsequences defined by additional entry points between additional sequences and additional entry points between adjacent packets of the video data sequence to produce separate decodable packets of the video data sequence, wherein the separate decodable packets are identified by the entry points for the video data sequence. As mentioned above, in some examples, the external bipartite enumeration limits the difference between the largest and smallest packets in the individually entropy-transcoded packets to less than a threshold difference.

[0141] In other examples of block 804, the operation can select the number of packets N by repeatedly bisecting video data sequences with values ​​of P bits, to define the set of individually entropy-transcoded packets according to the following formula: where the sum of their sizes is equal to the value of P bits. In such examples, each i is an individually entropy-transcoded packet, the first individually entropy-transcoded packet in the set of individually entropy-transcoded packets, and the last individually entropy-transcoded packet in the set of individually entropy-transcoded packets. Similarly, in such examples, each p is the bit size of the corresponding individually entropy-transcoded packet, the bit size of the first individually entropy-transcoded packet, and the bit size of the last individually entropy-transcoded packet.

[0142] At block 806, program 800 includes an operation to generate a sequence of video data that can be parallelized entropy-transcoded. In some examples, such an operation is implemented at least in part by performing a bipartite search, which encodes information associated with individually entropy-transcoded packets with position-determined data from the video data sequence to generate individually entropy-transcoded packets. In some versions, the operation associated with block 804 may include performing an internal bipartite search when determining the size of the individually entropy-transcoded packets, which encodes information from the video data sequence in parallel into individually entropy-transcoded packets.

[0143] In one example of block 806, the operation includes encoding the packet number N and the total number of bytes S using a general codeword set, wherein the operation can further initialize the stack in memory by pushing the set onto the stack and encoding the video data stream.

[0144] In some examples, encoding the stream in block 806 includes the following operations: analyzing the stack to retrieve the current interval set; defining a new index; encoding the stream using a bisection; updating the stack using the set when the interval is empty; and updating the stack using the set when the interval is full. These operations are repeated until the stack is empty (e.g., the entire stream or an available portion of the stream is decoded).

[0145] In some examples of this type of block 806, the general codeword set is Elias delta code. Furthermore, in some examples of block 806, the operation of encoding using bisection includes the following: initializing variables. Then, when , the following operations are repeated: when , bit "1" is encoded and set; when , bit "0" is encoded and set; and updated. When , the above repeated operations are completed and encoding is complete.

[0146] At block 808, program 800 includes: generating an index for a sequence of video data that can be transcoded in parallel entropy. This index uses information associated with an entry point to identify individually transcoded packets within the sequence of video data that can be transcoded in parallel entropy. As previously mentioned, the index can be constructed in different ways, using position and packet length data, entry point data, or any combination of such data to define the structure of the encoded bitstream to be used for parallel decoding.

[0147] In some examples of block 808, such as those described above with respect to Figure 8, the index can be constructed as pairs of packets, where each pair of packets has a single entry point and a packet size for each packet. In such examples, every other entry point is defined in the index, and unincluded entry points (not in the index) can be inferred from the included entry points and packet sizes. In one such example, block 808 may include the operation of defining packets that can be individually entropy-transcoded as forward or reverse packets, where the corresponding packet pairs include the corresponding forward and reverse packets defined by a single shared entry point in the index. Some such examples will operate when the number of individually entropy-transcoded packets is even, the index includes N / 2 entry points, and when the number of individually entropy-transcoded packets is odd, the index includes [(N-1) / 2]+1 entry points.

[0148] Then, some examples of program 800 may store individually entropy-transcoding packets in memory. The individually entropy-transcoding packets are constructed as files with a file structure including an index and a plurality of parts, each part including a plurality of blocks, each block including a plurality of segments, and each segment including a forward packet structure and a reverse packet structure for a corresponding forward packet and a corresponding reverse packet for a corresponding packet pair.

[0149] Then, other examples of program 800 may store the encoded data in memory or send the encoded data. Some examples operate in the following cases: program 800 includes the operation of sending a file including an index and a separate entropy-transcoding packet.

[0150] Additionally, in some examples of program 800, the size of the individually entropy-transcoded packet is an integer value of the number of bits in the individually entropy-transcoded packet. In some examples of program 800, the index identifies the individually entropy-transcoded packet using an entry point identified after encoding the video data sequence in the encoded bitstream. In some examples of program 800, the index identifies the individually entropy-transcoded packet by identifying the bit length and position of each packet in the encoded video bitstream generated from the video data sequence.

[0151] Figure 8B is a flowchart illustrating an example of a program 850 for decoding video using operations for parallel entropy transcoding according to the examples described herein. Program 850 can be considered as a corresponding decoding program to the encoding program 800 of Figure 8A. As described above for program 800, in some examples, program 850 is executed by a device including memory and one or more processors (such as a SOC 900 or a device according to architecture 1300). In other examples, program 850 is implemented as computer-readable instructions stored in a storage medium that, when executed by processing circuitry, cause the decoding operation to be performed. In some examples, the device uses a decoding device such as decoding device 112 to execute program 850. In some examples, neural networks can be utilized to perform various forms of entropy encoding in order to achieve complex massive parallelization with the decoding system, as further described below.

[0152] At block 852, program 850 includes the operation of obtaining an encoded video data stream including an index. The encoded video data, including the video data sequence from block 806 and the video data from block 808, can be encoded using the method 800 described above. As mentioned earlier, the video data sequence can be any video data to be encoded using the parallel entropy coding described herein for storage or transmission. Such parallel transcoding can be particularly beneficial when single-threaded decoding results in increased data bottlenecks, such as ultra-high resolution video (e.g., 8K video or higher), high frame rate video (e.g., 60 frames per second, 120 frames per second or higher), or data streams with wider color resolution or color gamut, high dynamic range, or high quality. Parallel transcoding of video data sequences can also be used for artificial intelligence analytics or machine learning, where machines analyze video data in a manner unconstrained by standard time display (e.g., where a computer can "watch" and / or process two hours of film in one second, compared to humans displaying and watching video frames over two hours). Such parallel entropy transcoding allows machine learning or artificial intelligence systems to decode and process video data at a much higher rate compared to single-threaded transcoding systems.

[0153] At block 854, program 850 includes an operation that decodes the size of a separately entropy transcoded packet based on an index.

[0154] At block 856, program 850 includes: an operation of identifying individually entropy-transcoded packets of a video data sequence from an encoded video data stream using an indexed entry point. The operation of block 854 can be viewed as the inverse operation of operation 804, wherein the packet size identified in block 854 is then used to identify the position identified in block 804 during encoding, in order to identify individually entropy-transcoded packets in block 856. These operations use index values ​​from operation 808 to allow the identification of relevant information to allow decoding of the parallelizable entropy-transcoded video data sequence.

[0155] At block 858, program 800 includes the operation of decoding individually entropy-transcoded packets in parallel from an encoded video data stream using the decoded size of the packets (e.g., using packet bit size data and / or entry point data at packet boundaries). As described herein, block 858 may include parallel operations to allow packets in a parallel decoding system to be decoded independently from different locations within the video data sequence. After all packets of the video data sequence have been decoded, the packets can then be integrated to produce a complete video data sequence that can be stored or output to a display.

[0156] In addition to the bipartite enumeration and bipartite search operations described in methods 800 and 850 above, other implementations may use the prefix codes described in Figures 5B-5F as alternatives to the bipartite enumeration and bipartite search operations. In either case, the index value is used to encode and decode individual entropy-transcoding packets as described herein.

[0157] In some examples, the programs described herein (e.g., program 800 and / or other programs described herein) may be executed by a computing device or apparatus (such as a computing device having computing device architecture 1300 as shown in FIG13). In one example, program 800 may be executed by a computing device having computing device architecture 1300 that implements any of the architectures described herein. In some examples, a computing device may include a mobile device (e.g., a mobile phone, a tablet computing device, etc.), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television, a vehicle (or a computing device of a vehicle), a robotic device, and / or any other computing device having the resource capability to execute the programs described herein (including program 800).

[0158] In some examples, the systems and techniques described herein can be used to perform image and / or video transcoding using one or more machine learning (ML) systems. As previously mentioned, parallel transcoding of video data sequences can also be used for artificial intelligence analytics or machine learning, where the machine analyzes the video data in a manner unconstrained by standard time display. Parallel decoding of files (e.g., files using the file structure 710 described above) can be performed by a machine learning system to analyze the video file at the highest possible speed. In some examples, parallel entropy transcoding can significantly improve machine processing time when the machine learning system is configured to "watch" and / or process ultra-high-definition movies at a speed at which the video data can be decoded. Such parallel entropy transcoding can allow machine learning or artificial intelligence systems to decode and process video data at a much higher rate than single-threaded transcoding systems.

[0159] Broadly speaking, ML is a subset of artificial intelligence (AI). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and reasoning without using explicit instructions. An example of an ML system is a neural network (also known as an artificial neural network), which can include a set of interconnected artificial neurons (e.g., a neuron model). Neural networks can be used in a variety of applications and / or devices, such as image and / or video transcoding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, etc.

[0160] The nodes in a neural network can be analogous to biological neurons by taking input data and performing simple operations on the data. The results of the simple operations performed on the input data are selectively passed to other neurons. Weights are associated with each vector and node in the network, and the values ​​constrain how the input data relates to the output data. For example, the input data of each node can be multiplied by the corresponding weight value, and the products can be summed. The sum of the products can be adjusted by an optional bias, and an activation function can be applied to the results to produce the node's output signal or "output activation" (sometimes called an activation map or feature map). The weights can initially be determined by the iterative flow of training data through the network (e.g., weights are established during the training phase, during which the network learns how to identify a particular category based on the typical characteristics of its input data).

[0161] Different types of neural networks exist, such as spiral neural networks (CNNs), recursive neural networks (RNNs), generative adversarial networks (GANs), and multilayer perceptron (MLP) neural networks. For example, a spiral neural network (CNN) is a type of feedforward artificial neural network. A spiral neural network can include an ensemble of artificial neurons, each with a receptive field (e.g., a spatially localized region of the input space) that collectively tile the input space. RNNs work by storing the output of a layer and feeding that output back to the input to help predict the outcome of the layer. A GAN is a form of generative neural network that learns patterns in the input data, allowing the neural network model to produce new synthetic outputs that may reasonably originate from the original dataset. A GAN can include two neural networks operating together: a generative neural network that produces the synthetic output and a discriminative neural network that evaluates the authenticity of the output. In an MLP neural network, data can be fed into the input layer, and one or more hidden layers provide a level of abstraction to the data. Then, predictions can be made on the output layer based on the abstracted data.

[0162] In a hierarchical neural network architecture (when multiple hidden layers exist, it is called a deep neural network), the output of the first layer of artificial neurons becomes the input of the second layer of artificial neurons, the output of the second layer of artificial neurons becomes the input of the third layer of artificial neurons, and so on. For example, a CNN can be trained to identify hierarchical structures of features. The computation in a CNN architecture can be distributed across a group of processing nodes that can be configured in one or more computation chains. Such multi-layer architectures can be trained one layer at a time and can be fine-tuned using backpropagation.

[0163] Conventional compression methods are designed for sequential decoding and computation, and therefore their structure consists of only a few independent data blocks. Neural network-based compression enables large-scale parallel computation and decoding, utilizing a large number of independent data blocks (e.g., data packets). The examples described herein are able to efficiently index large numbers of data packets. Without such indexing, the packet indexing management burden (e.g., the amount of additional data required to encode the index into a large number of packets) would significantly reduce overall compression. The examples described herein avoid such degradation in overall compression used for neural network-based decoding (e.g., video encoding and decoding).

[0164] Figure 9 illustrates an example implementation of a System-on-a-Chip (SOC) 900, which may include a Central Processing Unit (CPU) 902 or a multi-core CPU configured to perform one or more functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with the computing device (e.g., weighted neural networks), latency, frequency band information, task information, and other information may be stored in memory blocks associated with a Neural Processing Unit (NPU) 908, in memory blocks associated with the CPU 902, in memory blocks associated with a Graphics Processing Unit (GPU) 904, in memory blocks associated with a Digital Signal Processor (DSP) 906, in memory block 918, and / or may be distributed across multiple blocks. Instructions executed at the CPU 902 may be loaded from program memory associated with the CPU 902 or may be loaded from memory block 918.

[0165] The SOC 900 may also include additional processing blocks tailored to specific functions, such as a GPU 904, a DSP 906, a connectivity block 910 (which may include fifth-generation (5G) connectivity, fourth-generation LTE (4G) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth™ connectivity, etc.), and a multimedia processor 912 (e.g., capable of detecting and recognizing gestures). In one implementation, the NPU is implemented in the CPU 902, DSP 906, and / or GPU 904. The SOC 900 may also include a sensor processor 914, an image signal processor (ISP) 916, and / or a navigation module 920 (which may include a global positioning system).

[0166] In some examples, the SOC 900 may be based on the ARM instruction set. In one embodiment of this invention, the instructions loaded into the CPU 902 may include code for searching a stored multiplication result in a lookup table (LUT) corresponding to the multiplication product of the input values ​​and filter weights. The instructions loaded into the CPU 902 may also include code for disabling the multiplier during the multiplication operation of the multiplication product when a lookup table hit for the multiplication product is detected. Additionally, the instructions loaded into the CPU 902 may include code for storing the calculated multiplication product of the input values ​​and filter weights when a lookup table hit for the multiplication product is detected.

[0167] The SOC 900 and / or its components may be configured to perform video compression and / or decompression (also known as video encoding and / or decoding, collectively referred to as video transcoding) using various machine learning techniques according to the present invention discussed herein. By using a deep learning architecture to perform video compression and / or decompression, various forms of the present invention can improve the efficiency of video compression and / or decompression on the device. For example, a device using the described video transcoding techniques may use machine learning-based techniques to compress video more efficiently, send compressed video to another device, and the other device may use the machine learning-based techniques described herein to decompress the compressed video more efficiently.

[0168] As mentioned above, neural networks are examples of machine learning systems and may include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes in the input layer, processed by hidden nodes in one or more hidden layers, and output is generated by output nodes in the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network may include a feature map or a starter map, which may include artificial neurons (or nodes). Feature maps may include filters, cores, etc. Nodes may include one or more weights used to indicate the importance of nodes in one or more of these layers. In some cases, deep learning networks may have a series of many hidden layers, where early layers are used to determine simple and low-level characteristics of the input, and later layers build a hierarchical structure of more complex and abstract characteristics.

[0169] Deep learning architectures can learn hierarchical structures of features. For example, if presented with visual data, the first layer can learn to identify relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to identify spectral power at a specific frequency. The second layer, using the output of the first layer as input, can learn to identify combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to identify common visual objects or spoken phrases.

[0170] Deep learning architectures can perform particularly well when applied to problems with a natural hierarchical structure. For example, classifying motorized vehicles can benefit from first learning to identify wheels, windshields, and other features. These features can then be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0171] Neural networks can be designed with a wide variety of connection patterns. In feedforward networks, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As mentioned earlier, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be passed to another neuron in the same layer. Recurrent architectures can help identify patterns across large blocks of input data that are sequentially delivered to the neural network from more than one layer. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when identifying generalized concepts can help identify specific low-level features of the input.

[0172] The connections between layers of a neural network can be fully connected or partially connected. Figure 10A illustrates an example of a fully connected neural network 1002. In a fully connected neural network 1002, neurons in the first layer can transmit their outputs to each neuron in the second layer, such that each neuron in the second layer will receive inputs from each neuron in the first layer. Figure 10B illustrates an example of a partially connected neural network 1004. In a partially connected neural network 1004, neurons in the first layer can connect to a limited number of neurons in the second layer. More generally, the locally connected layers of a partially connected neural network 1004 can be configured such that each neuron in a layer will have the same or similar connection patterns, but with connection strengths of different values ​​(e.g., 1010, 1012, 1014, and 1016). The linear patterns of locally connected neural networks can create spatially different receptive fields in higher layers because neurons in higher layers in a given region can receive inputs that are tuned by training to a limited portion of the network's total input.

[0173] An example of a locally connected neural network is a spiral neural network. Figure 10C illustrates an example of a spiral neural network 1006. The spiral neural network 1006 can be configured such that the connection strength associated with the input to each neuron in the second layer is shared (e.g., 1008). Spiral neural networks are well-suited for problems where the spatial location of the input is meaningful. Depending on the specific embodiments of this invention, the spiral neural network 1006 can be used to perform one or more modes of video compression and / or decompression.

[0174] One type of spiral neural network is the deep spiral network (DCN). Figure 10D illustrates a detailed example of a DCN 1000, which is designed to identify visual features from an image 1026 input from an image capturing device 1030, such as an in-vehicle camera. The DCN 1000 of this example can be trained to recognize traffic signs and the numerical values ​​provided on the traffic signs. Of course, the DCN 1000 can be trained for other tasks, such as recognizing lane markings or traffic lights.

[0175] The DCN 1000 can be trained using supervised learning. During training, an image (e.g., image 1026 of a speed limit sign) can be presented to the DCN 1000, and the forward pass can then be computed to produce an output 1022. The DCN 1000 may include a feature extraction segment and a classification segment. Upon receiving image 1026, the cyclotron layer 1032 may apply a cyclotron kernel (not shown) to image 1026 to produce a first feature map set 1018. As an example, the cyclotron kernel used for the cyclotron layer 1032 may be a 5x5 kernel that produces a 108x28 feature map. In this example, because four different feature maps are produced in the first feature map set 1018, four different cyclotron kernels are applied to image 1026 at the cyclotron layer 1032. The cyclotron kernel may also be referred to as a filter or a cyclotron filter.

[0176] The first feature map set 1018 may be resampled by a max-pooling layer (not shown) to generate a second feature map set 1020. The max-pooling layer reduces the size of the first feature map set 1018. That is, the size of the second feature map set 1020 (e.g., 94x14) is smaller than the size of the first feature map set 1018 (e.g., 108x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second feature map set 1020 may be further spiraled via one or more subsequent spiral layers (not shown) to generate one or more subsequent feature map sets (not shown).

[0177] In the example of Figure 10D, the second feature map set 1020 is spiraled to produce a first feature vector 1024. Furthermore, the first feature vector 1024 is spiraled further to produce a second feature vector 1028. Each feature of the second feature vector 1028 may include a number corresponding to a possible feature of the image 1026 (e.g., "symbol", "60", and "100"). A softmax function (not shown) can convert the numerical values ​​in the second feature vector 1028 into probabilities. Thus, the output 1022 of the DCN 1000 is the probability that the image 1026 includes one or more features.

[0178] In this example, the probabilities for "symbol" and "60" in output 1022 are higher than the probabilities for other terms in output 1022 (e.g., "30", "40", "50", "70", "80", "90", and "100"). Before training, output 1022 generated by DCN 1000 may be incorrect. Therefore, the error between output 1022 and the target output can be calculated. The target output is the ground truth value of image 1026 (e.g., "symbol" and "60"). The weights of DCN 1000 can then be adjusted so that output 1022 of DCN 1000 is more closely aligned with the target output.

[0179] To adjust the weights, the learning algorithm can compute gradient vectors for the weights. The gradient indicates the amount by which the error will increase or decrease as the weights are adjusted. At the top layers, the gradient can directly correspond to the values ​​of the weights that connect the starting neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient can depend on the values ​​of the weights and the computed error gradients from the higher layers. The weights can then be adjusted to reduce the error. This method of adjusting weights can be called "backpropagation" because it involves "passing backward" through the neural network.

[0180] In practice, the error gradient of the weights can be calculated using a small number of examples, making the calculated gradient close to the true error gradient. This approximation method can be called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system has stopped decreasing or until the error rate has reached the target level. After learning, a new image can be presented to the DCN, and the forward pass through the network can produce an output that can be considered an inference or prediction of the DCN.

[0181] A Deep Belief Network (DBN) is a probabilistic model that includes multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. DBNs can be obtained by stacking layers of a Restricted Boltzmann Machine (RBM). An RBM is a type of artificial neural network that learns a probability distribution over a set of inputs. Because RBMs can learn a probability distribution even without information about the category to which each input should be classified, they are commonly used in unsupervised learning. Using a hybrid unsupervised and supervised paradigm, the bottom RBM of a DBN can be trained unsupervised and can act as a feature extractor, while the top RBM can be trained supervised (based on the joint distribution of inputs and target categories from previous layers) and can act as a classifier.

[0182] Deep spiral networks (DCNs) are spiral networks configured with additional pooling and regularization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning (where the input and output targets are known for many paradigms) and the network weights can be modified using gradient descent.

[0183] A DCN can be a feedforward network. Furthermore, as mentioned earlier, the connections from neurons in the first layer of a DCN to a group of neurons in the next higher layer are shared among neurons in the first layer. The feedforward and shared connections of a DCN can be used for fast processing. For example, the computational burden of a DCN can be much smaller than that of a similarly sized neural network that includes recurrent or feedback connections.

[0184] The processing of each layer of the spiral network can be considered as a spatially invariant paradigm or basis projection. If the input is first decomposed into multiple channels (e.g., red, green, and blue channels of a color image), the spiral network trained on that input can be considered three-dimensional, with two spatial dimensions along the image's axes and a third dimension capturing color information. The output of the spiral connections can be considered as forming feature maps in subsequent layers, where each element of the feature map (e.g., 1020) receives input from a series of neurons in the previous layer (e.g., feature map 1018) and from each of the multiple channels. The values ​​in the feature maps can be further processed using nonlinearities (e.g., rectification max(0,x)). The values ​​from neighboring neurons can be further pooled (which corresponds to downsampling) and can provide additional local invariance and dimensionality reduction.

[0185] Figure 11 is a block diagram illustrating an example of a deep spiral network 1150. The deep spiral network 1150 may include multiple layers of different types based on connectivity and weight sharing. As shown in Figure 11, the deep spiral network 1150 includes spiral blocks 1154A and 1154B. Each of the spiral blocks 1154A and 1154B may be configured to have a spiral layer (CONV) 1156, a normalization layer (LNorm) 1158, and a max pooling layer (MAX POOL) 1160.

[0186] The cyclotron layer 1156 may include one or more cyclotron filters, which may be applied to the input data 1152 to generate a feature map. Although only two cyclotron blocks 1154A and 1154B are illustrated, the scope of this invention is not limited thereto; rather, any number of cyclotron blocks (e.g., blocks 1154A and 1154B) may be included in the deep cyclotron network 1150, depending on design preference. The normalization layer 1158 may normalize the output of the cyclotron filters. For example, the normalization layer 1158 may provide whitening or lateral suppression. The max-pooling layer 1160 may provide spatial downsampling aggregation for local invariance and dimensionality reduction.

[0187] For example, the parallel filter bank of the deep cyclotron network can be loaded onto the CPU 902 or GPU 904 of the SOC 900 to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank can be loaded onto the DSP 906 or ISP 916 of the SOC 900. Furthermore, the deep cyclotron network 1150 can access other processing blocks that may exist on the SOC 900, such as a sensor processor 914 dedicated to sensors and a navigation module 920, respectively.

[0188] The deep spiral network 1150 may also include one or more fully connected layers, such as layer 1162A (labeled "FC1") and layer 1162B (labeled "FC2"). The deep spiral network 1150 may also include a logistic regression (LR) layer 1164. Weights (not shown) to be updated between each layer 1156, 1158, 1160, 1162A, 1162B, and 1164 of the deep spiral network 1150. The output of each of these layers (e.g., 1156, 1158, 1160, 1162A, 1162B, 1164) can serve as input to the next layer in the deep spiral network 1150 (e.g., 1156, 1158, 1160, 1162A, 1162B, 1164) to learn hierarchical feature representations based on the input data 1152 (e.g., image, audio, video, sensor data, and / or other input data) provided at the beginning of spiral block 1154A. The output of the deep spiral network 1150 is a classification score 1166 for the input data 1152. The classification score 1166 can be a set of probabilities, where each probability is the probability that the input data includes features from the feature set.

[0189] As mentioned above, digital video data can include large amounts of data, which can place a significant burden on communication networks and the devices that process and store the video data. For example, recording uncompressed video content generally results in a much larger file size, which increases significantly with the resolution of the recorded video content. In an illustrative example, uncompressed 16-bit video per channel recorded at 1080p / 24 (e.g., a resolution of 1920 pixels in width and 1080 pixels in height, capturing 24 frames per second) may occupy 12.4 megabytes per frame or 297.6 megabytes per second. Uncompressed 16-bit video per channel recorded at 12K resolution with 24 frames per second may occupy 129.8 megabytes per frame or 1195.2 megabytes per second.

[0190] Network bandwidth is another constraint that can make large video files problematic. For example, video content is often delivered over wireless networks (e.g., via LTE, improved LTE, New Radio (NR), WiFi™, Bluetooth™, or other wireless networks) and can constitute a large portion of consumer internet traffic. Despite advancements in the amount of available bandwidth in wireless networks, it may still be desirable to reduce the amount of bandwidth used to deliver video content over these networks.

[0191] Since uncompressed video content can result in large files, which may involve a considerable amount of memory for physical storage and a large amount of bandwidth for transmission, video transcoding technology can be used to compress and then decompress such video content.

[0192] To reduce the size of video content, and thus the amount of storage and bandwidth required for storing and transmitting video content, video transcoding techniques can be performed according to specific video transcoding standards (e.g., HEVC, AVC, MPEG, VVC, etc.). Video transcoding typically uses prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.), which utilize redundancy present in video images or sequences. The common goal of video transcoding techniques is to compress video data into a form using a lower bit rate while avoiding or minimizing degradation in video quality. As the demand for video services grows and new video services become available, there is a need for transcoding techniques with better transcoding efficiency, performance, and rate control.

[0193] Typically, encoding devices encode video data according to a video transcoding standard to produce an encoded video bitstream. In some examples, the encoded video bitstream (or "video bitstream" or "bitstream") is a series of one or more decoded video sequences. Encoding devices can produce a decoded representation of a picture by dividing each picture into multiple slices. A slice is independent of other slices, such that the information in that slice is decoded without relying on data from other slices within the same picture. A slice includes one or more segments, which include independent segments and (if present) one or more dependent segments that depend on previous segments. In HEVC, the slices are then divided into transcoding tree blocks (CTBs) for luma sampling and chroma sampling. The luma-sampled CTB and the chroma-sampled CTBs, along with the syntax used for sampling, are called decoding tree units (CTUs). CTUs can also be called "tree blocks" or "maximum transcoding units" (LCUs). The CTU is the basic processing unit used for HEVC encoding. A CTU can be divided into multiple transcoding units (CUs) of different sizes. A CU contains luminance and chrominance sample arrays called transcoding blocks (CBs).

[0194] Luminance and chrominance CBs can be further subdivided into prediction blocks (PBs). A PB is a sampled block of the luminance or chrominance component that uses the same motion parameters for inter-frame prediction or intra-block copy (IBC) prediction (when available or enabled for use). The luminance PB and one or more chrominance PBs, along with their associated syntax, form a prediction unit (PU). For inter-frame prediction, a set of motion parameters (e.g., one or more motion vectors, reference indices, etc.) is signaled in the bitstream for each PU and used for inter-frame prediction of the luminance PB and one or more chrominance PBs. Motion parameters may also be referred to as motion information. CBs can also be subdivided into one or more transform blocks (TBs). A TB represents a square block of sampled chrominance components where a residual transform (e.g., in some cases, the same two-dimensional transform) is applied to the TB to transcode the prediction residual signal. A transform unit (TU) represents the TB of luminance and chrominance samples and the corresponding syntax elements. Transform decoding is described in more detail below.

[0195] A TU can be used to perform the transformation. The size of the TU can be set based on the size of the PU within a given CU. The TU can have the same size as or smaller than the PU. In some examples, a quadtree structure called a residual quadtree (RQT) can be used to subdivide the residual samples corresponding to the CU into smaller units. The leaf nodes of the RQT can correspond to the TU. The primitive differences associated with the TU can be transformed to produce transform coefficients. The transform coefficients can then be quantized by an encoding device.

[0196] When the images of video data are segmented into CUs, the encoding device uses a prediction mode to predict each PU. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (described below). For each CU, the prediction mode can be signaled in the bitstream using syntax data. The prediction mode can include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (or inter-picture prediction). Intra-frame prediction utilizes the correlation between samples that are spatially adjacent within the image. For example, using intra-frame prediction, each PU is predicted from adjacent image data in the same image, using, for example, DC prediction to find an average value for the PU, using planar prediction to adapt a planar surface to the PU, using orientation prediction to extrapolate from adjacent data, or using any other appropriate prediction type. Inter-frame prediction uses temporal correlation between images to derive motion-compensated predictions for blocks of image samples. For example, using inter-frame prediction, each PU is predicted using motion-compensated prediction based on image data in one or more reference images (in the order of output before or after the current image). For example, a decision can be made at the CU level regarding whether to use inter-image prediction or intra-image prediction to decode image regions.

[0197] After performing prediction using intra-frame prediction and / or inter-frame prediction, the encoding device may perform transformation and quantization. For example, after prediction, the encoding device may calculate a residual value corresponding to the PU. The residual value may include the primitive difference between the current block (PU) of the primitives being decoded and the prediction block used to predict the current block (e.g., a prediction version of the current block). For example, after generating a prediction block (e.g., performing inter-frame prediction or intra-frame prediction), the encoding device may generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of primitive differences that quantize the difference between the primitive values ​​of the current block and the primitive values ​​of the prediction block. In some examples, the residual block may be represented in a two-dimensional block format (e.g., a two-dimensional matrix or array of primitive values). In such examples, the residual block is a two-dimensional representation of the primitive values.

[0198] Block transforms are used to transform any residual data that may remain after prediction is performed. These block transforms can be based on discrete cosine transform, discrete sine transform, integer transform, wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., sizes of 112 x 112, 16 x 16, 8 x 8, 12 x 12, or other suitable sizes) can be applied to the residual data in each CU. In some embodiments, TUs can be used for transform and quantization procedures implemented by the coding device. A given CU having one or more PUs may also include one or more TUs. As described in further detail below, residual values ​​can be transformed into transform coefficients using block transforms, and then quantized and scanned using TUs to produce serialized transform coefficients for entropy transcoding.

[0199] The encoding device can perform quantization of the transform coefficients. Quantization provides further compression by reducing the amount of data used to represent the coefficients. For example, quantization can reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient with an n-bit value can be rounded down to an m-bit value during quantization, where n is greater than m.

[0200] During quantization, the transcoded video bitstream includes quantized transform coefficients, prediction information (e.g., prediction patterns, motion vectors, block vectors, etc.), segmentation information, and any other suitable data (such as other grammatical data). The different elements of the transcoded video bitstream can then be entropy-encoded by an encoding device. In some examples, the encoding device may use a predefined scanning order to scan the quantized transform coefficients to produce a serialized vector that can be entropy-encoded. In some examples, the encoding device may perform adaptive scanning. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoding device may entropy-encode that vector. For example, the encoding device may use context-adaptive variable-length transcoding, context-adaptive binary arithmetic transcoding, grammar-based context-adaptive binary arithmetic transcoding, probabilistic interval segmentation entropy transcoding, or another suitable entropy coding technique.

[0201] The encoding device may store the encoded video bitstream and / or transmit the encoded video bitstream data to a receiving device (which may include a decoding device) over a communication link. The decoding device may decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more transcoded video sequences that make up the encoded video data. The decoding device may then rescale the encoded video bitstream data and perform an inverse transform on the encoded video bitstream data. The residual data is then passed to the prediction stage of the decoding device. The decoding device then uses intra-frame prediction, inter-frame prediction, IBS, and / or other types of prediction to predict blocks of primitives (e.g., PUs). In some examples, the prediction is added to the output of the inverse transform (the residual data). The decoding device may output the decoded video to a video destination device, which may include a display or other output device for displaying the decoded video data to a consumer of content.

[0202] The descriptive nature of the content of this case includes:

[0203] State 1. An apparatus for encoding video data, the apparatus comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: obtain a video data sequence; identify a position in the video data sequence associated with an entry point of an individually entropy-transcoded packet for parallel entropy transcoding of the video data sequence; generate the parallel entropy-transcoded video data sequence; and generate an index for the parallel entropy-transcoded video data sequence, the index identifying the individually entropy-transcoded packet within the parallel entropy-transcoded video data sequence.

[0204] State 2A, the apparatus according to state 1, wherein the index is generated by sequentially increasing the number of bits used to transcode the value from ⌊log2(Np)⌋ bits to ⌊log2(Np)⌋+1 in the first and last values ​​of the set of possible values ​​of the video data sequence.

[0205] State 2B, the apparatus according to State 1, wherein the entry point is identified at least in part by performing a bipartite enumeration of the video data sequence.

[0206] State 3, the apparatus according to State 2B, wherein the parallelizable entropy transcoded video data sequence is generated at least in part by performing a bipartite search, which encodes information associated with the individually entropy transcoded packet with position-determined data from the video data sequence to generate the individually entropy transcoded packet.

[0207] State 4, the apparatus according to State 3, wherein, in order to perform the bisection enumeration, the one or more processors are configured to: bisection the video data sequence to identify an initial position in the location, wherein the initial position is associated with a first entry point and a subsequence divided by the initial position; and bisection the subsequence hierarchy to identify the remaining portion of the location.

[0208] State 5, the device according to state 3, wherein the individually entropy-transcoded packet includes corresponding data determined by the position from the video data sequence, and information about the individually entropy-transcoded packet.

[0209] State 6, the apparatus according to state 5, wherein the information about the individually entropy-transcoded packet is information about the scale of the binary representation of the corresponding data from the video data sequence.

[0210] State 7, the apparatus according to state 3, wherein the difference between the largest and smallest packets in the individually entropy-transcoding packets is less than the threshold difference.

[0211] State 8, the apparatus according to State 3, wherein in order to generate the parallelizable entropy transcoding sequence, the one or more processors are configured to: identify multiple packets to define a set of individually entropy transcoding packets from the first packet to the last packet having packet sizes between the minimum and maximum packet sizes, wherein the sum of the packet sizes of the multiple packets is the size of the parallelizable entropy transcoding sequence.

[0212] State 9, the apparatus according to State 8, wherein, in order to generate the parallelizable entropy-encoded video data sequence, the one or more processors are configured to: initialize a stack in the memory comprising a plurality of packets and a scale for a binary representation of the corresponding data for the plurality of packets; and generate the parallelizable entropy-encoded video data sequence at least in part by performing the following operations until the stack is empty: analyze the stack to retrieve the current interval set; select the packet by defining a new index for the packet as the average of the current interval set; encode the bits of the packet using the average of the current interval set using a bisection; update the stack using a set from the first packet to the packet when the packet is not the first packet; and update the stack using a set from the packet to the last packet when the packet used is not the last packet.

[0213] State 10, the device according to state 9, wherein the stack is still initialized using a common encoding character set.

[0214] State 11. The apparatus according to State 9, wherein, in order to encode the bit of the packet using bisection, the one or more processors are configured to: initialize a first variable to the minimum packet size, initialize a second variable to the maximum packet size, and initialize a third variable to the average of the minimum packet size and the maximum packet size; access the value of the scale of the packet in binary representation; encode the first bit when the third variable is less than or equal to the value; encode the second bit when the third variable is greater than the value; and update the third variable to the new average of the first variable and the second variable.

[0215] State 12, the apparatus according to State 3, wherein the one or more processors are further configured to: define a packet in the individually entropy-transcoding packet as a forward packet or a reverse packet; wherein the corresponding packet pair includes a corresponding forward packet and a corresponding reverse packet defined using a single shared entry point in the index.

[0216] State 13, the apparatus according to state 11, wherein when the number of individually entropy-transcoding packets is even, the index includes N / 2 entry points, and when the number of individually entropy-transcoding packets is odd, the index includes [(N-1) / 2]+1 entry points.

[0217] State 14. The apparatus according to State 11, wherein the one or more processors are further configured to: store the individually entropy-transcoding packet in the memory, the individually entropy-transcoding packet being constructed as a file having a file structure including the index and a plurality of parts, each part including a plurality of blocks, each block including a plurality of segments, and each segment including a forward packet structure and a reverse packet structure for the corresponding forward packet and the corresponding reverse packet of the corresponding packet pair.

[0218] State 15, the apparatus according to state 13, wherein the one or more processors are further configured to: send the file including the index and the individually entropy-transcoded packet.

[0219] State 16, the apparatus according to State 3, wherein the index is used to identify the individually entropy transcoded packet by an entry point identified after encoding the video data sequence in the encoded bit stream.

[0220] State 17, the apparatus according to State 3, wherein the index identifies the individually entropy-transcoded packet, including identifying the bit length and position of each packet of the encoded video bitstream generated from the video data sequence.

[0221] State 18, the apparatus according to State 3, further includes: a transmitter coupled to the one or more processors and configured to transmit the video data sequence entropy encoded using the index and the individually entropy transcoding packet.

[0222] State 19. The apparatus according to State 3 further includes: a display coupled to the one or more processors and configured to display an image of the video data sequence.

[0223] Sample 20. A computer-implemented method comprising: obtaining a video data sequence; identifying a position in the video data sequence associated with an entry point of an individually entropy-transcoding packet for parallel entropy transcoding of the video data sequence; generating the parallel entropy-transcoding video data sequence; and generating an index for the parallel entropy-transcoding video data sequence, the index identifying the individually entropy-transcoding packet within the parallel entropy-transcoding video data sequence.

[0224] State 21, a computer-implemented method according to State 20, wherein the entry point is identified at least in part by performing a bipartite enumeration of the video data sequence; and wherein the parallelizable entropy transcoding video data sequence is generated at least in part by performing a bipartite search, which encodes information associated with the individually entropy transcoding packet and position-determined data from the video data sequence to generate the individually entropy transcoding packet.

[0225] State 22, the computer implementation of the method according to State 21, wherein in order to perform the bisection enumeration, the one or more processors are configured to: bisection the video data sequence to identify an initial position in the location, wherein the initial position is associated with a first entry point and a subsequence divided by the initial position; and bisection the subsequence hierarchy to identify the remaining portion of the location.

[0226] State 23, the computer-implemented method according to State 21, wherein the individually entropy-transcoding packet includes corresponding data determined by the position from the video data sequence, and information about the individually entropy-transcoding packet.

[0227] State 24, the computer implementation method according to State 21, wherein the information about the individually entropy transcoded packet is information about the scale of the binary representation of the corresponding data from the video data sequence.

[0228] State 25, the computer implementation of the method according to State 21, wherein in order to generate the parallelizable entropy transcoding sequence, the one or more processors are configured to: identify multiple packets to define a set of individually entropy transcoding packets from the first packet to the last packet having sizes between the minimum packet size and the maximum packet size, wherein the sum of the packet sizes of the multiple packets is the size of the parallelizable entropy transcoding sequence.

[0229] State 26. A computer-implemented method according to State 25, wherein, in order to generate the parallelizable entropy-encoded video data sequence, the one or more processors are configured to: initialize a stack in the memory comprising a plurality of packets and a scale for a binary representation of the corresponding data of the plurality of packets; and generate the parallelizable entropy-encoded video data sequence at least in part by performing the following operations until the stack is empty: analyze the stack to retrieve the current interval set; select the packet by defining a new index for the packet as the average of the current interval set; encode the bits of the packet using the average of the current interval set using a bisection; update the stack using a set from the first packet to the packet when the packet is not the first packet; and update the stack using a set from the packet to the last packet when the packet used is not the last packet.

[0230] State 27. The computer implementation method according to State 25, wherein, in order to encode the bit of the packet using bisection, the one or more processors are configured to: initialize a first variable to the minimum packet size, initialize a second variable to the maximum packet size, and initialize a third variable to the average of the minimum packet size and the maximum packet size; access the value of the scale of the binary representation of the packet; encode the first bit when the third variable is less than or equal to the value; encode the second bit when the third variable is greater than the value; and update the third variable to the new average of the first variable and the second variable.

[0231] State 28, the computer implementation of the method according to State 27, wherein the one or more processors are further configured to: send the file including the index and the individually entropy transcoding packet.

[0232] State Sample 29. An apparatus for decoding video data, the apparatus comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: obtain an encoded video data stream including an index and a sequence of video data that can be parallelized entropy-transcoded, the sequence of video data that can be parallelized entropy-transcoded including individually entropy-transcoded packets that can be identified using the index; decode the index for data associated with the individually entropy-transcoded packets; use the data from the index to identify the individually entropy-transcoded packets from the encoded video data stream; and use the data decoded according to the index to decode the individually entropy-transcoded packets in parallel from the encoded video data stream.

[0233] 30. A computer-implemented method comprising: obtaining an encoded video data stream including an index and a sequence of video data that can be parallelized entropy-transcoded, the sequence of video data including individually entropy-transcoded packets that can be identified using the index; decoding the index for data associated with the individually entropy-transcoded packets; using the data from the index to identify the individually entropy-transcoded packets from the encoded video data stream; and using the data decoded according to the index to decode the individually entropy-transcoded packets in parallel from the encoded video data stream.

[0234] State Sample 31, an apparatus for encoding video data, the apparatus comprising: a component for obtaining a sequence of video data; a component for identifying a position in the video data sequence associated with an entry point of an individually entropy-transcoded packet for parallel entropy transcoding of the video data sequence; a component for generating the parallel entropy-transcoded video data sequence; and a component for generating an index for the parallel entropy-transcoded video data sequence, the index identifying the individually entropy-transcoded packet within the parallel entropy-transcoded video data sequence.

[0235] State 32, the apparatus according to state 31 further includes: a component for performing a bipartite enumeration of the video data sequence.

[0236] State 33, the apparatus according to state 32 further includes: a component for generating the parallelizable entropy transcoded video data sequence using a component for performing a bipartite search, the bipartite search encoding information associated with the individually entropy transcoded packet with position-determined data from the video data sequence to generate the individually entropy transcoded packet.

[0237] 34. The apparatus according to 32 further includes a component for performing any operation according to 1-33 above.

[0238] 35. A non-transitory computer-readable medium, including instructions which, when executed by one or more processors, cause a device to perform any of the operations of the above-described formats 1-33.

[0239] Figure 12 illustrates system 1200, which includes device 1202 configured to perform video encoding and decoding system 1210. Device 1202 is coupled to camera 1207 and storage medium 1214 (e.g., data storage device). In some implementations, camera 1207 is configured to provide image data 1208 (e.g., video data stream) to processor 1204 for encoding by system 1210. In some implementations, device 1202 may be coupled to and / or may include multiple cameras (e.g., dual-camera system, three cameras, or other number of cameras). In some cases, device 1202 may be coupled to microphone and / or other input devices (e.g., keyboard, mouse, touch input device such as touch screen and / or touchpad, and / or other input devices). In some examples, camera 1207, storage medium 1214, microphone and / or other input devices may be part of device 1202.

[0240] Device 1202 is also coupled to second device 1290 via transmission medium 1218 (e.g., one or more wireless networks, one or more wired networks, or a combination thereof). For example, transmission medium 1218 may include a channel provided by a wireless network, a wired network, or a combination of wired and wireless networks. Transmission medium 1218 may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. Transmission medium 1218 may include a router, a switch, a base station, or any other device that can be used to facilitate communication from the source device to the receiving device. The wireless network may include any wireless interface or combination of wireless interfaces and may include any suitable wireless network (e.g., the Internet or other wide area networks, packet-based networks, WiFi™, radio frequency (RF), UWB, WiFi Direct, cellular, LTE, WiMax™, etc.). Wired networks can include any wired interface (e.g., fiber optic, Ethernet, powerline Ethernet, Ethernet via coaxial cable, digital signal line (DSL), etc.). Wired and / or wireless networks can be implemented using various devices, such as base stations, routers, access points, bridges, gateways, switches, etc. Encoded video bitstream data can be modulated according to communication standards such as wireless communication protocols and transmitted to receiving devices.

[0241] Device 1202 includes one or more processors (referred to herein as "processors") 1204 coupled to memory 1206, a first interface ("I / F 1") 1212, and a second interface ("I / F 2") 1216. Processor 1204 is configured to receive image data 1208 from camera 1207, from memory 1206, and / or from storage medium 1214. Processor 12012 is coupled to storage medium 1214 via first interface 1212 (e.g., via memory bus) and to transmission medium 1218 via second interface 1216 (e.g., network peripheral device, wireless transceiver and antenna, one or more other network peripheral devices, or combinations thereof).

[0242] Processor 1204 includes system 1210. System 1210 includes encoder portion 1262 and decoder portion 1266. In some implementations, system 1210 may include one or more auto encoders. Encoder portion 1262 is configured to receive input data 1270 and process the input data 1270 to generate output data 1274 based at least in part on the input data 1270.

[0243] In some implementations, the encoder portion 1262 of system 1210 is configured to perform distortion compression on input data 1270 to produce output data 1274, such that output data 1274 has fewer bits compared to input data 1270. The encoder portion 1262 can be trained to compress input data 1270 (e.g., an image or video frame) without using motion compensation based on any previous representation (e.g., one or more previously reconstructed frames). For example, the encoder portion 1262 can compress a video frame using only video data from the frame, without using any data from previously reconstructed frames. The video frame processed by the encoder portion 1262 may herein be referred to as an in-frame predicted frame (I-frame).

[0244] In some implementations, the encoder portion 1262 of system 1210 may be trained to compress input data 1270 (e.g., video frames) using motion compensation based on previous representations within independent entropy-based transcoding packets as described herein. For example, encoder portion 1262 may compress the video frame using video data from the video frame and data from previously reconstructed frames, or data from portions of the frames, or other data as part of a single packet that can be processed in parallel as described herein. The video frame processed by encoder portion 1262 may herein be referred to as an in-frame predicted frame (P-frame). Motion compensation may be used to determine the data of the current frame by describing how primitives from previously reconstructed frames move to new positions within the current frame and residual information.

[0245] As shown in the figure, the encoder portion 1262 of system 1210 may include a neural network 1263 and a quantizer 1264. The neural network 1263 may include one or more spiral neural networks (CNNs), one or more fully connected neural networks, one or more gated recurrent units (GRUs), one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and / or other types of neural network architectures that generate intermediate data 1272. The intermediate data 1272 is input to the quantizer 1264.

[0246] The quantizer 1264 is configured to perform quantization of intermediate data 1272 and, in some cases, entropy transcoding of intermediate data 1272 to produce output data 1274. Output data 1274 may include quantized (and, in some cases, entropy transcoded) data. The quantization operation performed by the quantizer 1264 may result in the generation of quantized code (or data representing the quantized code generated by the system 1210) from intermediate data 1272. The quantized code (or data representing the quantized code) may also be referred to as latent code or latent quantity (denoted as z). The entropy model applied to the latent quantity may be referred to herein as "a priori". In some examples, the quantization and entropy transcoding operations may be performed using existing quantization and / or entropy transcoding operations performed when encoding and / or decoding video data according to existing video transcoding standards. In some examples, the quantization and / or entropy transcoding operations may be performed by the system 1210 after a bisection operation for identifying independently entropy transcoded packets of video data. In an illustrative example, supervised training can be used to train system 1210, where residual data is used as input during training, and quantization codes and entropy codes are used as known outputs (labels).

[0247] The decoder portion 1266 of system 1210 is configured to receive output data 1274 (e.g., directly from quantizer 1264 and / or from storage medium 1214). The decoder portion 1266 can process the output data 1274 to produce a representation 1276 of input data 1270 based at least in part on the output data 1274. In some examples, the decoder portion 1266 of system 1210 includes a neural network 1268, which may include one or more CNNs, one or more fully connected neural networks, one or more GRUs, one or more Long Short-Term Memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and / or other types of neural network architectures.

[0248] Processor 1204 is configured to send output data 1274 to at least one of transmission medium 1218 or storage medium 1214. For example, output data 1274 may be stored on storage medium 1214 for later retrieval and decoding (or decompression) by decoder section 1266 to produce a representation 1276 of input data 1270 as reconstructed data. The reconstructed data may be used for various purposes, such as for replaying video data that has been encoded / compressed to produce output data 1274. In some implementations, output data 1274 may be decoded at another decoder device matched with decoder section 1266 (e.g., in device 1202, in second device 1290, or in another device) to produce a representation 1276 of input data 1270 as reconstructed data. For example, the second device 1290 may include a decoder that matches (or substantially matches) the decoder section 1266, and output data 1274 may be transmitted to the second device 1290 via transmission medium 1218. The second device 1290 may process the input data 1274 to produce a representation 1276 of the input data 1270 as reconstructed data.

[0249] The components of system 1200 may include electronic circuits or other electronic hardware and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0250] Although system 1200 is shown to include certain elements, those skilled in the art will understand that system 1200 may include more or fewer components than those shown in FIG. 12. For example, in some cases, system 1200 may also include input devices and output devices (not shown), or may be part of a computing device that includes input devices and output devices. In some implementations, system 1200 may also include or be part of a computing device that includes the following: one or more memory devices (e.g., one or more random access memory (RAM) elements, read-only memory (ROM) elements, cache memory elements, buffer elements, database elements, and / or other memory devices); one or more processing devices (e.g., one or more CPUs, GPUs, and / or other processing devices) that communicate with and / or are electrically connected to one or more memory devices; one or more wireless interfaces for performing wireless communication (e.g., including one or more transceivers and baseband processors for each wireless interface); one or more wired interfaces for performing communication over one or more hardwired connections (e.g., serial interfaces such as Universal Serial Bus (USB) inputs, lighting connectors, and / or other wired interfaces); and / or other elements not shown in FIG12.

[0251] In some implementations, system 1200 may be implemented locally by and / or included in a computing device. For example, a computing device may include a mobile device, a personal computer, a tablet computer, a virtual reality (VR) device (e.g., a head-mounted display (HMD) or other VR device), an augmented reality (AR) device (e.g., an HMD, AR glasses or other AR device), a wearable device, a server (e.g., in a Software as a Service (SaaS) system or other server-based system), a television set, and / or any other computing device with the resource capability to perform the techniques described herein.

[0252] In one example, system 1210 may be incorporated into a portable electronic device including: memory 1206 coupled to processor 1204 and configured to store instructions executable by processor 1204; and a wireless transceiver coupled to antenna and processor 1204 and operable to transmit output data 1274 to a remote device.

[0253] In some examples, the systems and techniques described herein can be used at the input of a neural network-based transcoding system that combines a combination of a spiral (e.g., a CNN) and a regularization stage, compatible with independent entropy transcoding packet structures for parallel transcoding.

[0254] In some cases, a computing device or apparatus may include various elements such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more transmitters, receivers or combinations thereof (e.g., referred to as transceivers), one or more cameras, one or more sensors, and / or other elements configured to perform the steps of the procedures described herein. In some examples, a computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or other elements. The network interface may be configured to transmit and / or receive Internet Protocol (IP) based data or other types of data.

[0255] The components of a computing device may be implemented in a circuit. For example, the components may include and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), neural processing unit (NPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0256] Programs 800 and 850 are shown as logic flowcharts, the actions of which represent a series of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, these actions represent computer-executable instructions stored on one or more computer-readable storage media, which, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the described operations can be combined in any order and / or in parallel to implement these programs.

[0257] Furthermore, the programs described herein (including program 800, program 850, and / or other programs described herein) can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, implemented by hardware, or a combination thereof. As described above, the code can be stored, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors on a computer-readable or machine-readable storage medium. The computer-readable or machine-readable storage medium can be non-transitory.

[0258] Figure 13 illustrates an example computing device architecture 1300 that can implement the various technologies described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device of a vehicle), or other devices. For example, computing device architecture 1300 can implement the system of Figure 6. The elements of computing device architecture 1300 are shown to be in electrical communication with each other using a connection 1305 (e.g., a bus). The example computing device architecture 1300 includes a processing unit (CPU or processor) 1310 and a computing device connection 1305 that couples various computing device elements, including computing device memory 1315 (e.g., read-only memory (ROM) 1313 and random access memory (RAM) 1325), to the processor 1310.

[0259] The computing device architecture 1300 may include a cache memory for high-speed memory, which is directly connected to, close to, or integrated into the processor 1310. The computing device architecture 1300 can copy data from memory 1315 and / or storage device 1330 to the cache memory 1312 for fast access by the processor 1310. In this way, the cache memory can provide performance improvements by avoiding delays for the processor 1310 while waiting for data. These and other modules can control or be configured to control the processor 1310 to perform various operations. Other computing device memories 1315 may also be available. Memory 1315 may include various different types of memory with different performance characteristics. Processor 1310 may include any general-purpose processor and hardware or software services (such as services 1 1332, 2 1334, and 3 1336 stored in storage device 1330), which are configured to control processor 1310 and dedicated processors in which software instructions are incorporated into the processor design. Processor 1310 may be a self-contained system containing multiple cores or processors, buses, memory controllers, cache memory, etc. Multi-core processors may be symmetric or asymmetric.

[0260] To enable user interaction with the computing device architecture 1300, the input device 1345 can represent any number of input mechanisms, such as a microphone for voice, a sensitive touchscreen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The output device 1335 can also be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, projector, television, speaker equipment, etc. In some cases, a multimodal computing device can enable the user to provide multiple types of input to communicate with the computing device architecture 1300. The communication interface 1340 can typically control and manage user input and computing device output. There are no limitations on operation on any particular hardware arrangement, and therefore, the basic features here can be easily replaced by improved hardware or firmware configurations (as they are developed).

[0261] Storage device 1330 is non-volatile memory and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as magnetic tape, flash memory card, solid-state memory device, digital multifunction disk, cassette tape, random access memory (RAM) 1325, read-only memory (ROM) 1320, and mixtures thereof. Storage device 1330 may include services 1332, 1334, 1336 for controlling processor 1310. Other hardware or software modules are contemplated. Storage device 1330 may be connected to computing device connection 1305. In one embodiment, a hardware module performing a specific function may include software elements stored in a computer-readable medium connected to hardware elements necessary to perform that function, such as processor 1310, connection 1305, output device 1335, etc.

[0262] The variations of this content apply to any suitable electronic device (such as a security system, smartphone, tablet, laptop, vehicle, drone or other device) that includes or is coupled to one or more active depth sensing systems. Although devices having or coupled to a light projector are described below, the variations of this content apply to devices having any number of light projectors and are therefore not limited to any particular device.

[0263] The term "device" is not limited to one or a specific number of physical objects (such as a smartphone, a controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more components that can implement at least some parts of the content of this application. Although the term "device" is used in the following description and examples to describe various aspects of the content of this application, the term "device" is not limited to a specific configuration, type, or number of objects. Furthermore, the term "system" is not limited to multiple elements or a specific embodiment. For example, a system can be implemented on one or more printed circuit boards or other substrates and can have movable or static elements. Although the term "system" is used in the description and examples herein to describe various aspects of the content of this application, the term "system" is not limited to a specific configuration, type, or number of objects.

[0264] Specific details are provided in the foregoing description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that these embodiments can be practiced without these specific details. For clarity, in some instances, the techniques described herein may be presented as comprising individual functional blocks including devices, device elements, steps or routines in a method embodied in software, or a combination of hardware and software. Additional elements may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, programs, and other elements may be shown as elements in block diagram form so as not to obscure these embodiments with unnecessary detail. In other instances, well-known circuits, programs, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring these embodiments.

[0265] The various embodiments described above can be presented as programs or methods, illustrated as flowcharts, schematic diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts can describe operations as a sequential program, many of these operations can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A program terminates when its operations are completed, but may have additional steps not included in the diagram. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0266] The programs and methods described in the above examples can be implemented using computer-executable instructions, which are stored in or otherwise accessible from a computer-readable medium. Such instructions may include, for example, instructions or data that cause a general-purpose computer, special-purpose computer, or processing device to perform a specific function or a specific set of functions, or otherwise configure it to perform a specific function or a specific set of functions. The computer resources used may be accessible via a network. Computer-executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc.

[0267] The term "computer-readable media" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored, but does not include carrier waves and / or transient electronic signals propagated wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic or magnetic discs, optical storage media (such as flash memory), memory or memory devices, magnetic or optical discs, flash memory, USB devices providing non-volatile memory, networked storage devices, optical discs (CDs) or digital versatile discs (DVDs), any suitable combinations thereof, etc. Computer-readable media may have code and / or machine-executable instructions stored thereon, which may represent any combination of programs, functions, subroutines, programs, routines, subroutines, modules, software packages, software components, or instructions, data structures, or program statements. Code segments can be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or sent via any appropriate means, including memory sharing, message passing, scepter passing, network transmission, etc.

[0268] In some embodiments, computer-readable storage devices, media, and memory may include cable or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0269] Devices implementing the programs and methods according to these disclosures may include hardware, software, firmware, intermediary software, microcode, hardware description languages, or any combination thereof, and may employ any form factor from a wide variety of form factors. When implemented in software, firmware, intermediary software, or microcode, program code or code fragments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mount devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By further example, such functionality may also be implemented on circuit boards between different chips or different programs executing in a single device.

[0270] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example units for providing the functionality described in the content of this document.

[0271] In the foregoing description, various aspects of this invention have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this invention is not limited thereto. Therefore, although illustrative embodiments of this invention have been described in detail herein, it should be understood that the inventive concept may be embodied and employed differently in other ways, and the appended claims are intended to be construed as including such variations, in addition to those limited by the prior art. Various features and aspects of the above applications may be used individually or collectively. Furthermore, embodiments may be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. For illustrative purposes, the methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in a different order than that described.

[0272] It will be understood by those skilled in the art that, without departing from the scope of this specification, the less than (“<”) and greater than (“>”) symbols or terms used herein may be replaced by the less than or equal to (“≦”) and greater than or equal to (“≧”) symbols, respectively.

[0273] When an element is described as being "configured" to perform certain operations, such configuration may be achieved, for example, by designing a circuit or other hardware to perform the operation, programming a programmable circuit (e.g., a microprocessor or other suitable circuit) to perform the operation, or any combination thereof.

[0274] The phrase “coupled to” means any element that is directly or indirectly physically connected to another element, and / or any element that communicates directly or indirectly with another element (e.g., connected to another element by wired or wireless connection and / or other suitable communication interface).

[0275] A request language or other language that states "at least one" and / or "one or more" in a set indicates that one or more members of the set satisfy the request. For example, a request language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, a request language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. "At least one" in a language set and / or "one or more" in a set does not limit the set to items listed in that set. For example, a request language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0276] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, the various illustrative elements, blocks, modules, circuits, and steps have been described above in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this document.

[0277] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any device across a variety of devices, such as general-purpose computers, wireless communication devices, mobile phones, or integrated circuit devices with multiple uses (including applications in wireless communication devices, mobile phones, and other devices). Any feature described as a module or element can be implemented together in an integrated logic device, or separately as individual but interoperable logic devices. If implemented in software, the technology can be implemented at least in part by a computer-readable data storage medium comprising program code that includes instructions, when executed, to perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging materials. Computer-readable media can include memory or data storage media, such as random access memory (RAM) (e.g., synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electronically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Alternatively or additionally, this technology can be implemented at least in part by computer-readable communication media (such as propagated signals or waves) that carry or transmit program code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer.

[0278] The code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or individual logic circuits. Such processors can be configured to perform any of the techniques described herein. A general-purpose processor may be a microprocessor, but alternatively, a processor may be any general processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, a combination of one or more microprocessors with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0279] The explanatory form of the content of this case is as follows:

[0280] State 1: An apparatus for encoding video data, the apparatus comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: obtain a sequence of video data; identify a position in the video data sequence associated with an entry point of an individually entropy-transcoded packet for parallel entropy transcoding of the video data sequence; generate the parallel entropy-transcoded video data sequence; and generate an index for the parallel entropy-transcoded video data sequence, the index identifying the individually entropy-transcoded packet within the parallel entropy-transcoded video data sequence.

[0281] State 2: The apparatus according to request 1, wherein the entry point is identified at least in part by performing a bipartite enumeration of the video data sequence.

[0282] State 3: The apparatus according to claim 2, wherein the parallelizable entropy transcoding video data sequence is generated at least in part by performing a bipartite search, which encodes information associated with the individually entropy transcoding packet with position-determined data from the video data sequence to generate the individually entropy transcoding packet.

[0283] State 4: The apparatus according to claim 3, wherein, in order to perform the bisection enumeration, the one or more processors are configured to: bisection the video data sequence to identify an initial position in the location, wherein the initial position is associated with a first entry point and a subsequence divided by the initial position; and bisection the subsequence hierarchy to identify the remaining portion of the location.

[0284] State 5: The apparatus according to any of claim 3 or 4, wherein the individually entropy-transcoded packet includes corresponding data determined by the position from the video data sequence, and information about the individually entropy-transcoded packet.

[0285] State 6: The device according to request 5, wherein the information about the individually entropy transcoded packet is information about the scale of the binary representation of the corresponding data from the video data sequence.

[0286] State 7: The apparatus according to any of the claims 1 to 6, wherein the difference between the largest and smallest packets in the individually entropy-transcoded packets is less than a threshold difference.

[0287] State 8: The apparatus according to any of claims 1 to 7, wherein, in order to generate the parallelizable entropy transcoding sequence, the one or more processors are configured to: identify a plurality of packets to define a set of individually entropy transcoding packets from a first packet having a size between a minimum packet size and a maximum packet size to a last packet, wherein the sum of the packet sizes of the plurality of packets is the size of the parallelizable entropy transcoding sequence.

[0288] State 9: The apparatus according to claim 8, wherein, in order to generate the parallelizable entropy-encoded video data sequence, the one or more processors are configured to: initialize a stack in the memory comprising a plurality of packets and a scale for a binary representation of the corresponding data for the plurality of packets; and generate the parallelizable entropy-encoded video data sequence at least in part by performing the following operations until the stack is empty: analyze the stack to retrieve the current interval set; select the packet by defining a new index for the packet as the average of the current interval set; encode the bits of the packet using the average of the current interval set using a bisection; update the stack using a set from the first packet to the packet when the packet is not the first packet; and update the stack using a set from the packet to the last packet when the packet used is not the last packet.

[0289] State 10: The apparatus according to request item 9, wherein the stack is still initialized using a common encoding character set.

[0290] State 11: The apparatus according to any one of claims 9 or 10, wherein, in order to encode the bit of the packet using bisection, the one or more processors are configured to: initialize a first variable to the minimum packet size, initialize a second variable to the maximum packet size, and initialize a third variable to the average of the minimum packet size and the maximum packet size; access a value of the scale of the binary representation of the packet; encode the first bit when the third variable is less than or equal to the value; encode the second bit when the third variable is greater than the value; and update the third variable to a new average of the first variable and the second variable.

[0291] State 12: The apparatus according to any of claims 1 to 11, wherein the one or more processors are further configured to: define a packet in the individually entropy-transcoded packet as a forward packet or a reverse packet; wherein the corresponding packet pair includes a corresponding forward packet and a corresponding reverse packet defined using a single shared entry point in the index.

[0292] State 13: The apparatus according to request 12, wherein when the number of individually entropy-transcoding packets is even, the index includes N / 2 entry points, and when the number of individually entropy-transcoding packets is odd, the index includes [(N-1) / 2]+1 entry points.

[0293] State 14: The apparatus according to any one of claims 12 or 13, wherein the one or more processors are further configured to: store the individually entropy-transcoded packet in the memory, the individually entropy-transcoded packet being constructed as a file having a file structure including the index and multiple parts, each part including multiple blocks, each block including multiple segments, and each segment including a forward packet structure and a reverse packet structure for the corresponding forward packet and the corresponding reverse packet of the corresponding packet pair.

[0294] State 15: The apparatus according to request 14, wherein the one or more processors are further configured to: send the file including the index and the individually entropy transcoding packet.

[0295] Style 16: The apparatus according to any of claims 1 to 15, wherein the index is used to identify the individually entropy-transcoded packet by an entry point identified after encoding the video data sequence in the encoded bitstream.

[0296] State 17: The apparatus according to any of claims 1 to 16, wherein the index identifies the individually entropy-transcoded packet, including identifying the bit length and position of each packet of the encoded video bitstream generated from the video data sequence.

[0297] Sample 18: The apparatus according to any one of claims 1 to 17 further includes: a transmitter coupled to the one or more processors and configured to transmit the video data sequence entropy encoded using the index and the individually entropy transcoding packet.

[0298] Format 19: The apparatus according to any one of claims 1 to 18 further includes: a display coupled to the one or more processors and configured to display an image of the video data sequence.

[0299] Sample 20: A method for encoding video data, comprising: obtaining a video data sequence; identifying a position in the video data sequence associated with an entry point of an individually entropy-transcoding packet for parallel entropy transcoding of the video data sequence; generating the parallel entropy-transcoding video data sequence; and generating an index for the parallel entropy-transcoding video data sequence, the index identifying the individually entropy-transcoding packet within the parallel entropy-transcoding video data sequence.

[0300] State 21: The apparatus according to request 20, wherein the entry point is identified at least in part by performing a bipartite enumeration of the video data sequence.

[0301] State 22: The apparatus according to claim 21, wherein the parallel entropy transcoding video data sequence is generated at least in part by performing a bipartite search, which encodes information associated with the individually entropy transcoding packet with position-determined data from the video data sequence to generate the individually entropy transcoding packet.

[0302] State 23: The apparatus according to claim 22, wherein, in order to perform the bisection enumeration, the one or more processors are configured to: bisection the video data sequence to identify an initial position in the location, wherein the initial position is associated with a first entry point and a subsequence divided by the initial position; and bisection the subsequence hierarchy to identify the remaining portion of the location.

[0303] State 24: The apparatus according to any of claims 22 or 23, wherein the individually entropy-transcoded packet includes corresponding data determined by the position from the video data sequence, and information about the individually entropy-transcoded packet.

[0304] State 25: The device according to request 24, wherein the information about the individually entropy transcoded packet is information about the scale of the binary representation of the corresponding data from the video data sequence.

[0305] State 26: The apparatus according to any of claims 20 to 25, wherein the difference between the largest and smallest packets in the individually entropy-transcoded packets is less than a threshold difference.

[0306] State 27: The apparatus according to any of claims 20 to 26, wherein, in order to generate the parallelizable entropy transcoding sequence, the one or more processors are configured to: identify a plurality of packets to define a set of individually entropy transcoding packets from a first packet having a size between a minimum packet size and a maximum packet size to a last packet, wherein the sum of the packet sizes of the plurality of packets is the size of the parallelizable entropy transcoding sequence.

[0307] State 28: The apparatus according to claim 27, wherein, in order to generate the parallelizable entropy-encoded video data sequence, the one or more processors are configured to: initialize a stack in the memory comprising a plurality of packets and a scale for a binary representation of the corresponding data for the plurality of packets; and generate the parallelizable entropy-encoded video data sequence at least in part by performing the following operations until the stack is empty: analyze the stack to retrieve the current interval set; select the packet by defining a new index for the packet as the average of the current interval set; encode the bits of the packet using the average of the current interval set using a bisection; update the stack using a set from the first packet to the packet when the packet is not the first packet; and update the stack using a set from the packet to the last packet when the packet used is not the last packet.

[0308] State 29: The apparatus according to request 28, wherein the stack is still initialized using a common encoding set.

[0309] State 30: The apparatus according to any of claims 28 or 29, wherein, in order to encode the bit of the packet using bisection, the one or more processors are configured to: initialize a first variable to the minimum packet size, initialize a second variable to the maximum packet size, and initialize a third variable to the average of the minimum packet size and the maximum packet size; access a value of the scale of the binary representation of the packet; encode the first bit when the third variable is less than or equal to the value; encode the second bit when the third variable is greater than the value; and update the third variable to a new average of the first variable and the second variable.

[0310] State 31: The apparatus according to any of claims 20 to 30, wherein the one or more processors are further configured to: define a packet in the individually entropy-transcoded packet as a forward packet or a reverse packet; wherein the corresponding packet pair includes a corresponding forward packet and a corresponding reverse packet defined using a single shared entry point in the index.

[0311] State 32: The apparatus according to request 31, wherein when the number of individually entropy-transcoding packets is even, the index includes N / 2 entry points, and when the number of individually entropy-transcoding packets is odd, the index includes [(N-1) / 2]+1 entry points.

[0312] State 33: The apparatus according to any one of claims 31 or 32, wherein the one or more processors are further configured to: store the individually entropy-transcoded packet in the memory, the individually entropy-transcoded packet being constructed as a file having a file structure including the index and multiple parts, each part including multiple blocks, each block including multiple segments, and each segment including a forward packet structure and a reverse packet structure for the corresponding forward packet and the corresponding reverse packet of the corresponding packet pair.

[0313] State 34: The apparatus according to request 33, wherein the one or more processors are further configured to: send the file including the index and the individually entropy transcoding packet.

[0314] Style 35: The apparatus according to any of claims 20 to 34, wherein the index is used to identify the individually entropy-transcoded packet by an entry point identified after encoding the video data sequence in the encoded bitstream.

[0315] Style 36: The apparatus according to any of claims 20 to 35, wherein the index identifies the individually entropy-transcoded packet, including identifying the bit length and position of each packet of the encoded video bitstream generated from the video data sequence.

[0316] Format 37: The apparatus according to any one of claims 20 to 36 further includes: a transmitter coupled to the one or more processors and configured to transmit the video data sequence entropy encoded using the index and the individually entropy transcoding packet.

[0317] 38: The apparatus according to any one of claims 20 to 37 further includes: a display coupled to the one or more processors and configured to display an image of the video data sequence.

[0318] State 39: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the operation according to any of the states 1 to 38.

[0319] Speech 40: An apparatus for encoding video data, comprising a component for performing the operation according to any one of Speech 1 to 38.

[0320] State Sample 41. An apparatus for decoding video data, the apparatus comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: obtain an encoded video data stream including an index and a sequence of video data that can be parallelized entropy-transcoded, the sequence of video data that can be parallelized entropy-transcoded including individually entropy-transcoded packets that can be identified using the index; decode the index for data associated with the individually entropy-transcoded packets; use the data from the index to identify the individually entropy-transcoded packets from the encoded video data stream; and use the data decoded according to the index to decode the individually entropy-transcoded packets in parallel from the encoded video data stream.

[0321] State 42. A computer-implemented method comprising: obtaining an encoded video data stream including an index and a sequence of video data that can be parallelized entropy-transcoded, the sequence of video data including individually entropy-transcoded packets that can be identified using the index; decoding the index for data associated with the individually entropy-transcoded packets; using the data from the index to identify the individually entropy-transcoded packets from the encoded video data stream; and using the data decoded according to the index to decode the individually entropy-transcoded packets in parallel from the encoded video data stream.

[0322] State 43: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform the operation according to any of states 1 to 38, 41 and 42.

[0323] Session 44: An apparatus for encoding video data, comprising a component for performing the operation according to any one of modes 1 to 38, 41 and 42. [Simplified Explanation of the Diagram]

[0031] The illustrative embodiments of this case will be described in detail below with reference to the accompanying drawings:

[0032] Figure 1 is a block diagram illustrating examples of encoding and decoding devices according to some examples;

[0033] Figure 2 is a diagram illustrating an example implementation of a filter unit used to perform the technology described in this case;

[0034] Figure 3 is a diagram illustrating various forms of parallel transcoding for independently entropy transcoding parcels according to the examples described herein;

[0035] Figure 4 is a diagram showing the various types of indexes of the description entry point according to the example described herein, which will separate independently entropy transcoding packets for parallel entropy transcoding;

[0036] Figure 5A illustrates the tier of a tree structure used in conjunction with the examples described herein to describe bisection operations;

[0037] Figure 5B illustrates various forms of prefix codes that can be used with the examples described herein;

[0038] Figure 5C illustrates various forms of prefix codes that can be used with the examples described herein;

[0039] Figure 5D illustrates the various forms of prefix codes that can be used with the examples described herein;

[0040] Figure 5E illustrates various forms of prefix codes that can be used with the examples described herein;

[0041] Figure 5F illustrates various forms of prefix codes that can be used with the examples described herein;

[0042] Figure 6 is a diagram illustrating various forms of parallel transcoding for independently entropy transcoding packets according to the example described herein;

[0043] Figure 7 illustrates a file structure for use in parallel transcoding of independently entropy-transcoded packets of video data sequences, according to the example described herein;

[0044] Figure 8A illustrates a method for encoding video data according to the example described herein;

[0045] Figure 8B illustrates a method for encoding video data according to the example described herein;

[0046] Figure 9 illustrates an example system, according to the examples described herein, that can be used in conjunction with a neural network system to encode video data;

[0047] Figure 10A illustrates an example of a fully connected neural network according to the example described herein;

[0048] Figure 10B illustrates an example of a locally connected neural network according to the example described herein;

[0049] Figure 10C illustrates an example of a spiral neural network according to the example described herein;

[0050] Figure 10D illustrates a detailed example of a Deep Rotary Network (DCN) designed to identify visual features from an image, according to the example described herein;

[0051] Figure 11 is a block diagram illustrating a deep spiral network (DCN) according to the example described herein;

[0052] Figure 12 is a diagram illustrating an example of a system according to some examples, the system including a device operable to perform image and / or video transcoding (encoding and decoding) using a neural network-based system;

[0053] Figure 13 illustrates an example computing device architecture that can implement the various technologies described herein.

Claims

1. An apparatus for encoding video data, the apparatus comprising: Memory; and one or more processors coupled to the memory, the processors being configured to: obtain a video data sequence; identify locations in the video data sequence associated with entry points of individually entropy-encoded packets for a parallelizable entropy-encoded video data sequence; initialize a stack in the memory, the stack including multiple packets and a scale of a binary representation of the corresponding data for the multiple packets; for each of the multiple packets, sequentially select a packet from the stack by defining a corresponding index for each packet, encode each of the multiple packets into a corresponding entropy-encoded packet, and update the stack based on each corresponding entropy-encoded packet; generate the parallelizable entropy-encoded video data sequence using each corresponding entropy-encoded packet; and generate an index for the parallelizable entropy-encoded video data sequence, the index using each corresponding index value of each of the multiple packets to identify the individually entropy-encoded packet within the parallelizable entropy-encoded video data sequence.

2. The apparatus according to claim 1, wherein the entry points are identified at least in part by performing a one-to-one enumeration of the video data sequence.

3. The apparatus according to claim 2, wherein the parallelizable entropy-transcoded video data sequence is generated at least in part by performing a pairwise search, the pairwise search encoding information associated with the individually entropy-transcoded packets with position-determined data from the video data sequence to generate the individually entropy-transcoded packets.

4. The apparatus according to claim 3, wherein, in order to perform the bisection enumeration, the one or more processors are configured to: bisection the video data sequence to identify an initial position in the location, wherein the initial position is associated with a first entry point and a subsequence divided by the initial position; and bisection the subsequence hierarchy to identify a remaining portion of the location.

5. The apparatus according to claim 3, wherein the individually entropy-transcoded packet includes positional corresponding data from the video data sequence and information about the individually entropy-transcoded packet.

6. The apparatus according to claim 5, wherein the information regarding the individually entropy-transcoded packets is information regarding a scale of the binary representation of the corresponding data from the video data sequence.

7. The apparatus according to claim 3, wherein the difference between a maximum packet and a minimum packet in the individually entropy-transcoded packets is less than a threshold difference.

8. The apparatus according to claim 3, wherein, in order to generate the parallelizable entropy transcoding sequence, the one or more processors are configured to: identify a plurality of packets to define a set of individually entropy transcoding packets from a first packet having a size between a minimum packet size and a maximum packet size to a last packet, wherein the sum of the packet sizes of the plurality of packets is a size of the parallelizable entropy transcoding sequence.

9. The apparatus according to claim 8, wherein the one or more processors are configured to: select a packet by defining a new index for one of the packets as an average of the current interval set; encode the bits of the packet using the average of the current interval set using a bisection; update the stack using a set from the first packet to the packet when the packet is not the first packet; and update the stack using a set from the packet to the last packet when the packet used is not the last packet.

10. The apparatus according to claim 9, wherein the stack is initialized using a general-purpose encoding set.

11. The apparatus according to claim 9, wherein, in order to encode the equal bits of the packet using bisection, the one or more processors are configured to: initialize a first variable to the minimum packet size, initialize a second variable to the maximum packet size, and initialize a third variable to an average of the minimum packet size and the maximum packet size; access a value of a scale in a binary representation of the packet; encode a first bit when the third variable is less than or equal to the value; encode a second bit when the third variable is greater than the value; and update the third variable to a new average of the first variable and the second variable.

12. The apparatus according to claim 3, wherein the one or more processors are further configured to: define packets in the individually entropy-transcoding packets as forward packets or reverse packets; wherein a corresponding packet pair includes a corresponding forward packet and a corresponding reverse packet defined using a single shared entry point in the index.

13. The apparatus according to claim 12, wherein when the number of individually entropy-transcoding packets is even, the index includes N / 2 entry points, and when the number of individually entropy-transcoding packets is odd, the index includes [(N-1) / 2]+1 entry points.

14. The apparatus according to claim 12, wherein the one or more processors are further configured to: store the individually entropy-transcoding packets in the memory, the individually entropy-transcoding packets being constructed as a file having a file structure including the index and a plurality of parts, each part including a plurality of blocks, each block including a plurality of segments, and each segment including a forward packet structure and a reverse packet structure for the corresponding forward packet and the corresponding reverse packet of the corresponding packet pair.

15. The apparatus according to request 14, wherein the one or more processors are further configured to: send the file including the index and the individually entropy-transcoded packet.

16. The apparatus according to claim 3, wherein the index identifies the individually entropy-transcoded packet using an entry point as identified after encoding the video data sequence in an encoded bitstream.

17. The apparatus according to claim 3, wherein the index identifies the individually entropy-transcoded packet, including identifying a bit length and a position of each packet of an encoded video bitstream generated from the video data sequence.

18. The apparatus according to claim 3 further includes: A transmitter coupled to one or more processors and configured to transmit the video data sequence entropy encoded using the index and the individually entropy-encoding packets.

19. The apparatus according to claim 3 further includes: A display coupled to the one or more processors and configured to display an image of the video data sequence.

20. A computer-implemented method, comprising: Obtain a video data sequence; Identify the location in the video data sequence associated with the entry point of an individually entropy-encoded packet for a parallelizable entropy-encoded video data sequence; initialize a stack in the memory, the stack including multiple packets and a scale of a binary representation of the corresponding data for the multiple packets; for each of the multiple packets, sequentially select a packet from the stack by defining a corresponding index for each packet, encode each of the multiple packets into a corresponding entropy-encoded packet, and update the stack based on each corresponding entropy-encoded packet; use each corresponding entropy-encoded packet to generate the parallelizable entropy-encoded video data sequence; and generate an index for the parallelizable entropy-encoded video data sequence, the index using each corresponding index value of each of the multiple packets to identify the individually entropy-encoded packet within the parallelizable entropy-encoded video data sequence.

21. The computer-implemented method of claim 20, wherein the entry points are identified at least in part by performing a pairwise enumeration of the video data sequence; and wherein the parallelizable entropy-transcoded video data sequence is generated at least in part by performing a pairwise search, which encodes information associated with the individually entropy-transcoded packet with position-determined data from the video data sequence to generate the individually entropy-transcoded packet.

22. The computer-implemented method according to request 21, wherein performing the partition enumeration includes: The video data sequence is divided to identify an initial position among the positions, wherein the initial position is associated with a first entry point and a subsequence divided by the initial position; and the subsequence is hierarchically divided to identify a remaining portion of the positions.

23. The computer-implemented method according to claim 21, wherein the individually entropy-transcoded packets include position-determined corresponding data from the video data sequence and information about the individually entropy-transcoded packets.

24. The computer-implemented method according to claim 21, wherein the information about the individually entropy-transcoded packet is information about a scale of the binary representation of the corresponding data from the video data sequence.

25. The computer-implemented method according to claim 21, wherein generating the parallelizable entropy-encoded video data sequence includes: Identify multiple packets to define a set of individually entropy-transcoded packets from a first packet to a last packet having a size between a minimum packet size and a maximum packet size, wherein the sum of the packet sizes of the multiple packets is a size of the parallelizable entropy-transcoded sequence.

26. The computer-implemented method according to request item 25 further includes: The package is selected by defining a new index for one of the multiple packages as the average of the current interval set; The packet's bits are encoded using the average of the current interval set; when the packet is not the first packet, the stack is updated using the set from the first packet to the packet; and when the packet used is not the last packet, the stack is updated using the set from the last packet to the packet.

27. The computer-implemented method according to request 26, wherein using bisection to encode the bits of the packet includes: Initialize a first variable to the minimum packet size, initialize a second variable to the maximum packet size, and initialize a third variable to an average of the minimum and maximum packet sizes; access a value of a scale in the binary representation of the packet; when the third variable is less than or equal to the value, encode a first bit; when the third variable is greater than the value, encode a second bit; and update the third variable to a new average of the first and second variables.

28. The computer-implemented method according to request item 27 further includes: Send a file that includes the index and the individually entropy-transcoded packets.

Images