Methods, apparatus and storage media for encoding and decoding transform blocks
By determining the transform type by decoding a single truncated unary binary content from the bitstream, the transform selection and signaling coding of the video coding block are optimized, solving the problem of low efficiency in the existing technology, achieving more efficient video compression and reducing implementation costs.
Patent Information
- Application Number
- CN202311312173.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-06-29
- Filing Date
- 2019-05-29
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2039-05-29
AI Technical Summary
Existing video coding standards suffer from inefficiencies in block transform selection and signaling coding, especially in high-resolution and high-frame-rate video compression, resulting in poor coding efficiency.
By decoding individual truncated unary binarized contents from the bitstream, the types of horizontal and vertical transforms are determined, and these transform types are applied to decode transform blocks in the image frame, optimizing block transform selection and signaling coding.
It improves the efficiency of video encoding, reduces implementation costs and processor load, and meets the compression requirements of high-resolution and high-frame-rate videos.
Smart Images

Figure CN117544791B_ABST
Abstract
Description
[0001] (This application is a divisional application of the application filed on May 29, 2019, with application number 2019800417241, entitled "Method, apparatus and system for transform block encoding and decoding of video samples".) Technical Field
[0002] This invention generally relates to digital video signal processing, and more particularly to methods, apparatus, and systems for encoding and decoding transform blocks of video samples. The invention also relates to a computer program product comprising a computer-readable medium containing computer programs for encoding and decoding transform blocks of video samples. Background Technology
[0003] Currently, there are numerous video coding applications, including those for transmitting and storing video data. Many video coding standards have been developed, and others are currently under development. Recent advancements in video coding standardization have led to the formation of a group known as the Joint Video Experts Group (JVET). The JVET includes members of Study Group 16, Problem 6 (SG16 / Q6) of the Telecommunication Standardization Sector (ITU-T) of the International Telecommunication Union (ITU), known as the Video Coding Experts Group (VCEG); and members of the Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), also known as the Moving Picture Experts Group (MPEG).
[0004] The Joint Video Experts Group (JVET) issued a Call for Proposals (CfP) and analyzed the responses at its 10th meeting in San Diego, USA. The submitted responses demonstrated video compression capabilities significantly superior to those of the current state-of-the-art video compression standard, namely, "High Efficiency Video Coding" (HEVC). Based on this superior performance, a project was initiated to develop a new video compression standard named "Various Video Coding" (VVC). VVC is expected to address the ongoing demand for even higher compression performance, particularly as video format capabilities increase (e.g., higher resolution and higher frame rates), and the growing market demand for services delivered over WANs (where bandwidth costs are relatively high). Simultaneously, VVC must be implementable in modern silicon processes and offer an acceptable trade-off between achieved performance and implementation costs (e.g., in terms of silicon area, CPU processor load, memory utilization, and bandwidth).
[0005] Video data consists of a sequence of frames, each containing one or more color channels. Typically, there is one primary color channel and two secondary color channels. The primary color channel is often referred to as the "luminance" channel, and the secondary color channels(s) are often referred to as the "chrominance" channels. Although video data is typically displayed in the RGB (Red-Green-Blue) color space, this color space is highly correlated between its three corresponding components. The video data representation seen by the encoder or decoder typically uses a color space such as YCbCr. YCbCr concentrates luminance in the Y (primary) channel and chrominance in the Cb and Cr (secondary) channels. Furthermore, the Cb and Cr channels can be sampled at a lower rate than the luminance channel (e.g., half horizontally and half vertically (referred to as the "4:2:0 chrominance format")).
[0006] VVC is a "block-based" codec where frames are divided into blocks and processed in a specific order. For each block, a prediction of the block's content is generated, forming a representation of the difference (or "residual" in the spatial domain) between this prediction and the actual block content considered as input to the encoder. This difference can be encoded as a sequence of residual coefficients, obtained by applying a forward principal transform, such as a Discrete Cosine Transform (DCT) or other transforms, to the block of residual values. This principal transform is applied separately to the block of residual samples, i.e., a two-dimensional transform is performed in two passes, where rows are first transformed by applying a one-dimensional transform to each row, and then the result is transformed by applying a one-dimensional transform to each column, to produce a block of residual coefficients that are essentially decorrelated on the residual samples. Furthermore, multiple transforms are available for this row and column processing, requiring signaling to indicate which transform to use. While the option to use different transforms in a block offers compression advantages, these advantages are slightly reduced by the need to encode indices to select which transform to use in the horizontal and vertical directions. The efficient coding of this signaling has a beneficial impact on the signaling overhead. Summary of the Invention
[0007] The object of this invention is to generally overcome or at least improve one or more disadvantages of existing configurations.
[0008] One aspect of the present invention provides a method for decoding a transform block in an image frame from a bitstream, the method comprising: decoding a single truncated unary binary content from the bitstream, the single truncated unary binary content being used for a horizontal and vertical transform of a transform block in the image frame; determining the types of the horizontal and vertical transforms based on the decoded single truncated unary binary content; and decoding the transform block in the image frame by applying the determined types of the horizontal and vertical transforms to the transform block of the image.
[0009] Another aspect of the present invention provides a non-transitory computer-readable medium including a computer application program executable by a computer device to enable a computer to perform a method for decoding transform blocks in an image frame from a bitstream, the method comprising: decoding a single truncated unary binary content from the bitstream, the single truncated unary binary content being used for horizontal and vertical transforms of transform blocks in the image frame; determining the types of the horizontal and vertical transforms based on the decoded single truncated unary binary content; and decoding the transform blocks in the image frame by applying the determined types of the horizontal and vertical transforms to the transform blocks of the image.
[0010] Other aspects were also disclosed. Attached Figure Description
[0011] At least one embodiment of the invention will now be described with reference to the following figures and appendices, wherein:
[0012] Figure 1 This is a schematic block diagram illustrating a video encoding and decoding system;
[0013] Figure 2A and 2B Composition can be put into practice Figure 1 A schematic block diagram of a general computer system that includes one or both of the video encoding and decoding systems;
[0014] Figure 3 This is a schematic block diagram showing the functional modules of a video encoder;
[0015] Figure 4 This is a schematic block diagram illustrating the functional modules of the video decoder;
[0016] Figure 5 This is a schematic block diagram illustrating the available segmentation of a block into one or more blocks in the tree structure of general video coding;
[0017] Figure 6 This is a schematic diagram illustrating the permissive segmentation of a data stream from one block to one or more blocks within the tree structure of general video coding;
[0018] Figure 7 shows an example of splitting a coding tree unit (CTU) into multiple coding units (CU);
[0019] Figure 8A This is a diagram illustrating the intra-frame prediction mode;
[0020] Figure 8B It is a table that shows the mapping from a set of transformations to a specific transformation based on the transformation index;
[0021] Figure 8CIt is a table showing the mapping from intra-prediction mode to transform set for transform block in the horizontal and vertical directions;
[0022] Figure 9 It is shown Figure 3 Video encoder or Figure 4 A schematic block diagram of the inverse transform module of the video decoder;
[0023] Figure 10A This is a diagram showing the relationship between truncated unary syntax elements and the transformation indices used for transformations indexed in the horizontal and vertical directions;
[0024] Figure 10B It is a diagram showing the substitution relationship between truncated unary syntax elements and the transformation indices used for transformations indexed in the horizontal and vertical directions;
[0025] Figure 11 This is a flowchart of a method for encoding the transform blocks of residual coefficients into a bitstream by selecting the primary transform based on explicit transform selection; and
[0026] Figure 12 This is a flowchart of a method for selecting transform blocks from a bit stream to decode residual coefficients via a primary inverse transform based on explicit transform selection. Detailed Implementation
[0027] Where steps and / or features with the same reference numerals are referenced in any one or more of the accompanying drawings, unless the contrary is intended, these steps and / or features have the same function(s) or operation(s) for the purposes of this specification.
[0028] Figure 1 This is a schematic block diagram illustrating the functional modules of a video encoding and decoding system 100. System 100 may utilize coefficient scanning methods to improve compression efficiency and / or achieve reduced implementation costs. System 100 includes a source device 110 and a destination device 130. A communication channel 120 is used to communicate encoded video information from the source device 110 to the destination device 130. In some configurations, one or both of the source device 110 and the destination device 130 may each include a mobile phone or "smartphone," in which case the communication channel 120 is a wireless channel. In other configurations, the source device 110 and the destination device 130 may include video conferencing equipment, in which case the communication channel 120 is typically a wired channel such as an Internet connection. Furthermore, the source device 110 and the destination device 130 may include a wide range of arbitrary devices, including those supporting over-the-air television broadcasting, cable television applications, Internet video applications (including streaming), and applications that capture encoded video data on some computer-readable storage medium (such as hard disk drives in a file server).
[0029] like Figure 1 As shown, source device 110 includes a video source 112, a video encoder 114, and a transmitter 116. Video source 112 typically includes a source of captured video frame data (denoted as 113), such as a camera sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote camera sensor. Video source 112 can also be the output of a computer graphics card (e.g., video output from a display operating system and various applications running on a computing device (e.g., a tablet computer)). Examples of source devices 110 that may include a camera sensor as video source 112 include smartphones, video camcorders, professional cameras, and webcams. See further reference... Figure 3 The video encoder 114 converts (or "encodes") the captured frame data (indicated by arrow 113) from the video source 112 into a bitstream (indicated by arrow 115). The bitstream 115 is transmitted by the transmitter 116 via the communication channel 120 as encoded video data (or "encoded video information"). The bitstream 115 may also be stored in a non-transitory storage device 122, such as a flash memory or a hard disk drive, until subsequently transmitted via the communication channel 120 or as a replacement for transmission via the communication channel 120.
[0030] Destination device 130 includes a receiver 132, a video decoder 134, and a display device 136. Receiver 132 receives encoded video data from communication channel 120 and transmits the received video data as a bitstream (indicated by arrow 133) to video decoder 134. Video decoder 134 then outputs decoded frame data (indicated by arrow 135) to display device 136. Examples of display device 136 include cathode ray tubes, liquid crystal displays (such as in smartphones, tablets, computer monitors, or standalone televisions). Alternatively, the functions of source device 110 and destination device 130 can be embodied in a single device, examples of which include mobile phones and tablets.
[0031] Although the example apparatus has been described above, the source apparatus 110 and the destination apparatus 130 can each typically be configured within a general-purpose computer system via a combination of hardware and software components. Figure 2AThis computer system 200 is illustrated and includes: a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 configured as a video source 112, and a microphone 280; and output devices including a printer 215, a display device 214 configured as a display device 136, and a speaker 217. The computer module 201 can communicate with a communication network 220 via a connection 221 using an external modulator-demodulator (modem) transceiver device 216. The communication network 220, which may represent the communication channel 120, can be a wide area network (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. If the connection 221 is a telephone line, the modem 216 can be a conventional dial-up modem. Alternatively, if the connection 221 is a high-capacity (e.g., cable or optical) connection, the modem 216 can be a broadband modem. A wireless modem can also be used for wireless connection to the communication network 220. The transceiver device 216 can provide the functions of a transmitter 116 and a receiver 132, and the communication channel 120 can be embodied in the wiring 221.
[0032] Computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, memory unit 206 may have semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). Computer module 201 also includes multiple input / output (I / O) interfaces, including: an audio-video interface 207 connected to a video display 214, speakers 217, and microphone 280; an I / O interface 213 connected to a keyboard 202, mouse 203, scanner 226, camera 227, and optionally a joystick or other human-machine interface device (not shown); and an interface 208 for external modem 216 and printer 215. Signals from audio-video interface 207 to computer monitor 214 are typically output from the computer's graphics card. In some implementations, modem 216 may be integrated within computer module 201, for example, within interface 208. Computer module 201 also has a local network interface 211, which allows computer system 200 to connect via cable 223 to a local area network (LAN) known as a local area network (LAN). Figure 2A As shown, the local area communication network 222 can also be connected to the wide area network 220 via cable 224, wherein the local area communication network 222 typically includes a so-called "firewall" device or a device with similar functionality. The local network interface 211 may include Ethernet. TM Circuit card, Bluetooth TMThe interface can be configured wirelessly or via IEEE 802.11; however, various other types of interfaces can be implemented for interface 211. The local network interface 211 can also provide the functionality of transmitter 116 and receiver 132, and the communication channel 120 can also be incorporated into the local communication network 222.
[0033] I / O interfaces 208 and 213 can provide either serial or parallel connections, or both, with the former typically implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 209 is provided, and storage device 209 typically includes a hard disk drive (HDD) 210. Other storage devices (not shown), such as floppy disk drives and magnetic tape drives, may also be used. An optical disc drive 212 is typically provided as a non-volatile source of data. Optical discs, such as CD-ROMs, DVDs, and Blu-ray discs, may be used. TM Portable storage devices such as USB-RAM, portable external hard disk drives, and floppy disks serve as suitable sources of data for computer system 200. Typically, any of HDD 210, optical disc drive 212, network 220, and 222 can also be configured to operate as video source 112 or as a destination for decoded video data to be stored for reproduction via display 214. Source device 110 and destination device 130 of system 100, or system 100's source device 110 and destination device 130, can be embodied in computer system 200.
[0034] Components 205-213 of computer module 201 typically communicate via interconnect bus 204 and in a manner known to those skilled in the art to operate computer systems 200 in a conventional mode of operation. For example, processor 205 is connected to system bus 204 via cable 218. Similarly, memory 206 and optical disc drive 212 are connected to system bus 204 via cable 219. Examples of computers in which this configuration may be practiced include IBM-PC and compatible machines, Sun SPARCstation, and Apple Macs. TM Or a similar computer system.
[0035] Where appropriate or desired, the video encoder 114 and video decoder 134, as well as the methods described below, can be implemented using the computer system 200, wherein the video encoder 114, video decoder 134, and the methods to be described can be implemented as one or more software applications 233 executable within the computer system 200. Specifically, instructions 231 (see [link to instructions]) executed within the computer system 200 in the software 233 are utilized. Figure 2BThe software instructions 231 can be configured as one or more code modules, each for performing one or more specific tasks. Alternatively, the software can be divided into two separate parts, with a first part and corresponding code modules performing the method, and a second part and corresponding code modules managing the user interface between the first part and the user.
[0036] For example, the software can be stored in a computer-readable medium including the storage means described below. The software is loaded from the computer-readable medium into computer system 200 and then executed by computer system 200. The computer-readable medium having such software, or the computer program recorded on such computer-readable medium, is a computer program product. Using the computer program product in computer system 200 preferably implements advantageous means for carrying out the video encoder 114, video decoder 134, and the methods.
[0037] Software 233 is typically stored in HDD 210 or memory 206. The software is loaded from a computer-readable medium into computer system 200 and executed by computer system 200. Thus, for example, software 233 can be stored on an optically readable disk storage medium (e.g., CD-ROM) 225 read by optical disk drive 212.
[0038] In some instances, application 233 is supplied to the user in a manner encoded on one or more CD-ROMs 225 and read via a corresponding drive 212, or alternatively, application 233 may be read by the user from network 220 or 222. Furthermore, software may also be loaded into computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, and Blu-ray Discs. TM This includes hard disk drives, ROMs or integrated circuits, USB storage devices, magneto-optical disks, or computer-readable cards such as PCMCIA cards, regardless of whether these devices are inside or outside the computer module 201. Examples of temporary or intangible computer-readable transmission media that may also participate in providing software, applications, instructions, and / or video data or encoded video data to the computer module 401 include: radio or infrared transmission channels and network connections to other computers or networked devices, as well as the Internet or intranets including email sending and information recorded on websites.
[0039] The second part of the aforementioned application 233 and the corresponding code modules can be executed to implement one or more graphical user interfaces (GUIs) to be drawn or otherwise presented on the display 214. By typically operating the keyboard 202 and mouse 203, users and applications of the computer system 200 can operate the interface in a functionally applicable manner to provide control commands and / or input to applications associated with these GUIs(s). Other functionally applicable forms of user interfaces can also be implemented, such as audio interfaces utilizing voice prompts output via speaker 217 and user voice commands input via microphone 280.
[0040] Figure 2B This is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents... Figure 2A The computer module 201 can access the logical aggregation of all memory modules (including HDD 209 and semiconductor memory 206).
[0041] When the computer module 201 is initially powered on, the power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in... Figure 2A The ROM 249 of the semiconductor memory 206 is sometimes referred to as firmware. The POST program 250 checks the hardware within the computer module 201 to ensure proper operation, and typically checks the processor 205, the memory 234 (209, 206), and the Basic Input / Output System Software (BIOS) module 251, which is also typically stored in the ROM 249, for correct operation. Once the POST program 250 has successfully run, the BIOS 251 boots. Figure 2A The hard disk drive 210 is booted. Booting the hard disk drive 210 causes the bootloader 252 residing on the hard disk drive 210 to be executed via the processor 205. This loads the operating system 253 into the RAM memory 206, where the operating system 253 begins operation. The operating system 253 is a system-level application executable by the processor 205 to implement various advanced functions, including processor management, memory management, device management, storage management, software application interfaces, and a general user interface.
[0042] The operating system 253 manages memory 234 (209, 206) to ensure that each process or application running on computer module 201 has sufficient memory to execute without conflicting with memory allocated to other processes. Furthermore, appropriate use of memory is essential. Figure 2AThe computer system 200 contains different types of memory so that each process can run efficiently. Therefore, aggregate memory 234 is not intended to illustrate how specific segments of memory are allocated (unless otherwise stated), but rather to provide an overview of the memory accessible to the computer system 200 and how that memory is used.
[0043] like Figure 2B As shown, processor 205 includes multiple functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and local or internal memory 248, sometimes referred to as cache memory. Cache memory 248 typically includes multiple storage registers 244-246 in a register segment. One or more internal buses 241 functionally interconnect these functional modules. Processor 205 also typically has one or more interfaces 242 for communicating with external devices via system bus 204 using wiring 218. Memory 234 is connected to bus 204 using wiring 219.
[0044] Application program 233 includes a sequence of instructions 231, which may contain conditional branch instructions and loop instructions. Program 233 may also include data 232 used when executing program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative size of instructions 231 and memory locations 228-230, as described by the instructions shown in memory location 230, a particular instruction may be stored in a single memory location. Alternatively, as described by the instruction segments shown in memory locations 228 and 229, an instruction may be divided into multiple parts, each stored in a separate memory location.
[0045] Typically, a set of instructions is assigned to processor 205, which is then executed within processor 205. Processor 205 waits for subsequent input, which it responds to by executing another set of instructions. Each input can be provided from one or more of a plurality of sources, including data generated by one or more of input devices 202, 203, data received from an external source via one of networks 220, 202, data retrieved from one of storage devices 206, 209, or data retrieved from storage medium 225 inserted into the corresponding reader 212 (all of which are within...). Figure 2A (As shown in the diagram). Executing a set of instructions may result in output data in some cases. Execution may also involve storing data or variables into memory 234.
[0046] The video encoder 114, video decoder 134, and the method can use input variables 254 stored in corresponding memory locations 255, 256, and 257 within memory 234. The video encoder 114, video decoder 134, and the method generate output variables 261 stored in corresponding memory locations 262, 263, and 264 within memory 234. Intermediate variables 258 can be stored in memory locations 259, 260, 266, and 267.
[0047] refer to Figure 2B The processor 205, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 work together to perform micro-operation sequences required for "fetch, decode, and execute" cycles of instructions in the instruction set constituting program 233. Each fetch, decode, and execute cycle includes:
[0048] (a) Fetch operation, used to fetch or read instruction 231 from memory locations 228, 229, 230;
[0049] (b) Decoding operation, wherein the control unit 239 determines which instruction was extracted; and
[0050] (c) Execute an operation, wherein the control unit 239 and / or ALU 240 execute the instruction.
[0051] Then, a further fetch, decode, and execute cycle for the next instruction can be performed. Similarly, a storage cycle can be performed, in which the control unit 239 stores or writes a value to or writes it to memory location 232.
[0052] To explain Figure 11 and Figure 12 Each step or subprocess in the method is associated with one or more segments of program 233 and is typically performed by working together in the register units 244, 245, 247, ALU 240 and control unit 239 in processor 205 to fetch, decode and execute cycles of instructions in the instruction set of the segment of program 233.
[0053] Figure 3 This is a schematic block diagram showing the functional modules of the video encoder 114. Figure 4 This is a schematic block diagram illustrating the functional modules of the video decoder 134. Typically, data is transferred between the functional modules within the video encoder 114 and the video decoder 134 in groups of samples or coefficients (such as the partitioning of blocks into fixed-size sub-blocks) or as arrays. Figure 2A and 2BAs shown, the video encoder 114 and video decoder 134 can be implemented using a general-purpose computer system 200, wherein various functional modules can be implemented using dedicated hardware within the computer system 200 and software executable within the computer system 200 (such as one or more software code modules of a software application 233 residing on the hard disk drive 205 and executed under the control of the processor 205). Alternatively, the video encoder 114 and video decoder 134 can be implemented using a combination of dedicated hardware and software executable within the computer system 200. Alternatively, the video encoder 114, video decoder 134, and the method can be implemented in dedicated hardware such as one or more integrated circuits that perform the functions or sub-functions of the method. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), a dedicated standard product (ASSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or one or more microprocessors and associated memory. Specifically, the video encoder 114 includes modules 322-386, and the video decoder 134 includes modules 420-496, wherein each of these modules can be implemented as one or more software code modules of the software application 233.
[0054] although Figure 3 The video encoder 114 is an example of a Universal Video Coding (VVC) video coding pipeline, but other video codecs can also be used for the processing stages described herein. The video encoder 114 receives captured frame data 113, such as a series of frames (each frame including one or more color channels). The block partitioner 310 first divides the frame data 113 into regions commonly referred to as “coding tree units” (CTUs), which are typically square in shape and configured to use a specific size for the CTUs. For example, the size of a coding tree unit can be 64×64, 128×128, or 256×256 luminance samples. The block partitioner 310 further divides each CTU into one or more coding units (CUs), where CUs have various sizes that can include both square and non-square aspect ratios. Thus, the current block 312 (“coding unit”) is output from the block partitioner 310, progressing according to iterations of one or more blocks of CTUs. However, the concept of CUs is not limited to the block partitions obtained from the block partitioner 310. Video decoder 134 can also be considered as generating CUs, which are typically approximations of blocks from block partitioner 310 due to the use of lossy compression techniques. Furthermore, video encoder 114 generates CUs with the same approximations seen in video decoder 134, thereby enabling accurate understanding of the sample data available for block prediction methods in video decoder 134. The following references... Figure 5 and 6To further explain the options for partitioning a CTU as a CU.
[0055] The coded tree units (CTUs) obtained from the first segmentation of frame data 113 can be scanned in raster scan order and divided into one or more "slices". Since frame data 113 typically includes multiple color channels, CTUs and CUs are associated with samples from all color channels overlapping the block region defined by the operation of block partitioner 310. A CU can be considered to include one or more coded blocks (CBs), where each CB occupies the same block region as the CU but is associated with each color channel in the frame data 113. Because the chroma and luma channels potentially have different sampling rates, the size of the CB for the chroma channel may differ from the size of the CB for the luma channel. When using a 4:2:0 chroma format, the size of the CB for the chroma channel of a CU is half the width and half the height of the CB for the luma channel of the CU.
[0056] While iterating over all CUs obtained from block partitioner 310, video encoder 114 generates a “prediction unit” (PU) for each block (e.g., block 312), indicated by arrow 320. PU 320 is a prediction of the content of the associated CU 312. Subtractor module 322 generates a difference (or “residual,” which refers to the difference in the spatial domain) between PU 320 and CU 312, represented as 324. Difference 324 is the block size difference between the corresponding samples in PU 320 and CU 312. Difference 324 is transformed, quantized, and represented as a transform unit (TU), indicated by arrow 336. PU 320 is typically selected as the “best” among several possible candidate PUs. Candidate PUs are obtained from one of the prediction modes available to video encoder 114. Each candidate PU receives a corresponding transform unit. Transform unit 336 is the quantized and transformed representation of difference 324. When combined with the prediction PU in decoder 114, transform unit 336 reduces the difference between decoded CU and original block 312 at the cost of additional signaling in the bit stream.
[0057] Therefore, each candidate PU has an associated coding cost (rate) and an associated difference (or "distortion"). The coding rate (cost) is typically measured in bits. The coding distortion of a block is typically estimated as the difference in sample values, such as the sum of absolute differences (SAD) or the sum of squared differences (SSD). The mode selector 386 uses the difference 324 to determine the estimate obtained from each candidate PU to determine the intra-frame prediction mode (indicated by arrow 388). The coding cost associated with each candidate prediction mode and the corresponding residual coding can be estimated at a significantly lower cost compared to entropy coding of the residuals. Therefore, multiple candidate modes can be evaluated to determine the optimal mode in a rate-distortion sense.
[0058] Determining the optimal mode is typically achieved using a variation of Lagrange optimization. Selecting an intra-prediction mode 388 generally involves determining the coding cost of the residual data obtained from applying a specific intra-prediction mode. The coding cost can be approximated using a "transform difference sum," thereby obtaining an estimated transform residual cost using a relatively simple transform such as the Hadamard transform. For implementations using relatively simple transforms, the simplified estimation method can be used to make the same decision (i.e., the intra-prediction mode) as long as the cost obtained from the simplified estimation method is monotonically correlated with the actual cost otherwise determined from a full evaluation, thus reducing the complexity of the video encoder 114. To allow for the possible non-monotonicity of the relationship between the estimated cost and the actual cost, which may be derived from more mode decisions available for encoding the residual data, this simplified estimation method can be used to generate a list of optimal candidates. The list of optimal candidates can have any number. A more complete search can be performed using the optimal candidates to establish the optimal mode selection for encoding the residual data of each candidate, thereby allowing for the final selection of the intra-prediction mode and other mode decisions.
[0059] Other pattern decisions include the ability to skip major and minor transformations (known as "transform skipping"). Transform skipping is suitable for residual data that lacks sufficient relevance to reduce encoding costs via expressions that serve as the fundamental functions of the transformations. Certain types of content, such as relatively simple computer-generated graphics, can exhibit similar behavior.
[0060] Another mode decision associated with the mode selector module 386 is the selection of the explicit multi-transformation index (indicated by arrow 390). The explicit multi-transformation index 390 has four possible values. These four possible values are (in...) Figure 10A (As shown in columns 1046 and 1048) is a combination of two indices, each ranging from 0 to 1. The first index is associated with the horizontal transform index H, and the second index is associated with the vertical transform index V. The horizontal transform index H is also called the horizontal transform selection, and the vertical transform index V is also called the vertical transform selection. Refer to Figure 10 to illustrate the selection and encoding of the multiple transform index 390.
[0061] Lagrange or similar optimization processes can be used to select both the optimal partitioning of the CTU to the CU (using block partitioner 310) and the optimal prediction mode from multiple possibilities. By applying Lagrange optimization of candidate modes to the mode selector module 386, the intra-prediction mode with the lowest cost metric is selected as the optimal mode. The optimal mode is the selected intra-prediction mode 388, and it is also encoded in bitstream 115 by entropy encoder 338. The selection of intra-prediction mode 388 through the operation of mode selector module 386 is extended to the operation of block partitioner 310. For example, the candidates for selecting intra-prediction mode 388 can include modes applicable to a given block and additional modes applicable to multiple smaller blocks that are co-located with the given block. In such a case, the candidate selection process is implicitly also a process for determining the optimal hierarchical decomposition of the CTU to the CU.
[0062] The entropy encoder 338 supports both variable-length encoding and arithmetic encoding of syntactic elements. Context-adaptive binary arithmetic encoding is used to support arithmetic encoding. Arithmetic-encoded syntactic elements consist of a sequence of one or more "bins". Like bits, bins have values of "0" or "1". However, bins are not encoded as discrete bits in the bitstream 115. Bins have associated possible values and associated probabilities (called "context"). When the actual bin to be encoded matches a possible value, the "maximum probability symbol" (MPS) is encoded. Encoding the maximum probability symbol is relatively inexpensive in terms of bit consumption. When the actual bin to be encoded does not match a possible value, the "minimum probability symbol" (LPS) is encoded. Encoding the minimum probability symbol is relatively expensive in terms of bit consumption. Bin encoding techniques enable efficient encoding of bins that can skew the probabilities of "0" vs. "1". For syntactic elements with two possible values (i.e., "flag"), a single bin is sufficient. For syntactic elements with many possible values, a sequence of bins is required. The presence of a later bin in the sequence can then be determined based on the value of the earlier bin in the sequence. Furthermore, each bin can be associated with more than one context, where a specific context is selected based on factors such as the earlier bin in the syntactic element and the bin values of adjacent syntactic elements (i.e., bin values from neighboring blocks). Each time a bin is encoded, the context is updated to accommodate the new bin value. Thus, the binary arithmetic coding scheme is considered adaptive.
[0063] Encoder 114 also supports bins lacking context (“bypass bins”). Bypass bins are encoded using an equal probability distribution between “0” and “1”. The absence of context saves memory and reduces complexity, thus using bypass bins with an unskewed distribution of values for a particular bin. An example of a context-adaptive entropy encoder is known in the art as CABAC (Context-Adaptive Binary Arithmetic Encoder), and many variations of this encoder are used in video coding.
[0064] Entropy encoder 338 uses a combination of context-coded and (optionally) bypass-coded bins to encode intra-prediction mode 388. Typically, a list of "maximum probability modes" is generated in video encoder 114. This list of maximum probability modes is usually a fixed length, such as three or six modes, and may include modes encountered in earlier blocks. The context-coded bin encodes a flag indicating whether the intra-prediction mode is one of the maximum probability modes. If intra-prediction mode 388 is one of the maximum probability modes, further signaling indicating which maximum probability mode corresponds to intra-prediction mode 388 is encoded, for example, using a truncated unary bin string. Otherwise, alternative syntax, such as fixed-length codes, is used to encode intra-prediction mode 388 as "remaining modes" to represent intra-prediction modes other than those present in the list of maximum probability modes.
[0065] Referring to Figure 10, the entropy encoder 338 also encodes the explicit multiple transform index 390 for specific encoding units or transform blocks. Specifically, the explicit multiple transform index 390 includes a vertical transform index V and a horizontal transform index H. Both the vertical transform index V and the horizontal transform index H are selected by the mode selector 386 (as described above). The combination of the vertical transform index V and the horizontal transform index H (see...) Figure 10A The entropy encoder 338 will (in) Figure 10A The corresponding bin string encoding is shown in column 1042 in bitstream 115.
[0066] The multiplexer module 384 outputs a PU 320 based on the optimal intra-frame prediction mode 388 determined from the candidate prediction modes tested. The candidate prediction modes do not need to include every conceivable prediction mode supported by the video encoder 114. Prediction modes are broadly categorized into two types. The first type is "intra-frame prediction" (or "intra prediction"). In intra-frame prediction, block predictions are generated using additional samples extracted from the current frame. The second type is "inter-frame prediction" (or "interprediction"). In intra-frame prediction, block predictions are generated using samples from frames preceding the current frame in the bitstream, in an order that may differ from the order of frames captured or displayed. Within each category (i.e., intra-frame and inter-frame prediction), different techniques can be applied to generate the PU. For example, intra-frame prediction can generate the PU using the direction of values from adjacent rows and columns of previously reconstructed samples according to a prescribed filtering process. Alternatively, a small number of parameters can be used to describe the PU. Inter-frame prediction methods can vary in the number and accuracy of motion parameters. Motion parameters typically include a reference frame offset plus translations of one or two reference frames, but can include more frames, special frames, or complex affine parameters such as scaling and rotation. Additionally, predetermined motion refinement processes can be applied to generate dense motion estimates based on reference sample blocks.
[0067] Having determined and selected the optimal PU 320, and subtracted it from the original sample block at subtractor 322, the residual 324 with the lowest encoding cost is obtained, and lossy compression is performed on this residual 324. The lossy compression process includes transform, quantization, and entropy coding steps. Transform module 326 applies a first transform to the difference 324, thereby converting the difference 324 to the frequency domain and producing intermediate transform coefficients indicated by arrow 328. The first transform is typically separable, transforming a set of rows and then a set of columns for each block. The transformations for each set of rows and columns are performed by applying repeated one-dimensional transforms to each row and column of the block. The transform spans the entire block.
[0068] Specifically, intra-frame prediction mode 388 and explicit multi-transform index 390 are used to select the horizontal and vertical one-dimensional transforms that constitute the main transforms performed by transform module 326. The horizontal and vertical one-dimensional transforms are each separable transforms. The horizontal and vertical one-dimensional transforms are performed over the entire transform block, thereby substantially decorrelating the residual samples.
[0069] The transformation types available to the transformation module 326 include DST-VII, DST-VIII, DCT-VIII, DST-I, and DCT-V. (See reference...) Figure 8B Based on this set of transformations, three "transformation sets" 842 are defined. For each direction in the horizontal or vertical direction, as referenced... Figure 8C The transform set 842 is selected based on the intra-frame prediction mode 388. (See reference...) Figure 11 A transformation type is selected and applied from each transformation group 842. Intermediate transformation coefficients 328 are passed to the secondary transformation module 330.
[0070] The secondary transformation module 330 operates on a subset of the intermediate transformation coefficients 328 (such as intermediate transformation coefficients occupying the upper left 4×4 or 8×8 region of the entire block). Other transformation coefficients in the intermediate transformation coefficients 328 are passed through module 330 unchanged. The secondary transformation module 330 applies one of various transformations to the subset of intermediate transformation coefficients 328 to produce the transformation coefficients indicated by arrow 332. The secondary transformation module 330 applies a forward secondary transformation selected in a manner similar to that of the inverse secondary transformation module 344. The transformations available to the secondary transformation module 330 are generally inseparable and therefore cannot be performed in two stages (i.e., rows and columns) as in the case of transformation module 326. Due to the higher complexity of the secondary transformations and the earlier decorrelation of the primary transformations, it is sufficient to perform secondary transformations only on the upper left region of the transformation block. The transformations performed by the secondary transformation module 330 may be at least partially dependent on the chosen prediction mode. Additionally, the video encoder 114 can consider the selection of the transform applied at module 330 as a further candidate test of the selection based on rate / distortion cost evaluation.
[0071] Transform coefficients 332 are passed to quantizer module 334. At module 334, quantization is performed according to the "quantization parameters" to produce residual coefficients, indicated by arrow 336. The quantization parameters are constant for a given transform block, thus resulting in a uniform scaling of the generated residual coefficients for the transform block. Non-uniform scaling can also be achieved by applying a "quantization matrix," whereby the scaling factor applied to each residual coefficient is derived from a combination of the quantization parameters and corresponding entries in a scaling matrix whose size is typically equal to the size of the transform block. Therefore, the expensive quantization matrix to be represented (if any) is only encoded infrequently in bit stream 115. Encoding the quantization matrix requires converting a two-dimensional matrix of scaling factors into a list of scaling factors to be entropy-encoded into bit stream 115. Existing Z-order scans can be reused for this purpose, thus avoiding the overhead associated with additional scan modes used to support the infrequent encoding of the quantization matrix. Residual coefficients 336 are fed to entropy encoder 338 for encoding in bit stream 115. Typically, the residual coefficients of the transform block are scanned according to a scan mode to generate an ordered list of values. The scan mode typically scans the transform block into a sequence of 4×4 “sub-blocks,” thus providing regular scan operations at a granularity of a 4×4 set of residual coefficients, where the configuration of the sub-blocks depends on the size of the transform block. Additionally, the prediction mode and corresponding block partitions are also encoded in bitstream 115. As described above, the video encoder 114 needs access to the frame representation corresponding to the frame representation seen in the video decoder 134. Therefore, the residual coefficients 336 are also inversely quantized by the dequantizer module 340 to produce the inverse transform coefficients indicated by arrow 342. The inverse transform coefficients 342 are passed to the inverse secondary transform module 344. The inverse secondary transform module 344 applies the selected secondary transform to produce intermediate inverse transform coefficients as indicated by arrow 346. The intermediate inverse transform coefficients 346 are fed to the inverse transform module 348 to produce residual samples of the transform unit, indicated by arrow 350. The summation module 352 adds the residual sample 350 and PU 320 to produce reconstructed samples of the CU (indicated by arrow 354). Reconstructed sample 354 is passed to reference sample cache 356 and in-loop filter module 368. Reference sample cache 356, typically implemented using static RAM on an ASIC (thus avoiding expensive off-chip memory access), provides minimal sample storage to satisfy the dependencies used to generate intra-prediction blocks for subsequent CUs in a frame. Minimal dependencies typically consist of a “line buffer” of samples along the bottom of a row of CTUs for use by the next row of CTUs and a column buffer whose range is set by the height of the CTU. Reference sample cache 356 feeds reference samples (indicated by arrow 358) to reference sample filter 360. Sample filter 360 applies a smoothing operation to produce filtered reference samples (indicated by arrow 362). Filtered reference samples 362 are used by intra-prediction module 364 to produce intra-prediction blocks of samples indicated by arrow 366.For each candidate intra-prediction mode, the intra-prediction module 364 generates a sample block, namely 366.
[0072] The in-loop filter module 368 applies several filtering stages to the reconstructed sample 354. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned with the CU boundaries to reduce artifacts caused by discontinuities. Another filtering stage present in the in-loop filter module 368 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. Another available filtering stage in the in-loop filter module 368 is a "sample adaptive offset" (SAO) filter. The SAO filter works by first classifying the reconstructed sample into one or more categories and then applying an offset at the sample level according to the assigned category. Filtered sample 370 is output from the in-loop filter module 368. Filtered sample 370 is stored in frame buffer 372. Frame buffer 372 typically has a capacity to store several (e.g., up to 16) images and is therefore stored in memory 206. Thus, access to frame buffer 372 is expensive in terms of memory bandwidth. Frame buffer 372 provides reference frames (indicated by arrow 374) to motion estimation module 376 and motion compensation module 380.
[0073] Motion estimation module 376 estimates multiple "motion vectors" (denoted as 378), each a Cartesian space offset relative to the current CU's position, thus referencing a block of one of the reference frames in frame buffer 372. A filtered block of reference samples (denoted as 382) is generated for each motion vector. The filtered reference samples 382 form further candidate modes for potential selection by mode selector 386. Furthermore, for a given CU, PU 320 can be formed using one reference block ("single prediction") or two reference blocks ("double prediction"). For the selected motion vector, motion compensation module 380 generates PU 320 based on filtering that supports sub-pixel precision in the motion vector. Thus, motion estimation module 376 (which operates on many candidate motion vectors) can conceivably perform simplified filtering compared to motion compensation module 380 (which operates only on the selected candidate), achieving reduced computational complexity.
[0074] Although the reference to Universal Video Coding (VVC) explains Figure 3 The video encoder 114 can be used, but other video encoding standards or implementations can also employ the processing stages of modules 310-386. Frame data 113 (and bit stream 115) can also be sourced from memory 206, hard disk drive 210, CD-ROM, Blu-ray disc. TMThe data can be read from (or written to) memory 206, hard disk drive 210, CD-ROM, Blu-ray disc, or other computer-readable storage media. Additionally, frame data 113 (and bit stream 115) can be received from (or sent to) an external source, such as a server or radio frequency receiver connected to the communication network 220.
[0075] exist Figure 4 The video decoder 134 is shown. Although... Figure 4 The video decoder 134 is an example of a Universal Video Coding (VVC) video bit-coding pipeline, but other video codecs can also be used for the processing stages described herein. Figure 4 As shown, bitstream 133 is input to video decoder 134. Bitstream 133 can be read from memory 206, hard disk drive 210, CD-ROM, Blu-ray disc, or other non-transitory computer-readable storage media. Alternatively, bitstream 133 can be received from an external source, such as a server connected to communication network 220 or an RF receiver. Bitstream 133 contains encoded syntax elements representing the captured frame data to be decoded.
[0076] Bitstream 133 is input to entropy decoder module 420. Entropy decoder module 420 extracts syntax elements from bitstream 133 and passes the values of the syntax elements to other modules in video decoder 134. Entropy decoder module 420 applies the CABAC algorithm to decode the syntax elements from bitstream 133. The decoded syntax elements are used to reconstruct parameters within video decoder 134. Parameters include residual coefficients (indicated by arrow 424) and mode selection information such as intra-frame prediction mode 458 and explicit multi-transform index 454. Mode selection information also includes information such as motion vectors and partitioning of each CTU to one or more CUs. Parameters are used to generate PUs, typically combined with sample data from previously decoded CUs.
[0077] Bitstream 133 also contains data from bin string 1042 (see...) Figure 10A The entropy decoder module 420 extracts the bin string and generates a vertical transform set index (from vertical transform set index column 1046) and a horizontal transform set index (from horizontal transform set index column 1048).
[0078] The residual coefficients 424 are input to the dequantizer module 428. The dequantizer module 428 inversely scales the residual coefficients 424 to generate reconstructed intermediate transform coefficients (indicated by arrow 432) based on the quantization parameters. If a non-uniform inverse quantization matrix is indicated in the bitstream 133, the video decoder 134 reads the quantization matrix from the bitstream 133 as a sequence of scaling factors and arranges the scaling factors into a matrix according to the Z-order scan used to encode the residual coefficients. The inverse scaling then combines the quantization matrix with the quantization parameters to create the reconstructed intermediate transform coefficients. Using a Z-order scan for the residual coefficients scans the quantization matrix scaling factors in the same way, avoiding the presence of additional scan patterns and the associated memory and complexity burden for infrequent scans. The reconstructed intermediate transform coefficients 432 are passed to the inverse secondary transform module 436. The inverse secondary module 436 performs a "secondary inverse transform" to produce the reconstructed transform coefficients (indicated by arrow 440). The secondary transform is performed based on the determined transform block size. See reference... Figure 9 The reconstructed transform coefficients 440 are passed to the inverse transform module 444. Module 444 transforms the coefficients from the frequency domain back to the spatial domain. The transform block is effectively based on the values of effective and ineffective residual coefficients. The result of the operation of module 444 is a block of residual samples, indicated by arrow 448. The residual samples 448 are equal in size to the corresponding CU. The residual samples 448 are fed to the summing module 450. At the summing module 450, the residual samples 448 are added to the decoding PU 452 to produce a block of reconstructed samples, indicated by arrow 456. The reconstructed samples 456 are fed to the reconstructed sample cache 460 and the in-loop filtering module 488. The in-loop filtering module 488 produces a reconstructed block of frame samples, represented as 492. Frame samples 492 are written to the frame buffer 496.
[0079] The reconstruction sample cache 460 operates in a manner similar to the reconstruction sample cache 356 of the video encoder 114. The reconstruction sample cache 460 provides storage for reconstructed samples required for intra-frame prediction of subsequent CUs in the absence of memory 206 (e.g., by using data 232, typically on-chip memory, instead). The reference sample, indicated by arrow 464, is obtained from the reconstruction sample cache 460 and supplied to the reference sample filter 468 to produce the filtered reference sample, indicated by arrow 472. The filtered reference sample 472 is supplied to the intra-frame prediction module 476. Module 476 generates blocks of intra-frame prediction samples, indicated by arrow 480, based on the intra-frame prediction mode parameters 458 represented in bitstream 133 and decoded by the entropy decoder 420.
[0080] When intra-prediction is indicated in the current CU bitstream 133, the intra-prediction sample 480 is formed into a decoding PU 452 via the multiplexer module 484.
[0081] When inter-frame prediction is indicated in the current CU bitstream 133, the motion compensation module 434 uses motion vectors and reference frame indices to select and filter sample blocks from the frame buffer 496 to generate blocks of inter-frame prediction samples 438. Sample blocks 498 are obtained from previously decoded frames stored in the frame buffer 496. For dual prediction, two sample blocks are generated and blended together to produce samples for the decoded PU 452. The frame buffer 496 is filled with filtered block data 492 from the in-loop filtering module 488. Similar to the in-loop filtering module 368 of the video encoder 114, the in-loop filtering module 488 applies any, at least, all, DBF, ALF, and SAO filtering operations. The in-loop filtering module 368 generates filtered block data 492 from the reconstructed samples 456.
[0082] Figure 5 This is a schematic block diagram illustrating a set 500 of available partitions or splits of blocks into one or more blocks within the tree structure of a general video coding system. (See reference...) Figure 3 The segmentation shown in set 500 can be used by block partitioner 310 of encoder 114 to segment each CTU into one or more CUs according to Lagrange optimization.
[0083] Although set 500 only shows the division of square blocks into other possible non-square blocks, it should be understood that set 500 shows potential divisions and does not constrain the containing block to be square. If the containing block is non-square, the size of the block obtained from the division is scaled according to the aspect ratio of the containing block. A specific sub-segment of one or more CUs using the CTU of block partitioner 310 is called the “coding tree” of the CTU. In the context of this invention, a leaf node is the node where the sub-segmentation process terminates. The sub-segmentation process must terminate when the region corresponding to the leaf node is equal to the minimum coding unit size. The leaf node that yields the minimum coding unit size is located at the deepest level of the decomposition of the coding tree. The sub-segmentation process can also terminate before the deepest level of the decomposition, thus yielding a coding unit larger than the minimum coding unit size.
[0084] In the absence of further sub-splitting, a CU exists at a leaf node of the coding tree. For example, leaf node 510 contains a CU. At non-leaf nodes of the coding tree, there are splits into two or more other nodes, each of which may contain a CU or further splits into smaller regions.
[0085] like Figure 5As shown, quadtree split 512 divides the containing region into four regions of equal size. Compared to HEVC, Universal Video Coding achieves additional flexibility by adding horizontal binary split 514 and vertical binary split 516. Splits 514 and 516 each divide the containing region into two regions of equal size. The splits are made along the horizontal boundary (514) or vertical boundary (516) within the containing block.
[0086] Further flexibility is achieved in general video coding by adding ternary horizontal splitting (518) and ternary vertical splitting (520). Ternary splitting (518) and (520) divides a block into segments along the width or height of the containing region. 1 / 4 and 3 / 4 forms three regions that are bounded in the horizontal direction (518) or the vertical direction (520). A combination of quadtrees, binary trees, and ternary trees is called a "QTBTT" or alternatively a multi-branch tree (MT).
[0087] Compared to HEVC, which only supports quadtrees and therefore only square blocks, QTBTT offers a wider range of possible CU sizes, particularly considering the potential recursive application of binary and / or ternary tree splits. The likelihood of anomalous (e.g., non-square) block sizes can be reduced by constraining split options to eliminate splits that result in block widths or heights less than four samples or that are not multiples of four samples. Typically, constraints will apply when considering luma samples. However, constraints can also be applied individually to chroma channel blocks, potentially resulting in different minimum block sizes for luma vs. chroma (e.g., when frame data is in a 4:2:0 chroma format).
[0088] Figure 6 This is a schematic flowchart illustrating the data stream 600 of the QTBTTT (or "coding tree") structure used in general video coding. The QTBTTT structure is used for each CTU to define the partitioning of the CTU into one or more CUs. The QTBTTT structure of each CTU is determined by the block partitioner 310 in the video encoder 114 and is encoded into bit stream 115 or decoded from bit stream 133 by the entropy decoder 420 in the video decoder 134. Figure 5 As shown in the diagram, data stream 600 further exhibits features that allow block partitioner 310 to partition the CTU into one or more CUs using a licensed combination.
[0089] Starting from the top level of the hierarchy, i.e., at CTU, zero or more quadtree partitions are performed first. Specifically, the block partitioner 310 makes a quadtree (QT) split decision 610. The decision at 610 returns a "1" sign, indicating that it is decided to split the current node into four child nodes according to the quadtree split 512. The result is the generation of four new nodes, such as at 620, and for each new node, the process recursively returns to the QT split decision 610. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates no further splitting (returns a "0" sign), the quadtree partitioning stops, and multitree (MT) splits are subsequently considered.
[0090] First, block partitioner 310 makes an MT split decision 612. At 612, a decision indicating whether to perform an MT split is made is given. Decision 612 returns a "0" sign, indicating that no further splitting of the node into its child nodes will occur. If no further splitting of the node occurs, the node is a leaf node of the coding tree and corresponds to a coding unit (CU). The leaf node is output at 622. Alternatively, if MT split 612 indicates a decision to perform an MT split (returning a "1" sign), block partitioner 310 proceeds to direction decision 614.
[0091] Direction decision 614 indicates the direction of MT splitting as horizontal (“H” or “0”) or vertical (“V” or “1”). If decision 614 returns “0” indicating the horizontal direction, block partitioner 310 proceeds to decision 616. If decision 614 returns “1” indicating the vertical direction, block partitioner 310 proceeds to decision 618.
[0092] In each of decisions 616 and 618, the number of partitions to be split in the MT partition during the BT / TT split is indicated as two (binary split or "BT" nodes) or three (ternary split or "TT" nodes). That is, when the direction indicated from 614 is horizontal, the block partitioner 310 makes the BT / TT split decision 616, and when the direction indicated from 614 is vertical, the block partitioner 310 makes the BT / TT split decision 618.
[0093] BT / TT split decision 616 indicates whether the horizontal split is a binary split 514 indicated by returning "0" or a ternary split 518 indicated by returning "1". When BT / TT split decision 616 indicates a binary split, at step 625 of generating HBT CTU nodes, block partitioner 310 generates two nodes based on binary horizontal split 514. When BT / TT split 616 indicates a ternary split, at step 626 of generating HTT CTU nodes, block partitioner 310 generates three nodes based on ternary horizontal split 518.
[0094] The BT / TT split decision 618 indicates whether the vertical split is a binary split 516 indicated by returning "0" or a ternary split 520 indicated by returning "1". When the BT / TT split 618 indicates a binary split, at step 627 of generating VBT CTU nodes, the block partitioner 310 generates two nodes according to the vertical binary split 516. When the BT / TT split 618 indicates a ternary split, at step 628 of generating VTT CTU nodes, the block partitioner 310 generates three nodes according to the vertical ternary split 520. For each node obtained from steps 625-628, the data flow 600 is applied according to direction 614 in a left-to-right or top-to-bottom order, recursively returning to the MT split decision 612. As a result, binary and ternary splits can be applied to generate CUs of various sizes.
[0095] Figure 7A and 7B Provides an example segmentation of CTU 710 into multiple coding units (CUs) 700. Figure 7A Example CU712 is shown in the figure. Figure 7A This illustrates the spatial arrangement of the CUs in a CTU 710. Example segmentation 700 in... Figure 7B It is also shown as coding tree 720.
[0096] exist Figure 7A At each non-leaf node in CTU 710 (e.g., nodes 714, 716, and 718), the contained nodes (which may be further splits or CUs) are scanned or traversed in "Z order" to create a list of nodes represented as columns in the coding tree 720. For quadtree splits, the Z-order scan results in a sequence from top left to right, followed by a sequence from bottom left to right. For horizontal and vertical splits, the Z-order scan (traversal) simplifies to scans from top to bottom and from left to right, respectively. Figure 7B The encoding tree 720 lists all nodes and CUs according to the applied scan order. Each split generates a list of 2, 3, or 4 new nodes at the next level of the tree, until a leaf node (CU) is reached.
[0097] As per reference Figure 3 In the case where the image is decomposed into CTUs and further decomposed into CUs using the block partitioner 310, and each residual block (324) is generated using the CUs, the residual blocks are forward transformed and quantized using the encoder 114. The resulting transform blocks (TBs) 336 are then scanned to form an ordered list of residual coefficients as part of the operation of the entropy coding module 338. Equivalent processing is performed in the video decoder 134 to obtain the transform blocks from the bitstream 133.
[0098] Figure 8AA set 800 of intra-prediction modes for transform blocks that can be indicated using intra-prediction modes 388 and 458 is shown. Figure 8A The document defines 67 intra-prediction modes. Mode 0 is a "planar" intra-prediction mode, Mode 1 is a "DC" intra-prediction mode, and Modes 2-66 are "angle" intra-prediction modes. The planar intra-prediction mode (Mode 0) fills the prediction block with samples based on a plane (i.e., with offsets and gradients in both the horizontal and vertical directions). The plane parameters are obtained from neighboring reference samples (if any). Similarly, the DC intra-prediction mode (Mode 1) also fills the prediction block with offsets using neighboring reference samples (if any).
[0099] The angular intra-frame prediction mode (modes 2-66) fills the block by generating textures aligned with one of 65 directions or "angles". For clarity, in Figure 8A The diagram shows a subset of 65 angles (i.e., modes 2, 18, 34, 50, and 66). For each mode, neighboring reference samples are used to generate a texture that fills the prediction block in the direction indicated by the arrow for the intra-frame prediction mode of the angle. Figure 8A Additional angles, not explicitly shown, are located at intermediate positions (i.e., patterns 3-17, 19-33, 35-49, and 51-65). From Figure 8A A first symmetry can be seen along angle pattern 34 and in the diagonal direction from the upper left to the lower right of the prediction block. From the perspective of the first symmetry, by transposing along this diagonal axis, patterns 2 to 34 are shown as corresponding to pattern 66 downwards up to pattern 34.
[0100] Figure 8B A mapping 840 is shown from each transform set 842 to a specific transform type according to transform index 844. The three transform sets 842 are defined and identified by corresponding transform set indices of 0, 1, and 2. Each transform set 842 includes two transform types that can be selected by transform index 844. As a result, mapping 840 allows selection of a given transform type based on the transform set index of the transform set 842 and transform index 844. The selected transform is a one-dimensional transform. Separate selection of transform types is made for the horizontal and vertical directions. The size (number of points) of the transform corresponds to the size of the transform block. That is, when a vertical transform is selected, the size of the transform corresponds to the height of the transform block, while when a horizontal transform is selected, the size of the transform corresponds to the width of the transform block.
[0101] Figure 8CThis is a table showing the mapping 880 from intra-prediction modes to transform set indices of transform set 842 for transform blocks based on whether the transform block is in the horizontal direction (“H”) or the vertical direction (“V”). For each intra-prediction mode, one of the transform set indices associated with the three transform sets 842 is selected for the horizontal direction H, and one of the transform set indices associated with the three transform sets 842 is selected for the vertical direction V.
[0102] For example, for a transform block in the vertical direction, intra-prediction mode 14 is mapped to transform set index 0, while for a transform block in the horizontal direction, intra-prediction mode 14 is mapped to transform set index 2. Figure 8B As shown in mapping 840, transform set index 0 is mapped to either DST-VII or DCT-VIII, selectable according to transform index 844. Figure 8B As shown in mapping 840, transform set index 2 is mapped to either DST-VII or DCT-V, selectable according to transform index 844. Therefore, for transform blocks in the vertical direction, intra-prediction mode 14 uses either DST-VII or DCT-VIII transform according to transform index 844. Similarly, for transform blocks in the horizontal direction, intra-prediction mode 14 uses either DST-VII or DCT-V transform according to transform index 844. Transform index 844 is based on explicit multiple transform indexes 390 or 454. As described above, explicit multiple transform index 390 is selected by mode selector 306, and explicit multiple transform index 454 is derived from bitstream 133 by entropy encoder 420.
[0103] Figure 9 This is a schematic block diagram showing the inverse transform module 348 of the video encoder 114 or the inverse transform module 444 of the video decoder 134. The two modules 348 and 444 operate identically, thus reflecting the need of the video encoder 114 to model the inverse quantization and inverse transform of the residual coefficients performed in the video decoder 134.
[0104] Inverse transformation modules 348 and 444 each include a transformation set table module 910, a transformation index deriver module 920, a horizontal transformation selector module 940, a vertical transformation selector module 945, a horizontal transformation module 960, and a vertical transformation module 980.
[0105] Transform set table module 910 receives intra-prediction mode 388 or 458. When implemented as inverse transform module 348, transform set table module 910 receives intra-prediction mode 388 from mode selector 386 (see [link]). Figure 3 When implemented as inverse transform module 444, transform set table module 910 receives intra-prediction mode 458 from entropy encoder 420 (see...). Figure 4 ).
[0106] The transform set table module 910 determines the horizontal transform set index (represented by arrow 915) and the vertical transform set index (represented by arrow 917) from the intra-frame prediction mode 388 or 458. The horizontal transform set index 915 and the vertical transform set index 917 are determined based on mapping 880 (see...). Figure 8C It is certain.
[0107] Transform index deriver 920 receives explicit multiple transform indices 390 or 454. When implemented as inverse transform module 348, transform index deriver 920 receives explicit multiple transform index 390 from mode selector 386 (see also...). Figure 3 When implemented as inverse transform module 444, transform index deriver 920 receives explicit multi-transform index 454 from entropy encoder 420 (see [link to entropy encoder 420]). Figure 4 ).
[0108] Transform index exporter 920 extracts horizontal transform index 920 and vertical transform index 925 from explicit multiple transform indexes 390 or 454. Horizontal transform index 920 and vertical transform index 925 are each in the range of 0 to 1.
[0109] The horizontal transform selector module 940 receives a horizontal transform set index 915 (from the transform set table module 910) and a horizontal transform index 920 (from the transform index exporter 920). The horizontal transform selector 940 then selects the horizontal transform index based on mapping 840 (see...). Figure 8B The selected horizontal transformation (represented by arrow 950) is generated from horizontal transformation set index 915 and horizontal transformation index 920. Horizontal transformation set index 915 is used to select one of the transformation sets 842, and horizontal transformation index 920 is used to select one of the transformation indices 844.
[0110] The vertical transform selector module 945 receives a vertical transform set index 917 (from the transform set table module 910) and a vertical transform index 925 (from the transform index deriver 920). The vertical transform selector module 945 then selects the vertical transform index according to mapping 840 (see...). Figure 8B The selected vertical transform 955 is generated from vertical transform set index 917 and vertical transform index 925. Vertical transform set index 917 is used to select one of transform sets 842, and vertical transform index 925 is used to select one of transform indices 844.
[0111] The horizontal transformation module 960 applies the selected horizontal transformation 950 to the intermediate inverse transformation coefficients 346 or the reconstructed transformation coefficients 440. The intermediate inverse transformation coefficients 346 are derived from the inverse secondary transformation module 344 (see [link to module 344]). Figure 3 The reconstructed transform coefficients 440 are derived from the inverse secondary transform module 436 (see [link]). Figure 4 ).
[0112] Transform coefficients 346 or 440 are applied based on the selected horizontal transform 950. That is, the selected horizontal transform 950, with a set of one-dimensional transforms, is applied to each row of the array of transform coefficients 346 or 440. Each one-dimensional transform spans the width of the transform block. The horizontal transform module 960 outputs the horizontal transform coefficients 970 as an array with the same size as the current transform block. The transform coefficients 970 are supplied to the vertical transform module 980. For ease of description, the transform coefficients 970 are referenced to both the horizontal transform coefficients 346 and 440.
[0113] The vertical transformation module 980 applies a selected vertical transformation 955, which has a set of one-dimensional transformations, to each column of the transformation coefficients 970 of the horizontal transformation. The vertical transformation module 980 uses the selected vertical transformation 955 to generate residual samples 350 or 448 from the corresponding transformation coefficients 346 or 440 of the horizontal transformation.
[0114] Figure 10A Table 1000 is shown, illustrating the relationship between truncated unary syntax elements and the transform indices used for transform selection. The transform indices are the horizontal transform index H (shown in column 1048) and the vertical transform index V (shown in column 1046). Table 1000 includes a bin string column 1042, a binarization content column 1044, a vertical transform index column 1046, and a horizontal transform index column 1048.
[0115] Bin string column 1042 has four possible bin strings, which are derived from truncated unary binarized content with a maximum value of 3 (i.e., ranging from 0 to 3). For encoder 114, the bin strings shown in column 1042 are the bin strings encoded by entropy encoder 338 into bitstream 115. The bin strings to be encoded into bitstream 115 are selected based on the horizontal transform index H and vertical transform index V selected by mode selector 386.
[0116] For decoder 134, the bin string shown in column 1042 is the bin string decoded from bit stream 133 by entropy decoder 420. Then, entropy decoder 420 outputs the horizontal transform index H and the vertical transform index V based on the decoded bin string.
[0117] Binarized content column 1044 comprises the components of each bin string in bin string column 1042. In this naming, "Ctx0" and "Ctx1" refer to two different CABAC contents used to encode the corresponding bin in bin string column 1042. The term "Byp" refers to the bypass encoded bin used to encode the corresponding bin in bin string column 1042.
[0118] Vertical transform index column 1046 contains index values (0 or 1) used to select transforms from the transform set used for vertical 1D transforms. Horizontal transform index column 1048 contains index values (0 or 1) used to select transforms from the transform set used for horizontal 1D transforms. In Table 1000, the shortest bin string is assigned to the cases where the vertical and horizontal indices are "0" and "0". The shortest bin string corresponds to the most frequently selected case using pattern selector 386. Then, the second shortest bin string is assigned to the cases where the vertical and horizontal indices are "1" and "1". This corresponds to the second most frequently selected case using pattern selector 386. Then, the cases where the combined index values of the vertical and horizontal transform indices are "01" or "10" are also impossible and are assigned to the longest bin string in column 1044. Furthermore, a bypass-coded bin is used to distinguish between these two cases, thus reflecting the approximately equal probability of occurrence between these two selections using pattern selector 386. The described statistical properties were found to be largely invariant with respect to the change in size, and thus the presence of these contexts, chosen based on their positions in the truncated unary bin string, was sufficient to capture the asymmetric probability distribution of bin values.
[0119] Figure 10B Table 1050 shows the substitution relationship between truncated unary syntax elements and the transformation indices used for horizontal and vertical transformation indexing, where the third bin of the truncated unary binarized content uses context encoding instead of bypass encoding.
[0120] Figure 11 This is a flowchart of method 1100 for encoding a block of transforms, in which the residual coefficients are encoded into bitstream 115, based on explicit transform selection of the primary transform. Method 1100 can be implemented by a device such as a configured FPGA, ASIC, or ASSP. Alternatively, method 1100 can be performed by video encoder 114 under the execution of processor 205. Thus, method 1100 can be stored on a computer-readable storage medium and / or in memory 206. Method 1100 begins at step 1110, which utilizes processor 205, to determine the intra-frame prediction mode.
[0121] Method 1100 begins with step 1110 of the intra-prediction mode. At step 1110 of the intra-prediction mode, mode selector 386, under the execution of processor 205, selects an intra-prediction mode 388 for the prediction unit associated with the current coding unit. This selection is typically performed in two passes. In the first pass, all intra-prediction modes for the luma prediction block are tested. For each mode, the residual cost is approximated using, for example, an "absolute transform difference sum" method such as the Hadamard transform. A list of "best" (lowest distortion) candidate prediction modes is derived from this test. Then, a full test of residual coding is performed on the list of candidate prediction modes. As a result of the full test, an intra-prediction mode 388 is selected for the current coding unit. Entropy encoder 338 encodes the intra-prediction mode 388 into bitstream 115. Then, control in processor 205 moves from step 1110 of the intra-prediction mode to step 1120 of determining the transform set.
[0122] At step 1120, in determining the transform set, the inverse transform module 348, under the execution of the processor 205, uses the determined intra-prediction mode 388 to determine the transform set according to the mapping 880 (see...). Figure 8C Determine the level transformation set index 915 (as mentioned above regarding...) Figure 9 The above) and vertical transform set index 917 (as mentioned above) Figure 9 (As stated above). From the horizontal transform set index 915, a set of transformations is determined for the horizontal transform according to mapping 840. From the vertical transform set index 917, a set of transformations is determined for the vertical transform according to mapping 840.
[0123] As described in the example above, if the intra-frame prediction mode 388 is mode 14, the possible transformations for the vertical transformation are DST-VII or DCT-VIII, and the possible transformations for the horizontal transformation are DST-VII or DCT-V.
[0124] Then, control in processor 205 moves from step 1120, which determines the transform set, to step 1130, which tests the transform.
[0125] At step 1130 of the transformation test, transformation module 326 and inverse transformation module 348, under the execution of processor 205, test various combinations of transformations from the transform set used for the horizontal transformation and the transform set used for the vertical transformation from step 1120. In the example where intra-frame prediction mode 388 is mode 14, the transformations of DST-VII and DCT-VIII are tested for the vertical transformation, and the transformation of DST-VII or DCT-V is tested for the horizontal transformation.
[0126] As described above, the tests are performed by the transform module 326 and the inverse transform module 348. Specific transforms in each set are referenced using the horizontal transform index 920 (used for horizontal transforms) and the vertical transform index 925 (used for vertical transforms). Combinations of vertical and horizontal transforms are tested to determine the distortion used to encode the residuals.
[0127] During testing, transformation module 326 and inverse transformation module 348 operate normally. Therefore, for every inverse transformation operation performed by inverse transformation module 348, transformation module 326 performs a corresponding forward transformation operation. The test process can be performed using an approximation method, which reduces complexity compared to performing the full transformation.
[0128] Then, control in processor 205 moves from step 1130 of testing transform set to selecting transform 1140.
[0129] At step 1140 of the transform selection, a combination of vertical and horizontal transforms that yields the lowest distortion for encoding the residual is selected. In the example where intra-prediction mode 388 is mode 14, the combination providing the lowest distortion could be DCT-VIII for the vertical transform and DST-VII for the horizontal transform. Then, control in processor 205 moves from transform selection step 1140 to applying the horizontal transform 1150.
[0130] At step 1150 of applying the horizontal transform, the inverse transform module 348 performs the horizontal transform. Specifically, the horizontal transform module 960 performs the selected horizontal transform 950. In the example where the intra-frame prediction mode 388 is mode 14, DST-VII is used for the horizontal transform. Then, control in the processor 205 moves from applying the horizontal transform 1150 to applying the vertical transform 1160.
[0131] At step 1160, where the vertical transform is applied, the inverse transform module 348 performs the vertical transform. Specifically, the vertical transform module 980 performs the selected vertical transform 955. In the example where the intra-prediction mode 388 is mode 14, DCT-VIII is used for the vertical transform. As a result of step 1150, where the horizontal transform is applied, and step 1160, where the vertical transform is applied, and after the optional application of the secondary transform (i.e., the operation of module 330), transform coefficients 332 are generated. Then, control in the processor 205 moves from step 1160, where the vertical transform is applied, to step 1170, where the residual coefficients are quantized.
[0132] At step 1170, where the residual coefficients are quantized, the quantization module 334, under the execution of the processor 205, quantizes the transform coefficients 332 to generate residual coefficients 336 according to the quantization parameters. Then, control in the processor 205 moves from step 1170, where the residual coefficients are quantized, to step 1180, where the residual coefficients are encoded.
[0133] At step 1180, where the residual coefficients are encoded, the entropy encoder 338, under the execution of the processor 205, encodes the residual coefficients 336 into the bitstream 115. The residual coefficients 336 of a given transform block are encoded by scanning a two-dimensional array of coefficients into a list. The scan typically divides the transform block into one or more sub-blocks, each of size 4×4 (although other sizes are also possible). The presence of valid coefficients in each sub-block is indicated by an "encoded sub-block flag," and a validity flag exists for each residual coefficient in the sub-block if at least one valid coefficient is present. The encoded sub-block flag and validity flag use a context encoding bin. Additional indications of size, such as "greater than 1 flag" and "greater than 2 flags," are also encoded using the context encoding bin. Finally, further indications of size are typically encoded using Golomb Rice encoding utilizing a bypass encoding bin. The bit cost when encoding the residual coefficients 336 is largely influenced by their size. As a result of selecting the optimal EMT index, these sizes are slightly reduced, leading to improved compression efficiency. The binarization of these indices performed at step 1190 further improves compression efficiency. Then, control in processor 205 moves from step 1180, which encodes the residual coefficients, to step 1190, which encodes the EMT bin string.
[0134] At step 1190, which encodes the EMT bin string, the entropy encoder 338, under the execution of the processor 205, uses truncated unary binarized content (maximum value 3), context modeling, and as referenced... Figure 10AThe mapping of bin strings to index values is used to encode the horizontal transform index 920 and the vertical transform index 925 into bitstream 115. Since the shortest bin string (“0”) is used to encode the most frequently chosen index pair (“0” in both the horizontal and vertical directions), the encoding efficiency is higher than the simple case of using one bin for each index. The second shortest bin string (i.e., “1 0”) is used to encode the second most frequently chosen index pair (“1” in both the horizontal and vertical directions), and the longest bin strings (“1 1 0” and “1 1 1”) are used to encode the two approximately equivalent least likely combinations (“1” and “0”, “0” and “1”). Since there is no statistical symmetry available for development using context encoding, the third bin uses bypass encoding. Method 1100 then terminates at the end of step 1190.
[0135] Figure 12 This is a flowchart of a method 1200 for decoding a transform block of residual coefficients from bitstream 133. Decoding in method 1200 is performed by selecting a primary inverse transform based on explicit transform selection. Method 1200 can be implemented by a device such as a configured FPGA, ASIC, or ASSP. Alternatively, method 1200 can be executed by a video decoder 134 under the processing of processor 205. Thus, method 1200 can be stored on a computer-readable storage medium and / or in memory 206. Method 1200 begins at step 1210 of decoding the intra-frame prediction mode using processor 205.
[0136] At step 1210, where the intra-prediction mode is decoded, entropy decoder 420, under the execution of processor 205, decodes intra-prediction mode 458 from bitstream 133. Entropy decoder 420 generates a list of maximum probability modes, including those considered most likely to be selected by mode selector 386. Context-coded bins are decoded to indicate the use of any of the maximum probability modes or other modes not in the maximum probability mode list (“remaining modes”). Further bins are then decoded to identify which maximum probability mode or which remaining mode is in use. Control in processor 205 then moves from step 1210, where the intra-prediction mode is decoded, to step 1220, where the residual coefficients are decoded.
[0137] At step 1220, where the residual coefficients are decoded, the entropy decoder 420, executed by the processor 205, decodes the list of residual coefficients and assembles them into an array for transform blocks. This assembly follows a scanning pattern that typically groups the residual coefficients into 4×4 sub-blocks. The list of residual coefficients is typically scanned in reverse direction, from the “last” valid residual coefficient back towards the upper left (“DC” residual coefficients, although this may not correspond to block offsets not based on cosine functions). A context code bin is decoded to indicate the presence of valid residual coefficients in each sub-block, and, if indicated, an additional context code bin is decoded to indicate the validity of each residual coefficient in the sub-block. The magnitude of each residual coefficient can be further characterized by decoding a “greater than 1” or “greater than 2” flag. Finally, for residual coefficients whose size is not determined by the previous context code bin, the “remaining size” is decoded using Golomb-Rice coding with a bypass code bin. As a result of this decoding, the residual coefficients 424 of the transformed block are decoded from bit stream 133. Then, control in processor 205 moves from step 1220 of decoding the residual coefficients to step 1230 of inverse quantization of the residual coefficients.
[0138] At step 1230, where the residual coefficients are inversely quantized, the dequantizer module 428, under the execution of the processor 205, generates intermediate transform coefficients 432 by scaling the residual coefficients 424 according to the quantization parameters. The intermediate transform coefficients 432 can be transformed by applying an inseparable minor transform, typically applied only to the upper left region of the transform block, to produce intermediate transform coefficients 432. As a result of this scaling operation, the intermediate transform coefficients 432 have approximately the same size as the coefficients in the video encoder before quantization (i.e., transform coefficients 332). The reconstructed transform coefficients 440 are obtained from the intermediate transform coefficients 432 by optionally applying an inverse inseparable minor transform (i.e., inverse minor transform module 436). Control in the processor 205 then proceeds from step 1230, where the residual coefficients are inversely quantized, to step 1240, where the EMT bin string is decoded.
[0139] At step 1240, during the decoding of the EMT bin string, the entropy decoder 420, under the execution of the processor 205, decodes the truncated unary bin string from the bit stream 133. The truncating of the unary bin string is... Figure 10A One of the bin strings listed in column 1042. For example... Figure 10AAs shown and described above, the unary bin string is limited to three bins in length (ranging from 0 to 3). The truncated unary bin string is of length from 1 to 3 bins, where the first bin is context-encoded, the second bin (if present) is context-encoded using a different context, and the third bin (if present) is bypass-encoded. This binarization is as described with reference to FIG10. Then, control in processor 205 moves from step 1240, which decodes the EMT bin string, to step 1250, which determines the EMT index.
[0140] At step 1250, when determining the EMT index, the transform index deriver 920 in the video decoder 134, under the execution of the processor 205, determines the transform index derives according to... Figure 10A Table 1000 in the processor 205 determines the horizontal transform index 920 and the vertical transform index 925. Then, control in the processor 205 moves from step 1250, which determines the EMT index, to step 1260, which determines the transform set.
[0141] At step 1260, which determines the transform set, the inverse transform module 444, under the execution of processor 205, determines which transform set in map 840 will be used for the horizontal and vertical transforms of the primary inverse transform. Specifically, using intra-frame prediction mode 458, transform set table 910 is used to determine horizontal transform set 915 and vertical transform set 917. Then, control in processor 205 moves from step 1260, which determines the transform set, to step 1270, which selects the transform.
[0142] At step 1270 of the transformation selection, the inverse transformation module 444, under the execution of the processor 205, determines the horizontal transformation 950 and the vertical transformation 955. The determination of the horizontal transformation 950 and the vertical transformation 955 is performed by the horizontal transformation selector 940 and the vertical transformation selector 945, respectively. These modules are as follows... Figure 9 The diagram shows the use of horizontal transform set index 915, vertical transform set index 917, horizontal transform index 920, and vertical transform index 925 as input, and based on... Figure 8B The selected horizontal transformation 950 and the selected vertical transformation 960 are generated. Then, the control in the processor 205 moves from the transformation selection step 1270 to the application of the horizontal transformation step 1280.
[0143] At step 1280 of applying the horizontal transform, the inverse transform module 444, under the execution of the processor 205, uses the horizontal transform module 960 to apply the selected horizontal transform 950. The selected horizontal transform 950 is applied to each row of the reconstructed transform coefficients 440 to generate the horizontal transform coefficients 970. The horizontal transform coefficients 970 are supplied to the vertical transform module 980. Then, the vertical transform module 980 generates the residual sample 448 by applying the selected vertical transform 955. (See reference...) Figure 4 The residual sample 950 is further used to generate a reconstructed sample by adding it to the intra-frame prediction PU. Then, method 1200 terminates.
[0144] In another configuration of methods 1100 and 1200, the binarization of the truncated unary bin string is modified so that only one context-encoded bin is used. Specifically, refer to... Figure 10A The first bin of the binarized content 1044 remains context-encoded, while the second bin of the binarized content 1044 uses a bypass-encoded bin instead of context encoding. This configuration using only one context-encoded bin saves implementation costs at the cost of a slight decrease in compression efficiency. However, this configuration achieves slightly higher throughput. The higher throughput is because, at least due to the lack of context state updates, the CABAC processing in the entropy encoder 338 and entropy decoder 420 can process the bypass-encoded bin at a higher rate compared to the context-encoded bin.
[0145] For reference Figure 10A The probabilities of selecting the horizontal transform index H and the vertical transform index V, which have values of "0" and "1" or "1" and "0" respectively, are approximately equal. Thus, Figure 10A The alternative binarized content shown in the diagram inverts the use of the third bypass coding bin. The configuration of inverting the use of the third bin maps the bin string "1 1 1" to "0" and "1" for the respective vertical and horizontal transform indices, and maps the bin string "1 1 0" to "1" and "0" for the respective vertical and horizontal transform indices. Since the probabilities between these two choices are approximately equal, equivalent compression advantages are achieved.
[0146] The result of binarization of explicit multi-transformation selection in the horizontal and vertical directions (especially the aspect of jointly encoding the two based on the expected probability of selecting each index value) improves encoding efficiency compared to using independent syntactic elements separately for the horizontal and vertical indices (e.g., each for a context encoding bin).
[0147] Industrial availability
[0148] The configuration described herein can be applied to the computer and data processing industries, and is particularly useful for digital signal processing that encodes or decodes signals such as video and image signals, thereby achieving high compression efficiency without incurring excessive costs in terms of memory bandwidth due to non-local scanning of residual coefficients.
[0149] The foregoing only illustrates some embodiments of the present invention, and modifications and / or changes can be made to the present invention without departing from the scope and spirit of the present invention, wherein these embodiments are merely exemplary and not restrictive.
[0150] In the context of this specification, the word "comprising" means "primarily but not necessarily only" or "has" or "includes," rather than "consisting of only." Suffixes such as "comprise" and "comprises" in the word "comprising" have corresponding variations in meaning.
Claims
1. A method for decoding a transform block in an image from a bitstream, the method comprising: Decode at least a single truncated unary codeword from the bit stream; By using the single truncated unary codeword, the horizontal and vertical transform types are determined for the transform block, wherein different transform types can be determined for the horizontal and vertical transform types by using the single truncated unary codeword; and The transform block is decoded by applying the horizontal transform type and the vertical transform type to the transform block. Wherein, the single truncated unary codeword represents a value used to specify the horizontal transform type and the vertical transform type, and In the case where the single truncated unary codeword corresponds to a bin string composed of multiple bins, the context values corresponding to different bins among the multiple bins are different from each other.
2. A method for encoding transform blocks in an image into a bitstream, the method comprising: Determine the horizontal transformation type for the transformation block; Determine the vertical transformation type for the transformation block; The transform block is encoded by applying the horizontal transform type and the vertical transform type to the transform block; and Encode at least a single truncated unary codeword. In the method for decoding the transform block, the horizontal transform type and the vertical transform type are determined for the transform block using the single truncated unary codeword. Specifically, by using the single truncated unary codeword, different transformation types can be determined for the horizontal transformation type and the vertical transformation type in the method of decoding the transform block. Wherein, the single truncated unary codeword represents a value used to specify the horizontal transform type and the vertical transform type, and In the case where the single truncated unary codeword corresponds to a bin string composed of multiple bins, the context values corresponding to different bins among the multiple bins are different from each other.
3. An apparatus for decoding transform blocks in an image from a bitstream, the apparatus comprising: A decoding unit is configured to decode at least a single truncated unary codeword from the bit stream; as well as A determining unit is configured to determine a horizontal transform type and a vertical transform type for the transform block using the single truncated unary codeword, wherein different transform types can be determined for the horizontal transform type and the vertical transform type by using the single truncated unary codeword. The decoding unit is configured to decode the transform block by applying the horizontal transform type and the vertical transform type to the transform block. Wherein, the single truncated unary codeword represents a value used to specify the horizontal transform type and the vertical transform type, and In the case where the single truncated unary codeword corresponds to a bin string composed of multiple bins, the context values corresponding to different bins among the multiple bins are different from each other.
4. An apparatus for encoding transform blocks in an image into a bitstream, the apparatus comprising: A determining unit is configured to determine a horizontal transformation type for the transform block and a vertical transformation type for the transform block; as well as An encoding unit is configured to encode the transform block by applying the horizontal transform type and the vertical transform type to the transform block. The encoding unit is configured to encode at least a single truncated unary codeword. In the device used to decode the transform block, the horizontal transform type and the vertical transform type are determined for the transform block using the single truncated unary codeword. Specifically, by using the single truncated unary codeword, different transformation types can be determined for the horizontal transformation type and the vertical transformation type in the device used to decode the transform block. Wherein, the single truncated unary codeword represents a value used to specify the horizontal transform type and the vertical transform type, and In the case where the single truncated unary codeword corresponds to a bin string composed of multiple bins, the context values corresponding to different bins among the multiple bins are different from each other.
5. A non-transitory computer-readable storage medium comprising a computer program, the computer program including instructions that, when executed by a computer, cause the computer to perform a method for decoding transform blocks in an image from a bitstream, the method comprising: Decode at least a single truncated unary codeword from the bit stream; By using the single truncated unary codeword, the horizontal and vertical transform types are determined for the transform block, wherein different transform types can be determined for the horizontal and vertical transform types by using the single truncated unary codeword; and The transform block is decoded by applying the horizontal transform type and the vertical transform type to the transform block. Wherein, the single truncated unary codeword represents a value used to specify the horizontal transform type and the vertical transform type, and In the case where the single truncated unary codeword corresponds to a bin string composed of multiple bins, the context values corresponding to different bins among the multiple bins are different from each other.
6. A non-transitory computer-readable storage medium including a computer program, the computer program including instructions that, when executed by a computer, cause the computer to perform a method for encoding transform blocks in an image into a bitstream, the method comprising: Determine the horizontal transformation type for the transformation block; Determine the vertical transformation type for the transformation block; The transform block is encoded by applying the horizontal transform type and the vertical transform type to the transform block; and Encode at least a single truncated unary codeword. In the method for decoding the transform block, the horizontal transform type and the vertical transform type are determined for the transform block using the single truncated unary codeword. Specifically, by using the single truncated unary codeword, different transformation types can be determined for the horizontal transformation type and the vertical transformation type in the method of decoding the transform block. Wherein, the single truncated unary codeword represents a value used to specify the horizontal transform type and the vertical transform type, and In the case where the single truncated unary codeword corresponds to a bin string composed of multiple bins, the context values corresponding to different bins among the multiple bins are different from each other.
7. A computer program product comprising a computer program, the computer program including instructions that, when executed by a computer, cause the computer to perform a method for decoding transform blocks in an image from a bitstream, the method comprising: Decode at least a single truncated unary codeword from the bit stream; By using the single truncated unary codeword, the horizontal and vertical transform types are determined for the transform block, wherein different transform types can be determined for the horizontal and vertical transform types by using the single truncated unary codeword; and The transform block is decoded by applying the horizontal transform type and the vertical transform type to the transform block. Wherein, the single truncated unary codeword represents a value used to specify the horizontal transform type and the vertical transform type, and In the case where the single truncated unary codeword corresponds to a bin string composed of multiple bins, the context values corresponding to different bins among the multiple bins are different from each other.
8. A computer program product comprising a computer program, the computer program including instructions that, when executed by a computer, cause the computer to perform a method for encoding transform blocks in an image into a bitstream, the method comprising: Determine the horizontal transformation type for the transformation block; Determine the vertical transformation type for the transformation block; The transform block is encoded by applying the horizontal transform type and the vertical transform type to the transform block; and Encode at least a single truncated unary codeword. In the method for decoding the transform block, the horizontal transform type and the vertical transform type are determined for the transform block using the single truncated unary codeword. Specifically, by using the single truncated unary codeword, different transformation types can be determined for the horizontal transformation type and the vertical transformation type in the method of decoding the transform block. Wherein, the single truncated unary codeword represents a value used to specify the horizontal transform type and the vertical transform type, and In the case where the single truncated unary codeword corresponds to a bin string composed of multiple bins, the context values corresponding to different bins among the multiple bins are different from each other.
Citation Information
Patent Citations
Image processing device and method
WO2017195476A1
Image encoding / decoding method and device, and recording medium in which bitstream is stored
WO2018016823A1
