Image decoding apparatus and method, image encoding apparatus and method, and storage medium

Abstract

This invention relates to an image decoding apparatus and method, an image encoding apparatus and method, and a storage medium. Additionally, a system and method for decoding encoded units in an image frame from a bitstream are disclosed. The method includes: determining the size of the encoded unit from the bitstream; and dividing the image frame into multiple equal-sized processing regions, each equal-sized processing region being smaller than the maximum available encoded unit size. The method further includes: selecting a motion vector corresponding to the encoded unit from a list of candidate motion vectors, the selection of the motion vector including: (i) decoding a merge index if the encoded unit is greater than or equal to the size of one of the determined processing regions, or (ii) decoding a skip flag to decode the merge index if the encoded unit is not greater than or equal to the size of one of the determined processing regions; and decoding the encoded unit according to the selected motion vector for the encoded unit.

Description

[0001] (This application is a divisional application of the application filed on June 25, 2019, with application number 2019800544976, entitled "Method, apparatus and system for transform block encoding and decoding of video samples".) Technical Field

[0002] This invention generally relates to digital video signal processing, and more particularly to methods, apparatus, and systems for encoding and decoding transform blocks of video samples. The invention also relates to a computer program product comprising a computer-readable medium containing computer programs for encoding and decoding transform blocks of video samples. Background Technology

[0003] Currently, there are numerous video coding applications, including those for transmitting and storing video data. Many video coding standards have been developed, and others are currently under development. Recent advancements in video coding standardization have led to the formation of a group known as the Joint Video Experts Group (JVET). The JVET includes members of Study Group 16, Problem 6 (SG16 / Q6) of the Telecommunication Standardization Sector (ITU-T) of the International Telecommunication Union (ITU), also known as the Video Coding Experts Group (VCEG); and members of the Joint Technical Committee 1 / Subcommittee 29 / Working Group 11 (ISO / IEC JTC1 / SC29 / WG11), also known as the Moving Picture Experts Group (MPEG).

[0004] The Joint Video Experts Group (JVET) issued a Call for Proposals (CfP) and analyzed the responses at its 10th meeting in San Diego, USA. The submitted responses demonstrated video compression capabilities significantly superior to those of the current state-of-the-art video compression standard, namely, "High Efficiency Video Coding" (HEVC). Based on this superior performance, a project was initiated to develop a new video compression standard named "Various Video Coding" (VVC). VVC is expected to address the ongoing demand for even higher compression performance, particularly as video format capabilities increase (e.g., higher resolution and higher frame rates), and the growing market demand for services delivered over WANs (where bandwidth costs are relatively high). Simultaneously, VVC must be implementable in modern silicon processes and offer an acceptable trade-off between achieved performance and implementation costs (e.g., in terms of silicon area, CPU processor load, memory utilization, and bandwidth).

[0005] Video data consists of a sequence of frames, each containing one or more color channels. Typically, one primary color channel and two secondary color channels are required. The primary color channel is often referred to as the "luminance" channel, and the secondary color channels(s) are often referred to as the "chrominance" channels. Although video data is typically displayed in the RGB (Red-Green-Blue) color space, this color space is highly correlated between its three corresponding components. The video data representation seen by the encoder or decoder typically uses a color space such as YCbCr. YCbCr concentrates luminance (mapped to "luminance" according to a transformation equation) in the Y (primary) channel and chrominance in the Cb and Cr (secondary) channels. Furthermore, the Cb and Cr channels can be spatially sampled at a lower rate than the luminance channel (e.g., half horizontally and half vertically (referred to as the "4:2:0 chrominance format")).

[0006] The VVC standard is a "block-based" codec, in which frames are first divided into an array of square regions called "Code Tree Units" (CTUs). CTUs typically occupy relatively large areas, such as 128 × 128 luma samples. However, the areas of the CTUs at the right and bottom edges of each frame may be smaller. Associated with each CTU is a "code tree," which defines the set of regions into which the CTU's area is divided, also known as "Code Units" (CUs). CUs are processed in a specific order for encoding or decoding. Due to the use of the code tree and the 4:2:0 chroma format, a given region in a frame is associated with a set of co-located blocks across color channels. The luma block is sized as width × height, while for each chroma block, the chroma block is sized as width / 2 × height / 2. The set of co-located blocks for a given region is often called a "unit," such as the CUs mentioned above, as well as "prediction units" (PUs) and "transformation units" (TUs).

[0007] Although chroma blocks and luma blocks differ in size for the same area, the size of a given "unit" is generally described in terms of the size of a luma block per unit. Individual blocks are typically identified by the type of unit associated with them. For example, a "coded block" (CB), a "transform block" (TB), and a prediction block (PB) are blocks for a single color channel and are associated with CU, TU, and PU, respectively. Despite the aforementioned distinction between "unit" and "block," the term "block" can be used as a general term for an area or region of a frame where operations are applied to all color channels.

[0008] For each CU, a prediction (PU) (“prediction unit”) is generated for the content (sample values) of the corresponding region of the generated frame data. Furthermore, a representation of the difference (or “residual”) between the prediction seen at the encoder input and the region content is formed. The differences between the color channels can be transformed and encoded into a sequence of residual coefficients, thus forming one or more TUs for a given CU. The applied transform can be a Discrete Cosine Transform (DCT) or other transform applied to individual blocks of residual values. The main transform is applied separately, i.e., the two-dimensional transform is performed in two passes. First, the block is transformed by applying a one-dimensional transform to each row of samples in the block. Then, a portion of the result is transformed by applying a one-dimensional transform to each column of the partial result to produce a final block of transform coefficients that are essentially decorrelated on the residual samples. The VVC standard supports transforms of various sizes, including transforms of rectangular blocks (each side being a power of 2). The transform coefficients are quantized for entropy encoding in the bitstream.

[0009] Implementations of the VVC standard typically use a pipeline to divide processing into a sequence of stages. The stages operate synchronously, and partially processed blocks are passed from one stage to the next before a fully processed (i.e., encoded or decoded) block is output. Efficient processing of transformed blocks within the context of the pipelined architecture is necessary to avoid excessive implementation costs for the VVC standard. This is due to both high memory consumption and the number of functional modules required to handle the "worst-case" scenario, as well as the speed required for pipeline stages to complete and the size of the data processed by each stage. Summary of the Invention

[0010] The object of the present invention is to substantially overcome or at least improve one or more disadvantages of the existing arrangement.

[0011] According to one aspect of the present invention, a method is provided for decoding encoded units in an image frame from a bitstream, the method comprising:

[0012] The size of the encoding unit is determined from the bit stream;

[0013] The image frame is divided into multiple processing regions of equal size, each of which is a block processed during a single stage of the pipeline for decoding the bitstream.

[0014] In the case where the boundary between the encoding unit and the determined processing region overlaps, a transformation size for the encoding unit is selected from a plurality of transformation sizes. This transformation size is chosen to fit within the encoding unit and is different in size from the processing region.

[0015] The encoding unit is decoded by applying an inverse transform to the residual coefficients of each transform unit in the encoding unit, each transform unit having a selected transform size.

[0016] According to another aspect of the present invention, a non-transitory computer-readable medium is provided, on which a computer program is stored to implement a method for decoding encoded units in an image frame from a bitstream, said program comprising:

[0017] Code used to determine the size of the encoded unit from the bit stream;

[0018] Code for dividing the image frame into multiple processing regions of equal size, each processing region being a block processed during a single stage of a pipeline for decoding the bitstream;

[0019] Code for selecting a transformation size from multiple transformation sizes for the encoding unit when the boundary between the encoding unit and the determined processing region overlaps, the transformation size being selected to fit within the encoding unit and having a different size than the processing region; and

[0020] Code for decoding the coded unit by applying an inverse transform to the residual coefficients of each transform unit in the coded unit, each transform unit having a selected transform size.

[0021] According to another aspect of the present invention, a system is provided, comprising:

[0022] Memory; and

[0023] A processor, wherein the processor is configured to execute code stored in the memory to implement a method for decoding encoded units in an image frame from a bitstream, the method comprising:

[0024] The size of the encoding unit is determined from the bit stream;

[0025] The image frame is divided into multiple processing regions of equal size, each of which is a block processed during a single stage of the pipeline for decoding the bitstream.

[0026] In the case where the boundary between the encoding unit and the determined processing region overlaps, a transformation size for the encoding unit is selected from a plurality of transformation sizes. This transformation size is chosen to fit within the encoding unit and is different in size from the processing region.

[0027] The encoding unit is decoded by applying an inverse transform to the residual coefficients of each transform unit in the encoding unit, each transform unit having a selected transform size.

[0028] According to another aspect of the present invention, a video decoder is provided, which is configured to:

[0029] Receive image frames from a bit stream;

[0030] The size of the encoding unit is determined from the bit stream;

[0031] The image frame is divided into multiple processing regions of equal size, each of which is a block processed during a single stage of the pipeline for decoding the bitstream.

[0032] In the case where the boundary between the encoding unit and the determined processing region overlaps, a transformation size for the encoding unit is selected from a plurality of transformation sizes. This transformation size is chosen to fit within the encoding unit and is different in size from the processing region.

[0033] The encoding unit is decoded by applying an inverse transform to the residual coefficients of each transform unit in the encoding unit, each transform unit having a selected transform size.

[0034] According to another aspect of the present invention, a method for decoding encoded units in an image frame from a bitstream is provided, the method comprising:

[0035] The size of the encoding unit is determined from the bit stream;

[0036] The image frame is divided into multiple processing regions of equal size, each of which is smaller than the maximum available encoding unit size.

[0037] Selecting a motion vector from the candidate motion vector list corresponding to the encoded unit, the selection of the motion vector includes: (i) decoding the merge index if the encoded unit is greater than or equal to the size of one of the determined processing regions, or (ii) decoding the skip flag to decode the merge index if the encoded unit is not greater than or equal to the size of one of the determined processing regions; and

[0038] The encoded unit is decoded based on the selected motion vector for the encoded unit.

[0039] According to another aspect of the present invention, a non-transitory computer-readable medium is provided, on which a computer program is stored to implement a method for decoding encoded units in an image frame from a bitstream, the program comprising:

[0040] Code used to determine the size of the encoded unit from the bit stream;

[0041] Code for dividing the image frame into multiple processing regions of equal size, each processing region being smaller than the maximum available encoding unit size;

[0042] Code for selecting a motion vector corresponding to the coded unit from a list of candidate motion vectors, the selection of the motion vector comprising: (i) decoding a merge index if the coded unit is greater than or equal to the size of one of the determined processing regions, or (ii) decoding a skip flag to decode the merge index if the coded unit is not greater than or equal to the size of one of the determined processing regions; and

[0043] Code for decoding the encoded unit based on a selected motion vector for the encoded unit.

[0044] According to another aspect of the present invention, a system is provided, comprising:

[0045] Memory; and

[0046] A processor, wherein the processor is configured to execute code stored on the memory to implement a method for decoding encoded units in an image frame from a bitstream, the method comprising:

[0047] The size of the encoding unit is determined from the bit stream;

[0048] The image frame is divided into multiple processing regions of equal size, each of which is smaller than the maximum available encoding unit size.

[0049] Selecting a motion vector from the candidate motion vector list corresponding to the encoded unit, the selection of the motion vector includes: (i) decoding the merge index if the encoded unit is greater than or equal to the size of one of the determined processing regions, or (ii) decoding the skip flag to decode the merge index if the encoded unit is not greater than or equal to the size of one of the determined processing regions; and

[0050] The encoded unit is decoded based on the selected motion vector for the encoded unit.

[0051] According to another aspect of the present invention, a video decoder is provided, which is configured to:

[0052] Receive image frames from the bit stream;

[0053] The size of the encoding unit is determined from the bit stream;

[0054] The image frame is divided into multiple processing regions of equal size, each of which is smaller than the maximum available encoding unit size.

[0055] Selecting a motion vector from the candidate motion vector list corresponding to the encoded unit, the selection of the motion vector includes: (i) decoding the merge index if the encoded unit is greater than or equal to the size of one of the determined processing regions, or (ii) decoding the skip flag to decode the merge index if the encoded unit is not greater than or equal to the size of one of the determined processing regions; and

[0056] The encoded unit is decoded based on the selected motion vector for the encoded unit.

[0057] Other aspects were also disclosed. Attached Figure Description

[0058] At least one embodiment of the invention will now be described with reference to the following figures and appendices, wherein:

[0059] Figure 1 This is a schematic block diagram illustrating a video encoding and decoding system;

[0060] Figure 2A and 2B Composition can be put into practice Figure 1 A schematic block diagram of a general computer system that includes one or both of the video encoding and decoding systems;

[0061] Figure 3 This is a schematic block diagram showing the functional modules of a video encoder;

[0062] Figure 4 This is a schematic block diagram illustrating the functional modules of the video decoder;

[0063] Figure 5 This is a schematic block diagram illustrating the available segmentation of a block into one or more blocks in the tree structure of general video coding;

[0064] Figure 6 This is a schematic diagram illustrating the permissive segmentation of a data stream from one block to one or more blocks within the tree structure of general video coding;

[0065] Figure 7A and 7B This illustrates an example of splitting a coding tree unit (CTU) into multiple coding units (CUs);

[0066] Figure 8A This shows an example sequence of coded tree units (CTUs) processed according to a pipelined architecture;

[0067] Figure 8B This shows an example "random access" image group structure for frames in a video;

[0068] Figure 9 This is a diagram showing the transformation size of the VVC standard;

[0069] Figure 10A This is a diagram showing a coding unit with a ternary split at the top level of the coding tree;

[0070] Figure 10B It is shown that... Figure 10A A graph of the coded tree associated with the alternative transformation units;

[0071] Figure 10C It is a diagram showing the transformation unit associated with a coding tree having two ternary splits in opposite directions;

[0072] Figure 10D This is a diagram showing the transformation units associated with a coding tree having vertical ternary splits, horizontal binary splits, and vertical ternary splits;

[0073] Figure 10E This is a diagram showing the transformation unit associated with a coding tree having two vertical ternary splits;

[0074] Figure 10F It is shown Figure 10E An alternative graph of the transform unit associated with a coding tree having two vertical ternary splits;

[0075] Figure 11 It is a flowchart of a method for determining the prediction pattern of a coding unit in a coding tree;

[0076] Figure 12 This is a flowchart of a method for encoding coding units using transforms, which enables the pipelined implementation of a video encoder; and

[0077] Figure 13 This is a flowchart of a method for decoding encoded units using a transform, the size of which is based on... Figure 12 The method was chosen. Detailed Implementation

[0078] Where steps and / or features with the same reference numerals are referenced in any one or more of the accompanying drawings, unless the contrary is intended, these steps and / or features have the same function(s) or operation(s) for the purposes of this specification.

[0079] Figure 1This is a schematic block diagram illustrating the functional modules of a video encoding and decoding system 100. System 100 can implicitly divide a large block or coding unit (CU) into multiple smaller blocks or transform units (TUs) to enable the processing of coding tree units (CTUs) in regions smaller than the CTU size (or "pipeline processing regions"). For example, system 100 can process a CTU into four quadrants, each quadrant potentially containing many CUs and / or a portion of CUs spanning multiple regions.

[0080] System 100 includes a source device 110 and a destination device 130. A communication channel 120 is used to communicate encoded video information from the source device 110 to the destination device 130. In some configurations, one or both of the source device 110 and the destination device 130 may each include a mobile phone or "smartphone," in which case the communication channel 120 is a wireless channel. In other configurations, the source device 110 and the destination device 130 may include video conferencing equipment, in which case the communication channel 120 is typically a wired channel such as an Internet connection. Furthermore, the source device 110 and the destination device 130 may include a wide range of arbitrary devices, including those supporting over-the-air television broadcasting, cable television applications, Internet video applications (including streaming), and applications that capture encoded video data on some computer-readable storage medium (such as a hard disk drive in a file server).

[0081] like Figure 1 As shown, source device 110 includes a video source 112, a video encoder 114, and a transmitter 116. Video source 112 typically includes a source of captured video frame data (denoted as 113), such as a camera sensor, a previously captured video sequence stored on a non-transitory recording medium, or a video feed from a remote camera sensor. Video source 112 can also be the output of a computer graphics card (e.g., video output from a display operating system and various applications running on a computing device (e.g., a tablet computer)). Examples of source devices 110 that may include a camera sensor as video source 112 include smartphones, video camcorders, professional cameras, and webcams.

[0082] Video encoder 114 converts (or “encodes”) the captured frame data (indicated by arrow 113) from video source 112 into a bitstream (indicated by arrow 115). Bitstream 115 is transmitted by transmitter 116 via communication channel 120 as encoded video data (or “encoded video information”). Bitstream 115 may also be stored in a non-transitory storage device 122 such as flash memory or hard disk drive until subsequently transmitted via communication channel 120 or as a replacement for transmission via communication channel 120.

[0083] Destination device 130 includes a receiver 132, a video decoder 134, and a display device 136. Receiver 132 receives encoded video data from communication channel 120 and transmits the received video data as a bitstream (indicated by arrow 133) to video decoder 134. Video decoder 134 then outputs decoded frame data (indicated by arrow 135) to display device 136. Examples of display device 136 include cathode ray tubes, liquid crystal displays (such as in smartphones, tablets, computer monitors, or standalone televisions). Alternatively, the functions of source device 110 and destination device 130 can be embodied in a single device, examples of which include mobile phones and tablets.

[0084] Although the example apparatus has been described above, the source apparatus 110 and the destination apparatus 130 can each typically be configured within a general-purpose computer system via a combination of hardware and software components. Figure 2A This computer system 200 is illustrated and includes: a computer module 201; input devices such as a keyboard 202, a mouse pointer device 203, a scanner 226, a camera 227 configured as a video source 112, and a microphone 280; and output devices including a printer 215, a display device 214 configured as a display device 136, and a speaker 217. The computer module 201 can communicate with a communication network 220 via a connection 221 using an external modulator-demodulator (modem) transceiver device 216. The communication network 220, which may represent the communication channel 120, can be a wide area network (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. If the connection 221 is a telephone line, the modem 216 can be a conventional dial-up modem. Alternatively, if the connection 221 is a high-capacity (e.g., cable or optical) connection, the modem 216 can be a broadband modem. A wireless modem can also be used for wireless connection to the communication network 220. The transceiver device 216 can provide the functions of a transmitter 116 and a receiver 132, and the communication channel 120 can be embodied in the wiring 221.

[0085] Computer module 201 typically includes at least one processor unit 205 and a memory unit 206. For example, memory unit 206 may have semiconductor random access memory (RAM) and semiconductor read-only memory (ROM). Computer module 201 also includes multiple input / output (I / O) interfaces, including: an audio-video interface 207 connected to a video display 214, speakers 217, and microphone 280; an I / O interface 213 connected to a keyboard 202, mouse 203, scanner 226, camera 227, and optionally a joystick or other human-machine interface device (not shown); and an interface 208 for external modem 216 and printer 215. Signals from audio-video interface 207 to computer monitor 214 are typically output from the computer's graphics card. In some implementations, modem 216 may be integrated within computer module 201, for example, within interface 208. Computer module 201 also has a local network interface 211, which allows computer system 200 to connect via cable 223 to a local area network (LAN) known as a local area network (LAN). Figure 2A As shown, the local area communication network 222 can also be connected to the wide area network 220 via wiring 224, wherein the local area communication network 222 typically includes a so-called "firewall" device or a device with similar functionality. The local network interface 211 may include an Ethernet™ circuit card, Bluetooth™ wireless configuration, or IEEE 802.11 wireless configuration; however, various other types of interfaces can be implemented for interface 211. The local network interface 211 can also provide the functionality of a transmitter 116 and a receiver 132, and the communication channel 120 can also be embodied in the local area communication network 222.

[0086] I / O interfaces 208 and 213 can provide either serial or parallel connections, with the former typically implemented according to the Universal Serial Bus (USB) standard and having a corresponding USB connector (not shown). A storage device 209 is provided, and storage device 209 typically includes a hard disk drive (HDD) 210. Other storage devices (not shown), such as floppy disk drives and tape drives, may also be used. An optical disk drive 212 is typically provided as a non-volatile source of data. Portable storage devices such as optical discs (e.g., CD-ROM, DVD, Blu-ray Disc™), USB-RAM, portable external hard disk drives, and floppy disks may be used as appropriate sources of data for computer system 200. Typically, any of HDD 210, optical disk drive 212, network 220, and 222 may also be configured to operate as a video source 112 or as a destination for decoded video data to be stored for reproduction via display 214. The source device 110 and destination device 130 of system 100 may be embodied in computer system 200.

[0087] Components 205-213 of computer module 201 typically communicate via interconnect bus 204 and in a manner known to those skilled in the art to operate computer systems 200 in a conventional mode of operation. For example, processor 205 is connected to system bus 204 via wiring 218. Similarly, memory 206 and optical disc drive 212 are connected to system bus 204 via wiring 219. Examples of computers in which this configuration may be practiced include IBM-PC and compatible machines, Sun SPARCstation, Apple Mac™, or similar computer systems.

[0088] Where appropriate or desired, the video encoder 114 and the video decoder 134, as well as the methods described below, can be implemented using the computer system 200. Specifically, the video encoder 114, the video decoder 134, and the methods to be described can be implemented as one or more software applications 233 executable within the computer system 200. In particular, instructions 231 (refer to...) executed within the computer system 200 in the software 233 are utilized. Figure 2B The software instructions 231 can be configured as one or more code modules, each for performing one or more specific tasks. Alternatively, the software can be divided into two separate parts, with a first part and corresponding code modules performing the method, and a second part and corresponding code modules managing the user interface between the first part and the user.

[0089] For example, the software can be stored in a computer-readable medium including the storage means described below. The software is loaded from the computer-readable medium into computer system 200 and then executed by computer system 200. The computer-readable medium having such software, or the computer program recorded on such computer-readable medium, is a computer program product. Using the computer program product in computer system 200 preferably implements advantageous means for carrying out the video encoder 114, video decoder 134, and the methods.

[0090] Software 233 is typically stored in HDD 210 or memory 206. The software is loaded from a computer-readable medium into computer system 200 and executed by computer system 200. Thus, for example, software 233 may be stored on an optically readable disk storage medium (e.g., CD-ROM) 225 read by optical disk drive 212.

[0091] In some instances, application 233 is supplied to the user in a manner encoded on one or more CD-ROMs 225 and read via a corresponding drive 212, or alternatively, application 233 may be read by the user from network 220 or 222. Furthermore, software may also be loaded into computer system 200 from other computer-readable media. Computer-readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to computer system 200 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray Disc™, hard disk drives, ROMs or integrated circuits, USB storage devices, magneto-optical disks, or computer-readable cards such as PCMCIA cards, regardless of whether these devices are internal or external to computer module 201. Examples of temporary or intangible computer-readable transmission media that may also be used to provide software, applications, instructions and / or video data or encoded video data to computer module 401 include: radio or infrared transmission channels and network connections to other computers or networked devices, as well as the Internet or intranet including email sending and information recorded on websites.

[0092] The second part of the aforementioned application 233 and the corresponding code modules can be executed to implement one or more graphical user interfaces (GUIs) to be drawn or otherwise presented on the display 214. By typically operating the keyboard 202 and mouse 203, users and applications of the computer system 200 can operate the interface in a functionally applicable manner to provide control commands and / or input to applications associated with these GUIs(s). Other functionally applicable forms of user interfaces can also be implemented, such as audio interfaces utilizing voice prompts output via speaker 217 and user voice commands input via microphone 280.

[0093] Figure 2B This is a detailed schematic block diagram of processor 205 and "memory" 234. Memory 234 represents... Figure 2A The computer module 201 can access the logical aggregation of all memory modules (including HDD 209 and semiconductor memory 206).

[0094] When the computer module 201 is initially powered on, the power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in... Figure 2A The ROM 249 of the semiconductor memory 206 is sometimes referred to as firmware. The POST program 250 checks the hardware within the computer module 201 to ensure proper operation, and typically checks the processor 205, the memory 234 (209, 206), and the Basic Input / Output System Software (BIOS) module 251, which is also typically stored in the ROM 249, for correct operation. Once the POST program 250 has successfully run, the BIOS 251 boots. Figure 2A The hard disk drive 210 is booted. Booting the hard disk drive 210 causes the bootloader 252 residing on the hard disk drive 210 to be executed via the processor 205. This loads the operating system 253 into the RAM memory 206, where the operating system 253 begins operation. The operating system 253 is a system-level application executable by the processor 205 to implement various advanced functions, including processor management, memory management, device management, storage management, software application interfaces, and a general user interface.

[0095] The operating system 253 manages memory 234 (209, 206) to ensure that each process or application running on computer module 201 has sufficient memory to execute without conflicting with memory allocated to other processes. Furthermore, appropriate use of memory is essential. Figure 2AThe computer system 200 contains different types of memory so that each process can run efficiently. Therefore, aggregate memory 234 is not intended to illustrate how specific segments of memory are allocated (unless otherwise stated), but rather to provide an overview of the memory accessible to the computer system 200 and how that memory is used.

[0096] like Figure 2B As shown, processor 205 includes multiple functional modules, including a control unit 239, an arithmetic logic unit (ALU) 240, and local or internal memory 248, sometimes referred to as cache memory. Cache memory 248 typically includes multiple storage registers 244-246 in a register segment. One or more internal buses 241 functionally interconnect these functional modules. Processor 205 also typically has one or more interfaces 242 for communicating with external devices via system bus 204 using wiring 218. Memory 234 is connected to bus 204 using wiring 219.

[0097] Application program 233 includes a sequence of instructions 231, which may contain conditional branch instructions and loop instructions. Program 233 may also include data 232 used when executing program 233. Instructions 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative size of instructions 231 and memory locations 228-230, as described by the instructions shown in memory location 230, a particular instruction may be stored in a single memory location. Alternatively, as described by the instruction segments shown in memory locations 228 and 229, an instruction may be divided into multiple parts, each stored in a separate memory location.

[0098] Typically, a set of instructions is assigned to processor 205, which is then executed within processor 205. Processor 205 waits for subsequent input, which it responds to by executing another set of instructions. Each input can be provided from one or more of a plurality of sources, including data generated by one or more of input devices 202, 203, data received from an external source via one of networks 220, 202, data retrieved from one of storage devices 206, 209, or data retrieved from storage medium 225 inserted into the corresponding reader 212 (all of which are within...). Figure 2A (As shown in the diagram). Executing a set of instructions may result in output data in some cases. Execution may also involve storing data or variables into memory 234.

[0099] The video encoder 114, video decoder 134, and the method can use input variables 254 stored in corresponding memory locations 255, 256, and 257 within memory 234. The video encoder 114, video decoder 134, and the method generate output variables 261 stored in corresponding memory locations 262, 263, and 264 within memory 234. Intermediate variables 258 can be stored in memory locations 259, 260, 266, and 267.

[0100] refer to Figure 2B The processor 205, registers 244, 245, 246, arithmetic logic unit (ALU) 240, and control unit 239 work together to perform micro-operation sequences required for "fetch, decode, and execute" cycles of instructions in the instruction set constituting program 233. Each fetch, decode, and execute cycle includes:

[0101] (a) Fetch operation, used to fetch or read instruction 231 from memory locations 228, 229, 230;

[0102] (b) Decoding operation, wherein the control unit 239 determines which instruction was extracted; and

[0103] (c) Execute an operation, wherein the control unit 239 and / or ALU 240 execute the instruction.

[0104] Then, a further fetch, decode, and execute cycle for the next instruction can be performed. Similarly, a storage cycle can be performed, in which the control unit 239 stores or writes a value to or writes it to memory location 232.

[0105] To explain Figure 12 and Figure 13 Each step or subprocess in the method is associated with one or more segments of program 233 and is typically performed by working together in the register units 244, 245, 247, ALU 240 and control unit 239 in processor 205 to fetch, decode and execute cycles of instructions in the instruction set of the segment of program 233.

[0106] Figure 3 This is a schematic block diagram showing the functional modules of the video encoder 114. Figure 4 This is a schematic block diagram illustrating the functional modules of the video decoder 134. Typically, data is transferred between the functional modules within the video encoder 114 and the video decoder 134 in groups of samples or coefficients (such as the partitioning of blocks into fixed-size sub-blocks) or as arrays. Figure 2A and 2BAs shown, the video encoder 114 and video decoder 134 can be implemented using a general-purpose computer system 200, wherein various functional modules can be implemented using dedicated hardware within the computer system 200 and software executable within the computer system 200 (such as one or more software code modules of a software application 233 residing on the hard disk drive 205 and executed under the control of the processor 205). Alternatively, the video encoder 114 and video decoder 134 can be implemented using a combination of dedicated hardware and software executable within the computer system 200. Alternatively, the video encoder 114, video decoder 134, and the method can be implemented in dedicated hardware such as one or more integrated circuits that perform the functions or sub-functions of the method. Such dedicated hardware may include a graphics processing unit (GPU), a digital signal processor (DSP), a dedicated standard product (ASSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or one or more microprocessors and associated memory. Specifically, the video encoder 114 includes modules 310-386, and the video decoder 134 includes modules 420-496, wherein each of these modules can be implemented as one or more software code modules of the software application 233.

[0107] although Figure 3 The video encoder 114 is an example of a Universal Video Coding (VVC) video coding pipeline, but other video codecs can also be used for the processing stages described herein. The video encoder 114 receives captured frame data 113, such as a series of frames (each frame including one or more color channels). The block partitioner 310 first partitions the frame data 113 into CTUs, which are typically square in shape and configured to use a specific size. For example, the size of a CTU can be 64×64, 128×128, or 256×256 luminance samples. The block partitioner 310 further partitions each CTU into one or more CUs, where CUs have various sizes that can include both square and non-square aspect ratios. However, in the VVC standard, CUs, PUs, and TUs always have side lengths that are powers of 2. Therefore, the current CU (denoted as 312) is output from the block partitioner 310, progressing according to iterations of one or more blocks of CTUs, based on the coding tree of the CTUs. The following references... Figure 5 and 6 To further explain the options for partitioning a CTU as a CU.

[0108] The CTUs obtained from the first segmentation of frame data 113 may be scanned in raster scan order and may be grouped into one or more "slices". A slice may be an "intra-frame" (or "I") slice. An intra-frame slice (I slice) indicates that each CU in the slice is intra-predicted. Alternatively, a slice may be single-predicted or double-predicted ("P" or "B" slices, respectively), indicating the additional availability of single and double prediction in the slice. Since frame data 113 typically includes multiple color channels, CTUs and CUs are associated with samples from all color channels overlapping the block region defined by the operation of block partitioner 310. Each CU for each color channel of frame data 113 includes a coded block (CB). Because the sampling rates of the chroma channels and luma channels may potentially differ, the size of the CB for the chroma channel may differ from the size of the CB for the luma channel. When using a 4:2:0 chroma format, the size of the CB for the chroma channel of the CU is half the width and half the height of the CB for the luma channel of the CU.

[0109] For each CTU, the video encoder 114 operates in two phases. In the first phase (referred to as the “search” phase), the block partitioner 310 tests various potential configurations of the coding tree. Each potential configuration of the coding tree has associated “candidate” CUs. The first phase involves testing various candidate CUs to select the CU that provides high compression efficiency and low distortion. This testing typically involves Lagrangian optimization, thereby evaluating the candidate CUs based on a weighted combination of rate (coding cost) and distortion (error with respect to the input frame data 113). The “best” candidate CU (the CU with the lowest rate / distortion) is selected for subsequent encoding in the bitstream 115. The evaluation of the candidate CUs includes options such as using the CU for a given region, or splitting the region according to various splitting options and utilizing other CUs to encode the individual smaller resulting regions or further splitting the region. As a result, both the CU and the coding tree itself are selected in the search phase.

[0110] Video encoder 114 generates a prediction unit (PU) indicated by arrow 320 for each CU (e.g., CU 312). PU 320 is a prediction of the content of the associated CU 312. Subtractor module 322 generates a difference (or “residual”, which refers to the difference in the spatial domain) between PU 320 and CU 312, represented as 324. Difference 324 is the block size difference between the corresponding samples in PU 320 and CU 312. Difference 324 is transformed, quantized, and represented as a transform unit (TU) indicated by arrow 336. PU 320 and the associated TU 336 are typically selected as the “best” CU among several possible candidate CUs.

[0111] A candidate coding unit (CU) is a CU obtained from one of the prediction modes available in the video encoder 114 for the associated PU and the resulting residual. Each candidate CU yields one or more corresponding TUs, as described below with reference to Figures 10-12. TU 336 is a quantized and transformed representation of the difference 324. When combined with the prediction PU in the decoder 114, TU 336 reduces the difference between the decoded CU and the original CU 312 at the cost of additional signaling in the bitstream.

[0112] Therefore, each candidate coding unit (CU) (i.e., a combination of a prediction unit (PU) and a transform unit (TU)) has an associated coding cost (or "rate") and an associated difference (or "distortion"). The rate is typically measured in bits. The distortion of the CU is typically estimated as the difference in sample values, such as the sum of absolute differences (SAD) or the sum of squared differences (SSD). The mode selector 386 uses the difference 324 to determine the estimate obtained from each candidate PU to determine the intra-frame prediction mode (indicated by arrow 388). The coding cost associated with each candidate prediction mode and the corresponding residual coding can be estimated at a significantly lower cost compared to entropy coding of the residuals. Therefore, multiple candidate modes can be evaluated to determine the optimal mode in a rate-distortion sense.

[0113] Determining the optimal mode is typically achieved using a variation of Lagrange optimization. Selecting an intra-prediction mode 388 generally involves determining the coding cost of the residual data obtained from applying a specific intra-prediction mode. The coding cost can be approximated using the Absolute Transform Difference (SATD), thereby obtaining an estimated transform residual cost using a relatively simple transform such as the Hadamard transform. In some implementations using relatively simple transforms, the cost obtained from the simplified estimation method is monotonically correlated with the actual cost otherwise determined from a full evaluation. In implementations with monotonically correlated estimated costs, this simplified estimation method can be used to make the same decision (i.e., the intra-prediction mode), where the complexity of the video encoder 114 is reduced. To allow for possible non-monotonicity in the relationship between the estimated cost and the actual cost, this simplified estimation method can be used to generate a list of optimal candidates. For example, non-monotonicity can be derived from other mode decisions that can be used to encode the residual data. The list of optimal candidates can have any number. A more complete search can be performed using the optimal candidates to establish the optimal mode selection for encoding the residual data for each candidate, thereby allowing for the final selection of the intra-prediction mode as well as other mode decisions.

[0114] Other modality decisions include the ability to skip forward transforms (known as "transform skipping"). Transform skipping is suitable for residual data lacking sufficient relevance to reduce encoding costs via expressions that serve as the transform's underlying function. Certain types of content, such as relatively simple computer-generated graphics, can exhibit similar behavior. With "skipped transforms," ​​the residual coefficients are still encoded even if the transform itself is not performed.

[0115] Lagrange or similar optimization processes can be used for both the selection of the CTU to CU partition (using block partitioner 310) and the selection of the best prediction mode from multiple possibilities. By applying Lagrange optimization of candidate modes to the mode selector module 386, the intra-prediction mode with the lowest cost metric is selected as the best mode. The best mode is the selected intra-prediction mode 388, which is also encoded in bitstream 115 by entropy encoder 338. The selection of intra-prediction mode 388 through the operation of mode selector module 386 is extended to the operation of block partitioner 310. For example, the candidates for selecting intra-prediction mode 388 can include modes applicable to a given block and additional modes applicable to multiple smaller blocks that are co-located with the given block. In the case of including modes applicable to the given block and smaller co-located blocks, the candidate selection process is implicitly also a process for determining the optimal hierarchical decomposition of the CTU to CU.

[0116] In the second phase of the operation of the video encoder 114 (referred to as the “encoding” phase), the selected coding tree (and thus each selected CU) is iterated within the video encoder 114. As further described here, during the iteration, the CUs are encoded in the bit stream 115.

[0117] The entropy encoder 338 supports both variable-length encoding and arithmetic encoding of syntactic elements. Context-adaptive binary arithmetic encoding is used to support arithmetic encoding. Arithmetic-encoded syntactic elements consist of a sequence of one or more "bins" (binary files). Like bits, bins have values ​​of "0" or "1". However, bins are not encoded as discrete bits in the bitstream 115. A bin has an associated prediction (or "possible" or "maximum probability") value and an associated probability (called "context"). The "maximum probability symbol" (MPS) is encoded when the actual bin to be encoded matches the predicted value. Encoding the maximum probability symbol is relatively inexpensive in terms of bit consumption. The "minimum probability symbol" (LPS) is encoded when the actual bin to be encoded does not match the possible value. Encoding the minimum probability symbol is relatively expensive in terms of bit consumption. Bin encoding techniques enable efficient encoding of bins that can skew the probabilities of "0" vs. "1". For syntactic elements with two possible values ​​(i.e., "flag"), a single bin is sufficient. For syntactic elements with many possible values, a sequence of bins is required.

[0118] The presence of a later bin in a sequence can be determined based on the value of an earlier bin. Furthermore, each bin can be associated with more than one context. A specific context can be selected based on the earlier bin in a syntactic element and the bin values ​​of adjacent syntactic elements (i.e., bin values ​​from adjacent blocks). Each time a context-encoded bin is encoded, the context selected for that bin (if any) is updated in a way that reflects the new bin value. Thus, the binary arithmetic coding scheme is considered adaptive.

[0119] Video encoder 114 also supports bins lacking context (“bypass bins”). Bypass bins are encoded using an equal probability distribution between “0” and “1”. Thus, each bin occupies one bit in bitstream 115. The absence of context saves memory and reduces complexity, thus using bypass bins with an unskewed distribution of values ​​for a particular bin. An example of a context-adaptive entropy encoder is known in the art as CABAC (Context-Adaptive Binary Arithmetic Encoder), and many variations of this encoder are used in video coding.

[0120] The entropy encoder 338 uses a combination of context-coded and bypass-coded bins to encode the intra-prediction mode 388. Typically, a list of "maximum probability modes" is generated in the video encoder 114. This list is usually a fixed length, such as three or six modes, and may include modes encountered in earlier blocks. The context-coded bin encodes a flag indicating whether the intra-prediction mode is one of the maximum probability modes. If the intra-prediction mode 388 is one of the maximum probability modes, the bypass-coded bin encodes further signaling. For example, the encoded further signaling uses a truncated unary bin string to indicate which maximum probability mode corresponds to the intra-prediction mode 388. Otherwise, the intra-prediction mode 388 is encoded as a "remaining mode." Encoding as a remaining mode uses an alternative syntax, such as a fixed-length code, and is also encoded using the bypass-coded bin to represent intra-prediction modes other than those present in the maximum probability mode list.

[0121] The multiplexer module 384 outputs the PU 320 based on the optimal intra-frame prediction mode 388 determined from the tested prediction modes selected from each candidate CU. The candidate prediction modes need not include every conceivable prediction mode supported by the video encoder 114.

[0122] Prediction modes are broadly categorized into two types. The first type is "intra-frame prediction" (also known as "intra prediction"). In intra-frame prediction, predictions are generated for blocks, and the generation method can utilize other samples obtained from the current frame. For intra-frame prediction (PU), it is possible to use different intra-frame prediction modes for luma and chroma, thus intra-frame prediction is primarily described in terms of operation on PB rather than PU.

[0123] The second category of prediction modes is "inter-frame prediction" (also known as "inter-prediction"). In inter-frame prediction, samples from one or two frames preceding the current frame in the bitstream are used to generate a block prediction. The order of the encoded frames in the bitstream may differ from the order of the frames when captured or displayed. When one frame is used for prediction, the block is called "single prediction" and has two associated motion vectors. When two frames are used for prediction, the block is called "double prediction" and has two associated motion vectors. For P-strips, each CU can be intra-frame predicted or single-predicted. For B-strips, each CU can be intra-frame predicted, single-predicted, or double-predicted. Frames are typically encoded using a "picture group" structure to achieve a temporal hierarchy of frames. The temporal hierarchy of frames allows frames to reference previous and subsequent pictures in the order they are displayed. Images are encoded in the order necessary to ensure that the dependencies between frames are satisfied during decoding.

[0124] The subcategory of inter-frame prediction is called "skip mode". Inter-frame prediction and skip mode are described as two distinct modes. However, both inter-frame prediction and skip mode involve motion vectors referencing blocks of samples from previous frames. Inter-frame prediction involves encoding a motion vector increment (delta), which specifies the motion vector relative to the motion vector predictor. The motion vector predictor is obtained from a list of one or more candidate motion vectors selected using a "merge index". Encoding the motion vector increment provides a spatial offset relative to the selected motion vector prediction. Inter-frame prediction also uses encoded residuals in bitstream 133. Skip mode uses only an index (also called a "merge index") to select one from several motion vector candidates. The selected candidate is used without any further signaling. Moreover, skip mode does not support encoding any residual coefficients. The absence of encoded residual coefficients when using skip mode means that no skip mode transformation is required. Therefore, skip mode generally does not cause pipeline problems. Pipeline problems may arise for intra-frame prediction CUs and inter-frame prediction CUs. Due to the limited signaling of skip mode, it is useful for achieving very high compression performance when relatively high-quality reference frames are available. Dual-prediction CUs in higher temporal layers of a random access image group structure typically possess high-quality reference images and motion vector candidates that accurately reflect the underlying motion. Consequently, skipping modes are more effective for referencing... Figure 8B The double prediction blocks in frames at higher temporal levels in the described random access image group structure are useful.

[0125] Samples are selected based on motion vectors and reference image indices. Motion vectors and reference image indices apply to all color channels, thus describing inter-frame prediction primarily in terms of operations on the PU (Planetary Unit) rather than the PB (Planetary Frame). Within each category (i.e., intra-frame and inter-frame prediction), different techniques can be applied to generate the PU. For example, intra-frame prediction can use the combined directions of values ​​from adjacent rows and columns of previously reconstructed samples to generate the PU according to a prescribed filtering and generation process. Alternatively, a small number of parameters can be used to describe the PU. Inter-frame prediction methods can vary in the number and accuracy of motion parameters. Motion parameters typically include reference frame indices (indicating which reference frames from the reference frame list will be used, plus their respective spatial translations), but can include more frames, special frames, or complex affine parameters such as scaling and rotation. Additionally, predetermined motion refinement processes can be applied to generate dense motion estimates based on reference sample blocks.

[0126] Having determined and selected the optimal PU 320, and subtracted it from the original sample block at subtractor 322, the residual with the lowest encoding cost (denoted as 324) is obtained, and this residual is then subjected to lossy compression. The lossy compression process includes transform, quantization, and entropy coding steps. Transform module 326 applies a forward transform to the difference 324, thereby converting the difference 324 to the frequency domain and generating transform coefficients indicated by arrow 332. The forward transform is typically separable, transforming a set of rows and then a set of columns for each block. The transformations of each set of rows and columns are performed by first applying a one-dimensional transform to each row of the block to produce partial results, and then applying a one-dimensional transform to each column of the partial results to produce the final result.

[0127] Transform coefficients 332 are passed to quantizer module 334. At module 334, quantization is performed according to the "quantization parameters" to produce residual coefficients, indicated by arrow 336. The quantization parameters are constant for a given TB, thus resulting in a uniform scaling of the generated residual coefficients for the TB. Non-uniform scaling can also be achieved by applying a "quantization matrix," whereby the scaling factor applied to each residual coefficient is derived from a combination of the quantization parameters and corresponding entries in a scaling matrix typically equal in size to the TB. Residual coefficients 336 are fed to entropy encoder 338 for encoding in bit stream 115. Typically, the residual coefficients of each TB and at least one valid residual coefficient of the TU are scanned according to a scan mode to generate an ordered list of values. The scan mode typically scans the TB as a sequence of 4×4 "sub-blocks," thus providing a regular scan operation at the granularity of a 4×4 set of residual coefficients, where the configuration of the sub-blocks depends on the size of the TB. Additionally, prediction mode 388 and corresponding block partitions are also encoded in bit stream 115.

[0128] As described above, the video encoder 114 needs access to the frame representation corresponding to the frame representation seen in the video decoder 134. Therefore, the residual coefficients 336 are also inversely quantized by the dequantizer module 340 to produce the inverse transform coefficients, indicated by arrow 342. The inverse transform coefficients 342 are passed through the inverse transform module 348 to produce residual samples of TU, ​​indicated by arrow 350. The summation module 352 adds the residual samples 350 and PU 320 to produce reconstructed samples of CU (indicated by arrow 354).

[0129] Reconstructed sample 354 is passed to reference sample cache 356 and in-loop filter module 368. Reference sample cache 356, typically implemented using static RAM on an ASIC (thus avoiding expensive off-chip memory access), provides the minimum sample storage required to generate intra-frame PB dependencies for subsequent CUs in a frame. Minimal dependencies typically consist of a “line buffer” of samples along the bottom of a row of CTUs for use by the next row of CTUs and a column buffer whose range is set by the height of the CTU. Reference sample cache 356 feeds reference samples (indicated by arrow 358) to reference sample filter 360. Sample filter 360 applies a smoothing operation to produce filtered reference samples (indicated by arrow 362). Filtered reference samples 362 are used by intra-frame prediction module 364 to generate an intra-frame prediction block of samples, indicated by arrow 366. For each candidate intra-frame prediction mode, intra-frame prediction module 364 generates sample block 366.

[0130] The in-loop filter module 368 applies several filtering stages to the reconstructed samples 354. The filtering stages include a "deblocking filter" (DBF), which applies smoothing aligned with the CU boundaries to reduce artifacts caused by discontinuities. Another filtering stage present in the in-loop filter module 368 is an "adaptive loop filter" (ALF), which applies a Wiener-based adaptive filter to further reduce distortion. Another available filtering stage in the in-loop filter module 368 is a "sample adaptive offset" (SAO) filter. The SAO filter works by first classifying the reconstructed samples into one or more categories and then applying an offset at the sample level based on the assigned category.

[0131] The filtered sample, indicated by arrow 370, is output from the in-loop filter module 368. The filtered sample 370 is stored in the frame buffer 372. The frame buffer 372 typically has a capacity to store several (e.g., up to 16) images and is therefore stored in memory 206. Due to the large memory consumption required, the frame buffer 372 is typically not stored in on-chip memory. Thus, access to the frame buffer 372 is expensive in terms of memory bandwidth. The frame buffer 372 provides reference frames (indicated by arrow 374) to the motion estimation module 376 and the motion compensation module 380.

[0132] Motion estimation module 376 estimates multiple "motion vectors" (denoted as 378), each a Cartesian space offset relative to the current CU's position, thus referencing a block of one of the reference frames in frame buffer 372. A filtered block of reference samples (denoted as 382) is generated for each motion vector. The filtered reference samples 382 form further candidate modes for potential selection by mode selector 386. Furthermore, for a given CU, PU 320 can be formed using one reference block ("single prediction") or two reference blocks ("double prediction"). For the selected motion vector, motion compensation module 380 generates PU 320 based on filtering that supports sub-pixel precision in the motion vector. Thus, motion estimation module 376 (which operates on many candidate motion vectors) can perform simplified filtering compared to motion compensation module 380 (which operates only on the selected candidate), achieving reduced computational complexity.

[0133] Although the reference to Universal Video Coding (VVC) explains Figure 3 The video encoder 114 can be used, but other video encoding standards or implementations can also utilize the processing stages of modules 310-386. Frame data 113 (and bit stream 115) can also be read from (or written to) memory 206, hard disk 210, CD-ROM, Blu-ray disc, or other computer-readable storage media. Additionally, frame data 113 (and bit stream 115) can be received from (or sent to) an external source, such as a server or RF receiver connected to communication network 220.

[0134] exist Figure 4 The video decoder 134 is shown. Although... Figure 4 The video decoder 134 is an example of a Universal Video Coding (VVC) video decoding pipeline, but other video codecs can also be used for the processing stages described herein. Figure 4As shown, bitstream 133 is input to video decoder 134. Bitstream 133 can be read from memory 206, hard disk drive 210, CD-ROM, Blu-ray disc, or other non-transitory computer-readable storage media. Alternatively, bitstream 133 can be received from an external source, such as a server connected to communication network 220 or an RF receiver. Bitstream 133 contains encoded syntax elements representing the captured frame data to be decoded.

[0135] Bitstream 133 is input to entropy decoder module 420. Entropy decoder module 420 extracts syntax elements from bitstream 133 and passes the values ​​of the syntax elements to other modules in video decoder 134. Entropy decoder module 420 applies the CABAC algorithm to decode the syntax elements from bitstream 133. The decoded syntax elements are used to reconstruct parameters within video decoder 134. Parameters include residual coefficients (indicated by arrow 424) and mode selection information such as intra-frame prediction modes (indicated by arrow 458). Mode selection information also includes information such as motion vectors and partitioning of each CTU to one or more CUs. Parameters are used to generate PUs, typically combined with sample data from previously decoded CUs.

[0136] The residual coefficients 424 are input to the dequantizer module 428. The dequantizer module 428 performs inverse quantization (or "scaling") on the residual coefficients 424 to create reconstructed transform coefficients (indicated by arrow 440) based on the quantization parameters. If a non-uniform inverse quantization matrix is ​​indicated in the bitstream 133, the video decoder 134 reads the quantization matrix from the bitstream 133 as a sequence of scaling factors and arranges the scaling factors into a matrix. Inverse scaling combines the quantization matrix with the quantization parameters to create reconstructed intermediate transform coefficients.

[0137] The reconstructed transform coefficients 440 are passed to the inverse transform module 444. Module 444 transforms the coefficients from the frequency domain back to the spatial domain. The TB is effectively based on the effective and ineffective residual coefficient values. The result of the operation of module 444 is a block of residual samples, indicated by arrow 448. The residual samples 448 are equal in size to the corresponding CU. The residual samples 448 are fed to the summing module 450. At the summing module 450, the residual samples 448 are added to the decoding PU, indicated by 452, to produce a block of reconstructed samples, indicated by arrow 456. The reconstructed samples 456 are fed to the reconstructed sample cache 460 and the in-loop filtering module 488. The in-loop filtering module 488 produces a reconstructed block of frame samples, indicated by 492. Frame samples 492 are written to the frame buffer 496.

[0138] The reconstruction sample cache 460 operates in a manner similar to the reconstruction sample cache 356 of the video encoder 114. The reconstruction sample cache 460 provides storage for reconstructed samples required for intra-frame prediction of subsequent CUs in the absence of memory 206 (e.g., by using data 232, typically on-chip memory, instead). The reference sample, indicated by arrow 464, is obtained from the reconstruction sample cache 460 and supplied to the reference sample filter 468 to produce the filtered reference sample, indicated by arrow 472. The filtered reference sample 472 is supplied to the intra-frame prediction module 476. Module 476 generates blocks of intra-frame prediction samples, indicated by arrow 480, based on the intra-frame prediction mode parameters 458 represented in bitstream 133 and decoded by the entropy decoder 420.

[0139] When intra-prediction is indicated in the current CU bitstream 133, the intra-prediction sample 480 is formed into a decoding PU 452 via the multiplexer module 484.

[0140] When inter-frame prediction is indicated in the current CU bitstream 133, the motion compensation module 434 selects and filters sample blocks from the frame buffer 496 using motion vectors and reference frame indices to produce a block of inter-frame prediction samples, represented as 438. Sample block 498 is obtained from previously decoded frames stored in the frame buffer 496. For dual prediction, two sample blocks are generated and combined to produce samples for the decoded PU 452. The frame buffer 496 is filled with filtered block data 492 from the in-loop filtering module 488. Similar to the in-loop filtering module 368 of the video encoder 114, the in-loop filtering module 488 applies any, at least, all, DBF, ALF, and SAO filtering operations. The in-loop filtering module 368 generates filtered block data 492 from the reconstructed samples 456.

[0141] Figure 5 This is a schematic block diagram illustrating a set 500 of available partitions or splits of a region into one or more sub-regions in a tree structure of general video coding. (See reference...) Figure 3 The segmentation shown in set 500 can be used by block partitioner 310 of encoder 114 to segment each CTU into one or more CUs according to the number of codes determined, such as by Lagrange optimization.

[0142] Although set 500 only shows the division of a square region into other possible non-square sub-regions, it should be understood that set 500 is showing potential divisions and does not require the containing region to be square. If the containing region is non-square, the size of the block obtained from the division is scaled according to the aspect ratio of the containing block. Once a region is not further divided, that is, at the leaf node of the coding tree, the CU occupies the region. The specific sub-segmentation of one or more CUs using the CTU of block partitioner 310 is called the "coding tree" of the CTU. The process of sub-segmenting a region into sub-regions must terminate when the resulting sub-regions reach the minimum CU size. In addition to constraining the CU to a size that is not smaller than, for example, 4×4, the CU is constrained to have a minimum width or height of four. Other minimum values ​​are also possible for width and height, or for both width and height. The sub-segmentation process may also terminate before the deepest level of decomposition, resulting in a CU larger than the minimum CU size. It is possible that no splitting occurs, resulting in a single CU occupying the entire CTU. A single CU occupying the entire CTU is the maximum available coding unit size. Furthermore, a CU that has not been split is larger than the processing region size. As a result of binary or ternary splitting at the highest level of the coding tree, CU sizes such as 64×128, 128×64, 32×128, and 128×32 are possible, each larger than the processing region size. (Reference) Figure 10A-10F An example of a CU larger than the processing area size is further described.

[0143] In the absence of further subsegments, a CU exists at a leaf node of the coding tree. For example, leaf node 510 contains a CU. At non-leaf nodes of the coding tree, there are splits into two or more other nodes, where each node may contain a leaf node (and thus a CU) or further splits into smaller regions.

[0144] like Figure 5 As shown, quadtree split 512 divides the containing region into four regions of equal size. Compared to HEVC, Universal Video Coding (VVC) achieves additional flexibility by adding horizontal binary split 514 and vertical binary split 516. Splits 514 and 516 each divide the containing region into two regions of equal size. The splits are made along the horizontal boundary (514) or vertical boundary (516) within the containing block.

[0145] Further flexibility is achieved in general video coding by adding ternary horizontal splitting 518 and ternary vertical splitting 520. Ternary splitting 518 and 520 divide a block into three regions that form boundaries in the horizontal direction (518) or vertical direction (520) along ¼ and ¾ of the width or height of the containing region. The combination of quadtree, binary tree, and ternary tree is called "QTBTTT" or alternatively, multi-branch tree (MT).

[0146] Compared to HEVC, which only supports quadtrees and therefore only square blocks, QTBTTT offers a wider range of possible CU sizes, particularly considering the potential recursive application of binary and / or ternary tree splits. The likelihood of anomalous (e.g., non-square) block sizes can be reduced by constraining split options to eliminate splits that result in block widths or heights less than four samples or that are not multiples of four samples. Typically, constraints are applied when considering luma samples. However, constraints can also be applied individually to blocks for chroma channels, potentially resulting in different minimum block sizes for luma vs. chroma (e.g., when frame data uses a 4:2:0 chroma format). Each split produces a sub-region with side sizes that are either bisected or quadrupled relative to the containing region. Then, since the CTU size is a power of 2, the side sizes of all CUs are also powers of 2.

[0147] Figure 6 This is a schematic flowchart illustrating the data stream 600 of the QTBTTT (or "coding tree") structure used in general video coding. The QTBTTT structure is used for each CTU to define the partitioning of the CTU into one or more CUs. The QTBTTT structure of each CTU is determined by the block partitioner 310 in the video encoder 114 and is encoded into bit stream 115 or decoded from bit stream 133 by the entropy decoder 420 in the video decoder 134. Figure 5 As shown in the diagram, data stream 600 further exhibits features that allow block partitioner 310 to partition the CTU into one or more CUs using a combination of licenses.

[0148] Starting from the top level of the hierarchy, i.e., at CTU, zero or more quadtree partitions are performed first. Specifically, the block partitioner 310 makes a quadtree (QT) split decision 610. The decision at 610 returns a "1" sign, indicating that it is decided to split the current node into four child nodes according to the quadtree split 512. The result is the generation of four new nodes, such as at 620, and for each new node, the process recursively returns to the QT split decision 610. Each new node is considered in raster (or Z-scan) order. Alternatively, if the QT split decision 610 indicates no further splitting (returns a "0" sign), the quadtree partitioning stops, and multitree (MT) splits are subsequently considered.

[0149] First, block partitioner 310 makes an MT split decision 612. At 612, a decision indicating whether to perform an MT split is made is given. Decision 612 returns a "0" sign, indicating that no further splitting of the node into its child nodes will be performed. If no further splitting of the node is performed, the node is a leaf node of the coding tree and corresponds to a CU. The leaf node is output at 622. Alternatively, if MT split 612 indicates a decision to perform an MT split (returning a "1" sign), block partitioner 310 proceeds to direction decision 614.

[0150] Direction decision 614 indicates the direction of MT splitting as horizontal (“H” or “0”) or vertical (“V” or “1”). If decision 614 returns “0” indicating the horizontal direction, block partitioner 310 proceeds to decision 616. If decision 614 returns “1” indicating the vertical direction, block partitioner 310 proceeds to decision 618.

[0151] In each of decisions 616 and 618, the number of partitions to be split in the MT partition during the BT / TT split is indicated as two (binary split or "BT" nodes) or three (ternary split or "TT" nodes). That is, when the direction indicated from 614 is horizontal, the block partitioner 310 makes the BT / TT split decision 616, and when the direction indicated from 614 is vertical, the block partitioner 310 makes the BT / TT split decision 618.

[0152] BT / TT split decision 616 indicates whether the horizontal split is a binary split 514 indicated by returning "0" or a ternary split 518 indicated by returning "1". When BT / TT split decision 616 indicates a binary split, at step 625 of generating HBT CTU nodes, block partitioner 310 generates two nodes based on binary horizontal split 514. When BT / TT split 616 indicates a ternary split, at step 626 of generating HTT CTU nodes, block partitioner 310 generates three nodes based on ternary horizontal split 518.

[0153] The BT / TT split decision 618 indicates whether the vertical split is a binary split 516 indicated by returning "0" or a ternary split 520 indicated by returning "1". When the BT / TT split 618 indicates a binary split, at step 627 of generating VBT CTU nodes, the block partitioner 310 generates two nodes according to the vertical binary split 516. When the BT / TT split 618 indicates a ternary split, at step 628 of generating VTT CTU nodes, the block partitioner 310 generates three nodes according to the vertical ternary split 520. For each node obtained from steps 625-628, the data flow 600 is applied according to direction 614 in a left-to-right or top-to-bottom order, recursively returning to the MT split decision 612. As a result, binary and ternary splits can be applied to generate CUs of various sizes.

[0154] Figure 7A and 7B Provides an example of CTU 710 splitting into multiple CUs, 700. Figure 7A Example CU 712 is shown in the figure. Figure 7A This illustrates the spatial arrangement of the CUs in a CTU 710. Example segmentation 700 in... Figure 7B It is also shown as coding tree 720.

[0155] exist Figure 7A At each non-leaf node in CTU 710 (e.g., nodes 714, 716, and 718), the contained nodes (which may be further splits or CUs) are scanned or traversed in "Z order" to create a list of nodes represented as columns in the coding tree 720. For quadtree splits, the Z-order scan results in a sequence from top left to right followed by a sequence from bottom left to right. For horizontal and vertical splits, the Z-order scan (traversal) simplifies to a top-to-bottom scan and a left-to-right scan, respectively. Figure 7B The encoding tree 720 lists all nodes and CUs according to the applied scan order. Each split generates a list of two, three, or four new nodes at the next level of the tree, until a leaf node (CU) is reached.

[0156] As per reference Figure 3 In the case where the image is decomposed into CTUs and further decomposed into CUs using block partitioner 310, and residual blocks (324) are generated using CUs, the residual blocks are forward transformed and quantized using video encoder 114. The resulting TBs 336 are then scanned to form an ordered list of residual coefficients as part of the operation of entropy coding module 338. Equivalent processing is performed in video decoder 134 to obtain TBs from bitstream 133.

[0157] Figure 8AAn example frame 800 is shown, comprising a sequence of CTUs (e.g., CTU 812, followed by subsequent CTUs). Each CTU has a size of 128 × 128 luminance samples. If frame 800 were to be processed in units of CTUs using the local memory or integrated circuits of processor 205, the resulting local memory requirements would be prohibitive due to the 128 × 128 luminance samples per CTU. The implementations of the video encoder 114 and video decoder 134 described herein can reduce on-chip memory consumption by processing image data or bitstreams in areas smaller than the area of ​​a CTU. On-chip memory is particularly expensive because it consumes a large area on the die. Software implementations can also benefit from reducing the need to access external memory by restricting more memory accesses to lower-level cache memories (e.g., L1 and L2 cache memories). Thus, to reduce memory consumption, the implementations of the video encoder 114 and video decoder 134 can process data at a granularity smaller than that of a single CTU.

[0158] Smaller granularities can be regions (or “pipeline processing regions”) the size of 64×64 luminance samples, similar to a quadtree subdivision of a CTU. Furthermore, smaller granularity definitions are regions considered indivisible. Indivisible regions are passed through the various processing stages of the pipeline architecture. A pipelined processing region is considered indivisible in the sense that the region definition corresponds to an aggregate or chunk of data (such as a collection of samples, blocks, and coefficients, a portion of a bitstream, etc.) on a specific region of a frame (such as frame 800) and traverses the pipeline. Within a region, various arrangements of CUs can exist, and CUs can span multiple smaller granularity regions. These regions allow each pipeline processing stage to locally store only the data associated with the smaller region (e.g., 64×64 luminance samples or less), rather than the data associated with the full 128×128 CTU size.

[0159] The pipelined processing areas are also used to reduce the corresponding local memory usage of chroma data. Within each CTU, regions are processed in Z-order. This processing proceeds from CTU to CTU in a raster scan manner, as shown in region scan 810. From the perspective of video decoder 134, the first pipeline stage is entropy decoder 420. Although bitstream 133 is parsed sequentially, the parsed syntactic elements can be grouped according to regions. For example, Figure 8ARegion 814 is first processed by entropy decoder 420. Once entropy decoder 420 has processed a region, the associated syntactic elements are passed to a second pipeline stage. The second pipeline stage may be an inverse quantizer 428 and an inverse transform 444. Modules 428 and 444 process all CUs in the region to produce residual samples 448 for the region. Once the second stage is complete, the residual samples 448 for the region are passed to a third stage. The third stage may include summation 450 (intra-frame reconstruction), a reference sample cache 460, a reference sample filter 468, and an intra-frame prediction module 476. The third-stage modules form a feedback loop, such as... Figure 4 The images and references shown Figure 4 The feedback loop exists between adjacent CUs, and thus exists both within a region and from one region to the next. This feedback loop requires a third-stage module within a pipeline stage. In-loop filtering 488 is typically performed in one or more subsequent pipeline stages.

[0160] A separate feedback loop for inter-frame prediction can also be implemented involving frame buffer 496 and motion compensation module 434. However, the feedback loop for inter-frame prediction runs from the current frame to the previous frame and therefore does not affect pipelined operations at the CTU level. Figure 8A Areas yet to be processed (e.g., 816) are shown in light shades.

[0161] Depending on the coding tree of each CTU, a region can contain different CUs of different sizes, for example, as per the information provided. Figure 7A and 7B As described. Figure 7A and 7B Examples include quadtree splitting at the top level of the encoding tree, in Figure 7B The value is represented as 726. The quadtree is divided into four 64×64 regions (which are further subdivided into CUs of different sizes), aligned with the 64×64 pipelined processing region size. When encoding image frames containing highly detailed textures and without available reference images, such as in the case of "intra-frame" frames, the likelihood of at least one quadtree split is high. Therefore, for intra-frame coding, a smaller CTU size (e.g., 64×64) can be used compared to the CTU size used for 128×128 inter-frame prediction frames, without imposing an unacceptable level of compression performance loss on the user.

[0162] Using a small CTU size (specifically, no larger than the area size used for pipelined processing) ensures no CUs and therefore no TUs spanning multiple areas. Transformations are operations that need to be performed within a pipeline stage because data dependencies within a transform span the TB, causing each residual coefficient to affect every output sample from the TB. As a result, transforms represent pipelined operations that must be at the smallest "atomic" (undivided) level, so the data for a given transform must be completely contained within a single pipelined processing area or data block.

[0163] However, within a pipelined processing area, multiple control units (CUs) and therefore multiple transformations are possible because each transformation is entirely contained within the processing area. Limitations on transformation size set the worst-case scenario for the number of transformations that may be encountered within the processing area. For example, for a 64×64 processing area and considering the luma channel, the maximum number of 4×4 transformations that may be encountered within the processing area is 16×16=256. The maximum number of 8×8 transformations that may be encountered within the processing area is 8×8=64, and so on, up to a 64×64 transformation. For a 64×64 transformation, only one transformation may be performed within a pipelined processing area. Similar calculations apply to the chroma channel.

[0164] For inter-frame prediction CUs, relatively large CUs are possible. The possibility of larger CUs arises from the availability of one or more reference frames that can contain highly matching reference blocks. The video encoder 114 can select a large CU and, in doing so, copy a large sample block from the reference frame to the current frame.

[0165] One approach to encoding inter-frame prediction CUs is via a "skip mode." A CU encoded in skip mode has no valid residual coefficients and obtains its corresponding motion vector from a spatial or temporal neighbor selected using a "merge index." The absence of any valid residual coefficients means no inverse quantization or inverse transform steps are required. Thus, the placement of skip mode CUs within the CTU is not constrained by pipelined processes, and the TU size for skip mode CUs does not need to be determined. Skip mode CUs do not introduce difficulties in aligning specific CUs relative to the pipelined processing region. Skip mode CUs are typically used when high-quality reference frames are available and motion parameters are easily modeled. Therefore, skip mode CUs are selected for blocks where the decoder can accurately predict motion vectors. The motion vector selection is based on reference blocks that closely match the desired output at the CU.

[0166] The use of skip mode is not limited to portions of an image containing relatively low detail. Image portions containing highly detailed textures can be reproduced at low cost because the coding tree terminates at a large CU size, and the encoding of motion vectors specifying spatial displacements is highly efficient, especially via merged index encoding. Frames in higher temporal layers of a random access image group structure are an example of how skip mode provides high compression performance. Figure 6 The described flexible block structure, combined with a relatively large CTU size, such as 128×128, allows for the flexible placement of large CUs within each CTU. Therefore, the decoder or encoder can adapt to changes in the motion field, such as those typically occurring at the boundary between foreground and background objects. Large CUs are generally common at low bit rates. Furthermore, these large CUs can span multiple pipelined processing regions without being constrained to avoid crossing pipelined processing regions. Applying constraints to avoid crossing pipelined processing regions is equivalent to reducing the CTU size. Reducing the CTU size limits the flexibility of both CU size and placement within each CTU, thus undesirably reducing compression efficiency.

[0167] Figure 8B This is an example "Random Access" Group of Pictures (GOP) structure 850 for frames in a video. A sequence of seventeen frames is shown. For each frame, the display order 851, encoding order 852, and temporal layer ID 853 are shown in structure 850. The video begins with an intraframe 860. Since the reference picture buffer is empty, intraframe 860 can only include intra-predictive CUs. The second frame to be encoded is a single-predictive frame (P-strip) 870 (because the second value of encoding order 852 is "16"). Single-predictive frame 870 is encoded after intraframe 860 and can be referenced only to frame 860, as indicated by arrow 872. However, the display order of frame 870 is 16, so relatively large changes to the content between frames 860 and 870 are possible. Therefore, the encoding cost of frame 870 is relatively high. However, the encoding cost of frame 870 is less than that of intraframe 860, in which no reference frame is available.

[0168] Intermediate frames can use double prediction, such as having two arrows relative to the available reference frame (e.g., Figure 8B As shown by arrows 891 and 892 in the diagram, the distance from a frame to its corresponding reference frame (the increment in display order) is smaller at higher temporal levels. Generally, compression performance is higher when the distance from a frame to its corresponding reference frame is smaller because the underlying image data varies less between frames. Frames at higher temporal levels can typically utilize larger CUs and more frequently the skip mode. An exception is when an occluded object becomes visible. When an occluded object becomes visible, there are usually no available reference blocks within the constraints of the group of pictures (GOP) structure, and the use of smaller CUs with intra-frame prediction becomes more likely.

[0169] Figure 9 The set of 900 supported transform sizes for the VVC standard used for the luma channel is shown. Supported transform sizes for the luma channel are as follows: 4×4, 4×8, 4×16, 4×32, 8×4, 8×8, 8×16, 8×32, 16×4, 16×8, 16×16, 16×32, 32×4, 32×8, 32×16, 32×32, and 64×64. For the chroma channel using the 4:2:0 chroma format, the corresponding chroma transform size is available for each luma transform size. The chroma transform size has half the width and half the height of the luma transform size. Two-dimensional (2D) transforms are separable, with one-dimensional (1D) DCT-2 transforms performed horizontally and vertically. Alternatively, a choice can be made between a 1D DCT-7 transform and a 1D DST-7 transform, where the horizontal and vertical phases of the transform are independently controllable.

[0170] Generally, a TU is associated with a CU. For each color channel, a TB is associated with a TU. However, the TB of a color channel can be said to be nonexistent when there are no valid coefficients for the transformation of a particular color channel. The color channel can be said to be nonexistent because there is no need to transform the array of all zero residual coefficients. Although one-dimensional (1D) transformations are usually defined in terms of matrix multiplication operations, implementations using butterfly steps and lifting steps are often used for complexity reduction. Due to the horizontal and vertical stages of the application, there are dependencies both within each 1D transformation and across 2D blocks. Therefore, each residual sample is affected by each residual coefficient (from the perspective of the inverse transformation) and there is a correspondence for the forward transformation. The work of performing the forward or inverse transformation cannot be divided into segments (e.g., transforming half of the TB and then transforming the other half later). The computational cost of determining half of the TB is almost the same as the cost of determining the entire TB. Thus, the architecture for determining the transformation in segments has much higher complexity than the architecture for determining the transformation (from the perspective of the processing pipeline) as "atomic" (indivisible) operations. In contrast, as described below, when processed in segments, the PU has a cost per segment that is approximately proportional to the segment size versus the total PU size.

[0171] To support large CUs spanning multiple pipeline regions, data dependencies for intra-frame and inter-frame prediction are considered. For intra-frame prediction, spatially adjacent reference samples and intra-frame prediction patterns are used to generate prediction blocks. When the CU is larger than the pipeline region size, PUs can be determined in multiple parts, such that the prediction pipeline stage operating on the pipeline region computes partial PBs (for one region) and determines additional partial PBs (for subsequent regions), which together form the entire PB.

[0172] For intra-prediction blocks, determining a partial prediction block requires using a reference sample of the PB. The reference sample does not need to be adjacent to the partial PB. For example, a 128×64 PB is segmented and processed into two 64×64 partial PBs. The resulting second (rightmost) 64×64 partial PB uses a reference sample corresponding to the original (complete) 128×64 PB, while the unused reference sample is the 64×64 prediction block present at the location of the second 64×64 partial prediction block. As a result, a pipelined processing architecture operating at a granularity smaller than the CTU size can perform intra-prediction on PBs larger than the pipeline processing region size, with the added cost of buffering additional reference samples for partial PBs. The additional buffered reference samples are rows and columns of samples whose size is determined according to the width of the CTU, but no additional frame-width row buffer is required.

[0173] For inter-frame prediction blocks, segmenting the PU into multiple partial PUs is relatively simple because the common information used includes one or more motion vectors and one or more reference frame indices. Thus, a PU can span multiple pipeline processing regions and be processed into multiple partial PUs, each contained within a separate pipeline processing region. Even if several PUs span multiple pipeline processing stages, the cost of storing the associated motion vectors for use across multiple pipeline processing regions is low. Using large PUs for inter-frame prediction is highly beneficial for low bit-rate applications and especially at higher levels when using group of picture (GOP) structures such as "random access". In such GOP structures, and especially in low-motion regions, relatively large PUs can be used. Large PUs are used to encode relatively large portions of the entire picture with the minimum syntax present in the bitstream.

[0174] Figure 10A The diagram shows the CUs and TUs of a CTU 1000, which has a vertical ternary split at the top level of the coding tree and is not further split. Splitting the coding tree yields three CUs, 1020, 1022, and 1024, with sizes of 32×128, 64×128, and 32×128, respectively. CUs 1020, 1022, and 1024 are located within the CTU at offsets of (0,0), (32,0), and (96,0), respectively. For each CU, there is a corresponding PU of the same size, and within the CTU 1000, the corresponding PU spans multiple pipeline processing regions. One or more TUs are also associated with each CU. When the CU size is equal to one of the transform sizes, a TU is associated with the CU and has a transform size equal to the corresponding size.

[0175] Figure 10B Showing with Figure 10AThe CU of the coding tree is associated with an alternative arrangement of TUs (CTUs) of 1040. When the CU size is larger than any transform size, multiple TUs are arranged in a "stitched" manner to occupy the entire CU. Given width and height constraints, the stitching uses the largest available transform within the CU to "fit". For example, as... Figure 10B As shown, the 32×128 CU1042 and 32×128 CU1046 use four 32×32 TUs in a stitched configuration. The 64×128 CU1044 uses two 64×64 TUs in a stitched configuration because 64×64 is the maximum transform size available for the CU 1044. As mentioned above, splitting the CTU 1040 does not cause performance issues for intra-frame or inter-frame prediction operations. However, the processing of the associated TUs needs to be adapted to the pipeline processing region size.

[0176] For reference Figure 9 The available transform sizes are limited to a specific set of sizes. From the perspective of a pipelined processing architecture, the processing of each TB for encoding or decoding is an indivisible operation. Because it is necessary to accommodate different CU placements within the CTU and to perform each transform for a region entirely within a single pipeline stage, two methods are described below.

[0177] In the first method, the pipeline processing region is not always a fixed size (e.g., 64×64). Instead, the size of the pipeline processing region is adapted to the coding tree of each CTU. Therefore, the first method can be referred to as a "flexible pipeline processing region". The term "flexible pipeline processing region" is used to distinguish it from the case of a fixed-size region, where a fixed grid of the pipeline processing region exists in the image frame, as discussed elsewhere in the invention. In particular, using a flexible pipeline processing region, CTU 1040 can be processed as follows:

[0178] Region 0: 32×64 (the upper half of CU 1042, containing two 32×32 TUs).

[0179] Region 1: 64×64 (the upper half of CU 1044, containing a 64×64 TU).

[0180] Region 2: 32×64 (the upper half of CU 1046, containing two 32×32 TUs).

[0181] Region 3: 32×64 (the lower half of CU 1042, containing two 32×32 TUs).

[0182] Region 4: 64×64 (the lower half of CU 1044, containing a 64×64 TU).

[0183] Region 5: 32×64 (the lower half of CU 1046, containing two 32×32 TUs).

[0184] Thus, the arrangement of the CUs in the CTU 1040 results in six flexible pipeline processing areas of sizes 32×64 and 64×64. This is the worst-case scenario for the number of flexible pipeline processing areas.

[0185] When the TU does not cross the boundary of the flexible pipeline processing area (e.g., due to quadtree splitting, as in...) Figure 7A In the example shown, the number of flexible pipelined processing regions is four, and each region has a size of 64×64 luminance samples. While the flexible pipelined processing region approach does achieve flexible placement of the TU within the CTU in the pipelined implementation, the worst-case processing rate of this example pipeline is increased by 50% compared to an architecture where the pipelined processing regions are fixed on each CTU and therefore on the image frame. Although the total sampling rate remains unchanged, in the pipelined architecture, the processing rate of each volume region is not necessarily solely related to the region size; therefore, smaller regions are not necessarily processed at a higher rate commensurate with their smaller size, but rather at a lower rate due to the overhead of processing each region. Thus, the worst-case design is superior to a system where all regions are of the same size. Furthermore, as referenced... Figure 10C Furthermore, there are cases where flexible processing areas require seven regions per CTU, thereby further increasing the worst-case region rate of this architecture.

[0186] In the second method, the relationship between CUs and TUs is altered, extending the splicing method for "large" CUs (CUs exceeding the width or height of available transforms). This extended splicing method is also applied to CUs that would otherwise have TUs spanning multiple pipeline processing regions. For example, CU 1022 is partitioned into a 2x4 array of 32×32 TUs. The implicit aspect of partitioning CU (1022) into smaller TUs is that the partition is determined by the coding tree and the placement of the CUs within the CTU, without requiring further signaling (e.g., additional flags) in the bitstream. However, the partition can be determined based on the implicit properties of the coding unit itself, which is the size of the coding unit. The partitioning of coding unit 1022 is implemented... Figure 10A The following are the automated processes for the CTU 1000:

[0187] Region 0: 64×64 (the upper half of CU 1020, the upper left quarter of CU 1022, a total of four 32×32TUs).

[0188] Region 1: 64×64 (the upper right quarter of CU 1022, the upper half of CU 1024, a total of four 32×32TU).

[0189] Region 2: 64×64 (the lower half of CU 1020, the lower left quarter of CU 1022, a total of four 32×32TUs).

[0190] Region 3: 64×64 (the lower right quarter of CU 1020, the lower half of CU 1022, a total of four 32×32TUs).

[0191] Thus, using the second method, regardless of the coding tree, four 64×64 pipelined processing regions are always used to process the 128×128 CTU. The shape and / or aspect ratio of the transformation unit differs from the shape or size of the CU. Therefore, the processing rate of the pipelined processing region is constant, regardless of the arrangement of the CUs within the CTU. In the second method, a 64×64 TU can only be used for CUs whose top-left position is aligned to a 64×64 grid relative to the top-left corner of the image frame. The required coding unit condition occurs for CTUs with coding trees that have no splitting operations (which have 256×256 CUs), or with at most one binary split in each of the horizontal and vertical directions (giving CUs of size 128×64, 64×128, or 64×64) or a single quadtree split (giving four CUs of size 64×64 each).

[0192] In the third method, the 64×64 TU is not present in the video encoder 114 and the video decoder 134, and therefore is not included in the set of available TU sizes. The 64×64 TU is relatively rarely used. However, the 64×64 TU can be beneficial at very low bit rates, so the absence of the 64×64 transform size does indeed cause a loss of compression performance for the VVC standard. However, even if the 64×64 transform is removed, TU splicing is still required when the CU spans multiple pipeline processing regions. For example, Figure 10E The CTU 10300 shown has a coding tree containing the results of two vertical ternary splits. The two vertical ternary splits result in a CU 10320 of size 32×128, vertically oriented along the center of the CTU. The CU 10320 is located at an offset of (48, 0) relative to the top-left corner of the CTU 10300. The CU 10320 occupies each of four 64×64 pipeline processing regions and uses a relatively small TU.

[0193] Figure 10EThe diagram illustrates a CTU 10300 segmented into regions according to a (first) method for flexible pipelined region processing. The coding tree has two vertical ternary partitions that can be adapted using two additional regions with 32×64 luminance samples each. Each additional region can contain two 32×32 TUs (arranged as a 1x4 column overall in the CU 10320), with the four regions occupying 48×64 luminance samples. The resulting segmentation of the CTU 10320 is determined by... Figure 10E The region boundary 10330 is shown. In the first method, the stitching of TUs within a CU is only constrained by the availability of transform sizes. The CU is stitched with TUs using the maximum available size.

[0194] Figure 10F CTU 10400 is shown. CTU 10400 relates to CTU 10300, which is divided into multiple regions according to a second method of further splicing TUs. Figure 10F In the example, the 32×128 CU 10420 (corresponding to CU 10320) is associated with a 2x4 array of 16×32 TUs, rather than a 1x4 array of 32×32 TUs. The 2x4 array of 16×32 TUs ensures that each TU is contained within one of the four pipelined processing regions of the CTU, as shown at region boundary 10430. Figure 10F In the example, no TU crosses boundary 10430, meaning no TU overlaps with or crosses two or more pipeline processing regions. The example with a coding tree having two vertical ternary splits demonstrates that different coding trees are possible for splitting the CU into multiple TUs to avoid TUs crossing multiple pipeline processing regions. Furthermore, the example case is not limited to cases involving 64×64 TUs. In the (second) method, the concatenation of TUs within the CU is further constrained by the requirement that each TU does not cross multiple pipeline processing regions. As a result, TUs smaller than the maximum available transform size would be used, which limits the concatenation of TUs within the CU. Since the transform is used to decorrelate residual samples on a given-size block, lower compression performance is expected due to the use of smaller TUs. However, for inter-frame prediction CUs, lower compression performance does not occur in some cases. A single transform is effective when decorrelating residual samples containing values ​​mapped to a relatively small number of basis functions. Therefore, a single transform is generally effective for decorrelation of low-frequency residual samples, which are those located in the frequency domain facing the top-left corner of the TB. However, residual samples containing highly discontinuous content (such as computer graphics or text) or content concentrated in a portion of a block are poorly decorrelated because all basis functions span the entire TB.

[0195] For example, it's common for residual samples to be concentrated in a portion of the CU (Cumulative Unit) for inter-frame prediction blocks and stems from differences such as the boundary between the foreground and background. Generally, using larger blocks improves compression performance, but also results in blocks that span the foreground and background (or other similar boundaries). When the foreground and background have different motions, a portion of the CU is typically predicted well from the reference image (with residual samples having zero or near-zero values). In contrast, other portions of the CU are predicted poorly from the reference image.

[0196] One solution is for the video encoder 114 to further split the coding tree, resulting in smaller CUs. Each smaller CU can be assigned a different prediction mode to better fit the underlying image features. Motion search often fails to find enough inter-frame prediction CUs at object boundaries or where occluded areas become visible. Instead, intra-frame prediction is typically used. The stitching of TUs within a CU limits the spatial influence of each residual coefficient to the size of the corresponding TB. Thus, the use of stitched TUs within a CU allows for the selection of larger CUs. Larger CUs can be CUs spanning occluded foreground and / or background objects across an image frame. Further stitching of TUs due to pipelined processing region boundaries may not unduly degrade compression performance and may even offer advantages due to the selection of CUs larger than would be the case with the video encoder 114 originally.

[0197] Figure 10C The diagram shows the TU associated with a coding tree 10100 having two ternary splits in opposite directions. The first ternary split vertically yields three regions 10101, 10102, and 10103. The middle region 10102 is further ternary split horizontally, resulting in an additional region. Specifically, a 64×64 CU 10122 originates from a further ternary split within region 10102. CU 10122 is located at an offset of (32, 32) relative to the top-left corner of CTU 10100. CU 10122 spans four pipeline processing regions, as seen from the processing region boundary 10110.

[0198] If a 64×64 transform were to be used for the CU 10122, seven processing regions would be required, consisting of one 64×64 region for the CU 10122 and six additional regions (two 32×64 and four 64×32) to process the remaining regions in the CTU. With the substantial increase in pipelined processing regions beyond the usual four, the 64×64 transform size becomes unsuitable for the CU 10122. Therefore, four 32×32 transforms are used for the CU 10122 to encode the residuals.

[0199] Figure 10DThe diagram shows the TU associated with the coding tree of CTU 10200, which has two ternary splits in the same direction and a middle binary split in the opposite direction. As a result of the coding tree of CTU 10200, CU10222, with a size of 32×64, is located at position (0, 48) relative to the upper left corner of CTU 10200. CU 10222 spans two pipeline processing regions, such that each region includes a 16×64 portion of CU 10222. Based on Figure 9 The available transform size, using a 16×32 size, is concatenated in a 2x2 manner to occupy CU 10200. The pipelined implementation can then process the CTUs in the region depicted by boundary 10210. Furthermore, the pipelined implementation can process CU 10200 by separately processing the leftmost two transforms for CU 10222 as part of one region and the rightmost two transforms for CU 10222 as part of another region.

[0200] Intra-frame stitching TUs can reconstruct the boundaries of each TU. Reconstructing the boundaries of each TU increases complexity due to additional feedback loops within each CU. However, reconstructing the boundaries of each TU does not increase worst-case complexity because choosing an alternative for a smaller CU will result in feedback loops of the same severity. For inter-frame prediction CUs, there are no additional feedback loops.

[0201] Figure 11 A method 1100 is shown for determining the prediction mode of a coding unit in a coding tree. Method 1100 enables pipelined implementation of a video encoder 114 and a video decoder 134. When determining the coding tree of a CTU, the video encoder 114 performs a search to determine the coding tree, as per [reference to...]. Figure 6 As described. Step 622, which generates leaf nodes, tests the option of including a CU in each “candidate region” of the CTU without further splitting. At step 622, method 1100 is invoked to generate one or more candidate CUs. Each candidate CU is evaluated based on a comparison between the best candidate CU for a region of the CTU and further splitting of the region. When considering the best candidate CU derived from the split, both the coding tree and the prediction mode of the resulting CU are determined. That is, the “best” candidate CU is selected for use in bitstream 115. The arrangement of the video encoder 114 and the video decoder 134 may limit the available prediction modes for a particular CU based on aspects such as CU size and / or time layer ID. This limitation affects the determination of the prediction mode. Furthermore, this limitation also reduces the necessary prediction mode signaling for the affected CU, as referenced. Figure 12 and 13The description is particularly relevant to steps 1222 and 1322. Method 1100 can be embodied by a device such as a configured FPGA, ASIC, or ASSP. Alternatively, method 1100 can be executed by a video encoder 114 under the processing of processor 205. Thus, method 1100 can be stored on a computer-readable storage medium and / or stored in memory 206. Method 1100 invoked for each candidate region begins at stripe type test step 1110.

[0202] At stripe type test step 1110, processor 205 tests the stripes of the current frame of video data 113. Typically, the stripe type is derived from the GOP structure (e.g., Figure 8B (Random access GOP structure). If the stripe type is intra-frame (“I”) stripe, an intra-frame candidate CU 1150 is added for evaluation. If the stripe type is inter-frame (“P” or “B”) stripe, method 1100 proceeds from step 1110 to candidate CU size test step 1120.

[0203] At candidate CU size testing step 1120, processor 205 tests the size of the CU to determine whether only the skip mode can be used to encode the CU, or whether other prediction modes can also be used. If either side of the CU (i.e., width or height) exceeds 64, step 1120 returns "large," and only skip mode candidate 1170 is added for evaluation. The return of "large" in step 1120 infers that the CU overlaps with the processing region, and steps 1210 and 1310 will be described. If the CU is the result of a vertical ternary partition, the overlap is likely vertical.

[0204] Since only one mode is added for evaluating any CU with a side length greater than 64, it is unnecessary to encode or decode the skip flag for CUs with a side length greater than 64. Since the size of the CU is sufficient for the video decoder 134 to determine the prediction mode of the CU, it is also unnecessary to encode or decode the skip flag. Furthermore, for a CTU size of 128×128, encoding CUs with a side length greater than 64 as skipped only prevents pipeline problems that occur with such large CUs. Pipeline problems may still occur with smaller CUs, as addressed by additional TU splicing within the CU. While allowing skip encoding only for large CUs may seem to limit the flexibility of the video encoder 114 in selecting the prediction mode of the CU, the ability to further split the CU to obtain smaller CUs where the larger CU would originally be located is available. Furthermore, the expected position of the intra-frame CU in the P or B stripe may be relative to the position where the occluded object becomes visible in the video sequence, as occluded objects tend to require smaller CUs to follow the contours of the occluded area. If the side length of any CU does not exceed 64, step 1120 returns to "normal". As a result, each of the intra-frame candidate mode 1150, inter-frame candidate mode 1160, and skip mode candidate 1170 is evaluated for the candidate CU.

[0205] After evaluating the candidate prediction modes of the CU (i.e., one or more of 1150, 1160, and 1170), the best mode is selected as the candidate CU. The "best" candidate prediction mode is selected based on the lowest rate or distortion cost. This is achieved using methods such as those described in the reference... Figure 6 The described traversal of the coding tree, by comparing the aggregate cost of the CUs obtained from the split with that of a CU in the containing region, enables the determination of the coding tree.

[0206] In another arrangement of method 1100, the restriction to test only skip modes for a given CU is limited to the case where the CU is the result of a ternary split of the CTU. For example, the test modes are restricted if the CU size is 32×128, 64×128 (a split center CU spanning four pipeline processing regions), 128×32, or 128×64 (a split center CU spanning four pipeline processing regions). This restriction on test skip modes reduces the number of cases where skip modes are inferred because multiple prediction modes are tested, and the best one signaled to the CU originates from a binary split of the CTU. In other words, using a CU of size 64×128 or 128×64 will not result in a TU spanning multiple pipeline processing regions. The following description... Figure 12 Steps 1222 and Figure 13 Step 1322 changes accordingly.

[0207] In another arrangement of method 1100, the restriction on testing only the skip mode is limited to the center CU of the ternary split of the CTU. That is, this restriction applies to either a 64×128 CU (split center CU spanning four pipeline processing regions) or a 128×64 CU (divided center CU spanning four pipeline processing regions). The following description... Figure 12 Steps 1222 and Figure 13 Step 1322 changes accordingly.

[0208] Figure 12 This is a flowchart of method 1200 for encoding the resulting CU of the CTU's encoding tree into bitstream 115, wherein the transform size is selected so that the method can be performed in a pipelined architecture, wherein the size of the pipelined processing region is smaller than the CTU size. In method 1200, the transform size is selected so that each transform can be fully processed within a region defined according to the processing grid. Method 1200 can be embodied by a device such as a configured FPGA, ASIC, or ASSP. Alternatively, method 1200 can be performed by video encoder 114 under the execution of processor 205. Thus, method 1200 can be stored on a computer-readable storage medium and / or stored in memory 206. Method 1200 begins at step 1210 of determining the processing region by processor 205.

[0209] At step 1210, the video encoder 114, under the execution of the processor 205, determines the segmentation of an image frame into a grid of equal-sized, square processing regions occupying the entire image frame. A processing region defines a portion of each image frame that is smaller than the size of the CTU. For example, in the case of a 128×128 CTU, the processing region size could be 64×64, or in the case of a 64×64 CTU, the processing region size could be 32×32. In each of these cases, each CTU is segmented into four pipelined processing regions arranged in a 2x2 array. The processing order of the processing regions is set to Z-order. The use of Z-order scanning is consistent with the scanning order of the CUs in the CTU, thus aligning with the order necessary to ensure data dependencies are satisfied when moving from one pipelined processing region to the next. Step 1210 is used to segment the image frame into multiple equal-sized processing regions, each equal-sized processing region being a block processed during a single stage of the pipeline encoding the bitstream. Control in processor 205 proceeds from step 1210 to step 1215, which involves encoding the coding tree.

[0210] In step 1215, where the coding tree is encoded, block partitioner 310, under the execution of processor 205, encodes the coding tree of the CTU determined by method 1100 into bitstream 115. (See reference...) Figure 5 and 6Describe and use Figure 7A and 7B For example, the coding tree breaks down the CTU into one or more CUs based on a series of splits. In method 1100, the block partitioner 310 tests many different combinations of splits to reach a specific coding tree that allows the CTU to be encoded at a high compression ratio while maintaining the fidelity of the decoded image, as shown in the reference. Figure 3 As described. Method 1100 efficiently determines the size of each coding unit (CU) by determining the coding tree. Control in processor 205 proceeds from step 1215 to step 1220 for selecting a CU.

[0211] At step 1220, the block partitioner, executed by processor 205, selects a CU from the coding tree of the CTU. When encoding a CU, step 1220 is performed for each CU in the coding tree encoded in step 1215. The selected CU has a specific size and position within the image frame, and thus a position relative to the upper left corner of the containing CTU. Therefore, it can be said that the selected CU occupies a given area within the containing CTU. Control in processor 205 proceeds from step 1220 to the prediction mode test and encoding step 1222.

[0212] At prediction mode testing and encoding step 1222, processor 205 tests the prediction mode of the selected CU as determined in method 1100. This is done if the frame's stripe type is (i) "P" or "B", or (ii) if a "skip flag" (or "cu_skip_flag") is encoded. The skip flag indicates whether the CU is encoded using skip mode. If the CU is not encoded using skip mode, the "pred_mode" flag is encoded, indicating whether inter-frame prediction or intra-frame prediction is used for the CU.

[0213] Such as about Figure 11 As described in step 1120, a skip pattern can be inferred based on the size of the CU. Therefore, for a “large” CU (for each step 1120), the CU is greater than or equal to the size of one of the processing regions, and the skip pattern is inferred through the implicit size properties of the CU. The CU will effectively overlap with the boundary of one of the processing regions. The skip pattern is inferred accordingly, and the skip flag is not encoded into the bitstream. Furthermore, a merge index is determined based on the implicit properties of the CU. Optionally, if the inferred skip is based on a ternary split or the center CU of a ternary split, the merge index is determined based on the implicit properties of the CU (which is the shape and / or position of the CU). If no skip pattern is inferred, step 1322 determines that the CU is not greater than or equal to the size of one of the processing regions and includes a skip flag. In an arrangement with an inferred skip pattern, the skip flag is encoded only if the CU size is less than a predetermined threshold, as referenced. Figure 11As described in step 1120. For example, the side length does not exceed 64 samples. The skip flag can be encoded only when the temporal layer ID of the frame (e.g., 853) is below a predetermined threshold. For example, the threshold for the temporal ID may be below the maximum temporal layer of the GOP structure size, such as below 4 when the GOP size is sixteen pictures. If the threshold is met (e.g., any CU with a side length greater than 64), the arrangement with the inferred skip pattern does not need to encode the skip flag, because skip encoding is only tested for such cases in method 1100. Furthermore, step 1222 is consistent with method 1100 in that the prediction mode information is encoded only when more than one prediction mode is tested for the CU. Thus, a corresponding reduction in the signaling present in bitstream 115 and therefore higher compression performance can be achieved.

[0214] As shown in the following arrangement, different partitioning of CUs between the "large" and "regular" sets is possible. Having more CUs in the "large" set results in fewer TU instances being stitched together to address pipeline processing issues, at the cost of providing less flexibility to the video encoder 114 when selecting prediction modes for CUs of these sizes.

[0215] In another arrangement of method 1100, the restriction on the side length of the CU to be used for the inference skip mode (step 1120 returns "large") applies to any CU with both sides greater than or equal to 64. The set of CUs that require both sides to be greater than or equal to 64 to result in skip inference is: 128×128, 64×128, and 128×64. Figure 12 Steps 1222 and Figure 13 Step 1322 changes accordingly.

[0216] In another arrangement of method 1100, the restriction on the side length of the CUs used for the inference skip mode (step 1120 returns "large") applies to any CU with a side length greater than 64, resulting in the set of CUs where skip inference occurs as: 128×128, 64×128, 128×64, 128×32, 32×128, 64×64. Again, Figure 12 Steps 1222 and Figure 13 Step 1322 changes accordingly. The threshold (or boundary) between the “large” and “regular” sets can depend on the “operating point” of system 100 (e.g., the desired bit rate of the bit stream). Instead of having a fixed boundary, the boundary can be signaled as a threshold in bit stream 115, thereby allowing video encoder 115 to select the boundary for system 100. The boundary can be signaled as log2 of the side length, and the requirement that “any” or “both” sides of a CU must match the signaled boundary of the CU to be considered in the “large” set can also be signaled.

[0217] If the prediction mode is determined (or inferred) to be a skip mode (step 1222 returns "skip"), then control in processor 205 proceeds from step 1222 to step 1270 to perform motion compensation. Otherwise, (if the prediction mode is inter-frame prediction or intra-frame prediction and step 1222 returns "intra-frame or inter-frame"), control in processor 205 proceeds from step 1222 to step 1225 to identify the processing region.

[0218] At step 1225 of identifying the processing region, processor 205 uses the region of the CU selected in step 1220 to identify which processing region(s) overlaps with the selected CU(s). For example, Figure 10A The CU 1022 overlaps with four 64×64 processing regions in the CTU 1000. Control in the processor 205 proceeds from step 1225 to step 1230, which determines the CU transform size constraint.

[0219] At step 1230, where the CU transformation size constraint is determined, the processor 205 determines the initial transformation size of the CU. The initial transformation size is set to the transformation size of a predetermined set (such as) whose width does not exceed the width of the selected CU and whose height does not exceed the height of the selected CU. Figure 9 The initial transform size is the largest transform size among the selected CUs (such as the transform size of the luminance channel). Therefore, the initial transform size is the largest size "fitted" into the selected CU. Considering the luminance channel, a single transform typically occupies the entire CU. The chroma channel has a similar relationship, where the chroma transform size corresponds to the transform size of the luminance channel adjusted for a 4:2:0 chroma format, and is half the width and height of each transform.

[0220] If a single transform does not completely occupy the CU, a "stitching" process is used at step 1230 to apply an initial transform size to occupy the entire CU. For example, for a CU 1020 with a size of 32×128, an initial transform size of 32×32, stitched together by one to four, is needed to occupy the entire CU. For a CU 1022 with a size of 64×128, the initial transform size is 64×64, and stitching together by one to two is used to occupy the entire CU. Control in processor 205 proceeds from step 1230 to the processing region boundary overlap test step 1235.

[0221] At the processing region boundary overlap test step 1235, processor 205 determines whether any transformations of the initial transform size and associated with the selected CU span two or more processing regions (or "cross" the boundaries of two or more processing regions). In other words, at step 1235, processor 205 determines whether the coding unit overlaps with the boundaries between processing regions. For example, for an initial transform size of 32×32 for CU 1020 located at position (0,0) relative to the top left corner of CTU 1000, each transform is completely contained within a 64×64 pipelined processing region. The top two 32×32 transforms are located in one processing region, and the bottom two 32×32 transforms are located in another processing region. In this case, step 1235 returns "No" and control in processor 205 proceeds to the CU transform size step 1240.

[0222] However, for CU 1022, which has an initial transform size of 64×64 and is located at position (32, 0) relative to the upper left corner of CTU 1000, the initial transform size occupies the region from (32, 0) to (95, 64). When the processing regions are aligned to a 64×64 grid, the initial first transform occupies two processing regions, and the second transform, which occupies the region from (32, 64) to (95, 127), occupies another two processing regions. When at least one of these proposed initial transforms (in) will result in a boundary spanning two or more processing regions... Figure 10A When there are two (in the example), step 1235 returns "yes", and control in processor 205 proceeds from step 1240 to step 1245, which involves resizing the processing region.

[0223] As a general rule, the possibility of a transform spanning two or more processing regions stems from the application of ternary splits at the top level of the coding tree. This is due to the fact that the CTU size, processing region size, and transform side size are all powers of 2, and the CTU size is twice the width and height of the processing region. Therefore, a ternary split at only the top level of the coding tree may result in a CU that spatially (horizontally or vertically) offsets by half the width or height of the processing region. When using transforms with an initial transform size, offsetting the CU may result in a transform that spans two or more processing regions, thus posing a substantial implementation challenge for pipelined architectures that operate at the granularity of processing regions.

[0224] Given the relationship between the CTU size, processing region size, and transform edge size, one solution could be to prohibit ternary splits of regions in the coding tree with edge lengths exceeding 64 samples. The remaining options for regions with edge lengths exceeding 64 samples are no further splits, binary splits, or quadtree splits. Without further splits, concatenating four 64×64 transforms would be possible, each fully contained within the pipelined processing region. If a binary split of a 128×128 CTU is performed in either direction, prohibiting ternary splits in the resulting sub-regions in the opposite direction would prevent possible 64×64 CUs from spanning two pipelined processing regions. However, further splitting the intermediate (64×64) CUs of the ternary split can solve the problem of transform placement in the pipelined processing. Initial prohibition would prevent searching the intermediate coding tree. If a binary split is performed in either direction (resulting in two regions of size 64×128 or 128×64), then due to a side length of 128, applying ternary splits to either resulting region in the same direction would also be impossible.

[0225] However, ternary partitioning in the same direction will not yield any transformations that cross the boundaries between pipelined regions. For example, a horizontal ternary partition of a 128×64 region (derived from a horizontal binary partitioning from CTU) will result in 16×128, 32×128, and another 16×128 region. A transformation of 32, typically applied along a 128-side, is used four times and does not result in any transformations that cross multiple pipelined regions. Finally, if a quadtree partition is performed, the resulting regions will be within separate pipelined regions and will not cause further pipelined problems, regardless of subsequent partitions.

[0226] Therefore, while prohibiting ternary splitting on any region with a side length exceeding 64 is one way to address pipelined processing performance, this prohibition does indeed restrict potentially useful block sizes, thus reducing compression performance. Furthermore, the restriction prevents the use of "skip-mode" inter-frame prediction CUs, which, without residual coefficients, do not introduce pipelined regions due to transform placement. Therefore, this restriction adversely affects compression performance, as flexible placement of skip-mode CUs (e.g., for large CUs) is particularly desirable at low bit rates.

[0227] As described, step 1235 is performed based on the CU size and position within the CTU. Therefore, step 1235 implements an implicit test, thus not increasing the "search space" of the video encoder 114. That is, the video encoder 114 is not given additional degrees of freedom in determining the TU configuration (e.g., the addition of flags). The absence of additional degrees of freedom in determining the TU configuration means that no additional signaling in the bit stream 115 is needed to store the results of the degrees of freedom. In other words, the operation of step 1235 is implicitly based on the nature of the CTU's coding tree. The output of step 1235 is independent of and does not involve the generation of explicit signals related to the TU size to be encoded in the bit stream.

[0228] At the CU transformation size step 1240, processor 205 selects the CU transformation size as determined in step 1230. Since no resulting TU spans multiple pipeline processing regions, there is no need to further divide the CU into additional TUs. Control in processor 205 proceeds from step 1240 to the application of forward transformation and quantization step 1250.

[0229] At the processing region resizing step 1245, the processor 205 determines the resizing size of the selected CU such that the resulting resizing does not cross two or more pipeline processing regions spanned by the selected CU. For example, a CU 1022 of size 64×128 is located at (32, 0) relative to the upper left of the containing CTU 1000. Thus, CU 1022 spans the region from (32, 0) to (95, 127). In the horizontal direction, the CU passes through the pipeline processing region with an X offset of 64. Therefore, in the horizontal direction, the TU width needs to be at most 32 to be the maximum TU width suitable for pipelined implementation. In the vertical direction, the TU width needs to be at most 64 to be the maximum TU width suitable for pipelined implementation. However, as Figure 9 As seen, no 32×64 TU is available. The largest usable TU is 32×32, therefore a 32×32 TU is selected. By selecting a 32×32 TU for CTU 1000, the CTU can be configured as shown in the reference. Figure 10A The process is carried out using the described pipeline method. Step 1245 effectively operates to extract from (e.g., as...) Figure 9 The transform size of the encoding unit is selected from the available set (multiple) of transform sizes (shown). The transform size is selected to fit within the encoding unit and may differ from the size of the processing region. Control in processor 205 proceeds from step 1245 to the application of the forward transform and quantization step 1250.

[0230] At step 1250, where the forward transform and quantization are applied, transform module 326 and quantizer module 334, under the execution of processor 205, apply the transform selected in step 1240 or 1245 to transform the difference 324 and generate residual coefficients 336. If the CU size is equal to the transform size, a single transform is performed. If the CU size is greater than the transform size, the transform is applied in a concatenated manner, such that all differences 324 are transformed. Furthermore, by means of the transform size selected in step 1245, individual transforms do not cover two or more regions spanning the pipeline processing regions.

[0231] Software implementations such as the "reference software" of video compression standards typically process each frame one CTU at a time, rather than using finer-grained processing (such as pipelined processing of regions larger than a CTU). Reference software implementations do not encounter problems such as the pipelined processing region issues identified above because they typically do not run in real-time or on resource-constrained devices. Practical implementations (especially hardware implementations utilizing pipelined architectures and some software implementations) benefit from transformations fully contained within different pipelined processing regions. For example, software implementations that benefit from transformations fully contained within different pipelined regions include multi-core implementations using the same pipelined architecture to improve locality. A significant benefit of transformations fully contained within different pipelined regions is the uniform size and rate of the pipelined processing regions. Control in processor 205 progresses from step 1250 to the encoding residual coefficients step 1255.

[0232] At the residual coefficient encoding step 1255, the entropy encoder 338, under the execution of the processor 205, encodes the residual coefficients from step 1250 into bitstream 115. First, a "root coding block flag" is encoded to indicate the presence of at least one valid residual coefficient obtained from the quantization of step 1250. The root coding block flag is encoded once for the CU, and all color channels of any TB across any TU of the CU signal the validity of any transform of the CU. Assuming that at least one valid residual coefficient exists for any transform across any color channel of the CU, a separate coding block flag is encoded for each transform applied in that color channel. Each coding block flag indicates the presence of at least one valid residual coefficient in the corresponding transform block. For a transform with at least one valid residual coefficient, the valid graph and the size and sign of the valid coefficient are also encoded. Control in the processor 205 proceeds from step 1255 to intra-frame mode test 1260.

[0233] At intra-frame mode test 1260, the processor 205 tests the prediction mode of the selected CU. If the prediction mode is intra-frame prediction (yes at step 1260), control in processor 205 proceeds to intra-frame prediction step 1265. Otherwise (the prediction mode is inter-frame prediction and step 1260 returns no), control in processor 205 proceeds to motion compensation step 1270.

[0234] At the intra-prediction step 1265, the intra-prediction module 364 generates an intra-prediction block (366) of the sample under the execution of the processor 205. The intra-prediction block 366 of the sample is generated using filtered reference samples 362, based on the intra-prediction mode for each PB of the selected CU. When multiple TUs are associated with the CU due to step 1245, intra-reconstruction processing is applied at the boundaries of each TU within the selected CU. In addition to the reconstructed samples at each CU boundary, the reference sample cache 356 is updated using the reconstructed samples at each TU boundary within the CU. Reconstruction at the TU boundaries within the CU allows the residual coefficients of TUs above or to the left of the current TU within the CU to contribute to the generation of a reference sample for a portion of the PB at the same position as the current TU. Therefore, reconstruction at the TU boundaries within the CU can reduce distortion and improve compression efficiency. Control in the processor 205 proceeds from step 1265 to the CU reconstruction step 1275.

[0235] At motion compensation step 1270, motion compensation module 380, under the execution of processor 205, generates filtered block samples 382. Filtered block samples 382 are generated by extracting one or two sample blocks 374 from frame buffer 372. For each sample block, a frame is selected based on a reference image index, and the spatial displacement of a pixel relative to the selected CU is specified based on a motion vector. For each sample block extracted from frame buffer 372, filtering is applied based on the "sub-pixel" displacement portion of the motion vector. The precision of the sub-pixel displacement portion of the motion vector can be one-quarter pixel precision or one-sixteenth pixel precision. In the case of using two blocks, the resulting filtered blocks are blended together. The reference image index and one or more motion vectors(s) are determined in method 1100. Control in processor 205 proceeds from step 1270 to CU reconstruction step 1275.

[0236] At CU reconstruction step 1275, summing module 352, executed by processor 205, generates reconstruction sample 354 by adding residual sample 350 to PU 320 used for inter-frame prediction or intra-frame prediction CUs. For skip mode CUs, no residual sample exists, and therefore reconstruction sample 354 is derived from PU 320. Reconstruction sample 354 can be used as a reference for subsequent intra-frame prediction CUs in the current frame. After applying intra-loop filtering (i.e., applying intra-loop filter 368), reconstruction sample 354 is written to frame buffer 372 for reference by inter-frame prediction CUs in subsequent frames. Deblocking filtering of intra-loop filter 368 is applied to the internal boundaries of the CU. That is, deblocking filtering is applied to the boundaries between TUs within the CU, which are generated by stitching due to both the CU size and the pipelined processing region boundaries. Control in processor 205 proceeds from step 1275 to final CU testing step 1285.

[0237] At the final CU test step 1285, the processor tests whether the selected CU is the last one in the CTU. If not ("No" in step 1160), control in processor 205 returns to step 1215. If the selected CU is the last one in the CTU scanned in the CU scan order, i.e., depth-first Z-order, then method 1200 terminates. After method 1200 terminates, the next CTU is encoded, or the video encoder 114 advances to the next image frame of the video.

[0238] Figure 13 A method 1300 for decoding a CU of a CTU from a bitstream 133 is shown. In method 1300, a transformation size is selected such that method 1300 can be performed in a pipelined architecture. The size of the corresponding pipelined processing region is smaller than the CTU size, and the rate of the pipelined processing region is independent on the coding tree of each CTU. Method 1300 can be embodied by a device such as a configured FPGA, ASIC, or ASSP. Alternatively, method 1300 can be performed by a video decoder 134 under the execution of processor 205. Thus, method 1300 can be stored on a computer-readable storage medium and / or stored in memory 206. Method 1300 begins at step 1310 of determining the processing region by processor 205.

[0239] At step 1310, in determining the processing region, the video decoder 134, under the execution of the processor 205, determines a grid to segment the image frame of the bitstream into square processing regions of equal size occupying the entire image frame. Step 1310 determines the segmentation of the image frame in a manner consistent with step 1210. Step 1310 is used to segment the image frame into multiple equal-sized processing regions, each of which is a block processed during a single stage of the decoding bitstream pipeline. Control in the processor 205 proceeds from step 1310 to step 1315, which decodes the coding tree.

[0240] At step 1315, which decodes the encoding tree, the entropy decoder 420, under the execution of the processor 205, decodes the encoding tree of the CTU from the bit stream 133. The encoding tree decomposes the CTU into one or more CUs according to a series of splits, as shown in the reference. Figure 5 and 6 And use Figure 7A and 7B The example described above. The code tree decoded from bitstream 133 is in Figure 12 Step 1215 determines the coding tree. Step 1315 efficiently determines the size of each coding unit (CU) by decoding the CTU using the coding tree. Control in processor 205 proceeds from step 1315 to CU selection step 1320.

[0241] At step 1320, the video decoder 134, under the execution of the processor 205, selects a CU from the decoded coding tree by iteratively traversing the coding tree in a forward direction corresponding to the direction in which the syntax associated with the coding tree exists in the bit stream 134. The forward direction involves a Z-order scan. The selected CU has a specific size and position in the image frame, and thus a position relative to the upper left corner containing the CTU. Thus, it can be said that the selected CU occupies a given area within the containing CTU. Control in the processor 205 proceeds from step 1320 to step 1322, which determines the prediction mode test.

[0242] At prediction mode determination test step 1322, processor 205 determines the prediction mode of the selected CU. If the frame stripe type is "P" or "B", entropy decoder 420 decodes a "skip flag" (or "cu_skip_flag") indicating whether skip mode is used to encode the CU. If skip mode is not used to encode the CU, entropy decoder 420 decodes the "pred_mode" flag. The "pred_mode" flag indicates which of inter-frame prediction or intra-frame prediction is used for the CU. (See also: Regarding...) Figure 11As described in step 1120, the skip pattern can be inferred based on the size of the CU. Therefore, for a “large” CU (for each step 1120), the CU is greater than or equal to the size of one of the processing regions, and the skip pattern is inferred through the implicit size properties of the CU. The skip pattern is inferred accordingly, and the skip flag is not encoded into the bitstream. Instead, a merge index is determined based on the implicit properties of the CU. Optionally, if the inferred skip is based on a ternary split or the center CU of a ternary split, the merge index is determined based on the implicit properties of the CU (which is the shape and / or position of the CU). If no skip pattern is inferred, step 1322 determines that the CU is not greater than or equal to the size of one of the processing regions and includes a skip flag.

[0243] In an arrangement with the inferred skip mode, the skip flag is decoded only if the CU size is less than a predetermined threshold (e.g., if neither side has more than 64 samples). Otherwise, the CU is judged as a "large CU," and the skip mode is inferred to be in use. The skip flag can be encoded only if the temporal layer ID is less than a predetermined threshold (e.g., less than the maximum temporal layer of the GOP structure size, e.g., less than four when the GOP size is 16 pictures). If the threshold test is met (e.g., a large CU size and / or temporal layer ID above the threshold), the arrangement with the inferred skip mode does not need to decode the skip flag i, because only the skip encoding is tested in method 1100 for such cases. Thus, the prediction mode is judged to be a skip mode. Furthermore, step 1322 is consistent with method 1100 in that the prediction mode information is decoded only when more than one prediction mode is tested for the CU. When only one prediction mode is tested for the CU, the video decoder 134 infers the prediction mode based on, for example, the CU size, rather than explicitly decoding the prediction mode.

[0244] If the prediction mode is determined (or inferred) to be a skip mode ("skip" at step 1322), control in processor 205 proceeds from step 1322 to step 1370 for decoding motion parameters. Otherwise (if the prediction mode is inter-frame prediction or intra-frame prediction), step 1322 returns to "intra-frame or inter-frame," and control in processor 205 proceeds to step 1325 for identifying the processing region.

[0245] At step 1325 of identifying the processing region, processor 205 uses the region of the CU selected in step 1320 to identify which processing region(s) overlaps with the selected CU(s). For example, Figure 10A The encoding unit 1022 overlaps with four 64×64 processing regions in CTU 1000. Step 1325 is to... Figure 12Step 1225 operates in a similar manner. Control in processor 205 proceeds from step 1325 to step 1330, which determines the encoding unit transform size constraint.

[0246] At step 1330, where the encoding unit transform size constraint is determined, processor 205 determines the initial transform size of the CU. The initial transform size is set in a similar manner to that determined in step 1230. Control in processor 205 then proceeds from step 1330 to step 1335, where the processing region boundary overlap test is performed.

[0247] Similar to overlap test step 1235, at processing region boundary overlap test step 1335, processor 205 determines whether any transforms of the initial transform size and associated with the selected CU span two or more processing regions. In other words, step 1335 determines whether the coding unit overlaps with the boundaries between processing regions. If each transform is completely contained within a processing region ("No" at step 1335), control in processor 205 proceeds to CU transform size step 1340. If at least one of the transforms originating from the initial transform size spans or "crosses" the boundaries between two or more processing regions ("Yes" at step 1335), control in processor 205 proceeds to processing region transform size step 1345. The result of test step 1335 depends on the CU size and its position within the CTU, which are entirely described by the coding tree of the CTU. Thus, it is not necessary to decode additional signaling from bitstream 133 to determine whether the CU spans two processing regions. Instead, the implicit properties of the CU (size and position) are used to test whether the processing region boundaries overlap.

[0248] At the CU transformation size step 1340, the processor 205 selects the CU transformation size as determined in step 1330 based on the transformation size selection in step 1240. Control in the processor 205 then proceeds from step 1340 to the decoding residual coefficient step 1350.

[0249] At the processing region transformation size step 1345, processor 205 determines the transformation size of the selected CU such that the resulting transformation does not cross two or more pipeline processing regions spanned by the selected CU. Step 1345 operates according to the transformation size selection in step 1245. Step 1345 effectively operates from (e.g., as...) Figure 9 The transform size of the encoding unit is selected from the available set (multiple) of transform sizes (shown). The transform size is selected to fit within the encoding unit and may differ from the size of the processing region. Control in processor 205 proceeds from step 1345 to the decoding residual coefficients step 1350.

[0250] At the residual coefficient decoding step 1350, the entropy decoder 420 decodes the residual coefficients from the bitstream 115 under the execution of the processor 205. The encoding unit is the residual coefficients of each transform unit within the encoding unit by applying an inverse transform. When decoding the residual coefficients, the "root coding block flag" is decoded first. The root coding block flag indicates the presence of at least one valid residual coefficient in any TU of the CU (i.e., across all color channels). When the root coding block flag indicates the presence of a valid residual coefficient in the CU, a separate coding block flag is decoded for each transform applied in that color channel within each color channel. Each coding block flag indicates the presence of at least one valid residual coefficient in the corresponding transform. For a transform with at least one valid residual coefficient, the valid graph and the size and sign of the valid coefficients are also decoded. Control in the processor 205 proceeds from step 1350 to the inverse quantization and inverse transform application step 1355.

[0251] At inverse quantization and inverse transform application step 1355, dequantizer module 428 and inverse transform module 444, under the execution of processor 205, inverse quantize the residual coefficients to produce scaled transform coefficients 440. At step 1355, the scaled transform coefficients 440 are transformed by applying the transform selected in step 1340 or step 1345 to produce residual samples 448. As in step 1250, the transformation is applied in a concatenated manner according to the determined transform size. Furthermore, by means of the transform size selected at step 1345, individual transforms do not cover two or more regions spanning pipelined processing regions. As with method 1200, practical implementations (especially hardware implementations utilizing pipelined architectures and some software implementations) benefit from transforms fully contained within different pipelined processing regions. An example software implementation beneficial to the described arrangement is a multi-core implementation that can use the same pipelined architecture to improve data locality. Control in processor 205 proceeds from step 1355 to intra-frame mode testing step 1360.

[0252] At intra-frame mode test 1360, processor 205 tests the determined prediction mode of the selected CU. If the prediction mode is intra-frame prediction ("Yes" at step 1360), control in processor 205 proceeds to intra-frame prediction step 1365. Otherwise (prediction mode is inter-frame prediction), step 1360 returns "No" and control in processor 205 proceeds to decoding motion parameters step 1370.

[0253] At intra-prediction step 1365, intra-prediction module 476 generates intra-prediction sample blocks (480) under the execution of processor 205. Intra-prediction sample blocks 480 are generated using filtered reference samples 472, based on the intra-prediction mode for each PB of the selected CU. When multiple TUs are associated with a CU due to step 1345, intra-reconstruction processing is applied at the boundaries of each TU within the selected CU. In addition to the reconstructed samples at each CU boundary, the reconstructed samples at the boundaries of each TU within the CU are used to update the reconstructed sample cache 460. The reconstruction at the TU boundaries within the CU allows the residuals of TUs above or to the left of the current TU within the CU to contribute to the generation of reference samples for a portion of a PB located at the same position as the current TU. The reconstruction at the TU boundaries within the CU can be operated to reduce distortion and improve compression efficiency. Control in processor 205 proceeds from step 1365 to CU reconstruction step 1380.

[0254] At step 1370, the entropy decoder 420, under the execution of processor 205, decodes the motion vector of the selected CU. Decoding the motion vector includes selecting the motion vector by: (i) decoding the merge index if a skip mode is inferred (as identified from the properties of the CU at 1120 and 1322), or (ii) decoding the skip flag to decode the merge index if no skip mode is inferred from the CU. A list of candidate motion vectors (referred to as the “merge list”) is created using spatially and temporally adjacent blocks. The merge index is decoded from bitstream 133 to select one of the candidates from the merge list. The merge index may be determined based on the implicit properties of the CU (as described above with respect to step 1322) or by decoding the split mode flag from the bitstream. If the selected CU is encoded using a skip mode, the selected candidate becomes the motion vector of the CU. If the selected CU is encoded using inter-frame prediction, the motion vector increment is decoded from bitstream 133 and added to the candidate selected based on the decoded merge index. The control in the processor proceeds from step 1370 to step 1375 to perform motion compensation.

[0255] At motion compensation step 1375, motion compensation module 434 generates filtered block samples 438 under the execution of processor 205. Filtered block samples 438 are generated by extracting one or two sample blocks 498 from frame buffer 496. For each sample block 498, a frame is selected according to a reference image index, and the spatial displacement of the pixel relative to the selected CU is specified according to the motion vector. For each sample block extracted from frame buffer 372, filtering is applied according to the "sub-pixel" displacement portion of the motion vector. The precision of the sub-pixel displacement portion of the motion vector can be one-quarter pixel precision or one-sixteenth pixel precision. In the case of using two blocks, the resulting filtered blocks are blended together. The reference image index and (one or more) motion vectors are decoded from bitstream 133 and determined in method 1100. Control in processor 205 proceeds from step 1375 to CU reconstruction step 1380.

[0256] At CU reconstruction step 1380, summation module 450 generates reconstruction sample 456 under the execution of processor 205. Reconstruction sample 456 is generated by adding residual sample 448 to PU 452 used for inter-frame prediction or intra-frame prediction CUs. For skip mode CUs, there is no residual, so reconstruction sample 456 is derived from PU 452. Reconstruction sample 456 can be used as a reference for subsequent intra-frame prediction CUs in the current frame. After applying in-loop filtering (i.e., applying in-loop filter 488), reconstruction sample 456 is written to frame buffer 496 for reference by inter-frame prediction CUs in subsequent frames. Deblocking filtering of loop filter 488 is applied to the internal boundaries of the CU. That is, deblocking filtering is applied to the boundaries between TUs within the CU (due to both CU size and pipeline processing region boundaries). Control in processor 205 proceeds from step 1380 to final CU testing step 1385.

[0257] At the final CU test step 1385, processor 205 tests whether the selected CU is the last CU in the CTU in CU scan order (depth-first Z-order scan). If not (No at step 1385), control in processor 205 returns to step 1315. If the selected CU is the last CU in the CTU (Yes at step 1385), method 1300 terminates. After method 1300 terminates, the next CTU is decoded, or video decoder 134 advances to the next image frame of the bitstream.

[0258] In an alternative arrangement of video encoder 114 and video decoder 134, CUs spanning multiple pipeline processing regions are inferred to be encoded in a "skip mode," thus having no associated residual coefficients and therefore requiring no transformation to encode or decode such blocks. Thus, when video encoder 114 determines the coding tree at step 1215, if testing yields such CUs spanning multiple processing regions (TUs), these CUs need to be encoded without any associated residual coefficients.

[0259] Industrial availability

[0260] The described configuration is applicable to the computer and data processing industries, and is particularly suitable for digital signal processing that encodes or decodes signals such as video and image signals, thereby achieving high compression efficiency without incurring excessive costs in terms of silicon area and memory consumption due to the possibility of providing a pipelined implementation with a processing region size smaller than the maximum support block size or CTU size. In some implementations, the arrangement is useful for the VVC standard because the splicing of the implementation region TU (e.g., as implemented at steps 1145 and 1245) helps prevent pipeline inefficiencies, especially for inter-frame prediction modes. As mentioned above, some implementations described herein allow the use of ternary coding trees for larger CUs, or the use of 64×64 CUs while reducing the impact on processing time and / or quality.

[0261] The foregoing only illustrates some embodiments of the present invention, and modifications and / or changes can be made to the present invention without departing from the scope and spirit of the present invention, wherein these embodiments are merely exemplary and not restrictive.

[0262] In the context of this specification, the word "comprising" means "primarily but not necessarily only" or "has" or "includes," rather than "consisting of only." Suffixes such as "comprise" and "comprises" in the word "comprising" have corresponding variations in meaning.

[0263] Cross-reference to related applications

[0264] This application claims priority to Australian Patent Application 2018217336, filed on August 17, 2018, pursuant to 35 USC § 119, the entire contents of which are incorporated herein by reference for all purposes.

Claims

1. An image decoding method for decoding encoded units from a bitstream according to a predetermined method, the image decoding method comprising: Decode information from the bitstream for determining the coding unit in the coding tree unit, wherein, in the predetermined manner, at least one edge of the coding unit can be 128 samples; With the first constraint applied, the transformation unit in the encoding unit is determined such that even when at least one side of the encoding unit has 128 samples, the maximum size that can be selected as the magnitude of the luminance component for the transformation unit is 32 samples. Under the second constraint, the transformation unit in the encoding unit is determined such that even when at least one side of the encoding unit has 128 samples, the maximum size that can be selected as the size of the luminance component for the transformation unit is 64 samples; and The encoding unit is decoded by performing an inverse transform based on the determined transform unit.

2. The method according to claim 1, wherein, At least one edge of the encoding unit is equal to 128 samples.

3. The method according to claim 1, wherein, Ternary splitting can be used to determine the coding unit in the coding tree unit.

4. The method according to claim 1, wherein, The shape of each transformation unit in the transformation unit is different from the shape of the encoding unit.

5. The method according to claim 1, wherein, The aspect ratio of each transformation unit in the transformation unit is different from that of the encoding unit.

6. The method according to claim 1, wherein, The first constraint and the second constraint are applied alternatively.

7. An image encoding method for encoding encoding units into a bitstream according to a predetermined method, the image encoding method comprising: Determine the coding unit in the coding tree unit, wherein, in the predetermined manner, at least one edge of the coding unit can be 128 samples; With the first constraint applied, the transformation unit in the encoding unit is determined such that even when at least one side of the encoding unit has 128 samples, the maximum size that can be selected as the magnitude of the luminance component for the transformation unit is 32 samples. Under the second constraint, the transformation unit in the encoding unit is determined such that even when at least one side of the encoding unit has 128 samples, the maximum size that can be selected as the size of the luminance component for the transformation unit is 64 samples; and The encoding unit is encoded by performing a transformation based on the determined transformation unit.

8. The method according to claim 7, wherein, At least one edge of the encoding unit is equal to 128 samples.

9. The method according to claim 7, wherein, Ternary splitting can be used to determine the coding unit in the coding tree unit.

10. The method according to claim 7, wherein, The shape of each transformation unit in the transformation unit is different from the shape of the encoding unit.

11. The method according to claim 7, wherein, The aspect ratio of each transformation unit in the transformation unit is different from that of the encoding unit.

12. The method according to claim 7, wherein, The first constraint and the second constraint are applied alternatively.

13. An image decoding apparatus for decoding encoded units from a bitstream according to a predetermined manner, the image decoding apparatus comprising: A decoding unit configured to decode from the bitstream information for determining coded units within a coded tree unit, wherein, in the predetermined manner, at least one edge of the coded unit can be 128 samples; and A determining unit is configured, under the application of a first constraint, to determine the transform unit in the encoding unit such that, even when at least one side of the encoding unit has 128 samples, the maximum size that can be selected as the magnitude of the luminance component for the transform unit is 32 samples. Wherein, when the second constraint is applied, the determining unit is configured to determine the transformation unit in the encoding unit in such a way that even when at least one side of the encoding unit has 128 samples, the maximum size that can be selected as the size of the luminance component for the transformation unit is 64 samples, and The decoding unit is configured to decode the encoding unit by performing an inverse transformation based on the determined transformation unit.

14. A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image decoding method for decoding encoded units from a bit stream according to a predetermined manner, the image decoding method comprising: Decode information from the bitstream for determining the coding unit in the coding tree unit, wherein, in the predetermined manner, at least one edge of the coding unit can be 128 samples; With the first constraint applied, the transformation unit in the encoding unit is determined such that even when at least one side of the encoding unit has 128 samples, the maximum size that can be selected as the magnitude of the luminance component for the transformation unit is 32 samples. Under the second constraint, the transformation unit in the encoding unit is determined such that even when at least one side of the encoding unit has 128 samples, the maximum size that can be selected as the size of the luminance component for the transformation unit is 64 samples; and The encoding unit is decoded by performing an inverse transform based on the determined transform unit.

15. An image encoding apparatus for encoding encoding units into a bitstream according to a predetermined method, the image encoding apparatus comprising: The first determining unit is configured to determine a coding unit in a coding tree unit, wherein, in the predetermined manner, at least one edge of the coding unit can be 128 samples. The second determining unit is configured, under the application of the first constraint, to determine the transformation unit in the encoding unit such that even when at least one side of the encoding unit has 128 samples, the maximum size that can be selected as the magnitude of the luminance component for the transformation unit is 32 samples. Wherein, when the second constraint is applied, the second determining unit is configured to determine the transformation unit in the encoding unit in such a way that even when at least one side of the encoding unit is 128 samples, the maximum size that can be selected as the size of the luminance component for the transformation unit is 64 samples; and An encoding unit is configured to encode the encoding unit by performing a transformation based on a determined transformation unit.

16. A non-transitory computer-readable storage medium storing a program for causing a computer to execute an image encoding method for encoding encoding units into a bitstream according to a predetermined manner, the image encoding method comprising: Determine the coding unit in the coding tree unit, wherein, in the predetermined manner, at least one edge of the coding unit can be 128 samples; With the first constraint applied, the transformation unit in the encoding unit is determined such that even when at least one side of the encoding unit has 128 samples, the maximum size that can be selected as the magnitude of the luminance component for the transformation unit is 32 samples. Under the second constraint, the transformation unit in the encoding unit is determined such that even when at least one side of the encoding unit has 128 samples, the maximum size that can be selected as the size of the luminance component for the transformation unit is 64 samples; and The encoding unit is encoded by performing a transformation based on the determined transformation unit.

Images