Motion compensation using size of reference picture

Abstract

A video decoder is configured to use motion information to determine a reference block of a reference image for predicting a current block of a current image, and to generate a reference sample set for the current block of the current image. To generate the reference sample set, the video decoder is configured to perform reference sample cropping on the reference block of the reference image based on the size of the reference image. The video decoder is further configured to generate a predicted block for the current block of the current image based on the reference sample set.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Application No. 17 / 125,486, filed December 17, 2020, which claims the benefits of U.S. Provisional Application No. 62 / 951,709, filed December 20, 2019, and U.S. Provisional Application No. 62 / 954,204, filed December 27, 2019, the entire contents of each of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to video encoding and video decoding. Background Technology

[0004] Digital video functionality can be incorporated into a wide variety of devices, including digital televisions, digital direct broadcasting systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite wireless phones, so-called "smartphones," video conferencing equipment, and video streaming devices. Digital video devices implement video decoding technologies, such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High-Efficiency Video Coding (HEVC), and extensions to such standards. By implementing such video decoding technologies, video devices can more efficiently transmit, receive, encode, decode, and / or store digital video information.

[0005] Video decoding techniques include spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove inherent redundancy in video sequences. For block-based video decoding, video slices (e.g., video pictures or portions of video pictures) can be divided into video blocks, which may also be referred to as decoding tree units (CTUs), decoding units (CUs), and / or decoding nodes. Video blocks in an intra-frame decoded (I) slice of a picture are encoded using spatial prediction with reference samples from adjacent blocks within the same picture. Video blocks in an inter-frame decoded (P or B) slice of a picture can use spatial prediction with reference samples from adjacent blocks within the same picture or temporal prediction with reference samples from other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame. Summary of the Invention

[0006] In summary, this disclosure describes techniques for improving the accuracy of video decoding when the current image and a reference image have different sizes and / or when using sub-images. For example, when generating a set of reference samples, a video decoder (e.g., a video encoder or video decoder) can perform reference sample cropping on a reference block having a different number of samples than the current image. In this example, when cropping the reference block, the video decoder can use the size of the reference image instead of the current image size to crop the reference samples. For example, a specific edge reference sample can be positioned at the rightmost position of the reference image size (e.g., picW-1). In this case, the video decoder can set positions for all samples to the right of that specific edge reference sample to set a value equal to that specific edge reference sample. In this way, the video decoder can "crop" the reference sample positions to be within the reference image. In other words, the video decoder can effectively "fill" the reference samples to the right of that specific edge reference sample using the specific edge reference sample value.

[0007] In one example, this disclosure describes a method for decoding video data, the method comprising: using motion information to determine a reference block of a reference image for predicting a current block of a current image; generating a reference sample set for the current block of the current image, wherein generating the reference sample set comprises: performing reference sample cropping on the reference block of the reference image based on the size of the reference image; and generating a predicted block for the current block of the current image based on the reference sample set.

[0008] In another example, this disclosure describes an apparatus including a memory configured to store video data, and one or more processors implemented in circuitry and configured to: determine a reference block of a reference image for predicting a current block of a current image using motion information. The one or more processors are further configured to generate a reference sample set for the current block of the current image, wherein, in order to generate the reference sample set, the one or more processors are configured to perform reference sample cropping on the reference block of the reference image based on the size of the reference image. The one or more processors are further configured to generate a predicted block for the current block of the current image based on the reference sample set.

[0009] In another example, this disclosure describes a computer-readable storage medium having instructions stored therein, which, when executed, cause a processor to use motion information to determine a reference block of a reference image for predicting a current block of a current image. The instructions further cause the processor to generate a reference sample set for the current block of the current image, wherein, in order to generate the reference sample set, the instructions cause the processor to perform reference sample cropping on the reference block of the reference image based on the size of the reference image; and to generate a predicted block for the current block of the current image based on the reference sample set.

[0010] In one example, this disclosure describes an apparatus comprising: a unit for determining, using motion information, a reference block of a reference image for predicting a current block of a current image. The apparatus further comprises: a unit for generating a reference sample set for the current block of the current image, wherein the unit for generating the reference sample set comprises: performing reference sample cropping on the reference block of the reference image based on the size of the reference image. The apparatus further comprises: a unit for generating a predicted block for the current block of the current image based on the reference sample set.

[0011] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0012] Figure 1 This is a block diagram illustrating an example video encoding and decoding system that can perform the techniques described in this disclosure.

[0013] Figure 2A and 2B This is a conceptual diagram illustrating an example quadtree binary tree (QTBT) structure and its corresponding decoding tree unit (CTU).

[0014] Figure 3 This is a block diagram illustrating an example video encoder that can perform the techniques described in this disclosure.

[0015] Figure 4 This is a block diagram illustrating an example video decoder that can perform the techniques described in this disclosure.

[0016] Figure 5 This is a concept map illustrating the reordering of sub-images.

[0017] Figure 6 This is a concept map illustrating the extracted sub-images after the sub-images have been reordered.

[0018] Figure 7 This is a flowchart illustrating an example method for encoding the current block.

[0019] Figure 8 This is a flowchart illustrating an example method for decoding the current block of video data.

[0020] Figure 9 This is a flowchart illustrating an example method for generating prediction blocks using the size of a reference image.

[0021] Figure 10 This is a flowchart illustrating an example method for generating prediction blocks using reference sub-image boundaries. Detailed Implementation

[0022] This disclosure describes techniques for improving the accuracy of video decoding when the current picture and a reference picture have different sizes and / or when using sub-pictures. In “Versatile Video Coding (Draft 7)” by Bross et al., Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 16th Meeting: Geneva, CH, 1-11 October 2019, JVET-P2001-v14 (hereinafter referred to as “VVC Draft 7”), techniques are described that allow prediction of the current picture using a reference picture with a different number of samples (e.g., resolution) than the current picture. That is, instead of using I-frames that do not use other frames for prediction, the video decoder (e.g., video encoder or video decoder) can use prediction frames (e.g., P-frames) or B-frames (bidirectional prediction frames) when the resolution changes. Using P-frames or B-frames can potentially reduce the amount of video data signal compared to using I-frames.

[0023] In video decoding, the current block in the current image can have a motion vector indicating a position in the reference image outside the image boundary of the reference image. This can occur in various situations, such as when a merge mode is used to determine the motion vector of the current block and the current block inherits motion vectors from neighboring blocks of the current image. In this case, the motion vectors of the neighboring blocks might indicate a position within the image boundary of the reference image. However, when the motion vectors of the neighboring blocks are taken to start from the current block, the resulting motion vector might indicate a position outside the image boundary of the reference image. Thus, a video decoder (e.g., a video encoder or video decoder) can perform reference sample cropping (which may also be referred to herein as "padded reference samples") to generate reference samples. In some cases, the term "cropping" refers to rounding, i.e., reducing the bit depth by a certain value. However, in this disclosure, the term cropping is generally used to refer to the process in which the video decoder replaces the sample value of the first sample (e.g., a non-usable pixel) with the sample value of the second sample (e.g., a usable sample) at the position of the first and second samples. That is, the term "cropping" in this disclosure can generally refer to rounding the position of a sample rather than rounding the bit depth value. For example, a video decoder can use the nearest sample value located within the image boundary of a reference image. For example, it can be assumed that each sample above a sample located just within the image boundary has the value of a sample within the image boundary. A similar principle can be applied to samples located to the left, right, and bottom of the image boundary of the reference image.

[0024] One or more problems may arise when performing reference sample cropping using a reference image with a different size than the current image. For example, when generating prediction blocks for the current image using a reference image with a different number of samples (e.g., resolution) than the current image, a video decoder (e.g., a video encoder or video decoder) may apply image boundaries of the reference image that can reduce decoding accuracy. For instance, if the image boundaries are generated using the current image size with fewer samples than the reference image, the video decoder may perform reference sample cropping on samples within the size of the reference image. In another example, if the image boundaries are generated using the current image size with more samples than the reference image, the video decoder may not perform reference sample cropping on samples outside the size of the reference image (e.g., unavailable reference samples).

[0025] This disclosure describes example techniques that can represent one or more solutions to the problems identified above and / or one or more other problems. For example, when performing reference sample cropping on a reference block, a video decoder (e.g., a video encoder or video decoder) can perform reference sample cropping using the size of a reference image instead of the current image size. For example, a specific edge reference sample can be positioned at the rightmost position of the reference image size (e.g., picW-1). In this case, the video decoder can set the positions of all samples to the right of that specific edge reference sample to correspond to the specific edge reference sample. In this way, the video decoder can "crop" the reference sample positions to be within the size of the reference image. In other words, the video decoder can "fill" the reference samples located to the right of the specific edge reference sample (e.g., outside the size of the reference image) with the specific edge reference sample values. Performing reference sample cropping using the size of the reference image can improve the decoding accuracy of the video decoder compared to techniques that perform reference sample cropping using the current image size.

[0026] In some examples, a video decoder (e.g., a video encoder or video decoder) can perform a surround processing to generate a set of reference samples. This surround processing, also referred to herein as “surround motion compensation,” can be used when decoding omnidirectional video content (also known as 360° video content). The surround processing can utilize video content surround in 360° video content for inter-frame prediction purposes. For example, a video decoder (e.g., a video encoder or video decoder) can generate a prediction block for inter-frame prediction of the current block based on samples along the relative boundaries of a reference image. For example, the prediction block could include samples from portions of the reference image that extend beyond the left boundary of the reference image and wrap back to the right boundary of the reference image. In this example, the prediction block includes samples located near the left boundary of the reference image and samples from near the right boundary of the reference image.

[0027] One or more problems may arise when performing wrap-around processing using a reference image with a different size than the current image. For example, when generating prediction blocks for the current image using a reference image with a different number of samples (e.g., resolution) than the current image, a video decoder (e.g., a video encoder or video decoder) may apply image boundaries of the reference image that can reduce decoding accuracy. For instance, if the image boundaries are generated using the current image size with fewer samples than the reference image, the video decoder may perform wrap-around processing on samples within the size of the reference image. In another example, if the image boundaries are generated using the current image size with more samples than the reference image, the video decoder may not perform wrap-around processing on samples outside the size of the reference image (e.g., unavailable reference samples).

[0028] This disclosure describes example techniques that can represent one or more solutions to the problems identified above and / or one or more other problems. For example, when performing wrap processing on a reference block, a video decoder (e.g., a video encoder or video decoder) can perform wrap processing based on the size of a reference image instead of the current image size. For example, the video decoder can generate samples for one or more sets of reference samples based on the size of the reference image and a wrap offset. Thus, a prediction block can be generated using reference samples from portions of the reference image, such as those extending beyond the left boundary of the reference image size and wrapping back to the right boundary of the reference image size. Performing wrap processing using the size of the reference image can improve the decoding accuracy of the video decoder compared to techniques that perform wrap processing using the current image size.

[0029] Figure 1 This is a block diagram illustrating an example video encoding and decoding system 100 that can perform the techniques of this disclosure. In general, the techniques of this disclosure are directed to decoding (encoding and / or decoding) video data. In general, video data includes any data used for processing video. Therefore, video data can include raw, unencoded video, encoded video, decoded (e.g., reconstructed) video, and video metadata such as signaling data.

[0030] As in Figure 1 As illustrated in this example, system 100 includes a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116. Specifically, source device 102 provides video data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 can include any of a variety of devices, including desktop computers, laptop computers, tablet computers, set-top boxes, mobile phones such as smartphones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, source device 102 and destination device 116 may be configured for wireless communication and may therefore be referred to as wireless communication devices.

[0031] exist Figure 1In the example, source device 102 includes a video source 104, memory 106, video encoder 200, and output interface 108. Destination device 116 includes an input interface 122, video decoder 300, memory 120, and display device 118. According to this disclosure, the video encoder 200 of source device 102 and the video decoder 300 of destination device 116 can be configured to apply techniques for generating predictive information using one or more boundaries of a reference sub-picture. Therefore, source device 102 represents an example of a video encoding device, and destination device 116 represents an example of a video decoding device. In other examples, the source and destination devices may include other components or arrangements. For example, source device 102 may receive video data from an external video source such as an external camera. Similarly, destination device 116 may interface with an external display device, rather than including an integrated display device.

[0032] like Figure 1 The system 100 shown is merely an example. In general, any digital video encoding and / or decoding device can perform techniques for generating predictive information using one or more boundaries of a reference sub-picture. Source device 102 and destination device 116 are merely examples of decoding devices in which source device 102 generates decoded video data for transmission to destination device 116. The term "decoding device" as used in this disclosure refers to a device that performs the decoding (encoding and / or decoding) of data. Therefore, video encoder 200 and video decoder 300 represent examples of decoding devices, specifically examples of a video encoder and a video decoder, respectively. In some examples, source device 102 and destination device 116 may operate in a substantially symmetrical manner such that each of source device 102 and destination device 116 includes video encoding and decoding components. Therefore, system 100 can support one-way or two-way video transmission between source device 102 and destination device 116, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0033] In summary, video source 104 represents the source of video data (i.e., raw, unencoded video data) and provides a sequential sequence of pictures (also referred to as "frames") of video data to video encoder 200, which encodes the data of the pictures. Video source 104 of source device 102 may include a video capture device such as a video camera, a video archive containing previously captured raw video, and / or a video feed interface for receiving video from a video content provider. As a further alternative, video source 104 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes the captured, pre-captured, or computer-generated video data. Video encoder 200 may rearrange the pictures from the received order (sometimes referred to as "display order") into a decoding order for decoding. Video encoder 200 may generate a bitstream comprising the encoded video data. The source device 102 can then output the encoded video data to a computer-readable medium 110 via the output interface 108 so that it can be received and / or retrieved by, for example, the input interface 122 of the destination device 116.

[0034] The memory 106 of source device 102 and the memory 120 of destination device 116 represent general-purpose memory. In some examples, memories 106 and 120 may store raw video data, such as raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106 and 120 may store software instructions executable by, for example, video encoder 200 and video decoder 300. Although memories 106 and 120 are shown in this example as separate from video encoder 200 and video decoder 300, it should be understood that video encoder 200 and video decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106 and 120 may store, for example, encoded video data output from video encoder 200 and input to video decoder 300. In some examples, portions of memories 106 and 120 may be allocated, for example, for storing one or more video buffers for storing raw, decoded, and / or encoded video data.

[0035] Computer-readable medium 110 can represent any type of medium or device capable of transmitting encoded video data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium enabling source device 102 to transmit encoded video data directly to destination device 116 in real time, for example, via a radio frequency network or a computer-based network. According to a communication standard such as a wireless communication protocol, output interface 108 can modulate the transmitted signal including encoded video data, and input interface 122 can demodulate the received transmitted signal. The communication medium can include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network such as a local area network, wide area network, or global network (such as the Internet). The communication medium can include a router, switch, base station, or any other device that may be useful for facilitating communication from source device 102 to destination device 116.

[0036] In some examples, computer-readable medium 110 may include storage device 112. Source device 102 may output encoded data to storage device 112 from output interface 108. Similarly, destination device 116 may access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0037] In some examples, computer-readable medium 110 may include file server 114 or another intermediate storage device that can store encoded video data generated by source device 102. Source device 102 may output encoded video data to file server 114 or another intermediate storage device that can store encoded video generated by source device 102. Destination device 116 may access the stored video data from file server 114 via streaming or download. File server 114 may be any type of server device capable of storing and sending encoded video data to destination device 116. File server 114 may represent a web server (e.g., for a website), a file transfer protocol (FTP) server, a content delivery network device, or a network attached storage (NAS) device. Destination device 116 may access the encoded video data from file server 114 via any standard data connection including an internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both, suitable for accessing encoded video data stored on file server 114. File server 114 and input interface 122 can be configured to operate according to a streaming protocol, a downloading protocol, or a combination thereof.

[0038] Output interface 108 and input interface 122 can represent a wireless transmitter / receiver, a modem, a wired network component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where output interface 108 and input interface 122 include wireless components, output interface 108 and input interface 122 can be configured to transmit data such as encoded video data according to cellular communication standards such as 4G, 4G-LTE (Long Term Evolution), Advanced LTE, 5G, etc. In some examples where output interface 108 includes a wireless transmitter, output interface 108 and input interface 122 can be configured to operate according to specifications such as IEEE 802.11, IEEE 802.15 (e.g., ZigBee). TM Bluetooth TM Other wireless standards, such as standard 102, transmit data such as encoded video data. In some examples, source device 102 and / or destination device 116 may include respective system-on-chip (SoC) devices. For example, source device 102 may include an SoC device for performing functions belonging to video encoder 200 and / or output interface 108, and destination device 116 may include an SoC device for performing functions belonging to video decoder 300 and / or input interface 122.

[0039] The techniques disclosed herein can be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission such as HTTP-based dynamic adaptive streaming, digital video encoded into a data storage medium, decoding of digital video stored in a data storage medium, or other applications.

[0040] The input interface 122 of the destination device 116 receives an encoded video bitstream from a computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, etc.). The encoded video bitstream may include signaling information defined by the video encoder 200 and also used by the video decoder 300. Such signaling information may be syntax elements having values ​​describing the characteristics and / or processing of video blocks or other decoded units (e.g., slices, pictures, picture groups, sequences, etc.). The display device 118 displays decoded images of the decoded video data to the user. The display device 118 may represent any of a variety of display devices, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or another type of display device.

[0041] Although not in Figure 1 As shown, but in some examples, the video encoder 200 and video decoder 300 may each integrate an audio encoder and / or audio decoder, and may include suitable MUX-DEMUX units or other hardware and / or software for processing multiplexed streams that include both audio and video in a common data stream. Where applicable, the MUX-DEMUX unit may conform to the ITU H.223 multiplexer protocol or other protocols such as User Datagram Protocol (UDP).

[0042] The video encoder 200 and video decoder 300 can each be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is partially implemented in software, the device may store the software instructions in a suitable non-transitory computer-readable medium and use one or more processors to execute the instructions in hardware to perform the technology of this disclosure. Each of the video encoder 200 and video decoder 300 may be included in one or more encoders or decoders, and either one may be integrated as part of a combined encoder / decoder (CODEC) in a separate device. Devices including the video encoder 200 and / or video decoder 300 may include integrated circuits, microprocessors, and / or wireless communication devices such as cellular phones.

[0043] The video encoder 200 and video decoder 300 can operate according to video decoding standards such as ITU-T H.265, also known as High Efficiency Video Decoding (HEVC), or its extensions (such as Multi-View and / or Scalable Video Decoding Extensions). Alternatively, the video encoder 200 and video decoder 300 can operate according to other proprietary or industry standards such as the Joint Exploratory Test Model (JEM) or ITU-T H.266, also known as Versatile Video Decoding (VVC). A draft of the VCC standard is described in “Versatile Video Coding (Draft 7)” by Bross et al., Joint Video Experts Group (JVET) of ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 16th Meeting: Geneva, CH, 1-11 October 2019, JVET-P2001-v14 (hereinafter referred to as “VVC Draft 7”). However, the technology of this disclosure is not limited to any particular decoding standard.

[0044] In summary, the video encoder 200 and video decoder 300 can perform block-based decoding of images. The term "block" generally refers to a structure comprising data to be processed (e.g., encoded, decoded, or used during encoding and / or decoding). For example, a block may comprise a two-dimensional matrix of samples of luminance and / or chrominance data. In summary, the video encoder 200 and video decoder 300 can decode video data represented in YUV (e.g., Y, Cb, Cr) format. That is, instead of decoding sampled red, green, and blue (RGB) data of an image, the video encoder 200 and video decoder 300 can decode both luminance and chrominance components, where the chrominance components may include both red and blue chrominance components. In some examples, the video encoder 200 converts the received RGB format data to a YUV representation before encoding, and the video decoder 300 converts the YUV representation back to RGB format. Alternatively, preprocessing and post-processing units (not shown) can perform these conversions.

[0045] This disclosure can be generally understood to refer to the decoding of images (e.g., encoding and decoding) to include the process of encoding or decoding the data of an image. Similarly, this disclosure can refer to the decoding of blocks of images to include the process of encoding or decoding the data of blocks (e.g., prediction and / or residual decoding). In general, an encoded video bitstream includes a series of values ​​for syntax elements representing decoding decisions (e.g., decoding modes) and the partitioning of images into blocks. Therefore, references to decoding images or blocks should be generally understood to mean decoding the values ​​of the syntax elements that form the images or blocks.

[0046] HEVC defines various blocks, including decoding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video decoder (such as a video encoder 200) divides the decoding tree units into CUs according to a quadtree structure. That is, the video decoder divides the CTUs and CUs into four equal, non-overlapping squares, and each node of the quadtree has zero or four child nodes. Nodes without child nodes can be called "leaf nodes," and the CU of such leaf nodes can include one or more PUs and / or one or more TUs. The video decoder can further divide the PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents the partitioning of the TUs. In HEVC, PUs represent inter-frame prediction data, while TUs represent residual data. CUs with intra-frame prediction include intra-frame prediction information such as intra-frame mode indication.

[0047] As another example, the video encoder 200 and video decoder 300 can be configured to operate according to JEM or VVC. According to JEM or VVC, the video decoder (such as the video encoder 200) partitions the image into multiple decoding tree units (CTUs). The video encoder 200 can partition the CTUs according to a tree structure such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure removes the concept of multiple partition types (such as the separation between CUs, PUs, and TUs in HEVC). The QTBT structure consists of two layers: a first layer partitioned according to a quadtree, and a second layer partitioned according to a binary tree. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to decoding units (CUs).

[0048] In the MTT partitioning structure, blocks can be partitioned using quadtree (QT), binary tree (BT), and one or more types of ternary tree (TT) partitioning (also known as tripartite tree (TT)). A ternary tree partition is a partition in which a block is split into three sub-blocks. In some examples, a ternary tree partition divides a block into three sub-blocks without crossing the center of the original block. The partitioning types in MTT (e.g., QT, BT, and TT) can be symmetric or asymmetric.

[0049] In some examples, the video encoder 200 and the video decoder 300 may use a single QTBT or MTT structure to represent each of the luma and chroma components, while in other examples, the video encoder 200 and the video decoder 300 may use two or more QTBT or MTT structures, such as using one QTBT / MTT structure for the luma component and another QTBT / MTT structure for both chroma components (or using two QTBT / MTT structures for each chroma component separately).

[0050] The video encoder 200 and video decoder 300 can be configured to use a quadtree partitioning, QTBT partitioning, MTT partitioning, or other partitioning structures according to HEVC. For illustrative purposes, the description of the technology of this disclosure is presented with reference to QTBT partitioning. However, it should be understood that the technology of this disclosure can also be applied to video decoders configured to use quadtree partitioning or other types of partitioning.

[0051] Blocks (e.g., CTUs or CUs) can be grouped in various ways within an image. As an example, a block can refer to a rectangular area of ​​a row of CTUs within a specific tile in an image. A tile can be a rectangular area of ​​CTUs within a specific tile column and a specific tile row in an image. A tile column refers to a rectangular area of ​​CTUs with a height equal to the image height and a width specified by a syntax element (e.g., as in the image parameter set). A tile row refers to a rectangular area of ​​CTUs with a height specified by a syntax element (e.g., as in the image parameter set) and a width equal to the image width.

[0052] In some examples, a tile can be divided into multiple bricks, each of which may include one or more CTU rows within the tile. A tile divided into multiple bricks can also be referred to as a brick. However, a brick that is a true subset of a tile cannot be referred to as a tile.

[0053] You can also arrange the bricks in an image into slices. A slice can be an integer number of bricks in an image that can be exclusively contained in a single Network Abstraction Layer (NAL) unit. In some examples, a slice consists of a continuous sequence of multiple complete tiles or a single complete tile.

[0054] This disclosure uses "NxN" and "N by N" interchangeably to refer to the sampling dimensions of a block (such as a CU or other video block) in terms of both vertical and horizontal dimensions, for example, 16x16 sampling or 16 by 16 sampling. In general, a 16x16 CU will have 16 samples in the vertical direction (y = 16) and 16 samples in the horizontal direction (x = 16). Similarly, an NxN CU generally has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. The samples in a CU can be arranged in rows and columns. Furthermore, a CU does not necessarily need to have the same number of samples in the horizontal direction as in the vertical direction. For example, a CU can include NxM samples, where M is not necessarily equal to N.

[0055] The video encoder 200 encodes video data of the CU (Complex Unit), which represents prediction and / or residual information and other information. Prediction information indicates how the CU should be predicted to form prediction blocks for the CU. Residual information generally represents the sample-by-sample difference between the samples of the CU before encoding and the prediction blocks.

[0056] To predict a CU, the video encoder 200 can generally form prediction blocks for the CU through inter-frame prediction or intra-frame prediction. Inter-frame prediction generally refers to predicting the CU based on data from previously decoded images, while intra-frame prediction generally refers to predicting the CU based on previously decoded data from the same image. To perform inter-frame prediction, the video encoder 200 can generate prediction blocks using one or more motion vectors. The video encoder 200 can generally perform motion search to identify, for example, a reference block that closely matches the CU based on the difference between the CU and a reference block. The video encoder 200 can use the sum of absolute differences (SAD), sum of squared differences (SSD), mean of absolute differences (MAD), mean squared error (MSD), or other such difference calculations to calculate a difference index to determine whether the reference block closely matches the current CU. In some examples, the video encoder 200 can use unidirectional or bidirectional prediction to predict the current CU.

[0057] Some examples of JEM and VVC also provide an affine motion compensation mode, which can be viewed as an inter-frame prediction mode. In affine motion compensation mode, the video encoder 200 can determine two or more motion vectors representing non-translational motion (such as zooming in or out, rotation, perspective motion, or other irregular motion types).

[0058] To perform intra-frame prediction, the video encoder 200 can select an intra-frame prediction mode to generate prediction blocks. Some examples of JEM and VVC provide 67 intra-frame prediction modes, including various directional modes as well as planar and DC modes. In general, the video encoder 200 selects an intra-frame prediction mode that describes the samples adjacent to the current block (e.g., a block of a CU) based on its prediction of the current block. Assuming the video encoder 200 encodes the CTU and CU in raster scan order (from left to right, from top to bottom), such samples can generally be located above, to the upper left, or to the left of the current block within the same frame.

[0059] The video encoder 200 encodes data representing the prediction mode of the current block. For example, for inter-frame prediction modes, the video encoder 200 may encode data indicating which of the various available inter-frame prediction modes is used, and the motion information used for the corresponding mode. For unidirectional or bidirectional inter-frame prediction, for example, the video encoder 200 may encode motion vectors using Advanced Motion Vector Prediction (AMVP) or merging modes. The video encoder 200 may use similar modes to encode motion vectors for affine motion compensation modes.

[0060] According to the technology of this disclosure, the video encoder 200 can generate a set of reference samples for the current block of the current image based on the size of a reference image. For example, the video encoder 200 can perform reference sample cropping on a reference block of the reference image based on the size of the reference image. In some examples, the video encoder 200 can perform wrapping based on the size of the reference image. For example, the video encoder 200 can generate samples from one or more reference sample sets based on the size of the reference image and a wrapping offset. Thus, compared to systems that can perform reference sample cropping on a reference block of the reference image based on the current image size and / or perform wrapping based on the current image size, the decoding accuracy of the video encoder 200 can be improved.

[0061] Following prediction (such as intra-frame or inter-frame prediction for a block), the video encoder 200 can compute residual data for the block. Residual data, such as a residual block, represents the sample-by-sample difference between the block formed using the corresponding prediction mode and the predicted block of the block. The video encoder 200 can apply one or more transforms to the residual block to produce transformed data in the transform domain, replacing the sampled domain. For example, the video encoder 200 can apply a Discrete Cosine Transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video data. Additionally, the video encoder 200 can apply a second transform after the first transform, such as a Mode Correlated Inseparable Second Transform (MDNSST), a Signal Correlation Transform, a Karhunen-Loeve Transform (KLT), etc. The video encoder 200 generates transform coefficients after applying one or more transforms.

[0062] As noted above, following any transform used to generate the transform coefficients, the video encoder 200 may perform quantization on the transform coefficients. Quantization, broadly speaking, refers to the process in which transform coefficients are quantized to potentially reduce the amount of data used to represent the transform coefficients by providing further compression. By performing the quantization process, the video encoder 200 may reduce the bit depth associated with some or all of the transform coefficients. For example, the video encoder 200 may round an n-bit value down to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, the video encoder 200 may perform a bitwise right shift of the value to be quantized.

[0063] Following quantization, the video encoder 200 can scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix including the quantized transform coefficients. The scan can be designed to place high-energy (and therefore low-frequency) transform coefficients at the beginning of the vector and low-energy (and therefore high-frequency) transform coefficients at the end. In some examples, the video encoder 200 can use a predefined scan order to scan the quantized transform coefficients to produce a serialized vector, and then entropy-encode the quantized transform coefficients of the vector. In other examples, the video encoder 200 can perform an adaptive scan. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder 200 can entropy-encode the one-dimensional vector, for example, according to context-adaptive binary arithmetic coding (CABAC). The video encoder 200 can also entropy-encode the values ​​of syntax elements describing metadata associated with the encoded video data for use by the video decoder 300 when decoding the video data.

[0064] To perform CABAC, the video encoder 200 can assign context within a context model to the symbols to be transmitted. The context can, for example, involve whether the neighboring values ​​of a symbol are zero. Probability determination can be based on the context assigned to the symbols.

[0065] The video encoder 200 can further generate syntax data (such as block-based syntax data, image-based syntax data, and sequence-based syntax data) for the video decoder 300, for example, in image headers, block headers, slice headers, or other syntax data (such as sequence parameter sets (SPS), picture parameter sets (PPS), or video parameter sets (VPS)). The video decoder 300 can similarly decode such syntax data to determine how to decode the corresponding video data.

[0066] In this way, the video encoder 200 can generate a bitstream comprising encoded video data, such as syntax elements describing the partitioning of images into blocks (e.g., CUs) and prediction and / or residual information of the blocks. Finally, the video decoder 300 can receive the bitstream and decode the encoded video data.

[0067] In summary, the video decoder 300 performs a process that is the inverse of the process performed by the video encoder 200 to decode the encoded video data bitstream. For example, the video decoder 300 can use the values ​​of syntax elements of the CABAC decoding bitstream in a manner that is substantially similar to but inverse of the CABAC encoding process of the video encoder 200. Syntax elements can define the partitioning information from picture to CTU and the partitioning of each CTU according to the corresponding partitioning structure (such as a QTBT structure) used to define the CTU. Syntax elements can further define the prediction and residual information of blocks (e.g., CUs) of the video data.

[0068] The residual information can be represented, for example, by quantized transform coefficients. The video decoder 300 can inversely quantize and inverse transform the quantized transform coefficients of the block to reproduce the residual block of the block. The video decoder 300 uses the prediction mode (intra-frame or inter-frame prediction) indicated by the signal and associated prediction information (e.g., motion information for inter-frame prediction) to form the prediction block of the block.

[0069] According to the technology of this disclosure, the video decoder 300 can generate a set of reference samples for the current block of the current image based on the size of a reference image. For example, the video decoder 300 can perform reference sample cropping on a reference block of the reference image based on the size of the reference image. In some examples, the video decoder 300 can perform wrap-around processing based on the size of the reference image. For example, the video decoder 300 can generate samples from one or more reference sample sets based on the size of the reference image and a wrap-around offset. Thus, compared to systems that can perform reference sample cropping on a reference block of the reference image based on the current image size and / or perform wrap-around processing based on the current image size, the decoding accuracy of the video decoder 300 can be improved.

[0070] The video decoder 300 can then merge the predicted block and the residual block (e.g., on a sample-by-sample basis) to reproduce the original block. The video decoder 300 can perform additional processing, such as performing a deblocking process to reduce visual artifacts along the block boundaries.

[0071] According to the technology of this disclosure, a video encoder 200 and / or a video decoder 300 can be configured to perform a method comprising: determining, using motion information, a reference block of a reference image for predicting a current block of a current image; and generating a reference sample set for the current block of the current image, wherein generating the reference sample set comprises: performing reference sample cropping on the reference block of the reference image based on the size of the reference image. The method further comprises: generating a predicted block for the current block of the current image based on the reference sample set.

[0072] This disclosure can be broadly referred to as "signaling notification," specifically information such as syntax elements. The term "signaling notification" can broadly refer to the transmission of values ​​for syntax elements and / or other data used to decode encoded video data. That is, the video encoder 200 can signal the values ​​of syntax elements in the bitstream. In general, signaling notification refers to the values ​​generated in the bitstream. As noted above, source device 102 can transmit the bitstream to destination device 116 substantially in real-time or non-real-time (e.g., when storing syntax elements in storage device 112 for later retrieval by destination device 116).

[0073] Figure 2A and 2B This is a conceptual diagram illustrating an example Quadtree Binary Tree (QTBT) structure 130 and a corresponding Decoding Tree Unit (CTU) 132. Solid lines represent quadtree splits, and dashed lines represent binary tree splits. In each split (i.e., non-leaf) node of the binary tree, a signal informs a flag indicating which split type (i.e., horizontal or vertical) is used, where, in this example, 0 indicates a horizontal split and 1 indicates a vertical split. For quadtree splits, it is not necessary to indicate the split type because the quadtree node splits the block horizontally and vertically into four sub-blocks of equal size. Accordingly, the video encoder 200 can encode and the video decoder 300 can decode the syntax elements (such as split information) of the region tree layer (i.e., solid lines) and the syntax elements (such as split information) of the prediction tree layer (i.e., dashed lines) of the QTBT structure 130. The video encoder 200 can encode and the video decoder 300 can decode the video data (such as prediction and transform data) of the CU represented by the terminal leaf nodes of the QTBT structure 130.

[0074] In summary, Figure 2B The CTU 132 may be associated with parameters that define the size of the block corresponding to the node of the QTBT structure 130 located in the first and second layers. These parameters may include the CTU size (representing the size of the CTU 132 in sampled terms), the minimum quadtree size (MinQTSize, representing the minimum allowed size of the leaf nodes of the quadtree), the maximum binary tree size (MaxBTSize, representing the maximum allowed size of the root node of the binary tree), the maximum binary tree depth (MaxBTDepth, representing the maximum allowed depth of the binary tree), and the minimum binary tree size (MinBTSize, representing the minimum allowed size of the leaf nodes of the binary tree).

[0075] The root node of the QTBT structure corresponding to CTU can have four child nodes at the first level of the QTBT structure, and each of the four child nodes can be partitioned according to the quadtree partitioning. That is, the nodes at the first level are either leaf nodes (with no child nodes) or have four child nodes. An example of QTBT structure 130 represents such a node as having solid lines for branching, including the parent node and child nodes. If the nodes at the first level are not larger than the maximum allowed binary tree root node size (MaxBTSize), the nodes can be further partitioned using separate binary trees. The binary tree split of a node can be iterated until the nodes produced by the split reach the minimum allowed binary tree leaf node size (MinBTSize) or the maximum allowed binary tree depth (MaxBTDepth). An example of QTBT structure 130 represents such a node as having dashed lines for branching. The binary tree leaf nodes are called decoding units (CUs), and CUs are used for prediction (e.g., intra-frame prediction or inter-frame prediction) and transformation without any further partitioning. As discussed above, CUs can also be called "video blocks" or "blocks".

[0076] In one example of a QTBT partitioning structure, the CTU size is set to 128x128 (luminance samples and two corresponding 64x64 chrominance samples), MinQTSize is set to 16x16, MaxBTSize is set to 64x64, MinBTSize (for both width and height) is set to 4, and MaxBTDepth is set to 4. First, a quadtree partitioning is applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes ranging from 16x16 (i.e., MinQTSize) to 128x128 (i.e., the CTU size). If a quadtree leaf node is 128x128, it is not further partitioned using a binary tree because the size exceeds MaxBTSize (in this example, 64x64). Otherwise, the quadtree leaf node is further partitioned using a binary tree. Therefore, the quadtree leaf node is also the root node of the binary tree and has a binary tree depth of 0. When the depth of the binary tree reaches MaxBTDepth (4 in this example), no further splitting is allowed. Similarly, when a binary tree node has a width equal to MinBTSize (4 in this example), no further vertical splitting is allowed. Likewise, a binary tree node with a height equal to MinBTSize means that no further horizontal splitting is allowed for that binary tree node. As noted above, the leaf nodes of the binary tree are called CUs and are further processed according to predictions and transformations without any further partitioning.

[0077] Figure 3This is a block diagram illustrating an example video encoder 200 that can perform the techniques described in this disclosure. Figure 3 This disclosure is provided for illustrative purposes and should not be construed as limiting the techniques as broadly illustrated and described herein. For illustrative purposes, this disclosure describes a video encoder 200 in the context of video decoding standards such as the HEVC video decoding standard and the H.266 video decoding standard under development. However, the techniques of this disclosure are not limited to these video decoding standards and are generally applicable to video encoding and decoding.

[0078] exist Figure 3 In the example, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filtering unit 216, a decoded picture buffer (DPB) 218, and an entropy decoding unit 220. Any or all of the video data memory 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filtering unit 216, DPB 218, and entropy decoding unit 220 can be implemented in one or more processors or in processing circuitry. For example, units of the video encoder 200 can be implemented as one or more circuits or as part of hardware circuitry or as logic elements as part of a processor, ASIC, or FPGA. Furthermore, the video encoder 200 may include additional or alternative processors or processing circuitry for performing these and other functions.

[0079] The video data storage device 230 can store video data to be encoded by components of the video encoder 200. The video encoder 200 can, for example, retrieve video data from the video source 104 (…). Figure 1 The video data memory 230 receives video data stored in the video data memory 230. The DPB 218 can act as a reference picture memory storing reference video data used by the video encoder 200 when predicting subsequent video data. The video data memory 230 and the DPB 218 can be constructed from any of a variety of storage devices, such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of storage devices. The video data memory 230 and the DPB 218 can be provided as a single storage device or as separate storage devices. In various examples, the video data memory 230 can be on-chip with other components of the video encoder 200 as illustrated, or off-chip relative to those components.

[0080] In this disclosure, references to video data memory 230 should not be construed as limited to memory located inside video encoder 200 (unless specifically described as such) or memory located outside video encoder 200 (unless specifically described as such). Rather, references to video data memory 230 should be understood as a reference memory storing video data received by video encoder 200 for encoding (e.g., video data of the current block to be encoded). Figure 1 The memory 106 can also provide temporary storage for the outputs from various units of the video encoder 200.

[0081] Explanation Figure 3 Various units are provided to aid in understanding the operations performed by the video encoder 200. Units can be implemented as fixed-function circuits, programmable circuits, or combinations thereof. A fixed-function circuit is one that provides a specific function and is predefined regarding the operations that can be performed. A programmable circuit is one that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by software or firmware instructions. A fixed-function circuit can execute software instructions (e.g., for receiving or outputting parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more units within the unit may be different circuit blocks (fixed-function or programmable), and in some examples, one or more units within the unit may be integrated circuits.

[0082] The video encoder 200 may include an arithmetic logic unit (ALU), an essential function unit (EFU), digital circuits, analog circuits, and / or a programmable core composed of programmable circuits. In an example where the operation of the video encoder 200 is performed using software executed by programmable circuits, memory 106 ( Figure 1 The video encoder 200 may store software instructions (e.g., object code) received and executed by the video encoder 200, or another memory (not shown) within the video encoder 200 may store such instructions.

[0083] The video data storage unit 230 is configured to store received video data. The video encoder 200 can retrieve images of the video data from the video data storage unit 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data storage unit 230 can be the raw video data to be encoded.

[0084] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra-frame prediction unit 226. The mode selection unit 202 may include additional functional units for performing video prediction based on other prediction modes. As an example, the mode selection unit 202 may include a panel unit, an intra-block copy unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, etc.

[0085] The mode selection unit 202 generally coordinates the rate-distortion values ​​of multiple coding passes for test combinations of coding parameters and the resulting combinations. Coding parameters may include the CTU-CU partitioning, the prediction mode for the CU, the transformation type of the residual data for the CU, the quantization parameters of the residual data for the CU, etc. The mode selection unit 202 can ultimately select a combination of coding parameters that has a better rate-distortion value than other tested combinations.

[0086] The video encoder 200 can divide images retrieved from the video data storage 230 into a series of CTUs, and encapsulate one or more CTUs within a slice. The mode selection unit 202 can divide the image's CTUs according to a tree structure such as the QTBT structure described above or the quadtree structure of HEVC. As described above, the video encoder 200 can form one or more CUs from the tree-structured CTUs. In general, such a CU can also be referred to as a "video block" or a "block".

[0087] In summary, mode selection unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra-frame prediction unit 226) to generate prediction blocks for the current block (e.g., the current CU, or the overlapping portion of PU and TU in HEVC). For inter-frame prediction of the current block, motion estimation unit 222 can perform a motion search to identify one or more closely matching reference blocks in one or more reference images (e.g., one or more previously decoded images stored in DPB 218). Specifically, motion estimation unit 222 can calculate values ​​representing how similar a potential reference block is to the current block, such as sum of absolute differences (SAD), sum of squared differences (SSD), mean of absolute differences (MAD), and mean squared error (MSD). Motion estimation unit 222 can generally perform these calculations using sample-wise differences between the current block and the reference blocks being considered. Motion estimation unit 222 can identify the reference block with the lowest value produced by these calculations (indicating the reference block most closely matching the current block).

[0088] Motion estimation unit 222 can generate one or more motion vectors (MVs) defining the position of a reference block in a reference image relative to the position of the current block in the current image. Motion estimation unit 222 can then provide the motion vectors to motion compensation unit 224. For example, for unidirectional inter-frame prediction, motion estimation unit 222 can provide a single motion vector, while for bidirectional inter-frame prediction, motion estimation unit 222 can provide two motion vectors. Motion compensation unit 224 can then use the motion vectors to generate prediction blocks. For example, motion compensation unit 224 can use the motion vectors to retrieve data for the reference blocks. As another example, if the motion vectors have fractional sampling precision, motion compensation unit 224 can interpolate the values ​​of the prediction blocks according to one or more interpolation filters. Furthermore, for bidirectional inter-frame prediction, motion compensation unit 224 can retrieve data for two reference blocks identified by separate motion vectors, and merge the retrieved data, for example, by averaging or weighted averaging.

[0089] As another example, for intra-prediction or intra-prediction decoding, intra-prediction unit 226 can generate a prediction block based on samples adjacent to the current block. For example, in directional mode, generally speaking, intra-prediction unit 226 can mathematically combine the values ​​of adjacent samples and fill these calculated values ​​across the current block in a defined direction to produce a prediction block. As another example, in DC mode, intra-prediction unit 226 can calculate the average of adjacent samples of the current block and generate a prediction block as the average of this result for each sample of the prediction block.

[0090] Mode selection unit 202 provides a prediction block to residual generation unit 204. Residual generation unit 204 receives the raw, uncoded version of the current block from video data memory 230 and the prediction block from mode selection unit 202. Residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines the residual block of the current block. In some examples, residual generation unit 204 may also determine the difference between sampled values ​​in the residual block to generate a residual block using residual differential pulse code modulation (RDPCM). In some examples, residual generation unit 204 may be constructed using one or more subtractor circuits that perform binary subtraction.

[0091] In the example where mode selection unit 202 divides a CU into PUs, each PU can be associated with a luma prediction unit and a corresponding chroma prediction unit. Video encoder 200 and video decoder 300 can support PUs of various sizes. As indicated above, the size of a CU can refer to the size of its luma decoding block, and the size of a PU can refer to the size of the luma prediction unit within that PU. Assuming a particular CU is 2Nx2N in size, video encoder 200 can support PU sizes of 2Nx2N or NxN for intra-frame prediction, and 2Nx2N, 2NxN, Nx2N, NxN, or similar symmetrical PU sizes for inter-frame prediction. Video encoder 200 and video decoder 300 can also support asymmetric partitioning of PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter-frame prediction.

[0092] In an example where the mode selection unit 202 does not further divide the CU into PUs, each CU may be associated with a luminance decoding block and a corresponding chrominance decoding block. As mentioned above, the size of the CU may refer to the size of the luminance decoding block of the CU. The video encoder 200 and the video decoder 300 may support CU sizes of 2Nx2N, 2NxN, or Nx2N.

[0093] For other video decoding techniques such as, for example, intra-block copy mode decoding, affine mode decoding, and linear model (LM) mode decoding, mode selection unit 202 generates a prediction block for the current block being encoded via units associated with the respective decoding technique. In some examples, such as panel mode decoding, mode selection unit 202 may not generate a prediction block and instead generate syntax elements indicating how the selected panel reconstruction block is sampled. In such modes, mode selection unit 202 can provide these syntax elements to entropy coding unit 220 for encoding.

[0094] According to the technology of this disclosure, the mode compensation unit 224 may include a sample generation unit 225. The sample generation unit 225 may generate a reference sample set for the current block of the current image based on the size of the reference image. For example, the sample generation unit 225 may perform reference sample cropping on the reference block of the reference image based on the size of the reference image. In some examples, the sample generation unit 225 may perform wrap processing based on the size of the reference image. For example, the sample generation unit 225 may generate samples from one or more reference sample sets based on the size of the reference image and a wrap offset. Thus, compared to systems that can perform reference sample cropping on the reference block of the reference image based on the current image size and / or perform wrap processing based on the current image size, the decoding accuracy of the video encoder 200 can be improved.

[0095] As described above, the residual generation unit 204 receives video data for the current block and the corresponding prediction block. The residual generation unit 204 then generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates the sample-by-sample difference between the prediction block and the current block.

[0096] Transform processing unit 206 applies one or more transformations to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transformations to the residual block to form the transform coefficient block. For example, transform processing unit 206 may apply discrete cosine transform (DCT), orientation transform, Karhunen-Loeve transform (KLT), or conceptually similar transformations to the residual block. In some examples, transform processing unit 206 may perform multiple transformations (e.g., elementary and quadratic transformations) on the residual block. In some examples, transform processing unit 206 does not apply any transformations to the residual block.

[0097] Quantization unit 208 can quantize the transform coefficients in a transform coefficient block to produce a quantized transform coefficient block. Quantization unit 208 can quantize the transform coefficients of the transform coefficient block based on the quantization parameter (QP) value associated with the current block. Video encoder 200 (e.g., via mode selection unit 202) can adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may introduce information loss, and therefore, the quantized transform coefficients may have lower precision than the original transform coefficients generated by transform processing unit 206.

[0098] The inverse quantization unit 210 and the inverse transform processing unit 212 can apply inverse quantization and inverse transform to the quantized transform coefficient block, respectively, to reconstruct the residual block from the transform coefficient block. The reconstruction unit 214 can generate a reconstructed block corresponding to the current block (but potentially with some degree of distortion) based on the reconstructed residual block and the prediction block generated by the mode selection unit 202. For example, the reconstruction unit 214 can add samples of the reconstructed residual block to corresponding samples of the prediction block generated by the mode selection unit 202 to generate the reconstructed block.

[0099] Filtering unit 216 can perform one or more filtering operations on the reconstructed block. For example, filtering unit 216 can perform a deblocking operation to reduce block artifacts along the edges of the CU. In some examples, the operation of filtering unit 216 can be skipped.

[0100] The video encoder 200 stores the reconstructed blocks in the DPB 218. For example, in an example where the operation of the filtering unit 216 is not required, the reconstruction unit 214 can store the reconstructed blocks in the DPB 218. In an example where the operation of the filtering unit 216 is required, the filtering unit 216 can store the filtered and reconstructed blocks in the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 can retrieve a reference picture composed of the reconstructed (and potentially filtered) blocks from the DPB 218 to perform inter-frame prediction of blocks in subsequently encoded pictures. Additionally, the intra-frame prediction unit 226 can use the reconstructed blocks in the DPB 218 of the current picture to perform intra-frame prediction of other blocks in the current picture.

[0101] In summary, entropy coding unit 220 can entropy-encode syntax elements received from other functional components of video encoder 200. For example, entropy coding unit 220 can entropy-encode quantized transform coefficient blocks from quantization unit 208. As another example, entropy coding unit 220 can entropy-encode predictive syntax elements (e.g., motion information for inter-frame prediction or intra-frame prediction intra-frame information) from mode selection unit 202. Entropy coding unit 220 can perform one or more entropy coding operations on syntax elements, another example of video data, to generate entropy-encoded data. For example, entropy coding unit 220 can perform context-adaptive variable-length decoding (CAVLC), CABAC, variable-to-variable (V2V) length decoding, syntax-based context-adaptive binary arithmetic decoding (SBAC), probability interval partitioning entropy (PIPE) decoding, exponential Golomb coding, or another type of entropy coding operation on the data. In some examples, entropy coding unit 220 can operate in a bypass mode in which entropy coding of syntax elements is not performed.

[0102] The video encoder 200 can output a bitstream, which includes entropy-encoded syntax elements required to reconstruct slices or blocks of images. Specifically, the entropy coding unit 220 can output a bitstream.

[0103] The operations described above are block-based. Such descriptions should be understood as operations applied to the luma decoding block and / or the chroma decoding block. As described above, in some examples, the luma decoding block and chroma decoding block are the luma and chroma components of the CU. In some examples, the luma decoding block and chroma decoding block are the luma and chroma components of the PU.

[0104] In some examples, it is not necessary to repeat the operations performed for the luma decoded block for the chroma decoded block. As an example, it is not necessary to repeat the operations used to identify the motion vector (MV) of the luma decoded block and the reference image in order to identify the MV of the chroma block. Instead, the MV of the luma decoded block can be scaled to determine the MV of the chroma block, and the reference image can be the same. As another example, the intra-frame prediction process can be the same for both the luma and chroma decoded blocks.

[0105] Video encoder 200 represents an example of a device configured to encode video data, the device including a memory configured to store the video data and one or more processing units implemented using circuitry and configured to perform the following operations: determining a reference block of a reference image for predicting a current block of the current image using motion information, and generating a reference sample set for the current block of the current image, wherein generating the reference sample set includes performing reference sample cropping on the reference block of the reference image based on the size of the reference image. The one or more processing units may be further configured to generate a predicted block for the current block of the current image based on the reference sample set.

[0106] Figure 4 This is a block diagram illustrating an example video decoder 300 that can perform the techniques described in this disclosure. Figure 4 This disclosure is provided for illustrative purposes and is not intended to limit the techniques as broadly illustrated and described herein. For illustrative purposes, this disclosure describes a video decoder 300 based on the techniques of JEM, VVC, and HEVC. However, the techniques of this disclosure can be implemented by video decoding devices configured for other video decoding standards.

[0107] exist Figure 4 In the example, the video decoder 300 includes a decoded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filtering unit 312, and a decoded picture buffer (DPB) 314. Any or all of the CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filtering unit 312, and DPB 314 can be implemented in one or more processors or in processing circuitry. For example, units of the video decoder 300 can be implemented as one or more circuits or as part of hardware circuitry or as logic elements as part of a processor, ASIC, or FPGA. Furthermore, the video decoder 300 may include additional or alternative processors or processing circuitry for performing these and other functions.

[0108] The prediction processing unit 304 includes a motion compensation unit 316 and an intra-frame prediction unit 318. The prediction processing unit 304 may include additional units for performing predictions based on other prediction modes. As an example, the prediction processing unit 304 may include a panel unit, an intra-block copy unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, the video decoder 300 may include more, fewer, or different functional components.

[0109] CPB memory 320 can store video data, such as encoded video bitstreams, that will be decoded by components of video decoder 300. This data can be stored, for example, from computer-readable medium 110 (…). Figure 1 The video data stored in the CPB memory 320 is retrieved. The CPB memory 320 may include a CPB storing encoded video data (e.g., syntax elements) from the encoded video bitstream. Furthermore, the CPB memory 320 may store video data different from the decoded image's syntax elements, such as temporary data representing the output from various units of the video decoder 300. The DPB 314 generally stores a decoded image that the video decoder 300 can output and / or use as reference video data when decoding subsequent data or images from the encoded video bitstream. The CPB memory 320 and DPB 314 can be constructed from any of a variety of storage devices, such as DRAM including SDRAM, MRAM, RRAM, or other types of storage devices. The CPB memory 320 and DPB 314 may be provided by the same storage device or separate storage devices. In various examples, the CPB memory 320 may be on-chip with other components of the video decoder 300, or off-chip relative to those components.

[0110] Additionally or alternatively, in some examples, the video decoder 300 can be drawn from the memory 120 ( Figure 1 Retrieving decoded video data. That is, memory 120 can store data as discussed above with respect to CPB memory 320. Similarly, when some or all of the functions of video decoder 300 are implemented by software to be executed by the processing circuitry of video decoder 300, memory 120 can store instructions to be executed by video decoder 300.

[0111] Explanation Figure 4 The various units shown are intended to aid in understanding the operations performed by the video decoder 300. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Figure 3Similarly, a fixed-function circuit refers to a circuit that provides a specific function and is predefined regarding the operations that can be performed. A programmable circuit refers to a circuit that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by instructions in the software or firmware. A fixed-function circuit can execute software instructions (e.g., for receiving or outputting parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more units in the unit may be different circuit blocks (fixed-function or programmable), and in some examples, one or more units in the unit may be integrated circuits.

[0112] The video decoder 300 may include an ALU, an EFU, digital circuitry, analog circuitry, and / or a programmable core composed of programmable circuitry. In an example where the operation of the video decoder 300 is performed by software executing on the programmable circuitry, on-chip or off-chip memory may store instructions (e.g., object code) of the software received and executed by the video decoder 300.

[0113] Entropy decoding unit 302 can receive encoded video data from the CPB and perform entropy decoding on the video data to reproduce the syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, and filtering unit 312 can generate decoded video data based on the syntax elements extracted from the bitstream.

[0114] In summary, the video decoder 300 reconstructs images on a block-by-block basis. The video decoder 300 can perform reconstruction operations on each block individually (where the block currently being reconstructed, i.e., decoded, can be referred to as the "current block").

[0115] Entropy decoding unit 302 can entropy decode the quantized transform coefficients that define the quantized transform coefficient block, as well as syntax elements containing transform information such as quantization parameters (QPs) and / or transform mode indications. Inverse quantization unit 306 can determine the degree of quantization using the QPs associated with the quantized transform coefficient block, and similarly determine the degree of inverse quantization that inverse quantization unit 306 will apply. Inverse quantization unit 306 can, for example, perform a bitwise left shift operation to inverse quantize the quantized transform coefficients. Inverse quantization unit 306 can thus form a transform coefficient block including the transform coefficients.

[0116] After the inverse quantization unit 306 forms the transform coefficient block, the inverse transform processing unit 308 can apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, the inverse transform processing unit 308 can apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotation transform, an inverse orientation transform, or another inverse transform to the transform coefficient block.

[0117] Furthermore, the prediction processing unit 304 generates a prediction block based on the prediction information syntax elements entropy-decoded by the entropy decoding unit 302. For example, if the prediction information syntax elements indicate that the current block is predicted inter-frame, the motion compensation unit 316 can generate the prediction block. In this case, the prediction information syntax elements may indicate a reference picture in the DPB 314 from which a reference block is to be retrieved, and a motion vector identifying the position of the reference block in the reference picture relative to the position of the current block in the current picture. The motion compensation unit 316 can be generally described as working in conjunction with the motion compensation unit 224 ( Figure 3 The inter-frame prediction process is performed in a manner roughly similar to that described above.

[0118] According to the technology of this disclosure, the motion compensation unit 316 may include a sampling generation unit 317. The sampling generation unit 317 may generate a set of reference samples for the current block of the current image based on the size of a reference image. For example, the sampling generation unit 317 may perform reference sample cropping on a reference block of the reference image based on the size of the reference image. In some examples, the sampling generation unit 317 may perform surround processing based on the size of the reference image. For example, the sampling generation unit 317 may generate samples from one or more reference sample sets based on the size of the reference image and a surround offset. Thus, compared to systems that can perform reference sample cropping on a reference block of the reference image based on the current image size and / or perform surround processing based on the current image size, the decoding accuracy of the video decoder 300 can be improved.

[0119] As another example, if the prediction information syntax element indicates that the current block is intra-predicted, then intra-prediction unit 318 can generate a prediction block according to the intra-prediction mode indicated by the prediction information syntax element. Again, intra-prediction unit 318 can be generally described as being combined with intra-prediction unit 226 ( Figure 3 The intra-prediction process is performed in a manner largely similar to that described above. The intra-prediction unit 318 can retrieve data from adjacent samples of the current block from the DPB 314.

[0120] Reconstruction unit 310 can reconstruct the current block using the prediction block and the residual block. For example, reconstruction unit 310 can add samples from the residual block to the corresponding samples from the prediction block to reconstruct the current block.

[0121] Filtering unit 312 can perform one or more filtering operations on the reconstructed block. For example, filtering unit 312 can perform a deblocking operation to reduce block artifacts along the edges of the reconstructed block. It is not necessary to perform the operation of filtering unit 312 in all examples.

[0122] The video decoder 300 can store the reconstructed blocks in the DPB 314. For example, in an example where the operation of the filtering unit 312 is not performed, the reconstruction unit 310 can store the reconstructed blocks in the DPB 314. In an example where the operation of the filtering unit 312 is performed, the filtering unit 312 can store the filtered, reconstructed blocks in the DPB 314. As discussed above, the DPB 314 can provide reference information to the prediction processing unit 304, such as samples of the current image for intra-frame prediction and previously decoded images for subsequent motion compensation. Furthermore, the video decoder 300 can output the decoded image (e.g., decoded video) from the DPB 314 for subsequent processing, such as... Figure 1 Displayed on a display device such as display device 118.

[0123] In this way, video decoder 300 represents an example of a video decoding device, which includes a memory configured to store video data and one or more processing units implemented using circuitry and configured to perform the following operations: determining a reference block of a reference image for predicting the current block of the current image using motion information, and generating a reference sample set for the current block of the current image, wherein generating the reference sample set includes performing reference sample cropping on the reference block of the reference image based on the size of the reference image. The one or more processing units may be further configured to generate a predicted block for the current block of the current image based on the reference sample set.

[0124] In VVC Draft 7, an image can be divided into one or more sub-images. A sub-image can be treated as an image by setting the flag `subpic_treated_as_pic_flag[i]` to 1. In this case, the sub-image can be decoded (e.g., extracted) as a separate image. For example, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to divide a single image into multiple sub-images. In this example, with the flag `subpic_treated_as_pic_flag[i]` equal to 1, the video decoder can decode (e.g., encode or decode) a sub-image from the multiple sub-images as an image. The video decoder can decode the sub-image as an image separate from the other sub-images in the multiple sub-images.

[0125] In some examples, although the video decoder (e.g., video encoder 200 or video decoder 300) can be configured to reorder sub-pictures within an image, some sub-pictures within the image may remain the same (e.g., unchanged). In some examples, sub-picture reordering may be performed image-by-image. For example, the video decoder may be configured to reorder sub-pictures within a first image using a first order, and reorder sub-pictures within a second image using a second order different from the first order. In some cases, the video decoder may be configured to reorder sub-pictures within a first image using a first order, and avoid reordering sub-pictures within a second image. Therefore, when the video decoder reorders sub-pictures, the decoding result of the reordered sub-pictures may differ depending on whether the sub-picture is decoded as part of the whole image or as a separate image.

[0126] The techniques described in this paper can help address the decoding problem of reordered sub-images by considering the boundaries of the reference sub-image in the motion compensation process and the Temporal Motion Vector Prediction (TMVP) derivation. Additionally, the techniques described in this paper can be used to replace the current image size with the size of the reference image during cropping (e.g., motion padding), since the reference image can have a different size than the current image when reference image resampling is enabled.

[0127] When using the techniques described in VVC Draft 7, a video decoder (video encoder 200 or video decoder 300) can divide an image into one or more sub-images (or determine their division). The video encoder (e.g., video encoder 200) can be configured to signal the sub-image division layout in the Slice Partition (SPS). The video decoder can be configured to signal the image sub-image layout mapped to the SPS layout in the PPS or Picture Header (PH). The video encoder can be configured to signal slices with the slice_subpic_id signaled in the Slice Header (SH) to identify which sub-image the slice belongs to.

[0128] When using the techniques described in VVC Draft 7, a video encoder (video encoder 200) can treat a subpic as a picture by setting the flag subpic_treated_as_pic_flag[i] to 1. In this case, a video decoder (e.g., video decoder 300) can decode (e.g., extract) the subpic as a separate picture.

[0129] Although a video decoder (e.g., video encoder 200 or video decoder 300) reorders sub-pictures within an image, some sub-pictures may remain identical (e.g., unchanged). In some examples, the video decoder may perform sub-picture reordering according to the image. For example, the video decoder may be configured to reorder sub-pictures within a first image using a first order, and reorder sub-pictures within a second image using a second order different from the first order. In some cases, the video decoder may be configured to reorder sub-pictures within a first image using a first order, and avoid reordering sub-pictures within a second image. Therefore, when the video decoder reorders sub-pictures, the decoding result of the reordered sub-pictures can differ depending on whether the sub-picture is decoded as part of the whole image or as a separate image.

[0130] exist Figure 5 (Decoding sub-images as part of the whole image) and Figure 6 An example of this case, where the MV points to a difference reference block, is shown in (decoding only the extracted sub-image). Figure 5 In the example, reference image 140 (e.g., image N-1) includes sub-image 0 that is sorted before sub-image 1 (e.g., sub-image 0 is to the left of sub-image 1). As shown, reference block 144 of reference image 140 is located within sub-image 1 of reference image 140. However, the current image 142 (e.g., image N) includes sub-image 0 that is sorted after sub-image 1 (e.g., sub-image 0 is to the right of sub-image 1). As described below, reordering sub-image 0 that is sorted after sub-image 1 may result in reference block 144 of reference image 140 being located outside sub-image 1 of reference image 140.

[0131] The video decoder (e.g., video encoder 200 or video decoder 300) can be configured to use the current image size during reference sampling cropping (e.g., padding) using, for example, the following equation.

[0132] Set the variable picW to pic_width_in_luma_samples and set the variable picH to be equal to pic_height_in_luma_samples.

[0133]

[0134] yInti=Clip3(0,picH-1,yIntL+i-3) (958)

[0135] in,

[0136]

[0137] Furthermore, `sps_ref_wraparound_enabled_flag` is a flag indicating whether wraparound processing is enabled, `MinCbSizeY` is the minimum decoded block size along the vertical direction, `picW` is the width of the current image in sample count, and `xInt`... L It represents the brightness position in the horizontal direction, picH is the height of the current image relative to the sampler, and yInt... L It refers to the brightness position in the vertical direction.

[0138] The video decoder (e.g., video encoder 200 or video decoder 300) can be configured to crop the reference sampling coordinates according to the current image width and height, but the video decoder should instead use the reference image width and height as the current image width and height, and the reference image width and height can be different when resampling with the reference image.

[0139] When using the techniques described in VVC Draft 7, the video encoder (video encoder 200) can divide an image into one or more sub-images. The video encoder 200 can signal the sub-image partition layout that is signaled in the SPS. The video encoder 200 can signal the image sub-image layout mapped to the SPS layout in the PPS or the image header (PH). The video encoder 200 can signal the slice_subpic_id in the slice header (SH) to identify which sub-image the slice belongs to.

[0140] SPS: ...

[0142]

[0143] ...

[0145] PPS:

[0146]

[0147] PH: ...

[0149]

[0150] ...

[0152] SH: ...

[0154] ...

[0156] The video decoder (e.g., video encoder 200 or video decoder 300) can be configured as follows to export the sub-image mapped to the SPS layout according to the SubpicIdList[i] list for the i-th sub-image:

[0157]

[0158] A video decoder (video encoder 200 or video decoder 300) can treat a subpicture as a picture (e.g., encode or decode it) when subpic_treated_as_pic_flag[SubPicIdx] equals 1. In some examples, video encoder 200 can notify the subpic_treated_as_pic_flag[SubPicIdx] flag according to the subpicture signal with subpicture index SubPicIdx. In this example, the video decoder can extract the subpicture into a separate bitstream, and the video decoder (e.g., video decoder 200) can decode the subpicture as a separate picture. In this example, the video decoder can perform the same technique regardless of whether the subpicture is decoded as a picture or as part of a larger picture with other subpictures.

[0159] A subpic_treated_as_pic_flag[i] equal to 1 indicates that the i-th subpic of each decoded image in the Layer-Decoded Video Sequence (CLVS) is treated as an image during decoding, excluding loop filtering. A subpic_treated_as_pic_flag[i] equal to 0 indicates that the i-th subpic of each decoded image in the CLVS is not treated as an image during decoding, excluding loop filtering. If it does not appear, the value of subpic_treated_as_pic_flag[i] is inferred to be 0.

[0160] A video decoder (e.g., video encoder 200 or video decoder 300) can derive the sub-image boundaries as the left, top, right, and bottom boundary positions from the layout of the signal notification.

[0161] A video decoder (e.g., video encoder 200 or video decoder 300) can derive the variables SubPicLeftBoundaryPos, SubPicTopBoundaryPos, SubPicRightBoundaryPos, and SubPicBotBoundaryPos as follows:

[0162]

[0163]

[0164] RefSubPicLeftBoundaryPos is the left boundary of one or more reference image boundaries, RefSubPicRightBoundaryPos is the right boundary of one or more reference image boundaries, RefSubPicTopBoundaryPos is the top boundary of one or more reference image boundaries, RefSubPicBotBoundaryPos is the bottom boundary of one or more reference image boundaries, subpic_ctu_top_left_x[RefSubPicIdx] is the top-left corner of the reference image, CtbSizeY is the height of the decoded tree block of the video data, picW is the width of the image, subpic_width_minus1[RefSubPicIdx] is the width of the subpic - 1, CtbSize is the size of the decoded tree block, PicH is the height of the image, and subpic_height_minus1[RefSubPicIdx] is the height of the subpic - 1.

[0165] During decoding, the video decoder (e.g., video encoder 200 or video decoder 300) can be configured to crop reference samples to the sub-image boundaries when treating a sub-image as an image (e.g., encoding or decoding). For example, during motion-compensated interpolation, the video decoder can crop integer sample positions to the sub-image boundaries.

[0166] A video decoder (e.g., video encoder 200 or video decoder 300) can be configured to derive the luminance position (xInt) in full-sample units for i = 0..1, as follows: i ,yInt i ):

[0167] – If subpic_treated_as_pic_flag[SubPicIdx] equals 1, the following equation, referred to in this paper as "Equation (644)", applies:

[0168] xInt i =Clip3(SubPicLeftBoundaryPos,SubPicRightBoundaryPos,xInt L +i)(642)

[0169] yInt i=Clip3(SubPicTopBoundaryPos,SubPicBotBoundaryPos,yInt L +i) (643)

[0170] Otherwise (subpic_treated_as_pic_flag[SubPicIdx] equals 0), the following applies:

[0171]

[0172] yInt i =Clip3(0,picH-1,yInt) L +i) (645)

[0173] clip3 is a function

[0174] (xInt i ,yInt i ) is the brightness position measured in full sampling units. RefSubPicLeftBoundaryPos is the left boundary of one or more reference image boundaries, RefSubPicRightBoundaryPos is the right boundary of one or more reference image boundaries, RefSubPicTopBoundaryPos is the top boundary of one or more reference image boundaries, and RefSubPicBotBoundaryPos is the bottom boundary of one or more reference image boundaries.

[0175] Figure 5 This is a conceptual diagram illustrating the reordering of sub-images. The video encoder 200 can signal the sub-image mapping in PPS, PH, or SPS. In some examples, such as in... Figure 5 As shown above, the sub-image ID can be changed according to the image.

[0176] exist Figure 5 In the example, in the SPS layout, the video encoder 200 can be configured to signal two sub-pictures 0 and 1. In PPS or PH, the video decoder (e.g., video encoder 200 or video decoder 300) can be configured to reorder the sub-pictures. For example, as in... Figure 5 As shown, the video decoder can be configured to reorder the sub-pictures such that SPS sub-picture index 0 can correspond to the sub-picture with id 0 in the previous picture 140 and to the sub-picture with id 1 in the current picture 142.

[0177] When a sub-image is treated as an image, it can be decoded individually because other sub-images will not exist. For example, when decoding sub-image 1 of the current image 142 as part of a larger image, the video decoder (e.g., video encoder 200 or video decoder 300) can determine that the motion vector points to a reference block with the top-left corner coordinates (sub_pic0_width, 0), while in the extracted sub-image 1, the motion vector will point to (sub_pic0_width, 0) corresponding to a different reference sample.

[0178] In other words, for example, when a video decoder (e.g., video encoder 200 or video decoder 300) decodes (e.g., encodes or decodes) a sub-picture as a picture, the video decoder can decode the sub-picture individually as if the other sub-pictures would not exist. For example, when decoding sub-picture 1 as part of a larger picture, the video decoder can generate a motion vector pointing to a reference block with top-left corner coordinates (sub_pic0_width, 0). In this example, when decoding sub-picture 1 as part of an extracted sub-picture 1, the video decoder can generate a motion vector that still points to (sub_pic0_width, 0), which corresponds to a different reference sample than when the video decoder decodes sub-picture 1 as part of a larger picture.

[0179] For consistent predictions, the motion vector should point as follows: Figure 6 The image shows (0, 0). In this case, decoding the sub-image 1 of the current image 142 as part of a larger image or as a separate image does not provide the same output. In other words, for example, when decoding the sub-image as an image and when decoding the sub-image as part of a larger image, the video decoder (e.g., video encoder 200 or video decoder 300) can be configured to replace (sub_pic0_width, 0) (e.g., MV in...). Figure 6 The position pointed to by the middle pointer is used to generate a pointer to (0, 0) (for example, Figure 6 The motion vector of the top left corner of sub-image 0 of image N-1 shown in the image.

[0180] Figure 6 This is a concept map illustrating the extracted sub-images after the sub-images have been reordered.

[0181] Another issue is that during decoding, when sub-images are not treated as images, samples are cropped to the current image boundary using picW-1 or picH-1 (where PicW corresponds to the current image width and PicH corresponds to the current image height), or motion wrap is applied considering the current image size, as described, for example, in equation (644). In other words, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to crop the reference image boundary using picW-1 (e.g., the width of image N) and / or picH-1 (e.g., the height of image N). In some examples, the video decoder can be configured to apply motion wrap considering the current image size. For example, to apply motion wrap considering the current image size, the video decoder can be configured to apply equation (644).

[0182] However, when reference image resampling is enabled, the size of the reference image can be different from the size of the current image. In this case, especially when the reference image is smaller than the current image, using the current image boundaries during decoding may provide incorrect results. In other words, for example, when the reference image is smaller than the current image, the video decoder (e.g., video encoder 200 or video decoder 300) may provide incorrect results when using the current image boundaries during decoding.

[0183] The techniques described herein can solve the aforementioned problems and / or one or more other problems described herein. The techniques described herein can be used independently or in any combination. The techniques described herein can be applied to all subpicks; alternatively, for example, when subpic_treated_as_pic_flag[SubPicIdx] equals 1 for those subpicks with subpick index SubPicIdx, the described methods can be applied only to the subpicks that are considered as pictures.

[0184] To help alleviate or resolve sub-image reordering issues and / or other problems, the techniques described herein introduce reference sub-image left, top, right, and bottom boundaries for a reference image. A video decoder (e.g., video encoder 200 or video decoder 300) can be configured to derive these reference sub-image boundaries from a sub-image mapping specified for the reference image, for example, signaled in the PPS or PH of the reference image.

[0185] To identify subpicks, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to derive a reference subpick index RefSubPicIdx for the SPS layout during the mapping process using the slice_subpic_id of the current image. For example, the video decoder can be configured to divide an image into multiple subpicks based on a subpick mapping specified for the image, and identify the subpicks within the multiple subpicks that contain blocks of video data based on the subpick identifier. The video decoder can be configured to determine the subpick identifier of a subpick among multiple subpicks containing blocks of video data. In some examples, the block of video data includes slices containing (or) video data. In some examples, the block of video data is included within a slice of video data.

[0186] In some examples, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to derive reference sub-image boundaries from a sub-image layout as shown below.

[0187] Export RefSubPicIdx such that SubpicIdList[RefSubPicIdx] of the reference image is equal to slice_subpic_id.

[0188] That is, the video decoder can be configured to derive a reference sub-picture index for a sub-picture based on the sub-picture identifier.

[0189] A video decoder (e.g., video encoder 200 or video decoder 300) can be configured to derive one or more reference picture boundaries of a reference picture based on a subpicture identifier. For example, the video decoder can be configured to derive variables RefSubPicLeftBoundaryPos, RefSubPicTopBoundaryPos, RefSubPicRightBoundaryPos, and RefSubPicBotBoundaryPos as follows:

[0190] RefSubPicLeftBoundaryPos=subpic_ctu_top_left_x[RefSubPicIdx]*CtbSizeY

[0191] RefSubPicRightBoundaryPos=Min(picW-1,(subpic_ctu_top_left_x[RefSubPicIdx]+subpic_width_minus1[RefSubPicIdx]+1)*CtbSizeY-1)

[0192] RefSubPicTopBoundaryPos=subpic_ctu_top_left_y[RefSubPicIdx]*CtbSize

[0193] RefSubPicBotBoundaryPos=Min(picH-1,(subpic_ctu_top_left_y[RefSubPicIdx]+subpic_height_minus1[RefSubPicIdx]+1)*CtbSizeY-1)

[0194] A video decoder (e.g., video encoder 200 or video decoder 300) can be configured to perform reference sampling cropping using a reference sub-image position instead of the current sub-image position. That is, the video decoder can be configured to perform reference sampling cropping on blocks of video data based on one or more reference image boundaries to generate a reference sample set for the blocks of video data. In some examples, the video decoder can be configured to generate prediction information for the blocks of video data based on the reference sample set for the blocks.

[0195] An example is shown where the cropping process in equations (642) and (643) is modified as follows. In other words, for example, a video decoder (e.g., a video encoder 200 or a video decoder 300) can be configured to modify the cropping process in equations (642) and / or (643) as follows.

[0196] xInt i =Clip3(RefSubPicLeftBoundaryPos,RefSubPicRightBoundaryPos,xInt L +i) (642)

[0197] yInt i =Clip3(RefSubPicTopBoundaryPos,RefSubPicBotBoundaryPos,yInt L +i)(643)

[0198] A video decoder (e.g., video encoder 200 or video decoder 300) can use the sub-image boundaries of both the current sub-image and the reference sub-image to derive the displacement offset in motion compensation between the reference sample and the correct position. In other words, for example, the video decoder can be configured to use the sub-image boundaries of both the current sub-image and the reference sub-image to derive the displacement offset in motion compensation between the reference sample and the correct position.

[0199] In some examples, the video decoder can alter the motion vector (MV). However, since the motion vector can already be encoded in the bitstream and can be used as an MV predictor, this example could use recoding, which is not desirable for sub-picture extraction or decoding as individual images.

[0200] A general formula for displacement calculation can be expressed, for example, by considering the difference between the current and reference sub-image boundary positions, as shown below. In other words, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured, for example, to calculate displacement by considering the difference between the current and reference sub-image boundary positions, as shown below.

[0201] refPos=curPos-curSubPicBoundaryPos+refSubPicBoundaryPos.

[0202] In a detailed example using the left boundary position, the left boundary position can be described as follows. In other words, for example, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to calculate the left boundary position as follows.

[0203] refLeftPos=curLeftPos–SubPicLeftBoundaryPos+RefSubPicLeftBoundaryPos.

[0204] In the example above, refPos and curPos are the sampling positions of the sample in the current sub-image and the reference sample in the reference sub-image, as indicated by MV, respectively. curSubPicBoundaryPos is the boundary position of the current sub-image, and refSubPicBoundaryPos is the boundary position of the reference sub-image. In one example, it can be the left, right, top, or bottom sub-image boundary.

[0205] The video decoder (e.g., video encoder 200 or video decoder 300) can be configured to apply similar displacement offsets to other sub-picture boundary positions, such as, but not limited to, SubPicTopBoundaryPos and RefSubPicBottomBoundaryPos.

[0206] During motion compensation, the video decoder (e.g., video encoder 200 or video decoder 300) can be configured to add the displacement offset to other existing offsets, rather than adding the displacement at the end of the reference sampling position offset. In this example, the accuracy of the calculation can be improved compared to a system that adds the displacement at the end of the reference sampling position offset, because the position offset can be completed with higher precision; for example, the MV accuracy can be 1 / 16 pixel (1 / 16 of an integer sampling position).

[0207] In some examples, the video decoder (e.g., video encoder 200 or video decoder 300) can be configured to add the displacement offset to the existing scaling window offset during interpolation. Additions relative to VVC Draft 7 are highlighted with "**" in Example 1. Subtractions relative to VVC Draft 7 are highlighted with "→" and "←" in Example 1.

[0208] To help address the issue of reference images of different sizes, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to use the size of the reference image (e.g., width and / or height) instead of the current image size during cropping and wrapping at the reference location.

[0209] In some examples, the video decoder (e.g., video encoder 200 or video decoder 300) can be configured to modify picW and picH to the reference image width and reference image height, respectively. In some examples, the aforementioned example can be expressed as follows:

[0210] **Set the variable picW to equal the pic_width_in_luma_samples of the reference image refPicLX, and set the variable picH to equal the pic_height_in_luma_samples of the reference image refPicLX.**

[0211] Furthermore, the redefined picW and picH are used in cropping and wrapping processes, for example, as shown below. In other words, for example, a video decoder (e.g., a video encoder 200 or a video decoder 300) can be configured to use the redefined picW and picH in cropping and wrapping processes, as shown below.

[0212] Otherwise (subpic_treated_as_pic_flag[SubPicIdx] equals 0), the following applies:

[0213]

[0214] yInt i=Clip3(0,picH-1,yInt) L +i) (645)

[0215] in,

[0216]

[0217] Furthermore, `sps_ref_wraparound_enabled_flag` is a flag indicating whether wraparound processing is enabled, `MinCbSizeY` is the minimum decoded block size along the vertical direction, `picW` is the width of the reference image in samples, and `xInt`... L It refers to the brightness position in the horizontal direction, the height of the picH reference image with the sampler, and yInt. L It refers to the brightness position in the vertical direction.

[0218] That is, a video decoder (e.g., video encoder 200 or video decoder 300) can generate a set of reference samples for the current block of the current image based on the size of a reference image. For example, the video decoder can perform reference sample cropping on a reference block of the reference image based on the size of the reference image. In some examples, the video decoder can perform wrapping based on the size of the reference image. For example, the video decoder can generate samples from one or more reference sample sets based on the size of the reference image and a wrapping offset. Thus, the decoding accuracy of the video decoder can be improved compared to systems that can perform reference sample cropping on a reference block of the reference image based on the current image size and / or perform wrapping based on the current image size.

[0219] More specifically, the video decoder (e.g., video encoder 200 or video decoder 300) may apply a clipping function (e.g., Clip 3) that restricts the position of the reference sample along the horizontal direction of the reference image to a minimum of 0 and a maximum of the width of the reference image in terms of samples minus 1 (e.g., picW-1), and along the vertical direction of the reference image to a minimum of 0 and a maximum of the height of the reference image in terms of samples minus 1 (e.g., picH-1).

[0220] In some examples, the video decoder (e.g., video encoder 200 or video decoder 300) may perform wraparound processing based on the size of a reference image. For example, the video decoder may generate samples from one or more reference sample sets based on the size of the reference image and a wraparound offset (e.g., sps_ref_wraparound_offset_minus1). For example, the video decoder may compute ClipH((sps_ref_wraparound_offset_minus1+1)*MinCbSizeY,picW,(xIntL+i)), where clipH is a function that adds the wraparound offset, modified by the minimum decoded block size (e.g., MinCbSizeY), to the luminance position (e.g., xIntL).

[0221] TMVP Export Example

[0222] In one example, when deriving co-location locations, a video decoder (e.g., video encoder 200 or video decoder 300) can use the co-location locations to derive a temporal motion vector predictor for regular modes such as merge or AMVP modes, or for a sub-block-based temporal motion vector predictor. The video decoder can find the correct co-location location using reference sub-image boundaries and the current sub-image boundary. For example, the video decoder can be configured to determine the correct co-location location using reference sub-image boundaries and the current sub-image boundary. For example, when the video decoder derives co-location locations, it can derive a temporal motion vector predictor for regular modes such as, but not limited to, merge or AMVP modes by using reference sub-image boundaries and the current sub-image boundary to determine the correct co-location location. In some examples, when the video decoder (e.g., video encoder 200 or video decoder 300) derives co-location locations, it can derive a temporal motion vector predictor for a sub-block-based temporal motion vector predictor by using reference sub-image boundaries and the current sub-image boundary to determine the correct co-location location.

[0223] For example, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to form colPos = blockPos - curSubPicBoundaryPos + refSubPicBoundaryPos. The video decoder can be configured to compare the common block position colPos with the lower and / or right boundary positions of a reference sub-image rather than the current sub-image, since those sub-image positions may differ within the image, where blockPos can represent the current block position in the current sub-image.

[0224] In some examples, the techniques described herein can be implemented using the VVC draft language as follows, with changes highlighted using "→" and "←". For example, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to perform one or more steps of the following process. Changes relative to VVC draft 7 are highlighted in the TMVP export examples using "→" and "←".

[0225] 8.5.2.11 Derivation process for temporal brightness motion vector prediction

[0226] The input to this process is:

[0227] - The brightness position (xCb, yCb) of the current brightness decoded block relative to the top-left brightness sample of the current image.

[0228] - Specifies the width of the current decoded block in terms of luminance sampler, cbWidth.

[0229] - Specifies the height of the current decoded block in terms of the luminance sampler, cbHeight.

[0230] - Refer to the index refIdxLX, where X is 0 or 1.

[0231] The output of this process is:

[0232] - Motion vector prediction mvLXCol with 1 / 16 fractional sampling accuracy

[0233] -Availability flag: availableFlagLXCol.

[0234] The variable currCb specifies the current luminance decoding block at the luminance position (xCb, yCb).

[0235] The variables mvLXCol and availableFlagLXCo are exported as follows:

[0236] - If pic_temporal_mvp_enabled_flag is equal to 0 or (cbWidth*cbHeight) is less than or equal to 32, then set all two components of mvLXCol to equal 0 and set availableFlagLXCol to equal 0.

[0237] - Otherwise (pic_temporal_mvp_enabled_flag equals 1), the following sorted steps apply:

[0238] →

[0239] 1. Set the variable picW to be equal to pic_width_in_luma_samples of the reference image refPicLX, and set the variable picH to be equal to pic_height_in_luma_samples of the reference image refPicLX.

[0240] RefSubPicIdx is exported such that SubpicIdList[RefSubPicIdx] of the reference image is equal to slice_subpic_id. The variables RefSubPicLeftBoundaryPos, RefSubPicTopBoundaryPos, RefSubPicRightBoundaryPos, and RefSubPicBotBoundaryPos are exported as follows:

[0241] RefSubPicLeftBoundaryPos=subpic_ctu_top_left_x[RefSubPicIdx]*CtbSizeY

[0242] RefSubPicRightBoundaryPos=Min(picW-1,(subpic_ctu_top_left_x[RefSubPicIdx]+subpic_width_minus1[RefSubPicIdx]+1)*CtbSizeY-1)

[0243] RefSubPicTopBoundaryPos=subpic_ctu_top_left_y[RefSubPicIdx]*CtbSize

[0244] RefSubPicBotBoundaryPos=Min(picH-1,(subpic_ctu_top_left_y[RefSubPicIdx]+subpic_height_minus1[RefSubPicIdx]+1)*CtbSizeY-1)

[0245] 2. The lower right co-located motion vector and the sampling positions of the lower and right boundaries are derived as follows:

[0246] leftBoundaryPos=subpic_treated_as_pic_flag[SubPicIdx]? SubPicLeftBoundaryPos:0

[0247] topBoundaryPos = subpic_treated_as_pic_flag[SubPicIdx]? SubPicTopBoundaryPos : 0

[0248] refRightBoundaryPos = subpic_treated_as_pic_flag[SubPicIdx]? RefSubPicRightBoundaryPos : pic_width_in_luma_samples - 1

[0249] refBotBoundaryPos = subpic_treated_as_pic_flag[SubPicIdx]? RefSubPicBotBoundaryPos : pic_height_in_luma_samples - 1

[0250] refLeftBoundaryPos = subpic_treated_as_pic_flag[SubPicIdx]? RefSubPicLeftBoundaryPos : 0

[0251] refTopBoundaryPos = subpic_treated_as_pic_flag[SubPicIdx]? RefSubPicTopBoundaryPos : 0

[0252] xColBr = xCb + cbWidth – leftBoundaryPos + refLeftBoundaryPos

[0253] yColBr = yCb + cbHeight – topBoundaryPos + refTopBoundaryPos

[0254] ←

[0255] - If yCb >> CtbLog2SizeY equals yColBr >> CtbLog2SizeY, and yColBr is less than or equal to →

[0256] refBotBoundaryPos ←, and xColBr is less than or equal to → refRightBoundaryPos ←, then the following applies:

[0257] - The variable colCb specifies the luminance decoding block that overwrites the position of the co-located image specified by ColPic, given by ((xColBr>>3)<<3,(yColBr>>3)<<3).

[0258] - Set the luminance position (xColCb, yColCb) to be equal to the top left sample of the co-location luminance decoder block specified by colCb relative to the top left luminance sample of the co-location image specified by ColPic.

[0259] - If the inputs currCb, colCb, (xColCb, yColCb), refIdxLX, and sbFlag are set to 0, invoke the derivation procedure for co-locating motion vectors as specified in Clause 8.5.2.12, and assign the output to mvLXCol and availableFlagLXCol.

[0260] - Otherwise, set both components of mvLXCol to 0 and set availableFlagLXCol to 0.

[0261] 3. When availableFlagLXCol equals 0, the centrally co-located motion vector is derived as follows:

[0262] xColCtr=xCb+(cbWidth>>1)→–leftBoundaryPos+refLeftBoundaryPos←

[0263] yColCtr=yCb+(cbHeight>>1)→–topBoundaryPos+refTopBoundaryPos←

[0264] - The variable colCb specifies the luminance decoding block that overwrites the modified position of the image specified by ColPic within the co-location image, given by ((xColCtr>>3)<<3,(yColCtr>>3)<<3).

[0265] - Set the luminance position (xColCb, yColCb) to be equal to the top left sample of the co-location luminance decoder block specified by colCb relative to the top left luminance sample of the co-location image specified by ColPic.

[0266] - If the inputs currCb, colCb, (xColCb, yColCb), refIdxLX, and sbFlag are set to 0, invoke the derivation procedure for co-locating motion vectors as specified in Clause 8.5.2.12, and assign the output to mvLXCol and availableFlagLXCol.

[0267] In VVC Draft 7, the size of the co-placed image should be equal to the size of the current image. If this is not the case, then when exporting the reference sub-image position, pic_width_in_luma_samples and pic_height_in_luma_samples should be the width and height of the co-placed image.

[0268] In some examples, TMVP should be disabled if sub-image reordering is applied, when a sub-image is treated as an image, or when the current and reference sub-images have different sizes. For example, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to disable TMVP in response to determining that sub-image reordering has been applied. In some examples, a video decoder can be configured to disable TMVP in response to determining that a sub-image is treated as an image. In some examples, a video decoder can be configured to disable TMVP in response to determining that the current and reference sub-images have different sizes.

[0269] Example 1

[0270] 8.5.6.3.1 General Case

[0271] The input to this process is:

[0272] - Specifies the brightness position (xSb, ySb) of the current decoded sub-block relative to the top-left brightness sample of the current image.

[0273] - The variable sbWidth specifies the width of the current decoded subblock.

[0274] - The variable sbHeight specifies the height of the current decoded subblock.

[0275] - Motion vector offset mvOffset

[0276] - Improved motion vector refMvLX,

[0277] - The selected reference image sampling array refPicLX,

[0278] - Half-sampling interpolation filter index hpelIfIdx,

[0279] -Bidirectional optical flow flag bdofFlag

[0280] - Specifies the cIdx variable as the color component index of the current block.

[0281] - List of two scaling ratios, horizontal and vertical.

[0282] The output of this process is:

[0283] -Predict the array of sampled values ​​(sbWidth+brdExtSize)x(sbHeight+brdExtSize)predSamplesLX.

[0284] The predicted block boundary extension size brdExtSize is derived as follows:

[0285] brdExtSize=(bdofFlag||(inter_affine_flag[xSb][ySb]&&!pic_disable_prof_flag))? 2:0 (936)

[0286] Set the variables leftBoundaryOffset to equal scaling_win_left_offset, topBoundaryOffset to equal scaling_win_top_offset, and refSubPicLeftBoundaryPos and refSubPicTopBoundaryPos to equal 0.

[0287] **If subpic_treated_as_pic_flag[SubPicIdx] equals 1, then the following applies:**

[0288] **leftBoundaryOffset+=SubPicLeftBoundaryPos**

[0289] **topBoundaryOffset+=SubPicTopBoundaryPos**

[0290] **RefSubPicIdx is exported such that SubpicIdList[RefSubPicIdx] of the reference image is equal to slice_subpic_id.**

[0291] **refSubPicLeftBoundaryPos=subpic_ctu_top_left_x[RefSubPicIdx]*CtbSizeY**

[0292] **refSubPicTopBoundaryPos=subpic_ctu_top_left_y[RefSubPicIdx]*CtbSizeY**

[0293] Set the variable fRefLeftOffset to equal ((scaling_win_left_offset**+refSubPicLeftBoundaryPos**)<<10), where scaling_win_left_offset is the scaling_win_left_offset used as the reference image.

[0294] Set the variable fRefTopOffset to equal ((scaling_win_top_offset**+refSubPicTopBoundaryPos**)<<10), where scaling_win_top_offset is the scaling_win_top_offset used as the reference image.

[0295] The array predSamplesLX of predicted sample values ​​(sbWidth + brdExtSize) x (sbHeight + brdExtSize) is derived as follows:

[0296] - Set the motion vector mvLX to be equal to (refMvLX-mvOffset).

[0297] - If cIdx equals 0, then the following applies:

[0298] -Let (xInt) L ,yInt L ) represents the brightness position given in full sampling units, and (xFrac) represents the position of the brightness position given in full sampling units. L ,yFrac L The offset is given in 1 / 16 sample units. These variables are used only in this clause to specify the fractional sampling position within the reference sampling array refPicLX.

[0299] - The top-left coordinate (xSbInt) of the boundary block used for reference sampling filling L ,ySbInt L ) is set to equal to (xSb+(mvLX[0]>>4),ySb+(mvLX[1]>>4)).

[0300] -For each luminance sampling location (x) within the predicted luminance sampling array predSamplesLX L=0..sbWidth-1+brdExtSize,y L =0..sbHeight-1+brdExtSize), and the corresponding predicted brightness sample value predSamplesLX[xL][yL] is derived as follows:

[0301] - Let (refxSbL, refySbL) and (refxL, refyL) be the brightness positions indicated by the motion vectors (refMvLX[0], refMvLX[1]) given in 1 / 16 sampling units. The variable refxSb... L refx L ,refySb L and refy L The following was exported:

[0302] refxSbL=(((xSb–**leftBoundaryOffset**→scaling_win_left_offset←)<<4)+refMvLX[0])*scalingRatio[0] (937)

[0303] refx L =((Sign(refxSbL)*((Abs(refxSbL)+128)>>8)+x L *((scalingRatio[0]+8)>>4))+fRefLeftOffset+32)>>6 (938)

[0304] refySb L =(((ySb–**topBoundaryOffset**→scaling_win_top_offset→)<<4)+refMvLX[1])*scalingRatio[1] (939)

[0305] refy L =((Sign(refySbL)*((Abs(refySbL)+128)>>8)+y L *((scalingRatio[1]+8)>>4))+fRefTopOffset+32)>>6 (940)

[0306] -variable xInt L yInt L xFrac L and yFrac L The following was exported:

[0307] xIntL =refx L >>4 (941)

[0308] yInt L =refy L >>4 (942)

[0309] xFrac L =refx L &15 (943)

[0310] yFrac L =refy L &15 (944)

[0311] -Predicted luminance sample values ​​predSamplesLX[x] L ][y L The following was exported:

[0312] - If bdofFlag equals TRUE or (pic_disable_prof_flag equals FALSE and inter_affine_flag[xSb][ySb] equals TRUE), and one or more of the following conditions are true, then by (xInt) L +(xFrac L >>3)-1), yInt L +(yFrac L >>3)-1) and refPicLX are used as inputs to call the luminance integer sampling acquisition procedure as specified in Clause 8.5.6.3.3 to derive the predicted luminance sample value predSamplesLX[x L ][y L ].

[0313] -x L It equals 0.

[0314] -x L It equals sbWidth+1.

[0315] -y L It equals 0.

[0316] -y L It equals sbHeight + 1.

[0317] - Otherwise, by using (xIntL-(brdExtSize>0?1:0), yIntL-(brdExtSize>0?1:0)), (xFrac L ,yFrac L (xSbInt) L,ySbInt L The predicted luminance sample values ​​predSamplesLX[xL][yL] are derived by calling the luminance sample 8-beat interpolation filtering procedure as specified in Clause 8.5.6.3.2 as inputs.

[0318] - Otherwise (cIdx is not equal to 0), the following applies:

[0319] - Let (xIntC, yIntC) be the chromaticity position given in full sample units, and (xFracC, yFracC) be the offset given in 1 / 32 sample units. These variables are used only in this clause to specify the general fractional sample position within the reference sample array refPicLX.

[0320] - Set the top left coordinates (xSbIntC, ySbIntC) of the boundary block used for reference sampling filling to equal ((xSb / SubWidthC)+(mvLX[0]>>5),(ySb / SubHeightC)+(mvLX[1]>>5)).

[0321] - For each chromaticity sampling position (xC = 0..sbWidth-1, yC = 0..sbHeight-1) within the predicted chromaticity sampling array predSamplesLX, the corresponding predicted chromaticity sampling value predSamplesLX[xC][yC] is derived as follows:

[0322] - Let (refxSbC, refySbC) and (refxC, refyC) be the chromaticity positions indicated by the motion vectors (refMvLX[0], refMvLX[1]) given in 1 / 32 sampling units. The variable refxSb... C ,refySb C refx C and refy C The following was exported:

[0323] addX=sps_chroma_horizontal_collocated_flag? 0:8*(scalingRatio[0]-(1<<14)) (945)

[0324] addY=sps_chroma_vertical_collocated_flag? 0:8*(scalingRatio[1]-(1<<14)) (946)

[0325] refxSb C =(((xSb - **leftBoundaryOffset** → scaling_win_left_offset ←) / SubWidthC << 5) + refMvLX[0]) * scalingRatio[0] + addX (947)

[0326] refx C =((Sign(refxSb C ) * ((Abs(refxSb C ) + 256) >> 9) + xC * ((scalingRatio[0] + 8) >> 4)) + fRefLeftOffset / SubWidthC + 16) >> 5 (948)

[0327] refySb C =(((ySb - **topBoundaryOffset → scaling_win_top_offset ←) / SubHeightC << 5) + refMvLX[1]) * scalingRatio[1] + addY (949)

[0328] refy C =((Sign(refySb C ) * ((Abs(refySb C ) + 256) >> 9) + y C * ((scalingRatio[1] + 8) >> 4)) + fRefTopOffset / SubHeightC + 16) >> 5 (950)

[0329] - The variables xInt C 、yInt C 、xFrac C and yFrac C are derived as follows:

[0330] xInt C =refx C >> 5 (951)

[0331] yInt C =refy C >> 5 (952)

[0332] xFrac C =refx C & 31 (953)

[0333] yFrac C =refy C &31 (954)

[0334] -by using (xInt) C ,yInt C (xFrac) C, yFrac C (xSbInt) C ,ySbInt C The predicted sample values ​​predSamplesLX[xC][yC] are derived from the procedure specified in Clause 8.5.6.3.4 using sbWidth, sbHeight, refPicLX, scalingRatio[0] and scalingRatio[1] as inputs.

[0335] Note - This procedure differs from the procedure specified in Clause 8.4.5.2.13; it uses both sps_chroma_vertical_collocated_flag and sps_chroma_horizontal_collocated_flag.

[0336] Example IIA

[0337] An example of a specification modification is shown below, where the change is indicated by "***".

[0338] 8.5.3.2.2 Brightness Sampling Bilinear Interpolation Process

[0339] The input to this process is:

[0340] –Use the brightness position (xInt) of the full sampling unit L ,yInt L ),

[0341] – Brightness position using fractional sampling units (xFrac) L ,yFrac L ),

[0342] –Luminance reference sampling array refPicLX L .

[0343] The output of this process is the predicted luminance sample value, predSampleLX. L

[0344] The variables shift1, shift2, shift3, shift4, offset1, offset2, and offset3 are exported as follows:

[0345] shift1 = BitDepth - 6 (635)

[0346] offset1 = 1 << (shift1 - 1) (636)

[0347] shift2 = 4 (637)

[0348] offset2 = 1 << (shift2 - 1) (638)

[0349] shift3 = 10-BitDepth (639)

[0350] shift4 = BitDepth - 10 (640)

[0351] offset4 = 1 << (shift4 - 1) (641)

[0352] Set the variable picW to equal the pic_width_in_luma_samples*** of the reference image refPicLX***, and set the variable picH to equal the pic_height_in_luma_samples*** of the reference image refPicLX***.

[0353] Used to equal xFrac L Or yFrac L The brightness interpolation filter coefficients fb at each 1 / 16 fractional sampling position p L [p] is specified in Table 28.

[0354] For i = 0..1, the brightness position (xInt) of the full sampling unit is used. i ,yInt i The following was exported:

[0355] – If subpic_treated_as_pic_flag[SubPicIdx] equals 1, then the following applies:

[0356] ***RefSubPicIdx is exported such that SubpicIdList[RefSubPicIdx] of the reference image is equal to slice_subpic_id.

[0357] The variables RefSubPicLeftBoundaryPos, RefSubPicTopBoundaryPos, RefSubPicRightBoundaryPos, and RefSubPicBotBoundaryPos are exported as follows:

[0358] RefSubPicLeftBoundaryPos = subpic_ctu_top_left_x[RefSubPicIdx] * CtbSizeY

[0359] RefSubPicRightBoundaryPos = Min(picW - 1, (subpic_ctu_top_left_x[RefSubPicIdx] + subpic_width_minus1[RefSubPicIdx] + 1) * CtbSizeY - 1)

[0360] RefSubPicTopBoundaryPos = subpic_ctu_top_left_y[RefSubPicIdx] * CtbSize

[0361] RefSubPicBotBoundaryPos = Min(picH - 1, (subpic_ctu_top_left_y[RefSubPicIdx] + subpic_height_minus1[RefSubPicIdx] + 1) * CtbSizeY - 1)***

[0362] xInt i = Clip3(***Ref***SubPicLeftBoundaryPos, ***Ref***SubPicRightBoundaryPos, xInt L + i) (642)

[0363] yInt i = Clip3(***Ref***SubPicTopBoundaryPos, ***Ref***SubPicBotBoundaryPos, yInt L + i) (643)

[0364] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] equals 0), the following applies:

[0365]

[0366] yInt i = Clip3(0, picH - 1, yInt L + i) (645)

[0367] The predicted luminance sample value predSampleLX L is derived as follows:

[0368] –If xFrac L and yFrac L If both are equal to 0, then predSampleLX L The value is exported as follows:

[0369] predSampleLX L =BitDepth<=10? (refPicLX) L [xInt0][yInt0]< <shift3):((refPicLX L [xInt0][yInt0]+offset4)>>shift4) (646)

[0370] Otherwise, if xFrac L Not equal to 0 and yFrac L If it equals 0, then predSampleLX L The value is exported as follows:

[0371]

[0372] Otherwise, if xFrac L 0 and yFrac L If it is not equal to 0, then predSampleLX L The value is exported as follows:

[0373]

[0374] Otherwise, if xFrac L Not equal to 0 and yFrac L If it is not equal to 0, then predSampleLX L The value is exported as follows:

[0375] –The sampling array temp[n] where n = 0..1 is derived as follows:

[0376]

[0377] – Predicted luminance sample value predSampleLX L The following was exported:

[0378]

[0379] 8.5.6.3 Fractional Sampling Interpolation Process

[0380] 8.5.6.3.1 General Case

[0381] The input to this process is:

[0382] – Specifies the brightness position (xSb, ySb) of the current decoded sub-block relative to the top-left brightness sample of the current image.

[0383] – The variable sbWidth specifies the width of the current decoded subblock.

[0384] – The variable sbHeight specifies the height of the current decoded subblock.

[0385] – Motion vector offset mvOffset

[0386] – Improved motion vector refMvLX,

[0387] – The selected reference image sampling array refPicLX,

[0388] – Half-sampling interpolation filter index hpelIfIdx,

[0389] – Bidirectional optical flow flag bdofFlag

[0390] – Specifies the color component index of the current block, cIdx.

[0391] - List of two scaling ratios, horizontal and vertical: scalingRatio.

[0392] The output of this process is:

[0393] – The array predSamplesLX of the predicted sample values ​​(sbWidth+brdExtSize)x(sbHeight+brdExtSize).

[0394] The predicted block boundary extension size brdExtSize is derived as follows:

[0395] brdExtSize=(bdofFlag||(inter_affine_flag[xSb][ySb]&&!pic_disable_prof_flag))? 2:0 (936)

[0396] ***Set the variable leftBoundaryOffset to equal scaling_win_left_offset, set topBoundaryOffset to equal scaling_win_top_offset, and set refSubPicLeftBoundaryPos and refSubPicTopBoundaryPos to equal 0.

[0397] If subpic_treated_as_pic_flag[SubPicIdx] equals 1, then the following applies:

[0398] leftBoundaryOffset+=SubPicLeftBoundaryPos

[0399] topBoundaryOffset+=SubPicTopBoundaryPos

[0400] RefSubPicIdx is exported such that SubpicIdList[RefSubPicIdx] of the reference image is equal to slice_subpic_id.

[0401] refSubPicLeftBoundaryPos=subpic_ctu_top_left_x[RefSubPicIdx]*CtbSizeY

[0402] refSubPicTopBoundaryPos=subpic_ctu_top_left_y[RefSubPicIdx]*CtbSizeY**

[0403] Set the variable fRefLeftOffset to equal (***(***scaling_win_left_offset***+refSubPicLeftBoundaryPos)***<<10), where scaling_win_left_offset is the scaling_win_left_offset used as the reference image.

[0404] Set the variable fRefTopOffset to equal (***(***scaling_win_top_offset***+refSubPicTopBoundaryPos)***<<10), where scaling_win_top_offset is the scaling_win_top_offset used as the reference image.

[0405] The array predSamplesLX of the predicted sample values ​​(sbWidth+brdExtSize)x(sbHeight+brdExtSize) is derived as follows: – Set the motion vector mvLX to be equal to (refMvLX-mvOffset).

[0406] – If cIdx equals 0, then the following applies:

[0407] – Let (xInt) L ,yInt L ) represents the brightness position given in full sampling units, and (xFrac) represents the position of the brightness position given in full sampling units. L ,yFrac L The offset is given in 1 / 16 sample units. These variables are used only in this clause to specify the fractional sampling position within the reference sampling array refPicLX.

[0408] – The top-left coordinate (xSbInt) of the boundary block used for reference sampling filling L ,ySbInt L ) is set to equal to (xSb+(mvLX[0]>>4),ySb+(mvLX[1]>>4)).

[0409] – For each luminance sampling location (x) within the predicted luminance sampling array predSamplesLX L =0..sbWidth-1+brdExtSize,y L =0..sbHeight-1+brdExtSize), the corresponding predicted brightness sample value predSamplesLX[x L ][y L The following was exported:

[0410] – Let (refxSb) L ,refySb L )and(refx L ,refy L ) represents the brightness position indicated by the motion vector (refMvLX[0], refMvLX[1]), given in 1 / 16 sampling units. The variable refxSb L refx L ,refySb L and refy L The following was exported:

[0411] refxSb L =(((xSb-***leftBoundaryOffset***)<<4)+refMvLX[0])*scalingRatio[0] (937)

[0412] refx L =((Sign(refxSb) L )*((Abs(refxSb L )+128)>>8)+x L*((scalingRatio[0]+8)>>4))+fRefLeftOffset+32)>>6 (938)

[0413] refySb L =(((ySb-***topBoundaryOffset***)<<4)+refMvLX[1])*scalingRatio[1] (939)

[0414] refy L =((Sign(refySb) L )*((Abs(refySb L )+128)>>8)+yL*((scalingRatio[1]+8)>>4))+fRefTopOffset+32)>>6 (940)

[0415] – Variable xInt L yInt L xFrac L and yFrac L The following was exported:

[0416] xInt L =refx L >>4 (941)

[0417] yInt L =refy L >>4 (942)

[0418] xFrac L =refx L &15 (943)

[0419] yFrac L =refy L &15 (944)

[0420] – Predicted luminance sample values ​​predSamplesLX[x] L ][y L The following was exported:

[0421] – If bdofFlag equals TRUE or (pic_disable_prof_flag equals FALSE and inter_affine_flag[xSb][ySb] equals TRUE), and one or more of the following conditions are true, then by (xInt) L +(xFrac L >>3)-1), yInt L+(yFrac L >>3)-1) and refPicLX are called as input as in the clause

[0422] The luminance integer sampling acquisition process specified in 8.5.6.3.3 derives the predicted luminance sample value predSamplesLX[x]. L ][y L ].

[0423] -x L It equals 0.

[0424] -x L It equals sbWidth+1.

[0425] –y L It equals 0.

[0426] –y L It equals sbHeight + 1.

[0427] Otherwise, by using (xIntL-(brdExtSize>0?1:0),yIntL-(brdExtSize>0?1:0)), (xFracL,yFracL), (xSbInt) L ,ySbInt L The predicted luminance sample values ​​predSamplesLX[xL][yL] are derived by calling the luminance sample 8-beat interpolation filtering procedure as specified in Clause 8.5.6.3.2 as inputs.

[0428] Otherwise (cIdx is not equal to 0), the following applies:

[0429] – Let (xIntC, yIntC) be the chromaticity position given in full sample units, and (xFracC, yFracC) be the offset given in 1 / 32 sample units. These variables are used only in this clause to specify the general fractional sample position within the reference sample array refPicLX.

[0430] – Set the top left coordinates (xSbIntC, ySbIntC) of the boundary block used for reference sampling filling to equal ((xSb / SubWidthC)+(mvLX[0]>>5),(ySb / SubHeightC)+(mvLX[1]>>5)).

[0431] – For each chromaticity sampling position (xC = 0..sbWidth-1, yC = 0..sbHeight-1) within the predicted chromaticity sampling array predSamplesLX, the corresponding predicted chromaticity sampling value predSamplesLX[xC][yC] is derived as follows:

[0432] – Let (refxSb) C ,refySb C ) and (refx C ,refy C ) represents the chromaticity position indicated by the motion vector (refMvLX[0], refMvLX[1]) using a 1 / 32 sampling unit. The variable refxSb C ,refySb C refx C and refy C The following was exported:

[0433] addX=sps_chroma_horizontal_collocated_flag? 0:8*(scalingRatio[0]-(1<<14)) (945)

[0434] addY=sps_chroma_vertical_collocated_flag? 0:8*(scalingRatio[1]-(1<<14)) (946)

[0435] refxSb C =(((xSb-***leftBoundaryOffset***) / SubWidthC<<5)+refMvLX[0])*scalingRatio[0]+addX (947)

[0436] refx C =((Sign(refxSb) C )*((Abs(refxSb C )+256)>>9)+xC*((scalingRatio[0]+8)>>4))+fRefLeftOffset / SubWidthC+16)>>5 (948)

[0437] refySb C =(((ySb-***topBoundaryOffset***) / SubHeightC<<5)+refMvLX[1])*scalingRatio[1]+addY (949)

[0438] refy C =((Sign(refySb) C )*((Abs(refySb C )+256)>>9)+yC*((scalingRatio[1]+8)>>4))+fRefTopOffset / SubHeightC+16)>>5 (950)

[0439] – Variable xInt C yInt C xFrac C and yFrac C The following was exported:

[0440] xInt C =refx C >>5 (951)

[0441] yInt C =refy C >>5 (952)

[0442] xFrac C =refx C &31 (953)

[0443] yFrac C =refy C &31 (954)

[0444] – Derive the predicted sample values ​​predSamplesLX[xC][yC] by calling the procedure specified in Clause 8.5.6.3.4 with (xIntC,yIntC), (xFracC,yFracC), (xSbIntC,ySbIntC), sbWidth, sbHeight, refPicLX, scalingRatio[0] and scalingRatio[1] as input.

[0445] Note – This procedure differs from the procedure specified in Clause 8.4.5.2.13, and uses both sps_chroma_vertical_collocated_flag and sps_chroma_horizontal_collocated_flag.

[0446] 8.5.6.3.2 Brightness Sampling Interpolation Filtering Process

[0447] The input to this process is:

[0448] –Use the brightness position (xInt) of the full sampling unitL ,yInt L ),

[0449] – Brightness position using fractional sampling units (xFrac) L ,yFrac L ),

[0450] – Specifies the top-left luminance position (xSbInt) of the boundary block used for reference sampling filling relative to the top-left luminance sample of the reference image, using full-sample units. L ,ySbInt L ),

[0451] –Luminance reference sampling array refPicLX L ,

[0452] – Half-sampling interpolation filter index hpelIfIdx,

[0453] – The variable sbWidth specifies the width of the current child block.

[0454] – The variable sbHeight specifies the height of the current child block.

[0455] – The fixed-point representation of the horizontal scaling factor is scalingRatio[0].

[0456] – The fixed-point representation of the vertical scaling factor, scalingRatio[1],

[0457] – Specifies the brightness position (xSb, ySb) of the current sub-block relative to the top-left brightness sample of the current image.

[0458] The output of this process is the predicted luminance sample value, predSampleLX. L

[0459] The variables shift1, shift2, and shift3 are exported as follows:

[0460] Set the variable shift1 to equal Min(4, BitDepth-8), the variable shift2 to equal 6, and the variable shift3 to equal Max(2, 14-BitDepth).

[0461] – Set the variable picW to be equal to the pic_width_in_luma_samples*** of the reference image refPicLX***, and set the variable picH to be equal to the pic_height_in_luma_samples*** of the reference image refPicLX***.

[0462] For each equal to xFrac or yFrac L The horizontal brightness interpolation filter coefficient f at the 1 / 16 fractional sampling position p LH [p] is exported as follows:

[0463] – If MotionModelIdc[xSb][ySb] is greater than 0, and both sbWidth and sbHeight are equal to 4, then the luminance interpolation filter coefficient fLH[p] is specified in Table 32.

[0464] Otherwise, if scalingRatio[0] is greater than 28672, then the luminance interpolation filter coefficient f LH [p] is specified in Table 30.

[0465] Otherwise, if scalingRatio[0] is greater than 20480, then the luminance interpolation filter coefficient f LH [p] is specified in Table 31.

[0466] Otherwise, depending on hpelIfIdx, the luminance interpolation filter coefficient f LH [p] is specified in Table 29.

[0467] For each equal to yFrac L The vertical brightness interpolation filter coefficient f at the 1 / 16 fractional sampling position p LV [p] is exported as follows:

[0468] – If MotionModelIdc[xSb][ySb] is greater than 0, and both sbWidth and sbHeight are equal to 4, then the luminance interpolation filter coefficient fLV[p] is specified in Table 32.

[0469] Otherwise, if scalingRatio[1] is greater than 28672, then the luminance interpolation filter coefficient f LV [p] is specified in Table 30.

[0470] Otherwise, if scalingRatio[1] is greater than 20480, then the luminance interpolation filter coefficient f LV [p] is specified in Table 31.

[0471] Otherwise, depending on hpelIfIdx, the luminance interpolation filter coefficient f LV [p] is specified in Table 29.

[0472] For i = 0..7, the brightness position (xInt) of the full sampling unit is used. i ,yInt i The following was exported:

[0473] – If subpic_treated_as_pic_flag[SubPicIdx] equals 1, then the following applies:

[0474] ***RefSubPicIdx is exported such that SubpicIdList[RefSubPicIdx] of the reference image is equal to slice_subpic_id.

[0475] The variables RefSubPicLeftBoundaryPos, RefSubPicTopBoundaryPos, RefSubPicRightBoundaryPos, and RefSubPicBotBoundaryPos are exported as follows:

[0476] RefSubPicLeftBoundaryPos=subpic_ctu_top_left_x[RefSubPicIdx]*CtbSizeY

[0477] RefSubPicRightBoundaryPos=Min(picW-1,(subpic_ctu_top_left_x[RefSubPicIdx]+subpic_width_minus1[RefSubPicIdx]+1)*CtbSizeY-1)

[0478] RefSubPicTopBoundaryPos=subpic_ctu_top_left_y[RefSubPicIdx]*CtbSize

[0479] RefSubPicBotBoundaryPos=Min(picH-1,(subpic_ctu_top_left_y[RefSubPicIdx]+subpic_height_minus1[RefSubPicIdx]+1)*CtbSizeY-1)***

[0480] xInt i =Clip3(***Ref***SubPicLeftBoundaryPos,***Ref***SubPicRightBoundaryPos,xInt L +i-3) (955)

[0481] yInt i=Clip3(***Ref***SubPicTopBoundaryPos,***Ref***SubPicBotBoundaryPos,yInt L +i-3) (956)

[0482] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] equals 0), the following applies:

[0483]

[0484] yInt i =Clip3(0,picH-1,yInt) L +i-3) (958)

[0485] For i = 0..7, the brightness position using the full sampling unit is further modified as follows:

[0486] xInt i =Clip3(xSbInt) L -3,xSbInt L +sbWidth+4,xInt i (959)

[0487] yInt i =Clip3(ySbInt) L -3,ySbInt L +sbHeight+4,yInt i (960)

[0488] Predicted luminance sample value predSampleLX L The following was exported:

[0489] –If xFrac L and yFrac L If both are equal to 0, and both scalingRatio[0] and scalingRatio[1] are less than 20481, then predSampleLX L The value is exported as follows:

[0490] predSampleLX L =refPicLX L [xInt3][yInt3]< <shift3 (961)

[0491] – Otherwise, if yFrac LIf the value is equal to 0 and scalingRatio[1] is less than 20481, then predSampleLX L The value is exported as follows:

[0492]

[0493] Otherwise, if xFrac L If the value is equal to 0 and scalingRatio[0] is less than 20481, then predSampleLX L The value is exported as follows:

[0494]

[0495] –Otherwise, predSampleLX L The value is exported as follows:

[0496] –The sampling array temp[n] where n = 0..7 is derived as follows:

[0497]

[0498] – Predicted luminance sample value predSampleLX L The following was exported:

[0499]

[0500] 8.5.6.3.3 Luminance Integer Sampling Extraction Process

[0501] The input to this process is:

[0502] –Use the brightness position (xInt) of the full sampling unit L ,yInt L ),

[0503] –Luminance reference sampling array refPicLX L ,

[0504] The output of this process is the predicted luminance sample value, predSampleLX. L

[0505] Set the variable shift to equal Max(2, 14-BitDepth).

[0506] Set the variable picW to be equal to the pic_width_in_luma_samples*** of the reference image refPicLX***, and set the variable picH to be equal to the pic_height_in_luma_samples*** of the reference image refPicLX***.

[0507] The brightness position (xInt, yInt) using full sampling units is derived as follows:

[0508] – If subpic_treated_as_pic_flag[SubPicIdx] equals 1, then the following applies:

[0509] ***RefSubPicIdx is exported such that SubpicIdList[RefSubPicIdx] of the reference image is equal to slice_subpic_id.

[0510] The variables RefSubPicLeftBoundaryPos, RefSubPicTopBoundaryPos, RefSubPicRightBoundaryPos, and RefSubPicBotBoundaryPos are exported as follows:

[0511] RefSubPicLeftBoundaryPos=subpic_ctu_top_left_x[RefSubPicIdx]*CtbSizeY

[0512] RefSubPicRightBoundaryPos=Min(picW-1,(subpic_ctu_top_left_x[RefSubPicIdx]+subpic_width_minus1[RefSubPicIdx]+1)*CtbSizeY-1)

[0513] RefSubPicTopBoundaryPos=subpic_ctu_top_left_y[RefSubPicIdx]*CtbSize

[0514] RefSubPicBotBoundaryPos=Min(picH-1,(subpic_ctu_top_left_y[RefSubPicIdx]+subpic_height_minus1[RefSubPicIdx]+1)*CtbSizeY-1)***

[0515] xInt=Clip3(***Ref***SubPicLeftBoundaryPos,***Ref***SubPicRightBoundaryPos,xInt L (966)

[0516] yInt=Clip3(***Ref***SubPicTopBoundaryPos,***Ref***SubPicBotBoundaryPos,yInt L (967)

[0517] –Otherwise, the following applies:

[0518]

[0519] yInt = Clip3(0, picH-1, yInt) L (969)

[0520] Predicted luminance sample value predSampleLX L The following was exported:

[0521] predSampleLX L =refPicLX L [xInt][yInt]< <shift3 (970)

[0522] 8.5.6.3.4 Chromaticity Sampling Interpolation Process

[0523] The input to this process is:

[0524] – Use chromaticity position (xInt) in full sampling units C ,yInt C ),

[0525] – Chromaticity position using 1 / 32 fractional sampling unit (xFrac) C ,yFrac C – Specifies the top-left chroma position (xSbIntC, ySbIntC) of the boundary block used for reference sampling fill relative to the top-left chroma sample of the reference image, using full sampling units.

[0526] – The variable sbWidth specifies the width of the current child block.

[0527] – The variable sbHeight specifies the height of the current child block.

[0528] – Chromaticity reference sampling array refPicLX C ,

[0529] – The fixed-point representation of the horizontal scaling factor is scalingRatio[0].

[0530] – The fixed-point representation of the vertical scaling factor is scalingRatio[1].

[0531] The output of this process is the predicted chromaticity sample value, predSampleLX. C

[0532] The variables shift1, shift2, and shift3 are exported as follows:

[0533] Set the variable shift1 to equal Min(4, BitDepth-8), the variable shift2 to equal 6, and the variable shift3 to equal Max(2, 14-BitDepth).

[0534] – Change variable picW C Set it to equal to pic_width_in_luma_samples / SubWidthC*** of the reference image refPicLX***, and set the variable picH C Set to equal to the reference image refPicLX***'s pic_height_in_luma_samples / SubHeightC***. Used for each equal to xFrac C The horizontal chromaticity interpolation filter coefficient f at the 1 / 32 fractional sampling position p CH [p] is exported as follows:

[0535] – If scalingRatio[0] is greater than 28672, then the chroma interpolation filter coefficient f CH [p] is specified in Table 35.

[0536] Otherwise, if scalingRatio[0] is greater than 20480, then the chroma interpolation filter coefficients f CH [p] is specified in Table 60.

[0537] Otherwise, the chroma interpolation filter coefficients f CH [p] is specified in Table 33.

[0538] For each equal to yFrac C The vertical chromaticity interpolation filter coefficient f at the 1 / 32 fractional sampling position p CV [p] is exported as follows:

[0539] – If scalingRatio[1] is greater than 28672, then the chroma interpolation filter coefficient f CV [p] is specified in Table 35.

[0540] Otherwise, if scalingRatio[1] is greater than 20480, then the chroma interpolation filter coefficients f CV [p] is specified in Table 60.

[0541] Otherwise, the chroma interpolation filter coefficients f CV [p] is specified in Table 33.

[0542] Set the variable xOffset to equal (sps_ref_wraparound_offset_minus1+1)*MinCbSizeY) / SubWidthC.

[0543] For i = 0..3, the chromaticity position (xInt) of the full sampling unit is used. i ,yInt i The following was exported:

[0544] – If subpic_treated_as_pic_flag[SubPicIdx] equals 1, then the following applies:

[0545] ***RefSubPicIdx is exported such that SubpicIdList[RefSubPicIdx] of the reference image is equal to slice_subpic_id.

[0546] The variables RefSubPicLeftBoundaryPos, RefSubPicTopBoundaryPos, RefSubPicRightBoundaryPos, and RefSubPicBotBoundaryPos are exported as follows:

[0547] RefSubPicLeftBoundaryPos=subpic_ctu_top_left_x[RefSubPicIdx]*CtbSizeY

[0548] RefSubPicRightBoundaryPos=Min(picW-1,(subpic_ctu_top_left_x[RefSubPicIdx]+subpic_width_minus1[RefSubPicIdx]+1)*CtbSizeY-1)

[0549] RefSubPicTopBoundaryPos=subpic_ctu_top_left_y[RefSubPicIdx]*CtbSize

[0550] RefSubPicBotBoundaryPos=Min(picH-1,(subpic_ctu_top_left_y[RefSubPicIdx]+subpic_height_minus1[RefSubPicIdx]+1)*CtbSizeY-1)***

[0551] xInt i =Clip3(***Ref***SubPicLeftBoundaryPos / SubWidthC,***Ref***SubPicRightBoundaryPos / SubWidthC,xInt*** C*** +i) (971)

[0552] yInt i =Clip3(***Ref***SubPicTopBoundaryPos / SubHeightC,***Ref***SubPicBotBoundaryPos / SubHeightC,yInt*** C*** +i) (972)

[0553] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] equals 0), the following applies:

[0554]

[0555] yInt i =Clip3(0,picH C -1,yInt C +i-1) (974)

[0556] For i = 0..3, the chromaticity position (xInt) of the full sampling unit is used. i ,yInt i The following has been further modified:

[0557] xInt i =Clip3(xSbIntC-1,xSbIntC+sbWidth+2,xInt i (975)

[0558] yInt i =Clip3(ySbIntC-1,ySbIntC+sbHeight+2,yInt i (976)

[0559] Predicted chromaticity sample value predSampleLX C The following was exported:

[0560] –If xFrac C and yFrac C If both are equal to 0, and both scalingRatio[0] and scalingRatio[1] are less than 20481, then predSampleLX C The value is exported as follows:

[0561] predSampleLX C =refPicLX C [xInt1][yInt1]< <shift3 (977)

[0562] – Otherwise, if yFrac C If the value is equal to 0 and scalingRatio[1] is less than 20481, then predSampleLX C The value is exported as follows:

[0563]

[0564] Otherwise, if xFrac C If the value is equal to 0 and scalingRatio[0] is less than 20481, then predSampleLX C The value is exported as follows:

[0565]

[0566] –Otherwise, predSampleLX C The value is exported as follows:

[0567] –The sampling array temp[n] where n = 0..3 is derived as follows:

[0568]

[0569] –Predicted chromaticity sample value predSampleLX C The following was exported:

[0570]

[0571] Example IIB

[0572] According to the technology of this disclosure, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to derive a reference sub-picture boundary in addition to the current sub-picture boundary. In this example, the video decoder can be configured to crop using the reference sub-picture boundary and add the difference between the current and reference boundary positions to motion compensation.

[0573] Similarly, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to modify the co-location position used for TMVP export by adding the difference between the reference and the left boundary position of the current sub-image to the horizontal co-location position. In this example, the video decoder can be configured to add the difference between the reference and the upper boundary position of the current sub-image to the vertical co-location position.

[0574] According to the technology disclosed herein, a video decoder (e.g., a video encoder 200 or a video decoder 300) can replace the current image width and height with the width and height of a reference image when performing reference sampling cropping.

[0575] Below is an example of such a change. In Example IIB, additions relative to VVC Draft 7 are highlighted with "**". Subtractions relative to VVC Draft 7 are highlighted with "→" and "←".

[0576] 8.5.6.3.2 Brightness Sampling Interpolation Filtering Process ...

[0578] Set the variable picW to equal the pic_width_in_luma_samples of the reference image refPicLX**, and set the variable picH to equal the pic_height_in_luma_samples of the reference image refPicLX**.

[0579] For each equal to xFrac L Or yFrac L The brightness interpolation filter coefficient fb at the 1 / 16 fractional sampling position p L [p] is specified in Table 28.

[0580] For i = 0..1, the brightness position (xInt) of the full sampling unit is used. i ,yInt i The following was exported:

[0581] – If subpic_treated_as_pic_flag[SubPicIdx] equals 1, then the following applies:

[0582] **

[0583] RefSubPicIdx is exported such that SubpicIdList[RefSubPicIdx] of the reference picture is equal to slice_subpic_id.

[0584] The variables RefSubPicLeftBoundaryPos, RefSubPicTopBoundaryPos, RefSubPicRightBoundaryPos

[0585] and RefSubPicBotBoundaryPos are exported as follows:

[0586] RefSubPicLeftBoundaryPos = subpic_ctu_top_left_x[RefSubPicIdx] * CtbSizeY

[0587] RefSubPicRightBoundaryPos = Min(picW - 1, (subpic_ctu_top_left_x[RefSubPicIdx] + subpic_width_minus1[RefSubPicIdx] + 1) * CtbSizeY - 1)

[0588] RefSubPicTopBoundaryPos = subpic_ctu_top_left_y[RefSubPicIdx] * CtbSize

[0589] [[ID=,20]]RefSubPicBotBoundaryPos = Min(picH - 1, (subpic_ctu_top_left_y[RefSubPicIdx] + subpic_height_minus1[RefSubPicIdx] + 1) * CtbSizeY - 1)

[0590] **

[0591] xInt i = Clip3(**Ref**SubPicLeftBoundaryPos, **Ref**SubPicRightBoundaryPos, xInt L + i) (642) <,

[0592] yInt i It should be noted that there seems to be a small formatting issue in the original text where "RefSubPicBotBoundaryPos" in line 20 has an extra comma in the translation. Also, there might be an issue with the "**Ref**" in lines 28 and could be a formatting error in the original. You may want to double-check the original text for accuracy.=Clip3(**Ref**SubPicTopBoundaryPos,**Ref**SubPicBotBoundaryPos,yInt L +i) (643)

[0593] – Otherwise (subpic_treated_as_pic_flag[SubPicIdx] equals 0), the following applies:

[0594]

[0595] yInt i =Clip3(0,picH-1,yInt) L +i) (645)

[0596] …

[0597] 8.5.6.3 Fractional Sampling Interpolation Process

[0598] 8.5.6.3.1 General Case

[0599] …

[0600] **

[0601] Set the variable leftBoundaryOffset to equal scaling_win_left_offset, and set topBoundaryOffset to equal...

[0602] In scaling_win_top_offset, set refSubPicLeftBoundaryPos and refSubPicTopBoundaryPos to equal 0.

[0603] If subpic_treated_as_pic_flag[SubPicIdx] equals 1, then the following applies:

[0604] leftBoundaryOffset+=SubPicLeftBoundaryPos

[0605] topBoundaryOffset+=SubPicTopBoundaryPos

[0606] RefSubPicIdx is exported such that SubpicIdList[RefSubPicIdx] of the reference image is equal to slice_subpic_id.

[0607] refSubPicLeftBoundaryPos=subpic_ctu_top_left_x[RefSubPicIdx]*CtbSizeY

[0608] refSubPicTopBoundaryPos=subpic_ctu_top_left_y[RefSubPicIdx]*CtbSizeY

[0609] **

[0610] Set the variable fRefLeftOffset to equal ((scaling_win_left_offset**+refSubPicLeftBoundaryPos**)<<10), where scaling_win_left_offset is the scaling_win_left_offset used as the reference image.

[0611] Set the variable fRefTopOffset to equal ((scaling_win_top_offset**+refSubPicTopBoundaryPos**)<<10), where scaling_win_top_offset is the scaling_win_top_offset used as the reference image.

[0612] …

[0613] – Let (refxSb) L ,refySb L ) and (refx L ,refy L ) represents the brightness position indicated by the motion vector (refMvLX[0], refMvLX[1]), given in 1 / 16 sampling units. The variable refxSb L refx L ,refySb L and refy L Exported

[0614] as follows:

[0615] refxSb L =(((xSb–**leftBoundaryOffset**→scaling_win_left_offset←)<<4)+refMvLX[0])*scalingRatio[0] (937)

[0616] refx L=((Sign(refxSb) L )*((Abs(refxSb L )+128)>>8)+x L *((scalingRatio[0]+8)>>4))+fRefLeftOffset+32)>>6 (938)

[0617] refySb L =(((ySb–topBoundaryOffset**→scaling_win_top_offset←)<<4)+refMvLX[1])*scalingRatio[1] (939)

[0618] refy L =((Sign(refySb) L )*((Abs(refySb L )+128)>>8)+yL*((scalingRatio[1]+8)>>4))+fRefTopOffset+32)>>6 (940)

[0619] – Variable xInt L yInt L xFrac L and yFrac L The following was exported:

[0620] xInt L =refx L >>4 (941)

[0621] yInt L =refy L >>4 (942)

[0622] xFrac L =refx L &15 (943)

[0623] yFrac L =refy L &15 (944)

[0624] …

[0625] – Let (refxSb) C ,refySb C ) and (refx C ,refy C) represents the chromaticity position indicated by the motion vector (refMvLX[0], refMvLX[1]) using a 1 / 32 sampling unit. The variable refxSb C ,refySb C refx C and refy C The following was exported:

[0626] addX=sps_chroma_horizontal_collocated_flag? 0:8*(scalingRatio[0]-(1<<14)) (945)

[0627] addY=sps_chroma_vertical_collocated_flag? 0:8*(scalingRatio[1]-(1<<14)) (946)

[0628] refxSb C =(((xSb–**leftBoundaryOffset**→scaling_win_left_offset←) / SubWidthC<<5)+refMvLX[0])*scalingRatio[0]+addX (947)

[0629] refx C =((Sign(refxSb) C )*((Abs(refxSb C )+256)>>9)+xC*((scalingRatio[0]+8)>>4))+fRefLeftOffset / SubWidthC+16)>>5 (948)

[0630] refySb C =(((ySb–**topBoundaryOffset**→scaling_win_top_offset←) / SubHeightC<<5)+refMvLX[1])*scalingRatio[1]+addY (949)

[0631] refy C =((Sign(refySb) C )*((Abs(refySb C )+256)>>9)+yC*((scalingRatio[1]+8)>>4))+fRefTopOffset / SubHeightC+16)>>5 (950)

[0632] – Variable xInt C yInt C xFrac C and yFrac C The following was exported:

[0633] xInt C =refx C >>5 (951)

[0634] yInt C =refy C >>5 (952)

[0635] xFrac C =refx C &31 (953)

[0636] yFrac C =refy C &31 (954)

[0637] Example of disabling sub-image reordering

[0638] In some examples, sub-picture reordering is not allowed for different pictures. For example, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to disallow (e.g., disable) sub-picture reordering for different pictures. For example, a video decoder can be configured to reorder sub-pictures of all pictures of video data and / or avoid reordering sub-pictures of the first picture of video data differently from the second picture of video data.

[0639] In some examples, it can be stated that pps_subpic_id[i] should appear in ascending order or in the same order as sps_subpic_id. For example, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to present pps_subpic_id[i] in ascending order. In this example, pps_subpic_id[i] specifies the subpicture ID of the i-th subpicture in the picture parameter set. In some examples, a video decoder can be configured to present pps_subpic_id[i] in the same order as sps_subpic_id[i]. In this example, pps_subpic_id[i] specifies the subpicture ID of the i-th subpicture in the picture parameter set, and sps_subpic_id[i] specifies the subpicture ID of the i-th subpicture in the sequence parameter set.

[0640] Similarly, for example, ph_subpic_id[i] should appear in ascending order or in the same order as pps_subpic_id or sps_subpic_id. For example, a video decoder (e.g., video encoder 200 or video decoder 300) can be configured to present ph_subpic_id[i] in ascending order or in the same order as pps_subpic_id or sps_subpic_id.

[0641] Figure 7 This is a flowchart illustrating an example method for encoding the current block. The current block may include the current CU. Although it is combined with a video encoder 200 ( Figure 1 and 3 This is as described, but it should be understood that other devices can be configured to perform the same actions. Figure 7 Similar to the method.

[0642] In this example, the video encoder 200 initially makes a prediction (350) for the current block. For example, the video encoder 200 can form a prediction block for the current block. For example, the sample generation unit 225 of the video encoder 200 can perform reference sample cropping on the reference block of the reference image based on the size of the reference image. In some examples, the sample generation unit 225 can perform wrap processing based on the size of the reference image. For example, the sample generation unit 225 can generate samples from one or more reference sample sets based on the size of the reference image and the wrap offset. Thus, the decoding accuracy of the video encoder 200 can be improved compared to systems that can perform reference sample cropping on the reference block of the reference image based on the current image size and / or perform wrap processing based on the current image size.

[0643] The video encoder 200 can then compute a residual block for the current block (352). To compute the residual block, the video encoder 200 can compute the difference between the original, uncoded block and the predicted block of the current block. The video encoder 200 can then transform and quantize the coefficients of the residual block (354). Next, the video encoder 200 can scan the quantized transform coefficients of the residual block (356). During or after the scan, the video encoder 200 can entropy-encode the transform coefficients (358). For example, the video encoder 200 can encode the transform coefficients using CAVLC or CABAC. The video encoder 200 can then output the entropy-encoded data of the block (360).

[0644] Figure 8 This is a flowchart illustrating an example method for decoding the current block of video data. The current block may include the current CU. Although it is combined with video decoder 300 ( Figure 1 and 4This is as described, but it should be understood that other devices can be configured to perform the same actions. Figure 8 Similar to the method.

[0645] The video decoder 300 can receive entropy-encoded data of the current block, such as entropy-encoded prediction information and entropy-encoded data of the coefficients of the residual block corresponding to the current block (370). The video decoder 300 can entropy decode the entropy-encoded data to determine the prediction information for the current block and the transform coefficients for reproducing the residual block (372). The video decoder 300 can, for example, use an intra-frame prediction mode or an inter-frame prediction mode indicated by the prediction information for the current block (374) to compute a prediction block for the current block.

[0646] For example, the sampling generation unit 317 of the video decoder 300 can perform reference sample cropping on reference blocks of the reference image based on the size of the reference image. In some examples, the sampling generation unit 317 can perform wrap processing based on the size of the reference image. For example, the sampling generation unit 317 can generate samples from one or more reference sample sets based on the size of the reference image and the wrap offset. Thus, compared to systems that can perform reference sample cropping on reference blocks of the reference image based on the current image size and / or perform wrap processing based on the current image size, the decoding accuracy of the video decoder 300 can be improved.

[0647] The video decoder 300 can then perform an inverse scan (376) on the reproduced transform coefficients to create a block of quantized transform coefficients. The video decoder 300 can then perform inverse quantization and inverse transform on the transform coefficients to produce a residual block (378). The video decoder 300 can finally decode the current block by merging the predicted block and the residual block (380).

[0648] Figure 9 This is a flowchart illustrating an example method for generating prediction blocks using the size of a reference image. Figure 9 The method can be performed by video encoder 200 or video decoder 300. For example, video encoder 200 can perform the decoding during the video encoding and / or decoding process (e.g., during...). Figure 7 The method is executed during step 350, or during the decoding loop of the encoding process. Figure 9 The method. As another example, the video decoder 300 can perform [the following] during the video decoding process. Figure 9 The method. In Figure 9 In the example, the video decoder (e.g., the mode selection unit 202 of the video encoder 200 or the prediction processing unit 304 of the video decoder 300) can use motion information to determine a reference block (402) of a reference image for predicting the current block of the current image.

[0649] A video decoder (e.g., the sample generation unit 215 of video encoder 200 or the sample generation unit 317 of video decoder 300) can generate a reference sample set for the current block of the current image, wherein generating the reference sample set includes performing reference sample cropping (404) on the reference block of the reference image based on the size of the reference image. The size of the reference image can indicate the width of the reference image in samples, the height of the reference image in samples, or both the width and height of the reference image in samples. For example, the video decoder can perform reference sample cropping on the reference block of the reference image based on the height of the reference image, the width of the reference image, or both the width and height of the reference image, wherein the width and height of the reference image are measured in samples.

[0650] In some examples, the video decoder (e.g., the sample generation unit 215 of the video encoder 200 or the sample generation unit 317 of the video decoder 300) may apply a cropping function that restricts the location of the reference sample. For example, the video decoder may apply a cropping function to restrict the sample to a set of samples (e.g., available samples) arranged within a reference image. For example, the video decoder may apply a cropping function that restricts the location of the reference sample along the horizontal direction of the reference image to a minimum of 0 and a maximum of the width of the reference image in samples minus 1 (e.g., picW-1), and along the vertical direction of the reference image to a minimum of 0 and a maximum of the height of the reference image in samples minus 1 (e.g., picH-1). In some examples, the size of the reference image is different from the size of the current image (e.g., larger or smaller).

[0651] In some examples, a video decoder (e.g., video encoder 200 or video decoder 300) may perform wraparound processing based on the size of a reference image. For example, the video decoder may generate samples from one or more reference sample sets based on the size of the reference image and the wraparound offset. For example, the video decoder may compute ClipH((sps_ref_wraparound_offset_minus1+1)*MinCbSizeY,picW,(xIntL+i)), where clipH is a function that adds the wraparound offset, modified according to the minimum decoded block size (e.g., MinCbSizeY), to the luma location (e.g., xIntL).

[0652] A video decoder (e.g., mode selection unit 202 of video encoder 200 or prediction processing unit 304 of video decoder 300) can generate a prediction block (406) for the current block of the current image based on a reference sample set.

[0653] In some examples, a video decoder (e.g., video decoder 300) can decode a residual block for the current block of the current image, and decode the current block of the current image using the prediction block and the residual block. For example, video decoder 300 can merge the prediction block and the residual block on a per-sample basis. Similarly, a video encoder (e.g., video encoder 200) can decode a residual block for the current block of the current image, and decode the current block of the current image using the prediction block and the residual block. In some examples, a video encoder (e.g., video encoder 200) can generate a residual block for the current block of the current image using the current block and the prediction block, and encode the residual block.

[0654] Figure 10 This is a flowchart illustrating an example method for generating prediction blocks using reference sub-image boundaries. Figure 10 The method can be performed by video encoder 200 or video decoder 300. For example, video encoder 200 can perform the decoding during the video encoding and / or decoding process (e.g., during...). Figure 7 The method is executed during step 350, or during the decoding loop of the encoding process. Figure 10 The method. As another example, the video decoder 300 can perform [the following] during the video decoding process. Figure 10 The method. It can be used with... Figure 9 Methods together or replacement Figure 9 The method used Figure 10 The method.

[0655] exist Figure 10 In some examples, a video decoder (e.g., mode selection unit 202 of video encoder 200 or prediction processing unit 304 of video decoder 300) can determine the partitioning of an image into multiple sub-images (502). For example, video decoder 300 can determine a sub-image mapping specified for a reference image based on a sub-image mapping specified for the image. For example, video decoder 300 can determine a sub-image mapping specified for a reference image based on signaling information in the picture parameter set (PPS) or picture header (PH) of the reference image, and determine the partitioning of the reference image based on the sub-image mapping specified for the reference image. In some examples, the video decoder can use signaling information in the sequence parameter set (SPS) to determine the sub-image partitioning layout for the image, and determine the partitioning of the reference image based on the sub-image partitioning layout. In some examples, a video encoder (e.g., video encoder 200) can partition an image into multiple sub-images.

[0656] A video decoder (e.g., video encoder 200 or video decoder 300) can determine a sub-block identifier (504) of one of a plurality of sub-pictures containing video data. In some examples, the sub-picture identifier includes slice_subpic_id. In some examples, the video decoder can determine the sub-picture identifier based on the slice_subpic_id signaled in the slice header (SH).

[0657] A video decoder (e.g., video encoder 200 or video decoder 300) can derive a reference subpicture index for a subpicture based on the subpicture identifier (506). For example, the video decoder can derive a reference subpicture index for the subpicture to which the subpicture identifier is mapped. For example, the video decoder can derive RefSubPicIdx such that SubpicIdList[RefSubPicIdx] of the reference picture is equal to slice_subpic_id. The slice_subpic_id can be signaled in the slice header to identify the subpicture.

[0658] A video decoder (e.g., video encoder 200 or video decoder 300) can derive one or more reference sub-picture boundaries (508) of a reference picture based on a reference sub-picture index. For example, the video decoder can calculate the following:

[0659] RefSubPicLeftBoundaryPos=subpic_ctu_top_left_x[RefSubPicIdx]*CtbSizeY

[0660] RefSubPicRightBoundaryPos=Min(picW-1,(subpic_ctu_top_left_x[RefSubPicIdx]+subpic_width_minus1[RefSubPicIdx]+1)*CtbSizeY-1)

[0661] RefSubPicTopBoundaryPos=subpic_ctu_top_left_y[RefSubPicIdx]*CtbSize

[0662] RefSubPicBotBoundaryPos=Min(picH-1,(subpic_ctu_top_left_y[RefSubPicIdx]+subpic_height_minus1[RefSubPicIdx]+1)*CtbSizeY-1)

[0663] Wherein, RefSubPicLeftBoundaryPos is the left boundary of one or more reference image boundaries, RefSubPicRightBoundaryPos is the right boundary of one or more reference image boundaries, RefSubPicTopBoundaryPos is the top boundary of one or more reference image boundaries, RefSubPicBotBoundaryPos is the bottom boundary of one or more reference image boundaries, subpic_ctu_top_left_x[RefSubPicIdx] is the top left corner of the reference image, CtbSizeY is the height of the decoding tree block used for video data, picW is the width of the image, subpic_width_minus1[RefSubPicIdx] is the width of the subpick minus 1, CtbSize is the size of the decoding tree block, PicH is the height of the image, and subpic_height_minus1[RefSubPicIdx] is the height of the subpick minus 1.

[0664] In some examples, the video decoder (e.g., video encoder 200 or video decoder 300) can perform reference sampling cropping using reference sub-image boundaries. For example, the video decoder may compute the following.

[0665] xInti=Clip3(RefSubPicLeftBoundaryPos,RefSubPicRightBoundaryPos,xIntL+i)

[0666] yInti=Clip3(RefSubPicTopBoundaryPos,RefSubPicBotBoundaryPos,yIntL+i)

[0667] Where clip3 is a function

[0668] Among them, (xInt) i ,yInt i ) is the brightness position using full sampling units. RefSubPicLeftBoundaryPos is the left boundary of one or more reference image boundaries. RefSubPicRightBoundaryPos is the right boundary of one or more reference image boundaries. RefSubPicTopBoundaryPos is the top boundary of one or more reference image boundaries. RefSubPicBotBoundaryPos is the bottom boundary of one or more reference image boundaries.

[0669] The video decoder can generate prediction blocks based on one or more reference sub-image boundaries and one or more current sub-image boundaries (510). For example, the video decoder can derive displacement offsets for motion compensation based on one or more reference image boundaries of a reference image. In this example, the video decoder can generate prediction blocks for blocks of video data generated based on displacement offsets for motion compensation. In this case, the accuracy of the computation can be improved compared to a system that does not use one or more reference sub-image boundaries and one or more current sub-image boundaries because the position derivation can be performed with higher precision; for example, the MV accuracy can be 1 / 16 pixel (1 / 16 of an integer sample position). In some examples, the video decoder can add the displacement offsets for motion compensation to the scaling window offset.

[0670] In some examples, the video decoder (e.g., video encoder 200 or video decoder 300) can generate a TMVP using one or more reference subpicture boundaries and one or more current subpicture boundaries. In some examples, the video decoder can add the difference between the positions of the current subpicture and the reference subpictures. For example, the video decoder can calculate colPos = blockPos - curSubPicBoundaryPos + refSubPicBoundaryPos, where colPos is the corrected co-occurrence block position, blockPos is the current block position in the current subpicture, curSubPicBoundaryPos is one or more current subpicture boundaries, and refSubPicBoundaryPos is one or more reference subpicture boundaries. Using the corrected co-occurrence block position can improve the decoding accuracy of the video decoder.

[0671] In some examples, a video decoder (e.g., video encoder 200 or video decoder 300) can determine, based on signaling information, whether a sub-picture is treated as a complete picture type (e.g., encoded or decoded). In this example, the video decoder can derive one or more reference picture boundaries in response to determining that a sub-picture is treated as a complete picture type.

[0672] In some examples, a video decoder (e.g., video encoder 200 or video decoder 300) can determine whether a subpic is treated as a full picture type when subpic_treated_as_pic_flag[SubPicIdx] equals 1. In this example, the video decoder can derive one or more reference picture boundaries in response to determining that the subpic is treated as a full picture type.

[0673] The following terms are a non-limiting list of examples of one or more technologies based on this disclosure.

[0674] Clause 1. A method for processing video data, the method comprising: dividing an image into a plurality of sub-images based on a sub-image mapping specified for the image; determining a sub-image identifier for one of the plurality of sub-images containing a block of video data; deriving a reference sub-image index for the sub-image based on the sub-image identifier; deriving one or more reference image boundaries of the reference image based on the reference sub-image index; performing reference sampling cropping on a reference block identified by motion information of a block of video data based on the one or more reference image boundaries of the reference image to generate a reference sample set for the block of video data; and generating a predicted block for the block of video data based on the reference sample set.

[0675] Clause 2. The method described in Clause 1, wherein the block of video data is included in a slice of video data.

[0676] Clause 3. The method of any combination of Clauses 1-2 further includes: determining a sub-picture mapping specified for the reference picture based on signaling information in the picture parameter set (PPS) or picture header (PH) of the reference picture; and dividing the reference picture based on the sub-picture mapping specified for the reference picture.

[0677] Clause 4. The method of any combination of Clauses 1-3 further includes: determining a sub-picture partitioning layout for a picture using signaling information in a sequence parameter set (SPS), wherein the partitioning of the picture is based on the sub-picture partitioning layout.

[0678] Clause 5. The method of any combination of Clauses 1-4, wherein the subpicture identifier includes slice_subpic_id, and wherein the determination of the subpicture identifier is based on slice_subpic_id signaled in the slice header (SH).

[0679] Clause 6. The method of any combination of Clauses 1-5, wherein deriving a reference sub-image index for a sub-image includes: deriving a reference sub-image index for the sub-image to which the sub-image identifier is mapped.

[0680] Clause 7. The method of any combination of Clauses 1-6, wherein deriving one or more reference picture boundaries of the reference picture includes calculating the following:

[0681] RefSubPicLeftBoundaryPos=subpic_ctu_top_left_x[RefSubPicIdx]*CtbSizeY

[0682] RefSubPicRightBoundaryPos=Min(picW-1,(subpic_ctu_top_left_x[RefSubPicIdx]+subpic_width_minus1[RefSubPicIdx]+1)*CtbSizeY-1)

[0683] RefSubPicTopBoundaryPos=subpic_ctu_top_left_y[RefSubPicIdx]*CtbSize

[0684] RefSubPicBotBoundaryPos=Min(picH-1,(subpic_ctu_top_left_y[RefSubPicIdx]+subpic_height_minus1[RefSubPicIdx]+1)*CtbSizeY-1),

[0685] Wherein, RefSubPicLeftBoundaryPos is the left boundary of one or more reference image boundaries, RefSubPicRightBoundaryPos is the right boundary of one or more reference image boundaries, RefSubPicTopBoundaryPos is the top boundary of one or more reference image boundaries, RefSubPicBotBoundaryPos is the bottom boundary of one or more reference image boundaries, subpic_ctu_top_left_x[RefSubPicIdx] is the top left corner of the reference image, CtbSizeY is the height of the decoding tree block used for video data, picW is the width of the image, subpic_width_minus1[RefSubPicIdx] is the width of the subpick minus 1, CtbSize is the size of the decoding tree block, PicH is the height of the image, and subpic_height_minus1[RefSubPicIdx] is the height of the subpick minus 1.

[0686] Clause 8. The method of any combination of Clauses 1-7, wherein performing reference sampling clipping includes calculating the following: xInti = Clip3(RefSubPicLeftBoundaryPos,RefSubPicRightBoundaryPos,xIntL+i)

[0687] yInt = Clip3(RefSubPicTopBoundaryPos,RefSubPicBotBoundaryPos,yIntL+i), where clip3 is a function Clip3(x,y,z) = {■("x"&";"&"z<x"@"y"&";"&"z> y"@"z"&";"&"otherwise")┤, where (xInti,yInti) is the brightness position using full sampling units, RefSubPicLeftBoundaryPos is the left boundary of one or more reference image boundaries, RefSubPicRightBoundaryPos is the right boundary of one or more reference image boundaries, RefSubPicTopBoundaryPos is the top boundary of one or more reference image boundaries, and RefSubPicBotBoundaryPos is the bottom boundary of one or more reference image boundaries.

[0688] Clause 9. The method of any combination of Clauses 1-8 further includes: determining, based on signaling information, whether a sub-image is treated as a complete image type, wherein deriving one or more reference image boundaries is performed in response to determining that the sub-image is treated as a complete image type.

[0689] Clause 10. The method of any combination of Clauses 1-9 further comprises: determining whether a subpic is treated as a full picture type when subpic_treated_as_pic_flag[SubPicIdx] equals 1, wherein deriving one or more reference picture boundaries is performed in response to determining that the subpic is treated as a full picture type.

[0690] Clause 11. The method of any combination of Clauses 1-10 further comprises: deriving a displacement offset for motion compensation based on one or more reference image boundaries of a reference image, wherein the predicted block of the video data is generated based on the displacement offset for motion compensation.

[0691] Clause 12. The method of Clause 11, wherein generating a predicted block for a block of video data comprises: adding a displacement offset for motion compensation to a scaling window offset.

[0692] Clause 13. The method of any combination of Clauses 1-12 further includes: determining a reference image size for a reference image, wherein performing reference sampling cropping includes using the reference image size.

[0693] Clause 14. The method of any combination of Clauses 1-13 further includes: determining a reference image size of a reference image; and using the reference image size in the wrapping process.

[0694] Clause 15. The method described in any combination of Clauses 1-14 further includes: determining a reference image size for a reference image; and modifying the image size of an image to correspond to the reference image size.

[0695] Clause 16. A method comprising: dividing an image into a plurality of current sub-images based on a sub-image mapping specified for the image; determining a sub-image identifier for one of the plurality of current sub-images containing a block of video data; deriving a reference sub-image index for the current sub-image based on the sub-image identifier; deriving one or more reference image boundaries of the reference image based on the reference sub-image index; determining a corrected co-location block position using the one or more reference sub-image boundaries and the one or more current sub-image boundaries of the sub-image; and generating a predicted block for the block of video data based on the corrected co-location block position.

[0696] Clause 17. The method described in Clause 16, wherein determining the corrected co-location of the block includes calculating the following:

[0697] colPos=blockPos-curSubPicBoundaryPos+refSubPicBoundaryPos,

[0698] Where colPos is the corrected co-positioned block position, blockPos is the current block position in the current sub-image, curSubPicBoundaryPos is one or more current sub-image boundaries, and refSubPicBoundaryPos is one or more reference sub-image boundaries.

[0699] The method described in any of Clauses 18.16-17 includes: comparing the corrected co-location block position with the lower boundary position and / or the right boundary position of the reference sub-image.

[0700] Clause 19. A method comprising: dividing a first image of video data into a first plurality of sub-images, the first plurality of sub-images being arranged in a first order corresponding to an ascending order and / or an order of sequence parameter sets used for the first plurality of sub-images; dividing a second image of video data into a second plurality of sub-images, the second plurality of sub-images being arranged in a second order corresponding to an ascending order and / or an order of sequence parameter sets used for the second plurality of sub-images; avoiding reordering the first plurality of sub-images in the first image from the first order, and avoiding reordering the second plurality of sub-images in the second image from the second order; generating a first prediction block for a first block of sub-images in the first plurality of sub-images using the first plurality of sub-images arranged in the first order; and generating a second prediction block for a second block of sub-images in the second plurality of sub-images using the second plurality of sub-images arranged in the second order.

[0701] The method described in any combination of Clauses 1-19, as described in Clause 20, further comprises: decoding residual blocks of a block of video data; and decoding a block of video data using prediction blocks and residual blocks.

[0702] Clause 21. The method of any combination of Clauses 1-20 further comprises: generating residual blocks for blocks of video data using blocks of video data and prediction blocks; and encoding the residual blocks.

[0703] Clause 22. An apparatus for decoding video data, the apparatus comprising one or more units for performing the method described in any one of Clauses 1-21.

[0704] Clause 23. The device described in Clause 22, wherein one or more units include one or more processors implemented in circuitry.

[0705] The device described in any of Clauses 22 and 23 further includes a memory for storing video data.

[0706] The device described in any one of Clauses 22-24, as described in Clause 25, further includes a display configured to display decoded video data.

[0707] Clause 26. The device described in any one of Clauses 22-25, wherein the device includes one or more of a camera, computer, mobile device, broadcast receiver device or set-top box.

[0708] Clause 27. The device described in any one of Clauses 22-26, wherein the device includes a video decoder.

[0709] Clause 28. The device described in any one of Clauses 22-27, wherein the device includes a video encoder.

[0710] Clause 29. A computer-readable storage medium having instructions stored therein, which, when executed, cause one or more processors to perform the method described in any one of Clauses 1-21.

[0711] It should be recognized that, depending on the example, specific actions or events of any of the techniques described herein may be performed in a different order, and may be added, combined, or excluded entirely (e.g., not all of the described actions or events are necessary to practice these techniques). Furthermore, in certain examples, actions or events may be performed concurrently rather than sequentially, for example, through multithreading, interrupt handling, or multiple processors.

[0712] In one or more examples, the described functionality can be implemented using hardware, software, firmware, or any combination thereof. If implemented in software, the functionality can be stored or transmitted as one or more instructions or code on a computer-readable medium, and executed by a hardware-based processing unit. A computer-readable medium can include a computer-readable storage medium or a communication medium corresponding to a tangible medium such as a data storage medium, including any medium that facilitates the transmission of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium can generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. A data storage medium can be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products can include computer-readable media.

[0713] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Furthermore, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead involve non-transient, tangible storage media. As used herein, disks and optical discs include compact discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically magnetically copy data, while optical discs optically copy data using lasers. Combinations of the above items should also be included within the scope of computer-readable media.

[0714] Instructions can be executed by one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functions described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, the techniques may be fully implemented in one or more circuit or logic elements.

[0715] The technologies disclosed herein can be implemented in a variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or collections of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed technologies, without necessarily requiring implementation by different hardware units. Rather, as described above, various units can be combined into codec hardware units, or provided as a collection of interoperable hardware units (including one or more processors as described above) in conjunction with suitable software and / or firmware.

[0716] Various examples have been described. These and other examples fall within the scope of the following claims.

Claims

1. A method for decoding video data, the method comprising: Use motion information to determine a reference block of a reference image for predicting the current block of the current image; Generate a reference sampling set for the current block of the current image, wherein generating the reference sampling set includes: When the size of the reference image is different from the size of the current image, wrapping processing is performed based on the size of the reference image. When the size of the reference image is different from the size of the current image, reference sampling cropping is performed on the reference block of the reference image based on the size of the reference image; and A predicted block for the current block of the current image is generated based on the reference sampling set.

2. The method according to claim 1, wherein, The size of the reference image includes the width of the reference image in terms of samples.

3. The method according to claim 1, wherein, The size of the reference image includes its height in sample measurements.

4. The method according to claim 1, wherein, Performing reference sampling cropping on the reference block includes applying a cropping function that restricts the position of the reference sample along the horizontal direction of the reference image to a minimum of 0 and a maximum of the width of the reference image in terms of samples minus 1, and along the vertical direction of the reference image to a minimum of 0 and a maximum of the height of the reference image in terms of samples minus 1.

5. The method according to claim 1, wherein, Performing the wrapping process includes generating one or more samples of the reference sample set based on the size of the reference image and the wrapping offset.

6. The method according to claim 1, further comprising: Decode the residual block of the current block of the current image; as well as The current block of the current image is decoded using the prediction block and the residual block.

7. The method according to claim 1, further comprising: Use the current block and the predicted block of the current image to generate a residual block of the current block of the current image; as well as The residual block is encoded.

8. An apparatus for decoding video data, comprising: The memory is configured to store video data; as well as One or more processors, implemented using circuitry, and configured to perform the following operations: Use motion information to determine a reference block of a reference image for predicting the current block of the current image; Generate a reference sample set for the current block of the current image, wherein, in order to generate the reference sample set, the one or more processors are configured to: When the size of the reference image is different from the size of the current image, wrapping processing is performed based on the size of the reference image. When the size of the reference image is different from the size of the current image, reference sampling cropping is performed on the reference block of the reference image based on the size of the reference image; and A predicted block for the current block of the current image is generated based on the reference sampling set.

9. The device according to claim 8, wherein, The size of the reference image includes the width of the reference image in terms of samples.

10. The device according to claim 8, wherein, The size of the reference image includes its height in sample measurements.

11. The device according to claim 8, wherein, In order to perform reference sampling clipping on the reference block, the one or more processors are configured to apply a clipping function that restricts the position of the reference sample along the horizontal direction of the reference image to a minimum of 0 and a maximum of 1 in the width of the reference image in terms of samples, and along the vertical direction of the reference image to a minimum of 0 and a maximum of 1 in the height of the reference image in terms of samples.

12. The device according to claim 8, wherein, In order to perform the wrap processing, the one or more processors are configured to generate one or more samples of the reference sample set based on the size of the reference image and the wrap offset.

13. The device according to claim 8, wherein, The one or more processors are configured to perform the following operations: Decode the residual block of the current block of the current image; and The current block of the current image is decoded using the prediction block and the residual block.

14. The device according to claim 8, wherein, The one or more processors are configured to perform the following operations: Use the current block and the predicted block of the current image to generate a residual block of the current block of the current image; and The residual block is encoded.

15. The device of claim 8, further comprising a display configured to display the video data.

16. The device according to claim 8, wherein, The device includes one or more of the following: camera, computer, mobile device, broadcast receiver, set-top box, integrated circuit, microprocessor, or wireless communication device.

17. The device of claim 8, further comprising a camera configured to capture the video data.

18. A computer-readable storage medium having instructions stored thereon, said instructions causing a processor to perform the following operations when executed: Use motion information to determine a reference block of a reference image for predicting the current block of the current image; Generate a reference sample set for the current block of the current image, wherein, In order to generate the reference sample set, the instructions cause the processor to perform the following operations: When the size of the reference image is different from the size of the current image, wrapping processing is performed based on the size of the reference image. When the size of the reference image is different from the size of the current image, reference sampling cropping is performed on the reference block of the reference image based on the size of the reference image; and A predicted block for the current block of the current image is generated based on the reference sampling set.

19. The computer-readable storage medium according to claim 18, wherein, The size of the reference image includes the width of the reference image in terms of samples.

20. The computer-readable storage medium of claim 18, wherein, The size of the reference image includes its height in sample measurements.

21. The computer-readable storage medium according to claim 18, wherein, In order to perform reference sampling clipping on the reference block, the instruction causes the processor to apply a clipping function that restricts the position of the reference sample along the horizontal direction of the reference image to a minimum of 0 and a maximum of the width of the reference image in terms of samples minus 1, and along the vertical direction of the reference image to a minimum of 0 and a maximum of the height of the reference image in terms of samples minus 1.

22. The computer-readable storage medium according to claim 18, wherein, In order to perform the wrap processing, the instructions cause the processor to generate one or more samples of the reference sample set based on the size of the reference image and the wrap offset.

23. The computer-readable storage medium according to claim 18, wherein, The instruction also causes the processor to perform the following operations: Decode the residual block of the current block of the current image; and The current block of the current image is decoded using the prediction block and the residual block.

24. The computer-readable storage medium according to claim 18, wherein, The instruction also causes the processor to perform the following operations: Use the current block and the predicted block of the current image to generate a residual block of the current block of the current image; and The residual block is encoded.

25. An apparatus for decoding video data, comprising: A unit used to determine a reference block of a reference image for predicting the current block of the current image using motion information; A unit for generating a reference sampling set for the current block of the current image, wherein the unit for generating the reference sampling set includes: A unit for performing wrap-around processing based on the size of the reference image when the size of the reference image is different from the size of the current image, and A unit for performing reference sampling cropping on the reference block of the reference image based on the size of the reference image when the size of the reference image is different from the size of the current image; and A unit for generating a prediction block for the current block of the current image based on the reference sampling set.

26. The device according to claim 25, wherein, The size of the reference image includes the width of the reference image in sample measure.

27. The device according to claim 25, wherein, The size of the reference image includes the height of the reference image in sample measurements.

28. The device according to claim 25, wherein, The unit for performing reference sampling clipping on the reference block includes a unit for applying a clipping function that restricts the position of the reference sample along the horizontal direction of the reference image to a minimum of 0 and a maximum of 1 (in terms of the width of the reference image minus 1), and along the vertical direction of the reference image to a minimum of 0 and a maximum of 1 (in terms of the height of the reference image minus 1).

29. The device according to claim 25, wherein, The unit for performing the wrap processing includes: a unit for generating samples of one or more of the reference sample sets based on the size and wrap offset of the reference image.

30. The apparatus of claim 25, further comprising: A unit for decoding the residual block of the current block of the current image; as well as A unit for decoding the current block of the current image using the prediction block and the residual block.

31. The device according to claim 25, wherein, Also includes: A unit for generating a residual block of the current block of the current image using the current block and the prediction block of the current image; as well as A unit used to encode the residual block.

Images