Method for padding processing with sub-region partition in video stream

Abstract

To provide a method, system, and computer program for video encoding.SOLUTION: Coded syntax elements corresponding to wrap-around padding process are decoded. At least one coded current picture is reconstructed using wrap-around padding process. The syntax elements indicate an offset value for wrap-around processing or left and right padding width information. A flag indicates whether syntax elements corresponding to wrap-around padding process are present in a parameter set. A pixel position for motion compensated prediction in a reference picture is determined by interpreting the syntax elements corresponding to wrap-around padding process with a clipping process.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] [Cross-references to related applications] This application claims priority to U.S. Provisional Patent Application No. 62 / 903,635, filed September 20, 2019, and U.S. Patent Application No. 17 / 019,692, filed September 14, 2020. The entire contents of these applications are incorporated herein by reference.

[0002] [Technical field] This disclosure relates in general to the field of data processing, and more specifically to video coding and decoding. [Background technology]

[0003] Video encoding and decoding using motion-compensated interpicture prediction have been known for decades. Uncompressed digital video can consist of a series of pictures, each picture having spatial dimensions of, for example, 1920×1080 luminance samples and associated chrominance samples. The series of pictures can have a fixed or variable picture rate (also informally known as frame rate), for example, 60 pictures per second or 60Hz. Uncompressed video has considerable bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at a frame rate of 60Hz) requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 GB of storage space.

[0004] One purpose of video encoding and decoding is to reduce the redundancy of the input video signal through compression. Compression can, in some cases, help reduce the bandwidth or storage space requirements by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to a technique in which an exact copy of the original signal can be restored from the compressed original signal. When using lossy compression, the restored signal may not be identical to the original signal, but the distortion between the original and restored signals is small enough to make the restored signal useful for the intended application. For video, lossy compression is widely used. The amount of acceptable distortion depends on the application. For example, users of a particular consumer's streaming application may tolerate higher distortion than users of a television distribution application. The achievable compression ratio can reflect that higher acceptable distortion / tolerance distortion can result in a higher compression ratio.

[0005] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transformation, quantization, and entropy coding, some of which are described below.

[0006] The concept of dividing an encoded video bitstream into packets for transport over a packet network has been in use for decades. In its early stages, video encoding standards and technologies were primarily optimized for bot-oriented transport, defining the bitstream accordingly. Packeting occurred, for example, at system layer interfaces specified in the Real-time Transport Protocol (RTP) payload format. With the emergence of internet connectivity suitable for the mass use of video over the internet, video encoding standards have reflected their prominent use cases through the conceptual distinction between the video coding layer (VCL) and the network abstraction layer (NAL). The NAL unit was introduced in H.264 in 2003 and has since been maintained in specific video encoding standards and technologies with only minor modifications.

[0007] A NAL unit can often be considered the smallest entity on which a decoder can operate without necessarily decoding all preceding NAL units in the encoded video sequence. To that extent, a NAL unit enables certain error prevention techniques and certain bitstream manipulation techniques, including bitstream pruning by a Media Aware Network Element (MANE), such as a Selective Forwarding Unit (SFU) or Multipoint Control Unit (MCU).

[0008] Figure 1 shows relevant parts of the syntax diagrams for NAL unit headers according to H.264(101) and H.265(102), without their respective extensions. In both cases, forbidden_zero_bit is a zero bit used to prevent start code emulation in a particular system layer environment. The nal_unit_type syntax element indicates the type of data carried by the NAL unit, which may be, for example, a specific slice type, parameter set type, or Supplementary Enhancement Information (SEI) message. The H.265 NAL unit header further includes nuh_layer_id and nuh_temporal_id_plus1, which indicate the spatial / SNR and temporal layers of the encoded picture to which the NAL unit belongs.

[0009] It can be seen that NAL unit headers contain only easily parsable fixed-length codewords, which do not have parsing dependencies on other data in the bitstream, such as other NAL unit headers or parameter sets. Since NAL unit headers are the first octets in a NAL unit, MANE can easily extract them, parse them, and act upon them. In contrast, other high-level syntax elements (e.g., slice headers or tile headers) are not easily accessible to MANE because they may require maintaining parameter set context and / or processing variable-length or arithmetic-encoded codepoints.

[0010] It can be further seen that the NAL unit header shown in Figure 1 does not contain any information that would allow a NAL unit to be associated with an encoded picture (for example, one containing multiple tiles or slices, of which at least some are packetized into individual NAL units).

[0011] Certain transport technologies, such as RTP (RFC3550), the MPEG-system standard, and the ISO file format, may include specific information that is readily accessible by MANE, often in the form of timing information such as presentation time (for MPEG and ISO file formats) or capture time (for RTP), which can help associate each transport unit with an encoded picture. However, the meaning of this information may differ depending on the transport / storage technology and may not have a direct relationship with the picture structure used in video encoding. Therefore, this information may be heuristic at best and may not be particularly suitable for identifying whether NAL units in a NAL unit stream belong to the same encoded picture. [Overview of the Initiative]

[0012] One embodiment relates to a method, system, and computer-readable medium for video encoding. According to one embodiment, a method for video encoding is provided. The method may include the step of decoding an encoding syntax element corresponding to a wrap-around padding process. At least one encoded current picture is restored using the wrap-around padding process. The syntax element indicates an offset value or left and right padding width information for the wrap-around process.

[0013] In another embodiment, a computer system for video encoding is provided. The computer system includes one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored in at least one of the storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, thereby enabling the computer system to perform a method. The method may include the step of decoding an encoding syntax element corresponding to a wrap-around padding process. At least one encoded current picture is restored using the wrap-around padding process. The syntax element indicates an offset value or left and right padding width information for the wrap-around process.

[0014] In yet another embodiment, a computer-readable medium for video encoding is provided. The computer-readable medium includes one or more computer-readable storage devices and program instructions stored in at least one of one or more tangible storage devices, the program instructions being executable by a processor. The program instructions are executable by the processor to perform a method which may include the step of decoding an encoding syntax element corresponding to a wrap-around padding process. At least one encoded current picture is restored using the wrap-around padding process. The syntax element indicates an offset value or left and right padding width information for the wrap-around process. [Brief explanation of the drawing]

[0015] The above and other purposes, features and advantages will become apparent from the following detailed description of exemplary embodiments, which will be read in conjunction with the attached drawings. The drawings are for clarification to facilitate understanding for those skilled in the art in connection with the detailed description; therefore, various features in the drawings are not to scale. [Figure 1]This is a schematic diagram of a NAL unit header according to H.264 and H.265. [Figure 2] This is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Figure 3] This is a schematic diagram of a simplified block diagram of a communication system according to one embodiment. [Figure 4] This is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 5] This is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 6] This is a schematic diagram of a syntax element for offset transmission according to one embodiment. [Figure 7] This is a schematic diagram of a syntax element for transmitting the padding width of an encoder according to one embodiment. [Figure 8] This is a schematic diagram of a simplified block diagram of the padding process for each sub-region (tile or sub-picture) according to one embodiment. [Figure 9] This is a schematic diagram of a syntax element for transmitting the padding offset of each subpicture according to one embodiment. [Figure 10] This is a schematic diagram of a syntax element for transmitting a unified padding offset for each subpicture according to one embodiment. [Figure 11] This is a schematic diagram of a computer system according to one embodiment. [Modes for carrying out the invention]

[0016] While detailed embodiments of the structures and methods of the claims are disclosed herein, it should be understood that the disclosed embodiments are merely examples of structures and methods of the claims that may be embodied in various forms. However, these structures and methods may be embodied in many different forms and should not be construed as being limited to the exemplary embodiments described herein. Rather, these exemplary embodiments are provided to ensure that this disclosure is complete and comprehensive and adequately conveys its scope to those skilled in the art. In this description, well-known features and technical details may be omitted to avoid unnecessarily obscuring the presented embodiments.

[0017] The embodiments generally relate to the field of data processing, and more specifically to video encoding and decoding. The exemplary embodiments described below provide, in particular, systems, methods, and computer programs for restoring an encoded current picture using wrap-around padding processing based on decoding encoded syntax elements. Thus, some embodiments have the ability to improve the field of computing by enabling the encoding of syntax elements used in the restoration of compressed images.

[0018] As mentioned above, one purpose of video encoding and decoding is to reduce the redundancy of the input video signal through compression. Compression can, in some cases, help reduce the bandwidth or storage space requirements by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to a technique in which an exact copy of the original signal can be restored from the compressed original signal. When using lossy compression, the restored signal may not be identical to the original signal, but the distortion between the original and restored signals is small enough to make the restored signal useful for the intended application. For video, lossy compression is widely used. The amount of acceptable distortion depends on the application. For example, users of a streaming application for a particular consumer may tolerate higher distortion than users of a television distribution application. The achievable compression ratio can reflect that higher acceptable distortion / tolerance distortion can result in a higher compression ratio. However, if a picture is divided into one or more sub-regions (tiles, slices, or sub-pictures), the boundary treatment of each sub-region can affect encoding efficiency and subjective visual quality. Adaptive control of boundary processing at each sub-region boundary is a crucial element in 360 media processing. Therefore, it can be advantageous to reconstruct the encoded current picture using wrap-around padding processing based on decoding the encoded syntax elements.

[0019] Multiple embodiments will be described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer-readable media according to various embodiments. It will be understood that each block in a flowchart and / or block diagram, and combinations of blocks in a flowchart and / or block diagram, can be implemented by computer-readable program instructions.

[0020] The exemplary embodiments described below provide a system, method, and computer program for restoring an encoded current picture using a wrap-around padding process based on decoding encoded syntax elements.

[0021] Figure 2 shows a simplified block diagram of a communication system (200) according to one embodiment of the present disclosure. The system (200) may include at least two terminals (210-220) interconnected via a network (250). For one-way data transmission, a first terminal (210) may encode video data at its local location for transmission to the other terminal (220) via the network (250). A second terminal (220) may receive the encoded video data from the other terminal via the network (250), decode the encoded video data, and display the restored video data. One-way data transmission may be common in media delivery applications and the like.

[0022] Figure 2 shows a second pair of terminals (230, 240) provided to support the bidirectional transmission of encoded video that may occur, for example, during a video conference. For bidirectional transmission of data, each terminal (230, 240) may encode video data captured at its local location for transmission to the other terminal over the network (250). Each terminal (230, 240) may also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the restored video data on a local display device.

[0023] In Figure 2, terminals (210-240) may be shown as servers, personal computers, and smartphones, but the principles of this disclosure are not limited to these. Embodiments of this disclosure are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (250) represents any number of networks that transmit encoded video data between terminals (210-240), including, for example, wired and / or wireless communication networks. Communication networks (250) may exchange data over circuit-switched channels and / or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this description, the architecture and topology of network (250) are not important to the operation of this disclosure unless described below herein.

[0024] Figure 3 shows an example of the application of the disclosed object, illustrating the arrangement of a video encoder and decoder in a streaming environment. The disclosed object is similarly applicable to other video-enabled applications, including, for example, video conferencing, digital TV, and the storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.).

[0025] The streaming system may include a capture subsystem (313), which may include, for example, a video source (301) (e.g., a digital camera) that generates an uncompressed video sample stream (302). The sample stream (302), which is drawn as a thick line to emphasize that it has a higher data volume compared to the encoded video bitstream, may be processed by an encoder (303) coupled to the camera (301). The encoder (303) may include hardware, software, or a combination thereof to enable or implement an aspect of the disclosed object, as will be described in more detail below. The encoded video bitstream (304), which is drawn as a thin line to emphasize that it has a lower data volume compared to the sample stream, may be stored in a streaming server (305) for future use. One or more streaming clients (306, 308) may access the streaming server (305) to obtain a copy (307, 309) of the encoded video bitstream (304). The client (306) may include a video decoder (310) that decodes an input copy (307) of an encoded video bitstream and generates an output video sample stream (311) that can be rendered on a display (312) or other rendering device (not shown). In some streaming systems, the video bitstreams (304, 307, 309) may be encoded according to a specific video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265. Video encoding standards under development are informally known as VVC (Versatile Video Coding). The disclosures may be used in the background of VVC.

[0026] Figure 4 shows a functional block diagram of a video decoder (310) according to one embodiment of the present disclosure.

[0027] The receiver (410) may receive one or more encoded video sequences to be decoded by the decoder (310), and in the same or other embodiments, may receive one encoded video sequence at a time, with the decoding of each encoded video sequence being independent of other encoded video sequences. The encoded video sequences may be received from a channel (412), which may be a hardware / software link to a storage device that stores encoded video data. The receiver (410) may receive the encoded video data together with other data (e.g., encoded audio data and / or auxiliary data streams), which may be transferred to their respective usage entities (not shown). The receiver (410) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (415) may be coupled between the receiver (410) and the entropy decoder / parser (420) (hereinafter referred to as the "parser"). If the receiver (410) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isochronous network, the buffer (415) may not be necessary or can be made small. For use in best-effort packet networks such as the Internet, the buffer (415) may be necessary and can be made relatively large, or advantageously, adaptively sized.

[0028] The video decoder (310) may include a parser (420) for recovering symbols (421) from an entropy-encoded video sequence. These symbol categories include information used to manage the operation of the decoder (310) and potentially include information for controlling a rendering device such as a display (312). The rendering device, as shown in Figure 3, is not an integral part of the decoder but may be coupled to it. The rendering device control information may be in the form of Supplementary Enhancement Information (SEI) (SEI messages) or Video Usability Information (VUI) parameter set fragments (not shown). The parser (420) may analyze / entropy-decode the received encoded video sequence. The encoding of the encoded video sequence may follow video coding techniques or standards, and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser (420) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the encoded video sequence, based on at least one parameter corresponding to a group. Subgroups may include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transformation units (TU), prediction units (PU), etc. The entropy decoder / parser may also extract information such as transformation coefficients, quantization parameter values, and motion vectors from the encoded video sequence.

[0029] The parser (420) may perform an entropy decoding / analysis operation on the video sequence received from the buffer (415) in order to generate a symbol (421).

[0030] The reconstruction of symbol (421) may involve multiple different units, depending on the type of encoded video picture or portion thereof (e.g., interpicture and intrapicture, interblock and intrablock) and other factors. Which units are involved and how may be controlled by subgroup control information analyzed from the encoded video sequence by the parser (420). The flow of such subgroup control information between the parser (420) and the following multiple units is not illustrated for clarity.

[0031] In addition to the functional blocks described above, the decoder (310) may be conceptually subdivided into several functional units, as described below. In practical implementations operating under commercial constraints, many of these units may interact closely with each other and be at least partially integrated. However, for the purpose of illustrating the subject matter of the disclosure, it is appropriate to conceptually subdivide it into the following functional units.

[0032] The first unit is the scaler / inverse unit (451). The scaler / inverse unit (451) receives quantized transformation coefficients as symbols (421) from the parser (420), along with control information (including which transformation to use, block size, quantization coefficients, quantization scaling matrix, etc.). The scaler / inverse unit may output a block containing sample values ​​that can be input to the aggregator (455).

[0033] In some cases, the output samples of the scaler / inverse transform (451) may relate to intra-encoded blocks, i.e., blocks that do not use prediction information from previously restored pictures but can use prediction information from previously restored portions of the current picture. Such prediction information may be provided by an intra-picture prediction unit (452). In some cases, the intra-picture prediction unit (452) generates blocks of the same size and shape as the block being restored using already restored surrounding information extracted from the (partially restored) current picture (458). In some cases, the aggregator (455) adds the prediction information generated by the intra-prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451) for each sample.

[0034] In other cases, the output samples of the scaler / inverse unit (451) may be associated with intercoded and potentially motion-compensated blocks. In such cases, the motion-compensated prediction unit (453) may access the reference picture memory (457) to retrieve samples to be used for prediction. After motion-compensating the retrieved samples according to the symbols (421) associated with the blocks, these samples may be added by the aggregator (455) to the output of the scaler / inverse unit (in this case, called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory from which the motion-compensated unit retrieves prediction samples, available to the motion-compensated unit, may be controlled by motion vectors, for example, in the form of symbols (421) which may have X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory when the exact motion vectors of the subsamples are used, motion vector prediction mechanisms, etc.

[0035] The output samples of the aggregator (455) may undergo various loop filtering techniques within the loop filtering unit (456). The video compression technique may include an in-loop filtering technique, which is controlled by parameters included in the encoded video bitstream and made available to the loop filtering unit (456) as symbols (421) from the parser (420), but may respond to metadata obtained during decoding of earlier portions (in decoding order) of the encoded picture or encoded video sequence, as well as to previously recovered and loop-filtered sample values.

[0036] The output of the loop filter unit (456) may also be a sample stream, which may be output to the rendering device (312) and stored in the reference picture memory (456) for use in future interpicture prediction.

[0037] A particular encoded picture, once fully restored, may be used as a reference picture for future predictions. Once an encoded picture is fully restored and identified as a reference picture (e.g., by a parser (420)), the current reference picture (456) may become part of the reference picture buffer (457), and new current picture memory may be reallocated before the restoration of subsequent encoded pictures begins.

[0038] The video decoder (420) may perform the decoding operation according to a predetermined video compression technique that may be documented in a standard such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the encoded video sequence conforms to the syntax of the video compression technique or standard specified in the video compression technique documentation or standard, and in particular in the syntax of the video compression technique or standard specified in these profile documents. Also, for compliance purposes, the complexity of the encoded video sequence must be within the range defined by the level of the video compression technique or standard. In some cases, the level may limit the maximum picture size, maximum frame rate, maximum recovery sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the level may be further limited through the Hypothetical Reference Decoder (HRD) specification and metadata about HRD buffer management transmitted in the encoded video sequence.

[0039] In one embodiment, the receiver (410) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (420) to properly decode the data and / or to more accurately restore the original video data. The additional data may be in the form of, for example, time, space or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0040] Figure 5 may also be a functional block diagram of a video encoder (303) according to one embodiment of the present disclosure.

[0041] The encoder (303) may receive video samples from a video source (301) (which is not part of the encoder), and the video source (301) may capture a video image to be encoded by the encoder (303).

[0042] The video source (301) may provide a source video sequence to be encoded by the encoder (303) in the form of a digital video sample stream, the digital video sample stream may be any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media delivery system, the video source (301) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (303) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that convey motion when viewed sequentially. The picture itself may be composed of a spatial array of pixels, and each pixel may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description will focus on samples.

[0043] According to one embodiment, the encoder (303) may encode and compress the pictures of a source video sequence into an encoded video sequence (543) in real time or under any other time constraints required by the application. Achieving an appropriate encoding speed is one function of the controller (550). The controller controls and is functionally coupled to other functional units, as described below. The couplings are not shown for clarity. Parameters set by the controller may include rate control-related parameters (such as picture skipping, quantization, lambda values ​​for rate distortion optimization techniques), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art will readily recognize other functions of the controller (550) that may be associated with a video encoder (303) optimized for a particular system design.

[0044] Some video encoders operate in a manner readily recognizable to those skilled in the art as a “coding loop.” In very simplified terms, the coding loop may consist of a coding portion of an encoder (530) (hereinafter referred to as the “source coder”) (responsible for generating symbols based on the input picture to be encoded and a reference picture) and a (local) decoder (533) embedded in the encoder (303). The decoder (533) reconstructs the symbols to generate sample data in the same way that a (remote) decoder would (such that any compression between the symbols and the encoded video bitstream is reversible in the video compression techniques considered in the disclosure). The reconstructed sample stream is fed into a reference picture memory (534). Since decoding the symbol stream yields bitwise-accurate results independent of the decoder’s location (local or remote), the contents of the reference picture buffer are also bitwise-accurate between the local and remote encoders. In other words, the predictive portion of the encoder “sees” the exact same sample values ​​as the reference picture samples that the decoder “sees” when using predictions during decoding. This basic principle of the synchronization of reference pictures (including, for example, drift resulting from the inability to maintain synchronization due to channel errors) is well known to those skilled in the art.

[0045] The operation of the “local” decoder (533) may be the same as that of the “remote” decoder (310), as has already been explained in detail above in relation to Figure 4. However, referring briefly to Figure 4, since symbols are available and the encoding / decoding of symbols to the encoded video sequence by the entropy coder (545) and parser (420) may be reversible, the entropy decoding portion of the decoder (310), including the channel (412), receiver (410), buffer (415), and parser (420), does not have to be fully implemented in the local decoder (533).

[0046] One consideration at this point is that any decoder technique, excluding analysis / entropy decoding, that exists within the decoder must necessarily exist in substantially the same functional form within the corresponding encoder. Therefore, the disclosure focuses on decoder operation. The description of encoder techniques can be omitted, as it is the reverse of the comprehensively described decoder techniques. More detailed explanations are necessary only in specific areas and are provided below.

[0047] As part of this operation, the source coder (530) may perform motion-compensated predictive coding, which predictively codes the input frame by referencing one or more previously coded frames from a video sequence designated as “reference frames”. In this way, the coding engine (532) codes the difference between the pixel blocks of the input frame and the pixel blocks of a reference frame that may be selected as a predictive reference for the input frame.

[0048] The local video decoder (533) may decode the encoded video data of a frame that may be designated as a reference frame based on the symbols generated by the source coder (530). The operation of the encoding engine (532) may, advantageously, be lossy. If the encoded video data can be decoded by a video decoder (not shown in Figure 5), the restored video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (533) may replicate the decoding process that may be performed by the video decoder on the reference frame and store the restored reference frame in a reference picture cache (534). Thus, the encoder (303) may locally store a copy of the restored reference frame having common content as a restored reference frame (without transmission errors) to be acquired by the far-end video decoder.

[0049] The predictor (535) may perform a predictive search for the coding engine (532). That is, for a new frame to be coded, the predictor (535) may search the reference picture memory (534) for sample data (as candidate reference pixel blocks) or specific metadata (reference picture motion vector, block shape, etc.). These may function as appropriate predictive references for the new picture. The predictor (535) may operate sample block-by-pixel block to find appropriate predictive references. In some cases, the input picture determined by the search results obtained by the predictor (535) may have predictive references drawn from multiple reference pictures stored in the reference picture memory (534).

[0050] The controller (550) may manage the encoding operation of the video coder (530), including, for example, setting parameters and subgroup parameters used to encode video data.

[0051] The outputs of all the above functional units may undergo entropy coding in the entropy coder (545). The entropy coder converts the symbols generated by the various functional units into coded video sequences by lossless compression of the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable-length coding, arithmetic coding, etc.

[0052] The transmitter (540) may buffer the encoded video sequence generated by the entropy coder (545) and prepare it for transmission over a communication channel (560), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (540) may merge the encoded video data from the video coder (530) with other data to be transmitted (e.g., encoded audio data and / or auxiliary data streams (not shown)).

[0053] The controller (550) may manage the operation of the encoder (303). During encoding, the controller (550) may assign a specific encoded picture type to each encoded picture. This encoded picture type may affect the encoding technique that can be applied to each picture. For example, a picture may often be assigned as one of the following frame types:

[0054] An intra-picture (I-picture) may be one that can be encoded and decoded without using other pictures in the sequence as a source for prediction. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will recognize these variations of I-pictures and their respective uses and characteristics.

[0055] The prediction picture (P-picture) may be encoded and decoded using intra-prediction or inter-prediction, with up to one motion vector and reference index used to predict the sample values ​​of each block.

[0056] A bidirectional predictive picture (B-picture) may be encoded and decoded using intra-prediction or inter-prediction, using up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures may use two or more reference pictures and associated metadata to reconstruct a single block.

[0057] Generally, a source picture may be subdivided spatially into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively), and each block may be encoded. Blocks may be predictively encoded by referencing other (already encoded) blocks, as determined by the encoding assignment applied to each picture in the block. For example, blocks of picture I may be non-predictively encoded, or they may be predictively encoded by referencing already encoded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be non-predictively encoded via spatial prediction or temporal prediction by referencing a previously encoded reference picture. Blocks of picture B may be predictively encoded via spatial prediction or temporal prediction by referencing a reference picture encoded one or two steps prior.

[0058] The video coder (303) may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec. H.265. In such operations, the video coder (303) may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to the syntax specified by the video coding technique or standard being used.

[0059] In one embodiment, the transmitter (540) may transmit additional data along with the encoded video. The video coder (530) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data, such as time / space / SNR enhancement layers, redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0060] In one embodiment, 360 video is captured by a set of cameras or camera devices having multiple lenses. The cameras typically cover all directions around a central point of the camera set. Pictures from the same point in time are stitched together, possibly rotated, projected, and mapped onto the picture. The packed pictures are encoded into an encoded video bitstream according to a specific media container file format and streamed. The file contains metadata such as projection information and packing information.

[0061] In one embodiment, 360 video may be projected onto 2D video using equirectangular projection (ERP). ERP projection can produce seam artifacts. Padded ERP (PERP) format can effectively reduce seam artifacts in the restored viewport that encompasses the left and right boundaries of the ERP picture. However, padding and blending may not be sufficient to completely solve the seam problem.

[0062] In one embodiment, horizontal geometric padding may be applied to ERP or PERP to reduce splice artifacts. The padding process for PERP may be the same as for ERP, except that the offset may be based on the unpaded ERP width instead of the picture width to account for the size of the padded area. If a reference block is outside the left (right) reference picture boundary, it may be replaced with a "wrapped-around" reference block shifted to the right (left) by the ERP width. Conventional repeating padding may be used vertically. Blending of left and right padding areas is removed from the loop as a post-processing operation.

[0063] In one embodiment, a syntax that enables horizontal geometric padding of reference pictures for ERP and PERP formats is shown in Figure 6.

[0064] A sps_ref_wrapaund_enabled_flag (602) equal to 1 specifies that horizontal wrap-around motion compensation is used for interpretation. A sps_ref_wrapaund_enabled_flag equal to 0 specifies that this motion compensation method is not applied.

[0065] ref_wraparound_offset(603) specifies the offset in the luminance samples used to calculate the horizontal wraparound position. ref_wraparound_offset must be greater than pic_width_in_luma_samples-1 and less than pic_width_in_luma_samples, and an integer multiple of MinCbSizeY.

[0066] In one embodiment, a syntax element that enables horizontal geometric padding of reference pictures for ERP and PERP formats is shown in Figure 7.

[0067] A sps_ref_wraparound_enabled_flag equal to 1 (702) specifies that horizontal wraparound motion compensation is used for interpretation. A sps_ref_wraparound_enabled_flag equal to 0 specifies that this motion compensation method is not applied.

[0068] left_wraparound_padding_width(703) specifies the width of the left-side padding area in the luminance samples. ref_wraparound_offset must be greater than or equal to 0, not greater than pic_width_in_luma_samples / 2, and an integer multiple of MinCbSizeY.

[0069] right_wraparound_padding_width(704) specifies the width of the right-side padding area in the luminance samples. ref_wraparound_offset must be greater than or equal to 0, not greater than pic_width_in_luma_samples / 2, and an integer multiple of MinCbSizeY.

[0070] Next, referring to Figure 8, an operational flowchart is shown illustrating the steps of method 900 for encoding video data.

[0071] In 901, method 900 includes analyzing the parameter set.

[0072] In 902, method 900 includes decoding picture segmentation information.

[0073] In 903, method 900 includes determining whether padding is applied to the boundary of a subregion.

[0074] In 904, method 900 includes analyzing the padding type when the padding is applied to the boundary of a sub-region.

[0075] In 905, method 900 includes decoding each subregion without padding if padding is not applied to the boundaries of the subregions.

[0076] In 906, method 900 includes determining whether wrap-around padding is applicable.

[0077] In 907, method 900 includes padding the boundaries of subregions with wrap-around padding when wrap-around padding is applied.

[0078] In 908, method 900 includes padding the boundaries of subregions with non-wrap-around padding when wrap-around padding is not applicable.

[0079] In one embodiment, the wrap-around offset value may be obtained by the following derivation process.

[0080] If ref_wraparound_offset exists, wrapAroundOffset=ref_wraparound_offset Otherwise, if left_wraparound_padding_width and right_wraparound_padding_width exist, wrapAroundOffset=pic_width_in_luma_samples-(left_wraparound_padding_width+right_wraparound_padding_width) Otherwise, wrapAroundOffset=pic_width_in_luma_samples In one embodiment, the luminance and color difference sample interpolation process may be modified to enable horizontal geometric padding of the reference picture for ERP and PERP formats.

[0081]

number

[0082] The variables shift1, shift2, and shift3 are derived as follows. The variable shift1 is set equal to Min(4, BitDepth Y - 8), the variable shift2 is set equal to 6, and the variable shift3 is set equal to Max(2, 14 - BitDepth Y ). The variable picW is set equal to pic_width_in_luma_samples, and the variable picH is set equal to pic_height_in_luma_samples. The variable xOffset is set equal to wrapAroundOffset. For each 1 / 16 fractional sample position p equal to xFrac L or yFrac L , the luminance interpolation filter coefficient f L [p] may be specified.

[0083] The predicted luminance sample value predSampleLX L is derived as follows. When both xFrac L and yFrac L are equal to 0, the following applies. When the sps_ref_wraparound_enabled_flag is equal to 0, the value of predSampleLX L is predSampleLX L = refPicLX L [Clip3(0, picW - 1, xInt L )][Clip3(0, picH - 1, yInt L )] << shift3. Otherwise, the value of predSampleLX L is predSampleLX L = refPicLX L [ClipH(xOffset, picW, xInt L )][Clip3(0, picH - 1, yInt L )] << shift3. Otherwise, when xFrac L is not equal to 0 and yFrac L is equal to 0, the following applies. The value of yPos L is yPosL =Clip3(0,picH-1,yInt L This is derived as follows.

[0084] If sps_ref_wraparound_enabled_flag is 0, predSampleLX L The value is derived as follows: predSampleLX L =( f L [xFrac L ][0]*refPicLX L [Clip3(0,picW-1,xInt L -3)][yPos L ]+ f L [xFrac L ][1]*refPicLX L [Clip3(0,picW-1,xInt L -2)][yPos L ]+ f L [xFrac L ][2]*refPicLX L [Clip3(0,picW-1,xInt L -1)][yPos L ]+ f L [xFrac L ][3]*refPicLX L [Clip3(0,picW-1,xInt L )][yPos L ]+ f L [xFrac L ][4]*refPicLX L [Clip3(0,picW-1,xInt L +1)][yPos L ]+ f L [xFrac L ][5]*refPicLX L [Clip3(0,picW-1,xInt L +2)][yPos L ]+ f L[xFrac L [6]*refPicLX L [Clip3(0,picW-1,xInt L +3)][yPos L + f L [xFrac L [7]*refPicLX L [Clip3(0,picW-1,xInt L +4)][yPos L )>>shift1 Otherwise, the value of predSampleLX L is derived as follows. predSampleLX L =(f L [xFrac L [0]*refPicLX L [ClipH(xOffset,picW,xInt L -3)][yPos L + f L [xFrac L [1]*refPicLX L [ClipH(xOffset,picW,xInt L -2)][yPos L + f L [xFrac L [2]*refPicLX L [ClipH(xOffset,picW,xInt L -1)][yPos L + f L [xFrac L [3]*refPicLX L [ClipH(xOffset,picW,xInt L )][yPos L + f L [xFrac L [4]*refPicLX L [ClipH(xOffset,picW,xInt L +1)][yPos L + f L [xFrac L ][5]*refPicLX L [ClipH(xOffset,picW,xInt L +2)][yPos L ]+ f L [xFrac L ][6]*refPicLX L [ClipH(xOffset,picW,xInt L +3)][yPos L ]+ f L [xFrac L ][7]*refPicLX L [ClipH(xOffset,picW,xInt L +4) [yPos L ])>>shift No, xFrac L If yFrac is equal to 0, L If it is not equal to 0, predSampleLX L The value is derived as follows: If sps_ref_wraparound_enabled_flag is equal to 0, xPos L The value of xPos L =Clip3(0,picW-1,xInt L ) is derived as. Otherwise, xPos L The value of xPos L =ClipH(xOffset,picW,xInt L This is derived as follows.

[0085] Predicted luminance sample value predSampleLX L This is derived as follows: predSampleLX L =( f L [yFrac L ][0]*refPicLX L [xPos L ][Clip3(0,picH-1,yInt L -3)]+ f L [yFracL ][1]*refPicLX L [xPos L ][Clip3(0,picH-1,yInt L -2)]+ f L [yFrac L ][2]*refPicLX L [xPos L ][Clip3(0,picH-1,yInt L -1)]+ f L [yFrac L ][3]*refPicLX L [xPos L ][Clip3(0,picH-1,yInt L )]+ f L [yFrac L ][4]*refPicLX L [xPos L ][Clip3(0,picH-1,yInt L +1)]+ f L [yFrac L ][5]*refPicLX L [xPos L ][Clip3(0,picH-1,yInt L +2)]+ f L [yFrac L ][6]*refPicLX L [xPos L ][Clip3(0,picH-1,yInt L +3)]+ f L [yFrac L ][7]*refPicLX L [xPos L ][Clip3(0,picH-1,yInt L +4)])>> shift1 No, xFrac L If yFrac is not equal to 0, L If it is not equal to 0, predSampleLX LThe value is derived as follows. If sps_ref_wraparound_enabled_flag is equal to 0, the sample array temp[n](n=0..7) is derived as follows. yPos L =Clip3(0,picH-1,yInt L +n-3) temp[n]=(f L [xFrac L ][0]*refPicLX L [Clip3(0,picW-1,xInt L -3)][yPos L ]+ f L [xFrac L ][1]*refPicLX L [Clip3(0,picW-1,xInt L -2)][yPos L ]+ f L [xFrac L ][2]*refPicLX L [Clip3(0,picW-1,xInt L -1)][yPos L ]+ f L [xFrac L ][3]*refPicLX L [Clip3(0,picW-1,xInt L )][yPos L ]+ f L [xFrac L ][4]*refPicLX L [Clip3(0,picW-1,xInt L +1)][yPos L ]+ f L [xFrac L ][5]*refPicLX L [Clip3(0,picW-1,xInt L +2)][yPos L ]+ f L [xFrac L ][6]*refPicLXL [Clip3(0,picW-1,xInt L +3)][yPos L ]+ f L [xFrac L ][7]*refPicLX L [Clip3(0,picW-1,xInt L +4) [yPos L ])>>shift1 Otherwise, the sample array temp[n](n=0..7) is derived as follows: yPos L =Clip3(0,picH-1,yInt L +n-3) temp[n]=(f L [xFrac L ][0]*refPicLX L [ClipH(xOffset,picW,xIntL-3)][yPos L ]+ f L [xFrac L ][1]*refPicLX L [ClipH(xOffset,picW,xInt L -2)][yPos L ]+ f L [xFrac L ][2]*refPicLX L [ClipH(xOffset,picW,xInt L -1)][yPos L ]+ f L [xFrac L ][3]*refPicLX L [ClipH(xOffset,picW,xInt L )][yPos L ]+ f L [xFrac L ][4]*refPicLX L [ClipH(xOffset,picW,xInt L +1)][yPos L ]+ f L [xFrac L ][5]*refPicLX L [ClipH(xOffset,picW,xInt L +2)][yPos L ]+ f L [xFrac L ][6]*refPicLX L [ClipH(xOffset,picW,xInt L +3)][yPos L ]+ f L [xFrac L ][7]*refPicLX L [ClipH(xOffset,picW,xInt L +4) [yPos L ])>>shift1 Predicted luminance sample value predSampleLX L This is derived as follows: predSampleLX L =( f L [yFrac L ][0]*temp[0]+ f L [yFrac L ][1]*temp[1]+ f L [yFrac L ][2]*temp[2]+ f L [yFrac L ][3]*temp[3]+ f L [yFrac L ][4]*temp[4]+ f L [yFrac L ][5]*temp[5]+ f L [yFrac L ][6]*temp[6]+ f L [yFrac L ][7]*temp[7])>>shift2 Color difference sample interpolation processing. The input to this process is the color difference position (xInt) for all samples. C, yInt C ) and the color difference position in 1 / 32 fractional sample units (xFrac C yFrac C ) and the color difference reference sample array refPicLX C It may also include the following. The output of this process is the predicted color difference sample value predSampleLX C It may include.

[0086] The variables shift1, shift2, and shift3 are derived as follows: Variable shift1 is Min(4, BitDepth C Variable shift2 is set to equal to 6, and variable shift3 is set to Max(2, 14-BitDepth C It is set to equal to ). Variable picW C is pic_width_in_luma_samples / SubWidth C Set to equal to, variable picH C is pic_height_in_luma_samples / SubHeight C It is set to be equal to xFrac C or yFrac C Brightness interpolation filter coefficient f for each 1 / 32 fractional sample position p equal to C [p] may be specified.

[0087] Predicted luminance sample value predSampleLX C xFrac is derived as follows: C and yFrac C If both are equal to 0, the following applies: If sps_ref_wraparound_enabled_flag is equal to 0, predSampleLX C The value is predSampleLX C =refPicLX C [Clip3(0,picW C -1, xInt C )][Clip3(0,picHC -1, y - intercept C )] is derived as <<shift3.

[0088] Otherwise, predSampleLX C 's value is predSampleLX C = refPicLX C [ClipH(xOffset C , picW C , x - intercept C )][Clip3(0, picH C - 1, y - intercept C )] is derived as <<shift3.

[0089] Otherwise, if xFrac C is not equal to 0 and yFrac C is equal to 0, the following applies. yPos C 's value is yPos C = Clip3(0, picH C - 1, y - intercept C ) is derived. When sps_ref_wraparound_enabled_flag is 0, the value of predSampleLX C is derived as follows. predSampleLX C =(f C [xFrac C )[0]*refPicLX C [Clip3(0, picW C - 1, x - intercept C - 1)][y - intercept C + f C [xFrac C )[1]*refPicLX C [Clip3(0, picW C - 1, x - intercept C )][y - intercept C + f C [xFrac C )[2]*refPicLX C [Clip3(0, picW C - 1, x - interceptC +1)][yInt C ]+ f C [xFrac C ][3]*refPicLX C [Clip3(0,picW C -1, xInt C +2)][yInt C ])>>shift1 Otherwise, predSampleLX C The value is derived as follows: predSampleLX C =( f C [xFrac C ][0]*refPicLX C [ClipH(xOffset C ,picW C xInt C -1)][yPos C ]+ f C [xFrac C ][1]*refPicLX C [ClipH(xOffset C ,picW C xInt C )][yPos C ]+ f C [xFrac C ][2]*refPicLX C [ClipH(xOffset C ,picW C xInt C +1)][yPos C ]+ f C [xFrac C ][3]*refPicLX C [ClipH(xOffset C ,picW C xInt C +2)][yPos C ])>>shift1 No, xFrac C If yFrac is equal to 0, C If it is not equal to 0, predSampleLXC The value of C is derived as follows. When sps_ref_wraparound_enabled_flag is equal to 0, xPos C = Clip3(0, picW C - 1, xInt C ) is derived. Otherwise, the value of xPos C is derived as xPos C = ClipH(xOffset C , picW C , xInt C ).

[0090] The predicted color difference sample value predSampleLX C is derived as follows. predSampleLX C =(f C [yFrac C [0] * refPicLX C [xPos C [Clip3(0, picH C - 1, yInt C - 1)] + f C [yFrac C [1] * refPicLX C [xPos C [Clip3(0, picH C - 1, yInt C )] + f C [yFrac C [2] * refPicLX C [xPos C [Clip3(0, picH C - 1, yInt C + 1)] + f C [yFrac C [3] * refPicLX C [xPos<用 C [Clip3(0, picH C - 1, yInt C + 2)]) >> shift1 Otherwise, xFrac Cis not equal to 0, yFrac C is not equal to 0, predSampleLX C The value of is derived as follows. When sps_ref_wraparound_enabled_flag is equal to 0, the sample array temp[n] (n = 0..3) is derived as follows. yPos C =Clip3(0, picH C -1, yInt C +n - 1) temp[n]=(f C [xFrac C [0] * refPicLX C [Clip3(0, picW C -1, xInt C -1)][yPos C + f C [xFrac C [1] * refPicLX C [Clip3(0, picW C -1, xInt C )][yPos C + f C [xFrac C [2] * refPicLX C [Clip3(0, picW C -1, xInt C +1)][yPos C + f C [xFrac C [3] * refPicLX C [Clip3(0, picW C -1, xInt C +2)][yPos C ) >> shift1 Otherwise, the sample array temp[n] (n = 0..3) is derived as follows. yPos C =Clip3(0, picH C -1, yInt C +n - 1) temp[n]=(f C [xFracC ][0]*refPicLX C [ClipH(xOffset C ,picW C xInt C -1)][yPos C ]+ f C [xFrac C ][1]*refPicLX C [ClipH(xOffset C ,picW C xInt C )][yPos C ]+ f C [xFrac C ][2]*refPicLX C [ClipH(xOffset C ,picW C xInt C +1)][yPos C ]+ f C [xFrac C ][3]*refPicLX C [ClipH(xOffset C ,picW C xInt C +2)][yPos C ])>>shift1 Predicted color difference sample value predSampleLX C This is derived as follows: predSampleLX C =( f C [yFrac C ][0]*temp[0]+ f C [yFrac C ][1]*temp[1]+ f C [yFrac C ][2]*temp[2]+ f C [yFrac C ][3]*temp[3])>>shift2 In one embodiment, if sps_ref_wraparound_enabled_flag is equal to 0 or does not exist, conventional repeating padding may be applied. Otherwise, wrap-around padding may be applied.

[0091] In one embodiment, wrap-around padding may be applied to both horizontal and vertical boundaries. A flag in the high-level syntax structure may indicate that wrap-around padding is applied both horizontally and vertically.

[0092] In one embodiment, wrap-around padding may be applied at tile or tile group boundaries. Flags in the high-level syntax structure may indicate that wrap-around padding is applied both horizontally and vertically.

[0093] In one embodiment, the reference picture may be the same as the current picture for motion compensation prediction. Wrap-around padding may be applied to the boundaries of the current picture if the current picture is a reference.

[0094] In one embodiment, when a picture is divided into one or more subpictures, which are rectangular regions of one or more slices, the boundaries of each subpicture may or may not be treated as picture boundaries. Treating subpictures as pictures indicates that the boundaries of each subpicture may be padded for motion compensation prediction.

[0095] Referring to Figure 9, in SPS (or any other parameter set), a subpic_treated_as_pic_flag[i] equal to 1 specifies that the i-th subpicture of each encoded picture in CVS is treated as a picture during the decoding process. A subpic_treated_as_pic_flag[i] equal to 0 specifies that the i-th subpicture of each encoded picture in CVS is not treated as a picture during the decoding process. If it does not exist, the value of subpic_treated_as_pic_flag[i] is assumed to be equal to 0.

[0096] In the same embodiment, the flag subpic_ref_wraparound_enabled_flag[i] is propagated if subpic_treated_as_pic_flag[i] is equal to 1, i.e., if the i-th subpicture is treated as a picture. subpic_ref_wraparound_enabled_flag[i] being equal to 1 specifies that horizontal wraparound motion compensation is applied in interpretation at the boundary of the i-th subpicture of each encoded picture. subpic_ref_wraparound_enabled_flag[i] being equal to 0 specifies that horizontal wraparound motion compensation is not applied at the boundary of the i-th subpicture of each encoded picture.

[0097] In the same embodiment, subpic_ref_wraparound_offset_minus1[i]+1 specifies the offset of the i-th subpicture of each encoded picture, which is used to calculate the horizontal wraparound position in units of MinCbSizeY luminance samples.

[0098] In the same embodiment, the interpolation filtering process is as follows: Brightness sample interpolation filtering process. The input to this process is the brightness position (xInt) for all sample units. L ,yInt L ) and the brightness position in fractional sample units (xFrac LyFrac L ) and the luminance position (xSbInt) which specifies the top-left sample of the boundary block for reference sample padding relative to the top-left luminance sample of the reference picture. L ,ySbInt L ) and the luminance reference sample array refPicLX L This may include a half-sample interpolation filter index hpelIfIdx, a variable sbWidth specifying the width of the current subblock, a variable sbHeight specifying the height of the current subblock, and a luminance position (xSb, ySb) specifying the top-left sample of the current subblock relative to the top-left luminance sample of the current picture. The output of this process is the predicted luminance sample value predSampleLX. L It may include.

[0099] The variables shift1, shift2, and shift3 are derived as follows: Variable shift1 is Min(4, BitDepth Y Variable shift2 is set to equal to 6, and variable shift3 is set to Max(2, 14-BitDepth Y The variable picW is set to equal pic_width_in_luma_samples, and the variable picH is set to equal pic_height_in_luma_samples.

[0100] xFrac L or yFrac L Brightness interpolation filter coefficient f for each 1 / 16 fractional sample position p equal to L [p] is derived as follows: When MotionModelIdc[xSb][ySb] is greater than 0 and both sbWidth and sbHeight are equal to 4, the luminance interpolation filter coefficient f L [p] may be specified. Otherwise, the luminance interpolation filter coefficient f L [p] is specified in Table 8-11, depending on hpelIfIdx.

[0101] Brightness position in all sample units (xInti ,yInt i The following is derived for i=0...7. If subpic_treated_as_pic_flag[SubPicIdx] is equal to 1, then the following applies. xCot i =Clip3(SubPicLeftBoundaryPos,SubPicRightBoundaryPos,subpic_ref_wraparound_enabled_flag[SubPicIdx] ClipH((subpic_ref_wraparound_offset_minus1[SubPicIdx]+1)*MinCbSizeY,SubPicRightBoundaryPos,xInt L +i-3): xCot L +i-3) yInt i =Clip3(SubPicTopBoundaryPos,SubPicBotBoundaryPos,yIntL+i-3) Otherwise (if subpic_treated_as_pic_flag[SubPicIdx] is equal to 0), the following applies: xCot i =Clip3(0,picW-1,sps_ref_wraparound_enabled_flag ClipH((sps_ref_wraparound_offset_minus1+1)*MinCbSizeY,picW,xInt L +i-3): xCot L +i-3) yInt i =Clip3(0,picH-1,yIntL+i-3) The luminance position in all sample units is further modified for i=0...7 as follows. xCot i =Clip3(xSbInt L -3, xSbInt L +sbWidth+4,xInt i ) yInt i = Clip3(ySbInt L - 3, ySbInt L + sbHeight + 4, yInt i ) The predicted luminance sample value predSampleLX L is derived as follows. xFrac L and yFrac L are both 0, predSampleLX L is derived as follows. predSampleLX L = refPicLX L [xInt3][yInt3] << shift3 Otherwise, xFrac L is not equal to 0 and yFrac L is equal to 0, predSampleLX L is

[0102]

Equation

[0103] Otherwise, xFrac L is equal to 0 and yFrac L is not equal to 0, predSampleLX L is

[0104]

Equation

[0105] <== 991 ==>Otherwise, xFrac L is not equal to 0 and yFrac L is not equal to 0, predSampleLX L is derived as follows.

[0106] The sample array temp[n] (n = 0..7) is

[0107]

number

[0108] Predicted luminance sample value predSampleLX L teeth,

[0109]

number

[0110] In the same embodiment or other embodiments, referring to Figure 10, in the SPS (or any other parameter set), all_subpic_treated_as_pic_flag equal to 1 specifies that any subpicture of each encoded picture in the CVS is treated as a picture during the decoding process. subpic_treated_as_pic_flag[i] equal to 0 specifies that any subpicture of each encoded picture in the CVS is not treated as a picture during the decoding process.

[0111] In the same embodiment, if all_subpic_treated_as_pic_flag[i] is equal to 1, the all_subpic_ref_wraparound_enabled_flag flag is propagated. all_subpic_ref_wraparound_enabled_flag equal to 1 specifies that horizontal wrap-around motion compensation is applied to interpretation at the boundaries of any subpicture of each encoded picture. all_sps_ref_wraparound_enabled_flag equal to 0 specifies that horizontal wrap-around motion compensation is not applied at the boundaries of any subpicture of each encoded picture.

[0112] In the same embodiment, all_subpic_ref_wraparound_offset_minus1[i]+1 specifies the offset of any subpicture in each encoded picture, which is used to calculate the horizontal wraparound position in units of MinCbSizeY luminance samples. In the same embodiment, if subpic_treated_as_pic_flag[i] is not present, the value of subpic_treated_as_pic_flag[i] is assumed to be equal to all_subpic_treated_as_pic_flag.

[0113] In the same embodiment, if subpic_ref_wraparound_enabled_flag[i] does not exist, the value of subpic_ref_wraparound_enabled_flag[i] is presumed to be equal to all_subpic_ref_wraparound_enabled_flag.

[0114] In the same embodiment, if subpic_ref_wraparound_offset_minus1[i] does not exist, the value of subpic_ref_wraparound_offset_minus1[i] is presumed to be equal to all_subpic_ref_wraparound_offset_minus1.

[0115] The techniques for the wrap-around padding process described above may be implemented as computer software using computer-readable instructions, or they may be physically stored on one or more computer-readable media. For example, Figure 11 shows a computer system 800 suitable for implementing a particular embodiment of the disclosed object.

[0116] Computer software may be encoded using any suitable machine code or computer language, which may undergo assembly, compilation, linking, or similar mechanisms to generate code containing instructions, which may be executed directly or through an interpreter, microcode execution, etc., by a central processing unit (CPU), graphics processing unit (GPU), etc.

[0117] The instructions may be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game devices, and Internet of Things devices.

[0118] The components of computer system 800 shown in Figure 11 are essentially illustrative and are not intended to imply any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. Furthermore, the configuration of the components should not be construed as having any dependency or requirement on any one or combination of components shown in the exemplary embodiments of computer system 800.

[0119] The computer system 800 may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users, for example, through haptic input (keystrokes, swipes, data glove movements, etc.), audio input (voices, applause, etc.), visual input (gestures, etc.), and olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., conversations, music, ambient sounds), images (scanned images, photographic images acquired from a still camera, etc.), and video (2D video, 3D video including stereoscopic pictures, etc.).

[0120] The input human interface device may include one or more of the following: keyboard 801, mouse 802, trackpad 803, touch screen 810, data glove 804, joystick 805, microphone 806, scanner 807, and camera 808.

[0121] Furthermore, the computer system 800 may include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touch screen 810, data glove 804, or joystick 805, although there may be tactile feedback devices that do not function as input devices), audio output devices (e.g., speaker 809, headphones (not shown)), visual output devices (screens 810 including CRT screens, LCD screens, plasma screens, OLED screens, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown). Each of these may have touch screen input functionality, each may have tactile feedback functionality, and some may be capable of outputting three or more dimensions through means such as two-dimensional visual output or stereoscopic output.

[0122] The computer system 800 may also include human-accessible storage devices and related media such as optical media including a CD / DVD ROM / RW 820 having a CD / DVD or similar medium 821, a thumb drive 822, a removable hard drive or solid-state drive 823, legacy magnetic media such as tape and floppy disks (not shown), and special ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0123] Furthermore, those skilled in the art should understand that the term “computer-readable medium” as used herein in relation to the subject matter of this disclosure does not include a transmission medium, carrier wave, or other non-transient signal.

[0124] Furthermore, the computer system 800 may include interfaces to one or more communication networks. These networks may be, for example, wireless, wired, or optical. They may also be local, wide-area, metropolitan, vehicle and industrial, real-time, or latency-tolerant. Examples of networks include Ethernet, wireless LAN, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), wired or wireless wide-area digital TV networks (including cable TV, satellite TV, and terrestrial TV), and vehicle and industrial networks (including CANBus). Certain networks generally require an external network interface adapter (e.g., a USB port on the computer system 800) attached to a specific general-purpose data port or peripheral bus 849, while other network interface adapters are generally integrated into the core of the computer system 800 by being attached to the system bus described below (e.g., an Ethernet interface to a PC computer system or a cellular network to a smartphone computer system). Using any of these networks, the computer system 800 can communicate with other entities. Such communication may be one-way reception only (e.g., broadcast television), one-way transmission only (e.g., CANbus to a specific CANbus device), or bidirectional to other computer systems using local or wide-area digital networks, for example. Specific protocols and protocol stacks may be used in each of the aforementioned networks and network interfaces.

[0125] The above-mentioned human interface devices, human-accessible memory devices, and network interfaces may be mounted on the core 840 of the computer system 800.

[0126] The core 840 may include one or more central processing units (CPUs) 841, graphics processing units (GPUs) 842, special programmable processing units in the form of field-programmable gate arrays (FPGAs) 843, hardware accelerators 844 for specific tasks, etc. These devices may be connected via a system bus 848, along with read-only memory (ROM) 845, random-access memory 846, and internal mass storage devices (internal, user-inaccessible hard drives, SSDs, etc.) 847. In some computer systems, the system bus 848 may be accessible in the form of one or more physical plugs to allow for expansion with further CPUs, GPUs, etc. Peripheral devices may be directly connected to the core's system bus 848, or they may be connected via a peripheral bus 849. The architecture of the peripheral bus may include PCI, USB, etc.

[0127] The CPU 841, GPU 842, FPGA 843, and accelerator 844 may execute specific instructions, and these specific instructions may, in combination, constitute the computer code described above. This computer code may be stored in ROM 845 or RAM 846. Temporary data may be stored in RAM 846, while persistent data may be stored, for example, in internal mass storage device 847. High-speed storage and retrieval to any of the memory devices may be enabled by using a cache memory that may be closely associated with one or more CPUs 841, GPUs 842, mass storage device 847, ROM 845, RAM 846, etc.

[0128] A computer-readable medium may contain computer code for performing operations implemented on various computers. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be readily available to those skilled in the art in the computer software field.

[0129] As an example, and not an limitation, an architecture 800, specifically a computer system having a core 840, can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) that runs software embodied in one or more tangible computer-readable media. Such computer-readable media may be media related to user-accessible mass storage as described above, as well as certain storage devices of the core 840 of a non-transient nature, such as the mass storage 847 or ROM 845 inside the core. Software implementing various embodiments of this disclosure may be stored in such devices and executed by the core 840. The computer-readable media may include one or more memory devices or chips according to specific needs. The software may cause the core 840, specifically the processor (including a CPU, GPU, FPGA, etc.) therein, to perform certain processing or specific parts of certain processing described herein, including defining data structures stored in RAM 846 and modifying such data structures according to processing defined by the software. Furthermore, or alternatively, a computer system may provide functionality as a result of logic wired within a circuit (e.g., accelerator 844) or otherwise embodied, such circuit may operate in place of or with software to perform specific processes or specific parts of specific processes described herein. References to software include logic, and vice versa, as applicable. References to computer-readable media may include, as applicable, circuits that store software for execution (such as integrated circuits (ICs)), circuits that embody logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0130] While this disclosure describes several exemplary embodiments, there are many variations, substitutions, and alternative equivalents that fall within the scope of this disclosure. Therefore, those skilled in the art will recognize that numerous systems and methods, not expressly illustrated or described herein, can be devised that embody the principles of this disclosure and thus fall within the true intent and scope of this disclosure.

Claims

1. A processor-implemented method of video encoding, comprising: generating a first flag indicating that the sub-picture is to be treated as a picture in a decoding process; generating a syntax element corresponding to a wrap-around padding process, the syntax element including a second flag indicating whether the wrap-around padding process is applied, a width of a left padding area in luma samples, and a width of a right padding area in luma samples; applying the wrap-around padding process to a sub-picture, which is a rectangular area divided from a current picture, at a boundary different from a boundary of the current picture when the first flag indicates that the sub-picture is treated as a picture in a decoding process and the second flag indicates that the wrap-around padding process is to be applied; encoding the first flag and the syntax element; Including, A method according to claim 1, wherein the wrap-around padding offset value is derived by subtracting a width of a left padding area in luma samples and a width of a right padding area in luma samples from a picture width in luma samples.

2. The method described in claim 1, wherein pixel positions for motion compensation prediction in a reference picture are determined by interpreting the syntax elements corresponding to the wrap-around padding process by a clipping process.

3. The method described in claim 2, wherein fractional pixels are interpolated for motion compensated prediction based on the determined pixel positions.

4. A method according to any one of claims 1 to 3, wherein the current picture is stored for use as a reference picture for motion compensation.

5. The method of claim 4, wherein the wraparound padding process is applied at a boundary of the current picture.

6. A method according to any one of claims 1 to 5, wherein the wraparound padding process is applied to horizontal or vertical boundaries, or to both horizontal and vertical boundaries.

7. The method described in claim 6, wherein a flag in a high-level syntax structure indicates whether the wrap-around padding process is applied at horizontal boundaries or vertical boundaries, or at both horizontal and vertical boundaries.

8. An apparatus for video decoding, comprising: one or more memories configured to store a computer program; one or more processors configured to access and execute said computer programs to implement the method of any one of claims 1 to 7; An apparatus comprising:

9. A computer program for video decoding, comprising: A computer program causing one or more computers to carry out the method according to any one of claims 1 to 7.

10. A processor-implemented method of video encoding, comprising: generating a bitstream; storing the bitstream; Including, the bitstream includes a first flag indicating that a sub-picture is to be treated as a picture in a decoding process and a syntax element corresponding to a wrap-around padding process; the syntax element includes a second flag indicating whether the wrap-around padding process is applied, a width of a left padding area in luma samples, and a width of a right padding area in luma samples; When the first flag indicates that a sub-picture is treated as a picture in a decoding process and the second flag indicates that the wrap-around padding process is applied, the wrap-around padding process is applied to a boundary of the sub-picture, which is a rectangular area divided from a current picture, that is different from a boundary of the current picture; A method according to claim 1, wherein the wrap-around padding offset value is derived by subtracting a width of a left padding area in luma samples and a width of a right padding area in luma samples from a picture width in luma samples.

Images