Video encoding and decoding methods, computer systems and computer-readable media

Abstract

This application provides a video coding method, a computer system, and a computer-readable medium. The method includes: receiving video data having one or more sub-pictures; extracting resampling parameters and spatial scalability parameters corresponding to the sub-pictures, wherein the resampling parameters and spatial scalability parameters correspond to one or more flags signaled in a parameter set associated with the video data; and decoding the video data based on the extracted resampling parameters and spatial scalability parameters.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 037,202 (filed June 10, 2020) and U.S. Patent Application No. 17 / 335,600 (filed June 1, 2021), the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of data processing, and in particular to video encoding and decoding. Background Technology

[0004] Video encoding and decoding using motion-compensated inter-picture prediction has been known for decades. Uncompressed digital video can comprise a series of pictures, each with a spatial dimension of, for example, 1920×1080 luminance samples and associated chrominance samples. This series of pictures can have a fixed or variable picture rate (informally also called frame rate), such as 60 pictures per second or 60Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at 60Hz frame rate) requires close to 1.5 Gbit / s of bandwidth. One hour of such video would require more than 600 GBytes of storage space.

[0005] One objective of video encoding and decoding is to reduce redundancy in the input video signal through compression. Compression can help reduce bandwidth or storage requirements by two orders of magnitude or more in some cases. Lossless compression and lossy compression, or combinations thereof, can be used. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal is useful for the intended application. In the case of video, lossy compression is widely used. The amount of distortion that is tolerated depends on the application; for example, users of some consumer streaming applications may tolerate higher distortion than users of television streaming applications. The achievable compression ratio can be reflected in the fact that higher allowable / tolerable distortion can result in a higher compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy coding, some of which will be described below.

[0007] In the past, video encoders and decoders tended to operate for a given picture size, which in most cases was defined and remained constant for frames with multiple pictures, such as coded video sequences (CVS), groups of pictures (GOPs), or similar structures. For example, in MPEG-2, system designs that relied on factors such as scene activity to change the horizontal resolution (and thus the picture size) were known, but these only changed the horizontal resolution at the I-picture and were therefore typically for the GOP. For example, resampling of reference pictures for different resolutions within a CVS is known from Annex P of the ITU-T H.263 Recommendation. However, here, the picture size remains unchanged, and only the reference picture is resampled, potentially resulting in only using a portion of the picture canvas (in the case of downsampling) or only capturing a portion of the scene (in the case of upsampling). Furthermore, Annex Q of H.263 allows resampling of individual macroblocks up or down by a factor of 2 (in each dimension). Again, the picture size remains the same. The macroblock size is fixed in H.263 and therefore does not require signaling.

[0008] The prediction of image size changes within images has become more mainstream in modern video coding. For example, VP9 allows for reference image resampling and changes in the resolution of the entire image. Similarly, certain proposals for VVC (e.g., "On adaptive resolution change (ARC) for VVC" by Hendry et al., Joint Video Team document JVET-M0135-v1, January 9-19, 2019, the entire contents of which are incorporated herein by reference) allow for the resampling of the entire reference image to different—higher or lower—resolutions. In that paper, it is proposed to encode different candidate resolutions in the sequence parameter set and to reference different candidate resolutions through per-image syntax elements in the image parameter set. Summary of the Invention

[0009] Implementations relate to methods, systems, and computer-readable media for video encoding. According to one aspect, a method for video encoding is provided. The method may include receiving video data having one or more sub-pictures; extracting resampling parameters and spatial scalability parameters corresponding to the sub-pictures; the resampling and spatial scalability parameters corresponding to one or more flags signaled in a parameter set associated with the video data; and decoding the video data based on the extracted resampling parameters and spatial scalability parameters.

[0010] According to another aspect, a computer system for video encoding is provided. The computer system may include one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices. The program instructions are executable by at least one of the one or more processors via at least one of the one or more memories, thereby enabling the computer system to perform a method. The method may include receiving video data having one or more sub-pictures; extracting resampling parameters and spatial scalability parameters corresponding to the sub-pictures; the resampling and spatial scalability parameters corresponding to one or more flags signaled in a parameter set associated with the video data; and decoding the video data based on the extracted resampling parameters and spatial scalability parameters.

[0011] According to another aspect, a computer-readable medium for video encoding is provided. The computer-readable medium may include one or more computer-readable storage devices and program instructions stored on at least one of the one or more tangible storage devices, the program instructions being executable by a processor. The program instructions, executable by a processor, are used to perform a method that may therefore include receiving video data having one or more sub-pictures. This involves extracting resampling parameters and spatial scalability parameters corresponding to the sub-pictures. The resampling parameters and spatial scalability parameters correspond to one or more flags signaled in a parameter set associated with the video data. The video data is then decoded based on the extracted resampling parameters and spatial scalability parameters. Attached Figure Description

[0012] These and other objects, features, and advantages will become apparent from the following detailed description of illustrative embodiments, which should be read in conjunction with the accompanying drawings. Various features in the drawings are not to scale, as the illustrations are provided for clarity and to facilitate understanding by those skilled in the art in conjunction with the detailed description. In the drawings:

[0013] Figure 1 This is a simplified block diagram of a communication system according to an embodiment;

[0014] Figure 2 This is a simplified block diagram of a communication system according to another embodiment;

[0015] Figure 3 This is a simplified block diagram of a decoder according to another embodiment;

[0016] Figure 4 This is a simplified block diagram of an encoder according to another embodiment;

[0017] Figure 5This is a schematic diagram of an option for sending ARC parameters by signaling, according to another embodiment;

[0018] Figure 6 This is an example of a syntax table according to another embodiment;

[0019] Figure 7 This is a schematic diagram of a computer system according to another embodiment;

[0020] Figure 8 This is an example of a predictive structure with scalability that adapts to changes in resolution;

[0021] Figure 9 This is an example of a syntax table according to another embodiment;

[0022] Figure 10 It is a simplified block diagram illustrating the parsing and decoding of the PoC cycle and the access unit count value of each access unit;

[0023] Figure 11 This is a schematic diagram of a video bitstream structure that includes multiple sub-images;

[0024] Figure 12 This is a schematic diagram showing the selected sub-image with enhanced resolution;

[0025] Figure 13 It is a block diagram for decoding and displaying video bitstreams that include multiple sub-images;

[0026] Figure 14 This is a schematic diagram of a 360-degree video display with an enhancement layer containing sub-images;

[0027] Figure 15 These are examples of the layout information of sub-images and their corresponding layers and image prediction structures;

[0028] Figure 16 It is an example of the layout information of a sub-image with a spatially scalable shape of a local region and its corresponding layer and image prediction structure;

[0029] Figure 17 This is an example of a syntax table used for sub-image layout information;

[0030] Figure 18 This is an example of the syntax table for the SEI message used for sub-image layout information;

[0031] Figure 19 This is an example of a syntax table that indicates the profile / layer / level information for the output layer and each set of output layers;

[0032] Figure 20 This is an example of a syntax table that indicates the output layer mode for each output layer set;

[0033] Figure 21 This is an example of a syntax table that indicates the current sub-image for each layer of each output layer set;

[0034] Figure 22 This is an example of the syntax table for a video parameter set RBSP;

[0035] Figure 23 This is an example of a syntax table indicating the output layer set in output layer setting mode;

[0036] Figure 24 This is an example of a syntax table that indicates resolution changes and reference image resampling within an encoded video sequence in SPS;

[0037] Figure 25 This is an example of a syntax table that indicates the size of images in PPS;

[0038] Figure 26 These are examples of inter-frame prediction within the same layer and inter-layer prediction with reference image resampling using a scaling window;

[0039] Figure 27 This is an example of sub-image bitstream extraction when resizing a window;

[0040] Figure 28 This is an example of sub-image sub-bitstream extraction without resizing the window, but only with offset shifting. Detailed Implementation

[0041] This document discloses detailed embodiments of the claimed structures and methods; however, it is to be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, which can be implemented in various forms. These structures and methods may be implemented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope to those skilled in the art. Details of well-known features and techniques may be omitted in the description to avoid unnecessarily obscuring the proposed embodiments.

[0042] The implementations generally relate to the field of data processing, and more specifically to video coding. The exemplary implementations described below provide a system, method, and computer program for bitstream extraction, etc., of sub-pictures in an encoded video stream having multiple layers. Therefore, some implementations have the ability to improve the computational domain by allowing improved video coding and decoding based on reference picture resampling within the video bitstream and spatial scalability parameters.

[0043] This document describes aspects with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer-readable media according to various embodiments. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0044] Figure 1 A simplified block diagram of a communication system 100 according to an embodiment of the present disclosure is shown. System 100 may include at least two terminals (110-120) interconnected via a network 150. For unidirectional data transmission, a first terminal 110 may encode video data locally for transmission to another terminal 120 via network 150. A second terminal 120 may receive the encoded video data from the other terminal from network 150, decode the encoded data, and display the recovered video data. Unidirectional data transmission is common in media service applications, etc.

[0045] Figure 1 A second pair of terminals (130, 140) is shown, which are provided to support bidirectional transmission of encoded video, for example, during video conferencing. For bidirectional data transmission, each terminal (130, 140) can encode video data captured at a local location for transmission to the other terminal via network 150. Each terminal (130, 140) can also receive encoded video data transmitted by the other terminal, can decode the encoded data, and can display the recovered video data on a local display device.

[0046] exist Figure 1 In this disclosure, terminals (110-140) may be illustrated as servers, personal computers, and smartphones, but the principles of this disclosure are not limited to these. Implementations of this disclosure can be applied to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 150 represents any number of networks transmitting encoded video data between terminals (110-140), including, for example, wired and / or wireless communication networks. Communication network 150 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this discussion, the architecture and topology of network 150 may be of little importance to the operation of this disclosure unless explained below.

[0047] Figure 2 The illustration shows the placement of a video encoder and decoder in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter can also be applied to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, Memory Sticks, etc.

[0048] A streaming system may include a capture subsystem 213, which may include a video source 201, such as a digital camera, creating, for example, an uncompressed video sample stream 202. The sample stream 202, depicted as a thick line to emphasize its high data volume when compared to an encoded video bitstream, may be processed by an encoder 203 coupled to the camera 201. The encoder 203 may include hardware, software, or a combination thereof to implement or enforce aspects of the disclosed subject matter as described in more detail below. The encoded video bitstream 204, depicted as a thin line to emphasize its low data volume when compared to the sample stream, may be stored on a streaming server 205 for future use. One or more streaming clients (206, 208) may access the streaming server 205 to retrieve copies (207, 209) of the encoded video bitstream 204. Client 206 may include video decoder 210, which decodes an incoming copy of the encoded video bitstream 207 and creates an outgoing video sample stream 211 that can be presented on display 212 or other presentation device (not depicted). In some streaming systems, the video bitstream (204, 207, 209) may be encoded according to certain video coding / compression standards. Examples of these standards include ITU-T Recommendation TH.265. Under development is a video coding standard informally known as Versatile Video Coding (VVC). The disclosed topics can be used in a VVC environment.

[0049] Figure 3 This can be a functional block diagram of the video decoder 210 according to an embodiment of the present invention.

[0050] Receiver 310 can receive one or more codec video sequences to be decoded by decoder 210; in the same or another embodiment, one encoded video sequence at a time, wherein the decoding of each encoded video sequence is independent of the others. Encoded video sequences can be received from channel 312, which can be a hardware / software link to a storage device storing the encoded video data. Receiver 310 can receive encoded video data with other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective user entities (not depicted). Receiver 310 can separate encoded video sequences from other data. To combat network jitter, buffer memory 315 can be coupled between receiver 310 and entropy decoder / parser 320 (hereinafter referred to as the "parser"). Buffer 315 may not be necessary or may be small when receiver 310 receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network. For use on best-effort packet networks such as the Internet, a buffer of 315 may be required. The buffer can be relatively large and can advantageously have an adaptive size.

[0051] Video decoder 210 may include parser 320 to reconstruct symbols 321 from an entropy-coded video sequence. These symbols may include information for managing the operation of decoder 210, and potentially information for controlling a presentation device such as display 212, which is not part of the decoder but may be... Figure 2The device is coupled to the decoder. Control information for the presentation device can take the form of Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not depicted). The parser 320 can parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence can be based on video coding techniques or standards and can follow principles known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 320 can extract a set of subgroup parameters for at least one subgroup of pixels in the encoded video sequence based on at least one parameter corresponding to a group, for use in the video decoder. Subgroups can include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The entropy decoder / parser can also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc., from the encoded video sequence.

[0052] Parser 320 can perform entropy decoding / parsing operations on the video sequence received from buffer 315 in order to create symbol 321.

[0053] Depending on the type of encoded video pictures or their portions (e.g., inter-frame pictures and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of symbol 321 can involve multiple distinct units. Which units are involved and how these units can be controlled by subgroup control information, which is parsed from the encoded video sequence by parser 320, are also factors to consider. For clarity, the flow of such subgroup control information between parser 320 and the multiple units is not depicted below.

[0054] In addition to the functional blocks already mentioned, decoder 210 can be conceptually subdivided into multiple functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and can be at least partially integrated into one another. However, for the purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate.

[0055] The first unit is the scaler / inverse transform unit 351. The scaler / inverse transform unit 351 receives quantized transform coefficients and control information from the parser 320, including which transform to use, block size, quantization factor, quantization scaling matrix, etc., as symbols 321. The scaler / inverse transform unit can output blocks containing sample values, which can be input to the aggregator 355.

[0056] In some cases, the output samples of the scaler / inverse transform unit 351 may belong to intra-coded blocks; that is, blocks that do not use prediction information from previously reconstructed images, but can use prediction information from previously reconstructed portions of the current image. Such prediction information can be provided by the intra-picture prediction unit 352. In some cases, the intra-picture prediction unit 352 uses surrounding already reconstructed information obtained from the current (partially reconstructed) image to generate blocks of the same size and shape as the blocks in the reconstruction. In some cases, the aggregator 355 adds the prediction information already generated by the intra-picture prediction unit 352 to the output sample information provided by the scaler / inverse transform unit 351 on a per-sample basis.

[0057] In other cases, the output samples of the scaler / inverse transform unit 351 may belong to inter-frame coded and potentially motion-compensated blocks. In such cases, the motion compensation prediction unit 353 can access the reference image memory 357 to obtain samples for prediction. After motion compensation of the obtained samples according to the symbols 321 belonging to the block, these samples can be added by the aggregator 355 to the output of the scaler / inverse transform unit (referred to in this case as residual samples or residual signals) to generate output sample information. The address in the reference image memory where the motion compensation unit obtains the predicted samples can be controlled by motion vectors, which can be used by the motion compensation unit in the form of symbols 321, which may have, for example, X components, Y components, and reference image components. Motion compensation may also include interpolation of sample values, such as those obtained from the reference image memory, when the precise motion vectors of the subsamples are in use, motion vector prediction mechanisms, etc.

[0058] The output samples of aggregator 355 can undergo various loop filtering techniques in loop filter unit 356. Video compression techniques may include in-loop filtering techniques, which are controlled by parameters included in the encoded video bitstream and may be used as symbol 321 from parser 320 for loop filter unit 356, but may also be in response to metadata obtained during decoding of previous (in decoding order) portions of the encoded picture or encoded video sequence, as well as sample values ​​in response to previous reconstruction and loop filtering.

[0059] The output of the loop filter unit 356 can be a sample stream, which can be output to the rendering device 212 and stored in the reference image buffer 357 for future inter-frame image prediction.

[0060] Once a certain encoded image has been fully reconstructed, it can be used as a reference image for future predictions. Once the encoded image has been fully reconstructed and has been identified as a reference image (e.g., by parser 320), the current reference image 356 can become part of the reference image buffer 357, and a new current image memory can be reallocated before reconstructing subsequent encoded images begins.

[0061] The video decoder 320 can perform decoding operations according to a predetermined video compression technique, such as that described in the ITU-T H.265 Recommendation. In the sense that the encoded video sequence conforms to the syntax of the video compression technique or standard, the encoded video sequence may conform to the syntax specified by the video compression technique or standard in use, such as as specified in the video compression technique document or standard and explicitly in the brief document therein. It may also be necessary for compliance that the complexity of the encoded video sequence is within the range defined by the level of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the hierarchy can be further restricted by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffers that are signaled in the encoded video sequence.

[0062] In this implementation, receiver 310 can receive additional (redundant) data with encoded video. The additional data can be included as part of the encoded video sequence. The additional data can be used by video decoder 320 to properly decode the data and / or more accurately reconstruct the original video data. The additional data can be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0063] Figure 4 This may be a functional block diagram of a video encoder 203 according to an embodiment of this disclosure.

[0064] Encoder 203 can receive video samples from video source 201 (which is not part of the encoder), which can capture video images to be encoded by encoder 203.

[0065] Video source 201 can provide a source video sequence to be encoded by encoder 203 in the form of a digital video sample stream. This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601YCrCb, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, video source 201 can be a storage device storing pre-prepared video. In a video conferencing system, video source 203 can be a camera capturing local image information as a video sequence. The video data can be provided as multiple individual pictures that provide motion when viewed sequentially. The pictures themselves can be organized as a spatial array of pixels, where each pixel can include one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

[0066] According to the implementation, encoder 203 can encode and compress images of the source video sequence into an encoded video sequence 443 in real time or under any other time constraints required by the application. Enforcing an appropriate encoding rate is a function of controller 450. The controller controls and is functionally coupled to other functional units as described below. For clarity, this coupling is not depicted. Parameters set by the controller may include rate control related parameters (image skipping, quantizer, λ value of rate distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Other functions of controller 450 can be readily identified by those skilled in the art, as they may relate to video encoder 203 optimized for a particular system design.

[0067] Some video encoders operate in a manner readily recognizable to those skilled in the art as an “encoding loop.” As an oversimplification, an encoding loop may include the encoding portion of encoder 430 (hereinafter referred to as the “source encoder”) (responsible for creating symbols based on the input picture to be encoded and the reference picture), and a (local) decoder 433 embedded in encoder 203, which reconstructs the symbols to create sample data that the (remote) decoder will also create (since any compression between the symbols and the encoded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). This reconstructed sample stream is fed into reference picture memory 434. Since decoding the symbol stream results in bit-precise results regardless of the decoder’s location (local or remote), the contents of the reference picture buffer are also bit-precise between the local and remote encoders. In other words, the reference picture samples “seen” by the encoder’s prediction portion are exactly the same sample values ​​that the decoder will “see” when using prediction during decoding. This fundamental principle of reference picture synchronization (and drift if, for example, synchronization cannot be maintained due to channel errors) is well known to those skilled in the art.

[0068] The operation of the "local" decoder 433 can be combined with what has already been mentioned above. Figure 3 The operation of the "remote" decoder 210 is the same as described in the detailed description; however, a brief reference is provided. Figure 3 Since the symbols are available and the encoding / decoding of the symbols of the encoded video sequence by the entropy encoder 445 and the parser 320 can be lossless, the entropy decoding part of the decoder 210, including the channel 312, receiver 310, buffer 315 and parser 320, may not be fully implemented in the local decoder 433.

[0069] It can be observed that any decoder technique present in the decoder, other than parsing / entropy decoding, must also exist in the corresponding encoder in essentially the same functional form. Therefore, the subject matter presented here focuses on decoder operation. The description of encoder techniques can be simplified because encoder techniques are inverses of fully described decoder techniques. More detailed descriptions are only provided in certain areas below.

[0070] As part of the operation of the source encoder 430, the source encoder 430 may perform motion-compensated predictive coding, which predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence designated as "reference frames". In this way, the encoding engine 432 encodes the differences between pixel blocks of the input frame and pixel blocks of the reference frame that can be selected as the prediction reference for the input frame.

[0071] The local video decoder 433 can decode encoded video data of frames that can be designated as reference frames based on symbols created by the source encoder 430. The operation of the encoding engine 432 can advantageously be lossy. When the video decoder ( Figure 4 When decoding encoded video data at a location (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder 433 can replicate the decoding process performed on the reference frame by the video decoder, and the reconstructed reference frame can be stored in the reference image memory 434. In this way, the encoder 203 can locally store a copy of the reconstructed reference frame, which has the same content as the reconstructed reference frame that will be obtained by the remote video decoder (no transmission errors).

[0072] Predictor 435 can perform a prediction search on encoding engine 432. That is, for a new frame to be encoded, predictor 435 can search the reference image memory 434 for sample data (as candidate reference pixel blocks) or specific metadata, such as reference image motion vectors, block shapes, etc., that can be used as appropriate prediction references for the new image. Predictor 435 can operate on sample blocks pixel by pixel to find appropriate prediction references. In some cases, as determined by the search results obtained by predictor 435, the input image may have prediction references extracted from multiple reference images stored in reference image memory 434.

[0073] The controller 450 can manage the encoding operations of the video encoder 430, including, for example, the setting of parameters and subgroup parameters for encoding video data.

[0074] The outputs of all the aforementioned functional units can undergo entropy encoding in the entropy encoder 445. The entropy encoder converts the symbols generated by the various functional units into an encoded video sequence by performing lossless compression on the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable-length coding, arithmetic coding, etc.

[0075] Transmitter 440 can buffer the encoded video sequence created by entropy encoder 445, thereby preparing it for transmission via communication channel 460, which can be a hardware / software link to a storage device storing the encoded video data. Transmitter 440 can combine encoded video data from video encoder 430 with other data to be transmitted (e.g., encoded audio data and / or auxiliary data streams (sources not shown)).

[0076] The controller 450 can manage the operation of the encoder 203. During encoding, the controller 450 can assign a specific encoded image type to each encoded image, which can affect the encoding techniques that can be applied to the corresponding image. For example, an image can typically be assigned to one of the following frame types:

[0077] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other frames in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, pictures refreshed by a separate decoder. Those skilled in the art will recognize those variations of I-pictures and their respective applications and characteristics.

[0078] A predicted image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and a reference index to predict the sample value for each block.

[0079] A bidirectional prediction image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict the sample values ​​for each block. Similarly, a multi-prediction image can reconstruct a single block using more than two reference images and associated metadata.

[0080] Source images can typically be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded block by block. Blocks can be predictedly encoded by referencing other (already encoded) blocks, which are determined by the encoding assignment applied to the corresponding images of the block. For example, blocks of an I-image can be unpredictably encoded, or blocks of an I-image can be predicted (spatial prediction or intra-frame prediction) by referencing already encoded blocks of the same image. Pixel blocks of a P-image can be unpredictably encoded by referencing a previously encoded reference image via spatial prediction or temporal prediction. Blocks of a B-image can be unpredictably encoded by referencing one or two previously encoded reference images via spatial prediction or temporal prediction.

[0081] Video encoder 203 can perform encoding operations according to a predetermined video coding technique or standard (e.g., ITU-T H.265 Recommendation). During operation, video encoder 203 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0082] In one implementation, transmitter 440 may send additional data along with the encoded video. Video encoder 430 may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplementary enhancement information (SEI) messages, fragments of visual usability information (VUI) parameter sets, etc.

[0083] Before describing certain aspects of the disclosed subject matter in more detail, it is necessary to introduce some terms that will be referred to in the remainder of this specification.

[0084] Subsequently, a sub-image, in some cases, refers to a rectangular arrangement of samples, blocks, macroblocks, coding units, or similar entities that are semantically grouped and can be independently encoded at varying resolutions. One or more sub-images can be used for an image. One or more encoded sub-images can form an encoded image. One or more sub-images can be assembled into an image, and one or more sub-images can be extracted from an image. In some environments, one or more encoded sub-images can be assembled in a compressed domain without transcoding the sample level into an encoded image, and in the same or some other cases, one or more encoded sub-images can be extracted from an encoded image in a compressed domain.

[0085] Adaptive Resolution Change (ARC) hereafter refers to a mechanism that allows the resolution of images or sub-images within an encoded video sequence to be changed by means of, for example, resampling of a reference image. ARC parameters hereafter refer to the control information required to perform adaptive resolution change, which may include, for example, filter parameters, scaling factors, the resolution of the output and / or reference images, various control flags, etc.

[0086] The above description focuses on encoding and decoding a single semantically independent encoded video image. Before describing the implications of encoding / decoding multiple sub-images with independent ARC parameters and their implied additional complexity, the options for signaling ARC parameters can be described.

[0087] refer to Figure 5This paper illustrates several novel options for signaling ARC parameters. As noted for each option, these options have certain advantages and disadvantages from the perspective of coding efficiency, complexity, and architecture. Video coding standards or technologies may choose one or more of these options, or options known from the prior art, for signaling ARC parameters. These options are not mutually exclusive, and it is conceivable that they may be interchanged based on application needs, the standard technology involved, or the choice of encoder.

[0088] The classes of ARC parameters can include:

[0089] - Upsampling / downsampling factors that are separated or combined in the X and Y dimensions.

[0090] - Added an upsampling / downsampling factor in the time dimension, indicating a constant rate of zooming in / out for a given number of images.

[0091] Either of the above may involve encoding one or more short syntax elements that can point to a table containing factors.

[0092] - The resolution in the X or Y dimension of the input, output, reference, or encoded images, either in units of samples, blocks, macroblocks, CUs, or any other suitable granularity. If more than one resolution exists (e.g., one for the input image and one for the reference image), then in some cases, another set of values ​​can be inferred from one set of values. This can be done, for example, by using flags for gating. See below for more detailed examples.

[0093] - The "distorted" coordinates are similar to those used in H.263 Appendix P, also at the appropriate granularity as described above. H.263 Appendix P defines an efficient way to encode such distorted coordinates, but it is conceivable that other potentially more efficient ways can be designed. For example, the variable-length, reversible "Huffman" type encoding of the distorted coordinates in Appendix P could be replaced by a binary encoding of appropriate length, where the length of the binary codeword could be derived, for example, from the maximum image size, possibly multiplied by a specific factor and offset by a specific value, to allow "distortion" outside the boundaries of the maximum image size.

[0094] - Upsampling or downsampling filter parameters. In the simplest case, there may only be a single filter for upsampling and / or downsampling. However, in some cases, allowing more flexibility in filter design can be advantageous, and this may require sending filter parameters with a signal. Such parameters can be selected by indexing from a list of possible filter designs, the filter can be fully specified (e.g., by a list of filter coefficients using appropriate entropy coding techniques), the filter can be implicitly selected by the upsampling / downsampling ratio, such parameters can be sent with a signal based on the upsampling / downsampling ratio, and so on, according to any of the mechanisms mentioned above.

[0095] Therefore, this specification assumes encoding of a finite set of upsampling / downsampling factors (using the same factors in both the X and Y dimensions) indicated by the codeword. This codeword can advantageously be used for variable-length encoding, for example, using Ext-Golomb codes, which are common to certain syntax elements in video coding specifications such as H.264 and H.265. A suitable mapping from values ​​to upsampling / downsampling factors can be, for example, according to the following table.

[0096] Typing Index Columbus Code Original / Target Resolution 0 1 1 / 1 1 010 1 / 1.5 (scaled 50%) 2 011 1.5 / 1 (scaled down by 50%) 3 00100 1 / 2 (scaled up to 100%) 4 00101 2 / 1 (scaled down by 100%)

[0097] Depending on the application requirements and the capabilities of upscaling and downscaling mechanisms available in video compression techniques or standards, many similar mappings can be designed. This table can be extended to more values. Values ​​can also be represented by entropy coding mechanisms other than exponential Golomb codes, such as binary coding. This can have certain advantages when the resampling factor is of interest outside of the video processing engine (most importantly the encoder and decoder) itself (e.g., by MANE). It should be noted that for the (probably) most common cases where resolution does not need to be changed, a short exponential Golomb code can be chosen; in the table above, only a single bit. This can have a coding efficiency advantage over using binary codes for the most common cases.

[0098] The number of entries in a table and their semantics can be fully or partially configurable. For example, a basic summary of the table can be transmitted in a "high" parameter set, such as a sequence or decoder parameter set. Alternatively or additionally, one or more such tables can be defined in a video coding technique or standard, and said tables can be selected, for example, by a decoder or sequence parameter set.

[0099] We then describe how the upsampling / downsampling factors (ARC information) encoded as described above can be included in video coding techniques or standard syntax. Similar considerations can be applied to one or more codewords controlling the upsampling / downsampling filter. See the following discussion on when filters or other data structures require a relatively large amount of data.

[0100] H.263 Appendix P includes ARC information 502 in four twisted coordinates into the image header 501 (specifically in the H.263PLUSPTYPE 503 header extension). This may be a wise design choice when a) an image header is available and b) frequent changes to the ARC information are expected. However, the overhead when using H.263-style signaling can be quite high, and the scaling factor may not be within the image boundaries because the image header can be transient.

[0101] The JVCET-M135-v1 referenced above includes ARC reference information 505 (index) located in picture parameter set 504, which includes a table 506 of the target resolution, which in turn is located within sequence parameter set 507. According to the authors' oral statements, the placement of possible resolutions in table 506 within sequence parameter set 507 can be justified by using SPS as an interoperability negotiation point during capability exchange. The resolution can be changed picture-by-picture by referencing the appropriate picture parameter set 504, within the constraints set by the values ​​in table 506.

[0102] Still referencing Figure 5 The following additional options may exist to transmit ARC information in the video bitstream. Each of these options has certain advantages over the existing technologies described above. These options may coexist in the same video coding technology or standard.

[0103] In implementations, ARC information 509, such as a resampling (scaling) factor, can be present in the title block, GOB header, tile header, or tile group header (hereinafter referred to as tile group header) 508. This is sufficient when the ARC information is small (e.g., a single variable-length ue(v) or a fixed-length codeword of a few bits, as shown above). Having the ARC information directly in the tile group header has the additional advantage that the ARC information can be applied to a sub-picture represented by, for example, the tile group, rather than the entire picture. See also below. Additionally, even if the video compression technique or standard only anticipates adaptive resolution changes to the entire picture (e.g., the opposite of adaptive resolution changes based on tile groups), placing the ARC information in the tile group header has certain advantages from an error recovery point of view compared to placing it in an H.263-style picture header.

[0104] In the same or another implementation, the ARC information 512 itself may reside in a suitable parameter set 511, such as a picture parameter set, header parameter set, tile parameter set, adaptation parameter set, etc. (as depicted in the adaptation parameter set). The scope of this parameter set may advantageously be no larger than a picture, for example, a tile group. The use of the ARC information is implicit through the activation of the relevant parameter set. For example, a picture parameter set or equivalent may be suitable when the video coding technique or standard only considers picture-based ARC.

[0105] In the same or another implementation, ARC reference information 513 may reside in tile group header 514 or a similar data structure. This reference information 513 may refer to a subset of ARC information 515 available in a parameter set 516 that extends beyond a single image, such as a sequence parameter set or decoder parameter set.

[0106] The additional layer of indirect implicit activation from the PPS of the tile group header PPS and SPS used in JVET-M0135-v1 appears unnecessary, as the picture parameter set (as a sequence parameter set only) can (and is present in some standards such as RFC3984) be used for capability negotiation or announcement. However, if the ARC information should also apply to, for example, subpictures represented by tile groups, then a parameter set limited to the activation range of the tile group, such as an adaptation parameter set or a header parameter set, might be a better choice. Moreover, if the ARC information has a negligible size—for example, containing filter control information such as multiple filter coefficients—then from a coding efficiency perspective, the parameters may be a better choice than directly using header 508, because these settings can be reused by future pictures or subpictures by referencing the same parameter set.

[0107] When using a sequence parameter set that spans multiple images or another, higher parameter set, certain considerations may be applied:

[0108] 1. In some cases, the set of parameters used to store the ARC information table 516 may be a sequence parameter set, but in other cases, it is advantageous to use a decoder parameter set. The decoder parameter set may have an active range of multiple CVSs, i.e., the encoded video stream, i.e., all encoded video bits from the start of the session to the end of the session. Such a range may be more appropriate because possible ARC factors may be decoder features, possibly implemented in hardware, and hardware features tend not to change with any CVS (which in at least some entertainment systems is a group of pictures of one second or less in length). That is to say, placing the table in a sequence parameter set is explicitly included in the placement options described herein (particularly in conjunction with point 2 below).

[0109] 2. The ARC reference information 513 can advantageously be placed directly in the picture / piece tile / GOB / tile group header (hereinafter, the tile group header) 514, instead of in the picture parameter set as in JVCET-M0135-v1, for the following reasons: When the encoder wants to change a single value in the picture parameter set, such as the ARC reference information, the encoder must create a new PPS and reference that new PPS. Assume only the ARC reference information changes, but other information, such as the quantization matrix information in the PPS, remains. Such information can be quite large and will need to be retransmitted to make the new PPS complete. Since the ARC reference information can be a single codeword, such as an index in Table 513, and this information will be the only value changed, retransmitting all, for example, quantization matrix information, would be cumbersome and wasteful. Within this scope, from a coding efficiency point of view, it is clearly better to avoid the indirection via the PPS as proposed in JVET-M0135-v1. Similarly, incorporating ARC reference information into PPS has an additional drawback: since the scope of the image parameter set activation is the image, the ARC information referenced by ARC reference information 513 must necessarily be applied to the entire image rather than sub-images.

[0110] In the same or another implementation, the signal transmission of the ARC parameters can follow the following... Figure 6 Detailed examples are outlined in the text. Figure 6 This depicts a syntax diagram of the representation used in video coding standards since at least 1993. The notation of such a syntax diagram generally follows C-type programming. Bold lines indicate syntax elements present in the bitstream, while lines without bold lines typically indicate control flow or variable settings.

[0111] A tile group header 601, as an exemplary syntax structure applicable to the header of a (possibly rectangular) portion of an image, may conditionally include a variable-length, exponentially Golomb-encoded syntax element dec_pic_size_idx 602 (depicted in bold). The presence of this syntax element in the tile group header can be gated when using adaptive resolution 603; here, the value of the flag is not shown in bold, meaning that the flag exists in the bitstream at the point where it appears in the syntax graph. Whether adaptive resolution is used for the image or a portion thereof can be signaled in any high-level syntax structure, either inside or outside the bitstream. In the example shown, as described below, whether adaptive resolution is used for the image or a portion thereof is signaled in the sequence parameter set.

[0112] Still referencing Figure 6Also shown is an excerpt of the sequence parameter set 610. The first syntax element shown is adaptive_pic_resolution_change_flag 611. When true, this flag can indicate the use of an adaptive resolution, which may in turn require some control information. In this example, such control information is conditionally presented based on the value of the flag, which is based on the parameter set 612 and the if() statement in the tile group header 601.

[0113] When using adaptive resolution, in this example, the encoded output resolution is 613 in units of samples. The number 613 refers to both `output_pic_width_in_luma_samples` and `output_pic_height_in_luma_samples`, which together define the resolution of the output image. Elsewhere in video coding techniques or standards, certain constraints on either value can be defined. For example, a hierarchy definition can limit the total number of output samples, which can be the product of the values ​​of those two syntax elements. Furthermore, certain video coding techniques or standards, or external techniques or standards such as system standards, may restrict the numbering range (e.g., one or two dimensions must be divisible by powers of 2) or the aspect ratio (e.g., width and height must be in a relationship such as 4:3 or 16:9). Such constraints can be introduced for hardware implementation convenience or for other reasons, and such constraints are well known in the art.

[0114] In some applications, the following would be preferable: the encoder instructs the decoder to use a specific reference image size, rather than implicitly assuming that size is the output image size. In this example, the syntax element `reference_pic_size_present_flag 614` gates the reference image size to 615 (again, the numbers refer to both width and height).

[0115] Finally, a table showing possible decoded image widths and heights is presented. For example, such a table could be represented by the table indicator num_dec_pic_size_in_luma_samples_minus1 616.

[0116] "minus1" can refer to the interpretation of the value of this syntax element. For example, if the encoded value is zero, there is one table entry. If the value is five, there are six table entries. For each "row" in the table, the decoded image width and height are then included in the syntax 617.

[0117] The rendered table entry 617 can be indexed using the syntax element dec_pic_size_idx 602 in the tile group header, thus allowing for different decoded sizes—actually, scaling factors—for each tile group.

[0118] Some video coding techniques or standards (such as VP9) support spatial scalability by incorporating temporal scalability to achieve certain forms of reference image resampling (using signals entirely different from the published subject). Specifically, ARC-type techniques can be used to upsample certain reference images to higher resolutions to form the basis of spatial enhancement layers. High-resolution normal prediction mechanisms can then be used to refine those upsampled images to add detail.

[0119] The disclosed subject matter can be used in such environments. In some cases, in the same or another implementation, the value in the NAL unit header, such as the time ID field, can be used to indicate not only the temporal layer but also the spatial layer. Doing so has certain advantages for some system designs; for example, for scalable environments, existing Selected Forwarding Units (SFUs) created and optimized for temporal layers selected based on the time ID value in the NAL unit header can be used without modification. To achieve this, it may be necessary that the mapping between the encoded image size and the temporal layer is indicated by the time ID field in the NAL unit header.

[0120] In some video coding techniques, an access unit (AU) can refer to an encoded image, slice, tile, or NAL unit that is captured at a given time instance and incorporated into the corresponding image / piece / tile / NAL unit bitstream. This time instance can be a composite time.

[0121] In HEVC and some other video coding techniques, the picture order count (POC) value can be used to indicate a selected reference picture among multiple reference pictures stored in a decoded picture buffer (DPB). When an access unit (AU) comprises one or more pictures, slices, or tiles, each picture, slice, or tile belonging to the same AU can carry the same POC value, from which it can be deduced that each picture, slice, or tile belonging to the same AU was created from content at the same composition time. In other words, when two pictures / slices / tiles carry the same given POC value, the same given POC value can indicate two pictures / slices / tiles belonging to the same AU and having the same composition time. Conversely, two pictures / slices / tiles with different POC values ​​can indicate those pictures / slices / tiles belonging to different AUs and having different composition times.

[0122] In embodiments of the disclosed subject matter, the aforementioned rigidity can be relaxed because access units can include images, slices, or tiles with different POC values. By allowing different POC values ​​within an AU, it becomes possible to use the POC values ​​to identify potentially independently decodable images / slices / tiles with the same rendering time. This, in turn, enables support for multiple scalable layers without altering the reference image selection signaling (e.g., reference image set signaling or reference image list signaling), as described in more detail below.

[0123] However, it is still necessary to be able to identify the AU to which an image / image / tile belongs based solely on its POC value, relative to other images / pieces / tiles with different POC values. This can be achieved as described below.

[0124] In the same or other implementations, the access unit count (AUC) can be signaled in a high-level syntax structure, such as a NAL unit header, slice header, tile group header, SEI message, parameter set, or AU delimiter. The AUC value can be used to identify which NAL units, slices, or tiles belong to a given AU. The AUC value can correspond to different composition time instances. The AUC value can be equal to a multiple of the POC value. The AUC value can be calculated by dividing the POC value by an integer value. In some cases, division can impose a burden on the decoder implementation. In such cases, the small limitation of the AUC value number space can allow shift operations to replace division. For example, the AUC value can be equal to the most significant bit (MSB) value within the POC value range.

[0125] In the same implementation, the POC cycle value (poc_cycle_au) for each AU can be signaled in a high-level syntax structure, such as a NAL unit header, slice header, tile group header, SEI message, parameter set, or AU delimiter. poc_cycle_au can indicate how many distinct and consecutive POC values ​​can be associated with the same AU. For example, if the value of poc_cycle_au is equal to 4, then pictures, slices, or tiles with POC values ​​of 0 to 3 (inclusive) are associated with an AU with an AUC value of 0, and pictures, slices, or tiles with POC values ​​of 4 to 7 (inclusive) are associated with an AU with an AUC value of 1. Therefore, the AUC value can be inferred by dividing the POC value by the value of poc_cycle_au.

[0126] In the same or another implementation, the value of poc_cycle_au can be derived from information located in, for example, a video parameter set (VPS) that identifies the number of spatial or SNR layers in the encoded video sequence. Such a possible relationship is briefly described below. While the derivation described above can save a few bits in the VPS and thus improve coding efficiency, it would be advantageous to explicitly encode poc_cycle_au in a suitable high-level syntax structure, at a level lower than the video parameter set, so that poc_cycle_au can be minimized for a given small portion of the bitstream (e.g., a picture). This optimization can save more bits than could be saved through the above derivation process because the POC value (and / or the value of syntax elements indirectly referencing the POC) can be encoded in a low-level syntax structure.

[0127] In the same or another implementation, Figure 9 Example of a syntax table for signaling the following syntax elements: a syntax element indicating `vps_poc_cycle_au` in the VPS (or SPS) for `poc_cycle_au` of all pictures / slices in an encoded video sequence, and a syntax element indicating `slice_poc_cycle_au` for `poc_cycle_au` of the current slice in the slice header. If the POC value increases uniformly per AU, `vps_contant_poc_cycle_per_au` in the VPS is set to 1, and `vps_poc_cycle_au` is signaled in the VPS. In this case, `slice_poc_cycle_au` is not explicitly signaled, and the AUC value for each AU is calculated by dividing the POC value by `vps_poc_cycle_au`. If the POC value does not increase uniformly per AU, then `vps_contant_poc_cycle_per_au` in the VPS is set to 0. In this case, instead of signaling `vps_access_unit_cnt`, `slice_access_unit_cnt` is signaled in the slice header of each slice or image. Each slice or image can have a different `slice_access_unit_cnt` value. The AUC value for each AU is calculated by dividing the POC value by `slice_poc_cycle_au`. Figure 10 A block diagram illustrating the relevant workflow is shown.

[0128] In the same or other implementations, even though the POC values ​​of images, slices, or tiles may differ, images, slices, or tiles corresponding to AUs with the same AUC value can be associated with the same decoding or output time instance. Therefore, without any inter-frame resolution / decoding dependencies across images, slices, or tiles within the same AU, all or a subset of images, slices, or tiles associated with the same AU can be decoded in parallel and output at the same time instance.

[0129] In the same or other implementations, even if the POC values ​​of images, slices, or tiles may differ, images, slices, or tiles corresponding to AUs with the same AUC value can be associated with the same composition / display time instance. When composition time is included in a container format, even if images correspond to different AUs, images can be displayed in the same time instance if they have the same composition time.

[0130] In the same or other implementations, each image, tile, or patch can have the same temporal identifier (temporal_id) within the same AU. All or a subset of images, tiles, or patches corresponding to a time instance can be associated with the same temporal sublayer. In the same or other implementations, each image, tile, or patch can have the same or different spatial layer IDs (layer_id) within the same AU. All or a subset of images, tiles, or patches corresponding to a time instance can be associated with the same or different spatial layers.

[0131] The technique described throughout for transmitting adaptive resolution parameters using signals can be implemented as computer software using computer-readable instructions and can be physically stored in one or more computer-readable media. For example, Figure 7 A computer system 700 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0132] Computer software can be encoded using any suitable machine code or computer language, and can be subjected to mechanisms such as assembly, compilation, and linking to create code containing instructions that can be executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU), or through interpretation, microcode execution, etc.

[0133] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0134] Figure 7The components shown for computer system 700 are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. Nor should the configuration of the components be construed as having any dependency or requirement associated with any component or combination thereof shown in the exemplary embodiments of computer system 700.

[0135] Computer system 700 may include certain human-machine interface input devices. Such human-machine interface input devices can respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, waving, movement of a data glove), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not depicted). Human-machine interface devices can also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0136] The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard 701, mouse 702, trackpad 703, touch screen 710, data glove 704, joystick 705, microphone 706, scanner 707, and camera 708.

[0137] Computer system 700 may also include certain human-machine interface (HMI) output devices. Such HMI output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such HMI output devices may include tactile output devices (e.g., tactile feedback via touchscreen 710, data glove 704, or joystick 705, but may also include tactile feedback devices that are not used as input devices), audio output devices (e.g., speaker 709, headphones (not depicted)), visual output devices (e.g., screen 710, including CRT screens, LCD screens, plasma screens, OLED screens, each screen may or may not have touchscreen input capability, each may or may not have tactile feedback capability—some of which may be able to output two-dimensional or more than three-dimensional visual output in a manner such as stereoscopic image output; virtual reality glasses (not depicted), holographic displays, and ashtrays (not depicted)) and printers (not depicted).

[0138] The computer system 700 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 720 with media such as CD / DVD 721, thumb drives 722, removable hard disk drives or solid-state drives 723, conventional magnetic media such as magnetic tapes and floppy disks (not depicted), devices based on dedicated ROM / ASIC / PLD such as security dongles (not depicted), etc.

[0139] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transient signals.

[0140] The computer system 700 may also include interfaces to one or more communication networks. These networks can be, for example, wireless, wired, or optical. Networks can also be local, wide area, metropolitan, vehicular, and industrial; real-time; or latency-tolerant. Examples of networks include local area networks (LANs) such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, and vehicle and industrial networks including CANbus, etc. Some networks typically require an external network interface adapter that attaches to a general-purpose data port or peripheral bus 749 (e.g., a USB port such as that of computer system 700); other networks are typically integrated into the core of computer system 700 by attaching to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, computer system 700 can communicate with other entities. Such communication can be unidirectional, receive-only (e.g., broadcasting TV), transmit-only (e.g., to a CANbus device), or bidirectional, such as to other computer systems using local or wide-area digital networks. As mentioned above, certain protocols and protocol stacks can be used on each of these networks and network interfaces.

[0141] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces can be attached to the core 740 of the computer system 700.

[0142] The core 740 may include one or more Central Processing Units (CPUs) 741, Graphics Processing Units (GPUs) 742, Dedicated Programmable Processing Units (FPGAs) 743, hardware accelerators 744 for certain tasks, etc. These devices, along with Read-Only Memory (ROM) 745, Random Access Memory (RAM) 746, and internal mass storage devices 747 such as internal non-user-accessible hard disk drives (HDDs) or SSDs, can be connected via a system bus 748. In some computer systems, the system bus 748 may be accessed as one or more physical connectors to allow for expansion by adding CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 748 or via a peripheral bus 749. Peripheral bus architectures include PCI, USB, etc.

[0143] The CPU 741, GPU 742, FPGA 743, and accelerator 744 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 745 or RAM 746. Transient data can also be stored in RAM 746, while permanent data can be stored, for example, in internal mass storage 747. Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs 741, GPUs 742, mass storage devices 747, ROM 745, RAM 746, etc.

[0144] Computer-readable media may contain computer code for performing operations of various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of types known and available to those skilled in the art of computer software.

[0145] By way of example and not limitation, a computer system having architecture 700, particularly core 740, can provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as described above, as well as specific storage devices of core 740 with non-transient characteristics, such as internal mass storage device 747 or ROM 745. Software implementing various embodiments of this disclosure can be stored in such devices and executed by core 740.

[0146] Depending on specific needs, a computer-readable medium may include one or more memory devices or chips. Software may enable core 740, and in particular its processors (including CPUs, GPUs, FPGAs, etc.), to perform the specific processing described herein or specific portions of the specific processing, including defining data structures stored in RAM 746 and modifying such data structures according to the software-defined processing. Alternatively or as an alternative, the computer system may be provided with functionality by hard-wired or otherwise embodied logic (e.g., accelerator 744) that may replace or operate with software to perform the specific processing described herein or specific portions of the specific processing.

[0147] Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry storing software for execution (such as integrated circuits (ICs)), circuitry containing logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0148] In the same or another implementation, a sub-image having (W, H) can be encoded and included in the encoded bitstream corresponding to layer 0, wherein W indicates the width of the sub-image and H indicates the height of the sub-image, respectively, while a sub-image having (W*S) is upsampled (or downsampled) from the sub-image having the original spatial resolution. w,k H*S h,k The sub-images of S can be encoded and contained in the encoded bitstream corresponding to layer k, where S w,k S h,k Indicates the horizontal and vertical resampling ratios. If S w,k S h,k If the value of S is greater than 1, then resampling is equal to upsampling. However, if S... w,k S h,k If the value is less than 1, then resampling is equal to downsampling.

[0149] In the same or another implementation, the encoded sub-images in a layer may have a different visual quality than the encoded sub-images in another layer of the same or different sub-images. For example, sub-image i in layer n is encoded using quantization parameter Q. i,n The encoding is done using the quantization parameter Q, while the sub-image j in layer m is encoded using the quantization parameter Q. j,m It's used for encoding.

[0150] In the same or another embodiment, an encoded sub-image in a layer may be independently decodeable without any parsing or decoding dependency on an encoded sub-image in another layer of the same local region. A sub-image layer that can be independently decoded without reference to another sub-image layer of the same local region is an independent sub-image layer. An encoded sub-image in an independent sub-image layer may or may not have a decoding or parsing dependency on a previous encoded sub-image in the same sub-image layer, but an encoded sub-image may not have any dependency on an encoded image in another sub-image layer.

[0151] In the same or another embodiment, an encoded sub-image in a layer can be dependently decoded in the presence of any parsing or decoding dependency on an encoded sub-image in another layer of the same local region. A sub-image layer that can be dependently decoded in the presence of another sub-image layer of the same local region is a dependent sub-image layer. An encoded sub-image in a dependent sub-image can refer to an encoded sub-image belonging to the same sub-image, a previously encoded sub-image in the same sub-image layer, or both.

[0152] In the same or another implementation, the encoded sub-image consists of one or more independent sub-image layers and one or more dependent sub-image layers. However, for an encoded sub-image, at least one independent sub-image layer may exist. An independent sub-image layer may have a layer identifier (layer_id) value equal to 0, which may exist in the NAL unit header or another high-level syntax structure. The sub-image layer with layer_id equal to 0 is the base sub-image layer.

[0153] In the same or another implementation, the image may include one or more foreground sub-images and a background sub-image. The region supported by the background sub-image may be equal to the region of the image. The region supported by the foreground sub-image may overlap with the region supported by the background sub-image. The background sub-image may be a base sub-image layer, while the foreground sub-image may be a non-base (enhanced) sub-image layer. One or more non-base sub-image layers may reference the same base layer for decoding. Each non-base sub-image layer with layer_id equal to 'a' may reference a non-base sub-image layer with layer_id equal to 'b', where 'a' is greater than 'b'.

[0154] In the same or another implementation, the image may include one or more foreground sub-images, with or without a background sub-image. Each sub-image may have its own base sub-image layer and one or more non-base (enhancement) layers. Each base sub-image layer may be referenced by one or more non-base sub-image layers. Each non-base sub-image layer with layer_id equal to a may reference a non-base sub-image layer with layer_id equal to b, where a is greater than b.

[0155] In the same or another embodiment, an image may include one or more foreground sub-images, with or without a background sub-image. Each coded sub-image in a (basic or non-basic) sub-image layer may be referenced by one or more non-basic layer sub-images belonging to the same sub-image and one or more non-basic layer sub-images not belonging to the same sub-image.

[0156] In the same or another embodiment, an image may include one or more foreground sub-images, with or without a background sub-image. A sub-image in layer a may be further segmented into multiple sub-images within the same layer. One or more coded sub-images in layer b may refer to the segmented sub-images in layer a.

[0157] In the same or another implementation, a coded video sequence (CVS) can be a set of coded pictures. A CVS can include one or more coded sub-picture sequences (CSPS), where a CSPS can be a set of coded sub-pictures covering the same local region of a picture. CSPS can have the same or different temporal resolution as the coded video sequence.

[0158] In the same or another implementation, the CSPS can be encoded and contained in one or more layers. The CSPS may include one or more CSPS layers. Decoding one or more CSPS layers corresponding to the CSPS can reconstruct a sub-image sequence corresponding to the same local region.

[0159] In the same or another implementation, the number of CSPS layers corresponding to a CSPS may be the same as or different from the number of CSPS layers corresponding to another CSPS.

[0160] In the same or another implementation, the CSPS layer may have a different temporal resolution (e.g., frame rate) than another CSPS layer. The original (uncompressed) sub-image sequence can be temporally resampled (upsampled or downsampled), encoded using different temporal resolution parameters, and included in the bitstream corresponding to the layer.

[0161] In the same or another embodiment, a sub-picture sequence having a frame rate F can be encoded and included in the encoded bitstream corresponding to layer 0, while a sequence from F*S can be included. t,k The original sub-image sequence is temporally upsampled (or downsampled) and the resulting sub-image sequence is encoded and included in the encoded bitstream corresponding to layer k, where S t,k Indicates the time sampling ratio of layer k. If S t,kIf the value is greater than 1, the temporal resampling process is equivalent to frame rate up-conversion. If the S t,k value is less than 1, the temporal resampling process is equivalent to frame rate down-conversion.

[0162] In the same or another embodiment, when a sub-picture with a CSPS layer a is referenced by a sub-picture with a CSPS layer b for motion compensation or any inter-layer prediction, if the spatial resolution of the CSPS layer a is different from the spatial resolution of the CSPS layer b, the decoded pixels in the CSPS layer a are resampled and used for reference. The resampling process may require upsampling filtering or downsampling filtering.

[0163] Figure 11 An example video stream including a background video CSPS with layer_id equal to 0 and multiple foreground CSPS layers is shown. Although the encoded sub-picture may include one or more CSPS layers, the background area that does not belong to any foreground CSPS layer may include a base layer. The base layer may contain both the background area and the foreground area, while the enhanced CSPS layer contains the foreground area. In the same area, the enhanced CSPS layer may have better visual quality than the base layer. The enhanced CSPS layer may reference the motion vectors and reconstructed pixels of the base layer corresponding to the same area.

[0164] In the same or another embodiment, in a video file, the video bitstream corresponding to the base layer is contained in a track, while the CSPS layer corresponding to each sub-picture is contained in a separate track.

[0165] In the same or another embodiment, the video bitstream corresponding to the base layer is contained in a track, while the CSPS layers with the same layer_id are contained in separate tracks. In this example, the track corresponding to layer k only includes the CSPS layer corresponding to layer k.

[0166] In the same or another embodiment, each CSPS layer of each sub-picture is stored in a separate track. Each track may or may not have any parsing or decoding dependencies with one or more other tracks.

[0167] In the same or another embodiment, each track may contain the bitstreams corresponding to layers i to j of the CSPS layers of all sub-pictures or a subset of sub-pictures, where 0 < i = < j = < k, and k is the highest layer of the CSPS.

[0168] In the same or another embodiment, a picture includes one or more associated media data, and the media data includes depth maps, alpha maps, 3D geometric data, occupancy maps, etc. Such associated timed media data can be divided into one or more data sub-streams, and each data sub-stream corresponds to a sub-picture.

[0169] In the same or another implementation, Figure 12 An example of a video conferencing method based on multi-layer subpictures is shown. The video stream contains a base layer video bitstream corresponding to a background image and one or more enhancement layer video bitstreams corresponding to foreground subpictures. Each enhancement layer video bitstream corresponds to a CSPS layer. On the display, the image corresponding to the base layer is shown by default. It contains picture-in-picture (PIP) images of one or more users. When a specific user is selected via client controls, the enhanced CSPS layer corresponding to the selected user is decoded and displayed with enhanced quality or spatial resolution. Figure 13 A diagram illustrating the operation is shown.

[0170] In the same or another implementation, a network middleware box (e.g., a router) can select a subset of layers to send to the user based on its bandwidth. Image / sub-image organization can be used for bandwidth adaptation. For example, if the user has no bandwidth, the router may strip layers or select some sub-images based on the importance of the layers or sub-images or based on the settings used, and this can be done dynamically to adapt to the bandwidth.

[0171] Figure 14 This illustrates the use of 360-degree video. When a spherical 360-degree image is projected onto a planar image, the projected 360-degree image can be segmented into multiple sub-images as a base layer. Enhancement layers for specific sub-images can be encoded and sent to the client. The decoder is capable of decoding the base layer containing all sub-images and the enhancement layer for the selected sub-image. When the current viewport matches the selected sub-image, the displayed image can have higher quality due to the decoded sub-images with enhancement layers. Otherwise, the decoded image with the base layer can be displayed at lower quality.

[0172] In the same or another implementation, any layout information used for display may exist in the file as supplementary information (e.g., SEI messages or metadata). One or more decoded sub-images can be repositioned and displayed based on layout information signaled by a stream server or broadcasting device, or can be regenerated by a network entity or cloud server, or determined through user-defined settings.

[0173] In this implementation, when the input image is divided into one or more (rectangular) sub-regions, each sub-region can be encoded as an independent layer. Each independent layer corresponding to a local region can have a unique layer_id value. For each independent layer, the size and position information of the sub-image can be signaled. For example, image size (width, height) and offset information of the top-left corner (x_offset, y_offset). Figure 15An example is shown of the layout of the divided sub-images, their size and position information, and their corresponding image prediction structure. Layout information, including sub-image size and position, can be signaled in high-level syntax structures (e.g., headers or SEI messages of parameter sets, slices, or tile groups).

[0174] In the same implementation, each sub-image corresponding to an independent layer can have its unique POC value within the AU. The POC value of each sub-image corresponding to a layer can be used when a reference image among the images stored in the DPB is indicated by using syntax elements in the RPS or RPL structure.

[0175] In the same or another implementation, to indicate the (interlayer) predicted structure, layer_id may be omitted, and the POC (incremental) value may be used instead.

[0176] In the same implementation, a sub-image with a POC value equal to N corresponding to a layer (or local region) can be used or not used as a reference image for a sub-image with a POC value equal to N+K corresponding to the same layer (or the same local region) used for motion compensation prediction. In most cases, the value of the number K can be equal to the maximum number of (independent) layers, which can be the same as the number of sub-regions.

[0177] In the same or another implementation, Figure 16 It shows Figure 15 This extends to other cases. When the input image is divided into multiple (e.g., four) sub-regions, each local region can be encoded using one or more layers. In this case, the number of independent layers can be equal to the number of sub-regions, and one or more layers can correspond to a sub-region. Therefore, each sub-region can be encoded using one or more independent layers and zero or more dependent layers.

[0178] In the same implementation, Figure 16 In this approach, the input image can be divided into four sub-regions. The upper right sub-region can be encoded into two layers, namely layer 1 and layer 4, while the lower right sub-region can be encoded into two layers, namely layer 3 and layer 5. In this case, layer 4 can refer to layer 1 for motion compensation prediction, while layer 5 can refer to layer 3 for motion compensation.

[0179] In the same or another implementation, in-loop filtering across layer boundaries (e.g., deblocking filtering, adaptive in-loop filtering, shaper, bilateral filtering, or any deep learning-based filtering) can be (optionally) disabled.

[0180] In the same or another implementation, motion compensation prediction across layer boundaries or intra-block replication can be (optionally) disabled.

[0181] In the same or another implementation, boundary filling for in-loop filtering or motion compensation prediction at the boundaries of sub-images may optionally be processed. A flag indicating whether boundary filling is to be processed can be signaled in a high-level syntax structure, such as a parameter set (VPS, SPS, PPS, or APS), a slice or tile group header, or an SEI message.

[0182] In the same or another implementation, the layout information of the sub-region (or sub-image) can be notified by signal in the VPS or SPS. Figure 17 Examples of syntax elements in VPS and SPS are shown. In this example, `vps_sub_picture_division_flag` is signaled in the VPS. This flag indicates whether the input image is divided into multiple sub-regions. When the value of `vps_sub_picture_division_flag` is equal to 0, the input image in the encoded video sequence corresponding to the current VPS may not be divided into multiple sub-regions. In this case, the input image size may be equal to the encoded image size (`pic_width_in_luma_samples`, `pic_height_in_luma_samples`), which is signaled in the SPS. When the value of `vps_sub_picture_dividing_flag` is equal to 1, the input image can be divided into multiple sub-regions. In this case, the syntax elements `vps_full_pic_width_in_luma_samples` and `vps_full_pic_height_in_luma_samples` are signaled in the VPS. The values ​​of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples can be equal to the width and height of the input image, respectively.

[0183] In the same implementation, the values ​​of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples may not be used for decoding, but rather for compositing and display.

[0184] In the same implementation, when the value of `vps_sub_picture_dividing_flag` is equal to 1, the syntax elements `pic_offset_x` and `pic_offset_y`, corresponding to (a) a specific layer, can be signaled in the SPS. In this case, the encoded image size (`pic_width_in_luma_samples`, `pic_height_in_luma_samples`) signaled in the SPS can be equal to the width and height of the sub-region corresponding to the specific layer. Furthermore, the position of the top-left corner of the sub-region (`pic_offset_x`, `pic_offset_y`) can be signaled in the SPS.

[0185] In the same implementation, the position information (pic_offset_x, pic_offset_y) of the top left corner of the sub-region can be used for compositing and display instead of decoding.

[0186] In the same or another implementation, the layout information (size and position) of all sub-regions or subsets of sub-regions of the input image, as well as the dependency information between layers, can be signaled in the parameter set or SEI message. Figure 18 This example illustrates syntax elements used to indicate information about the layout of subregions, dependencies between layers, and relationships between subregions and one or more layers. In this example, the syntax element `num_sub_region` indicates the number of (rectangular) subregions in the current encoded video sequence. The syntax element `num_layers` indicates the number of layers in the current encoded video sequence. The value of `num_layers` can be equal to or greater than the value of `num_sub_region`. When any subregion is encoded as a single layer, the value of `num_layers` can be equal to the value of `num_sub_region`. When one or more subregions are encoded as multiple layers, the value of `num_layers` can be greater than the value of `num_sub_region`. The syntax element `direct_dependency_flag[i][j]` indicates the dependency from layer j to layer i. `num_layers_for_region[i]` indicates the number of layers associated with subregion i. `sub_region_layer_id[i][j]` indicates the layer_id of layer j associated with subregion i. `sub_region_offset_x[i]` and `sub_region_offset_y[i]` indicate the horizontal and vertical positions of the top-left corner of the `i`-th subregion, respectively. `sub_region_width[i]` and `sub_region_height[i]` indicate the width and height of the `i`-th subregion, respectively.

[0187] In one implementation, one or more syntax elements can be signaled within a high-level syntax structure, such as VPS, DPS, SPS, PPS, APS, or SEI messages, specifying an output layer set of one or more layers to be output with or without profile tier level information. See also... Figure 19 In a VPS, a syntax element `num_output_layer_sets` can be used to signal the number of output layer sets (OLS) in the encoded video sequence of the reference VPS. For each output layer set, the `output_layer_flag` can be signaled as many times as the number of output layers.

[0188] In the same implementation, output_layer_flag[i] equal to 1 specifies that the i-th layer is output. vps_output_layer_flag[i] equal to 0 specifies that the i-th layer is not output.

[0189] In the same or another implementation, one or more syntax elements specifying the level information for each output layer set can be signaled in a high-level syntax structure (e.g., VPS, DPS, SPS, PPS, APS, or SEI messages). Still referring to... Figure 19 In the VPS, a syntax element `num_profile_tile_level` can be used to signal the number of tile level information per OLS in the encoded video sequence of the reference VPS. For each output layer set, as many sets of syntax elements as output layers can be signaled for the tile level information, or for the index of a specific tile level information among the entries in the tile level information.

[0190] In the same implementation, profile_tier_level_idx[i][j] specifies the index of the profile_tier_level() syntax structure applied to the j-th layer of the i-th OLS in the list of profile_tier_level() syntax structures in the VPS.

[0191] In the same or another embodiment, refer to Figure 20 When the maximum number of layers is greater than 1 (vps_max_layers_minus1 > 0), the syntax elements num_profile_tile_level and / or num_output_layer_sets can be notified by signals.

[0192] In the same or another embodiment, refer to Figure 20 The syntax element vps_output_layers_mode[i], which indicates the mode of output layer signaling for the i-th output layer set, can exist in the VPS.

[0193] In the same implementation, `vps_output_layers_mode[i]` equal to 0 specifies that only the highest layer is output with the i-th output layer set. `vps_output_layer_mode[i]` equal to 1 specifies that all layers are output with the i-th output layer set. `vps_output_layer_mode[i]` equal to 2 specifies that the output layers are those with `vps_output_layer_flag[i][j]` equal to 1 and output with the i-th output layer set. More values ​​can be reserved.

[0194] In the same implementation, depending on the value of vps_output_layers_mode[i] of the i-th output layer set, output_layer_flag[i][j] can be signaled or not.

[0195] In the same or another embodiment, refer to Figure 20 For the i-th output layer set, a flag vps_ptl_signal_flag[i] may exist. Depending on the value of vps_ptl_signal_flag[i], the level information of the i-th output layer set may or may not be notified by a signal.

[0196] In the same or another embodiment, refer to Figure 21 The number of subpics in the current CVS can be signaled in a high-level syntax structure (such as VPS, DPS, SPS, PPS, APS, or SEI messages) as max_subpics_minus1.

[0197] In the same implementation, refer to Figure 21 When the number of subpics is greater than 1 (max_subpics_minus1>0), the subpics identifier sub_pic_id[i] of the i-th subpic can be signaled.

[0198] In the same or another implementation, one or more syntax elements indicating the sub-image identifiers belonging to each layer of each output layer set can be signaled in the VPS. See also... Figure 22 `sub_pic_id_layer[i][j][k]` indicates the k-th sub-picture present in the j-th layer of the i-th output layer set. Using this information, the decoder can identify which sub-picture can be decoded and output for each layer of a specific output layer set.

[0199] In this implementation, the picture header (PH) is a syntactic structure containing syntactic elements applied to all slices of the encoded picture. A picture unit (PU) is a collection of NAL units that are related to each other according to specified classification rules, are sequentially decoded, and contain exactly one decoded picture. A PU may contain a picture header (PH) and one or more VCLNAL units that make up the encoded picture.

[0200] In an implementation, the SPS (RBSP) can be used for decoding processing before it is referenced, and it is included in at least one AU where TemporalId equals 0, or provided by an external device.

[0201] In an implementation, the SPS (RBSP) can be used for decoding processing before it is referenced, and it is included in at least one AU in a CVS containing one or more PPSs of the reference SPS where TemporalId is equal to 0, or it is provided by an external device.

[0202] In an implementation, the SPS (RBSP) can be used for decoding processing before it is referenced by one or more PPSs, and it is included in at least one PU of the lowest nuh_layer_id value of the PPS NAL unit of the SPS NAL unit in the reference CVS (the CVS contains one or more PPSs of the reference SPS), or is provided by an external device.

[0203] In an implementation, the SPS (RBSP) can be used for decoding processing before it is referenced by one or more PPSs. It is included in at least one PU where the TemporalId is equal to 0 and the nuh_layer_id is equal to the lowest nuh_layer_id value of the PPS NAL unit that references the SPS NAL unit, or is provided by an external device.

[0204] In an implementation, the SPS (RBSP) can be used for decoding processing before it is referenced by one or more PPSs, and it is included in at least one PU of the lowest nuh_layer_id value of the PPS NAL unit of the SPS NAL unit in the reference CVS (the CVS contains one or more PPSs of the reference SPS), or provided by an external device or provided by an external device.

[0205] In the same or another implementation, pps_seq_parameter_set_id specifies the value of sps_seq_parameter_set_id of the referenced SPS. The value of pps_seq_parameter_set_id can be the same across all PPSs referenced by the encoded image in CLVS.

[0206] In the same or another implementation, all SPS NAL cells in CVS with a specific value of sps_seq_parameter_set_id can have the same content.

[0207] In the same or another implementation, regardless of the nuh_layer_id value, SPS NAL units can share the same value space of sps_seq_parameter_set_id.

[0208] In the same or another implementation, the nuh_layer_id value of the SPS NAL cell can be equal to the lowest nuh_layer_id value of the PPS NAL cell of the reference SPS NAL cell.

[0209] In an implementation, when an SPS with nuh_layer_id equal to m is referenced by one or more PPSs with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m can be the same as the (direct or indirect) reference layer of the layer with nuh_layer_id equal to n or the layer with nuh_layer_id equal to m.

[0210] In an implementation, the PPS (RBSP) may be available for decoding processing before it is referenced, and may be included in at least one AU of TemporalId equal to the PPS NAL unit, or may be provided by an external device.

[0211] In an implementation, the PPS (RBSP) may be available for decoding processing before it is referenced, and may be included in at least one AU of the TemporalId of the PPS NAL unit in the CVS (which contains one or more PHs (or encoded chip NAL units) of the reference PPS), or may be provided by an external device.

[0212] In an implementation, the PPS (RBSP) may be available for decoding processing before it is referenced by one or more PHs (or encoded slice NAL units), and may be included in at least one PU (PU) whose nuh_layer_id is equal to the lowest nuh_layer_id value of the encoded slice NAL unit of the PPS NAL unit in the reference CVS (the CVS contains one or more PHs (or encoded slice NAL units) of the reference PPS), or may be provided by an external device.

[0213] In an implementation, the PPS (RBSP) may be available for decoding processing before it is referenced by one or more PHs (or encoded slice NAL units), and may be included in at least one PU (PU) whose TemporalId is equal to the TemporalId of the PPS NAL unit and whose nuh_layer_id is equal to the lowest nuh_layer_id value of the encoded slice NAL unit of the PPS NAL unit in the reference CVS (the CVS contains one or more PHs (or encoded slice NAL units) of the reference PPS), or may be provided by an external device.

[0214] In the same or another implementation, ph_pic_parameter_set_id in PH specifies the value of pps_pic_parameter_set_id of the referenced PPS in use. The value of pps_seq_parameter_set_id can be the same across all PPSs referenced by the encoded image in CLVS.

[0215] In the same or another implementation, all PPS NAL cells within the PU that have a specific value of pps_pic_parameter_set_id can have the same content.

[0216] In the same or another implementation, regardless of the nuh_layer_id value, PPS NAL units can share the same value space of pps_pic_parameter_set_id.

[0217] In the same or another implementation, the nuh_layer_id value of the PPS NAL cell can be equal to the lowest nuh_layer_id value of the encoded slice NAL cell of the reference PPS NAL cell.

[0218] In an implementation, when a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m can be the same (direct or indirect) reference layer as the layer with nuh_layer_id equal to n or the layer with nuh_layer_id equal to m.

[0219] In an implementation, the PPS (RBSP) may be available for decoding processing before it is referenced, and may be included in at least one AU of TemporalId equal to the PPS NAL unit, or may be provided by an external device.

[0220] In an implementation, the PPS (RBSP) may be available for decoding processing before it is referenced, and may be included in at least one AU of the TemporalId of the PPS NAL unit in the CVS (which contains one or more PHs (or encoded chip NAL units) of the reference PPS), or may be provided by an external device.

[0221] In an implementation, the PPS (RBSP) may be available for decoding processing before it is referenced by one or more PHs (or encoded slice NAL units), and may be included in at least one PU (PU) whose nuh_layer_id is equal to the lowest nuh_layer_id value of the encoded slice NAL unit of the PPS NAL unit in the reference CVS (the CVS contains one or more PHs (or encoded slice NAL units) of the reference PPS), or may be provided by an external device.

[0222] In an implementation, the PPS (RBSP) may be available for decoding processing before it is referenced by one or more PHs (or encoded slice NAL units), and may be included in at least one PU (PU) whose TemporalId is equal to the TemporalId of the PPS NAL unit and whose nuh_layer_id is equal to the lowest nuh_layer_id value of the encoded slice NAL unit of the PPS NAL unit in the reference CVS (the CVS contains one or more PHs (or encoded slice NAL units) of the reference PPS), or may be provided by an external device.

[0223] In the same or another implementation, ph_pic_parameter_set_id in PH specifies the value of pps_pic_parameter_set_id of the referenced PPS in use. The value of pps_seq_parameter_set_id can be the same across all PPSs referenced by the encoded image in CLVS.

[0224] In the same or another implementation, all PPS NAL cells within the PU that have a specific value of pps_pic_parameter_set_id can have the same content.

[0225] In the same or another implementation, regardless of the nuh_layer_id value, PPS NAL units can share the same value space of pps_pic_parameter_set_id.

[0226] In the same or another implementation, the nuh_layer_id value of the PPS NAL cell can be equal to the lowest nuh_layer_id value of the encoded slice NAL cell of the reference PPS NAL cell.

[0227] In an implementation, when a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m can be the same (direct or indirect) reference layer as the layer with nuh_layer_id equal to n or the layer with nuh_layer_id equal to m.

[0228] Output layers indicate the layers that are output from the output layer set. An output layer set (OLS) indicates a set of layers that includes a specified set of layers, where one or more layers in the set are designated as output layers. The output layer set (OLS) layer index is an index from the layers in the OLS to the list of layers in the OLS.

[0229] A sublayer indicates a time-scalable layer of a time-scalable bitstream, which includes VCL NAL units with specific values ​​of the TemporalId variable and associated non-VCL NAL units. A sublayer representation indicates a subset of the bitstream that includes lower sublayers and NAL units of a specific sublayer.

[0230] VPS RBS can be used for decoding processing before it is referenced; it is included in at least one AU with TemporalId equal to 0, or it is provided via an external device. All VPS NAL units in CVS with a specific value of vps_video_parameter_set_id can have the same content.

[0231] The `vps_video_parameter_set_id` provides an identifier for the VPS, which is then referenced by other syntax elements. The value of `vps_video_parameter_set_id` can be greater than 0.

[0232] The increment of 1 in vps_max_layers_minus1 specifies the maximum number of layers allowed in each CVS of the reference VPS.

[0233] The increment of vps_max_sublayers_minus1 by 1 specifies the maximum number of time-limited sublayers that can exist in each CVS of the reference VPS. The value of vps_max_sublayers_minus1 can be in the range of 0 to 6 (inclusive).

[0234] A `vps_all_layers_same_num_sublayers_flag` value of 1 indicates that the number of time sublayers is the same for all layers in each CVS of the reference VPS. A `vps_all_layers_same_num_sublayers_flag` value of 0 indicates that layers in each CVS of the reference VPS may or may not have the same number of time sublayers. When `vps_all_layers_same_num_sublayers_flag` is not present, its value is inferred to be 1.

[0235] A `vps_all_independent_layers_flag` value of 1 indicates that all layers in the CVS are encoded independently without using inter-layer prediction. A `vps_all_independent_layers_flag` value of 0 indicates that one or more layers in the CVS can use inter-layer prediction. When `vps_all_independent_layers_flag` is not present, its value is inferred to be 1.

[0236] vps_layer_id[i] specifies the nuh_layer_id value of the i-th layer. For any two non-negative integer values ​​of m and n, when m is less than n, the value of vps_layer_id[m] can be less than vps_layer_id[n].

[0237] A value of 1 for `vps_independent_layer_flag[i]` indicates that the layer with index `i` does not use inter-layer prediction. A value of 0 for `vps_independent_layer_flag[i]` indicates that the layer with index `i` can use inter-layer prediction, and the syntax element `vps_direct_ref_layer_flag[i][j]` for `j` in the range 0 to `i-1` (inclusive) exists in the VPS. When `vps_independent_layer_flag[i]` does not exist, its value is inferred to be 1.

[0238] `vps_direct_ref_layer_flag[i][j]` equal to 0 indicates that the layer with index `j` is not a direct reference layer for the layer with index `i`. `vps_direct_ref_layer_flag[i][j]` equal to 1 indicates that the layer with index `j` is a direct reference layer for the layer with index `i`. When `vps_direct_ref_layer_flag[i][j]` does not exist for `i` and `j` in the range 0 to `vps_max_layer_minus1` (inclusive), `vps_direct_ref_layer_flag[i][j]` is inferred to be equal to 0. When `vps_independent_layer_flag[i]` equals 0, there exists at least one value for `j` in the range 0 to `i-1` (inclusive) such that the value of `vps_direct_ref_layer_flag[i][j]` is equal to 1.

[0239] The variables NumDirectRefLayers[i], DirectRefLayerIdx[i][d], NumRefLayers[i], RefLayerIdx[i][r], and LayerUsedAsRefLayerFlag[j] are derived as follows:

[0240]

[0241] The variable GeneralLayerIdx[i], which specifies the layer index of the layer whose nuh_layer_id is equal to vps_layer_id[i], is obtained as follows:

[0242] for(i=0;i<=vps_max_layers_minus1;i++)

[0243] GeneralLayerIdx[vps_layer_id[i]]=i

[0244] For any two distinct values ​​of i and j, both within the range of 0 to vps_max_layers_minus1 (inclusive), when dependencyFlag[i][j] equals 1, the requirement for bitstream consistency is that the values ​​of chroma_format_idc and bit_depth_minus8 applied to layer i can be equal to the values ​​of chroma_format_idc and bit_depth_minus8 applied to layer j, respectively.

[0245] A value of 1 for `max_tid_ref_present_flag[i]` indicates the existence of the syntax element `max_tid_il_ref_pics_plus1[i]`. A value of 0 for `max_tid_ref_present_flag[i]` indicates the non-existence of the syntax element `max_tid_il_ref_pics_plus1[i]`.

[0246] A value of 0 for `max_tid_il_ref_pics_plus1[i]` indicates that inter-layer predictions are not used by the non-IRAP image of layer i. A value greater than 0 for `max_tid_il_ref_pics_plus1[i]` indicates that, for the decoded image of layer i, images with a TemporalId greater than `max_tid_il_ref_pics_plus1[i] - 1` are not used for ILRP. When `max_tid_il_ref_pics_plus1[i]` does not exist, its value is inferred to be equal to 7.

[0247] `each_layer_is_an_ols_flag` equal to 1 indicates that each OLS contains only one layer, and that each layer in the reference VPS's CVS is itself a single layer included as the only output layer. `each_layer_is_an_ols_flag` equal to 0 indicates that the OLS can contain more than one layer. If `vps_max_layers_minus1` equals 0, the value of `each_layer_is_an_ols_flag` is inferred to be 1. Otherwise, when `vps_all_independent_layers_flag` equals 0, the value of `each_layer_is_an_ols_flag` is inferred to be 0.

[0248] The value of ols_mode_idc equals 0, which indicates that the total number of OLS specified by the VPS is equal to vps_max_layers_minus1+1. The i-th OLS includes layers with layer indices from 0 to i (inclusive), and for each OLS, only the highest layer in the OLS is output.

[0249] The value of ols_mode_idc equals 1, which specifies that the total number of OLS specified by the VPS is equal to vps_max_layers_minus1+1. The i-th OLS includes layers with layer indices from 0 to i (inclusive), and for each OLS, all layers in the OLS are output.

[0250] The value of ols_mode_idc equal to 2 specifies that the total number of OLSs specified by the VPS is explicitly signaled, and for each OLS, the output layer is explicitly signaled, and other layers are layers that are direct or indirect reference layers to the output layer of the OLS.

[0251] The value of ols_mode_idc can be in the range of 0 to 2 (inclusive). The value 3 of ols_mode_idc is reserved for future use by ITU-T|ISO / IEC.

[0252] When vps_all_independent_layers_flag equals 1 and each_layers_is_an_ols_flag equals 0, the value of ols_mode_idc is inferred to be equal to 2.

[0253] The increment of 1 in num_output_layer_set_minus1 specifies the total number of OLS specified by the VPS when ols_mode_idc equals 2.

[0254] The variable TotalNumOlss, which specifies the total number of OLS values ​​assigned by the VPS, is obtained as follows:

[0255]

[0256] `ols_output_layer_flag[i][j]` equal to 1 indicates that when `ols_mode_idc` equals 2, the layer whose `nuh_layer_id` equals `vps_layer_id[j]` is the output layer of the i-th OLS. `ols_output_layer_flag[i][j]` equal to 0 indicates that when `ols_mode_idc` equals 2, the layer whose `nuh_layer_id` equals `vps_layer_id[j]` is not the output layer of the i-th OLS.

[0257] The variables NumOutputLayersInOls[i] specifying the number of output layers in the i-th OLS, NumSubLayersInLayerInOLS[i][j] specifying the number of sublayers in the j-th layer of the i-th OLS, OutputLayerIdInOls[i][j] specifying the nuh_layer_id value of the j-th output layer in the i-th OLS, and LayerUsedAsOutputLayerFlag[k] specifying whether the k-th layer is used as an output layer in at least one OLS are derived as follows:

[0258]

[0259]

[0260]

[0261] For each value of i in the range from 0 to vps_max_layers_minus1 (inclusive), the values ​​of LayerUsedAsRefLayerFlag[i] and LayerUsedAsOutputLayerFlag[i] may not be equal to 0. In other words, there may not be a layer that is neither the output layer of at least one OLS nor a direct reference layer of any other layer.

[0262] For each OLS, there can be at least one layer that serves as the output layer. In other words, for any value of i in the range of 0 to TotalNumOlss-1 (inclusive), the value of NumOutputLayersInOls[i] can be greater than or equal to 1.

[0263] The variables NumLayersInOls[i], which specify the layer number in the i-th OLS, and LayerIdInOls[i][j], which specify the nuh_layer_id value of the j-th layer in the i-th OLS, are obtained as follows:

[0264]

[0265]

[0266] The variable OlsLayerIdx[i][j], which specifies the OLS layer index of the layer whose nuh_layer_id is equal to LayerIdInOls[i][j], is obtained as follows:

[0267] for(i=0; i <TotalNumOlss;i++)

[0268] for j = 0; j <NumLayersInOls[i];j++)

[0269] OlsLayerIdx[i][LayerIdInOls[i][j]]=j

[0270] The lowest layer in each OLS can be an independent layer. In other words, for each i in the range of 0 to TotalNumOlss-1 (inclusive), the value of vps_independent_layer_flag[GeneralLayerIdx[LayerIdInOls[i][0]]] can be equal to 1.

[0271] Each layer can be included in at least one OLS specified by the VPS. In other words, for each layer where the specific value of nuh_layer_id nuhLayerId is equal to one of vps_layer_id[k] in the range of 0 to vps_max_layer_minus1 (inclusive), there can be at least one pair of values ​​i and j, where i is in the range of 0 to TotalNumOlss-1 (inclusive) and j is in the range of NumLayersInOls[i]-1 (inclusive) such that the value of LayerIdInOls[i][j] is equal to nuhLayerId.

[0272] In this implementation, the decoding process operates on the current image CurrPic as follows: PictureOutputFlag is set as follows: If one of the following conditions is true, PictureOutputFlag is set to 0. Otherwise, PictureOutputFlag is set to pic_output_flag.

[0273] - The current image is a RASL image, and the NoOutputBeforeRecoveryFlag of the associated IRAP image is equal to 1.

[0274] -gdr_enabled_flag equals 1, and the current image is a GDR image with NoOutputBeforeRecoveryFlag equal to 1.

[0275] -gdr_enabled_flag equals 1, the current image is associated with a GDR image where NoOutputBeforeRecoveryFlag equals 1, and the PicOrderCntVal of the current image is less than the RpPicOrderCntVal of the associated GDR image.

[0276] -sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 0, and the current AU contains image picA that meets all of the following conditions:

[0277] -PicA has a PictureOutputFlag equal to 1.

[0278] -PicA has a nuh_layer_id nuhLid that is larger than the nuh_layer_id nuhLid of the current image.

[0279] -PicA belongs to the output layer of OLS (i.e.,

[0280] OutputLayerIdInOls[TargetOlsIdx][0] is equal to nuhLid).

[0281] -sps_video_parameter_set_id is greater than 0, ols_mode_idc is equal to 2, and ols_output_layer_flag[TargetOlsIdx][GeneralLayerIdx[nuh_layer_id]] is equal to 0.

[0282] After all slices of the current image have been decoded, the current decoded image is marked as “for short-term reference”, and each ILRP entry in RefPicList[0] or RefPicList[1] is marked as “for short-term reference”.

[0283] In the same or another implementation, when each layer is an output layer set, PictureOutputFlag is set to be equal to pic_output_flag, regardless of the value of ols_mode_idc.

[0284] In the same or another implementation, when sps_video_parameter_set_id is greater than 0, each_layer_is_an_ols_flag is equal to 0, ols_mode_idc is equal to 0, and the current AU contains a picture picA that satisfies all of the following conditions: PicA has a PictureOutputFlag equal to 1, PicA has a nuh_layer_id nuhLid greater than the nuh_layer_id nuhLid of the current picture, and PicA belongs to the output layer of OLS (i.e., OutputLayerIdOLs[TargetOlsIdx][0] equals nuhLid).

[0285] In the same or another implementation, when sps_video_parameter_set_id is greater than 0, each_layer_is_an_ols_flag is equal to 0, ols_mode_idc is equal to 2, and ols_output_layer_flag[TargetOlsIdx][GeneralLayerIdx[nuh_layer_id]] is equal to 0, PictureOutputFlag is set to equal to 0.

[0286] Reference image resampling enables adaptive resolution changes within the encoded (layered) video sequence and enables spatial scalability across layers with dependencies belonging to the same output layer set.

[0287] In the implementation method, such as Figure 24 As shown, the `sps_ref_pic_resampling_enabled_flag` is signaled in the parameter set (e.g., the sequence parameter set). The `sps_ref_pic_resampling_enabled_flag` indicates whether reference picture resampling is used for adaptive resolution changes within the coded video sequence of the reference SPS, or for spatial scalability across layers. `sps_ref_pic_resampling_enabled_flag` equal to 1 indicates that reference picture resampling is enabled and one or more slices of pictures in the CLVS reference reference pictures in the active entry of the reference picture list with different spatial resolutions. `sps_ref_pic_resampling_enabled_flag` equal to 0 indicates that reference picture resampling is disabled and none of the slices of pictures in the CLVS reference reference pictures in the active entry of the reference picture list with different spatial resolutions.

[0288] In the same or another implementation, when sps_ref_pic_resampling_enabled_flag equals 1, for the current image, reference images with different spatial resolutions belong to the same layer or a different layer than the layer containing the current image.

[0289] In another implementation, `sps_ref_pic_resampling_enabled_flag` equal to 1 indicates that reference image resampling is disabled and one or more slices of an image in the CLVS reference reference images in the active entry of the reference image list with different spatial resolutions or different scaling windows. `sps_ref_pic_resampling_enabled_flag` equal to 0 indicates that reference image resampling is disabled and none of the slices of an image in the CLVS reference reference images in the active entry of the reference image list with different spatial resolutions or different scaling windows.

[0290] In the same or another implementation, when sps_ref_pic_resampling_enabled_flag equals 1, for the current image, reference images with different spatial resolutions or different scaling windows belong to the same layer or a different layer than the layer containing the current image.

[0291] In the same or another implementation, `sps_res_change_in_clvs_allowed_flag` indicates whether the image resolution changes within a CLVS or CVS. `sps_res_change_in_clvs_allowed_flag` equal to 1 indicates that the image spatial resolution can change within a CLVS of the reference SPS. `sps_res_change_in_clvs_allowed_flag` equal to 0 indicates that the image spatial resolution does not change within any CLVS of the reference SPS. When `sps_res_change_in_clvs_allowed_flag` is not present, its value is inferred to be 0.

[0292] In the same or another implementation, when sps_ref_pic_resampling_enabled_flag equals 1 and sps_res_change_in_clvs_allowed_flag equals 0, reference image resampling can be used only for spatial scalability and not for adaptive resolution changes within CLVS.

[0293] In the same or another implementation, when sps_ref_pic_resampling_enabled_flag equals 1 and sps_res_change_in_clvs_allowed_flag equals 1, reference image resampling can be used for both adaptive resolution changes and spatial scalability within CLVS.

[0294] When `sps_ref_pic_resampling_enabled_flag` equals 1, `sps_res_change_in_clvs_allowed_flag` equals 0, and `sps_video_parameter_set_id` equals 0, `pps_scaling_window_explicit_signalling_flag` can equal 1. This implies that when the image resolution is constant within CLVS or CVS and resampling is used with reference image, the scaling window parameters need to be explicitly signaled, rather than inferred from the consistency window parameters.

[0295] In the implementation, the sps_virtual_boundaries_present_flag is notified by a signal in the SPS, such as... Figure 24 As shown. The flag `sps_virtual_boundaries_present_flag` indicates whether virtual boundary information is signaled in SPS.

[0296] In the same or another implementation, the sps_virtual_boundaries_present_flag is conditionally signaled only when sps_res_change_in_clvs_allowed_flag is equal to 0, because when resampling with a reference image, the virtual boundary information may not be signaled in the SPS.

[0297] In the same implementation, `sps_virtual_boundaries_present_flag` equal to 1 indicates that information about virtual boundaries is signaled in the SPS. `sps_virtual_boundaries_present_flag` equal to 0 indicates that information about virtual boundaries is not signaled in the SPS. When one or more virtual boundaries are signaled in the SPS, in-loop filtering operations are disabled for virtual boundaries in the image across the reference SPS. In-loop filtering operations include unblocking filters, sample adaptive offset filters, and adaptive loop filter operations. When `sps_virtual_boundaries_present_flag` is not present, its value is inferred to be 0.

[0298] In the implementation, the sps_subpic_info_present_flag is notified by a signal in the SPS, such as... Figure 24As shown. The flag `sps_subpic_info_present_flag` indicates whether subpicture segmentation information is signaled in SPS.

[0299] In the same or another implementation, the sps_subpic_info_present_flag is conditionally signaled only when sps_res_change_in_clvs_allowed_flag is equal to 0, because when resampling with a reference image, the subpicture segmentation information may not be signaled in the SPS.

[0300] In the same implementation, `sps_subpic_info_present_flag` equal to 1 indicates that subpicture information exists for CLVS, and each image in CLVS can contain one or more subpictures. `sps_subpic_info_present_flag` equal to 0 indicates that subpicture information does not exist for CLVS, and each image in CLVS contains only one subpicture. When `sps_subpic_info_present_flag` is not present, its value is inferred to be 0.

[0301] In the implementation, the pps_res_change_in_clvs_allowed_flag can be notified via a signal in the PPS, such as... Figure 25 As shown, the value of pps_res_change_in_clvs_allowed_flag in PPS can be equal to the value of sps_res_change_in_clvs_allowed_flag in the SPS referenced by PPS.

[0302] In the same implementation, the image width and height information is signaled in PPS only when the value of pps_res_change_in_clvs_allowed_flag is equal to 1. When pps_res_change_in_clvs_allowed_flag is equal to 0, the image width and height values ​​are inferred to be equal to the maximum image width and height values ​​signaled in SPS.

[0303] In the same implementation, `pps_pic_width_in_luma_samples` specifies the width of each decoded image of the reference PPS in units of luminance samples. `pps_pic_width_in_luma_samples` can be non-zero, can be an integer multiple of Max(8, MinCbSizeY), and can be less than or equal to `sps_pic_width_max_in_luma_samples`. When `pps_pic_width_in_luma_samples` does not exist, its value is inferred to be equal to `sps_pic_width_max_in_luma_samples`. When `sps_ref_wraparound_enabled_flag` equals 1, the value of `(CtbSizeY / MinCbSizeY+1)` can be less than or equal to the value of `(pps_pic_width_in_luma_samples / MinCbSizeY-1)`. `pps_pic_height_in_luma_samples` specifies the height of each decoded image of the reference PPS in units of luminance samples. `pps_pic_height_in_luma_samples` can be non-zero, can be an integer multiple of `Max(8, MinCbSizeY)`, and can be less than or equal to `sps_pic_height_max_in_luma_samples`. When `pps_pic_height_in_luma_samples` does not exist, its value is inferred to be equal to `sps_pic_height_max_in_luma_samples`.

[0304] In the list of reference images, all valid reference images of an image have the same sub-image layout as the image itself, and all valid reference images are inter-layer reference images with a single sub-image.

[0305] In the same or another implementation, each active entry in RefPicList[0] or RefPicList[1] references an image with the same image size and the same subpicture layout as the current image (i.e., the SPS referenced by this image and the current image have the same value of sps_num_subpics_minus1, and for each value of j in the range of 0 to sps_num_subpics_minus1 (inclusive), the values ​​of sps_subpic_ctu_top_left_x[j], sps_subpic_ctu_top_left_y[j], sps_subpic_width_minus1[j], and sps_subpic_height_minus1[j] are the same). The image referenced by each valid entry in RefPicList[0] or RefPicList[1] is an ILRP, for which the value of sps_num_subpics_minus1 is equal to 0.

[0306] In the same or another implementation, when sps_num_subpics_minus1 is greater than 0 and sps_subpic_treated_as_pic_flag[i] is equal to 1, for each CLVS of the current layer of the reference SPS, let targetAuSet be all AUs (including endpoints) from the AU containing the first picture of the CLVS in decoding order to the AU containing the last picture of the CLVS in decoding order. The requirement for bitstream consistency is that for targetLayerSet including the current layer and all layers with the current layer as the reference layer, all of the following conditions are true:

[0307] - For each AU in targetAuSet, all images of the layers in targetLayerSet can have the same values ​​for pps_pic_width_in_luma_samples and pps_pic_height_in_luma_samples.

[0308] - All SPS referenced by the layer in targetLayerSet that has the current layer as the reference layer can have the same value of sps_num_subpics_minus1, and for each value of j in the range of 0 to sps_num_subpics_minus1 (inclusive), they can have the same values ​​of sps_subpic_ctu_top_left_x[j], sps_subpic_ctu_top_left_y[j], sps_subpic_width_minus1[j], sps_subpic_height_minus1[j], and sps_subpic_treated_as_pic_flag[j].

[0309] - For each AU in targetAuSet, for each value of j in the range of 0 to sps_num_subpics_minus1 (inclusive), all pictures in targetLayerSet that have the current layer as the reference layer can have the same value of SubpicIdVal[j].

[0310] In the same or another implementation, pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset specify offsets applied to the image size for scaling ratio calculation. When pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset are not present, their values ​​are inferred to be equal to pps_conf_win_left_offset, pps_conf_win_right_offset, pps_conf_win_top_offset, and pps_conf_win_bottom_offset, respectively.

[0311] The value of SubWidthC*(Abs(pps_scaling_win_left_offset)+Abs(pps_scaling_win_right_offset)) can be less than pps_pic_width_in_luma_samples, and the value of SubHeightC*(Abs(pps_scaling_win_top_offset)+Abs(pps_scaling_win_bottom_offset)) can be less than pps_pic_height_in_luma_samples.

[0312] The variables CurrPicScalWinWidthL and CurrPicScalWinHeightL are obtained as follows: CurrPicScalWinWidthL=pps_pic_width_in_luma_samples-SubWidthC*(pps_scaling_win_right_offset+pps_s caling_win_left_offset)CurrPicScalWinHeightL=pps_pic_height_in_luma_samples-SubHeightC*(pps_scaling_win_bottom_offset+pps_scaling_win_top_offset)

[0313] Let refPicScalWinWidthL and refPicScalWinHeightL be CurrPicScalWinWidthL and CurrPicScalWinHeightL, respectively, of the reference image to the current image of this PPS. The bitstream consistency requirement is that all of the following conditions must be met:

[0314] -CurrPicScalWinWidthL*2 is greater than or equal to refPicScalWinWidthL.

[0315] -CurrPicScalWinHeightL*2 is greater than or equal to refPicScalWinHeightL.

[0316] -CurrPicScalWinWidthL is less than or equal to refPicScalWinWidthL*8.

[0317] -CurrPicScalWinHeightL is less than or equal to refPicScalWinHeightL*8.

[0318] -CurrPicScalWinWidthL*sps_pic_width_max_in_luma_samples is greater than or equal to refPicScalWinWidthL*(pps_pic_width_in_luma_samples-Max(8,MinCbSizeY)).

[0319] -CurrPicScalWinHeightL*sps_pic_height_max_in_luma_samples is greater than or equal to refPicScalWinHeightL*(pps_pic_height_in_luma_samples-Max(8,MinCbSizeY)).

[0320] The value of SubWidthC*(Abs(pps_scaling_win_left_offset)+Abs(pps_scaling_win_right_offset)) can be less than pps_pic_width_in_luma_samples, and the value of SubHeightC*(Abs(pps_scaling_win_top_offset)+Abs(pps_scaling_win_bottom_offset)) can be less than pps_pic_height_in_luma_samples.

[0321] In the same or another implementation, the value of SubWidthC*(pps_scaling_win_left_offset+pps_scaling_win_right_offset) can be greater than or equal to -pps_pic_width_in_luma_samples*15 and less than pps_pic_width_in_luma_samples, and the value of SubHeightC*(pps_scaling_win_top_offset+pps_scaling_win_bottom_offset) can be greater than or equal to -pps_pic_height_in_luma_samples*15 and less than pps_pic_height_in_luma_samples.

[0322] In the same or another implementation, the value of SubWidthC*(pps_scaling_win_left_offset+pps_scaling_win_right_offset) can be greater than or equal to -pps_pic_width_in_luma_samples*7 and less than pps_pic_width_in_luma_samples, and the value of SubHeightC*(pps_scaling_win_top_offset+pps_scaling_win_bottom_offset) can be greater than or equal to -pps_pic_height_in_luma_samples*7 and less than pps_pic_height_in_luma_samples.

[0323] In the same or another implementation, when sps_ref_pic_resampling_enabled_flag equals 1, sps_res_change_in_clvs_allowed_flag equals 0 and sps_subpic_info_present_flag equals 1, the value of SubWidthC*(Abs(pps_scaling_win_left_offset)+Abs(pps_scaling_win_right_offset)) can be less than the minimum value of sps_subpic_width_minus1[i]+1 in the range of 0 to sps_num_subpics_minus1, and the value of SubHeightC*(Abs(pps_scaling_top_offset)+Abs(pps_scaling_win_bottom_offset)) can be less than the minimum value of sps_subpic_height_minus1[i]+1 in the range of 0 to sps_num_subpics_minus1.

[0324] In the same or another implementation, when the value of the current layer, sps_res_change_in_clvs_allowed_flag, is equal to 1, the value of sps_subpic_info_present_flag can be equal to 0.

[0325] In the same or another implementation, when the value of the current layer, sps_res_change_in_clvs_allowed_flag, is equal to 1, the value of the layer referencing the current layer, sps_subpic_info_present_flag, can be equal to 0.

[0326] In the same or another implementation, when the value of the current layer, sps_res_change_in_clvs_allowed_flag, is equal to 1, the value of sps_subpic_info_present_flag for the current layer and all layers referencing the current layer can be equal to 0.

[0327] In the same or another implementation, when the value of sps_subpic_info_present_flag of the current layer is equal to 1, the value of sps_res_change_in_clvs_allowed_flag of the current layer can be equal to 0.

[0328] In the same or another implementation, when the value of sps_subpic_info_present_flag of the current layer is equal to 1, the value of sps_res_change_in_clvs_allowed_flag of the reference layer of the current layer can be equal to 0.

[0329] In the same or another implementation, when the value of sps_subpic_info_present_flag of the current layer can be equal to 1, the value of sps_res_change_in_clvs_allowed_flag of the current layer and all reference layers of the current layer can be equal to 0.

[0330] In the same or another implementation, when the value of sps_subpic_info_present_flag of the current layer is equal to 1 and the value of sps_ref_pic_resampling_enabled_flag can be equal to 1, the value of sps_ref_pic_resampling_enabled_flag of the reference layer of the current layer can be equal to 1.

[0331] In an implementation, the coded image in layer k can be divided into one or more sub-images, such as... Figure 26 As shown, one or more reference images can be referenced within the same layer, and one or more reference images within the reference layer of layer k can be referenced. Figure 26 In the example, when sps_ref_pic_resampling_enabled_flag( Figure 24 When the value of () is equal to 1, the current image and each reference image can have different scaled windows, even if the images are the same size.

[0332] In the same or another implementation, when extracting a sub-image, the scaling window size and its offset value calculated for the scaling ratio used to resample the reference image can be updated based on the sub-image size and position. When updating the scaling window size and its offset value of the current image, the scaling window sizes and their offset values ​​of one or more reference images of the current image can be updated accordingly. Figure 27 This example shows the scaling window update for reference images within the same layer and interlayer reference images in different layers.

[0333] In the same or another implementation, when an image in layer k is segmented into one or more sub-images with the same segmentation layout, an image in another layer that layer k refers to as a reference layer may not be segmented into multiple sub-images.

[0334] In the same or another implementation, when in Figure 27 In the example, when updating the scaling window size and its offset, the scaling window size can be rescaled relative to the scaling ratio between the original image size and the extracted sub-image size. When updating the scaling window size, the updated scaling window size can have fractional pixel values ​​based on the original image size and the sub-image size. These fractional pixel values ​​cannot be determined by factors such as... Figure 25 The scaling window offset values ​​(pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, pps_scaling_win_bottom_offset) are represented by signals in the PPS shown. Furthermore, this method can update all scaling offset values ​​of the reference image. This is a considerable overhead.

[0335] In the same or another implementation, such as Figure 28 As shown, in order to signal the same scaling ratio between the current image and the reference image when extracting sub-images, the size of the scaling window does not need to be changed compared to the original scaling window before extraction. Instead, the position of the scaling window is moved only by updating the values ​​of the scaling window offsets (pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, pps_scaling_win_bottom_offset) signaled in PPS.

[0336] In the same implementation, when the scaling window offset values ​​of the original image are orgScalingWinLeft, orgScalingWinRight, orgScalingWinTop, and orgScalingWinBottom (orgScalingWinLeft, orgScalingWinRight, orgScalingWinTop, and orgScalingWinBottom are equal to the values ​​of pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset of the original image, respectively), the position and size of the extracted sub-image are represented by SubpicLeftBoundaryPos, SubpicRightBoundaryPos, SubpicTopBoundaryPos, and SubpicBotBoundaryPos, where these values ​​are derived as follows:

[0337]

[0338] The values ​​of pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset of the extracted sub-images are obtained as follows:

[0339] pps_scaling_win_left_offset=orgScalingWinLeft-(SubpicLeftBoundaryPos / SubWidthC);

[0340] pps_scaling_win_right_offset=orgScalingWinRight-(SubpicLeftBoundaryPos / SubWidthC);

[0341] pps_scaling_win_top_offset=orgScalingWinTop-(SubpicTopBroundaryPos / SubWidthC);

[0342] pps_scaling_win_bottom_offset=orgScalingWinBottom-(SubpicTopBroundaryPos / SubWidthC).

[0343] In the same or another implementation, the sub-picture sub-bitstream extraction process is as follows. The input to this process is the bitstream inBitstream, the target OLS index targetOlsIdx, the highest target TemporalId value tIdTarget, and an array of target subpicture index values ​​for each layer, subpicIdxTarget[]. The output of this process is the sub-bitstream outBistream.

[0344] The requirement for bitstream conformance of the input bitstream is that any output sub-bitstream that satisfies all of the following conditions should be a conforming bitstream. An output sub-bitstream is the output of the processing specified in this clause whose input is a bitstream whose targetOlsIdx equals an index to the OLS list specified by the VPS and whose subpicIdxTarget[] equals the index of a subpicture existing in the OLS. An output sub-bitstream contains at least one VCL NAL unit whose nuh_layer_id equals each of the nuh_layer_id values ​​in LayerldInOls[targetOlsIdx]. An output sub-bitstream contains at least one VCL NAL unit whose TemporalId equals tIdTarget. A conforming bitstream contains one or more coded slice NAL units whose TemporalId equals 0, but does not necessarily contain coded slice NAL units whose nuh_layer_id equals 0. The output sub-bitstream contains at least one VCL NAL unit containing the value of SubpicIdVal[subpicIdxTarget[i] for each i in the range from nuh_layer_id equal to LayerldInOls[targetOlsIdx]-1 (inclusive) where sh_subpic_id equals 0 to NumLayerlInOls[targetOlsIdx]-1 (inclusive of end values).

[0345] The output sub-bitstream `outBitstream` is derived as follows. The sub-bitstream extraction process is invoked with `inBitstream`, `targetOlsIdx`, and `tIdTarget` as inputs, and the output of this process is assigned to `outBitstream`. If some external means not specified in this specification are available to provide a replacement parameter set for the sub-bitstream `outBitstream`, then all parameter sets are replaced with the replacement parameter set.

[0346] Otherwise, when the subpicture level information SEI message exists in inBitstream, the following applies: Set the variable subpicIdx to the value equal to subpicIdxTarget[[NumLayersInOls[targetOlsIdx]-1]]. For a subpicture set that includes a subpicture whose subpicture index is equal to subpicIdx, rewrite the value of general_level_idc in the vps_ols_ptl_idx[targetOlsIdx] entry of the profile_tier_level() syntax structure list in all referenced VPS NAL units to be equal to SubpicLevelIdc. When VCL HRD or NAL HRD parameters exist, in the vps_ols_hrd_idx[MultiLayerOlsIdx[targetOlsIdx]]ols_hrd_parameters() syntax structure in all referenced VPS NAL units and in the ols_hrd_parameters() syntax structure in all SPS NAL units referenced by the i-th layer, rewrite the corresponding values ​​of cpb_size_value_minus1[tIdTarget][j] and bit_rate_value_minus1[tIdTarget][j] of the j-th CPB, such that they correspond to SubpicCpbSizeVcl[SubpicSetLevelIdx][subpicIdx] and SubpicCpbSizeNal[SubpicSetLevelIdx][subpicIdx].

[0347] The subpic index is equal to subPicIdx, and its subpicBitrateVcl[SubpicSetLevelIdx][subpicIdx] and SubpicBitrateNal[SubpicSetLevelIdx][subpicIdx] are subpic indexes, j is in the range of 0 to hrd_cpb_cnt_minus1 (inclusive), and i is in the range of 0 to NumLayerInOls[targetOlsIdx]-1 (inclusive).

[0348] For the i-th layer where i is in the range 0 to NumLayersInOls[targetOlsIdx]-1, the following conditions apply: Set the variable subpicIdx to the value equal to subpicIdxTarget[i]. For a subpicture set including subpictures whose subpicture index is equal to subpicIdx, rewrite the value of general_level_idc in the profile_tier_level() syntax structure of all references where sps_ptl_dpb_hrd_params_present_flag is equal to 1 to be equal to SubpicLevelIdc.

[0349] The variables subpicWidthInLumaSamples and subpicHeightInLumaSamples are derived as follows:

[0350]

[0351] Rewrite the values ​​of `sps_pic_width_max_in_luma_samples` and `sps_pic_height_max_in_luma_samples` in all referenced SPS NAL units, and the values ​​of `pps_pic_width_in_luma_samples` and `pps_pic_height_in_luma_samples` in all referenced PPS NAL units, to be equal to `subpicWidthInLumaSamples` and `subpicHeightInLumaSamples`, respectively. Rewrite the values ​​of `sps_num_subpics_minus1` in all referenced SPS NAL units and `pps_num_subpics_minus1` in all referenced PPS NAL units to 0. Rewrite the syntax elements `sps_subpic_ctu_top_left_x[subpicIdx]` and `sps_subpic_ctu_top_left_y[subpicIdx]` (if present) in all referenced SPS NAL units to 0. Remove the syntax elements sps_subpic_ctu_top_left_x[j], sps_subpic_ctu_top_left_y[j], sps_subpic_width_minus1[j], sps_subpic_height_minus1[j], sps_subpic_treated_as_pic_flag[j], sps_loop_filter_across_subpic_enabled_flag[j], and sps_subpic_id[j] from all referenced SPS NAL cells for each j that is not equal to subpicIdx. Rewrite the syntax elements for tile and slice signal notification in all referenced PPS to remove all tile rows, tile columns, and slices that are not associated with a subpic at a subpic index equal to subpicIdx.

[0352] The variables subpicConfWinLeftOffset, subpicConfWinRightOffset, subpicConfWinTopOffset, and subpicConfWinBottomOffset are derived as follows:

[0353] subpicConfWinLeftOffset=sps_subpic_ctu_top_left_x[subticIdx]==0?

[0354] sps_conf_win_left_offset:0

[0355] subpicConfWinRightOffset=(sps_subpic_ctu_top_left_x[subpicIdx]+sps_subpic_width_minus1[subpicIdx]+1)*CtbSizeY>=sps_pic_width_max_in_luma_samples?sps_conf_win_right_offset:0

[0356] subpicConfWinTopOffset=sps_subpic_ctu_top_left_y[subpicIdx]==0?

[0357] sps_conf_win_top_offset:0

[0358] subpicConfWinBottomOffset=(sps_subpic_ctu_top_left_y[subpicIdx]+sps_subpic_height_minus1[subpicIdx]+1)*CtbSizeY>=sps_subpic_height_max_in_luma_samples?

[0359] sps_conf_win_bottom_offset:0

[0360] Rewrite the values ​​of `sps_conf_win_left_offset`, `sps_conf_win_right_offset`, `sps_conf_win_top_offset`, and `sps_conf_win_bottom_offset` in all referenced SPS NAL units, and the values ​​of `pps_conf_win_left_offset`, `pps_conf_win_right_offset`, `pps_conf_win_top_offset`, and `pps_conf_win_bottom_offset` in all referenced PPS NAL units, respectively, to be equal to `subpicConfWinLeftOffset`, `subpicConfWinRightOffset`, `subpicConfWinTopOffset`, and `subpicConfWinBottomOffset`. The values ​​of `pps_scaling_win_left_offset`, `pps_scaling_win_right_offset`, `pps_scaling_win_top_offset`, and `pps_scaling_win_bottom_offset` are rewritten as follows:

[0361] pps_scaling_win_left_offset=orgScalingWinLeft-(SubpicLeftBoundaryPos / SubWidthC)

[0362] pps_scaling_win_right_offset=orgScalingWinRight-(SubpicLeftBoundaryPos / SubWidthC)

[0363] pps_scaling_win_top_offset=orgScalingWinTop-(SubpicTopBoundaryPos / SubWidthC)

[0364] pps_scaling_win_bottom_offset=orgScalingWinBottom-(SubpicTopBoundaryPos / SubWidthC),

[0365] Where orgScalingWinLeft, orgScalingWinRight, orgScalingWinTop, and orgScalingWinBottom are equal to the values ​​of pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset of the original encoded image. Remove all VCL NAL units from outBitStream where nuh_layer_id equals nuh_layer_id of the i-th layer and sh_subpic_id is not equal to SubpicIdVal[subpicIdx].

[0366] When sli_cbr_constraint_flag equals 1, remove all NAL units with nal_unit_type equal to FD_NUT and the padding payload SEI message not associated with the VCL NAL unit of the subpicIdTarget[], and set cbr_flag[tIdTarget][j] of the j-th CPB in the vps_ols_hrd_idx[MultilayerOlsIdx[targetOlsIdx]]ols_hrd_parameters() syntax structure in all referenced VPS NAL units and SPS NAL units where j is in the range 0 to hrd_cpb_cnt_minus1 to equal 1. Otherwise,

[0367] (sli_cbr_constraint_flag equals 0), remove all NAL units with nal_unit_type equal to FD_NUT and fill the payload SEI message, and set cbr_flag[tIdTarget][j] equal to 0.

[0368] When outBitstream contains a SEINAL unit—which contains a scalable nested SEI message with sn_ols_flag equal to 1 and sn_subpic_flag equal to 1 for outBitstream—the appropriate non-scalable nested SEI message with payloadType equal to 1 (PT), 130 (DUI), or 132 (decoded image hash) is extracted from the scalable nested SEI message and placed in outBitstream.

[0369] Some implementations may relate to systems, methods, and / or computer-readable media at any possible level of integration technical detail. A computer-readable medium may include a computer-readable non-transitory storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform operations.

[0370] A computer-readable storage medium can be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures containing the above-mentioned instructions, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.

[0371] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0372] Computer-readable program code / instructions used to perform operations can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). Computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network, or can be connected to an external computer (e.g., via the Internet using an Internet service provider), any type of network including a local area network (LAN) or a wide area network (WAN). In some implementations, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can perform various aspects or operations by personalizing the electronic circuits with state information of computer-readable program instructions.

[0373] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other equipment to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0374] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0375] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. The method, computer system, and computer-readable medium may include additional blocks, fewer blocks, different blocks, or blocks arranged differently compared to those depicted in the drawings. In some alternative embodiments, the functions indicated in the blocks may occur in a non-consecutive order as indicated in the drawings. For example, two blocks shown consecutively may actually be executed simultaneously or substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0376] It will be apparent that the systems and / or methods described herein can be implemented in various forms, including hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit these implementations. Therefore, this document describes the operation and behavior of the systems and / or methods without referring to any specific software code—it should be understood that software and hardware can be designed to implement the systems and / or methods based on the descriptions herein.

[0377] Unless explicitly stated otherwise, the elements, actions, or instructions used herein should not be construed as critical or necessary. Furthermore, as used herein, the articles “a” and “one” are intended to include one or more items and may be used interchangeably with “one or more.” Additionally, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and may be used interchangeably with “one or more.” The term “one” or similar language is used when referring to only one item. Furthermore, as used herein, the terms “has,” “have,” “having,” etc., are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.”

[0378] Descriptions of various aspects and implementations have been presented for illustrative purposes; however, these descriptions are not intended to be exhaustive or limited to the disclosed implementations. Even though combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. While each dependent claim listed below may be directly dependent on only one claim, the disclosure of possible implementations includes combinations of each dependent claim with each other claim in the claim set. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described implementations. The terminology used herein has been chosen to best explain the principles, practical application, or relative technical improvements available in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A video encoding method, characterized in that, include: Receive video data with one or more sub-images; Generate a sub-bitstream associated with the resampling parameters and spatial scalability parameters corresponding to the sub-image, wherein the sub-bitstream is generated from the bitstream associated with the video data based on an array of target output layer set indices, target highest time signature values, and target sub-image index values ​​associated with the video data; and The video data is encoded based on the extracted resampling parameters and spatial scalability parameters; Specifically, the video data is scaled based on the extraction of sub-bitstreams associated with the resampling parameters and the spatial scalability parameters; the left and right offset values ​​of the scaling window of the extracted sub-image are updated based on the left boundary position of the extracted sub-image; and the upper and lower offset values ​​of the scaling window of the sub-image are updated based on the upper boundary position of the extracted sub-image, so as to move the position of the scaling window based on the updated offset values.

2. The method according to claim 1, characterized in that, The resampling parameters correspond to one or more flags signaled in the parameter set associated with the video data.

3. The method according to claim 1, characterized in that, The spatial scalability parameter corresponds to one or more flags signaled in the parameter set associated with the video data.

4. A video decoding method, characterized in that, include: Receive video data with one or more sub-images; Extracting a sub-bitstream associated with the resampling parameters and spatial scalability parameters corresponding to the sub-image, wherein the sub-bitstream is extracted from the bitstream associated with the video data based on an array of target output layer set indices, target highest time signature values, and target sub-image index values ​​associated with the video data; and The video data is decoded based on the extracted resampling parameters and spatial scalability parameters; Specifically, the video data is scaled based on the extraction of sub-bitstreams associated with the resampling parameters and the spatial scalability parameters; the left and right offset values ​​of the scaling window of the extracted sub-image are updated based on the left boundary position of the extracted sub-image; and the upper and lower offset values ​​of the scaling window of the sub-image are updated based on the upper boundary position of the extracted sub-image, so as to move the position of the scaling window based on the updated offset values.

5. The method according to claim 4, characterized in that, The resampling parameters correspond to one or more flags signaled in the parameter set associated with the video data.

6. The method according to claim 4, characterized in that, The spatial scalability parameter corresponds to one or more flags signaled in the parameter set associated with the video data.

7. A computer system for video encoding, characterized in that, The computer system includes: One or more computer-readable non-transitory storage media are configured to store computer program code; and One or more computer processors are configured to access the computer program code and operate as instructed by the computer program code to implement the method as described in any one of claims 1 to 3.

8. A computer system for video decoding, characterized in that, The computer system includes: One or more computer-readable non-transitory storage media are configured to store computer program code; and One or more computer processors are configured to access the computer program code and operate as instructed by the computer program code to implement the method as described in any one of claims 4 to 6.

9. A non-transitory computer-readable medium having a computer program stored thereon for video encoding, the computer program being configured to cause one or more computer processors to perform the method as described in any one of claims 1 to 6.

10. A method for storing video bitstreams, characterized in that, The processor executes the encoding method according to any one of claims 1 to 3, generates the bit stream, and stores the bit stream in a computer-readable medium.

Images