Coding and decoding method and device, computer system and readable medium

By dynamically adjusting the number of sub-pictures and rectangular slices in video encoding and decoding, adaptive resolution changes and spatial scalability are achieved, and the problem of high bandwidth and storage requirements of video data in the prior art is solved, and encoding efficiency is improved. flexibility.

CN118921462APending Publication Date: 2024-11-08TENCENT AMERICA LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411332855.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2020-11-16
Filing Date
2021-02-15
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

When existing video encoding technology deals with uncompressed video, bandwidth and storage space requirements are high, and the fixed image size limits encoding efficiency, making it difficult to effectively utilize the time and space redundant information of the video.

Method used

By notifying the number of sub-pictures and the number of rectangular slices and their incremental values ​​in the advanced syntax fields, the encoding and decoding process of video data is dynamically adjusted, adaptive resolution changes and spatial scalability are realized, and video encoding and decoding are optimized.

Benefits of technology

It reduces the bandwidth and storage requirements of video data, improves encoding efficiency, adapts to the compression ratio and resolution requirements of different application scenarios, and improves the flexibility of video encoding and decoding.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118921462A_ABST
    Figure CN118921462A_ABST
Patent Text Reader

Abstract

The invention provides a coding and decoding method and device, a computer system and a readable medium. A method of encoding and decoding includes receiving video data including one or more sub-pictures. In a high-level syntax field, a number of the one or more sub-pictures and an increment value between the number of the one or more sub-pictures and a number of rectangular slices are signaled. The number of the rectangular slices is derived based on the number of the one or more sub-pictures and the delta value.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the Chinese patent application with application number 202180003583.1 filed with the China Patent Office on February 15, 2021. Technical Field

[0002] The disclosed subject matter relates to video encoding and decoding, and more particularly to methods, apparatus, computer systems, and readable media for encoding and decoding. Background Art

[0003] The encoding and decoding of video using inter-picture prediction techniques with motion compensation has been known for decades. An uncompressed digital video may include a series of pictures, each picture having spatial dimensions of, for example, 1920×1080 luma samples and associated chroma samples. The series of pictures has a fixed or variable picture rate (also informally called frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed video has very large bitrate requirements. For example, 1080p60 4:2:0 video (1920×1080 luma sample resolution, 60 Hz frame rate) with 8 bits per sample requires close to 1.5 Gbit / s of bandwidth. One hour of such video requires more than 600 GB of storage space.

[0004] One goal of video encoding and decoding is to reduce redundant information in the input video signal through compression. Video compression can help reduce the bandwidth or storage space requirements mentioned above, in some cases by two or more orders of magnitude. Both lossless and lossy compression, and combinations of the two, can be used. Lossless compression refers to techniques that reconstruct an exact replica of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be exactly the same as the original signal, but the distortion between the original and the reconstructed signal is small enough that the reconstructed signal is usable for the intended application. Lossy compression is widely used for video. The amount of distortion allowed depends on the application. For example, users of some consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio reflects that higher allowed / tolerable distortion produces higher compression ratios.

[0005] There are several broad categories of techniques that video encoders and decoders can use, including, for example, motion compensation, transforms, quantization, and entropy coding and decoding. Some of these techniques are described below.

[0006] Historically, video encoders and decoders have tended to operate on a given picture size, which in most cases is defined and kept constant for a Coded Video Sequence (CVS), Group of Pictures (GOP), or similar multi-picture time frame. For example, in MPEG-2, the system design changes the horizontal resolution (and thus the picture size) based on factors such as scene activity, but only at I pictures, so this usually applies to GOPs. For example, from Annex P of the ITU-T Recommendation H.263, it is known that in order to use different resolutions in CVS, the reference picture can be resampled. However, because the picture size does not change here, only the reference picture is resampled, which may result in only a portion of the picture canvas being used (in the case of downsampling) or only a portion of the scene being captured (in the case of upsampling). In addition, Annex Q of H.263 allows resampling of a single macroblock up or down by a factor of two (in each dimension). Again, the picture size remains unchanged. The size of the macroblock is fixed in H.263 and therefore does not need to be signaled.

[0007] In modern video codecs, changing the picture size in the predicted picture has become more mainstream. For example, VP9 allows reference picture resampling and resolution change for the entire picture. Similarly, certain proposals for VVC (including, for example, Hendry et al., "Adaptive Resolution Change (ARC) for VVC", Joint Video Team document JVET-M0135-v1, January 9-19, 2019, which is incorporated herein in its entirety) allow the entire reference picture to be resampled to a different higher or lower resolution. In this document, it is proposed that different candidate resolutions be encoded in a sequence parameter set and referenced by a syntax element for each picture in the picture parameter set. Summary of the invention

[0008] Embodiments relate to methods, apparatus, computer systems, and readable media for encoding and decoding. According to one aspect, a method for encoding and decoding is provided. The method includes receiving video data including one or more sub-pictures. In a high-level syntax field, a number of the one or more sub-pictures and a delta value between the number of the one or more sub-pictures and the number of rectangular slices are signaled. Based on the number of the one or more sub-pictures and the delta value, the number of rectangular slices is derived.

[0009] According to another aspect, a computer system for encoding and decoding is provided. The computer system may include one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored in one or more storage devices, and the program instructions are executed by at least one processor through at least one of the one or more computer-readable memories, so that the computer system can perform a method. The method includes receiving video data including one or more sub-pictures. In a high-level syntax field, the number of the one or more sub-pictures and the incremental value between the number of the one or more sub-pictures and the number of rectangular slices are signaled. Based on the number of the one or more sub-pictures and the incremental value, the number of rectangular slices is derived.

[0010] According to another aspect, a computer-readable medium computer for encoding and decoding is provided. The computer-readable medium may include one or more computer-readable storage devices and program instructions stored on at least one of the one or more tangible storage devices, and the program instructions may be executed by a processor. The program instructions may be executed by a processor to perform a method. The method accordingly includes receiving video data including one or more sub-pictures. In a high-level syntax field, the number of the one or more sub-pictures and the incremental value between the number of the one or more sub-pictures and the number of rectangular slices are signaled. Based on the number of the one or more sub-pictures and the incremental value, the number of rectangular slices is derived.

[0011] According to another aspect, a coding and decoding device is provided. The device includes: a receiving device configured to cause the one or more computer processors to receive video data including one or more sub-pictures; a signaling device configured to cause the one or more computer processors to signal the number of the one or more sub-pictures and the delta value between the number of the one or more sub-pictures and the number of rectangular slices in a high-level syntax field; and a deriving device configured to cause the one or more computer processors to derive the number of the rectangular slices based on the number of the one or more sub-pictures and the delta value. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] These and other objects, features and advantages will become apparent from the following detailed description of illustrative embodiments read in conjunction with the accompanying drawings. Since the illustrations are intended to facilitate a clear understanding by those skilled in the art in conjunction with the detailed description, the various features of the drawings are not drawn to scale. In the drawings:

[0013] Figure 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment;

[0014] Figure 2is a schematic diagram of a simplified block diagram of a streaming media system according to an embodiment;

[0015] Figure 3 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment;

[0016] Figure 4 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment;

[0017] Figure 5 is a schematic diagram of options for signaling ARC parameters according to an embodiment;

[0018] Figure 6 is an example of a syntax table according to an embodiment;

[0019] Figure 7 is a schematic diagram of a computer system according to an embodiment;

[0020] Figure 8 is an example of a prediction structure for scalability with adaptive resolution change;

[0021] Fig. 9 is an example of a syntax table according to an embodiment;

[0022] Fig.10 is a schematic diagram of a simplified block diagram for parsing and decoding the poc period and access unit count value of each access unit;

[0023] Fig.11 is a schematic diagram of a video bitstream structure including multiple layers of sub-pictures;

[0024] Fig.12 is a schematic diagram of a display of a selected sub-picture with enhanced resolution;

[0025] Fig.13 is a block diagram of a decoding and display process of a video stream including multiple layers of sub-pictures;

[0026] Fig.14 is a schematic diagram of a 360-degree video display with an enhancement layer of a sub-picture;

[0027] Fig.15 is an example of layout information of a sub-picture and its corresponding layers and picture prediction structure;

[0028] Fig.16 is an example of layout information of a sub-picture having a spatial scalability morphology of a local region and its corresponding layer and picture prediction structure;

[0029] Fig.17 is an example of a syntax table for sub-picture layout information;

[0030] Fig.18is an example of a syntax table of a SEI message for sub-picture layout information;

[0031] Fig.19 is an example of a syntax table for indicating output layers and profile / tier / level information for each output layer set;

[0032] Fig. 20 is an example of a syntax table for indicating the output layer mode for each output layer set;

[0033] Fig.21 is an example of a syntax table for indicating a current sub-picture for each layer of each output layer set;

[0034] Fig. 22 is an example of a syntax table representing a sub-picture identifier;

[0035] Fig.23 is an example of an SPS syntax table indicating the number of sub-pictures;

[0036] Fig.24 is an example of a PPS syntax table indicating slice partitioning;

[0037] Fig.25 is an example of a PPS syntax table indicating the number of slices in a picture;

[0038] Fig.26 is another example of a PPS syntax table, using a modified syntax to indicate the number of slices in a picture. DETAILED DESCRIPTION

[0039] Detailed embodiments of the claimed structures and methods are disclosed herein; however, it is understood that the disclosed embodiments are merely illustrative of the claimed structures and methods that may be embodied in various forms. However, these structures and methods may be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that the present disclosure will be thorough and complete and will fully convey the scope to those skilled in the art. In the description, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

[0040] As previously mentioned, uncompressed digital video may consist of a series of pictures, each picture having a spatial dimension of, for example, 1920×1080 luma samples and associated chroma samples. The series of pictures may have a fixed or variable picture rate (also informally referred to as a frame rate), such as 60 pictures per second or 60Hz. Uncompressed video has significant bit rate requirements. For example, 1080p60 4:2:0 video (1920x1080 luma sample resolution, 60Hz frame rate) with 8 bits per sample requires a bandwidth close to 1.5Gbit / s. One hour of such video requires more than 600GB of storage space. However, a picture may be partitioned into one or more sub-pictures, and each sub-picture may be further partitioned into one or more slices. Therefore, as a more efficient signaling method, it may be advantageous to signal the slice partitioning of each sub-picture or picture.

[0041] Various aspects are described herein according to flowchart examples and / or block diagrams of methods, devices (systems) and computer-readable media of various embodiments. It should be understood that each frame of the flowchart example and / or block diagram, and the combination of frames in the flowchart example and / or block diagram can be implemented by computer-readable program instructions.

[0042] Figure 1 The present invention is a simplified block diagram of a communication system (100) according to an embodiment disclosed in the present application. The communication system (100) includes at least two terminal devices (110, 120), and the at least two terminal devices can be interconnected with each other through a network (150). For unidirectional data transmission, a first terminal device (110) can encode video data locally for transmission to another terminal device (120) through a network (150). A second terminal device (120) can receive encoded video data of another terminal device from the network (150), decode the encoded video data and display the recovered video data. Unidirectional data transmission is common in applications such as media services.

[0043] Figure 1 A second pair of terminal devices (130, 140) for supporting bidirectional transmission of encoded video is illustrated, and the bidirectional transmission may occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (130) and the fourth terminal device (140) may encode video data collected locally for transmission to the other of the third terminal device (130) and the fourth terminal device (140) via a network (150). Each of the third terminal device (130) and the fourth terminal device (140) may also receive the encoded video data transmitted by the other of the third terminal device (130) and the fourth terminal device (140), decode the encoded video data, and display the restored video data on a local display device.

[0044] exist Figure 1 In the embodiment of the present invention, the first terminal device (110), the second terminal device (120), the third terminal device (130) and the fourth terminal device (140) may be servers, personal computers and smart phones, but the principles disclosed in the present application may not be limited thereto. The embodiments disclosed in the present application are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. The network (150) represents any number of networks that transmit encoded video data between the first terminal device (110), the second terminal device (120), the third terminal device (130) and the fourth terminal device (140), including, for example, wired and / or wireless communication networks. The communication network (150) may exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purposes of the present application, unless explained below, the architecture and topology of the network (150) may be irrelevant to the operation disclosed in the present application.

[0045] As an embodiment of the application of the disclosed subject matter, Figure 2 The video encoder and video decoder are shown in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0046] The streaming system may include an acquisition subsystem (213), which may include a video source (201) such as a digital camera, which creates, for example, an uncompressed video sample stream (202). The video sample stream (202) is depicted as a thick line to emphasize the high data volume of the video sample stream compared to the encoded video stream, and the video sample stream (202) may be processed by a video encoder (203) coupled to the digital camera. The video encoder (203) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. The encoded video code stream (204) is depicted as a thin line to emphasize the lower data volume of the encoded video code stream (204) compared to the video sample stream (202), which may be stored on a streaming server (205) for future use. At least one streaming client (206, 208) may access a streaming server (205) to retrieve a copy (207) and a copy (209) of an encoded video stream (204). The client (206) may include a video decoder (210) that decodes the incoming copy (207) of the encoded video stream and produces an output video sample stream (211) that can be presented on a display (212) or other presentation device (not depicted). In some streaming systems, the video streams (204, 207, 209) may be encoded according to certain video encoding / compression standards. Examples of such standards include ITU-T Recommendation H.265. The video codec standard under development is informally referred to as Versatile Video Coding (VVC), and embodiments of the disclosed subject matter may be used in the context of the VVC standard.

[0047] Figure 3 is a block diagram of a video decoder (210) according to an embodiment disclosed in the present application.

[0048] The receiver (310) may receive at least one codec video sequence to be decoded by the video decoder (210). In the same or another embodiment, the receiver (301) may receive one coded video sequence at a time, wherein the decoding of each coded video sequence is independent of the other coded video sequences. The coded video sequence may be received from a channel (312), which may be a hardware / software link to a storage device storing the coded video data. The receiver (310) may receive the coded video data as well as other data, such as coded audio data and / or auxiliary data streams that may be forwarded to their respective consuming entities (not shown). The receiver (310) may separate the coded video sequence from the other data. To prevent network jitter, a buffer memory (315) may be coupled between the receiver (310) and the entropy decoder / parser (320) (hereinafter referred to as the "parser"). When the receiver (310) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, it may not be necessary to configure the buffer memory (315), or the buffer memory may be made smaller. For use on a best-effort packet network such as the Internet, the buffer memory (315) may also be required, and the buffer memory may be relatively large and may advantageously have an adaptive size.

[0049] The video decoder (210) may include a parser (320) to reconstruct symbols (321) from an entropy-encoded video sequence. The types of symbols include information used to manage the operation of the video decoder (210) and potential information for controlling a display device such as a display (212) that is not part of the video decoder but may be coupled to the video decoder, such as Figure 2As shown in . The control information for the display device may be a parameter set fragment (not indicated) of Supplementary Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (320) may parse / entropy decode the received coded video sequence. The encoding of the coded video sequence may be performed according to a video coding technique or standard, and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (320) may extract a subgroup parameter set of at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (Group of Pictures, GOP), a picture, a tile, a slice, a macroblock, a coding unit (Coding Unit, CU), a block, a transform unit (Transform Unit, TU), a prediction unit (Prediction Unit, PU), and the like. The entropy decoder / parser can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0050] The parser (320) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (315) to create symbols (321).

[0051] Depending on the type of coded video picture or portion of coded video picture (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors, the reconstruction of the symbol (321) may involve multiple different units. Which units are involved and how they are involved may be controlled by subgroup control information parsed by the parser (320) from the coded video sequence. For the sake of brevity, such subgroup control information flow between the parser (320) and the multiple units below is not described.

[0052] In addition to the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into several functional units as described below. In a practical embodiment operating under commercial constraints, many of these units interact closely with each other and can be integrated with each other. However, for the purpose of describing the disclosed subject matter, the conceptual subdivision into the following functional units is appropriate.

[0053] The first unit is a sealer / inverse transform unit (351). The sealer / inverse transform unit (351) receives quantized transform coefficients as symbols (321) from the parser (320) and control information, including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (351) can output a block including sample values, which can be input into an aggregator (355).

[0054] In some cases, the output samples of the scaler / inverse transform unit (351) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (352). In some cases, the intra-picture prediction unit (352) uses surrounding reconstructed information extracted from the current (partially reconstructed) picture memory (358) to generate a block of the same size and shape as the block being reconstructed. In some cases, the aggregator (355) adds the prediction information generated by the intra-picture prediction unit (352) to the output sample information provided by the scaler / inverse transform unit (351) on a per-sample basis.

[0055] In other cases, the output samples of the scaler / inverse transform unit (351) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (353) may access the reference picture memory (357) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (321), these samples may be added to the output of the scaler / inverse transform unit (351) (in this case referred to as residual samples or residual signals) by the aggregator (355) to generate output sample information. The acquisition of the predicted samples by the motion compensated prediction unit (353) from the address in the reference picture memory (357) may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (353) in the form of the symbols (321), for example, including X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory (357) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, and the like.

[0056] The output samples of the aggregator (355) may be used by various loop filtering techniques in a loop filter unit (356). The video compression techniques may include in-loop filter techniques controlled by parameters included in the encoded video bitstream and available to the loop filter unit (356) as symbols (321) from the parser (320), and may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, and to previously reconstructed and loop filtered sample values.

[0057] The output of the loop filter unit (356) may be a sample stream that may be output to the display (212) and stored in the current picture memory (358) for subsequent inter-picture prediction.

[0058] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. For example, once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (320)), the current picture memory (358) may become part of the reference picture memory (357), and a new current picture memory (358) may be reallocated before starting reconstruction of a subsequent coded picture.

[0059] The video decoder (210) may perform decoding operations according to a predetermined video compression technique as recorded, for example, in the ITU-T Recommendation H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence adheres to the syntax of the video compression technique or standard, as specified in the video compression technology document or standard, particularly the profiles therein. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, mega samples per second), maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata of the HRD buffer management signaled in the encoded video sequence.

[0060] In an embodiment, the receiver (310) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (320) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0061] Figure 4 is a block diagram of a video encoder (203) according to an embodiment disclosed in the present application.

[0062] The video encoder (203) may receive video samples from a video source (201) (not part of the encoder), which may capture video images to be encoded by the video encoder (203).

[0063] The video source (201) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (203), wherein the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (201) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (201) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that are given motion when viewed sequentially. The pictures themselves may be constructed as a spatial pixel array, wherein each pixel may include at least one sample depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.

[0064] According to an embodiment, the video encoder (203) may encode and compress pictures of a source video sequence into an encoded video sequence in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (450). The controller (450) controls other functional units as described below and is functionally coupled to these units. For the sake of brevity, coupling is not indicated in the figure. The parameters set by the controller (450) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology, etc.), picture size, GOP layout, maximum motion vector search range, etc. Those skilled in the art can easily identify other functions of the controller (450) that are related to the video encoder (203) optimized for a certain system design.

[0065] Some video decoders operate in a manner that is recognized by those skilled in the art as a "coding loop". As a simple description, the coding loop may include an encoder (430) (hereinafter referred to as the "source encoder"), which is responsible for the coding portion of creating symbols based on the input picture to be encoded and the reference picture, and a (local) decoder (433) embedded in the video encoder (203). The decoder (433) reconstructs the symbols to create sample data that the (remote) decoder also creates (because in the video compression techniques considered in the presently disclosed subject matter, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream is input to a reference picture memory (434). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the buffered contents of the reference picture memory (434) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction portion of the encoder are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. Those skilled in the art are familiar with the basic principles of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, such as due to channel errors).

[0066] The operation of the "local" decoder (433) can be combined with the above Figure 3 The "remote" decoder (210) is identical to the one described in detail. However, additional brief reference is made to Figure 3 , when symbols are available and the entropy encoder (445) and the parser (320) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (210) including the channel (312), the receiver (310), the buffer memory (315) and the parser (320) may not be fully implemented in the local decoder (433).

[0067] At this point, it can be observed that any decoder technology other than the parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the present application focuses on decoder operation. The description of encoder technology can be simplified because encoder technology is mutually inverse to the decoder technology described comprehensively. A more detailed description is only needed in certain areas and is provided below.

[0068] As part of its operation, the source encoder (430) may perform motion compensated predictive coding. The motion compensated predictive coding predictively encodes an input frame with reference to at least one previously encoded frame from a video sequence designated as a "reference frame." In this manner, the encoding engine (432) encodes the differences between pixel blocks of an input frame and pixel blocks of a reference frame that may be selected as a prediction reference for the input frame.

[0069] The local video decoder (433) may decode the encoded video data of the frame that may be designated as the reference frame based on the symbol created by the source encoder (430). The operation of the encoding engine (432) may be a lossy process. When the encoded video data is available at the video decoder ( Figure 4 When the video sequence is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (433) replicates the decoding process that may be performed by the video decoder on the reference frame and may cause the reconstructed reference frame to be stored in the reference picture memory (434). In this way, the video encoder (203) may store a copy of the reconstructed reference frame locally that has common content (absent transmission errors) with the reconstructed reference frame to be obtained by the remote video decoder.

[0070] The predictor (435) may perform a prediction search for the encoding engine (432). That is, for a new frame to be encoded, the predictor (435) may search the reference picture memory (434) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new frame. The predictor (435) may operate pixel-by-pixel based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (435), it may be determined that the input picture may have prediction references taken from a plurality of reference pictures stored in the reference picture memory (434).

[0071] The controller (450) may manage encoding operations of the source encoder (430), including, for example, setting parameters and subgroup parameters for encoding video data.

[0072] The outputs of all the above functional units may be entropy encoded in an entropy encoder (445). The entropy encoder (445) performs lossless compression on the symbols generated by the various functional units according to techniques well known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0073] The transmitter (440) may buffer the encoded video sequence created by the entropy encoder (445) in preparation for transmission over a communication channel (460), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (440) may combine the encoded video data from the source encoder (430) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0074] The controller (450) may manage the operation of the video encoder (203). During encoding, the controller (450) may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following frame types:

[0075] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0076] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0077] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0078] The source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or the blocks may be predictively coded (spatial prediction or intra prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0079] The video encoder (203) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (203) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0080] In an embodiment, the transmitter (440) may transmit additional data when transmitting the encoded video. The source encoder (430) may include such data as part of the encoded video sequence. The additional data may include time / space / SNR enhancement layers, redundant pictures and slices and other forms of redundant data, SEI messages, VUI parameter set fragments, etc. Before describing certain aspects of the subject matter disclosed in this application in more detail, it is necessary to introduce several terms that will be referred to in the rest of this specification.

[0081] "Sub-picture", hereinafter, refers in some cases to a semantically grouped rectangular arrangement of samples, blocks, macroblocks, coding units, or similar entities that can be independently encoded with varying resolutions. A picture can have one or more sub-pictures. One or more coded sub-pictures can form a coded picture. One or more sub-pictures can be assembled into a picture, and one or more sub-pictures can be extracted from a picture. In some circumstances, one or more coded sub-pictures can be assembled into a coded picture in the compressed domain without transcoding them to the sample level, and in the same case or in some other cases, one or more coded sub-pictures can be extracted from a coded picture in the compressed domain.

[0082] "Adaptive Resolution Change (ARC)" refers to a mechanism that allows the resolution of a picture or sub-picture within a coded video sequence to be changed, for example, by resampling a reference picture. "ARC parameters" refers to the control information required to perform adaptive resolution change, which may include, for example, filter parameters, scaling factors, resolution of output pictures and / or reference pictures, various control flags, etc.

[0083] The above description focuses on encoding and decoding of a single, semantically independent coded video picture.Before describing the implications of encoding / decoding of multiple sub-pictures with independent ARC parameters and the additional complexity it implies, the options for signaling ARC parameters will be described.

[0084] refer to Figure 5 , several new options for signaling ARC parameters are shown. As indicated by each option, they have certain advantages and disadvantages from the perspective of coding efficiency, complexity, and architecture. Video coding standards or technologies may select one or more of these options, or select options known in the prior art, for signaling ARC parameters. These options may not be mutually exclusive and may be interchanged based on application needs, the standard technologies involved, or the selection of encoders.

[0085] Categories of ARC parameters can include:

[0086] - Upsampling factors and / or downsampling factors in the X and Y dimensions, either separately or combined.

[0087] - An upsampling factor and / or downsampling factor in the time dimension is added to indicate that a given number of pictures are enlarged and / or reduced at a constant speed.

[0088] - Either of the above may involve encoding one or more possibly shorter syntax elements, which may point to a table containing the factor(s).

[0089] - Resolution: The resolution of the input picture, output picture, reference picture, coded picture, combined or individual, in the X or Y dimension, in units of samples, blocks, macroblocks, CUs, or any other suitable granularity. If more than one resolution is present (e.g., one for the input picture and one for the reference picture), then in some cases one set of values ​​can be inferred from another. For example, the resolution can be gated using a flag. See below for a more detailed example.

[0090] - "Warping" coordinates: similar to the coordinates used in Annex P of the H.263 standard, which can have an appropriate granularity as described above. Annex P of the H.263 standard defines an efficient way to encode such warping coordinates, but it is conceivable that other, potentially more efficient methods can also be envisioned. For example, the variable-length reversible "Huffman" encoding of the warping coordinates of Annex P can be replaced by a binary encoding of an appropriate length, where the length of the binary codeword can be derived, for example, from the maximum picture size (possibly multiplied by a factor and offset by a value) to allow "warping" outside the boundaries of the maximum picture size.

[0091] - Upsampling filter parameters or downsampling filter parameters: In the simplest case, there may be only a single filter used to perform upsampling and / or downsampling. However, in some cases it may be advantageous to allow greater flexibility in the design of the filter, which may need to be achieved by signaling filter parameters. Such parameters may be selected by an index into a list of possible filter designs, the filter may be fully specified (e.g., by a list of filter coefficients using an appropriate entropy coding technique), the filter may be selected implicitly by the upsampling ratio and / or downsampling ratio, which in turn may be signaled according to any of the mechanisms mentioned above, etc.

[0092] In the following, the present specification assumes that a finite set of upsampling factors and / or downsampling factors (using the same factors in the X and Y dimensions) are encoded by codeword indication. The codewords can be advantageously variable length encoded, for example, using Exp-Golomb coding common to certain syntax elements in video coding specifications (such as H.264 and H.265). A suitable mapping of values ​​to upsampling factors / downsampling factors can be, for example, according to the following table:

[0093] Codeword Exp-Golomb encoding Original resolution / target resolution 0 1 1 / 1 1 010 1 / 1.5 (50% zoom) 2 011 1.5 / 1 (50% smaller) 3 00100 1 / 2 (100% zoom) 4 00101 2 / 1 (reduced by 100%)

[0094] Many similar mappings between the values ​​of codewords and upsampling factors and / or downsampling factors can be designed according to the needs of the application and the capabilities of the magnification and reduction mechanisms available in the video compression technology or standard. The table corresponding to the mapping can be extended to more values. The values ​​of the codewords can also be represented by entropy coding mechanisms other than Exp-Golomb codes, for example using binary encoding. When the resampling factors are of interest outside the video processing engine (most importantly the encoder and decoder) itself, such as the MANE (Media Aware Network Element), the use of binary encoding may have certain advantages. It should be noted that for the (probably) most common case where no resolution change is required, a shorter Exp-Golomb code can be selected; in the above table, the Exp-Golomb code is, for example, only a single bit. For this most common case, the Exp-Golomb code can have the advantage of coding efficiency over the use of binary codes.

[0095] The number of entries in the above tables and their semantics may be fully or partially configurable. For example, the basic form of the above tables may be conveyed in a "higher layer" parameter set such as a sequence parameter set or a decoder parameter set. Alternatively or additionally, one or more such tables may be defined in a video codec or standard and may be selected by, for example, a decoder or sequence parameter set.

[0096] The following describes how to include the upsampling factor / downsampling factor (ARC information) encoded as described above in a video codec or standard syntax. Similar considerations can be applied to one or more codewords that control the upsampling filter / downsampling filter. See the discussion below for when a filter or other data structure requires a relatively large amount of data.

[0097] Annex P of the H.263 standard includes ARC information 502 in the form of four deformation coordinates in the picture header 501, more specifically, in the H.263 standard enhanced P type (PLUSPTYPE) (503) header extension. This may be a wise design choice when a) there is a picture header available, and b) the ARC information is expected to change frequently. However, the overhead of using H.263-style signaling may be quite high, and the scaling factor may not apply to picture boundaries because picture headers may have a transient nature.

[0098] The JVCET-M135-v1 mentioned above includes ARC reference information (505) (index) located in a picture parameter set (504) for indexing a table (506) including target resolutions, and the table (506) is located in a sequence parameter set (507). According to the oral statement made by the author, the location of the possible resolutions in the table (506) in the sequence parameter set (SPS) (507) can be justified by using the SPS as an interoperability negotiation point during capability exchange. Within the limits set by the values ​​in the table (506), the resolution can be changed from picture to picture by referencing the appropriate picture parameter set (PPS) (504).

[0099] Still refer to Figure 5 , there may be the following additional options to transmit ARC information in the video bitstream. Each of these options has certain advantages over the above prior art. These options may exist simultaneously in the same video coding technology or standard.

[0100] In one embodiment, ARC information (509) such as a resampling (scaling) factor may appear in a slice header, a GOB (Group of Blocks) header, a tile header, or a tile group header (hereinafter referred to as a tile group header) (508). It may be sufficient that the ARC information is small, for example, the ARC information is a single variable length ue(v) codeword of a few bits or a fixed length codeword, for example, as shown above. The ARC information may be applied to, for example, a sub-picture represented by a tile group, rather than to the entire picture, and including the ARC information directly in the tile group header has additional advantages. See also below. In addition, even if the video compression technology or standard only envisions adaptive resolution changes for the entire picture (as opposed to, for example, adaptive resolution changes based on tile groups), from a fault tolerance perspective, placing the ARC information in the tile group header has certain advantages over placing it in the H.263-type picture header.

[0101] In the same or another embodiment, the ARC information (512) itself may be present in an appropriate parameter set (511), such as a picture parameter set, a header parameter set, a tile parameter set, an Adaptation Parameter Set (APS), etc. (Adaptation parameter sets are described herein). The scope of the parameter set may advantageously be no larger than one picture, such as a tile group. The use of the ARC information (512) may be implicit by activating the relevant parameter set. For example, when a video coding technique or standard only considers picture-based ARC, a picture parameter set or an equivalent parameter set may be applicable.

[0102] In the same or another embodiment, ARC reference information (513) may be present in a tile group header (514) or similar data structure. The ARC reference information (513) may refer to a subset of the ARC information (515) available in a parameter set (516) (e.g., a sequence parameter set or a decoder parameter set) that extends beyond a single picture.

[0103] It seems unnecessary to indirectly imply activation of an additional level of PPS from the tile group header, PPS, SPS, as used in JVET-M0135-v1, since picture parameter sets can be used for capability negotiation or notification (as already used in some standards such as RFC3984) in the same way as sequence parameter sets. However, if the ARC information should apply, for example, to sub-pictures that are also represented by tile groups, parameter sets whose activation scope is limited to tile groups (e.g., adaptation parameter sets or header parameter sets) may be a better choice. Moreover, if the ARC information has more than a negligible size - for example, containing filter control information such as multiple filter coefficients - then from the perspective of coding efficiency, choosing to use parameter sets may be a better choice than choosing to use the tile group header (508) directly, since those settings can be reused by future pictures or sub-pictures by referencing the same parameter set.

[0104] When using a sequence parameter set or another higher layer parameter set that spans multiple pictures, the following points may need to be considered:

[0105] 1. The parameter set for storing the ARC information table may be a sequence parameter set (516) in some cases, but may advantageously be a decoder parameter set in other cases. The decoder parameter set may have an activation range of multiple CVS (Coded Video Sequences) (i.e., coded video streams), i.e., all coded video bits during the session start until the end of the session. Such a range may be more appropriate because possible ARC factors may be decoder features that may be implemented in hardware, and hardware features tend not to change with any CVS (which in at least some entertainment systems is a group of pictures with a length of one second or less). That is, placing the table in a sequence parameter set is explicitly included in the placement options described in this application, especially in conjunction with point 2 below.

[0106] 2. The ARC reference information (513) can be advantageously placed directly in the picture / slice / tile / GOB / tile group header (hereinafter referred to as the tile group header) (514) instead of being placed in the picture parameter set as in JVCET-M0135-v1. The reason is as follows: When the encoder wants to change a single value in the picture parameter set, such as the ARC reference information, the encoder may have to create a new PPS and reference the new PPS. Assume that only the ARC reference information changes, but other information such as the quantization matrix information in the PPS is retained. Such information may be large and may need to be retransmitted to make the new PPS complete. Since the ARC reference information may be a single codeword, such as an index in the ARC reference information (513), and this will be the only value that changes, it will be very cumbersome and wasteful to retransmit all of the quantization matrix information, for example. Therefore, from the perspective of coding efficiency, it is better to avoid indirect access through the PPS as proposed in JVET-M0135-v1. Similarly, placing the ARC reference information in the PPS has the additional disadvantage that the ARC information referenced by the ARC reference information (513) must be applied to the entire picture rather than a sub-picture, since the scope of picture parameter set activation is a picture.

[0107] In the same or another embodiment, the signaling of ARC parameters may follow Figure 6 A detailed example is outlined in . Figure 6 A syntax diagram depicting a representation used in video coding standards since at least 1993. The notation for such syntax diagrams roughly follows C-style programming. Bold lines indicate syntax elements present in the codestream, and non-bold lines generally indicate control flow or the setting of variables.

[0108] The tile group header (601), an exemplary syntax structure for a header applicable to a (possibly rectangular) portion of a picture, may conditionally contain a variable length Exp-Golomb coded syntax element dec_pic_size_idx (602) (shown in bold). The presence of this syntax element in the tile group header may be gated by using adaptive resolution (603). Here, the value of the flag is not shown in bold, which means that the point at which the flag appears in the codestream is when it appears in the syntax table. Whether adaptive resolution is used for the picture or a portion thereof may be signaled in any high-level syntax structure, either internal or external to the codestream. In the example shown, the adaptive resolution is signaled in the sequence parameter set as described below.

[0109] Still reference Figure 6 , an excerpt of a sequence parameter set (610) is also shown. The first syntax element shown is adaptive_pic_resolution_change_flag (611). When true, this flag may indicate that adaptive resolution is used, which may require specific control information. In this example, such control information is conditionally present based on the value of the flag, which is based on an if() statement in the sequence parameter set (612) and the tile group header (601).

[0110] When adaptive resolution is used, in this example, the output resolution is encoded in units of samples (613). The number 613 refers to output_pic_width_in_luma_samples and output_pic_height_in_luma_samples, which together can define the resolution of the output picture. Elsewhere in the video coding technology or standard, certain restrictions on any value may be defined. For example, a level definition may limit the number of total output samples, which may be the product of the values ​​of the two syntax elements described above. In addition, certain video coding technologies or standards, or external technologies or standards (e.g., system standards) may limit the range of values ​​(e.g., one or both dimensions must be divisible by a power of 2) or aspect ratio (e.g., width and height must have a relationship such as 4:3 or 16:9). Such restrictions may be introduced to facilitate hardware implementation or for other reasons and are well known in the art.

[0111] In some applications, it is recommended that the encoder instruct the decoder to use a certain reference picture size, rather than implicitly assuming it to be the output picture size. In this example, the syntax element reference_pic_size_present_flag (614) enables the conditional presence of the reference picture size (615) (again, this number refers to both width and height).

[0112] Finally, a table of possible decoded picture widths and heights is shown. This table may be represented, for example, by a table indication (num_dec_pic_size_in_luma_samples_minus1) (616). "Minus1" (minus 1) may refer to the interpretation of the value of this syntax element. For example, if the coded value is zero, there is one table entry. If the value is five, there are six table entries. For each "row" in the table, the decoded picture width and height is then contained in the syntax (617).

[0113] The presented table entry (617) can be indexed using the syntax element dec_pic_size_idx (602) in the tile group header, allowing each tile group to have a different decode size (effectively a scaling factor).

[0114] Some video coding techniques or standards, such as VP9, ​​support spatial scalability by implementing some form of reference picture resampling (which is signaled in a very different way than disclosed in this application) in conjunction with temporal scalability to achieve spatial scalability. More specifically, certain reference pictures can be upsampled to a higher resolution using ARC-type techniques to form the basis of a spatial enhancement layer. These upsampled pictures can be refined using standard prediction mechanisms at a higher resolution to increase detail.

[0115] The embodiments discussed in this application can be used in such environments. In some cases, in the same or another embodiment, the value in the Network Abstract Layer (NAL) unit header (e.g., the Temporal ID field) can be used to indicate not only the temporal layer, but also the spatial layer. Doing so may have certain advantages for certain system designs; for example, an existing Selected Forwarding Unit (SFU) created and optimized for forwarding selected for the temporal layer based on the Temporal ID value in the NAL unit header can be used in a scalable environment without modification. To achieve this, it may be necessary to map between the coded picture size and the temporal layer indicated by the Temporal ID field in the NAL unit header.

[0116] In some video coding techniques, an access unit (AU) may refer to one or more coded pictures, one or more slices, one or more tiles, one or more NAL units, etc., which are captured and combined into corresponding pictures, slices, tiles and / or NAL units at a given time instance. The time instance may be, for example, a composition time.

[0117] In HEVC and certain other video codecs, a picture order count (POC) value can be used to indicate a reference picture selected from a plurality of reference pictures stored in a decoded picture buffer (DPB). When an access unit (AU) includes one or more pictures, slices, or tiles, each picture, slice, or tile belonging to the same AU can carry the same POC value, from which it can be derived that they are created from content at the same composition time. In other words, in a scenario where two pictures / slices / tiles carry the same given POC value, the same given POC value can indicate that the two pictures / slices / tiles belong to the same AU and have the same composition time. Conversely, two pictures / tiles / slices with different POC values ​​can indicate that those pictures / slices / tiles belong to different AUs and have different composition times.

[0118] In an embodiment, this strict relationship can be relaxed because an access unit may include pictures, slices or tiles with different POC values. By allowing different POC values ​​within an AU, the POC value can be used to identify potentially independently decodable pictures / slices / tiles with the same presentation time. This in turn enables support for multiple scalable layers without changing the reference picture selection signaling (e.g., reference picture set signaling or reference picture list signaling), as described in more detail below.

[0119] However, it is still desirable to be able to identify the AU to which a picture / slice / tile belongs, relative to other pictures / slices / tiles having different POC values, simply from the POC value. This can be achieved as described below.

[0120] In an embodiment, an access unit count (AUC) is signaled in a high-level syntax structure such as a NAL unit header, a slice header, a tile group header, a SEI message, a parameter set, or an AU delimiter. The value of the AUC can be used to identify which NAL units, pictures, slices, or tiles belong to a given AU. The value of the AUC can correspond to different synthesis time instances. The AUC value can be equal to a multiple of the POC value. The AUC value can be calculated by dividing the POC value by an integer value. In some cases, the division operation may impose a certain burden on the decoder implementation. In this case, a small restriction in the numbering space of the AUC value can allow a shift operation to be used instead of a division operation. For example, the AUC value can be equal to the most significant bit (MSB) value of the POC value range.

[0121] In an embodiment, the value of the POC cycle (poc_cycle_au) for each AU is signaled in a high-level syntax structure (such as a NAL unit header, a slice header, a tile group header, a SEI message, a parameter set, or an AU delimiter). poc_cycle_au may indicate how many different and consecutive POC values ​​may be associated with the same AU. For example, if the value of poc_cycle_au is equal to 4, pictures, slices, or tiles with POC values ​​equal to 0 to 3 (inclusive) are associated with AUs with AUC values ​​equal to 0, and pictures, slices, or tiles with POC values ​​equal to 4 to 7 (inclusive) are associated with AUs with AUC values ​​equal to 1. Therefore, the value of AUC may be inferred by dividing the POC value by the value of poc_cycle_au.

[0122] In an embodiment, the value of poc_cycle_au may be derived from information, such as that located in a video parameter set (VPS), identifying the number of spatial or SNR layers in a coded video sequence. An example of such a possible relationship is briefly described below. Although the derivation as described above may save some bits in the VPS and may therefore improve codec efficiency, in some embodiments, poc_cycle_au may be explicitly encoded in an appropriate high-level syntax structure at a level lower than the video parameter set so that poc_cycle_au may be minimized for a given small portion of a codestream such as a picture. Because POC values ​​and / or values ​​of syntax elements that indirectly reference the POC may be encoded in a low-level syntax structure, this optimization may save more bits than may be saved by the derivation process described above.

[0123] The techniques for signaling adaptive resolution parameters described above may be implemented as computer software using computer-readable instructions and physically stored in at least one computer-readable storage medium. Figure 7 A computer system 700 suitable for implementing certain embodiments of the presently disclosed subject matter is shown.

[0124] The computer software may be encoded using any suitable machine code or computer language, which may be subjected to assembly, compilation, linking or similar mechanisms to create code comprising instructions that may be executed directly or through interpretation, microcode execution, etc., by a computer central processing unit (CPU), graphics processing unit (GPU), etc.

[0125] The instructions may be executed on various types of computers or computer components, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0126] Figure 7 The components for computer system 700 shown in the figure are exemplary in nature and are not intended to imply any limitation on the scope of use or functionality of computer software implementing embodiments of the present application. Nor should the configuration of components be interpreted as having any dependency or requirement on any one or combination of components shown in the exemplary embodiment of computer system 700.

[0127] Computer system 700 may include certain human-machine interface input devices. Such human-machine interface input devices may be responsive to input from at least one human user, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, taps), visual input (e.g., gestures), olfactory input (not depicted). Human-machine interface devices may also be used to capture certain media that may not be directly related to a person's conscious input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0128] Input human interface devices may include at least one of the following (only one of each is depicted): keyboard 701 , mouse 702 , track pad 703 , touch screen 710 , data gloves ( 704 ), joystick 705 , microphone 706 , scanner 707 , camera 708 .

[0129] The computer system 700 may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate at least one sense of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen 710, a data glove (704), or a joystick 705, but there may also be tactile feedback devices that do not serve as input devices), audio output devices (e.g., speakers 709, headphones (not depicted)), visual output devices (e.g., screens 710, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities, some of which are capable of outputting two-dimensional visual outputs or outputs greater than three dimensions through, for example, stereographic output; virtual reality glasses (not depicted), holographic displays, and smoke boxes (not depicted)), and printers (not depicted).

[0130] The computer system 700 may also include human-accessible storage devices and associated media for the storage devices, such as optical media, including CD / DVD ROM / RW720 with CD / DVD and other media 721, thumb drive 722, removable hard disk drive or solid state drive 723, traditional magnetic media such as tapes and floppy disks (not depicted), ROM / ASIC / PLD-based special devices, such as security protection devices (not depicted), and the like.

[0131] Those skilled in the art will also appreciate that the term "computer-readable media" used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transient signals.

[0132] The computer system 700 may also include an interface to at least one communication network. The network may be, for example, wireless, wired, optical. The network may also be local, wide area, metropolitan, vehicle-mounted and industrial, real-time, delay-tolerant, and the like. Examples of the network include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicle-mounted networks and industrial networks including CAN buses, and the like. Some networks typically require external network interface adapters attached to some universal data ports or peripheral buses 749 (e.g., USB ports of the computer system 700); other networks are typically integrated into the core of the computer system 700 by attaching to the system bus as described below (e.g., integrated into a PC computer system via an Ethernet interface, or integrated into a smart phone computer system via a cellular network interface). By using any of these networks, the computer system 700 can communicate with other entities. Such communications may be one-way receive only (e.g. broadcast TV), one-way send only (e.g. a CAN bus connected to certain CAN bus devices), or bidirectional, such as connecting to other computer systems using a local area digital network or a wide area digital network. Certain protocols and protocol stacks may be used on each of the networks and network interfaces such as those described above.

[0133] The above-mentioned human-machine interface device, human-accessible storage device, and network interface may be attached to the core 740 of the computer system 700 .

[0134] The core 740 may include at least one central processing unit (CPU) 741, a graphics processing unit (GPU) 742, a dedicated programmable processing unit in the form of a field programmable gate area (FPGA) 743, a hardware accelerator for certain tasks 744, etc. These devices, along with a read-only memory (ROM) 745, a random access memory 746, and an internal mass storage device 747 such as an internal non-user accessible hard drive, SSD, etc., may be connected via a system bus 748. In some computer systems, the system bus 748 may be accessible via at least one physical plug form to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached to the core's system bus 748 directly or via a peripheral bus 749. Architectures for peripheral buses include PCI, USB, etc.

[0135] The CPU 741, GPU 742, FPGA 743, and accelerator 744 may execute certain instructions, which in combination may constitute the above-mentioned computer code. The computer code may be stored in ROM 745 or RAM 746. Transient data may also be stored in RAM 746, while permanent data may be stored, for example, in an internal mass storage device 747. Fast storage and retrieval of any memory device may be achieved by using a cache memory, which may be closely associated with at least one CPU 741, GPU 742, mass storage device 747, ROM 745, RAM 746, etc.

[0136] The computer readable medium may have computer codes for performing various computer-implemented operations. The media and computer codes may be media and computer codes designed and constructed specifically for the purpose of this application, or may be of a type well known and available to technicians in the computer software field.

[0137] By way of example and not limitation, a computer system having architecture 700 and in particular core 740 may provide functionality resulting from the execution of software embodied in at least one tangible computer-readable medium by a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such computer-readable media may be media associated with the user-accessible mass storage device introduced above and certain storage devices of the non-volatile nature of core 740 (e.g., core internal mass storage device 747 or ROM 745). Software implementing various embodiments of the present application may be stored in such devices and executed by core 740. Depending on specific needs, the computer-readable medium may include at least one memory device or chip. The software may enable core 740, and in particular, the processor therein (including CPU, GPU, FPGA, etc.) to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM 746 and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality produced by logic hardwired or otherwise embodied in circuits (e.g., accelerator 744) that may operate in place of or in conjunction with software to perform specific processes or specific portions of specific processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuits (e.g., integrated circuits (ICs)) storing software for execution, circuits embodying logic for execution, or both, where appropriate. The present application encompasses any suitable combination of hardware and software.

[0138] Figure 8 An example of a video sequence structure with a combination of temporal_id, layer_id, POC and AUC values ​​with adaptive resolution change is shown. In this example, a picture, slice or tile in the first AU with AUC=0 may have temporal_id=0 and layer_id=0 or 1, while a picture, slice or tile in the second AU with AUC=1 may have temporal_id=1 and layer_id=0 or 1, respectively. Regardless of the values ​​of temporal_id and layer_id, the value of POC for each picture increases by 1. In this example, the value of poc_cycle_au may be equal to 2. Preferably, the value of poc_cycle_au may be set equal to the number of (spatial scalability) layers. Therefore, in this example, the value of POC increases by 2, while the value of AUC increases by 1.

[0139] In the above embodiments, all or a subset of inter-picture or inter-layer prediction structures and reference picture indications can be supported by using the existing reference picture set (RPS) signaling or reference picture list (RPL) signaling in HEVC. In the RPS or RPL, the selected reference picture is indicated by signaling the value of the POC between the current picture and the selected reference picture or the incremental value of the POC. For the disclosed subject matter, RPS and RPL can be used to indicate the inter-picture or inter-layer prediction structure without changing the signaling, but with the following limitations. If the value of the temporal_id of the reference picture is greater than the value of the temporal_id of the current picture, the current picture may not use the reference picture for motion compensation or other prediction. If the value of the layer_id of the reference picture is greater than the value of the layer_id of the current picture, the current picture may not use the reference picture for motion compensation or other prediction.

[0140] In the same and other embodiments, motion vector scaling for temporal motion vector prediction based on POC differences may be disabled across multiple pictures within an access unit. Thus, although each picture may have a different POC value within an access unit, the motion vectors are not scaled and used for temporal motion vector prediction within an access unit. This is because reference pictures with different POCs in the same AU may be considered reference pictures with the same temporal instance. Thus, in this embodiment, the motion vector scaling function may return 1 when the reference picture belongs to the AU associated with the current picture.

[0141] In the same and other embodiments, when the spatial resolution of the reference picture is different from the spatial resolution of the current picture, motion vector scaling based on POC difference for temporal motion vector prediction can be optionally disabled across multiple pictures. When motion vector scaling is enabled, the motion vector is scaled based on the ratio of the POC difference and the spatial resolution between the current picture and the reference picture.

[0142] In the same or another embodiment, for temporal motion vector prediction, especially when poc_cycle_au has non-uniform values ​​(when vps_contant_poc_cycle_per_au==0), the motion vector may be scaled based on AUC difference instead of POC difference. Otherwise (when vps_contant_poc_cycle_per_au==1), the motion vector scaling based on AUC difference may be the same as the motion vector scaling based on POC difference.

[0143] In the same or another embodiment, when the motion vector is scaled based on the AUC difference, the reference motion vector in the same AU as the current picture (having the same AUC value) is not scaled based on the AUC difference, and is not scaled or is scaled based on the ratio of the spatial resolution between the current picture and the reference picture for motion vector prediction.

[0144] In the same and other embodiments, the AUC value is used to identify the boundaries of the AU and for the operation of the hypothetical reference decoder (HRD), which requires input timing and output timing with AU granularity. In most cases, the decoded picture with the highest layer in the AU can be output for display. The AUC value and the layer_id value can be used to identify the output picture.

[0145] In an embodiment, a picture may be composed of one or more sub-pictures. Each sub-picture may cover a partial area or the entire area of ​​the picture. The area supported by a sub-picture may overlap or not overlap with the area supported by another sub-picture. The area composed of one or more sub-pictures may cover or not cover the entire area of ​​the picture. If a picture is composed of one sub-picture, the area supported by the sub-picture is the same as the area supported by the picture.

[0146] In the same embodiment, a sub-picture may be encoded by a codec similar to that used for an encoded picture. A sub-picture may be encoded independently or may be encoded based on another sub-picture or an encoded picture. A sub-picture may or may not have any parsing dependency on another sub-picture or an encoded picture.

[0147] In the same embodiment, the coded sub-pictures may be included in one or more layers. The coded sub-pictures in a layer may have different spatial resolutions. The original sub-pictures may be spatially resampled (upsampled or downsampled), encoded with different spatial resolution parameters, and included in the codestream corresponding to the layer.

[0148] In the same or another embodiment, a sub-picture with (W, H) may be encoded and included in the encoded bitstream corresponding to layer 0, where W indicates the width of the sub-picture and H indicates the height of the sub-picture. w,k ,H*S h,k ), the upsampled (or downsampled) sub-picture can be encoded and included in the encoded bitstream corresponding to layer k, where S w,k , S h,k Indicates the horizontal resampling rate and the vertical resampling rate. If S w,k , S h,k If the value of is greater than 1, resampling is equal to upsampling. However, if S w,k , Sh,k If the value of is less than 1, resampling is equal to downsampling.

[0149] In the same or another embodiment, an encoded sub-picture in a layer may have a different visual quality than an encoded sub-picture in another layer in the same sub-picture or in a different sub-picture. i,n is encoded, and the sub-image j in layer m is quantized with parameter Q j,m to encode.

[0150] In the same or another embodiment, a coded sub-picture in a layer may be independently decodable without any parsing dependency or decoding dependency on a coded sub-picture in another layer of the same local region. A sub-picture layer that is independently decodable without reference to another sub-picture layer of the same local region is an independent sub-picture layer. A coded sub-picture in an independent sub-picture layer may or may not have a decoding dependency or parsing dependency on a previously coded sub-picture in the same sub-picture layer, but the coded sub-picture may not have any dependency on a coded picture from another sub-picture layer.

[0151] In the same or another embodiment, a coded sub-picture in a layer may be dependently decodable, with any parsing dependency or decoding dependency on a coded sub-picture in another layer from the same local region. A sub-picture layer that is dependently decoded by reference to another sub-picture layer of the same local region is a dependent sub-picture layer. A coded sub-picture in a dependent sub-picture may reference a coded sub-picture belonging to the same sub-picture, a previously coded sub-picture in the same sub-picture layer, or both reference sub-pictures.

[0152] In the same or another embodiment, a coded sub-picture consists of one or more independent sub-picture layers and one or more dependent sub-picture layers. However, for a coded sub-picture, there may be at least one independent sub-picture layer. An independent sub-picture layer may have a layer identifier (layer_id) value equal to 0, which may be present in a NAL unit header or another high-level syntax structure. A sub-picture layer with layer_id equal to 0 is a base sub-picture layer.

[0153] In the same or another embodiment, a picture may consist of one or more foreground sub-pictures and one background sub-picture. The area supported by the background sub-picture may be equal to the area of ​​the picture. The area supported by the foreground sub-picture may overlap with the area supported by the background sub-picture. The background sub-picture may be a base sub-picture layer and the foreground sub-picture may be a non-base (enhancement) sub-picture layer. One or more non-base sub-picture layers may be decoded with reference to the same base layer. Each non-base sub-picture layer with layer_id equal to a may reference a non-base sub-picture layer with layer_id equal to b, where a is greater than b.

[0154] In the same or another embodiment, a picture may consist of one or more foreground sub-pictures with or without background sub-pictures. Each sub-picture may have its own base sub-picture layer and one or more non-base (enhancement) layers. Each base sub-picture layer may be referenced by one or more non-base sub-picture layers. Each non-base sub-picture layer with layer_id equal to a may reference a non-base sub-picture layer with layer_id equal to b, where a is greater than b.

[0155] In the same or another embodiment, a picture may consist of one or more foreground sub-pictures with or without background sub-pictures. Each coded sub-picture in a (base or non-base) sub-picture layer may be referenced by one or more non-base layer sub-pictures belonging to the same sub-picture, and one or more non-base layer sub-pictures not belonging to the same sub-picture.

[0156] In the same or another embodiment, a picture may consist of one or more foreground sub-pictures with or without background sub-pictures. A sub-picture in layer a may be further partitioned into multiple sub-pictures in the same layer. One or more coded sub-pictures in layer b may reference a partitioned sub-picture in layer a.

[0157] In the same or another embodiment, a coded video sequence (CVS) may be a set of coded pictures. A CVS may be composed of one or more coded sub-picture sequences (CSPS), where a CSPS may be a set of coded sub-pictures covering the same local area of ​​a picture. A CSPS may have the same or different temporal resolution as the coded video sequence.

[0158] In the same or another embodiment, the CSPS may be encoded and included in one or more layers. The CSPS may be composed of one or more CSPS layers. Decoding one or more CSPS layers corresponding to the CSPS may reconstruct a sub-picture sequence corresponding to the same local area.

[0159] In the same or another embodiment, the number of CSPS layers corresponding to one CSPS may be the same as or different from the number of CSPS layers corresponding to another CSPS.

[0160] In the same or another embodiment, one CSPS layer may have a different temporal resolution (e.g., frame rate) than another CSPS layer. The original (uncompressed) sub-picture sequence may be temporally resampled (upsampled or downsampled), encoded with different temporal resolution parameters, and included in the codestream corresponding to the layer.

[0161] In the same or another embodiment, a sub-picture sequence having a frame rate F may be encoded and included in the encoded bitstream corresponding to layer 0, while a sub-picture sequence having a frame rate F*S may be encoded and included in the encoded bitstream corresponding to layer 0. t,k The temporal upsampled (or downsampled) sub-picture sequence of the original sub-picture sequence can be encoded and included in the encoded bitstream corresponding to layer k, where S t,k indicates the temporal sampling rate of layer k. If S t,k If the value of S is greater than 1, the temporal resampling process is equivalent to frame rate upconversion. However, if S t,k If the value of is less than 1, the temporal resampling process is equivalent to frame rate down-conversion.

[0162] In the same or another embodiment, when a sub-picture with CSPS layer a is referenced by a sub-picture with CSPS layer b for motion compensation or any inter-layer prediction, if the spatial resolution of CSPS layer a is different from the spatial resolution of CSPS layer b, the decoded pixels in CSPS layer a are resampled and used as reference. The resampling process may require upsampling filtering or downsampling filtering.

[0163] Fig. 9An example of a syntax table in a slice header is shown, which is used to signal a syntax element vps_poc_cycle_au in a VPS (or SPS), which indicates the poc_cycle_au for all pictures / slices in a coded video sequence, and the syntax table signals a syntax element slice_poc_cycle_au, which indicates the poc_cycle_au of the current slice in the slice header. If the POC value of each AU increases uniformly, vps_contant_poc_cycle_per_au in the VPS is set to 1, and vps_poc_cycle_au is signaled in the VPS. In this case, slice_poc_cycle_au is not explicitly signaled, and the AUC value of each AU is calculated by dividing the value of the POC by vps_poc_cycle_au. If the POC value of each AU increases unevenly, vps_contant_poc_cycle_per_au in the VPS is set equal to 0. In this case, vps_access_unit_cnt is not signaled, but slice_access_unit_cnt is signaled in the slice header of each slice or picture. Each slice or picture can have a different slice_access_unit_cnt value. The AUC value of each AU is calculated by dividing the value of the POC by slice_poc_cycle_au. Fig.10 A block diagram illustrating the relevant workflow is shown.

[0164] In the same or other embodiments, pictures, slices or tiles corresponding to an AU having the same AUC value may be associated with the same decoding or output time instance, even though the values ​​of the POC of the pictures, slices or tiles may be different. Thus, all or a subset of pictures, slices or tiles associated with the same AU may be decoded in parallel and may be output at the same time instance, without any mutual parsing dependency / decoding dependency between the pictures, slices or tiles in the same AU.

[0165] In the same or other embodiments, pictures, slices or tiles corresponding to AUs with the same AUC value may be associated with the same composition time instance / display time instance, even though the POC values ​​of the pictures, slices or tiles may be different. When the composition time is included in the container format, pictures may be displayed at the same time instance, even if they correspond to different AUs, if the pictures have the same composition time.

[0166] In the same or other embodiments, each picture, slice, or tile may have the same temporal identifier (temporal_id) in the same AU. All or a subset of the pictures, slices, or tiles corresponding to a time instance may be associated with the same temporal sub-layer. In the same or other embodiments, each picture, slice, or tile may have the same or different spatial layer id (layer_id) in the same AU. All or a subset of the pictures, slices, or tiles corresponding to a time instance may be associated with the same or different spatial layers.

[0167] Fig.11 An example video bitstream is shown that includes a background video CSPS layer with layer_id equal to 0 and multiple foreground CSPS layers. Although the encoded sub-pictures may consist of one or more CSPS layers, the background region that does not belong to any foreground CSPS layer may include the base layer. The base layer may contain both the background region and the foreground region, while the enhanced CSPS layers contain the foreground regions. At the same region, the enhanced CSPS layers may have better visual quality than the base layer. The enhanced CSPS layers may reference the reconstructed pixels and motion vectors of the base layer corresponding to the same region.

[0168] In the same or another embodiment, in a video file, the video bitstream corresponding to the base layer is contained in a track, while the CSPS layers corresponding to each sub-picture are contained in separate tracks.

[0169] In the same or another embodiment, the video bitstream corresponding to the base layer is contained in a track, while the CSPS layers with the same layer_id are contained in separate tracks. In this example, the track corresponding to layer k includes only the CSPS layers corresponding to layer k.

[0170] In the same or another embodiment, each CSPS layer of each sub-picture is stored in a separate track. Each track may or may not have parsing dependencies or decoding dependencies on one or more other tracks.

[0171] In the same or another embodiment, each track may contain the bitstreams of layers i to j in the CSPS layers corresponding to all or a subset of the sub-pictures, where 0 < i = < j = < k, and k is the highest layer of the CSPS.

[0172] In the same or another embodiment, a picture consists of one or more associated media data, which include depth maps, alpha maps, 3D geometric data, occupancy maps, etc. Such associated timed media data may be divided into one or more data sub-bitstreams, each corresponding to a sub-picture.

[0173] In the same or another embodiment, Fig.12An example of a video conference based on a multi-layer sub-picture method is shown. In the video stream, a base layer video stream corresponding to a background picture and one or more enhancement layer video streams corresponding to a foreground sub-picture are included. Each enhancement layer video stream corresponds to a CSPS layer. In the display, the picture corresponding to the base layer is displayed by default. It contains a picture-in-picture (PIP) of one or more users. When a client control selects a specific user, the enhanced CSPS layer corresponding to the selected user is decoded and displayed with enhanced quality or spatial resolution. Fig.13 An operational schematic diagram is shown.

[0174] In the same or another embodiment, a network middlebox (such as a router) can select a subset of layers to send to a user based on its bandwidth. The picture / sub-picture organization can be used for bandwidth adaptation. For example, if a user does not have bandwidth, the router strips off layers or selects some sub-pictures due to the importance of the layers and sub-pictures or based on usage settings. This can be done dynamically to adapt to the bandwidth.

[0175] Fig.14 A use case for 360 video is shown. When a spherical 360 picture is projected onto a flat picture, the projected 360 picture can be partitioned into multiple sub-pictures as a base layer. The enhancement layer of a specific sub-picture is encoded and transmitted to the client. The decoder is able to decode the base layer including all sub-pictures and the enhancement layer of the selected sub-picture. When the current viewport is the same as the selected sub-picture, the displayed picture includes the decoded sub-picture with the enhancement layer, and the displayed picture has a higher quality. Otherwise, the decoded picture with the base layer can be displayed at a low quality.

[0176] In the same or another embodiment, any layout information for display may be present in the file as supplemental information (such as SEI messages or metadata). One or more decoded sub-pictures may be repositioned and displayed based on the signaled layout information. The layout information may be signaled by the streaming server or broadcaster, or may be regenerated by a network entity or cloud server, or may be determined by a user's customized settings.

[0177] In an embodiment, when the input image is divided into one or more (rectangular) sub-regions, each sub-region can be encoded as an independent layer. Each independent layer corresponding to a local area can have a unique layer_id value. For each independent layer, a signal can be sent to indicate the sub-image size and position information. For example, the image size (width, height), the offset information of the upper left corner (x_offset, y_offset). Fig.15An example of the layout of the divided sub-pictures, their sub-picture size and position information, and the picture prediction structure corresponding to the sub-pictures is shown. The layout information is signaled in a high-level syntax structure (such as a parameter set, a slice header or a tile group header, or an SEI message), and the layout information includes the sub-picture size and the sub-picture position.

[0178] In the same embodiment, each sub-picture corresponding to an independent layer may have its unique POC value within the AU. When indicating a reference picture in a picture stored in the DPB by using one or more syntax elements in the RPS or RPL structure, one or more POC values ​​corresponding to each sub-picture of the layer may be used.

[0179] In the same or another embodiment, to indicate the (inter-layer) prediction structure, layer_id may not be used and the POC (delta) value may be used.

[0180] In the same embodiment, a sub-picture having a POC value equal to N corresponding to a layer (or a local area) may or may not be used as a reference picture for a sub-picture having a POC value equal to N+K, the sub-picture having a POC value equal to N+K corresponding to the same layer (or the same local area), the reference picture being used for motion compensated prediction. In most cases, the value of the number K may be equal to the maximum number of (independent) layers, and the value of the number K may be the same as the number of sub-regions.

[0181] In the same or another embodiment, Fig.16 Shows Fig.15 When the input picture is divided into multiple (e.g., four) sub-regions, each local region can be encoded with one or more layers. In this case, the number of independent layers can be equal to the number of sub-regions, and one or more layers can correspond to one sub-region. Therefore, each sub-region can be encoded with one or more independent layers and zero or more dependent layers.

[0182] In the same embodiment, Fig.16 In , the input picture can be divided into four sub-regions. The upper right sub-region can be encoded into two layers, namely layer 1 and layer 4, and the lower right sub-region can be encoded into two layers, namely layer 3 and layer 5. In this case, layer 4 can refer to layer 1 for motion compensation prediction, and layer 5 can refer to layer 3 for motion compensation.

[0183] In the same or another embodiment, loop filtering (such as deblocking filtering, adaptive loop filtering, shaper, bilateral filtering, or any deep learning based filtering) across layer boundaries may (optionally) be disabled.

[0184] In the same or another embodiment, motion compensated prediction or intra-block copy across layer boundaries may (optionally) be disabled.

[0185] In the same or another embodiment, boundary filling or loop filtering at sub-picture boundaries for motion compensated prediction may be optionally performed. A flag indicating whether boundary filling is performed is signaled in a high-level syntax structure such as one or more parameter sets (VPS, SPS, PPS, or APS), a slice header or a tile group header, or an SEI message.

[0186] In the same or another embodiment, in a VPS or SPS, layout information of one or more sub-regions (or one or more sub-pictures) is signaled. Fig.17 Examples of syntax elements in VPS and SPS are shown. In this example, vps_sub_picture_dividing_flag is signaled in the VPS. This flag may indicate whether one or more input pictures are divided into multiple sub-areas. When the value of vps_sub_picture_dividing_flag is equal to 0, one or more input pictures in one or more encoded video sequences corresponding to the current VPS may not be divided into multiple sub-areas. In this case, the input picture size may be equal to the encoded picture size (pic_width_in_luma_samples, pic_height_in_luma_samples), and the input picture size is signaled in the SPS. When the value of vps_sub_picture_dividing_flag is equal to 1, one or more input pictures may be divided into multiple sub-areas. In this case, the syntax elements vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples are signaled in the VPS. The values ​​of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples may be equal to the width and height of one or more input pictures, respectively.

[0187] In the same embodiment, the values ​​of vps_full_pic_width_in_luma_samples and vps_full_pic_height_in_luma_samples may not be used for decoding, but may be used for synthesis and display.

[0188] In the same embodiment, when the value of vps_sub_picture_dividing_flag is equal to 1, the syntax elements pic_offset_x and pic_offset_y may be signaled in the SPS corresponding to one or more specific layers. In this case, the coded picture size (pic_width_in_luma_samples, pic_height_in_luma_samples) signaled in the SPS may be equal to the width and height of the sub-region corresponding to the specific layer. Furthermore, the position of the upper left corner of the sub-region (pic_offset_x, pic_offset_y) may be signaled in the SPS.

[0189] In the same embodiment, the position information (pic_offset_x, pic_offset_y) of the upper left corner of the sub-region may not be used for decoding, but may be used for synthesis and display.

[0190] In the same or another embodiment, layout information (size and location) of all or a subset of sub-regions of an input picture, dependency information between layers is signaled in a parameter set or SEI message. Fig.18 An example of a syntax element is shown, which is used to indicate layout information of a sub-region, dependency information between layers, and information about the relationship between a sub-region and one or more layers. In this example, the syntax element num_sub_region indicates the number of (rectangular) sub-regions in the current encoded video sequence. The syntax element num_layers indicates the number of layers in the current encoded video sequence. The value of num_layers may be equal to or greater than the value of num_sub_region. When any sub-region is encoded as a single layer, the value of num_layers may be equal to the value of num_sub_region. When one or more sub-regions are encoded as multiple layers, the value of num_layers may be greater than the value of num_sub_region. The syntax element direct_dependency_flag[i][j] indicates a dependency from the jth layer to the ith layer. num_layers_for_region[i] indicates the number of layers associated with the i-th sub-region. sub_region_layer_id[i][j] indicates the layer_id of the jth layer associated with the i-th sub-region. sub_region_offset_x[i] and sub_region_offset_y[i] indicate the horizontal position and vertical position of the upper left corner of the i-th sub-region, respectively. sub_region_width[i] and sub_region_height[i] indicate the width and height of the i-th sub-region, respectively.

[0191] In one embodiment, one or more syntax elements are signaled in a high-level syntax structure (e.g., VPS, DPS, SPS, PPS, APS, or SEI message). The one or more syntax elements specify an output layer set to indicate one of a plurality of layers to be output with or without profile level information. Fig.19 , a syntax element num_output_layer_sets is signaled in the VPS, which indicates the number of output layer sets (OLS) in the coded video sequence that references the VPS. For each output layer set, as many output_layer_flags as the number of output layers may be signaled.

[0192] In the same embodiment, output_layer_flag[i] equal to 1 specifies that the i-th layer is output. vps_output_layer_flag[i] equal to 0 specifies that the i-th layer is not output.

[0193] In the same or another embodiment, one or more syntax elements may be signaled in a high-level syntax structure (e.g., a VPS, DPS, SPS, PPS, APS, or SEI message) that specify profile level information for each output layer set. Fig.19 , a syntax element num_profile_tile_level may be signaled in the VPS, indicating the number of profile level information for each OLS in the coded video sequence referring to the VPS. For each output layer set, a set of syntax elements for profile level information may be signaled with the same number as the number of output layers, or an index indicating a specific profile level information in an entry in the profile level information may be signaled.

[0194] In the same embodiment, profile_tier_level_idx[i][j], in the list of profile_tier_level() syntax structures in the VPS, specifies the index of the profile_tier_level() syntax structure that applies to the j-th tier of the i-th OLS.

[0195] In the same or another embodiment, reference Fig. 20 , when the maximum number of layers is greater than 1 (vps_max_layers_minus1>0), the syntax elements num_profile_tile_level and / or num_output_layer_sets may be signaled.

[0196] In the same or another embodiment, reference Fig. 20, the syntax element vps_output_layers_mode[i] may be present in the VPS, indicating the mode of output layer signaling for the i-th output layer set.

[0197] In the same embodiment, vps_output_layers_mode[i] equal to 0 specifies: only the highest layer with the i-th output layer set is output. vps_output_layer_mode[i] equal to 1 specifies: all layers with the i-th output layer set are output. vps_output_layer_mode[i] equal to 2 specifies: the layers output are the layers with vps_output_layer_flag[i][j] equal to 1 and with the i-th output layer set. More values ​​may be reserved.

[0198] In the same embodiment, output_layer_flag[i][j] may or may not be signaled depending on the value of vps_output_layers_mode[i] for the i-th output layer set.

[0199] In the same or another embodiment, reference Fig. 20 , there is a flag vps_ptl_signal_flag[i] for the i-th output layer set. Depending on the value of vps_ptl_signal_flag[i], the profile level information for the i-th output layer set may be signaled or not.

[0200] In the same or another embodiment, reference Fig.21 , the number of sub-pictures in the current CVS, max_subpics_minus1, can be signaled in a high-level syntax structure (eg, VPS, DPS, SPS, PPS, APS, or SEI message).

[0201] In the same embodiment, reference Fig.21 , when the number of sub-pictures is greater than 1 (max_subpics_minus1>0), the sub-picture identifier sub_pic_id[i] for the i-th sub-picture may be signaled.

[0202] In the same or another embodiment, in the VPS, one or more syntax elements are signaled that indicate the sub-picture identifiers of each layer belonging to each output layer set. Fig. 22 , sub_pic_id_layer[i][j][k] indicates the kth sub-picture present in the jth layer of the i-th output layer set. Using this information, the decoder can identify which sub-picture can be decoded and output for each layer of a specific output layer set.

[0203] In an embodiment, a picture header (PH) is a syntax structure containing syntax elements that apply to all slices of a coded picture. A picture unit (PU) is a group of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and contain exactly one coded picture. A PU may contain a picture header (PH) and one or more VCL NAL units that make up a coded picture.

[0204] In an embodiment, the SPS (Raw Byte Sequence Payload, RBSP) is available for the decoding process before it is referenced, included in at least one AU with TemporalId equal to 0, or provided by an external device.

[0205] In an embodiment, the SPS (RBSP) is available to the decoding process before it is referenced, included in at least one AU with TemporalId equal to 0 in the CVS (including one or more PPSs referencing the SPS), or provided by an external device.

[0206] In an embodiment, the SPS (RBSP) is available to the decoding process before it is referenced by one or more PPSs, included in at least one PU, or provided by an external device, wherein the nuh_layer_id of the at least one PU is equal to the lowest nuh_layer_id value of the PPS NAL units that reference the SPS NAL units in the CVS (including the one or more PPSs that reference the SPS).

[0207] In an embodiment, the SPS (RBSP) is available to the decoding process before it is referenced by one or more PPSs, included in at least one PU or provided by an external device. The at least one PU has TemporalId equal to 0 and nuh_layer_id equal to the lowest nuh_layer_id value of the PPS NAL units that reference the SPS NAL unit.

[0208] In an embodiment, the SPS (RBSP) is available to the decoding process before it is referenced by one or more PPSs, included in at least one PU or provided by an external device. The at least one PU has a TemporalId equal to 0 and nuh_layer_id equal to the lowest nuh_layer_id value of the PPS NAL units that reference the SPS NAL units in the CVS (including one or more PPSs that reference the SPS).

[0209] In the same or another embodiment, pps_seq_parameter_set_id specifies the value of sps_seq_parameter_set_id for the referenced SPS.The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by coded pictures in the CLVS.

[0210] In the same or another embodiment, all SPSNAL units in a CVS having a particular sps_seq_parameter_set_id value may have the same content.

[0211] In the same or another embodiment, SPS NAL units may share the same value space of sps_seq_parameter_set_id regardless of the nuh_layer_id value.

[0212] In the same or another embodiment, the nuh_layer_id value of the SPS NAL unit may be equal to the lowest nuh_layer_id value of the PPS NAL units that reference the SPS NAL unit.

[0213] In an embodiment, when an SPS with nuh_layer_id equal to m is referenced by one or more PPSs with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n, or may be the same as the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0214] In an embodiment, the PPS (RBSP) may be available to the decoding process before it is referenced, included in at least one AU having a TemporalId equal to the TemporalId of the PPS NAL unit, or provided by an external device.

[0215] In an embodiment, the PPS (RBSP) is available to the decoding process before it is referenced, included in at least one AU or provided by an external device. The TemporalId of the at least one AU is equal to the TemporalId of the PPS NAL unit in the CVS (which contains one or more PHs (or coded slice NAL units) that reference the PPS).

[0216] In an embodiment, the PPS (RBSP) is available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units), included in at least one PU or provided by an external device. The nuh_layer_id of the at least one PU is equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the PPS NAL unit in the CVS (which contains one or more PHs (or coded slice NAL units) that reference the PPS).

[0217] In an embodiment, the PPS (RBSP) is available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units), included in at least one PU or provided by an external device. The TemporalId of the at least one PU is equal to the TemporalId of the PPS NAL unit, and the nuh_layer_id is equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the PPS NAL unit in the CVS (including one or more PHs (or coded slice NAL units) that reference the PPS).

[0218] In the same or another embodiment, ph_pic_parameter_set_id in the PH specifies the value of pps_pic_parameter_set_id of the reference PPS used.The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by coded pictures in the CLVS.

[0219] In the same or another embodiment, all PPSNAL units within a PU with a particular pps_pic_parameter_set_id value shall have the same content.

[0220] In the same or another embodiment, PPS NAL units may share the same value space of pps_pic_parameter_set_id regardless of the nuh_layer_id value.

[0221] In the same or another embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the NAL unit (which references the PPS NAL unit).

[0222] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n, or may be the same as the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0223] In an embodiment, the PPS (RBSP) is available for the decoding process before it is referenced, included in at least one AU, or provided by an external device. The TemporalId of the at least one AU is equal to the TemporalId of the PPS NAL unit.

[0224] In an embodiment, the PPS (RBSP) is available to the decoding process before it is referenced, included in at least one AU or provided by an external device. The TemporalId of the at least one AU is equal to the TemporalId of the PPS NAL unit in the CVS (which contains one or more PHs (or coded slice NAL units) that reference the PPS).

[0225] In an embodiment, the PPS (RBSP) is available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units), included in at least one PU or provided by an external device. The nuh_layer_id of the at least one PU is equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the PPS NAL unit in the CVS (which contains one or more PHs (or coded slice NAL units) that reference the PPS).

[0226] In an embodiment, the PPS (RBSP) is available to the decoding process before it is referenced by one or more PHs (or coded slice NAL units), included in at least one PU, or provided by an external device. The TemporalId of the at least one PU is equal to the TemporalId of the PPS NAL unit, and the nuh_layer_id is equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the PPS NAL unit in the CVS (including one or more PHs (or coded slice NAL units) that reference the PPS).

[0227] In the same or another embodiment, ph_pic_parameter_set_id in the PH specifies the value of pps_pic_parameter_set_id for the referenced PPS. The value of pps_seq_parameter_set_id may be the same in all PPSs referenced by coded pictures in the CLVS.

[0228] In the same or another embodiment, all PPSNAL units within a PU with a particular pps_pic_parameter_set_id value shall have the same content.

[0229] In the same or another embodiment, PPS NAL units may share the same value space of pps_pic_parameter_set_id regardless of the nuh_layer_id value.

[0230] In the same or another embodiment, the nuh_layer_id value of the PPS NAL unit may be equal to the lowest nuh_layer_id value of the coded slice NAL units that reference the NAL unit (which references the PPS NAL unit).

[0231] In an embodiment, when a PPS with nuh_layer_id equal to m is referenced by one or more coded slice NAL units with nuh_layer_id equal to n, the layer with nuh_layer_id equal to m may be the same as the layer with nuh_layer_id equal to n, or may be the same as the (direct or indirect) reference layer of the layer with nuh_layer_id equal to m.

[0232] like Fig. 22 As shown, in an embodiment, pps_subpic_id[i] in the picture parameter set specifies the sub-picture ID of the i-th sub-picture. The length of the pps_subpic_id[i] syntax element is pps_subpic_id_len_minus1+1 bits.

[0233] For each value of i in the range from 0 to sps_num_subpics_minus1 (inclusive), the variable SubpicIdVal[i] is derived as follows:

[0234]

[0235] In the same or another embodiment, for any two different values ​​of i and j in the range of 0 to sps_num_subpics_minus1, inclusive, SubpicIdVal[i] may not be equal to SubpicIdVal[j].

[0236] In the same or another embodiment, when the current picture is not the first picture of the CLVS, for each value of i in the range of 0 to sps_num_subpics_minus1 (inclusive), if the value of SubpicIdVal[i] is not equal to the value of SubpicIdVal[i] of the previous picture in decoding order in the same layer, then in the current picture with sub-picture index i, the nal_unit_type of all coded slice NAL units of the sub-picture may be equal to a specific value in the range of IDR_W_RADL to CRA_NUT (inclusive).

[0237] In the same or another embodiment, when the current picture is not the first picture of the CLVS, for each value of i in the range of 0 to sps_num_subpics_minus1 (inclusive), if the value of SubpicIdVal[i] is not equal to the value of SubpicIdVal[i] of the previous picture in decoding order in the same layer, sps_independent_subpics_flag may be equal to 1.

[0238] In the same or another embodiment, when the current picture is not the first picture of the CLVS, for each value of i in the range of 0 to sps_num_subpics_minus1 (inclusive), if the value of SubpicIdVal[i] is not equal to the value of SubpicIdVal[i] of the previous picture in decoding order in the same layer, subpic_treated_as_pic_flag[i] and loop_filter_across_subpic_enabled_flag[i] may be equal to 1.

[0239] In the same or another embodiment, when the current picture is not the first picture of the CLVS, for each value of i in the range of 0 to sps_num_subpics_minus1 (inclusive), if the value of SubpicIdVal[i] is not equal to the value of SubpicIdVal[i] of the previous picture in the same layer in decoding order, then sps_independent_subpics_flag shall be equal to 1, or subpic_treated_as_pic_flag[i] and loop_filter_across_subpic_enabled_flag[i] shall be equal to 1.

[0240] In the same or another embodiment, when a sub-picture is encoded independently without any reference to another sub-picture, the value of the sub-picture identifier of a region may change within the encoded video sequence.

[0241] The number of sub-pictures in a picture can be signaled in the SPS. Fig.23 In the CLVS, sps_num_subpics_minus1 plus 1 specifies the number of sub-pictures in each picture in the CLVS. The value of sps_num_subpics_minus1 shall be in the range of 0 to Ceil(pic_width_max_in_luma_samples÷CtbSizeY)*Ceil(pic_height_max_in_luma_samples÷CtbSizeY)-1, inclusive. When sps_num_subpics_minus1 is not present, the value of sps_num_subpics_minus1 is inferred to be equal to 0.

[0242] In the same or another embodiment, the number of sub-pictures in a picture may be signaled in the PPS. Fig.24 In , pps_num_subpics_minus1 specifies the number of sub-pictures in each picture and can be equal to sps_num_subpics_minus1.

[0243] The number of slices in a picture can be signaled in the PPS. Fig.24 , num_slices_in_pic_minus1 plus 1 specifies the number of rectangular slices in each picture that references the PPS. The value of num_slices_in_pic_minus1 may be in the range of 0 to MaxSlicesPerPicture-1, inclusive, where MaxSlicesPerPicture is specified in Appendix A. When no_pic_partition_flag is equal to 1, the value of num_slices_in_pic_minus1 is inferred to be equal to 0. When single_slice_per_subpic_flag is equal to 1, the value of num_slices_in_pic_minus1 is inferred to be equal to pps_num_subpics_minus1.

[0244] The number of sub-pictures may not be greater than the number of slices. When the value of single_slice_per_subpic_flag in the PPS referenced by the picture is equal to 1, the number of sub-pictures is equal to the number of slices in the picture.

[0245] In the same embodiment, the value of the variable NumSlicesInPic indicating the number of slices in a picture is derived as follows: NumSlicesInPic = num_slices_in_pic_minus1 + 1.

[0246] The difference between the number of sub-pictures and the number of slices may be signaled in the PPS, and the number of slices in a picture may be derived from the number of sub-pictures and the difference between the signaled number of sub-pictures and the number of slices.

[0247] like Fig.25 As shown, in an embodiment, num_slices_in_pic_minus_num_sub_pics plus pps_num_subpics_minus1+1 specifies the number of rectangular slices in each picture that references the PPS. The value of num_slices_in_pic_minus_num_sub_pics shall be in the range of 0 to MaxSlicesPerPicture - pps_num_subpics_minus1 - 1 (inclusive), where MaxSlicesPerPicture is specified in Appendix A. When single_slice_per_subpic_flag is equal to 1, the value of num_slices_in_pic_minus_num_sub_pics is inferred to be equal to 0.

[0248] In the same embodiment, the value of the variable NumSlicesInPic indicating the number of slices in a picture is derived as follows:

[0249]

[0250]

[0251] like Fig.26 As shown, in another embodiment, num_slices_in_pic_minus_num_sub_pics_minus1 plus pps_num_subpics_minus1+2 specifies the number of rectangular slices in each picture that references the PPS. The value of num_slices_in_pic_minus_num_sub_pics_minus1 shall be in the range of 0 to MaxSlicesPerPicture - pps_num_subpics_minus1 - 2) (inclusive), where MaxSlicesPerPicture is specified in Appendix A. When single_slice_per_subpic_flag is equal to 1, the value of num_slices_in_pic_minus_num_sub_pics_minus1 is inferred to be equal to 0.

[0252] In the same embodiment, the value of the variable NumSlicesInPic indicating the number of slices in a picture is derived as follows:

[0253]

[0254] In the same or another embodiment, when rect_slice_flag is equal to 1, for i in the range of 0 to NumSlicesInPic-1 (inclusive), the list NumCtusInSlice[i] specifies the number of CTUs in the i-th slice; for i in the range of 0 to NumSlicesInPic-1 (inclusive), the list SliceTopLeftTileIdx[i] specifies the tile index of the tile that contains the first CTU in the slice; and for i in the range of 0 to NumSlicesInPic-1 (inclusive), and j in the range of 0 to NumCtusInSlice[i]-1 (inclusive), the matrix CtbAddrInSlice[i][j] specifies the picture raster scan address of the j-th CTB in the i-th slice; and the variable NumSlicesInTile[i] specifies the number of slices in the tile that contains the i-th slice, and these lists, matrices, and variables are derived as follows:

[0255]

[0256]

[0257]

[0258]

[0259]

[0260]

[0261] Among them, the function AddCtbsToSlice(sliceIdx,startX,stopX,startY,stopY) is specified as follows:

[0262]

[0263] Codestream conformance requires that for i in the range 0 to num_slices_in_pic_minus1 (inclusive), the value of NumCtusInSlice[i] shall be greater than 0. Furthermore, codestream conformance requires that for i in the range 0 to num_slices_in_pic_minus1 (inclusive), and j in the range 0 to NumCtusInSlice[i]-1 (inclusive), the matrix CtbAddrInSlice[i][j] shall include once and only once each of the CTB addresses in the range 0 to PicSizeInCtbsY-1 (inclusive).

[0264] The list NumSlicesInSubpic[i] specifies the number of rectangular slices in the i-th sub-picture. The list NumSlicesInSubpic[i] is derived as follows:

[0265]

[0266]

[0267] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various substitutions of equivalents that fall within the scope of the present disclosure. It should therefore be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. A method of encoding, characterized in that: The method comprises: Receiving video data including one or more sub-pictures in a current picture; signaling, in a picture parameter set PPS, a number of one or more sub-pictures in the current picture and a delta value between the number of one or more sub-pictures in the current picture and a number of rectangular slices; and Based on the number of the one or more sub-pictures and the delta value, the number of the rectangular slices is derived, wherein: When the current picture is not the first picture of the coding layer video sequence CLVS, for each sub-picture in the current picture, if the value of the sub-picture identifier of the sub-picture with sub-picture index i in the current picture is not equal to the value of the sub-picture identifier of the sub-picture with sub-picture index i in the previous picture in the same layer in decoding order, the sequence parameter set flag sps_independent_subpics_flag in which the sub-picture of the current picture is independently encoded is equal to 1, the flag subpic_treated_as_pic_flag corresponding to the sub-picture of the current picture and the flag loop_filter_acros_subpic_enabled_flag corresponding to the loop filtering operation across sub-pictures are equal to 1, and the value range of i is from 1 to the maximum number of sub-pictures in the current picture.

2. The method according to claim 1, characterized in that Further including: The number of the rectangular slices is signaled in a picture parameter set.

3. The method according to claim 2, characterized in that Further including: Based on a flag corresponding to a picture in which no partition is set, the number of the rectangular slices is inferred to be 1.

4. The method according to claim 1, characterized in that: Further including: Based on a flag corresponding to setting a single slice for each of the one or more sub-pictures, it is inferred that the delta value between the number of the one or more sub-pictures and the number of the rectangular slices is 0.

5. The method according to any one of claims 1 to 4, characterized in that in, The number of the one or more sub-pictures is smaller than the number of the rectangular slices.

6. A decoding method, characterized in that: The method comprises: Receiving a video stream including one or more sub-pictures in a current picture; Determine, according to a picture parameter set PPS, the number of one or more sub-pictures in the current picture and an incremental value between the number of one or more sub-pictures in the current picture and the number of rectangular slices; and Based on the number of the one or more sub-pictures and the delta value, the number of the rectangular slices is derived, wherein: When the current picture is not the first picture of the coding layer video sequence CLVS, for each sub-picture in the current picture, if the value of the sub-picture identifier of the sub-picture with sub-picture index i in the current picture is not equal to the value of the sub-picture identifier of the sub-picture with sub-picture index i in the previous picture in the same layer in decoding order, the sequence parameter set flag sps_independent_subpics_flag in which the sub-picture of the current picture is independently encoded is equal to 1, the flag subpic_treated_as_pic_flag corresponding to the sub-picture of the current picture and the flag loop_filter_acros_subpic_enabled_flag corresponding to the loop filtering operation across sub-pictures are equal to 1, and the value range of i is from 1 to the maximum number of sub-pictures in the current picture.

7. A computer system for encoding and decoding, characterized in that: The computer system comprises: one or more computer readable non-volatile storage media configured to store computer program code; and One or more computer processors are configured to access the computer program code and operate according to the instructions of the computer program code to perform the method according to any one of claims 1 to 6.

8. A coding device, characterized in that The device comprises: A receiving module, receiving video data including one or more sub-pictures in a current picture; a signaling module, signaling, in a picture parameter set (PPS), the number of one or more sub-pictures in the current picture and an incremental value between the number of one or more sub-pictures in the current picture and the number of rectangular slices; and A deriving module, based on the number of the one or more sub-pictures and the increment value, deriving the number of the rectangular slices, wherein When the current picture is not the first picture of the coding layer video sequence CLVS, for each sub-picture in the current picture, if the value of the sub-picture identifier of the sub-picture with sub-picture index i in the current picture is not equal to the value of the sub-picture identifier of the sub-picture with sub-picture index i in the previous picture in the same layer in decoding order, the sequence parameter set flag sps_independent_subpics_flag in which the sub-picture of the current picture is independently encoded is equal to 1, the flag subpic_treated_as_pic_flag corresponding to the sub-picture of the current picture, and the flag loop_filter_acros_subpic_enabled_flag corresponding to the loop filtering operation across sub-pictures are equal to 1, and the value range of i is from 1 to the maximum number of sub-pictures in the current picture.

9. A decoding device, characterized in that: comprising a processing circuit configured to: process the method of claim 6.

10. A method for processing a video code stream, characterized in that: The video code stream is generated according to the encoding method according to any one of claims 1 to 5, or is decoded based on the decoding method according to claim 6.

11. A non-volatile computer-readable medium, characterized in that: A computer program is stored thereon, the computer program being configured to cause one or more computer processors to execute the method according to any one of claims 1 to 5.