Reference picture management in video coding

Abstract

A method for decoding a coded video bitstream includes obtaining a first reference picture list and a second reference picture list for a current slice, wherein the current slice is represented in the coded video bitstream; when the current slice is a first slice in a current picture to which the current slice belongs, identifying a status of a reference picture in a decoded picture buffer (DPB) according to the first reference picture list and the second reference picture list. The status of the reference picture in the decoded picture buffer is one of used for short-term reference, used for long-term reference, or unused for reference. The method further includes using the reference picture in a reconstruction process of the current picture or a picture subsequent to the current picture when the status of the reference picture is used for short-term reference or used for long-term reference.

Description

[0001] This application is a divisional application. The original application has the application number 201980054208.2 and the original application date is August 16, 2019. The entire contents of the original application are incorporated herein by reference.

[0002] Cross-referencing related applications This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 719,360, filed August 17, 2018, entitled “Reference Picture Management in Video Coding,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention generally relates to reference image management techniques for video decoding. More specifically, this invention describes techniques for constructing reference image lists and identifying reference images. Background Technology

[0004] Even with shorter videos, a large amount of video data needs to be described, which can be challenging when the data needs to be sent over communication networks with limited bandwidth or otherwise transmitted. Therefore, video data is typically compressed before being sent over modern telecommunications networks. Video size can also be an issue when storing video on storage devices due to potentially limited memory resources. Video compression devices typically use software and / or hardware on the source side to encode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination by a video decompression device that decodes the video data. Given limited network resources and the growing demand for higher video quality, there is a need to improve compression and decompression techniques that can increase compression ratios with minimal impact on image quality. Summary of the Invention

[0005] The first aspect relates to a method for decoding a decoded video stream. The method includes: obtaining a first reference image list and a second reference image list for a current stripe, wherein the current stripe is represented in the decoded video stream; when the current stripe is the first stripe in a current image to which the current stripe belongs, identifying the state of reference images in a decoded picture buffer (DPB) according to the first reference image list and the second reference image list, wherein the state of the reference images in the decoded picture buffer is one of short-term reference, long-term reference, or no reference; when the state of the reference images is short-term reference or long-term reference, using the reference images in the reconstruction process of the current image or images following the current image.

[0006] The second aspect relates to a method for decoding a decoded video stream implemented by a video decoder. The method includes: the video decoder decoding a first image to obtain a first decoded image, wherein the first image includes all its stripes; after decoding, the video decoder storing the first decoded image in a decoded picture buffer (DPB); after all stripes in the first image have been decoded, the video decoder identifying the first image as a short-term reference; the video decoder decoding a first stripe in a current image, wherein decoding the first stripe in the current image includes: the video decoder obtaining a first reference image list and a second reference image list; when the first image is referenced by a long-term reference picture (LTRP) entry in either the first or second reference image list, the video decoder re-identifies the first image as a long-term reference; when the first image is not referenced in either the first or second reference image list, the video decoder re-identifies the first image as not for reference.

[0007] The techniques provided by the above methods simplify the signaling of the reference image list and improve the efficiency of the reference image list signaling, thereby improving the overall decoding process.

[0008] According to the first or second aspect, in a first implementation of the method, the first reference image list is RefPictList[0].

[0009] According to the first or second aspect or any of the above implementations of the first or second aspect, in the second implementation of the method, the second reference image list is RefPictList[1].

[0010] According to the first or second aspect or any of the above implementations of the first or second aspect, in a third implementation of the method, the current image includes an intra random access point (IRAP) image.

[0011] According to the first or second aspect or any of the above implementations of the first or second aspect, in a fourth implementation of the method, when the first image is identified as being used for short-term reference, the first image is called a short-term reference picture (STRP).

[0012] According to the first or second aspect or any of the above implementations of the first or second aspect, in a fifth implementation of the method, the STRP is identified by the image sequence number value (PicOrderCntVal).

[0013] According to the first or second aspect or any of the above implementations of the first or second aspect, in the sixth implementation of the method, LTRP is identified by the logarithmic function (Log2(MaxLtPicOrderCntLsb)) of the least significant bit of the maximum long-term image order number value (PicOrderCntVal).

[0014] According to the first or second aspect or any of the above implementations of the first or second aspect, in a seventh implementation of the method, in addition to the first decoded image obtained after decoding the current image, the DPB also includes a plurality of reference images.

[0015] According to the first or second aspect or any of the above implementations of the first or second aspect, in an eighth implementation of the method, at any given time, the first image is identified or re-identified only as one of short-term reference, long-term reference, and no reference.

[0016] According to the first or second aspect or any of the above implementations of the first or second aspect, in a ninth implementation of the method, the step of identifying or re-identifying the first image as for short-term reference, for long-term reference, and not for reference excludes any other identifier of the first image.

[0017] The third and fourth aspects relate to a decoding device. The decoding device includes: a receiver for receiving an image for encoding or receiving a bitstream for decoding; a transmitter coupled to the receiver, wherein the transmitter is configured to send the bitstream to a decoder or send a decoded image to a display; a memory coupled to at least one of the receiver and the transmitter, wherein the memory is configured to store instructions; and a processor coupled to the memory, wherein the processor is configured to execute the instructions stored in the memory to perform the method described in any of the foregoing aspects or implementations.

[0018] The technology provided by the decoding device simplifies the indication of the reference image list and improves its efficiency. This, in turn, improves the overall decoding process. According to the third and fourth aspects, in a first implementation of the decoding device, the decoding device further includes a display for displaying the current image.

[0019] The fifth aspect relates to a decoding apparatus. The decoding apparatus includes: a receiver for receiving a code stream for decoding; a transmitter coupled to the receiver, wherein the transmitter is configured to send a decoded image to a display; a memory coupled to at least one of the receiver and the transmitter, wherein the memory is configured to store instructions; and a processor coupled to the memory, wherein the processor is configured to execute the instructions stored in the memory to perform the method described in any of the foregoing aspects or implementations.

[0020] The sixth aspect relates to a system. The system includes an encoder and a decoder in communication with the encoder. The encoder or the decoder includes any of the decoding devices or decoding apparatuses described in the foregoing aspects or implementations.

[0021] The technology provided by the system simplifies and improves the efficiency of the reference image list indication, thereby improving the overall decoding process.

[0022] The seventh aspect relates to an encoding module. The encoding module includes: a receiving module for receiving an image for encoding or receiving a bitstream for decoding; a transmitting module coupled to the receiving module, wherein the transmitting module is used to transmit the bitstream to a decoder or to a display module for transmitting a decoded image; a storage module coupled to at least one of the receiving module or the transmitting module, wherein the storage module is used to store instructions; and a processing module coupled to the storage module, wherein the processing module is used to execute the instructions stored in the storage module to perform the method described in any of the above aspects or implementations.

[0023] The decoding module provides a technique that simplifies and improves the efficiency of the reference image list indication, thereby improving the overall decoding process. Attached Figure Description

[0024] To gain a more complete understanding of the invention, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein similar reference numerals denote similar parts.

[0025] Figure 1 This is a block diagram of an exemplary decoding system that can employ bidirectional prediction techniques.

[0026] Figure 2 This is a block diagram of an exemplary video encoder that can implement bidirectional prediction technology.

[0027] Figure 3 This is a block diagram of an exemplary video decoder that can implement bidirectional prediction technology.

[0028] Figure 4 This is a schematic diagram of a reference picture set (RPS) that includes entries from all subsets of the RPS.

[0029] Figure 5 This is an example of a method for decoding a decoded video stream.

[0030] Figure 6 This is an example of a method for decoding a decoded video stream.

[0031] Figure 7 This is a schematic diagram of a video decoding device.

[0032] Figure 8 This is a schematic diagram of one embodiment of the decoding module. Detailed Implementation

[0033] Figure 1 This is a block diagram of an exemplary decoding system 10 that can employ the video decoding technology described herein. Figure 1 As shown, the decoding system 10 includes a source device 12, which provides encoded video data that is later decoded by a destination device 14. Specifically, the source device 12 can provide video data to the destination device 14 via a computer-readable medium 16. The source device 12 and the destination device 14 can include any of a variety of devices, including desktop computers, laptops, tablets, set-top boxes, mobile phones such as smartphones and tablets, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, the source device 12 and the destination device 14 can be used for wireless communication.

[0034] Destination device 14 can receive encoded video data to be decoded via computer-readable medium 16. Computer-readable medium 16 can include any type of medium or device capable of transmitting encoded video data from source device 12 to destination device 14. In one example, computer-readable medium 16 can include a communication medium enabling source device 12 to transmit encoded video data directly to destination device 14 in real time. The encoded video data can be modulated according to communication standards such as wireless communication protocols and transmitted to destination device 14. The communication medium can include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet network (e.g., a local area network, a wide area network, or a global network such as the Internet). The communication medium can include a router, switch, base station, or any other device that can be used for communication from source device 12 to destination device 14.

[0035] In some examples, encoded data can be output from output interface 22 to a storage device. Similarly, encoded data can be accessed from a storage device via an input interface. The storage device can include any of a variety of distributed or locally accessible data storage media, such as hard disk drives, Blu-ray discs, digital video discs (DVDs), compact disc read-only memory (CD-ROMs), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device can correspond to a file server or another intermediate storage device that can store the encoded video generated by source device 12. Destination device 14 can access the stored video data from the storage device via streaming or downloading. The file server can be any type of server capable of storing and sending encoded video data to destination device 14. Exemplary file servers include World Wide Web servers (e.g., for websites), file transfer protocol (FTP) servers, network attached storage (NAS) devices, or local disk drives. Destination device 14 can access encoded video data via any standard data connection, including an Internet connection. Standard data connections may include wireless channels (e.g., Wi-Fi connections), wired connections (e.g., digital subscriber line (DSL), cable modems, etc.), or a combination of both suitable for accessing encoded video data stored on a file server. Transfer of encoded video data from the storage device can be streaming, downloading, or a combination thereof.

[0036] The technologies of this invention are not necessarily limited to wireless applications or setups. These technologies can be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission (e.g., dynamic adaptive streaming over HTTP (DASH)), encoding digital video into data storage media, decoding digital video stored on data storage media, or other applications. In some examples, the decoding system 10 can be used to support one-way or two-way video transmission, thereby supporting applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0037] exist Figure 1In one example, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Destination device 14 includes an input interface 28, a video decoder 30, and a display device 32. According to the invention, the video encoder 20 in source device 12 and / or the video decoder 30 in destination device 14 can be used to employ video decoding technology. In other examples, the source device and destination device may include other components or devices. For example, source device 12 may receive video data from an external video source (e.g., an external camera). Similarly, destination device 14 may be connected to an external display device, rather than including an integrated display device.

[0038] Figure 1 The decoding system 10 shown is merely an example. Video decoding technology can be performed by any digital video encoding and / or decoding device. Although the techniques in this invention are typically performed by video decoding devices, they can also be performed by video encoders / decoders (commonly referred to as "codecs (CODECs)"). Furthermore, the techniques in this invention can also be performed by a video preprocessor. The video encoder and / or decoder can be a graphics processing unit (GPU) or a similar device.

[0039] Source device 12 and destination device 14 are merely examples of such decoding devices, where source device 12 generates decoded video data to be sent to destination device 14. In some examples, source device 12 and destination device 14 can operate substantially symmetrically, such that both source device 12 and destination device 14 include video encoding and decoding components. Therefore, decoding system 10 can support one-way or two-way video transmission between video devices 12 and 14, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0040] The video source 18 in source device 12 may include a video capture device (e.g., a camera), a video archive containing previously captured video, and / or a video input interface for receiving video from a video content provider. Alternatively, the video source 18 may generate computer graphics-based data as source video, or as a combination of live video, archived video, and computer-generated video.

[0041] In some cases, when the video source 18 is a camera, the source device 12 and the destination device 14 can form a camera phone or videophone. However, as described above, the techniques described in this invention are generally applicable to video decoding and can also be applied to wireless and / or wired applications. In each case, the video encoder 20 can encode the captured, pre-captured, or computer-generated video. The encoded video information can then be output to a computer-readable medium 16 via the output interface 22.

[0042] Computer-readable medium 16 may include transient media such as wireless broadcasting or wired network transmissions, and may also include storage media (i.e., non-transient storage media), such as hard disks, flash drives, optical discs, digital video optical discs, Blu-ray discs, or other computer-readable media. In some examples, a network server (not shown) may receive encoded video data from source device 12 and provide the encoded video data to destination device 14, for example, via network transmission. Similarly, a computing device in a media production facility (e.g., an optical disc stamping facility) may receive encoded video data from source device 12 and produce an optical disc including the encoded video data. Therefore, in various examples, computer-readable medium 16 can be understood to include one or more computer-readable media of various forms.

[0043] The input interface 28 in the destination device 14 receives information from the computer-readable medium 16. The information received by the computer-readable medium 16 may include grammatical information defined by the video encoder 20, which is also used by the video decoder 30, including grammatical elements describing the features and / or processing methods of descriptive blocks and other decoded units (e.g., groups of pictures, GOPs). The display device 32 displays the decoded video data to the user and may include any of a variety of display devices, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light emitting diode (OLED) display, or other types of display devices.

[0044] The video encoder 20 and video decoder 30 can operate according to video decoding standards (such as the Efficiency Video Coding (HEVC) standard currently under development) and can comply with the HEVC Test Model (HM). Alternatively, the video encoder 20 and video decoder 30 can operate according to other proprietary or industry standards, such as the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) H.264 standard (or Motion Picture Expert Group (MPEG)-4 Part 10, Advanced Video Coding (AVC)), H.265 / HEVC, and extended versions of such standards. However, the technology of this invention is not limited to any particular decoding standard. Other examples of video decoding standards include MPEG-2 and ITU-T H.263. Although Figure 1 Not shown, but in some aspects, the video encoder 20 and video decoder 30 may be integrated with the audio encoder and audio decoder, respectively, and may include suitable multiplexer-demultiplexer (MUX-DEMUX) units or other hardware and software to encode both audio and video in a shared data stream or separate data streams. Where applicable, the MUX-DEMUX unit may comply with the ITU H.223 multiplexer protocol or other protocols such as the User Datagram Protocol (UDP).

[0045] The video encoder 20 and video decoder 30 can each be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the foregoing technology is implemented in software, a device may store the instructions of the software in a suitable non-transitory computer-readable medium and execute these instructions in hardware using one or more processors to perform the technology of the present invention. Both the video encoder 20 and video decoder 30 can be included in one or more encoders or decoders, which can be integrated as part of a combined encoder / decoder (codec) in the respective device. A device including the video encoder 20 and / or video decoder 30 may include integrated circuits, microprocessors, and / or wireless communication devices such as cellular phones.

[0046] Figure 2 This is a block diagram of an exemplary video encoder 20 that can implement video decoding technology. The video encoder 20 can perform intra-frame decoding and inter-frame decoding on video blocks within a video slice. Intra-frame decoding relies on spatial prediction to reduce or remove spatial redundancy in the video within a given video frame or image. Inter-frame decoding relies on temporal prediction to reduce or remove temporal redundancy in the video within adjacent frames or images of a video sequence. The intra-frame mode (I-mode) can be any of several spatial-based decoding modes. The inter-frame mode (e.g., uni-directional / uni prediction (P-mode) or bi-prediction / bi prediction (B-mode)) can be any of several temporal-based decoding modes.

[0047] like Figure 2 As shown, the video encoder 20 receives the current video block within the video frame to be encoded. Figure 2 In the example, the video encoder 20 includes a mode selection unit 40, a reference frame memory 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy decoding unit 56. The mode selection unit 40 further includes a motion compensation unit 44, a motion estimation unit 42, an intra-frame prediction unit 46, and a segmentation unit 48. To reconstruct video blocks, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform unit 60, and a summer 62. It may also include a deblocking filter (…). Figure 2(Not shown) to filter block boundaries, thereby removing block artifacts from the reconstructed video. If necessary, the deblocking filter typically filters the output of summer 62. Other filters (in-loop or post-loop) besides the block filter can also be used. Such filters are not shown for simplicity, but if needed, they can filter the output of summer 50 (as in-loop filters).

[0048] During encoding, video encoder 20 receives video frames or stripes to be decoded. These frames or stripes can be divided into multiple video blocks. Motion estimation component 42 and motion compensation component 44 perform inter-frame prediction decoding on the received video blocks relative to one or more blocks in one or more reference frames to achieve temporal prediction. Intra-frame prediction unit 46 can also perform intra-frame prediction decoding on the received video blocks relative to one or more adjacent blocks in the same frame or strip as the video block to be decoded to achieve spatial prediction. Video encoder 20 can perform multiple decoding rounds, for example, selecting an appropriate decoding mode for each video data block.

[0049] Furthermore, segmentation unit 48 can segment video data blocks into sub-blocks based on evaluations of previous segmentation schemes in previous decoding rounds. For example, segmentation unit 48 can initially segment frames or stripes into largest coding units (LCUs), and then segment each LCU into sub-coding units (sub-CUs) based on rate-distortion analysis (e.g., rate-distortion optimization). Mode selection unit 40 can also generate a quadtree data structure indicating the segmentation of LCUs into sub-CUs. Leaf nodes (CUs) in the quadtree can include one or more prediction units (PUs) and one or more transform units (TUs).

[0050] This invention uses the term "block" to refer to any of the CU, PU, ​​or TU in an HEVC context or a similar data structure in other standard contexts (e.g., macroblocks and their subblocks in H.264 / AVC). A CU includes a decoder node, the PUs associated with that decoder node, and TUs. The size of the CU corresponds to the size of the decoder node and is square. The size of the CU can range from 8×8 pixels up to a maximum of 64×64 pixels or larger, representing a tree block size. Each CU can include one or more PUs and one or more TUs. The syntax data associated with the CU can describe, for example, dividing the CU into one or more PUs. The segmentation mode may differ depending on whether the CU is encoded in skip mode or direct mode, intra-prediction mode, or inter-prediction mode. PUs can be segmented into non-square blocks. The syntax data associated with the CU can also describe, for example, dividing the CU into one or more TUs according to a quadtree. TUs can be square or non-square (e.g., rectangular).

[0051] The mode selection unit 40 can select one of the intra-frame or inter-frame decoding modes based on error results, etc., and provide the obtained intra-frame or inter-frame decoded blocks to the adder 50 to generate residual block data, and provide them to the summer 62 to reconstruct the coded blocks for use as reference frames. The mode selection unit 40 also provides syntax elements (such as motion vectors, intra-frame mode indicators, segmentation information, and other such syntax information) to the entropy decoding unit 56.

[0052] The motion estimation unit 42 and the motion compensation unit 44 can be highly integrated, but are described separately for conceptual purposes. Motion estimation performed by the motion estimation unit 42 is the process of generating motion vectors used to estimate the motion of video blocks. These motion vectors can, for example, indicate the displacement of the PU of a video block within the current video frame or image relative to a predicted block within a reference frame (or other decoded unit) (relative to the currently decoded block within the current frame (or other decoded unit)). The predicted block is a block that closely matches the block to be decoded in terms of pixel differences, which can be determined by the sum of absolute differences (SAD), the sum of square differences (SSD), or other difference metrics. In some examples, the video encoder 20 can compute values ​​for sub-integer pixel positions of a reference image stored in the reference frame memory 64. For example, the video encoder 20 can interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference image. Therefore, the motion estimation unit 42 can perform motion searches relative to full pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision.

[0053] The motion estimation unit 42 calculates the motion vector of a PU by comparing the position of the PU in the video block in the inter-frame decoded strip with the position of the predicted block in the reference image. The reference image can be selected from a first reference image list (list 0) or a second reference image list (list 1), each list identifying one or more reference images stored in the reference frame memory 64. The motion estimation unit 42 sends the calculated motion vector to the entropy coding unit 56 and the motion compensation unit 44.

[0054] Motion compensation performed by motion compensation unit 44 may involve acquiring or generating prediction blocks based on motion vectors determined by motion estimation unit 42. Additionally, in some examples, motion estimation unit 42 and motion compensation unit 44 may be functionally integrated. After receiving the motion vector of the PU for the current video block, motion compensation unit 44 can immediately locate the prediction block pointed to by the motion vector in one of the reference image lists. Summer 50 forms a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being decoded, thus forming a pixel difference, as described below. Generally, motion estimation unit 42 performs motion estimation relative to the luminance component, and motion compensation unit 44 uses the motion vector calculated based on the luminance component for both the chroma and luminance components. Mode selection unit 40 may also generate syntax elements associated with video blocks and video stripes for use by video decoder 30 when decoding video blocks within a video stripe.

[0055] As described above, instead of the inter-frame prediction performed by the motion estimation unit 42 and the motion compensation unit 44, the intra-frame prediction unit 46 can perform intra-frame prediction for the current block. Specifically, the intra-frame prediction unit 46 can determine the intra-frame prediction mode to be used for encoding the current block. In some examples, the intra-frame prediction unit 46 can, for example, use various intra-frame prediction modes to encode the current block in separate coding rounds, and the intra-frame prediction unit 46 (or, in some examples, the mode selection unit 40) can select a suitable intra-frame prediction mode from the tested modes for use.

[0056] For example, intra-prediction unit 46 can use rate-distortion analysis for various tested intra-prediction modes to calculate rate-distortion values ​​and select the intra-prediction mode with the best rate-distortion characteristics among the tested modes. Rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and the original, uncoded block (which was previously encoded to produce a coded block), and determines the bit rate (i.e., the number of bits) used to produce the coded block. Intra-prediction unit 46 can calculate the ratio based on the distortion and rate of various coded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for that block.

[0057] Additionally, the intra-prediction unit 46 can be used to decode depth blocks in the depth image using a depth modeling mode (DMM). The mode selection unit 40 can determine whether the available DMM mode produces better decoding results than the intra-prediction mode and other DMM modes (e.g., using rate-distortion optimization (RDO)). Data of the texture image corresponding to the depth image can be stored in the reference frame memory 64. The motion estimation unit 42 and the motion compensation unit 44 can also be used to perform inter-frame prediction of depth blocks in the depth image.

[0058] After selecting the intra prediction mode for a block (e.g., a conventional intra prediction mode or one of the DMM modes), the intra prediction unit 46 can provide information to the entropy decoding unit 56 indicating the intra prediction mode selected for the block. The entropy decoding unit 56 can encode the information indicating the selected intra prediction mode. The video encoder 20 can carry the following in the transmitted bitstream: configuration data, which may include multiple intra prediction mode index tables and multiple modified intra prediction mode index tables (also called codeword maps); definitions of the coding contexts for various blocks; and indications of the most likely intra prediction mode to be used for each coding context, the intra prediction mode index tables, and the modified intra prediction mode index tables.

[0059] The video encoder 20 forms a residual video block by subtracting the prediction data from the mode selection unit 40 from the original video block being decoded. The summer 50 is one or more components that perform this subtraction operation.

[0060] Transform processing unit 52 applies a transform (such as a discrete cosine transform (DCT) or a conceptually similar transform) to the residual block, thereby producing a video block that includes the residual transform coefficient values. Transform processing unit 52 can perform other transforms conceptually similar to DCT. Wavelet transform, integer transform, subband transform, or other types of transforms can also be used.

[0061] Transform processing unit 52 applies a transform to the residual block, thereby generating a residual transform coefficient block. This transform converts the residual information from the pixel domain to the transform domain (e.g., the frequency domain). Transform processing unit 52 can then send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes these transform coefficients to further reduce the bit rate. This quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameters. In some examples, quantization unit 54 can then perform a scan on the matrix including the quantized transform coefficients. Alternatively, entropy coding unit 56 can perform the scan.

[0062] After quantization, entropy decoding unit 56 performs entropy decoding on the quantized transform coefficients. For example, entropy decoding unit 56 can perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) decoding, or other entropy decoding techniques. In the case of context-based entropy decoding, the context can be based on adjacent blocks. After entropy decoding is performed by entropy decoding unit 56, the encoded bitstream can be sent to another device (e.g., video decoder 30) or archived for later transmission or retrieval.

[0063] Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct a residual block in the pixel domain, for example, to be used later as a reference block. Motion compensation unit 44 can compute the reference block by adding the residual block to a prediction block of one of the frames in reference frame memory 64. Motion compensation unit 44 can also apply one or more interpolation filters to the reconstructed residual block to compute sub-integer pixel values ​​used in motion estimation. Summer 62 adds the reconstructed residual block to the motion-compensated prediction block generated by motion compensation unit 44 to produce a reconstructed video block, which is stored in reference frame memory 64. The reconstructed video block can be used by motion estimation unit 42 and motion compensation unit 44 as a reference block for inter-frame decoding of blocks in subsequent video frames.

[0064] Figure 3 This is a block diagram of an exemplary video decoder 30 that can implement video decoding technology. Figure 3 In the example, video decoder 30 includes an entropy decoding unit 70, a motion compensation unit 72, an intra-frame prediction unit 74, an inverse quantization unit 76, an inverse transform unit 78, a reference frame memory 82, and a summer 80. In some examples, video decoder 30 can perform decoding rounds, which are generally associated with a reference video encoder 20. Figure 2 The encoding rounds described are inverses. Motion compensation unit 72 can generate prediction data based on the motion vector received from entropy decoding unit 70, while intra-frame prediction unit 74 can generate prediction data based on the intra-frame prediction mode indicator received from entropy decoding unit 70.

[0065] During decoding, video decoder 30 receives an encoded video stream from video encoder 20, which represents video blocks and associated syntax elements within an encoded video stripe. Entropy decoding unit 70 in video decoder 30 performs entropy decoding on the stream to generate quantization coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. Entropy decoding unit 70 forwards the motion vectors and other syntax elements to motion compensation unit 72. Video decoder 30 can receive syntax elements at the video stripe level and / or video block level.

[0066] When a video strip is decoded into an intra-decoded (I) strip, the intra-prediction unit 74 generates prediction data for video blocks in the current video strip based on the indicated intra-prediction mode and data from previously decoded blocks in the current frame or image. When a video frame is decoded into an inter-decoded (i.e., B, P, or GPB) strip, the motion compensation unit 72 generates prediction blocks for video blocks in the current video strip based on motion vectors and other syntax elements received from the entropy decoding unit 70. These prediction blocks can be generated based on one of the reference images in one of the reference image lists. The video decoder 30 constructs the reference frame lists 0 and 1 using the default construction technique based on the reference images stored in the reference frame memory 82.

[0067] Motion compensation unit 72 determines prediction information for video blocks in the current video strip by parsing motion vectors and other syntax elements, and uses this prediction information to generate prediction blocks for the current video block being decoded. For example, motion compensation unit 72 uses some received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction) used to decode video blocks in the video strip, the inter-frame prediction strip type (e.g., B strip, P strip, or GPB strip), the construction information of one or more reference image lists in the strip, the motion vector of each inter-frame coded video block in the strip, the inter-frame prediction state of each inter-frame decoded video block in the strip, and other information used to decode video blocks in the current video strip.

[0068] The motion compensation unit 72 can also perform interpolation based on an interpolation filter. The motion compensation unit 72 can use the interpolation filter used by the video encoder 20 during the encoding of the video block to calculate the interpolated values ​​of the sub-integer pixels of the reference block. In this case, the motion compensation unit 72 can determine the interpolation filter used by the video encoder 20 based on the received syntax elements and use the interpolation filter to generate the prediction block.

[0069] Data corresponding to the texture image in the depth image can be stored in the reference frame memory 82. The motion compensation unit 72 can also be used to perform inter-frame prediction of depth blocks in the depth image.

[0070] Image and video compression has developed rapidly, with a diversification of decoding standards. These video decoding standards include ITU-T H.261, MPEG-1 Part 2 (International Organization for Standardization / International Electrotechnical Commission / IEC), ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, Advanced Video Coding (AVC) (also known as ITU-T H.264 or ISO / IEC MPEG-4 Part 10), and High Efficiency Video Coding (HEVC) (also known as ITU-T H.265 or MPEG-H Part 2). AVC includes extended versions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC), Multiview Video Coding plus Depth (MVC+D), and 3D AVC (3D-AVC). HEVC includes extended versions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC).

[0071] Versatile Video Coding (VVC) is a new video decoding standard developed by the Joint Video Experts Team (JVET), a consortium of ITU-T and ISO / IEC. At the time of writing, the latest working draft (WD) of VVC was included in JVET-K1001-v1. JVET document JVET-K0325-v3 includes updates to the advanced syntax of VVC.

[0072] Generally speaking, this invention describes technologies based on the VVC standard, which is currently under development. However, these technologies are also applied to other video / media codec specifications.

[0073] Video compression techniques perform spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove inherent redundancy in video sequences. For block-based video decoding, video strips (e.g., video images or portions of video images) can be segmented into video blocks, which may also be referred to as tree blocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or decoding nodes. Video blocks within an intra-frame decoded (I) strip of an image are encoded using spatial prediction with reference to reference pixels in adjacent blocks of the same image. Video blocks within an inter-frame decoded (P or B) strip of an image are encoded using spatial prediction with reference to reference pixels in adjacent blocks of the same image, or using temporal prediction with reference to reference pixels in other reference images. An image may be called a frame, and a reference image may be called a reference frame.

[0074] A prediction block for the block to be decoded is generated through spatial or temporal prediction. Residual data represents the pixel difference between the original block and the prediction block. The inter-frame decoded block is encoded based on motion vectors and residual data, where the motion vectors point to the block of reference pixels constituting the prediction block, and the residual data indicates the difference between the decoded block and the prediction block. The intra-frame decoded block is encoded based on the intra-frame decoding mode and residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain, generating residual transform coefficients that are then quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned to generate a one-dimensional vector of transform coefficients, and entropy decoding can be applied for further compression.

[0075] In video codec specifications, identification images serve multiple purposes, including serving as reference images in inter-frame prediction, images output from the decoded picture buffer (DPB), scaling motion vectors, and weighted prediction. In AVC and HEVC, images are identified by their picture order count (POC). In AVC and HEVC, images in the DPB can be marked as "for short-term reference," "for long-term reference," or "not for reference." Once an image is marked as "not for reference," it can no longer be used for prediction. When an image is no longer needed for output, it can be removed from the DPB.

[0076] In AVC, there are two types of reference images: short-term reference images and long-term reference images. When a reference image is no longer used for prediction, it can be marked as "not used for reference." The transitions between these three states (short-term reference, long-term reference, not used for reference) are controlled through the decoded reference image marking process. Two alternative decoded reference image marking mechanisms exist: an implicit sliding window procedure and an explicit memory management control operation (MMCO) procedure. When the number of reference frames equals a given maximum number (max_num_ref_frames in the sequence parameter set (SPS),) the sliding window procedure marks short-term reference images as "not used for reference." Short-term reference images are stored in a first-in, first-out (FIFO) manner, ensuring that the most recently decoded short-term image is stored in the DPB.

[0077] An explicit MMCO procedure can include multiple MMCO commands. An MMCO command can mark one or more short-term or long-term reference images as "not for reference", mark all images as "not for reference", or mark the current reference image or an existing short-term reference image as a long-term reference image, and then assign a long-term image index to the long-term reference image.

[0078] In AVC, reference image identification operations, as well as the processes of outputting and deleting images from the DPB, are performed after the image has been decoded.

[0079] HEVC introduces a different method for managing reference pictures, called the Reference Pictureset (RPS). The fundamental difference between the RPS concept and the MMCO / sliding window procedure in AVC is that each specific strip has a complete set of reference pictures used by the current picture or any subsequent pictures. Therefore, it indicates the complete set of all pictures that must be kept in the DPB for use by the current or subsequent pictures. This differs from the AVC scheme, which only indicates relative changes in the DPB. Using the RPS concept, information from earlier pictures in the decoding sequence is not needed to maintain the correct state of the reference pictures in the DPB.

[0080] To fully leverage the advantages of RPS and improve error resilience, the order of image decoding and DPB operations in HEVC has been changed compared to AVC. In AVC, image identification and buffering operations (both outputting and deleting the decoded image from the DPB) are typically performed after decoding the current image. In HEVC, the RPS is first decoded from the stripe header of the current image, and then image identification and buffering operations are typically performed before decoding the current image.

[0081] Each slice header in HEVC must include a parameter indicating the RPS (Reference Position Segment) of the image containing the slice. The only exception is when no RPS is indicated for an Instantaneous Decoding Refresh (IDR) slice, in which case the RPS is inferred to be empty. For I-slices that do not belong to IDR images, an RPS can be provided even if these I-slices belong to I-images, because there may be images decoded after the I-images, which are used for inter-frame prediction based on images decoded before the I-images. The number of images in the RPS does not exceed the DPB (Device Per Frame) size limit indicated by the syntax element sps_max_dec_pic_buffering in the SPS.

[0082] Each image is associated with a POC value indicating the output order. The strip header includes a fixed-length codeword `pic_order_cnt_lsb`, representing the least significant bit (LSB) of the complete POC value, also known as the POC LSB. The length of the codeword is indicated in the SPS, and the length can be, for example, between 4 and 16 bits. The RPS uses the POC to identify the reference image. In addition to the POC value, each strip header also directly includes or uses the decoded representation of the POC value (or LSB) of each image in the RPS within the SPS.

[0083] Each image's RPS includes five distinct lists of reference images, also known as five RPS subsets. RefPicSetStCurrBefore includes all short-term reference images that precede the current image in both decoding and output order and can be used in the inter-frame prediction of the current image. RefPicSetStCurrAfter includes all short-term reference images that precede the current image in decoding order and follow the current image in output order and can be used in the inter-frame prediction of the current image. RefPicSetStFoll includes all short-term reference images that can be used in the inter-frame prediction of one or more images following the current image in decoding order but are not used in the inter-frame prediction of the current image. RefPicSetLtCurr includes all long-term reference images that can be used in the inter-frame prediction of the current image. RefPicSetLtFoll includes all long-term reference images that can be used in the inter-frame prediction of one or more images following the current image in decoding order but are not used in the inter-frame prediction of the current image.

[0084] Up to three loops are used to indicate the RPS, iterating over different types of reference images: short-term reference images with a POC value less than the current image, short-term reference images with a POC value greater than the current image, and long-term reference images. Additionally, a flag (used_by_curr_pic_X_flag) is sent for each reference image indicating whether it is referenced by the current image (included in the lists RefPicSetStCurrBefore, RefPicSetStCurrAfter, or RefPicSetLtCurr, or in the lists RefPicSetStFoll or RefPicSetLtFoll).

[0085] Figure 4 RPS 400 is shown, which includes the current image B14, and the current image B14 includes table entries (e.g., images) in all subsets 402 of RPS 400. Figure 4 In the example, the current image B14 includes only one image from each of the five subsets 402 (also known as RPS subsets). P8 is the image in subset 402 (called RefPicSetStCurrBefore) because it is earlier in the output order and used by B14. P12 is the image in subset 402 (called RefPicSetStCurrAfter) because it is later in the output order and used by B14. P13 is the image in subset 402 (called RefPicSetStFoll) because it is a short-term reference image that B14 does not use (but must be retained in the DPB because B15 will use it). P4 is the image in subset 402 (called RefPicSetStCurrBefore) because it is a long-term reference image used by B14. I0 is the image in subset 402 (called RefPicSetLtFoll) because it is a long-term reference image that is not used by the current image (but must be kept in DPB because B15 will use it).

[0086] Short-term reference images in RPS 400 can be directly included in the stripe header. Alternatively, the stripe header can include only syntax elements indicating the index, thus referencing a predefined list of RPSs sent in the active SPS. Short-term reference images in RPS 402 can be indicated using either of two different schemes: inter-frame RPS as described below or intra-frame RPS as described here. When using intra-frame RPS, the indicators `num_negative_pics` and `num_positive_pics` represent the lengths of two distinct lists of reference images. These lists respectively include reference images with negative and positive POC differences from the current image. Each element in both lists is encoded using a variable-length code that represents the difference between each element in the list and the POC value of the previous element minus one. For the first image in each list, the indicator is the POC value of the current image minus one.

[0087] When encoding a cyclic RPS in a sequence parameter set, it is possible to encode elements in an RPS (e.g., RPS 400) by referencing another RPS already encoded in the sequence parameter set. This is called inter-frame RPS. Since all RPSs in the sequence parameter set reside in the same network abstraction layer (NAL) unit, this method does not introduce error stability issues. The inter-frame RPS syntax is based on the fact that the RPS of the current image can be predicted from the RPS of previously decoded images. This is because all reference images for the current image are either reference images of the previous image or previously decoded images. It is only necessary to indicate which of these images are reference images and used to predict the current image. Therefore, the syntax includes the following: an index pointing to the RPS to be used as the prediction value; delta_POC, the delta_POC to be added to the prediction value, the increment POC for the current RPS; and a set of indicators indicating which images are reference images and whether these images are only used to predict subsequent images.

[0088] Encoders wishing to use long-term reference images must set the SPS syntax element `long_term_ref_pics_present_flag` to 1. The long-term reference images can then be indicated in the strip header via fixed-length codewords `poc_lsb_lt`, which represent the least significant bits of the complete POC value for each long-term image. Each `poc_lsb_lt` is a copy of the codeword `pic_order_cnt_lsb` indicating a specific long-term image. It is also possible to indicate a set of long-term images in the SPS as a list of POC LSB values. The POC LSBs of the long-term images can then be indicated in the strip header as indices to this list.

[0089] Additionally, the syntax element delta_poc_msb_cycle_lt_minus1 can be specified, enabling the calculation of the complete POC distance of the long-term reference image relative to the current image. It is required that the codeword delta_poc_msb_cycle_lt_minus1 be specified for each long-term reference image, and that each long-term reference image has the same POCLSB value as any other reference image in the RPS.

[0090] For reference image identification in HEVC, many images typically exist in the DPB before image decoding. Some of these images can be used for prediction and are therefore marked as "used for reference." Other images cannot be used for prediction but are awaiting output and are therefore marked as "not used for reference." After parsing the strip header, the image identification process is performed first, and then the strip data is decoded. Images in the DPB that are marked as "used for reference" but not included in the RPS are marked as "not used for reference." When used_by_curr_pic_X_flag is 0, images that are not in the DPB but are included in the reference image set are ignored. However, when used_by_curr_pic_X_flag is 1, the reference image is expected to be used to predict the current image, but this reference image is missing. In this case, it is inferred that the image was unintentionally lost, and the decoder needs to take appropriate measures to resolve this issue.

[0091] After decoding the current image, the reference image is identified as "for short-term reference".

[0092] The following describes the construction of the reference image list in HEVC. In HEVC, the term "inter-frame prediction" is used to refer to a prediction derived from data elements (such as pixel values ​​or motion vectors) of reference images other than the currently decoded image. As in AVC, an image can be predicted based on multiple reference images. The reference images used for inter-frame prediction are arranged in one or more reference image lists. A reference index is used to identify which reference images in the list need to be used to generate the prediction signal.

[0093] P-strips use a single list of reference images (list 0), while B-strips use two lists of reference images (list 0 and list 1). Similar to AVC, reference image list construction in HEVC includes reference image list initialization and reference image list modification.

[0094] In AVC, the initialization process for list 0 of P-strips (using decoding order) and B-strips (using output order) is different. In HEVC, both cases use output order.

[0095] The reference image list initialization creates default lists 0 and 1 (if the strip is a B strip) based on the following three RPS subsets: RefPicSetStCurrBefore, RefPicSetStCurrAfter, and RefPicSetLtCurr. First, the shorter images that appear earlier (later) in the output order are inserted into list 0 (list 1) in ascending order of their POC distance from the current image. Then, the shorter images that appear later (earlier) in the output order are inserted into list 0 (list 1) in ascending order of their POC distance from the current image. Finally, the longer images are inserted at the end of list 0 (list 1). In terms of RPS, for list 0, entries from RefPicSetStCurrBefore are inserted into the initial list, followed by entries from RefPicSetStCurrAfter. Afterward, entries from RefPicSetLtCurr are added (if they exist).

[0096] In HEVC, when the number of entries in the list is less than the number of targets for the active reference image (indicated in the image parameter set or strip header), the above process is repeated (the reference images already added to the reference image list are added again). When the number of entries exceeds the number of targets, the list is truncated.

[0097] After initializing the reference image list, it can be modified to arrange the reference images of the current image in any order (including cases where a particular reference image can appear in multiple positions in the list) according to the reference image list modification commands. When the flag indicating the existence of list modification is set to 1, a fixed number of commands (equal to the target number of entries in the reference image list) are executed, and each command inserts one entry into the reference image list. In the commands, the reference image is identified by its index in the reference image list of the current image, derived from the RPS indicator. This differs from reference image list modification in H.264 / AVC, where images are identified by image number (derived from the syntax element frame_num) or long-term reference image index, and may require fewer commands, such as those used to swap the first two entries in the initial list or to insert an entry at the beginning of the initial list and move other entries.

[0098] The reference image list cannot include any reference image with a TemporalId greater than the current image. The HEVC bitstream may consist of several temporal sub-layers. Each NAL unit belongs to a specific sub-layer and is indicated by its TemporalId (equal to temporal_id_plus1 – 1).

[0099] Reference image management is based directly on a list of reference images. JCT-VC document JCTVC-G643 includes a method for managing reference images in DPB by directly using three lists of reference images (reference image list 0, reference image list 1, and idle reference image list), thereby avoiding the instruction and decoding process, including (1) the sliding window and MMCO process in AVC and the reference image list initialization and modification process, or (2) the reference image set and reference image list initialization and modification process in HEVC.

[0100] Reference image management methods may have several problems. The AVC method involves sliding windows, the MMCO process, and the initialization and modification of the reference image list, all of which are highly complex. Furthermore, image loss can lead to the loss of the DPB's state—that is, which images in the DPB are used for further inter-frame prediction reference purposes. The HEVC method does not suffer from the DPB state loss problem. However, the HEVC method involves a complex reference image set indication and derivation process, as well as a complex reference image list initialization and modification process. The method in JCTVC-G643 directly uses three reference image lists (Reference Image List 0, Reference Image List 1, and Idle Reference Image List) to manage reference images in the DPB, involving the following aspects: a third reference image list, i.e., the idle reference image list; decoding the POC difference into two parts: a "short-term" part and a "long-term" part decoded by ue(v); the POC granularity based on TemporalId used for POC difference decoding, using the two parts of the POC difference decoding to determine whether it is identified as "for short-term reference" or "for long-term reference"; a subset description of the reference image list, which can indicate the reference image list by removing a reference image from a reference image list description that was added earlier; the reference image list copy mode enabled by the syntax element ref_pic_list_copy_flag; and the reference image list description process. Each of the above aspects makes the method complex. In addition, the decoding process of the reference image list in JCTVC-G643 is also complex. Indicating long-term reference images may require indicating the POC cycle in the stripe header. This is inefficient.

[0101] To address the aforementioned issues, this paper discloses the following technical solutions, each of which can be applied individually, while some of these solutions can be applied in combination. (1) Directly identifying reference images based on two reference image lists (i.e., reference image list 0 and reference image list 1). (1a) Indicating information used to derive the two reference image lists based on syntax elements and syntax structures in the SPS, PPS, and / or strip header. (1b) Explicitly indicating two reference image lists for an image in the reference image list structure. (1b.i) One or more reference image list structures can be indicated in the SPS, and each reference image list structure can be referenced by an index in the strip header. (1b.ii) Reference image lists 0 and 1 can be directly indicated in the strip header. (2) Indicating information used to derive the two reference image lists for all types of stripes, i.e., B (bidirectional prediction) stripes, P (unidirectional prediction) stripes, and I (intra-frame) stripes. The term "slice" refers to a set of decoding tree units, such as slices in HEVC or the latest VVC WD; slices can also refer to some other set of decoding tree units, such as tiles in HEVC. (3) Generate two reference image lists for all types of slices, namely B slices, P slices and I slices. (4) Directly construct two reference image lists without the need for reference image list initialization and modification processes. (5) In the two reference image lists, the reference image used for inter-frame prediction of the current image can only be referenced by multiple entries at the beginning of the list. These entries are called active entries in the list, while other entries are called inactive entries in the list. The total number of entries in the list and the number of active entries can be derived. (6) An image referenced by an inactive entry in the reference image list cannot be referenced by other entries in the reference image list or any entry in any other reference image list. (7) Long-term reference images are identified only by a certain number of POC LSBs, which may be greater than the number of POC LSBs used to derive the POC value indicated in the strip header and also indicated in the SPS. (8) The reference image list structure is indicated only in the strip header; both short-term and long-term reference images are identified by their POC LSBs, which are represented by a number of bits, which is different from the number of bits used to derive the POC value indicated in the strip header; the number of bits used to identify the short-term and long-term reference images may be different. (9) The reference image list structure is indicated only in the strip header, without distinguishing between short-term and long-term reference images; all reference images are referred to as reference images, and reference images are identified by their POC LSBs, which may be represented by a number of bits, which is different from the number of bits used to derive the POC value indicated in the strip header.

[0102] The first embodiment of the present invention is provided below. The following description relates to the latest VVC WD. In this embodiment, the SPS indicates the two sets of reference image list structures corresponding to reference image list 0 and reference image list 1, respectively.

[0103] The following are definitions of some terms used in this paper. Intra-random access point (IRAP) image: A decoded image whose nal_unit_type is IRAP_NUT for each video coding layer (VCL) NAL unit. Non-IRAP image: A decoded image whose nal_unit_type is NON_IRAP_NUT for each VCL NAL unit. Reference image list: A list of reference images used for inter-frame prediction in P-slices or B-slices. Two reference image lists are generated for each slice in a non-IRAP image: Reference image list 0 and Reference image list 1. The unique set of images referenced by all entries in the two reference image lists associated with an image includes all reference images that can be used for inter-frame prediction of the associated image or any image whose decoding order follows the associated image. When decoding P-slice slice data, only Reference image list 0 is used for inter-frame prediction. When decoding B-slice slice data, both reference image lists are used for inter-frame prediction. When decoding I-slice slice data, no reference image list is used for inter-frame prediction. Long-term reference picture (LTRP): An image identified as "used for long-term reference". Short-term reference picture (STRP): An image identified as "used for short-term reference".

[0104] Section 8.3.3, “Decoding process for reference picture marking,” in VVC, Section 8.3.2, “Decoding process for reference picture set,” in HEVC, and Section 7.4.3.3, “Decoded reference picture marking semantics,” in AVC define the terms “for short-term reference,” “for long-term reference,” or “not for reference.” The terms used herein have the same meaning.

[0105] The relevant syntax and semantics used in the first embodiment are provided below.

[0106] NAL Unit Header Syntax

[0107] Sequence Parameter Set Raw Byte Sequence Payload (RBSP) Syntax

[0108] Image Parameter Set (RBSP) Syntax

[0109] Striped header syntax

[0110] Reference image list structure syntax

[0111] NAL Unit Header Semantics The forbidden_zero_bit is equal to 0. The nal_unit_type indicates the type of RBSP data structure included in the NAL unit.

[0112] Table 7-1: NAL Unit Type Codes and NAL Unit Type Categories

[0113] `nuh_temporal_id_plus1` minus 1 represents the time identifier of the NAL unit. The value of `nuh_temporal_id_plus1` is not equal to 0. The variable `TemporalId` is represented as follows: `TemporalId = nuh_temporal_id_plus1 – 1`. When `nal_unit_type` is `IRAP_NUT`, the decoded strip belongs to the IRAP image, and `TemporalId` equals 0. All VCL NAL units in the access unit have the same `TemporalId` value. The `TemporalId` value of the decoded image or access unit is the `TemporalId` value of the VCL NAL unit in that decoded image or access unit. The `TemporalId` value of non-VCL NAL units is constrained as follows: if `nal_unit_type` is `SPS_NUT`, then `TemporalId` equals 0, and the `TemporalId` of the access unit including the NAL unit is equal to 0. Otherwise, if nal_unit_type is EOS_NUT or EOB_NUT, then TemporalId equals 0. Otherwise, TemporalId is greater than or equal to the TemporalId of the access unit that includes the NAL unit. When the NAL unit is a non-VCLNAL unit, the TemporalId value is equal to the minimum of the TemporalId values ​​of all access units to which the non-VCL NAL unit applies. When nal_unit_type is PPS_NUT, TemporalId can be greater than or equal to the TemporalId of the access unit that includes the NAL unit because all picture parameter sets (PPS) can be included at the beginning of the bitstream, where the TemporalId of the first decoded picture is equal to 0. When nal_unit_type is PREFIX_SEI_NUT or SUFFIX_SEI_NUT, TemporalId can be greater than or equal to the TemporalId of the access unit that includes the NAL unit, because the SEI NAL unit can include information applied to a subset of the bitstream that includes the access unit, where the TemporalId values ​​of these access units are greater than the TemporalId of the access unit that includes the SEI NAL unit. nuh_reserved_zero_7bits is equal to '0000000'. Other values ​​for nuh_reserved_zero_7bits may be specified by ITU-T or ISO / IEC in the future. The decoder ignores (i.e., removes and discards) NAL units whose nuh_reserved_zero_7bits value is not equal to '0000000'.

[0114] Sequence Parameter Set RBSP Semantics log2_max_pic_order_cnt_lsb_minus4 represents the value of the variable MaxPicOrderCntLsb used in the decoding process for image sequence numbering, as shown below: MaxPicOrderCntLsb = 2 ( log2_max_pic_order_cnt_lsb_minus4 + 4 ) The value of `log2_max_pic_order_cnt_lsb_minus4` ranges from 0 to 12 (inclusive). `sps_max_dec_pic_buffering_minus1` incremented by 1 indicates the maximum size of the decoded image buffer required by CVS, in units of image storage buffers. The value of `sps_max_dec_pic_buffering_minus1` ranges from 0 to `MaxDpbSize` – 1 (inclusive), where `MaxDpbSize` is as shown elsewhere. `long_term_ref_pics_flag` being 0 indicates that no LTRP is used for inter-frame prediction of any decoded images in CVS. `long_term_ref_pics_flag` being 1 indicates that LTRP can be used for inter-frame prediction of one or more decoded images in CVS. `additional_lt_poc_lsb` represents the value of the variable `MaxLtPicOrderCntLsb` used in the decoding process for the reference image list, as shown below: `MaxLtPicOrderCntLsb = 2`. ( log2_max_pic_order_cnt_lsb_minus4 + 4 + additional_lt_poc_lsb )The value of additional_lt_poc_lsb ranges from 0 to 32 – log2_max_pic_order_cnt_lsb_minus4 – 4 (inclusive). If it does not exist, the value of additional_lt_poc_lsb is assumed to be 0. num_ref_pic_lists_in_sps[i] represents the number of syntax structures ref_pic_list_struct(listIdx,rplsIdx, ltrpFlag) with listIdx equal to i included in the SPS. The value of num_ref_pic_lists_in_sps[i] ranges from 0 to 64 (inclusive). For each value of listIdx (equal to 0 or 1), the decoder needs to allocate memory for all syntax structures num_ref_pic_lists_in_sps[i] + 1 ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag), because a syntax structure ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag) can be directly indicated in the strip header of the current image.

[0115] Image parameter set RBSP semantics When i equals 0, incrementing 1 in num_ref_idx_default_active_minus1[i] indicates the inferred value of the variable NumRefIdxActive[0] for the P-band or B-band where num_ref_idx_active_override_flag is 0; when i equals 1, incrementing 1 in num_ref_idx_default_active_minus1[i] indicates the inferred value of NumRefIdxActive[1] for the B-band where num_ref_idx_active_override_flag is 0. The value range of num_ref_idx_default_active_minus1[i] is 0~14 (inclusive).

[0116] Strip Header Semantics If present, the values ​​of each of the slice_pic_parameter_set_id and slice_pic_order_cnt_lsb syntax elements are the same across all slice headers of the decoded image. slice_type indicates the decoding type of the slice, as shown in Table 7-3.

[0117] Table 7-3: Relationship between name and slice_type

[0118] When nal_unit_type is IRAP_NUT, i.e., when the image is an IRAP image, slice_type is 2. slice_pic_order_cnt_lsb represents the image sequence number of the current image modulo MaxPicOrderCntLsb. The length of the syntax element slice_pic_order_cnt_lsb is (log2_max_pic_order_cnt_lsb_minus4 + 4) bits. The value range of slice_pic_order_cnt_lsb is 0 to MaxPicOrderCntLsb – 1 (inclusive). If it does not exist, slice_pic_order_cnt_lsb is assumed to be equal to 0. ref_pic_list_sps_flag[i] is equal to 1, indicating that the reference image list i of the current image is derived from one of the syntax structures ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag) in the activated SPS where listIdx is equal to i. `ref_pic_list_sps_flag[i]` equals 0, indicating that the reference image list `i` for the current image is derived from the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)` whose `listIdx` is directly included in the strip header of the current image. When `num_ref_pic_lists_in_sps[i]` equals 0, the value of `ref_pic_list_sps_flag[i]` is 0. `ref_pic_list_idx[i]` represents the index of the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)` used to derive the reference image list `i` for the current image, whose `listIdx` is equal to `i`, within the list of syntax structures `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)` whose `listIdx` is equal to `i` included in the active SPS. The syntax element `ref_pic_list_idx[i]` is represented by `Ceil(Log2(num_ref_pic_lists_in_sps[i]))` bits. If it does not exist, the value of `ref_pic_list_idx[i]` is assumed to be 0. The value of `ref_pic_list_idx[i]` ranges from 0 to `num_ref_pic_lists_in_sps[i] – 1` (inclusive).The value of num_ref_idx_active_override_flag is 1, indicating that the P-strip and B-strip have the syntax element num_ref_idx_active_minus1[0], while the B-strip has the syntax element num_ref_idx_active_minus1[1]. The value of num_ref_idx_active_override_flag is 0, indicating that the syntax elements num_ref_idx_active_minus1[0] and num_ref_idx_active_minus1[1] do not exist. If they exist, then num_ref_idx_active_minus1[i] represents the value of the variable NumRefIdxActive[i], as shown below: NumRefIdxActive[i] = num_ref_idx_active_minus1[i] + 1. The value range of num_ref_idx_active_minus1[i] is 0~14 (inclusive).

[0119] The value of NumRefIdxActive[i] – 1 represents the maximum reference index in the reference image list i that can be used to decode the slice. When the value of NumRefIdxActive[i] is equal to 0, there is no reference index in the reference image list i that can be used to decode the slice. When i is equal to 0 or 1, if the current slice is a B slice and num_ref_idx_active_override_flag is 0, then it is inferred that NumRefIdxActive[i] is equal to num_ref_idx_default_active_minus1[i] + 1. If the current slice is a P slice and num_ref_idx_active_override_flag is 0, then it is inferred that NumRefIdxActive[0] is equal to num_ref_idx_default_active_minus1[0] + 1. If the current slice is a P slice, then it is inferred that NumRefIdxActive[1] is equal to 0. If the current stripe is an I stripe, then it is inferred that NumRefIdxActive[0] and NumRefIdxActive[1] are both equal to 0.

[0120] Alternatively, if i equals 0 or 1, assume rplsIdx1 is set to ref_pic_list_sps_flag[i] ? ref_pic_list_idx[i] : num_ref_pic_lists_in_sps[i], and numRpEntries[i] is set to num_strp_entries[i][rplsIdx1] + num_ltrp_entries[i][rplsIdx1]. When NumRefIdxActive[i] is greater than numRpEntries[i], the value of NumRefIdxActive[i] is set to numRpEntries[i].

[0121] Reference image list structure semantics The syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)` can exist in either the SPS or the strip header. The specific meaning of this syntax structure depends on whether it is included in the strip header or the SPS: if it exists in the strip header, then the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)` represents the list of reference images `listIdx` for the current image (including the images in the strip). Otherwise (in the SPS), the syntax structure ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag) represents a candidate for the reference image list listIdx. The term "current image" in the semantics defined in the remainder of this section refers to: (1) each image including one or more stripes, wherein the ref_pic_list_idx[listIdx] included by one or more stripes is equal to the index in the list of the syntax structure ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag) included in the SPS; and (2) each image in the CVS, wherein the CVS has the same SPS as the active SPS. num_strp_entries[listIdx][rplsIdx] represents the number of STRP entries in the syntax structure ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag). `num_ltrp_entries[ listIdx ][ rplsIdx ]` represents the number of LTRP entries in the syntax structure `ref_pic_list_struct( listIdx, rplsIdx, ltrpFlag )`. If it does not exist, the value of `num_ltrp_entries[ listIdx ][ rplsIdx ]` is inferred to be 0. The derivation of the variable `NumEntriesInList[ listIdx ][ rplsIdx ]` is as follows: `NumEntriesInList[ listIdx ][ rplsIdx ] = num_strp_entries[ listIdx ][ rplsIdx ] + num_ltrp_entries[ listIdx ][ rplsIdx ]`. The value range of `NumEntriesInList[ listIdx ][ rplsIdx ]` is 0 to `sps_max_dec_pic_buffering_minus1` (inclusive).If `lt_ref_pic_flag[listIdx][rplsIdx][i]` equals 1, it indicates that the i-th entry in the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)` is an LTRP entry. If `lt_ref_pic_flag[listIdx][rplsIdx][i]` equals 0, it indicates that the i-th entry in the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)` is a STRP entry. If it does not exist, then it is inferred that the value of `lt_ref_pic_flag[listIdx][rplsIdx][i]` is 0. The requirement for stream consistency is: the sum of lt_ref_pic_flag[listIdx][rplsIdx][i] (where all values ​​of i are in the range 0 to NumEntriesInList[listIdx][rplsIdx] – 1 (inclusive)) is equal to num_ltrp_entries[listIdx][rplsIdx]. When the i-th entry is the first STRP entry in the syntax structure ref_pic_list_struct( rplsIdx, ltrpFlag ), delta_poc_st[ listIdx ][ rplsIdx ][ i ] represents the difference between the image sequence number value of the current image and the image referenced by the i-th entry. Alternatively, when the i-th entry is a STRP entry in the syntax structure ref_pic_list_struct( rplsIdx, ltrpFlag ) but not the first STRP entry, delta_poc_st[ listIdx ][ rplsIdx ][ i ] represents the difference between the image sequence number value of the image referenced by the i-th entry in the syntax structure ref_pic_list_struct( listIdx,rplsIdx, ltrpFlag ) and the image referenced by the previous STRP entry. The value range of delta_poc_st[ listIdx ][ rplsIdx ][ i ] is -2. 15 ~2 15– 1 (inclusive). `poc_lsb_lt[listIdx][rplsIdx][i]` represents the value modulo `MaxLtPicOrderCntLsb`, which is the image sequence number of the image referenced by the `i`th entry in the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)`. The length of the syntax element `poc_lsb_lt[listIdx][rplsIdx][i]` is Log2(MaxLtPicOrderCntLsb) bits.

[0122] The decoding process is described below. The decoding process for the current image CurrPic is as follows. The decoding of the NAL unit is described in detail below. The following process uses the syntax elements of the stripe header layer and higher layers to describe the following decoding process. Derivation of variables and functions related to the image sequence number. Called only for the first strip in the image. At the beginning of the decoding process for each strip in a non-IRAP image, the decoding process for constructing the reference image list is called to derive reference image list 0 (RefPicList[0]) and reference image list 1 (RefPicList[1]). The decoding process for reference image identification is called, where the reference image can be identified as "not for reference" or "for long-term reference". Called only for the first strip in the image. The decoding process for decoding tree unit, scaling, transformation, in-loop filtering, etc. is called. After all strips in the current image have been decoded, the current decoded image is identified as "for short-term reference".

[0123] The NAL unit decoding process is described below. The input to this process is the NAL unit of the current image and its associated non-VCL NAL units. The output of this process is the parsed RBSP syntax structure encapsulated within the NAL unit. The decoding process for each NAL unit extracts the RBSP syntax structure from the NAL unit and then parses the RBSP syntax structure.

[0124] The following describes the stripe decoding process, including the decoding process for image sequence numbering. The output of this process is the image sequence number PicOrderCntVal for the current image. The image sequence number is used for image identification, derivation of motion parameters in merge mode, motion vector prediction, and decoder consistency checks. Each decoded image is associated with an image sequence number variable (denoted as PicOrderCntVal). If the current image is not an IRAP image, the variables prevPicOrderCntLsb and prevPicOrderCntMsb are derived as follows: Assume prevTid0Pic is the previous image in the decoding sequence where TemporalId equals 0. Set the variable prevPicOrderCntLsb to slice_pic_order_cnt_lsb of prevTid0Pic. Set the variable prevPicOrderCntMsb to PicOrderCntMsb of prevTid0Pic.

[0125] The derivation of the variable PicOrderCntMsb for the current image is as follows: If the current image is an IRAP image, then PicOrderCntMsb is set to 0. Otherwise, the derivation of PicOrderCntMsb is as follows: if( ( slice_pic_order_cnt_lsb <prevpicordercntlsb )&& ( prevpicordercntlsb – slice_pic_order_cnt_lsb )≥(maxpicordercntlsb 2 ) picordercntmsb="prevPicOrderCntMsb" + maxpicordercntlsb else if( (slice_pic_order_cnt_lsb>prevPicOrderCntLsb )&& ( ( slice_pic_order_cnt_lsb – prevPicOrderCntLsb )>(MaxPicOrderCntLsb / 2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb – MaxPicOrderCntLsb else PicOrderCntMsb = prevPicOrderCntMsb The derivation of PicOrderCntVal is as follows: PicOrderCntVal = PicOrderCntMsb + slice_pic_order_cnt_lsb.

[0126] Since the slice_pic_order_cnt_lsb of the inferred IRAP image is 0, and both prevPicOrderCntLsb and prevPicOrderCntMsb are set to 0, the PicOrderCntVal of all IRAP images is equal to 0. The value range of PicOrderCntVal is -2. 31 ~2 31 – 1 (inclusive of end values). In a CVS, the PicOrderCntVal values ​​of any two decoded images are not the same.

[0127] At any point during the decoding process, the values ​​of PicOrderCntVal & (MaxLtPicOrderCntLsb – 1) for any two reference images in the DPB are not identical. The function PicOrderCnt(picX) is expressed as: PicOrderCnt(picX) = PicOrderCntVal of image picX. The function DiffPicOrderCnt(picA, picB) is expressed as: DiffPicOrderCnt(picA, picB) = PicOrderCnt(picA) – PicOrderCnt(picB). The bitstream does not include values ​​of DiffPicOrderCnt(picA, picB) used during decoding that are outside the range – 2. 15 ~2 15 - Data within 1 (inclusive). Assuming X is the current image, and Y and Z are two other images in the same coded video sequence (CVS), when both DiffPicOrderCnt(X, Y) and DiffPicOrderCnt(X, Z) are positive or both are negative, it is assumed that Y and Z are in the same output order direction relative to X.

[0128] The following describes the decoding process used for constructing the reference image list. This process is invoked at the beginning of the decoding process for each strip in a non-IRAP image. Reference images are addressed by reference indices. A reference index is an index in the reference image list. When decoding I stripe, the reference image list is not used to decode the stripe data. When decoding P stripe, only reference image list 0 (i.e., RefPicList[0]) is used to decode the stripe data. When decoding B stripe, both reference image list 0 and reference image list 1 (i.e., RefPicList[1]) are used to decode the stripe data. At the beginning of the decoding process for each strip in a non-IRAP image, the reference image lists RefPicList[0] and RefPicList[1] are derived. These two reference image lists are used to identify reference images or decode stripe data. If the I-band in a non-IRAP image is not the first band in the image, then RefPicList[0] and RefPicList[1] can be derived to check bitstream consistency. However, it is not necessary to derive RefPicList[0] and RefPicList[1] when decoding the current image or images that are decoded after the current image. If the P-band is not the first band in the image, then RefPicList[1] can be derived to check bitstream consistency. However, it is not necessary to derive RefPicList[1] when decoding the current image or images that are decoded after the current image. The construction methods of the reference image lists RefPicList[0] and RefPicList[1] are as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] for( j = 0, pocBase = PicOrderCntVal; j<NumEntriesInList[ i ][RplsIdx[ i ]]; j++) { if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ]][ j ]) { RefPicPocList[ i ][ j ]= pocBase – delta_poc_st[ i ][ RplsIdx[ i ]][j ] if( there is a reference picture picA in the DPB with PicOrderCntValequal to RefPicPocList[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” pocBase = RefPicPocList[ i ][ j ] } else { if( there is a reference picA in the DPB with PicOrderCntVal&(MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ i ][ RplsIdx[ i ]][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” } } } In each case where i equals 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are called active entries in RefPicList[i], and the other entries in RefPicList[i] are called inactive entries in RefPicList[i]. If lt_ref_pic_flag[i][RplsIdx[i]][j] equals 0, then each entry in RefPicList[i][j] (j is in the range 0~NumEntriesInList[i][RplsIdx[i]]–1 (inclusive)) is called a STRP entry; otherwise, it is called an LTRP entry. A particular image may be referenced by both entries in RefPicList[0] and entries in RefPicList[1]. A particular image may also be referenced by multiple entries in RefPicList[0] or multiple entries in RefPicList[1]. The active entries in RefPicList[0] and RefPicList[1] collectively reference all reference images that can be used for inter-frame prediction of the current image and one or more images whose decoding order is after the current image. The inactive entries in RefPicList[0] and RefPicList[1] collectively reference all reference images that are not used for inter-frame prediction of the current image but can be used for inter-frame prediction of one or more images whose decoding order is after the current image. There may be one or more entries in RefPicList[0] or RefPicList[1] that are equal to "no reference image" because there is no corresponding image in the DPB. Ignore each inactive entry in RefPicList[0] or RefPicList[0] that is equal to "no reference image". Infer unintentional image loss for each active entry in RefPicList[0] or RefPicList[1] that is equal to "no reference image".

[0129] The requirement for stream consistency is to use the following constraints: for each i equal to 0 or 1, NumEntriesInList[i][RplsIdx[i]] is not less than NumRefIdxActive[i]. The image referenced by each active entry in RefPicList[0] or RefPicList[1] exists in the DPB, and its TemporalId is less than or equal to the TemporalId of the current image. Optionally, the following constraint can also be specified: the entry index of any inactive entry in RefPicList[0] or RefPicList[1] is not used as a reference index for decoding the current image. Optionally, the following constraint can also be specified: the inactive entry in RefPicList[0] or RefPicList[1] does not reference the same image as any other entry in RefPicList[0] or RefPicList[1]. The STRP entries in RefPicList[0] or RefPicList[1] of a strip in an image do not reference the same image as LTRP entries in RefPicList[0] or RefPicList[1] of the same or different strips in the same image. The current image is not referenced by any entry in RefPicList[0] or RefPicList[1]. The difference between the PicOrderCntVal of the current image and the PicOrderCntVal of the image referenced by the LTRP entry is greater than or equal to 224, and such an entry does not exist in RefPicList[0] or RefPicList[1]. Assume that setOfRefPics is a unique set of images referenced by all entries in RefPicList[0] and all entries in RefPicList[1]. The number of images in setOfRefPics is less than or equal to sps_max_dec_pic_buffering_minus1, and all stripes in an image have the same setOfRefPics.

[0130] Decoding process for reference image identifiers This process is called once per image after the decoding process that reconstructs the list of reference images for the stripe, but before decoding the stripe data. This process may result in one or more reference images in the DPB being marked as "not for reference" or "for long-term reference." Decoded images in the DPB can be marked as "not for reference," "for short-term reference," or "for long-term reference," but at any given moment during the decoding process, only one of these three marking states can exist. Assigning one of these marking states to an image implicitly removes the others (if applicable). When an image is said to be marked as "for reference," this uniformly means marking the image as either "for short-term reference" or "for long-term reference" (but not simultaneously). If the current image is an IRAP image, then all reference images currently included in the DPB (if any) are marked as "not for reference." STRPs are identified by their PicOrderCntVal values. LTRPs are identified by Log2(MaxLtPicOrderCntLsb) LSBs of their PicOrderCntVal values. For each LTRP entry in RefPicList[0] or RefPicList[1], if the referenced image is a STRP, the image is marked as "for long-term reference". Each reference image in DPB that is not referenced by any entry in RefPicList[0] or RefPicList[1] is marked as "not for reference".

[0131] The following provides a detailed description of a second embodiment of the present invention. This section describes a second embodiment of the above disclosure. The following description relates to the latest VVC WD. In this embodiment, a set of reference image list structures shared by reference image list 0 and reference image list 1 is indicated in the SPS.

[0132] Sequence Parameter Set RBSP Syntax

[0133] Image Parameter Set (RBSP) Syntax

[0134] Striped header syntax

[0135] Reference image list structure syntax

[0136] The semantics of the NAL unit header are described below.

[0137] Sequence Parameter Set RBSP Semantics log2_max_pic_order_cnt_lsb_minus4 represents the value of the variable MaxPicOrderCntLsb used in the decoding process for image sequence numbering, as shown below: MaxPicOrderCntLsb = 2 ( log2_max_pic_order_cnt_lsb_minus4 + 4 ) The value of `log2_max_pic_order_cnt_lsb_minus4` ranges from 0 to 12 (inclusive). `sps_max_dec_pic_buffering_minus1` plus 1 indicates the maximum size of the decoding image buffer required by CVS, in units of image storage buffers. The value of `sps_max_dec_pic_buffering_minus1` ranges from 0 to `MaxDpbSize` – 1 (inclusive), where `MaxDpbSize` is as shown elsewhere. `num_ref_pic_lists_in_sps` indicates the number of syntax structures `ref_pic_list_struct( rplsIdx, ltrpFlag )` included in SPS. The value of `num_ref_pic_lists_in_sps` ranges from 0 to 128 (inclusive). The decoder needs to allocate memory for all syntax structures `num_short_term_ref_pic_sets + 2 ref_pic_list_struct( rplsIdx, ltrpFlag )`, because the two syntax structures `ref_pic_list_struct( rplsIdx, ltrpFlag )` can be directly indicated in the stripe header of the current image. `long_term_ref_pics_flag` being 0 indicates that no LTRP is used for inter-frame prediction of any decoded image in CVS. `long_term_ref_pics_flag` being 1 indicates that LTRP can be used for inter-frame prediction of one or more decoded images in CVS. `additional_lt_poc_lsb` represents the value of the variable `MaxLtPicOrderCntLsb` used in the decoding process for the reference image list, as follows: `MaxLtPicOrderCntLsb = 2`. ( log2_max_pic_order_cnt_lsb_minus4 + 4 + additional_lt_poc_lsb ) The value of additional_lt_poc_lsb ranges from 0 to 32 – log2_max_pic_order_cnt_lsb_minus4 – 4 (inclusive). If it does not exist, then the value of additional_lt_poc_lsb is assumed to be 0.

[0138] The semantics of the Image Parameter Set (RBSP) are described below.

[0139] Strip Header Semantics If present, the values ​​of each of the slice_pic_parameter_set_id and slice_pic_order_cnt_lsb syntax elements are the same across all slice headers of the decoded image. slice_type indicates the decoding type of the slice, as shown in Table 7-3.

[0140] Table 7-3: Relationship between name and slice_type

[0141] When nal_unit_type is IRAP_NUT, i.e., when the image is an IRAP image, slice_type is 2. slice_pic_order_cnt_lsb represents the image sequence number of the current image modulo MaxPicOrderCntLsb. The length of the syntax element slice_pic_order_cnt_lsb is (log2_max_pic_order_cnt_lsb_minus4 + 4) bits. The value range of slice_pic_order_cnt_lsb is 0 to MaxPicOrderCntLsb – 1 (inclusive). If it does not exist, slice_pic_order_cnt_lsb is assumed to be equal to 0. ref_pic_list_sps_flag[i] is equal to 1, indicating that the reference image list i of the current image is derived from one of the syntax structures ref_pic_list_struct(rplsIdx, ltrpFlag) in the activation SPS. `ref_pic_list_sps_flag[i]` equals 0, indicating that the reference image list `i` for the current image is derived from the syntax structure `ref_pic_list_struct(rplsIdx, ltrpFlag)` directly included in the strip header of the current image. When `num_ref_pic_lists_in_sps` equals 0, the value of `ref_pic_list_sps_flag[i]` is 0. `ref_pic_list_idx[i]` represents the index of the syntax structure `ref_pic_list_struct(rplsIdx, ltrpFlag)` used to derive the reference image list `i` for the current image within the list of syntax structures `ref_pic_list_struct(rplsIdx, ltrpFlag)` included in the active SPS. The syntax element `ref_pic_list_idx[i]` is represented by `Ceil(Log2(num_ref_pic_lists_in_sps))` bits. If it does not exist, it is inferred that the value of ref_pic_list_idx[i] is equal to 0. The value range of ref_pic_list_idx[i] is 0 to num_ref_pic_lists_in_sps – 1 (inclusive). num_ref_idx_active_override_flag is 1, indicating that the syntax element num_ref_idx_active_minus1[0] exists in both P and B stripes, while the syntax element num_ref_idx_active_minus1[1] exists in B stripe.The value of num_ref_idx_active_override_flag is 0, indicating that the syntax elements num_ref_idx_active_minus1[0] and num_ref_idx_active_minus1[1] do not exist.

[0142] If it exists, then num_ref_idx_active_minus1[i] represents the value of the variable NumRefIdxActive[i], as follows: NumRefIdxActive[i] = num_ref_idx_active_minus1[i] + 1. The value range of num_ref_idx_active_minus1[i] is 0~14 (inclusive). The value of NumRefIdxActive[i] – 1 represents the maximum reference index in the reference image list i that can be used to decode the stripe. When the value of NumRefIdxActive[i] is equal to 0, there is no reference index in the reference image list i that can be used to decode the stripe. When i is equal to 0 or 1, if the current stripe is a B stripe and num_ref_idx_active_override_flag is 0, then it is inferred that NumRefIdxActive[i] is equal to num_ref_idx_default_active_minus1[i] + 1. If the current stripe is a P stripe and num_ref_idx_active_override_flag is 0, then it is inferred that NumRefIdxActive[0] is equal to num_ref_idx_default_active_minus1[0] + 1. If the current stripe is a P stripe, then it is inferred that NumRefIdxActive[1] is equal to 0. If the current stripe is an I stripe, then it is inferred that both NumRefIdxActive[0] and NumRefIdxActive[1] are equal to 0.

[0143] Alternatively, if i equals 0 or 1, assume rplsIdx1 is set to ref_pic_list_sps_flag[i] ? ref_pic_list_idx[i] : num_ref_pic_lists_in_sps[i], and numRpEntries[i] is set to num_strp_entries[i][rplsIdx1] + num_ltrp_entries[i][rplsIdx1]. When NumRefIdxActive[i] is greater than numRpEntries[i], the value of NumRefIdxActive[i] is set to numRpEntries[i].

[0144] Reference image list structure semantics The syntax structure ref_pic_list_struct(rplsIdx, ltrpFlag) can exist in either the SPS or the strip header. The specific meaning of this syntax structure depends on whether it is included in the strip header or in the SPS: if it exists in the strip header, the syntax structure ref_pic_list_struct(rplsIdx, ltrpFlag) represents a list of reference images for the current image (including the images in the strip). Otherwise (if it exists in the SPS), the syntax structure ref_pic_list_struct(rplsIdx, ltrpFlag) represents a list of candidate reference images, and the term "current image" in the semantics defined in the remainder of this section means: (1) each image that includes one or more strips, wherein the ref_pic_list_idx[i] included by one or more strips is equal to the index in the list of the syntax structure ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag) included in the SPS, and (2) each image in the CVS, wherein the CVS has the same SPS as the active SPS. `num_strp_entries[ rplsIdx ]` represents the number of STRP entries in the syntax structure `ref_pic_list_struct( rplsIdx, ltrpFlag )`. `num_ltrp_entries[ rplsIdx ]` represents the number of LTRP entries in the syntax structure `ref_pic_list_struct( rplsIdx, ltrpFlag )`. If it does not exist, the value of `num_ltrp_entries[ rplsIdx ]` is assumed to be 0.

[0145] The derivation of the variable NumEntriesInList[rplsIdx] is as follows: NumEntriesInList[rplsIdx] = num_strp_entries[rplsIdx] + num_ltrp_entries[rplsIdx]. The value range of NumEntriesInList[rplsIdx] is 0 to sps_max_dec_pic_buffering_minus1 (inclusive). lt_ref_pic_flag[rplsIdx][i] equals 1, indicating that the i-th entry in the syntax structure ref_pic_list_struct(rplsIdx, ltrpFlag) is an LTRP entry. lt_ref_pic_flag[rplsIdx][i] equals 0, indicating that the i-th entry in the syntax structure ref_pic_list_struct(rplsIdx, ltrpFlag) is a STRP entry. If it does not exist, then the value of lt_ref_pic_flag[ rplsIdx ][ i ] is inferred to be equal to 0. The requirement for bitstream consistency is: the sum of lt_ref_pic_flag[ rplsIdx ][ i ] (all values ​​of i in the range of 0 to NumEntriesInList[ rplsIdx ] – 1 (inclusive)) is equal to num_ltrp_entries[ rplsIdx ]. When the i-th entry is the first STRP entry in the syntax structure ref_pic_list_struct( rplsIdx, ltrpFlag ), delta_poc_st[ rplsIdx ][ i ] represents the difference between the image sequence number value of the current image and the image referenced by the i-th entry. Alternatively, when the i-th entry is a STRP entry in the syntax structure ref_pic_list_struct( rplsIdx, ltrpFlag ) but not the first STRP entry, delta_poc_st[ rplsIdx ][ i ] represents the difference between the image sequence number value of the image referenced by the i-th entry in the syntax structure ref_pic_list_struct( rplsIdx, ltrpFlag ) and the image referenced by the previous STRP entry. The value of delta_poc_st[ rplsIdx ][ i ] ranges from 0 to 2. 15 – 1 (inclusive of end values). poc_lsb_lt[ rplsIdx ][ i ] represents the value modulo MaxLtPicOrderCntLsb of the image sequence number of the image referenced by the i-th entry in the syntax structure ref_pic_list_struct( rplsIdx,ltrpFlag ). The length of the syntax element poc_lsb_lt[ rplsIdx ][ i ] is Log2( MaxLtPicOrderCntLsb ) bits.

[0146] A general decoding process is used, which is part of the detailed description of the first embodiment of the present invention. The NAL unit decoding process is described below. The NAL unit decoding process is used, which is part of the detailed description of the first embodiment of the present invention.

[0147] The stripe decoding process is described below.

[0148] Decoding process for image sequential numbering A decoding process for image sequence numbering is used, which is part of a detailed description of the first embodiment of the present invention.

[0149] Decoding process used for constructing the reference image list This procedure is invoked at the start of the decoding process for each strip in a non-IRAP image. Reference images are addressed by reference indices. A reference index is an index in a list of reference images. When decoding I stripes, the list of reference images is not used to decode the strip data. When decoding P stripes, only reference image list 0 (i.e., RefPicList[0]) is used to decode the strip data. When decoding B stripes, both reference image list 0 and reference image list 1 (i.e., RefPicList[1]) are used to decode the strip data. At the start of the decoding process for each strip in a non-IRAP image, the reference image lists RefPicList[0] and RefPicList[1] are derived. These two reference image lists are used to identify reference images or decode strip data. If the I-band in a non-IRAP image is not the first band in the image, then RefPicList[0] and RefPicList[1] can be derived to check bitstream consistency. However, it is not necessary to derive RefPicList[0] and RefPicList[1] when decoding the current image or images that are decoded after the current image. If the P-band is not the first band in the image, then RefPicList[1] can be derived to check bitstream consistency. However, it is not necessary to derive RefPicList[1] when decoding the current image or images that are decoded after the current image.

[0150] The reference image lists RefPicList[ 0 ] and RefPicList[ 1 ] are constructed as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps for( j = 0, pocBase = PicOrderCntVal; j<NumEntriesInList[ RplsIdx[ i] ]; j++) { if( !lt_ref_pic_flag[ RplsIdx[ i ] ][ j ] ) { RefPicPocList[ i ][ j ]= pocBase – delta_poc_st[ RplsIdx[ i ] ][ j ] if( there is a reference picture picA in the DPB with PicOrderCntVal equal to RefPicPocList[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” pocBase = RefPicPocList[ i ][ j ] } else { if( there is a reference picA in the DPB with PicOrderCntVal&(MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ RplsIdx[ i ] ][ j ] ) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture" } } } In each case where i equals 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are called active entries in RefPicList[i], and the other entries in RefPicList[i] are called inactive entries in RefPicList[i]. If lt_ref_pic_flag[RplsIdx[i]][j] equals 0, then each entry in RefPicList[i][j] (j is in the range 0~NumEntriesInList[RplsIdx[i]]–1 (inclusive)) is called a STRP entry; otherwise, it is called an LTRP entry. A particular image may be referenced by both entries in RefPicList[0] and entries in RefPicList[1]. A particular image may also be referenced by multiple entries in RefPicList[0] or multiple entries in RefPicList[1]. The active entries in RefPicList[0] and RefPicList[1] refer to all reference images that can be used for inter-frame prediction of the current image and one or more images whose decoding order is after the current image. The inactive entries in RefPicList[0] and RefPicList[1] refer to all reference images that are not used for inter-frame prediction of the current image but can be used for inter-frame prediction of one or more images whose decoding order is after the current image. There may be one or more entries in RefPicList[0] or RefPicList[1] that are equal to "no reference image" because there is no corresponding image in the DPB. Ignore each inactive entry in RefPicList[0] or RefPicList[1] that is equal to "no reference image". Infer unintentional image loss for each active entry in RefPicList[0] or RefPicList[1] that is equal to "no reference image".

[0151] The requirement for stream consistency is to use the following constraints: for each i equal to 0 or 1, NumEntriesInList[RplsIdx[i]] is not less than NumRefIdxActive[i]. The image referenced by each active entry in RefPicList[0] or RefPicList[1] exists in the DPB, and its TemporalId is less than or equal to the TemporalId of the current image. Optionally, the following constraint can also be specified: the entry index of any inactive entry in RefPicList[0] or RefPicList[1] is not used as a reference index for decoding the current image. Optionally, the following constraint can also be specified: the inactive entry in RefPicList[0] or RefPicList[1] does not reference the same image as any other entry in RefPicList[0] or RefPicList[1]. The STRP entries in RefPicList[0] or RefPicList[1] of a strip in an image do not reference the same image as LTRP entries in RefPicList[0] or RefPicList[1] of the same or different strips in the same image. The current image is not referenced by any entry in RefPicList[0] or RefPicList[1]. The difference between the PicOrderCntVal of the current image and the PicOrderCntVal of the image referenced by the LTRP entry is greater than or equal to 224, and such an entry does not exist in RefPicList[0] or RefPicList[1]. Assume that setOfRefPics is a unique set of images referenced by all entries in RefPicList[0] and all entries in RefPicList[1]. The number of images in setOfRefPics is less than or equal to sps_max_dec_pic_buffering_minus1, and all stripes in an image have the same setOfRefPics.

[0152] The following describes the decoding process used for reference image identification.

[0153] This process is called once per image after the decoding process that reconstructs the list of reference images for the stripe, but before decoding the stripe data. This process may result in one or more reference images in the DPB being marked as "not for reference" or "for long-term reference." Decoded images in the DPB can be marked as "not for reference," "for short-term reference," or "for long-term reference," but at any given moment during the decoding process, only one of these three marking states can exist. Assigning one of these marking states to an image implicitly removes the others (if applicable). When an image is said to be marked as "for reference," this uniformly means marking the image as either "for short-term reference" or "for long-term reference" (but not simultaneously). If the current image is an IRAP image, then all reference images currently included in the DPB (if any) are marked as "not for reference." STRPs are identified by their PicOrderCntVal values. LTRPs are identified by Log2(MaxLtPicOrderCntLsb) LSBs of their PicOrderCntVal values.

[0154] For each LTRP entry in RefPicList[0] or RefPicList[1], if the referenced image is a STRP, the image is marked as "for long-term reference". Each reference image in DPB that is not referenced by any entry in RefPicList[0] or RefPicList[1] is marked as "not for reference".

[0155] Figure 5 This is an embodiment of a method 500 for decoding a decoded video stream, implemented by a video decoder (e.g., video decoder 30). Method 500 can be executed after receiving the decoded stream directly or indirectly from a video encoder (e.g., video encoder 20). Method 500 can be used to improve the decoding process (e.g., to make the decoding process more efficient and faster than conventional decoding processes).

[0156] In step 502, a first reference image list and a second reference image list for the current stripe are obtained, wherein the current stripe is represented in the decoded video stream. In one embodiment, the first reference image list is RefPictList[0] and the second reference image list is RefPictList[1].

[0157] In step 504, when the current stripe is the first stripe in the current image to which the current stripe belongs, the state of the reference image in the decoded picture buffer (DPB) is identified according to the first reference image list and the second reference image list. In one embodiment, the state of the reference image in the decoded picture buffer is one of short-term reference, long-term reference, or no reference.

[0158] In step 506, when the reference image is in a state of short-term reference or long-term reference, the reference image is used in the reconstruction process of the current image (or an image after the current image). After the reconstruction process, the video decoder is able to output video or images. In one embodiment, the video or images can be displayed on the display of an electronic device (e.g., a smartphone, tablet, laptop, etc.).

[0159] Figure 6 This is an embodiment of a method 600 for decoding a decoded video stream, implemented by a video decoder (e.g., video decoder 30). Method 600 can be executed after receiving the decoded stream directly or indirectly from a video encoder (e.g., video encoder 20). Method 600 can be used to improve the decoding process (e.g., to make the decoding process more efficient and faster than conventional decoding processes).

[0160] In step 602, the video decoder decodes the first image (including all its stripes) to obtain a first decoded image. In step 604, after decoding, the video decoder stores the first decoded image in a decoded image buffer (DPB). In one embodiment, in addition to the first decoded image, the DPB also includes multiple reference images. The content of the DPB is managed to ensure synchronization between the encoder and the decoder. The set of images held in the DPB is called the Reference Picture Set (RPS).

[0161] In step 606, after all stripes in the first image have been decoded, the first image is identified as a short-term reference. In one embodiment, when the first image is identified as a short-term reference, the first image is called a STRP. In one embodiment, the STRP is identified by an image sequence number value (PicOrderCntVal).

[0162] In step 608, the first stripe in the current image is decoded. In one embodiment, the decoding process of the first stripe in the current image includes: obtaining a first reference image list and a second reference image list; when the first image is referenced by a long-term reference picture (LTRP) entry in the first or second reference image list, the video decoder re-identifies the first image as being used for long-term reference; when the first image is not referenced in the first or second reference image list, the video decoder re-identifies the first image as not being used for reference. Therefore, when the first image is not referenced by an LTRP entry in the first or second reference image list, the state of the first image changes from STRP or is updated to not being used for reference.

[0163] In one embodiment, the first reference image list is RefPictList[0], and the second reference image list is RefPictList[1]. In one embodiment, the current image includes a non-IRAP image. In one embodiment, LTRP is identified by the logarithmic function (Log2(MaxLtPicOrderCntLsb)) of the least significant bit of the maximum long-term image sequence number value (PicOrderCntVal).

[0164] In one embodiment, at any given time, the first image is identified or re-identified only for one of the following: for short-term reference, for long-term reference, and not for reference. In one embodiment, the step of identifying or re-identifying the first image for one of the following excludes any other identification of the first image.

[0165] The following describes an overview of alternative embodiments based on the first and second embodiments.

[0166] This section provides a brief overview of other alternative embodiments of the invention. These overviews relate to the description of the first embodiment. However, the basic concepts of the invention used in the following optional embodiments are also applicable to implementations of the invention based on the second embodiment. This implementation shares the same spirit as how these aspects are implemented based on the first embodiment.

[0167] Semantics of incremental POCs for short-term reference image entries In an alternative embodiment of the invention, the semantics of the syntax element representing the incremental POC of the i-th entry in the reference image list structure ref_pic_list_struct() is defined as the POC difference between the current image and the reference image associated with the i-th entry. Some descriptions used herein refer to this draft standard (e.g., the VVC working draft), where only the increment is shown or described. Deleted text is indicated by strikethrough, and any newly added text is highlighted.

[0168] The semantics of delta_poc_st[listIdx][rplsIdx][i] are defined as follows: delta_poc_st[listIdx][rplsIdx][i] represents the difference between the image sequence number of the current image and the image referenced by the i-th entry. The value range of delta_poc_st[listIdx][rplsIdx][i] is -2. 15 ~2 15 – 1 (including end value).

[0169] The equations in the process of constructing the reference image list need to be updated. The construction methods of the reference image lists RefPicList[0] and RefPicList[1] are as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] for( j = 0, pocBase = PicOrderCntVal; j <NumEntriesInList[ i ][RplsIdx[ i ]]; j++) { if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ]][ j ]) { RefPicPocList[ i ][ j ]= pocBasePicOrderCntVal – delta_poc_st[ i ][RplsIdx[ i ]][ j ] if( there is a reference picture picA in the DPB with PicOrderCntValequal to RefPicPocList[ i ][ j ]) RefPicList[i][j] = picA else RefPicList[ i ][ j ]= "no reference picture" pocBase = RefPicPocList[ i ][ j ] } else { if( there is a reference picA in the DPB with PicOrderCntVal&(MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ i ][ RplsIdx[ i ]][ j ]) RefPicList[i][j] = picA else RefPicList[ i ][ j ]= "no reference picture" } } } Long-term reference image entry indication In an alternative embodiment of the invention, long-term reference image entries are not indicated within the same reference image list structure that includes short-term reference image entries. Instead, long-term reference image entries are indicated in a separate structure, and each entry in this structure has a syntax element that describes the expected position of the long-term reference image entry for deriving the corresponding entry index in the final reference image list.

[0170] Sequence Parameter Set RBSP Syntax

[0171] Striped header syntax

[0172] Reference image list structure syntax

[0173] Long-term reference image list structure syntax

[0174] Sequence Parameter Set RBSP Semantics `num_ref_pic_lists_lt_in_sps` represents the number of `ref_pic_list_lt_struct(ltRplsIdx)` syntax structures included in the SPS. The value of `num_ref_pic_lists_lt_in_sps` ranges from 0 to 64 (inclusive). If it does not exist, the value of `num_ref_pic_lists_lt_in_sps` is assumed to be 0.

[0175] Strip Header Semantics `ref_pic_list_lt_idx[i]` represents the index of the reference image list `i` used to deduce the current image in the list of syntax structures `ref_pic_list_lt_struct(ltRplsIdx)` included in the active SPS. The syntax element `ref_pic_list_lt_idx[i]` is represented by `Ceil(Log2(num_ref_pic_lists_lt_in_sps))` bits. The value of `ref_pic_list_lt_idx` ranges from 0 to `num_ref_pic_lists_lt_in_sps – 1` (inclusive).

[0176] Reference image list structure semantics The syntax structure `ref_pic_list_struct(listIdx, rplsIdx)` can exist in either the SPS or the strip header. The specific meaning of this syntax structure depends on whether it is included in the strip header or the SPS: if it exists in the strip header, the syntax structure `ref_pic_list_struct(listIdx, rplsIdx)` represents a short-term reference image list `listIdx` for the current image (including the images in the strip). Otherwise (in the SPS), the syntax structure ref_pic_list_struct(listIdx, rplsIdx) represents a candidate short-term reference image list listIdx, and the term "current image" in the semantics defined in the remainder of this section refers to: (1) each image including one or more stripes, wherein the ref_pic_list_idx[listIdx] included by one or more stripes is equal to the index in the list of the syntax structure ref_pic_list_struct(listIdx, rplsIdx) included in the SPS; (2) each image in the CVS, wherein the CVS has the same SPS as the active SPS. num_strp_entries[listIdx][rplsIdx] represents the number of STRP entries in the syntax structure ref_pic_list_struct(listIdx, rplsIdx).

[0177] `num_ltrp_entries[ listIdx ][ rplsIdx ]` represents the number of LTRP entries in the syntax structure `ref_pic_list_struct( listIdx, rplsIdx, ltrpFlag )`. If it does not exist, the value of `num_ltrp_entries[ listIdx ][ rplsIdx ]` is assumed to be 0.

[0178] The derivation of the variable NumEntriesInList[ listIdx ][ rplsIdx ] is as follows: NumRefPicEntriesInRpl[ listIdx ][ rplsIdx ]= num_strp_entries[listIdx ][ rplsIdx ]+ num_ltrp_entries[listIdx][rplsIdx](7-34) The value range of NumRefPicEntries[ listIdx ][ rplsIdx ] is 0 to sps_max_dec_pic_buffering_minus1 (inclusive).

[0179] If `lt_ref_pic_flag[listIdx][rplsIdx][i]` equals 1, it means that the i-th entry in the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)` is an LTRP entry. If `lt_ref_pic_flag[listIdx][rplsIdx][i]` equals 0, it means that the i-th entry in the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)` is a STRP entry. If it does not exist, then the value of `lt_ref_pic_flag[listIdx][rplsIdx][i]` is inferred to be 0.

[0180] The bitstream consistency requirement is: the sum of lt_ref_pic_flag[listIdx][rplsIdx][i] (all values ​​of i are in the range 0 to NumRefPicEntries[listIdx][rplsIdx] – 1 (inclusive)) is equal to num_ltrp_entries[listIdx][rplsIdx].

[0181] When the i-th entry is the first STRP entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx), delta_poc_st[listIdx][rplsIdx][i] represents the difference between the image sequence number values ​​of the current image and the image referenced by the i-th entry. Alternatively, when the i-th entry is a STRP entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx) but not the first STRP entry, delta_poc_st[listIdx][rplsIdx][i] represents the difference between the image sequence number values ​​of the i-th entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx) and the image referenced by the previous STRP entry. The value range of delta_poc_st[listIdx][rplsIdx][i] is -2. 15 ~2 15 – 1 (including end value).

[0182] `poc_lsb_lt[listIdx][rplsIdx][i]` represents the value of `maxLtPicOrderCntLsb` modulo the image sequence number of the image referenced by the `i`th entry in the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)`. The length of the syntax element `poc_lsb_lt[listIdx][rplsIdx][i]` is Log2(MaxLtPicOrderCntLsb) bits.

[0183] Long-term reference image list structure semantics The syntax structure ref_pic_list_lt_struct( ltRplsIdx ) can exist in either the SPS or the strip header. The specific meaning of this syntax structure depends on whether it is included in the strip header or in the SPS: if it exists in the strip header, the syntax structure ref_pic_list_lt_struct( ltRplsIdx ) represents a long-term reference image list for the current image (including images in the strip). Otherwise (if it exists in the SPS), the syntax structure ref_pic_list_struct( listIdx, rplsIdx) represents a candidate for the long-term reference image list, and the term "current image" in the semantics defined in the remainder of this section means: (1) each image that includes one or more strips, wherein the ref_pic_list_lt_idx[ i ] included by one or more strips is equal to the index in the list of the syntax structure ref_pic_list_lt_struct( ltRplsIdx ) included in the SPS; (2) each image in the CVS, wherein the CVS has the same SPS as the active SPS. `num_ltrp_entries[ltRplsIdx]` represents the number of LTRP entries in the syntax structure `ref_pic_list_lt_struct(ltRplsIdx)`. `poc_lsb_lt[rplsIdx][i]` indicates the modulo of the image sequence number of the image referenced by the i-th entry in the syntax structure `ref_pic_list_lt_struct(rplsIdx)`, divided by `MaxLtPicOrderCntLsb`. The syntax element `poc_lsb_lt[rplsIdx][i]` has a length of Log2(MaxLtPicOrderCntLsb) bits. `lt_pos_idx[rplsIdx][i]` represents the index of the i-th entry in the syntax structure `ref_pic_list_lt_struct(rplsIdx)` within the reference image list after the reference image list has been constructed. The length of the syntax element lt_pos_idx[ rplsIdx ][ i] is Log2( sps_max_dec_pic_buffering_minus1 + 1 ) bits. When num_ltrp_entries[ ltRplsIdx ] is greater than 1, poc_lsb_lt[ rplsIdx ][ i ] and lt_pos_idx[ rplsIdx ][ i ] are sorted in descending order of the value of lt_pos_idx[ rplsIdx ][ i ].

[0184] The decoding process is described below.

[0185] Decoding process used for constructing the reference image list This procedure is invoked at the start of the decoding process for each strip in a non-IRAP image. Reference images are addressed by reference indices. A reference index is an index in a list of reference images. When decoding I stripes, the list of reference images is not used to decode the strip data. When decoding P stripes, only reference image list 0 (i.e., RefPicList[0]) is used to decode the strip data. When decoding B stripes, both reference image list 0 and reference image list 1 (i.e., RefPicList[1]) are used to decode the strip data. At the start of the decoding process for each strip in a non-IRAP image, the reference image lists RefPicList[0] and RefPicList[1] are derived. These two reference image lists are used to identify reference images or decode strip data. If the I-band in a non-IRAP image is not the first band in the image, then RefPicList[0] and RefPicList[1] can be derived to check bitstream consistency. However, it is not necessary to derive RefPicList[0] and RefPicList[1] when decoding the current image or images that are decoded after the current image. If the P-band is not the first band in the image, then RefPicList[1] can be derived to check bitstream consistency. However, it is not necessary to derive RefPicList[1] when decoding the current image or images that are decoded after the current image. The construction methods of the reference image lists RefPicList[0] and RefPicList[1] are as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] for( j = 0, pocBase = PicOrderCntVal; j <NumEntriesInList[ i ][RplsIdx[ i ]]; j++) { if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ]][ j ]) { RefPicPocList[ i ][ j ]= pocBase – delta_poc_st[ i ][ RplsIdx[ i ]][j ] if( there is a reference picture picA in the DPB with PicOrderCntValequal to RefPicPocList[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” pocBase = RefPicPocList[ i ][ j ] } else { if( there is a reference picA in the DPB with PicOrderCntVal&(MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ i ][ RplsIdx[ i ]][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” } if( ref_pic_list_lt_sps_flag[ i ] ) LtRplsIdx = ref_pic_list_lt_idx[ i ] else LtRplsIdx = num_ref_pic_lists_lt_in_sps[ i ] for( j = 0; j<num_ltrp_entries[ LtRplsIdx[ i ] ]; j++) { if( there is a reference picA in the DPB with PicOrderCntVal&(MaxLtPicOrderCntLsb – 1 ) { equal to poc_lsb_lt[ LtRplsIdx[ i ] ][ j ] ) for( k = sps_max_dec_pic_buffering_minus1; k>lt_pos_idx[ LtRplsIdx[ i] ][ j ]; k– –) RefPicList[ i ][ k ]= RefPicList[ i ][ k – 1 ] RefPicList[ i ][ lt_pos_idx[ LtRplsIdx[ i ]] [ j ]] = picA } else { for( k = sps_max_dec_pic_buffering_minus1; k>lt_pos_idx[ LtRplsIdx[ i] ][ j ]; k– –) RefPicList[ i ][ k ]= RefPicList[ i ][ k – 1 ] RefPicList[ i ][ lt_pos_idx[ RplsIdx[ i ]] [ j ]] = "no referencepicture” } } } In each case where i equals 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are called active entries in RefPicList[i], and the other entries in RefPicList[i] are called inactive entries in RefPicList[i]. If lt_ref_pic_flag[i][RplsIdx[i]][j] equals 0, then each entry in RefPicList[i][j] (j is in the range 0~NumEntriesInList[i][RplsIdx[i]]–1 (inclusive)) is called a STRP entry; otherwise, it is called an LTRP entry. A particular image may be referenced by both entries in RefPicList[0] and entries in RefPicList[1]. A particular image may also be referenced by multiple entries in RefPicList[0] or multiple entries in RefPicList[1]. The active entries in RefPicList[0] and RefPicList[1] collectively reference all reference images that can be used for inter-frame prediction of the current image and one or more images whose decoding order is after the current image. The inactive entries in RefPicList[0] and RefPicList[1] collectively reference all reference images that are not used for inter-frame prediction of the current image but can be used for inter-frame prediction of one or more images whose decoding order is after the current image. There may be one or more entries in RefPicList[0] or RefPicList[1] that are equal to "no reference image" because there is no corresponding image in the DPB. Ignore each inactive entry in RefPicList[0] or RefPicList[0] that is equal to "no reference image". Infer unintentional image loss for each active entry in RefPicList[0] or RefPicList[1] that is equal to "no reference image".

[0186] The requirement for stream consistency is to use the following constraints: for each i equal to 0 or 1, the number of entries in RefPicList[i] is not less than NumRefIdxActive[i]. The image referenced by each active entry in RefPicList[0] or RefPicList[1] exists in the DPB, and its TemporalId is less than or equal to the TemporalId of the current image. Optionally, the following constraint can also be specified: the entry index of any inactive entry in RefPicList[0] or RefPicList[1] is not used as a reference index for decoding the current image. Optionally, the following constraint can also be specified: the inactive entry in RefPicList[0] or RefPicList[1] does not reference the same image as any other entry in RefPicList[0] or RefPicList[1]. The STRP entries in RefPicList[0] or RefPicList[1] of a strip in an image do not reference the same image as the LTRP entries in RefPicList[0] or RefPicList[1] of the same or different strips in the same image. The current image is not referenced by any entry in RefPicList[0] or RefPicList[1]. The difference between the PicOrderCntVal of the current image and the PicOrderCntVal of the image referenced by the LTRP entry is greater than or equal to 224, and such an entry does not exist in RefPicList[0] or RefPicList[1]. Assume that setOfRefPics is a unique set of images referenced by all entries in RefPicList[0] and all entries in RefPicList[1]. The number of images in setOfRefPics is less than or equal to sps_max_dec_pic_buffering_minus1, and all stripes in an image have the same setOfRefPics.

[0187] The following describes the number of entries in the short-term reference image table.

[0188] In an alternative embodiment of the invention, the syntax element representing the number of entries associated with short-term reference images in the reference image list structure ref_pic_list_struct() is defined as num_strp_entries_minus1[listIdx][rplsIdx] instead of num_strp_entries[listIdx][rplsIdx]. This change has two effects on the reference image list indication: it saves bits indicating the number of entries associated with short-term reference images in the reference image list structure when decoding elements using ue(v). There is an implicit constraint that each reference image list includes at least one short-term reference image. To achieve this, some modifications to the first embodiment are required.

[0189] For the reference image list indication in the strip header, only the necessary reference image list is indicated according to the strip type; that is, one reference image list (i.e., reference image list 0) corresponds to I or P strips, and two reference image lists (i.e., reference image list 0 and reference image list 1) correspond to B strips. The changes to the strip header syntax are as follows:

[0190] Making the above changes to the strip header (i.e., I stripe or P stripe corresponding to reference image list 0, B stripe corresponding to reference image 0 and reference image 1) will avoid the problem that the scheme has only one short-term reference image for P stripe. However, duplicate short-term reference images cannot be indicated in reference image list 0 and reference image list 1, where the entries in reference image list 1 are inactive entries, because the number of active entries in reference image list 1 must be equal to 0. The semantic change of num_strp_entries_minus1[listIdx][rplsIdx] is as follows: num_strp_entries_minus1[listIdx][rplsIdx] plus 1 indicates the number of STRP entries in the syntax structure ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag). The derivation of the variable NumEntriesInList[listIdx][rplsIdx] is as follows: NumRefPicEntriesInRpl[listIdx][rplsIdx] = num_strp_entries_minus1[listIdx][rplsIdx] + 1 + num_ltrp_entries[listIdx][rplsIdx]. The value range of NumRefPicEntries[listIdx][rplsIdx] is 1 to sps_max_dec_pic_buffering_minus1 (inclusive).

[0191] The current image can be included in the list of reference images.

[0192] In an alternative embodiment of the invention, the current image may be included in its reference image list. To support this feature, no changes to the syntax and semantics are required relative to the descriptions in the first and second embodiments. However, the bitstream consistency constraints described in the decoding process used for constructing the reference image list need to be modified as follows: The bitstream consistency requirement is to use the following constraint: for each i equal to 0 or 1, NumEntriesInList[i][RplsIdx[i]] is not less than NumRefIdxActive[i]. The image referenced by each active entry in RefPicList[0] or RefPicList[1] exists in the DPB, and its TemporalId is less than or equal to the TemporalId of the current image. Alternatively, the following constraint may also be specified: the entry index of any non-active entry in RefPicList[0] or RefPicList[1] is not used as a reference index for decoding the current image. Optionally, the following constraints may also be specified: Inactive entries in RefPicList[0] or RefPicList[1] do not reference the same image as any other entry in RefPicList[0] or RefPicList[1]. STRP entries in RefPicList[0] or RefPicList[1] for a strip in an image do not reference the same image as LTRP entries in RefPicList[0] or RefPicList[1] for the same or different strips in the same image. The current image is not referenced by any entry in RefPicList[0] or RefPicList[1]. When the current image is referenced by an entry in RefPicList[i], the entry index is less than NumRefIdxActive[i] if i is equal to 0 or 1. If the difference between the PicOrderCntVal of the current image and the PicOrderCntVal of the image referenced by the LTRP entry is greater than or equal to 224, then such an entry does not exist in RefPicList[0] or RefPicList[1]. Assume that setOfRefPics is a unique set of images referenced by all entries in RefPicList[0] and all entries in RefPicList[1].If the current image is not included in setOfRefPics, the number of images in setOfRefPics is less than or equal to sps_max_dec_pic_buffering_minus1; otherwise, the number of images in setOfRefPics is less than or equal to sps_max_dec_pic_buffering_minus1 + 1. All stripes in an image have the same setOfRefPics.

[0193] The LTRP entries in the reference image list use different POC LSB bits. In an alternative embodiment of the invention, the number of bits used to identify long-term reference images in the reference image list may differ before reference image list 0 and reference image list 1. To support this feature, the following changes are required:

[0194] additional_lt_poc_lsb[i] represents the value of the variable MaxLtPicOrderCntLsb[i] used in the decoding process for the list of reference images where listIdx equals i, as shown below: MaxLtPicOrderCntLsb[i] = 2 ( log2_max_pic_order_cnt_lsb_ minus4 + 4 + additional_lt_poc_lsb[ i ] ) The value of additional_lt_poc_lsb[i] ranges from 0 to 32 – log2_max_pic_order_cnt_lsb_minus4 – 4 (inclusive). If it does not exist, then the value of additional_lt_poc_lsb[i] is assumed to be 0.

[0195] `poc_lsb_lt[listIdx][rplsIdx][i]` represents the value of the image sequence number of the image referenced by the i-th entry in the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)` modulo `MaxLtPicOrderCntLsb[listIdx]`. The length of the syntax element `poc_lsb_lt[listIdx][rplsIdx][i]` is Log2(MaxLtPicOrderCntLsb[listIdx]) bits.

[0196] The reference image lists RefPicList[0] and RefPicList[1] are constructed as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] for( j = 0, pocBase = PicOrderCntVal; j<NumEntriesInList[ i ][RplsIdx[ i ]]; j++) { if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ]][ j ]) { RefPicPocList[ i ][ j ]= pocBase – delta_poc_st[ i ][ RplsIdx[ i ]][j ] if( there is a reference picture picA in the DPB with PicOrderCntValequal to RefPicPocList[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” pocBase = RefPicPocList[ i ][ j ] } else { if( there is a reference picA in the DPB with PicOrderCntVal&(MaxLtPicOrderCntLsb[ i ] – 1 ) equal to poc_lsb_lt[ i ][ RplsIdx[ i ] ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” } } } Reference image lists 0 and 1 use the same ref_pic_list_sps_flag.

[0197] In an alternative embodiment of the invention, reference image list 0 and reference image list 1 use a single flag to indicate whether the two reference image lists are derived from the syntax structure ref_pic_list_struct() in the active SPS, instead of using two separate flags. This alternative constraint dictates that the two reference image lists are derived either from ref_pic_list_struct() in the active SPS or from the syntax structure ref_pic_list_struct() directly included in the strip header of the current image. To support this feature, the following changes are required:

[0198] ref_pic_list_sps_flag[i] equals 1, indicating that the reference image list i of the current image is derived from one of the syntax structures ref_pic_list_struct(listIdx, rplsIdx,ltrpFlag) in the active SPS where listIdx equals i. ref_pic_list_sps_flag[i] equals 0, indicating that the reference image list i of the current image is derived from the syntax structure ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag) directly included in the strip header of the current image. When num_ref_pic_lists_in_sps[0] or num_ref_pic_lists_in_sps[1] equals 0, the value of ref_pic_list_sps_flag[i] is equal to 0, pic_lists_in_sps[1] is equal to 0, and the value of ref_pic_list_sps_flag is equal to 0.

[0199] The reference image lists RefPicList[0] and RefPicList[1] are constructed as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] for( j = 0, pocBase = PicOrderCntVal; j < NumEntriesInList[ i ][RplsIdx[ i ]]; j++) { if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ]][ j ]) { RefPicPocList[ i ][ j ]= pocBase – delta_poc_st[ i ][ RplsIdx[ i ]][j ] if( there is a reference picture picA in the DPB with PicOrderCntVal equal to RefPicPocList[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” pocBase = RefPicPocList[ i ][ j ] } else { if( there is a reference picA in the DPB with PicOrderCntVal&(MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ i ][ RplsIdx[ i ]][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” } } } Indication of the Most Significant Bit (MSB) of delta POC for long-term reference picture entries In an alternative embodiment of the invention, instead of using extra bits to represent the POC MSB of long-term reference images, a POC MSB cycle is indicated to distinguish long-term reference images. When indicated, the POC MSB cycle information is indicated for each entry of the reference long-term reference image in the ref_pic_list_struct(). The syntax structure ref_pic_list_struct() is not indicated in the SPS, but only in the stripe header. To support this feature, the following changes are required:

[0200]

[0201]

[0202] The syntax structure `ref_pic_list_struct(listIdx, ltrpFlag)` can exist in the strip header. If it exists in the strip header, then `ref_pic_list_struct(listIdx, ltrpFlag)` represents the list of reference images `listIdx` for the current image (including the images in the strip). `num_strp_entries[listIdx][rplsIdx]` represents the number of STRP entries in the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)`. `num_ltrp_entries[listIdx][rplsIdx]` represents the number of LTRP entries in the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)`. If it does not exist, then the value of `num_ltrp_entries[listIdx][rplsIdx]` is inferred to be 0.

[0203] The derivation of the variable NumEntriesInList[listIdx][rplsIdx] is as follows: NumRefPicEntriesInRpl[ listIdx ] [ rplsIdx ]= num_strp_entries[listIdx ] [ rplsIdx ]+ num_ltrp_entries[listIdx][rplsIdx].

[0204] The value range of NumRefPicEntries[listIdx][rplsIdx] is 0 to sps_max_dec_pic_buffering_minus1 (inclusive). lt_ref_pic_flag[listIdx][rplsIdx][i] equals 1, indicating that the i-th entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag) is an LTRP entry. lt_ref_pic_flag[listIdx][rplsIdx][i] equals 0, indicating that the i-th entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag) is a STRP entry. If it does not exist, then the value of lt_ref_pic_flag[listIdx][rplsIdx][i] is inferred to be 0. The requirement for stream consistency is: the sum of lt_ref_pic_flag[listIdx][rplsIdx][i] (where all values ​​of i are in the range 0 to NumRefPicEntries[listIdx][rplsIdx] – 1 (inclusive)) is equal to num_ltrp_entries[listIdx][rplsIdx]. When the i-th entry is the first STRP entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag), delta_poc_st[listIdx][rplsIdx][i] represents the difference between the image sequence number values ​​of the current image and the image referenced by the i-th entry. Alternatively, when the i-th entry is a STRP entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag) but not the first STRP entry, delta_poc_st[listIdx][rplsIdx][i] represents the difference between the image sequence number values ​​of the image referenced by the i-th entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag) and the image referenced by the previous STRP entry. The value range of delta_poc_st[listIdx][rplsIdx][i] is -2. 15 ~2 15 – 1 (inclusive). `poc_lsb_lt[listIdx][rplsIdx][i]` represents the modulo of the image sequence number of the image referenced by the i-th entry in the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)`, modulo `MaxLtPicOrderCntLsb`. The length of the syntax element `poc_lsb_lt[listIdx][rplsIdx][i]` is Log2(MaxLtPicOrderCntLsb) bits. `delta_poc_msb_present_flag[listIdx][i]` equals 1, indicating that `delta_poc_msb_cycle_lt[listIdx][i]` exists. `delta_poc_msb_present_flag[listIdx][i]` equals 0, indicating that `delta_poc_msb_cycle_lt[listIdx][i]` does not exist. When num_ltrp_entries[listIdx] is greater than 0 and multiple reference images exist in the DPB whose value of PicOrderCntVal modulo MaxPicOrderCntLsb equals poc_lsb_lt[listIdx][i] during stripe header decoding, delta_poc_msb_present_flag[listIdx][i] is equal to 1. If it does not exist, delta_poc_msb_cycle_lt[listIdx][i] is inferred to be equal to 0. delta_poc_msb_cycle_lt[listIdx][i] is used to determine the most significant bit of the image sequence number value of the i-th entry in the syntax structure ref_pic_list_struct(listIdx,ltrpFlag). If it does not exist, delta_poc_msb_cycle_lt[listIdx][i] is inferred to be equal to 0. The modification to the decoding process used for image sequence numbering is as follows: at any point during the decoding process, the values ​​of PicOrderCntVal&(MaxLtPicOrderCntLsb – 1) for any two reference images in the DPB are not the same.

[0205] The reference image lists RefPicList[0] and RefPicList[1] are constructed as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] for( j = 0, pocBase = PicOrderCntVal; j<NumEntriesInList[ i ][RplsIdx[ i ]]; j++) { if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ]][ j ]) { RefPicPocList[ i ][ j ]= pocBase – delta_poc_st[ i ][ RplsIdx[ i ]][j ] if( there is a reference picture picA in the DPB with PicOrderCntValequal to RefPicPocList[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” pocBase = RefPicPocList[ i ][ j ] } else { if( delta_poc_msb_cycle_lt[ i ][ j ]is equal to 0&& there is a reference picture picA in the DPB with PicOrderCntVal& ( MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ i ][ j ] ) RefPicList[ i ][ j ]= picA else if( delta_poc_msb_cycle_lt[ i ][ j ]is equal to 1&& there is a reference picture picA in the DPB with PicOrderCntValequal to ( MaxPicOrderCntLsb * delta_poc_msb_cycle_lt[ i ][ j ]) + poc_lsb_lt[ i ][ j ]) ) RefPicList[i][j] = picA else RefPicList[ i ][ j ]= "no reference picture" } } } Alternatively, the semantics of delta_poc_msb_cycle_lt[listIdx][i] can be represented as an increment of increments, so that the reference image list can be updated as follows: The reference image lists RefPicList[0] and RefPicList[1] are constructed as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] prevMsbCycle = 0 for( j = 0, pocBase = PicOrderCntVal; j <NumEntriesInList[ i ][RplsIdx[ i ]]; j++) {(8-5) if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ]][ j ]) { RefPicPocList[ i ][ j ]= pocBase – delta_poc_st[ i ][ RplsIdx[ i ]][j ] if( there is a reference picture picA in the DPB with PicOrderCntValequal to RefPicPocList[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” pocBase = RefPicPocList[ i ][ j ] } else { if( delta_poc_msb_cycle_lt[ i ][ j ]is equal to 0&& there is a reference picture picA in the DPB with PicOrderCntVal& ( MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ i ][ j ]) RefPicList[ i ][ j ]= picA else if( delta_poc_msb_cycle_lt[ i ][ j ]is equal to 1&& there is a reference picA in the DPB with PicOrderCntVal equal to ( MaxPicOrderCntLsb * ( delta_poc_msb_cycle_lt[ i ][ j ]+prevMsbCycle ) ) + poc_lsb_lt[ i ][ j ]) ) { RefPicList[ i ][ j ]= picA prevMsbCycle+=delta_poc_msb_cycle_lt[ i ][ j ] } else RefPicList[ i ][ j ]= "no reference picture” } } } The requirement for stream consistency is to use the following constraints: for each i equal to 0 or 1, NumEntriesInList[i][RplsIdx[i]] is not less than NumRefIdxActive[i]. The image referenced by each active entry in RefPicList[0] or RefPicList[1] exists in the DPB, and its TemporalId is less than or equal to the TemporalId of the current image. Optionally, the following constraint can also be specified: the entry index of any inactive entry in RefPicList[0] or RefPicList[1] is not used as a reference index for decoding the current image. Optionally, the following constraint can also be specified: the inactive entry in RefPicList[0] or RefPicList[1] does not reference the same image as any other entry in RefPicList[0] or RefPicList[1]. The STRP entries in RefPicList[0] or RefPicList[1] of a strip in an image do not reference the same image as LTRP entries in RefPicList[0] or RefPicList[1] of the same or different strips in the same image. The current image is not referenced by any entry in RefPicList[0] or RefPicList[1]. The difference between the PicOrderCntVal of the current image and the PicOrderCntVal of the image referenced by the LTRP entry is greater than or equal to 224, and such an entry does not exist in RefPicList[0] or RefPicList[1]. Assume that setOfRefPics is a unique set of images referenced by all entries in RefPicList[0] and all entries in RefPicList[1]. The number of images in setOfRefPics is less than or equal to sps_max_dec_pic_buffering_minus1, and all stripes in an image have the same setOfRefPics.

[0206] Each STRP is identified by its PicOrderCntVal value. If each LTRP is referenced by an entry in RefPicList[0] or RefPicList[1] where delta_poc_msb_present_flag[listIdx][i] equals 1, it is identified by its PicOrderCntVal value; otherwise, it is identified by Log2(MaxPicOrderCntLsb) LSBs of its PicOrderCntVal value.

[0207] Alternative Embodiment 1 for Incremental POC MSB Indication of Long-Term Reference Image Entries This embodiment is an alternative to the embodiment described in the preceding section. Similar to the idea in the preceding section, the POC MSB is indicated to distinguish the long-term reference image, instead of using extra bits to represent the POC LSB of the long-term reference image in ref_pic_list_struct(). However, in this alternative embodiment, the POC MSB cycle information is not indicated in ref_pic_list_struct() at the time of indication, but rather in the strip header when the POC MSB cycle information is required. The syntax structure ref_pic_list_struct() can be indicated in both the SPS and the strip header.

[0208]

[0209]

[0210] If delta_poc_msb_present_flag[i][j] is equal to 1, it means that delta_poc_msb_cycle_lt[i][j] exists. If delta_poc_msb_present_flag[i][j] is equal to 0, it means that delta_poc_msb_cycle_lt[i][j] does not exist. When NumLtrpEntries[i] is greater than 0, and for the j-th LTRP entry in the syntax structure ref_pic_list_struct(i, rplsIdx, 1), there exist multiple reference images in the DPB where the value of PicOrderCntVal modulo MaxPicOrderCntLsb equals poc_lsb_lt[i][rplsIdx][jj] during stripe header decoding, where jj is the entry index of the j-th LTRP entry in the syntax structure ref_pic_list_struct(i, rplsIdx, 1), delta_poc_msb_present_flag[i][j] equals 1. If it does not exist, then it is inferred that the value of delta_poc_msb_cycle_lt[i][j] is equal to 0. delta_poc_msb_cycle_lt[i][j] is used to determine the value of the most significant bit of the image sequence number value of the j-th LTRP entry in the syntax structure ref_pic_list_struct(i, rplsIdx, 1). If it does not exist, delta_poc_msb_cycle_lt[i][j] is assumed to be equal to 0.

[0211]

[0212] If delta_poc_msb_present_flag[i][j] equals 1, it indicates that delta_poc_msb_cycle_lt[i][j] exists. If delta_poc_msb_present_flag[i][j] equals 0, it indicates that delta_poc_msb_cycle_lt[i][j] does not exist. When NumLtrpEntries[i] is greater than 0 and there are multiple reference images in the DPB where the value of PicOrderCntVal modulo MaxPicOrderCntLsb equals poc_lsb_lt[i][rplsIdx][j] during stripe header decoding, delta_poc_msb_present_flag[i][j] equals 1. If it does not exist, it is inferred that the value of delta_poc_msb_cycle_lt[i][j] is equal to 0. `delta_poc_msb_cycle_lt[i][j]` is used to determine the most significant bit of the image sequence number value of the j-th entry in the syntax structure `ref_pic_list_struct(i, rplsIdx, 1)`. If it does not exist, `delta_poc_msb_cycle_lt[i][j]` is assumed to be 0. `poc_lsb_lt[listIdx][rplsIdx][i]` represents the modulo value of the image sequence number of the image referenced by the i-th entry in the syntax structure `ref_pic_list_struct(listIdx, rplsIdx, ltrpFlag)`, modulo `MaxLtPicOrderCntLsb`. The length of the syntax element `poc_lsb_lt[listIdx][rplsIdx][i]` is Log2(MaxLtPicOrderCntLsb).

[0213] The modification to the decoding process used for image sequence numbering is as follows: at any point during the decoding process, the values ​​of PicOrderCntVal&(MaxLtPicOrderCntLsb – 1) for any two reference images in the DPB are not the same.

[0214] In strip header design 1, the reference image lists RefPicList[0] and RefPicList[1] are constructed as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] msbCycleIdx = 0 for( j = 0, pocBase = PicOrderCntVal; j<NumEntriesInList[ i ][RplsIdx[ i ]]; j++) { if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ]][ j ]) { RefPicPocList[ i ][ j ]= pocBase – delta_poc_st[ i ][ RplsIdx[ i ]][j ] if( there is a reference picture picA in the DPB with PicOrderCntValequal to RefPicPocList[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” pocBase = RefPicPocList[ i ][ j ] } else { if( delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ]is equal to 0&& there is a reference picture picA in the DPB with PicOrderCntVal& ( MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ i ][ RplsIdx[ i ]][j ]) RefPicList[ i ][ j ]= picA else if( delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ]is equal to 1&& there is a reference picture picA in the DPB with PicOrderCntValequal to ( MaxPicOrderCntLsb * delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ]) +poc_lsb_lt[ i ][ RplsIdx[ i ]][ j ]) ) { RefPicList[i][j] = picA msbCycleIdx++ else RefPicList[ i ][ j ]= "no reference picture" } } } Alternatively, in strip header design 1, the semantics of delta_poc_msb_cycle_lt[listIdx][i] can be represented as an increment of increments, so that the reference image list can be updated as follows: The reference image lists RefPicList[0] and RefPicList[1] are constructed as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] pevMsbCycle = 0 msbCycleIdx = 0 for( j = 0, pocBase = PicOrderCntVal; j <NumEntriesInList[ i ][RplsIdx[ i ]]; j++) { if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ]][ j ]) { RefPicPocList[ i ][ j ]= pocBase – delta_poc_st[ i ][ RplsIdx[ i ]][j ] if( there is a reference picture picA in the DPB with PicOrderCntValequal to RefPicPocList[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” pocBase = RefPicPocList[ i ][ j ] } else { if( delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ]is equal to 0&& there is a reference picutre picA in the DPB with PicOrderCntVal& ( MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ i ][ RplsIdx[ i ]][j ]) RefPicList[ i ][ j ]= picA else if( delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ]is equal to 1&& there is a reference picture picA in the DPB with PicOrderCntValequal to ( MaxPicOrderCntLsb * ( delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ]+ prevMsbCycle ) + poc_lsb_lt[ i ][ RplsIdx[ i ]][ j ]) ) { RefPicList[i][j] = picA prevMsbCycle+=delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ] msbCycleIdx++ else RefPicList[ i ][ j ]= "no reference picture" } } } In strip header design 2, the reference image lists RefPicList[0] and RefPicList[1] are constructed as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] for( j = 0, pocBase = PicOrderCntVal; j <NumEntriesInList[ i ][RplsIdx[ i ]]; j++) { if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ]][ j ]) { RefPicPocList[ i ][ j ]= pocBase – delta_poc_st[ i ][ RplsIdx[ i ]][j ] if( there is a reference picture picA in the DPB with PicOrderCntValequal to RefPicPocList[ i ][ j ]) RefPicList[i][j] = picA else RefPicList[ i ][ j ]= "no reference picture" pocBase = RefPicPocList[ i ][ j ] } else { if( delta_poc_msb_cycle_lt[ i ][ j ]is equal to 0&& there is a reference picture picA in the DPB with PicOrderCntVal& ( MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ i ][ RplsIdx[ i ]][j ]) RefPicList[i][j] = picA else if( delta_poc_msb_cycle_lt[ i ][ j ]is equal to 1&& there is a reference picture picA in the DPB with PicOrderCntValequal to ( MaxPicOrderCntLsb * delta_poc_msb_cycle_lt[ i ][ j ]) + poc_lsb_lt[ i ][ RplsIdx[ i ]][ j ]) ) RefPicList[i][j] = picA else RefPicList[ i ][ j ]= "no reference picture" } } } Alternatively, in strip header design 2, the semantics of delta_poc_msb_cycle_lt[listIdx][i] can be represented as an increment of increments, so that the reference image list construction can be updated as follows: The construction methods of reference image lists RefPicList[0] and RefPicList[1] are as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] prevMsbCycle = 0 for( j = 0, pocBase = PicOrderCntVal; j<NumEntriesInList[ i ][RplsIdx[ i ]]; j++) { if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ]][ j ]) { RefPicPocList[ i ][ j ]= pocBase – delta_poc_st[ i ][ RplsIdx[ i ]][j ] if( there is a reference picture picA in the DPB with PicOrderCntValequal to RefPicPocList[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” pocBase = RefPicPocList[ i ][ j ] } else { if( delta_poc_msb_cycle_lt[ i ][ j ]is equal to 0&& there is a reference picture picA in the DPB with PicOrderCntVal& ( MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ i ][ RplsIdx[ i ]][j ]) RefPicList[ i ][ j ]= picA else if( delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ]is equal to 1&& there is a reference picture picA in the DPB with PicOrderCntValequal to ( MaxPicOrderCntLsb * ( delta_poc_msb_cycle_lt[ i ][ j ]+prevMsbCycle ) + poc_lsb_lt[ i ][ RplsIdx[ i ]][ j ]) ) { RefPicList[ i ][ j ]= picA prevMsbCycle += delta_poc_msb_cycle_lt[ i ][ j ] } else RefPicList[ i ][ j ]= "no reference picture” } } } The requirement for stream consistency is to use the following constraints: for each i equal to 0 or 1, NumEntriesInList[i][RplsIdx[i]] is not less than NumRefIdxActive[i]. The image referenced by each active entry in RefPicList[0] or RefPicList[1] exists in the DPB, and its TemporalId is less than or equal to the TemporalId of the current image. Optionally, the following constraint can also be specified: the entry index of any inactive entry in RefPicList[0] or RefPicList[1] is not used as a reference index for decoding the current image. Optionally, the following constraint can also be specified: the inactive entry in RefPicList[0] or RefPicList[1] does not reference the same image as any other entry in RefPicList[0] or RefPicList[1]. The STRP entries in RefPicList[0] or RefPicList[1] of a strip in an image do not reference the same image as LTRP entries in RefPicList[0] or RefPicList[1] of the same or different strips in the same image. The current image is not referenced by any entry in RefPicList[0] or RefPicList[1]. The difference between the PicOrderCntVal of the current image and the PicOrderCntVal of the image referenced by the LTRP entry is greater than or equal to 224, and such an entry does not exist in RefPicList[0] or RefPicList[1]. Assume that setOfRefPics is a unique set of images referenced by all entries in RefPicList[0] and all entries in RefPicList[1]. The number of images in setOfRefPics is less than or equal to sps_max_dec_pic_buffering_minus1, and all stripes in an image have the same setOfRefPics.

[0215] Each STRP is identified by its PicOrderCntVal value. If each LTRP is referenced by an entry in RefPicList[0] or RefPicList[1] where delta_poc_msb_present_flag[i][j] equals 1, it is identified by its PicOrderCntVal value; otherwise, it is identified by Log2(MaxPicOrderCntLsb) LSBs of its PicOrderCntVal value.

[0216] Alternative Embodiment 2 for Incremental POC MSB Indication of Long-Term Reference Image Entries In an alternative embodiment of the invention, the disclosure described in the first or second embodiment can be combined with the above-mentioned "Incremental POC MSB Indication for Long-Term Reference Image Entries" and "Alternative Embodiment 1 of Incremental POC MSB Indication for Long-Term Reference Image Entries," respectively. Aspects of the disclosure to be combined include the additional_lt_poc_lsb indication (i.e., according to the first or second embodiment) and the POC MSB cycle information indication (i.e., according to the above-mentioned embodiments titled "Incremental POC MSB Indication for Long-Term Reference Image Entries" or "Alternative Embodiment 1 of Incremental POC MSB Indication for Long-Term Reference Image Entries"). An example of how to combine the first embodiment with the above-mentioned embodiment titled "Alternative Embodiment 1 of Incremental POC MSB Indication for Long-Term Reference Image Entries" is as follows:

[0217] If delta_poc_msb_present_flag[i][j] equals 1, it means that delta_poc_msb_cycle_lt[i][j] exists. If delta_poc_msb_present_flag[i][i] equals 0, it means that delta_poc_msb_cycle_lt[i][j] does not exist. When NumLtrpEntries[i] is greater than 0, and for the j-th LTRP entry in the syntax structure ref_pic_list_struct(i, rplsIdx, 1), there exist multiple reference images in the DPB where the value of PicOrderCntVal modulo MaxPicOrderLtCntLsb equals poc_lsb_lt[i][rplsIdx][jj] during stripe header decoding, where jj is the entry index of the j-th LTRP entry in the syntax structure ref_pic_list_struct(i, rplsIdx, 1), delta_poc_msb_present_flag[i][j] equals 1. If it does not exist, then it is inferred that the value of delta_poc_msb_cycle_lt[i][j] is equal to 0. delta_poc_msb_cycle_lt[i][j] is used to determine the value of the most significant bit of the image sequence number value of the j-th LTRP entry in the syntax structure ref_pic_list_struct(i, rplsIdx, 1). If it does not exist, delta_poc_msb_cycle_lt[i][j] is assumed to be equal to 0.

[0218] The modification to the decoding process used for image sequence numbering is as follows: at any point during the decoding process, the values ​​of PicOrderCntVal&(MaxLtPicOrderCntLsb – 1) for any two reference images in the DPB are not the same.

[0219] The reference image lists RefPicList[0] and RefPicList[1] are constructed as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] msbCycleIdx = 0 for( j = 0, pocBase = PicOrderCntVal; j<NumEntriesInList[ i ][RplsIdx[ i ]]; j++) { if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ]][ j ]) { RefPicPocList[ i ][ j ]= pocBase – delta_poc_st[ i ][ RplsIdx[ i ]][j ] if( there is a reference picture picA in the DPB with PicOrderCntValequal to RefPicPocList[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” pocBase = RefPicPocList[ i ][ j ] } else { if( delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ]is equal to 0&& there is a reference picture picA in the DPB with PicOrderCntVal& ( MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ i ][ RplsIdx[ i ]][j ]) RefPicList[ i ][ j ]= picA else if( delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ]is equal to 1&& there is a reference picture picA in the DPB with PicOrderCntValequal to ( MaxLtPicOrderCntLsb * delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ])+ poc_lsb_lt[ i ][ RplsIdx[ i ]][ j ]) ) { RefPicList[i][j] = picA msbCycleIdx++ else RefPicList[ i ][ j ]= "no reference picture" } } } Alternatively, the semantics of delta_poc_msb_cycle_lt[listIdx][i] can be represented as an increment of increments, so that the reference image list can be updated as follows: The reference image lists RefPicList[0] and RefPicList[1] are constructed as follows: for( i = 0; i<2; i++ ) { if( ref_pic_list_sps_flag[ i ] ) RplsIdx[ i ] = ref_pic_list_idx[ i ] else RplsIdx[ i ] = num_ref_pic_lists_in_sps[ i ] prevMsbCycle = 0 msbCycleIdx = 0 for( j = 0, pocBase = PicOrderCntVal; j <NumEntriesInList[ i ][RplsIdx[ i ]]; j++) { if( !lt_ref_pic_flag[ i ][ RplsIdx[ i ]][ j ]) { RefPicPocList[ i ][ j ]= pocBase – delta_poc_st[ i ][ RplsIdx[ i ]][j ] if( there is a reference picture picA in the DPB with PicOrderCntValequal to RefPicPocList[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” pocBase = RefPicPocList[ i ][ j ] } else { if( delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ]is equal to 0&& there is a reference picture picA in the DPB with PicOrderCntVal& ( MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ i ][ RplsIdx[ i ]][j ]) RefPicList[ i ][ j ]= picA else if( delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ]is equal to 1&& there is a reference picture picA in the DPB with PicOrderCntValequal to ( MaxLtPicOrderCntLsb * ( delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ]+ prevMsbCycle ) + poc_lsb_lt[ i ][ RplsIdx[ i ]][ j ]) ) { RefPicList[ i ][ j ]= picA prevMsbCycle+=delta_poc_msb_cycle_lt[ i ][ msbCycleIdx ] msbCycleIdx++ else RefPicList[ i ][ j ]= "no reference picture" } } } The requirement for stream consistency is to use the following constraints: for each i equal to 0 or 1, NumEntriesInList[i][RplsIdx[i]] is not less than NumRefIdxActive[i]. The image referenced by each active entry in RefPicList[0] or RefPicList[1] exists in the DPB, and its TemporalId is less than or equal to the TemporalId of the current image. Optionally, the following constraint can also be specified: the entry index of any inactive entry in RefPicList[0] or RefPicList[1] is not used as a reference index for decoding the current image. Optionally, the following constraint can also be specified: the inactive entry in RefPicList[0] or RefPicList[1] does not reference the same image as any other entry in RefPicList[0] or RefPicList[1]. The STRP entries in RefPicList[0] or RefPicList[1] of a strip in an image do not reference the same image as LTRP entries in RefPicList[0] or RefPicList[1] of the same or different strips in the same image. The current image is not referenced by any entry in RefPicList[0] or RefPicList[1]. The difference between the PicOrderCntVal of the current image and the PicOrderCntVal of the image referenced by the LTRP entry is greater than or equal to 224, and such an entry does not exist in RefPicList[0] or RefPicList[1]. Assume that setOfRefPics is a unique set of images referenced by all entries in RefPicList[0] and all entries in RefPicList[1]. The number of images in setOfRefPics is less than or equal to sps_max_dec_pic_buffering_minus1, and all stripes in an image have the same setOfRefPics.

[0220] Each STRP is identified by its PicOrderCntVal value. If each LTRP is referenced by an entry in RefPicList[0] or RefPicList[1] where delta_poc_msb_present_flag[i][j] equals 1, it is identified by its PicOrderCntVal value; otherwise, it is identified by Log2(MaxLtPicOrderCntLsb) LSBs of its PicOrderCntVal value.

[0221] The list of reference images is always indicated in the strip header, distinguishing between short-term and long-term reference images.

[0222] This section describes another alternative embodiment of the invention. The following description refers to the latest VVC WD (i.e., only the increments related to the latest VVC WD in JVET-K1001-v1 are described, while the text in the latest VVC WD mentioned below still applies). The overview of this alternative embodiment is as follows: the reference image list structure is indicated only in the strip header. Both short-term and long-term reference images are identified by their POC LSBs, which may be represented by a number of bits that differs from the number of bits used to represent the POC LSBs indicated in the strip header for deriving the POC values. Furthermore, the number of bits used to represent the POC LSBs used to identify the short-term and long-term reference images may differ.

[0223] NAL Unit Header Syntax

[0224] Sequence Parameter Set RBSP Syntax

[0225] Image Parameter Set (RBSP) Syntax

[0226] Striped header syntax

[0227] Reference image list structure syntax

[0228] NAL Unit Header Semantics The forbidden_zero_bit is equal to 0. The nal_unit_type indicates the type of RBSP data structure included in the NAL unit.

[0229] Table 7-1: NAL Unit Type Codes and NAL Unit Type Categories

[0230] minus 1 from nuh_temporal_id_plus1 represents the time identifier of the NAL unit. The value of nuh_temporal_id_plus1 is not equal to 0. The variable TemporalId is represented as follows: TemporalId = nuh_temporal_id_plus1 – 1.

[0231] When nal_unit_type is IRAP_NUT, the decoded stripe belongs to the IRAP image, and TemporalId equals 0. All VCL NAL units in the access unit have the same TemporalId value. The TemporalId value of the decoded image or access unit is the TemporalId value of the VCL NAL unit in that decoded image or access unit. The TemporalId value of non-VCL NAL units is constrained as follows: If nal_unit_type is SPS_NUT, then TemporalId equals 0, and the TemporalId of the access unit including the NAL unit equals 0. Otherwise, if nal_unit_type is EOS_NUT or EOB_NUT, then TemporalId equals 0. Otherwise, TemporalId is greater than or equal to the TemporalId of the access unit including the NAL unit. When the NAL unit is a non-VCL NAL unit, the TemporalId value is equal to the minimum TemporalId value among all access units to which this non-VCL NAL unit applies. When nal_unit_type is PPS_NUT, the TemporalId can be greater than or equal to the TemporalId of the access unit that includes the NAL unit, because all picture parameter sets (PPS) can be included at the beginning of the bitstream, where the TemporalId of the first decoded picture is equal to 0. When nal_unit_type is PREFIX_SEI_NUT or SUFFIX_SEI_NUT, the TemporalId can be greater than or equal to the TemporalId of the access unit that includes the NAL unit, because the supplemental enhancement information (SEI) NAL unit can include a subset of the bitstream to which the access unit includes the access unit, where the TemporalId values ​​of these access units are greater than the TemporalId of the access unit that includes the SEI NAL unit. nuh_reserved_zero_7bits equals '0000000'. Other values ​​for nuh_reserved_zero_7bits may be specified by ITU-T or ISO / IEC in the future. The decoder ignores (i.e. removes and discards) NAL units whose nuh_reserved_zero_7bits value is not equal to '0000000'.

[0232] Sequence Parameter Set RBSP Semantics log2_max_pic_order_cnt_lsb_minus4 represents the value of the variable MaxPicOrderCntLsb used in the decoding process for image sequence numbering, as shown below: MaxPicOrderCntLsb = 2 ( log2_max_pic_order_cnt_lsb_minus4 + 4 ) .

[0233] The value of `log2_max_pic_order_cnt_lsb_minus4` ranges from 0 to 12 (inclusive). `sps_max_dec_pic_buffering_minus1` plus 1 indicates the maximum size of the decoding image buffer required by CVS, in units of image storage buffers. The value of `sps_max_dec_pic_buffering_minus1` ranges from 0 to `MaxDpbSize` – 1 (inclusive), where `MaxDpbSize` is as shown elsewhere. `additional_st_poc_lsb` represents the value of the variable `MaxStPicOrderCntLsb` used in the decoding process for the reference image list, as shown below: MaxStPicOrderCntLsb = 2 ( log2_max_pic_order_cnt_lsb_minus4 + 4 + additional_st_poc_lsb ) .

[0234] The value of `additional_st_poc_lsb` ranges from 0 to 32 – log2_max_pic_order_cnt_lsb_minus4 – 4 (inclusive). `long_term_ref_pics_flag` being 0 indicates that no LTRP is used for inter-frame prediction of any decoded images in CVS. `long_term_ref_pics_flag` being 1 indicates that LTRP can be used for inter-frame prediction of one or more decoded images in CVS. `additional_lt_poc_lsb` represents the value of the variable `MaxLtPicOrderCntLsb` used in the decoding process for the reference image list, as shown below: MaxLtPicOrderCntLsb = 2 ( log2_max_pic_order_cnt_lsb_minus4 + 4 + additional_st_poc_lsb + additional_lt_poc_lsb ) .

[0235] The value range of additional_lt_poc_lsb is 0~32 – log2_max_pic_order_cnt_lsb_minus4 – 4 – additional_st_poc_lsb (inclusive). If it does not exist, it is assumed that the value of additional_lt_poc_lsb is equal to 0.

[0236] Image parameter set RBSP semantics When i equals 0, incrementing 1 in num_ref_idx_default_active_minus1[i] indicates the inferred value of the variable NumRefIdxActive[0] for the P-band or B-band where num_ref_idx_active_override_flag is 0; when i equals 1, incrementing 1 in num_ref_idx_default_active_minus1[i] indicates the inferred value of NumRefIdxActive[1] for the B-band where num_ref_idx_active_override_flag is 0. The value range of num_ref_idx_default_active_minus1[i] is 0~14 (inclusive).

[0237] Strip Header Semantics If present, the values ​​of each of the slice_pic_parameter_set_id and slice_pic_order_cnt_lsb syntax elements are the same across all slice headers of the decoded image. slice_type indicates the decoding type of the slice, as shown in Table 7-3.

[0238] Table 7-3: Relationship between name and slice_type

[0239] When nal_unit_type is IRAP_NUT, that is, when the image is an IRAP image, slice_type is 2.

[0240] slice_pic_order_cnt_lsb represents the value of the current image sequence number modulo MaxPicOrderCntLsb. The length of the syntax element slice_pic_order_cnt_lsb is (log2_max_pic_order_cnt_lsb_minus4 + 4) bits. The value range of slice_pic_order_cnt_lsb is 0 to MaxPicOrderCntLsb – 1 (inclusive). If it does not exist, it is inferred that slice_pic_order_cnt_lsb is equal to 0. num_ref_idx_active_override_flag is 1, indicating that the P-slice and B-slice have the syntax element num_ref_idx_active_minus1[0], while the B-slice has the syntax element num_ref_idx_active_minus1[1]. The value of num_ref_idx_active_override_flag is 0, indicating that the syntax elements num_ref_idx_active_minus1[0] and num_ref_idx_active_minus1[1] do not exist. If they exist, num_ref_idx_active_minus1[i] represents the value of the variable NumRefIdxActive[i], as shown below: NumRefIdxActive[ i ] = num_ref_idx_active_minus1[ i ]+ 1.

[0241] The value of num_ref_idx_active_minus1[i] ranges from 0 to 14 (inclusive). The value of NumRefIdxActive[i] – 1 represents the maximum reference index in the reference image list i that can be used to decode the stripe. When the value of NumRefIdxActive[i] is equal to 0, there is no reference index in the reference image list i that can be used to decode the stripe. When i is equal to 0 or 1, if the current stripe is a B stripe and num_ref_idx_active_override_flag is 0, then it is inferred that NumRefIdxActive[i] is equal to num_ref_idx_default_active_minus1[i] + 1. If the current stripe is a P stripe and num_ref_idx_active_override_flag is 0, then it is inferred that NumRefIdxActive[0] is equal to num_ref_idx_default_active_minus1[0] + 1. If the current stripe is a P stripe, then it is inferred that NumRefIdxActive[1] is equal to 0. If the current stripe is an I stripe, then it is inferred that both NumRefIdxActive[0] and NumRefIdxActive[1] are equal to 0. Alternatively, if i is equal to 0 or 1, assume that rplsIdx1 is set to ref_pic_list_sps_flag[i] ? ref_pic_list_idx[i]: num_ref_pic_lists_in_sps[i], and numRpEntries[i] is set to num_strp_entries[i][rplsIdx1] + num_ltrp_entries[i][rplsIdx1]. When NumRefIdxActive[i] is greater than numRpEntries[i], the value of NumRefIdxActive[i] is set to numRpEntries[i].

[0242] Reference image list structure semantics The syntax structure `ref_pic_list_struct(listIdx, ltrpFlag)` can exist in the strip header. If it exists in the strip header, then `ref_pic_list_struct(listIdx, ltrpFlag)` represents the list of reference images `listIdx` for the current image (including the images in the strip). `num_strp_entries[listIdx]` represents the number of STRP entries in the syntax structure `ref_pic_list_struct(listIdx, ltrpFlag)`. `num_ltrp_entries[listIdx]` represents the number of LTRP entries in the syntax structure `ref_pic_list_struct(listIdx, ltrpFlag)`. If it does not exist, then `num_ltrp_entries[listIdx]` is inferred to be 0. The derivation of the variable `NumEntriesInList[listIdx]` is as follows: NumEntriesInList[listIdx] = num_strp_entries[listIdx]+ num_ltrp_entries[listIdx].

[0243] The value range of NumEntriesInList[listIdx] is 0 to sps_max_dec_pic_buffering_minus1 (inclusive). lt_ref_pic_flag[listIdx][i] equals 1, indicating that the i-th entry in the syntax structure ref_pic_list_struct(listIdx, ltrpFlag) is an LTRP entry. lt_ref_pic_flag[listIdx][i] equals 0, indicating that the i-th entry in the syntax structure ref_pic_list_struct(listIdx, ltrpFlag) is a STRP entry. If it does not exist, then the value of lt_ref_pic_flag[listIdx][i] is inferred to be 0. The requirement for bitstream consistency is that the sum of lt_ref_pic_flag[listIdx][i] (where all values ​​of i are in the range 0 to NumEntriesInList[listIdx] – 1 (inclusive)) equals num_ltrp_entries[listIdx][rplsIdx]. When lt_ref_pic_flag[listIdx][i] equals 0, poc_lsb_st[listIdx][i] represents the value modulo MaxStPicOrderCntLsb of the image sequence number of the image referenced by the i-th entry in the syntax structure ref_pic_list_struct(listIdx, ltrpFlag). The length of the syntax element poc_lsb_st[listIdx][i] is Log2(MaxStPicOrderCntLsb) bits. When lt_ref_pic_flag[listIdx][i] equals 1, poc_lsb_lt[listIdx][i] represents the value modulo MaxLtPicOrderCntLsb of the image sequence number of the image referenced by the i-th entry in the syntax structure ref_pic_list_struct(listIdx, ltrpFlag). The length of the syntax element poc_lsb_lt[listIdx][i] is Log2(MaxLtPicOrderCntLsb) bits.

[0244] The decoding process is described below.

[0245] General Decoding Process The decoding process of the current image CurrPic is as follows. The decoding of the NAL unit is described in detail below. The following process uses the syntax elements of the stripe header layer and higher layers to describe the following decoding process. Derive the variables and functions related to the image sequence number. Call only for the first strip in the image. At the beginning of the decoding process of each strip in a non-IRAP image, the decoding process for constructing the reference image list is called to derive reference image list 0 (RefPicList[0]) and reference image list 1 (RefPicList[1]). Call the decoding process for reference image identification, where the reference image can be identified as "not for reference" or "for long-term reference". Call only for the first strip in the image. Call the decoding process for the decoding tree unit, scaling, transformation, in-loop filtering, etc. After all strips in the current image have been decoded, the current decoded image is identified as "for short-term reference".

[0246] NAL unit decoding process The input to this process is the NAL units of the current image and their associated non-VCL NAL units. The output of this process is the parsed RBSP syntax structure encapsulated within the NAL units. The decoding process for each NAL unit extracts the RBSP syntax structure from the NAL unit and then parses the RBSP syntax structure.

[0247] strip decoding process Decoding process for image sequential numbering The output of this process is the image sequence number PicOrderCntVal for the current image. The image sequence number is used for image recognition, deriving motion parameters in merge mode, motion vector prediction, and decoder consistency checks. Each decoded image is associated with an image sequence number variable (denoted as PicOrderCntVal). If the current image is not an IRAP image, the variables prevPicOrderCntLsb and prevPicOrderCntMsb are derived as follows: Assume prevTid0Pic is the previous image in the decoding order where TemporalId equals 0. Set the variable prevPicOrderCntLsb to the slice_pic_order_cnt_lsb of prevTid0Pic. Set the variable prevPicOrderCntMsb to the PicOrderCntMsb of prevTid0Pic. The variable PicOrderCntMsb for the current image is derived as follows: If the current image is an IRAP image, set PicOrderCntMsb to 0. Otherwise, the derivation of PicOrderCntMsb is as follows: if( ( slice_pic_order_cnt_lsb <prevpicordercntlsb )&& ( prevpicordercntlsb – slice_pic_order_cnt_lsb )≥(maxpicordercntlsb 2 ) picordercntmsb="prevPicOrderCntMsb" + maxpicordercntlsb else if( (slice_pic_order_cnt_lsb>prevPicOrderCntLsb )&& ( ( slice_pic_order_cnt_lsb – prevPicOrderCntLsb )>(MaxPicOrderCntLsb / 2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb – MaxPicOrderCntLsb else PicOrderCntMsb = prevPicOrderCntMsb The derivation of PicOrderCntVal is as follows: PicOrderCntVal = PicOrderCntMsb + slice_pic_order_cnt_lsb.

[0248] Since the slice_pic_order_cnt_lsb of the inferred IRAP image is 0, and both prevPicOrderCntLsb and prevPicOrderCntMsb are set to 0, the PicOrderCntVal of all IRAP images is equal to 0. The value range of PicOrderCntVal is -2. 31 ~2 31 – 1 (inclusive of end values). Within a CVS, the PicOrderCntVal values ​​of any two decoded images are different. At any point during the decoding process, the values ​​of PicOrderCntVal & (MaxStPicOrderCntLsb – 1) of any two short-term reference images in the DPB are different. At any point during the decoding process, the values ​​of PicOrderCntVal & (MaxLtPicOrderCntLsb – 1) of any two reference images in the DPB are different.

[0249] The function PicOrderCnt(picX) is represented as follows: PicOrderCnt(picX) = PicOrderCntVal of image picX.

[0250] The function DiffPicOrderCnt(picA, picB) is represented as follows: DiffPicOrderCnt( picA, picB ) = PicOrderCnt( picA ) – PicOrderCnt(picB ).

[0251] The bitstream does not include values ​​that cause the values ​​of DiffPicOrderCnt(picA, picB) used during decoding to be out of range – 2. 15 ~2 15 - Data within 1 (inclusive). Assuming X is the current image, and Y and Z are two other images in the same CVS, when both DiffPicOrderCnt(X, Y) and DiffPicOrderCnt(X, Z) are positive or both are negative, it is assumed that Y and Z are in the same output order direction relative to X.

[0252] Decoding process used for constructing the reference image list This procedure is invoked at the start of the decoding process for each strip in a non-IRAP image. Reference images are addressed by reference indices. A reference index is an index in a list of reference images. When decoding I stripes, the list of reference images is not used to decode the strip data. When decoding P stripes, only reference image list 0 (i.e., RefPicList[0]) is used to decode the strip data. When decoding B stripes, both reference image list 0 and reference image list 1 (i.e., RefPicList[1]) are used to decode the strip data. At the start of the decoding process for each strip in a non-IRAP image, the reference image lists RefPicList[0] and RefPicList[1] are derived. These two reference image lists are used to identify reference images or decode strip data. If the I-band in a non-IRAP image is not the first band in the image, then RefPicList[0] and RefPicList[1] can be derived to check bitstream consistency. However, it is not necessary to derive RefPicList[0] and RefPicList[1] when decoding the current image or images that are decoded after the current image. If the P-band is not the first band in the image, then RefPicList[1] can be derived to check bitstream consistency. However, it is not necessary to derive RefPicList[1] when decoding the current image or images that are decoded after the current image.

[0253] The reference image lists RefPicList[0] and RefPicList[1] are constructed as follows: for( i = 0; i<2; i++ ) { for( j = 0; j <NumEntriesInList[ i ]; j++) { if( lt_ref_pic_flag[ i ][ j ]) { if( there is a reference picA in the DPB with PicOrderCntVal&(MaxLtPicOrderCntLsb – 1 ) equal to poc_lsb_lt[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” } } } for( i = 0; i<2; i++ ) { for( j = 0; j<NumEntriesInList[ i ]; j++) { if( !lt_ref_pic_flag[ i ][ j ]) { if( there is a short-term reference picture picA in the DPB withPicOrderCntVal&( MaxStPicOrderCntLsb – 1 ) equal to poc_lsb_st[ i ][ j ]) RefPicList[ i ][ j ]= picA else RefPicList[ i ][ j ]= "no reference picture” } } } In each case where i equals 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are called active entries in RefPicList[i], and the other entries in RefPicList[i] are called inactive entries in RefPicList[i]. If lt_ref_pic_flag[i][j] equals 0, then each entry in RefPicList[i][j] (j is in the range 0 to NumEntriesInList[i] – 1 (inclusive)) is called a STRP entry; otherwise, it is called an LTRP entry. A particular image may be referenced by both entries in RefPicList[0] and entries in RefPicList[1]. A particular image may also be referenced by multiple entries in RefPicList[0] or multiple entries in RefPicList[1]. The active entries in RefPicList[0] and RefPicList[1] collectively reference all reference images that can be used for inter-frame prediction of the current image and one or more images whose decoding order is after the current image. The inactive entries in RefPicList[0] and RefPicList[1] collectively reference all reference images that are not used for inter-frame prediction of the current image but can be used for inter-frame prediction of one or more images whose decoding order is after the current image. There may be one or more entries in RefPicList[0] or RefPicList[1] that are equal to "no reference image" because there is no corresponding image in the DPB. Ignore each inactive entry in RefPicList[0] or RefPicList[0] that is equal to "no reference image". Infer unintentional image loss for each active entry in RefPicList[0] or RefPicList[1] that is equal to "no reference image".

[0254] The requirement for stream consistency is to use the following constraints: for each i equal to 0 or 1, NumEntriesInList[i] is not less than NumRefIdxActive[i]. The image referenced by each active entry in RefPicList[0] or RefPicList[1] exists in the DPB, and its TemporalId is less than or equal to the TemporalId of the current image. Optionally, the following constraint can also be specified: the entry index of any inactive entry in RefPicList[0] or RefPicList[1] is not used as a reference index for decoding the current image. Optionally, the following constraint can also be specified: the inactive entry in RefPicList[0] or RefPicList[1] does not reference the same image as any other entry in RefPicList[0] or RefPicList[1]. The STRP entries in RefPicList[0] or RefPicList[1] of a strip in an image do not reference the same image as the LTRP entries in RefPicList[0] or RefPicList[1] of the same or different strips in the same image. The current image is not referenced by any entry in RefPicList[0] or RefPicList[1]. The difference between the PicOrderCntVal of the current image and the PicOrderCntVal of the image referenced by the LTRP entry is greater than or equal to 224, and such an entry does not exist in RefPicList[0] or RefPicList[1]. Assume that setOfRefPics is a unique set of images referenced by all entries in RefPicList[0] and all entries in RefPicList[1]. The number of images in setOfRefPics is less than or equal to sps_max_dec_pic_buffering_minus1, and all stripes in an image have the same setOfRefPics.

[0255] Decoding process for reference image identifiers This process is called once for each image after the decoding process of the strip header and the list of reference images for the strip, but before decoding the strip data. This process may result in one or more reference images in the DPB being marked as "not for reference" or "for long-term reference". Decoded images in the DPB can be marked as "not for reference", "for short-term reference", or "for long-term reference", but at any given moment during the decoding process, only one of these three marking states can exist. Assigning one of the marking states to an image implicitly removes the other marking states (if applicable). When an image is said to be marked as "for reference", this uniformly means marking the image as "for short-term reference" or "for long-term reference" (but not simultaneously). If the current image is an IRAP image, then all reference images currently included in the DPB (if any) are marked as "not for reference". STRPs are identified by Log2(MaxStPicOrderCntLsb) LSBs of their PicOrderCntVal values. LTRPs are identified by their PicOrderCntVal value of Log2(MaxLtPicOrderCntLsb) LSBs.

[0256] For each LTRP entry in RefPicList[0] or RefPicList[1], if the referenced image is a STRP, the image is marked as "for long-term reference". Each reference image in DPB that is not referenced by any entry in RefPicList[0] or RefPicList[1] is marked as "not for reference".

[0257] The list of reference images is always indicated in the strip header, without distinguishing between short-term and long-term reference images.

[0258] This section describes another alternative embodiment of the invention. The following description refers to the latest VVC WD (i.e., only the increments related to the latest VVC WD in JVET-K1001-v1 are described, while the text in the latest VVC WD mentioned below still applies). The overview of this alternative embodiment is as follows: The reference image list structure is indicated only in the strip header. No distinction is made between short-term and long-term reference images. All reference images are simply referred to as reference images. Reference images are identified by their POC LSBs, which are represented by a number of bits that differs from the number of bits used to represent the POC LSBs indicated in the strip header for deriving the POC values.

[0259] Abbreviations. Use text from Chapter 4 of VVC WD.

[0260] NAL Unit Header Syntax

[0261] Sequence Parameter Set RBSP Syntax

[0262] Image Parameter Set (RBSP) Syntax

[0263] Striped header syntax

[0264] Reference image list structure syntax

[0265] NAL Unit Header Semantics The forbidden_zero_bit is equal to 0. The nal_unit_type indicates the type of RBSP data structure included in the NAL unit.

[0266] Table 7-1: NAL Unit Type Codes and NAL Unit Type Categories

[0267] minus 1 from nuh_temporal_id_plus1 represents the time identifier of the NAL unit. The value of nuh_temporal_id_plus1 is not equal to 0. The variable TemporalId is represented as follows: TemporalId = nuh_temporal_id_plus1 – 1.

[0268] When nal_unit_type is IRAP_NUT, the decoded stripe belongs to the IRAP image, and TemporalId equals 0. All VCL NAL units in the access unit have the same TemporalId value. The TemporalId value of the decoded image or access unit is the TemporalId value of the VCL NAL unit in that decoded image or access unit. The TemporalId value constraints for non-VCL NAL units are as follows: If nal_unit_type is SPS_NUT, then TemporalId equals 0, and the TemporalId of the access unit including the NAL unit is also 0. Otherwise, if nal_unit_type is EOS_NUT or EOB_NUT, then TemporalId equals 0. Otherwise, TemporalId is greater than or equal to the TemporalId of the access unit including the NAL unit. When the NAL unit is a non-VCLNAL unit, the TemporalId value is equal to the minimum of the TemporalId values ​​of all access units to which the non-VCL NAL unit applies. When nal_unit_type is PPS_NUT, TemporalId can be greater than or equal to the TemporalId of the access unit including the NAL unit because all picture parameter sets (PPS) can be included at the beginning of the bitstream, where the TemporalId of the first decoded picture is equal to 0. When nal_unit_type is PREFIX_SEI_NUT or SUFFIX_SEI_NUT, TemporalId can be greater than or equal to the TemporalId of the access unit that includes the NAL unit, because the SEI NAL unit can include information applied to a subset of the bitstream that includes the access unit, where the TemporalId values ​​of these access units are greater than the TemporalId of the access unit that includes the SEI NAL unit. nuh_reserved_zero_7bits is equal to '0000000'. Other values ​​for nuh_reserved_zero_7bits may be specified by ITU-T or ISO / IEC in the future. The decoder ignores (i.e., removes and discards) NAL units whose nuh_reserved_zero_7bits value is not equal to '0000000'.

[0269] Sequence Parameter Set RBSP Semantics log2_max_pic_order_cnt_lsb_minus4 represents the value of the variable MaxPicOrderCntLsb used in the decoding process for image sequence numbering, as shown below: MaxPicOrderCntLsb = 2 ( log2_max_pic_order_cnt_lsb_minus4 + 4 ) .

[0270] The value of `log2_max_pic_order_cnt_lsb_minus4` ranges from 0 to 12 (inclusive). `sps_max_dec_pic_buffering_minus1` plus 1 indicates the maximum size of the decoding image buffer required by CVS, in units of image storage buffers. The value of `sps_max_dec_pic_buffering_minus1` ranges from 0 to `MaxDpbSize` – 1 (inclusive), where `MaxDpbSize` is as shown elsewhere. `additional_ref_poc_lsb` represents the value of the variable `MaxRefPicOrderCntLsb` used in the decoding process for the reference image list, as shown below: MaxRefPicOrderCntLsb = 2 ( log2_max_pic_order_cnt_lsb_minus4 + 4 + additional_ref_poc_lsb ) .

[0271] The value range of additional_ref_poc_lsb is 0~32 – log2_max_pic_order_cnt_lsb_minus4 – 4 (inclusive).

[0272] Image parameter set RBSP semantics When i equals 0, incrementing 1 in num_ref_idx_default_active_minus1[i] indicates the inferred value of the variable NumRefIdxActive[0] for the P-band or B-band where num_ref_idx_active_override_flag is 0; when i equals 1, incrementing 1 in num_ref_idx_default_active_minus1[i] indicates the inferred value of NumRefIdxActive[1] for the B-band where num_ref_idx_active_override_flag is 0. The value range of num_ref_idx_default_active_minus1[i] is 0~14 (inclusive).

[0273] Strip Header Semantics If present, the values ​​of each of the slice_pic_parameter_set_id and slice_pic_order_cnt_lsb syntax elements are the same across all slice headers of the decoded image. slice_type indicates the decoding type of the slice, as shown in Table 7-3.

[0274] Table 7-3: Relationship between name and slice_type

[0275] When nal_unit_type is IRAP_NUT, that is, when the image is an IRAP image, slice_type is 2. slice_pic_order_cnt_lsb represents the value of the current image sequence number modulo MaxPicOrderCntLsb. The length of the syntax element slice_pic_order_cnt_lsb is (log2_max_pic_order_cnt_lsb_minus4 + 4) bits. The value range of slice_pic_order_cnt_lsb is 0 to MaxPicOrderCntLsb – 1 (inclusive). If it does not exist, it is inferred that slice_pic_order_cnt_lsb is equal to 0. num_ref_idx_active_override_flag is 1, indicating that the P slice and B slice have the syntax element num_ref_idx_active_minus1[0], while the B slice has the syntax element num_ref_idx_active_minus1[1]. The value of num_ref_idx_active_override_flag is 0, indicating that the syntax elements num_ref_idx_active_minus1[0] and num_ref_idx_active_minus1[1] do not exist. If they exist, num_ref_idx_active_minus1[i] represents the value of the variable NumRefIdxActive[i], as shown below: NumRefIdxActive[ i ] = num_ref_idx_active_minus1[ i ]+ 1.

[0276] The value of num_ref_idx_active_minus1[i] ranges from 0 to 14 (inclusive). The value of NumRefIdxActive[i] – 1 represents the maximum reference index in the reference image list i that can be used to decode the stripe. When the value of NumRefIdxActive[i] is equal to 0, there is no reference index in the reference image list i that can be used to decode the stripe. When i is equal to 0 or 1, if the current stripe is a B stripe and num_ref_idx_active_override_flag is 0, then it is inferred that NumRefIdxActive[i] is equal to num_ref_idx_default_active_minus1[i] + 1. If the current stripe is a P stripe and num_ref_idx_active_override_flag is 0, then it is inferred that NumRefIdxActive[0] is equal to num_ref_idx_default_active_minus1[0] + 1. If the current stripe is a P stripe, then it is inferred that NumRefIdxActive[1] is equal to 0. If the current stripe is an I stripe, then it is inferred that both NumRefIdxActive[0] and NumRefIdxActive[1] are equal to 0. Alternatively, if i is equal to 0 or 1, assume that rplsIdx1 is set to ref_pic_list_sps_flag[i] ? ref_pic_list_idx[i]: num_ref_pic_lists_in_sps[i], and numRpEntries[i] is set to num_strp_entries[i][rplsIdx1] + num_ltrp_entries[i][rplsIdx1]. When NumRefIdxActive[i] is greater than numRpEntries[i], the value of NumRefIdxActive[i] is set to numRpEntries[i].

[0277] Reference image list structure semantics The syntax structure `ref_pic_list_struct(listIdx)` can exist in the strip header. If it does, `ref_pic_list_struct(listIdx)` represents the list of reference images `listIdx` for the current image (including the images in the strip). `num_ref_entries[listIdx]` represents the number of entries in the syntax structure `ref_pic_list_struct(listIdx)`. The derivation of the variable `NumEntriesInList[listIdx]` is as follows: NumRefPicEntriesInRpl[ listIdx ] = num_ref_entries[ listIdx ].

[0278] The value range of NumRefPicEntries[listIdx] is 0 to sps_max_dec_pic_buffering_minus1 (inclusive). poc_ref_lsb[listIdx][i] represents the value of MaxRefPicOrderCntLsb modulo the image sequence number of the image referenced by the i-th entry in the syntax structure ref_pic_list_struct. The length of the syntax element poc_ref_lsb[listIdx][i] is Log2(MaxRefPicOrderCntLsb) bits.

[0279] The decoding process is described below.

[0280] General Decoding Process The decoding process of the current image CurrPic is as follows. The decoding of the NAL unit is described in detail below. The following process uses the syntax elements of the stripe header and higher layers to describe the following decoding process. Derive the variables and functions related to the image sequence number. Call only for the first strip in the image. At the beginning of the decoding process of each strip in a non-IRAP image, the decoding process for constructing the reference image list is called to derive reference image list 0 (RefPicList[0]) and reference image list 1 (RefPicList[1]). Call the decoding process for reference image identification, where the reference image can be identified as "not for reference". Call only for the first strip in the image. Call the decoding process for the decoding tree unit, scaling, transformation, in-loop filtering, etc. After all strips in the current image have been decoded, the current decoded image is identified as "for reference".

[0281] NAL unit decoding process The input to this process is the NAL units of the current image and their associated non-VCL NAL units. The output of this process is the parsed RBSP syntax structure encapsulated within the NAL units. The decoding process for each NAL unit extracts the RBSP syntax structure from the NAL unit and then parses the RBSP syntax structure.

[0282] strip decoding process Decoding process for image sequential numbering The output of this process is the image sequence number PicOrderCntVal for the current image. The image sequence number is used for image recognition, deriving motion parameters in merge mode, motion vector prediction, and decoder consistency checks. Each decoded image is associated with an image sequence number variable (denoted as PicOrderCntVal). If the current image is not an IRAP image, the variables prevPicOrderCntLsb and prevPicOrderCntMsb are derived as follows: Assume prevTid0Pic is the previous image in the decoding order where TemporalId equals 0. Set the variable prevPicOrderCntLsb to the slice_pic_order_cnt_lsb of prevTid0Pic. Set the variable prevPicOrderCntMsb to the PicOrderCntMsb of prevTid0Pic. The variable PicOrderCntMsb for the current image is derived as follows: If the current image is an IRAP image, then set PicOrderCntMsb to 0. Otherwise, the derivation of PicOrderCntMsb is as follows: if( ( slice_pic_order_cnt_lsb <prevpicordercntlsb )&& ( prevpicordercntlsb – slice_pic_order_cnt_lsb )≥(maxpicordercntlsb 2 ) picordercntmsb="prevPicOrderCntMsb" + maxpicordercntlsb else if( (slice_pic_order_cnt_lsb>prevPicOrderCntLsb )&& ( ( slice_pic_order_cnt_lsb – prevPicOrderCntLsb )>(MaxPicOrderCntLsb / 2 ) ) ) PicOrderCntMsb = prevPicOrderCntMsb – MaxPicOrderCntLsb else PicOrderCntMsb = prevPicOrderCntMsb The derivation of PicOrderCntVal is as follows: PicOrderCntVal = PicOrderCntMsb + slice_pic_order_cnt_lsb.

[0283] Since the slice_pic_order_cnt_lsb of the inferred IRAP image is 0, and both prevPicOrderCntLsb and prevPicOrderCntMsb are set to 0, the PicOrderCntVal of all IRAP images is equal to 0. The value range of PicOrderCntVal is -2. 31 ~2 31 – 1 (inclusive of end values). In a CVS, the PicOrderCntVal values ​​of any two decoded images are different. At any point during the decoding process, the values ​​of PicOrderCntVal & (MaxRefPicOrderCntLsb – 1) of any two reference images in the DPB are different.

[0284] The function PicOrderCnt(picX) is represented as follows: PicOrderCnt(picX) = PicOrderCntVal of image picX.

[0285] The function DiffPicOrderCnt(picA, picB) is represented as follows: DiffPicOrderCnt( picA, picB ) = PicOrderCnt( picA ) – PicOrderCnt(picB ).

[0286] The bitstream does not include values ​​that cause the values ​​of DiffPicOrderCnt(picA, picB) used during decoding to be out of range – 2. 15 ~2 15 - Data within 1 (inclusive). Assuming X is the current image, and Y and Z are two other images in the same CVS, when both DiffPicOrderCnt(X, Y) and DiffPicOrderCnt(X, Z) are positive or both are negative, it is assumed that Y and Z are in the same output order direction relative to X.

[0287] Decoding process used for constructing the reference image list This procedure is invoked at the start of the decoding process for each strip in a non-IRAP image. Reference images are addressed by reference indices. A reference index is an index in a list of reference images. When decoding I stripes, the list of reference images is not used to decode the strip data. When decoding P stripes, only reference image list 0 (i.e., RefPicList[0]) is used to decode the strip data. When decoding B stripes, both reference image list 0 and reference image list 1 (i.e., RefPicList[1]) are used to decode the strip data. At the start of the decoding process for each strip in a non-IRAP image, the reference image lists RefPicList[0] and RefPicList[1] are derived. These two reference image lists are used to identify reference images or decode strip data. If the I-band in a non-IRAP image is not the first band in the image, then RefPicList[0] and RefPicList[1] can be derived to check bitstream consistency. However, it is not necessary to derive RefPicList[0] and RefPicList[1] when decoding the current image or images that are decoded after the current image. If the P-band is not the first band in the image, then RefPicList[1] can be derived to check bitstream consistency. However, it is not necessary to derive RefPicList[1] when decoding the current image or images that are decoded after the current image. The construction methods of the reference image lists RefPicList[0] and RefPicList[1] are as follows: for( i = 0; i<2; i++ ) { for( j = 0, pocBase = PicOrderCntVal; j <NumEntriesInList[ i ]; j++) { if( there is a reference picA in the DPB with PicOrderCntVal&(MaxRefPicOrderCntLsb – 1 ) equal to poc_ref_lsb[ i ][ j ]) RefPicList[i][j] = picA else RefPicList[ i ][ j ]= "no reference picture" } } In the case where i equals 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are called the active entries in RefPicList[i], and the other entries in RefPicList[i] are called the inactive entries in RefPicList[i]. A particular image may be referenced by both entries in RefPicList[0] and entries in RefPicList[1]. A particular image may also be referenced by multiple entries in RefPicList[0] or multiple entries in RefPicList[1]. The active entries in RefPicList[0] and the active entries in RefPicList[1] jointly reference all reference images that can be used for inter-frame prediction of the current image and one or more images whose decoding order follows the current image. The inactive entries in RefPicList[0] and RefPicList[1] collectively refer to a reference image for all inter-frame predictions not used for the current image but which can be used for decoding one or more images in the order of the current image. One or more entries equal to "No reference image" may exist in RefPicList[0] or RefPicList[1] because no corresponding image exists in the DPB. Each inactive entry equal to "No reference image" in RefPicList[0] or RefPicList[0] is ignored. Unintentional image loss is inferred for each active entry equal to "No reference image" in RefPicList[0] or RefPicList[1].

[0288] The requirement for stream consistency is to use the following constraints: For each i equal to 0 or 1, NumEntriesInList[i] is not less than NumRefIdxActive[i]. The image referenced by each active entry in RefPicList[0] or RefPicList[1] exists in the DPB, and its TemporalId is less than or equal to the TemporalId of the current image. Optionally, the following constraint can also be specified: the entry index of any inactive entry in RefPicList[0] or RefPicList[1] is not used as a reference index for decoding the current image. Optionally, the following constraint can also be specified: inactive entries in RefPicList[0] or RefPicList[1] do not reference the same image as any other entry in RefPicList[0] or RefPicList[1]. The current image is not referenced by any entry in RefPicList[0] or RefPicList[1]. The difference between the PicOrderCntVal of the current image and the PicOrderCntVal of the image referenced by the entry is greater than or equal to 224, and such an entry does not exist in RefPicList[0] or RefPicList[1]. Assume that setOfRefPics is a unique set of images referenced by all entries in RefPicList[0] and all entries in RefPicList[1]. The number of images in setOfRefPics is less than or equal to sps_max_dec_pic_buffering_minus1, and all stripes in an image have the same setOfRefPics.

[0289] Decoding process for reference image identifiers This process is called once per image after the decoding process of the strip header and the list of reference images for the strip, but before the strip data is decoded. This process may result in one or more reference images in the DPB being marked as "not for reference". Decoded images in the DPB can be marked as "not for reference" or "for reference", but at any given moment during the decoding process, only one of these two marking states can exist. Assigning one of the marking states to an image implicitly removes the other marking states (if applicable). If the current image is an IRAP image, all reference images currently included in the DPB (if any) are marked as "not for reference". Reference images in the DPB are identified by Log2(MaxRefPicOrderCntLsb) LSBs of their PicOrderCntVal values. Each reference image in the DPB that is not referenced by any entry in RefPicList[0] or RefPicList[1] is marked as "not for reference".

[0290] Another alternative embodiment This section is an alternative embodiment of the method detailed above, titled "Always Indicating a List of Reference Images in the Strip Header While Differentiating Between Short-Term and Long-Term Reference Images." In this alternative embodiment, the POC MSB cycle can be indicated for each LTRP entry in the strip header, similar to HEVC or the method described above, and the following constraint is removed: at any point during the decoding process, the values ​​of PicOrderCntVal&(MaxLtPicOrderCntLsb – 1) for any two reference images in the DPB are not identical.

[0291] Figure 7 This is a schematic diagram of a video decoding device 700 (e.g., video encoder 20 or video decoder 30) provided according to an embodiment of the present invention. The video decoding device 700 is suitable for implementing the disclosed embodiments described herein. The video decoding device 700 includes: an input port 710 and a receiving unit (Rx) 720 for receiving data; a processor, logic unit, or central processing unit (CPU) 730 for processing the data; a transmitting unit (Tx) 740 and an output port 750 for transmitting the data; and a memory 760 for storing the data. The video decoding device 700 may further include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the input port 710, receiving unit 720, transmitting unit 740, and output port 750 for the entry or exit of optical or electrical signals.

[0292] Processor 730 is implemented in both hardware and software. Processor 730 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). Processor 730 communicates with input port 710, receiving unit 720, transmitting unit 740, output port 750, and memory 760. Processor 730 includes a decoding module 770. Decoding module 770 implements the embodiments disclosed above. For example, decoding module 770 performs, processes, prepares, or provides various network connectivity functions. Therefore, including decoding module 770 provides a substantial improvement to the functionality of video decoding device 700 and affects the transitions of video decoding device 700 to different states. Alternatively, decoding module 770 can be implemented with instructions stored in memory 760 and executed by processor 730.

[0293] The video decoding device 700 may also include input and / or output (I / O) devices 780 for data communication with the user. I / O devices 780 may include output devices such as a display for showing video data and speakers for outputting audio data. I / O devices 780 may also include input devices such as a keyboard, mouse, and trackball, as well as corresponding interfaces for interacting with such output devices.

[0294] Memory 760 includes one or more disks, tape drives, or solid-state drives and can be used as an overflow data storage device to store such programs when a program is selected for execution, or to store instructions and data read during program execution. Memory 760 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), or static random-access memory (SRAM).

[0295] Figure 8 This is a schematic diagram of one embodiment of the decoding module 800. In this embodiment, the decoding module 800 is implemented in a video decoding device 802 (e.g., a video encoder 20 or a video decoder 30). The video decoding device 802 includes a receiving module 801. The receiving module 801 is used to receive images for encoding or to receive bitstreams for decoding. The video decoding device 802 includes a transmitting module 807 coupled to the receiving module 801. The transmitting module 807 is used to transmit the bitstream to the decoder or to transmit the decoded image to a display module (e.g., one of a plurality of I / O devices 780).

[0296] The video decoding device 802 includes a storage module 803. The storage module 803 is coupled to at least one of a receiving module 801 or a transmitting module 807. The storage module 803 is used to store instructions. The video decoding device 802 also includes a processing module 805. The processing module 805 is coupled to the storage module 803. The processing module 805 is used to execute the instructions stored in the storage module 803 to perform the methods disclosed herein.

[0297] While several embodiments have been provided in this invention, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the invention. The examples of this invention should be considered illustrative rather than restrictive, and the invention is not limited to the details set forth herein. For example, various elements or components may be combined or integrated into another system, or certain features may be omitted or not implemented.

[0298] Furthermore, without departing from the scope of this invention, the technologies, systems, subsystems, and methods described and illustrated as discrete or separate in the various embodiments can be combined or integrated with other systems, modules, technologies, or methods. Other items shown or described as coupled or directly coupled or communicating with each other may also be indirectly coupled or communicating via an interface, device, or intermediate component in an electrical, mechanical, or other manner. Other variations, substitutions, and alterations can be determined by those skilled in the art without departing from the spirit and scope of this disclosure.< / prevpicordercntlsb> < / prevpicordercntlsb> < / prevpicordercntlsb>

Claims

1. A method for decoding a decoded video stream using a video decoder, characterized in that, The method includes: Obtain a first reference image list and a second reference image list for the current stripe, wherein the current stripe is represented in the decoded video stream; When the current stripe is the first stripe in the current image to which the current stripe belongs, the state of the reference image in the decoded picture buffer (DPB) is identified according to the first reference image list and the second reference image list, wherein the state of the reference image in the decoded picture buffer is one of short-term reference, long-term reference, or no reference. When the reference image is in the state of being used for short-term reference or for long-term reference, the reference image is used in the reconstruction process of the current image or an image after the current image.

2. The method according to claim 1, characterized in that, The first reference image list is RefPictList[0].

3. The method according to claim 1, characterized in that, The second reference image list is RefPictList[1].

4. The method according to any one of claims 1 to 3, characterized in that, The current image includes images from non-intra-random access points (IRAP).

5. The method according to claim 1, characterized in that, The state of the reference image is identified by the image sequence number value (PicOrderCntVal).

6. A decoding device, characterized in that, The decoding device includes: Receiver, used to receive decoded video streams; A memory coupled to the receiver, wherein the memory stores instructions; A processor coupled to the memory and the receiver, wherein the processor is configured to execute the instructions stored in the memory such that the processor: Obtain a first reference image list and a second reference image list for the current stripe, wherein the current stripe is represented in the decoded video stream; When the current stripe is the first stripe in the current image to which the current stripe belongs, the state of the reference image in the decoded picture buffer (DPB) is identified according to the first reference image list and the second reference image list, wherein the state of the reference image in the decoded picture buffer is one of short-term reference, long-term reference, or no reference. When the reference image is in the state of being used for short-term reference or for long-term reference, the reference image is used in the reconstruction process of the current image or an image after the current image.

7. The decoding device according to claim 6, characterized in that, The decoding device further includes a display for displaying the current image.

8. A decoding device, characterized in that, The decoding device includes: Receiver, used to receive and decode the bitstream; A transmitter coupled to the receiver, wherein the transmitter is used to send a decoded image to the display; A memory coupled to at least one of the receiver and the transmitter, wherein the memory is used to store instructions; A processor coupled to the memory, wherein the processor is configured to execute the instructions stored in the memory to perform the method according to any one of claims 1 to 5.

9. A system, characterized in that, The system includes: Encoder; A decoder communicating with the encoder, wherein the encoder or the decoder includes a decoding device or decoding apparatus according to any one of claims 6 to 8.

10. A decoding module, characterized in that, The decoding module includes: The receiving module is used to receive the bitstream and decode it. A transmitting module coupled to the receiving module, wherein the transmitting module is used to transmit a decoded image to the display module; A storage module coupled to at least one of the receiving module or the transmitting module, wherein the storage module is used to store instructions; A processing module coupled to the storage module, wherein the processing module is configured to execute instructions stored in the storage module to perform the method according to any one of claims 1 to 5.

Images