Method and apparatus for reconstructing encoded enhancement layer picture

Abstract

A method of reconstructing an encoded enhancement layer picture, comprising: decoding at least one independent picture header of an encoded picture of a reference layer, the encoded picture of the reference layer being referenced by the encoded enhancement layer picture; decoding a subordinate picture header of the encoded enhancement layer picture; and performing picture header prediction comprising one or both of: predicting at least one value of at least one syntax element of the slave picture header from at least one value of at least one corresponding syntax element of the at least one independent picture header; and inferring at least one value of an inexistent syntax element of the slave picture header to be equal to at least one value of an existent syntax element of the at least one independent picture header.

Description

[0001] Priority information

[0002] This application claims priority to U.S. Provisional Applications No. 62 / 947,236 and 62 / 947,226, filed December 12, 2019, and U.S. Application No. 17 / 071,479, filed October 15, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to video encoding and decoding technologies, and more specifically, to methods, apparatus, and storage media for reconstructing encoded enhancement layer images. Background Technology

[0004] Using motion-compensated inter-frame prediction for video encoding and decoding has been well-established for decades. Uncompressed digital video can comprise a series of images, each with spatial dimensions, for example, 1920×1080 luminance samples and correlated chrominance samples. This series of images has a fixed or variable image rate (also informally referred to as the frame rate), such as 60 images per second or 60Hz. Uncompressed video has very high bitrate requirements. For example, a 1080p60 4:2:0 video with 8 bits per sample (1920x1080 luminance sample resolution, 60Hz frame rate) requires close to 1.5 Gbit / s of bandwidth. One hour of such video would require over 600 GB of storage space.

[0005] One goal of video encoding and decoding is to reduce redundant information in the input video signal through compression. Video compression can help reduce bandwidth or storage requirements by two or more orders of magnitude in some cases. Lossless and lossy compression, as well as combinations of both, can be used. Lossless compression refers to the technique of reconstructing an exact copy of the original signal from the compressed original signal. When using lossy compression, the reconstructed signal may not be exactly the same as the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application. Lossy compression is widely used in video. The amount of distortion tolerated depends on the application. For example, users of some consumer streaming applications may tolerate higher distortion than users of television applications. The achievable compression ratio reflects that higher allowable / tolerable distortion results in a higher compression ratio.

[0006] Video encoders and decoders can utilize several major categories of techniques, including motion compensation, transform, quantization, and entropy coding, some of which will be discussed below.

[0007] The concept of dividing an encoded video stream into multiple packets for transmission over a packet network has been used for decades. Early video codec standards and technologies were mostly optimized for robot-oriented transmission and defined streams. Packetization occurred, for example, at system-level interfaces specified in Real-Time Transport Protocol (RTP) payload formats. With the advent of Internet connections suitable for the extensive use of video over the Internet, video codec standards reflected this prominent use case through the concepts of the Video Codec Layer (VCL) and the Network Abstraction Layer (NAL). The NAL unit was introduced in H.264 in 2003 and has remained in some video codec standards and technologies since, with only minor modifications.

[0008] In many cases, a NAL unit can be viewed as the smallest entity on which a decoder can operate, without having to decode all previous NAL units of the encoded video sequence. Within this scope, NAL units, through Media Aware Network Units (MANEs) such as Selective Forward Units (SFUs) or Multipoint Control Units (MCUs), enable certain error resilience techniques and certain stream processing techniques to include stream pruning.

[0009] Figure 1 Relevant portions of the syntax diagrams for the NAL unit header according to H.264(101) and H.265(102) are shown, in both cases without their respective extensions. In both cases, the forbidden_zero_bit is a zero bit used to prevent startup code emulation in certain system-level environments. The nal_unit_type syntax element indicates the type of data carried by the NAL unit, which can be, for example, some slice type, parameter set type, or Supplemental Enhancement Information (SEI) message. The H.265 NAL unit header also includes nuh_layer_id and nuh_temporal_id_plus1, which indicate the spatial / SNR and temporal layer of the encoded picture to which the NAL unit belongs.

[0010] It can be observed that the NAL unit header only includes fixed-length codewords that are easy to parse, and it has no parsing dependency on other data in the bitstream (such as other NAL unit headers, parameter sets, etc.). Since the NAL unit header is the first octet in the NAL unit, MANE can easily extract, parse, and operate on it. In contrast, other high-level syntax elements (such as slice or tile headers) are less convenient for MANE to access because they may require maintaining the parameter set context and / or need to handle variable-length or arithmetic-encoded code points.

[0011] It can be further observed that Figure 1 The NAL unit header shown does not include information that can associate the NAL unit with an encoded image consisting of multiple NAL units (e.g., multiple tiles or slices, where at least some tiles or slices are packaged in separate NAL units).

[0012] Some transmission technologies, such as RTP (RFC 3550), MPEG system standards, and ISO file formats, may include information, usually in the form of timing information, such as presentation time (in the case of MPEG and ISO file formats) or capture time (in the case of RTP). This information can be easily accessed by the MANE and helps associate their respective transmission units with encoded pictures. However, the semantics of this information may differ depending on the transmission / storage technology and may not be directly related to the picture structure used in video encoding and decoding. Therefore, this information is at best exploratory and may not be particularly suitable for identifying whether NAL units in a NAL unit stream belong to the same encoded picture.

[0013] Conventional video syntax redundantly uses signals to notify parameters referenced by one or more slices or VCL NAL units. Summary of the Invention

[0014] To address the aforementioned issues, embodiments of this application provide a method, apparatus, and storage medium for reconstructing images with encoded enhancement layers.

[0015] According to an embodiment, a method for reconstructing an encoded enhancement layer image is provided, comprising: decoding at least one independent image header of an encoded image of a reference layer, the encoded image of the reference layer being referenced by the encoded enhancement layer image; decoding a subordinate image header of the encoded enhancement layer image; and performing image header prediction, which includes one or both of the following: predicting at least one value of at least one syntax element of the subordinate image header based on at least one value of at least one corresponding syntax element of the at least one independent image header; and inferring at least one value of a non-existent syntax element of the subordinate image header as equal to at least one value of an existing syntax element of the at least one independent image header.

[0016] According to an embodiment, an apparatus for reconstructing an encoded enhancement layer image is provided, comprising: at least one memory for storing program code; and at least one processor for accessing the at least one memory and operating according to the program code, the program code comprising: first decoding code for decoding at least one independent image header of an encoded image of a reference layer, the encoded image of the reference layer being referenced by the encoded enhancement layer image; second decoding code for decoding a dependent image header of the encoded enhancement layer image; and execution code for performing image header prediction, comprising one or both of the following: predicting at least one value of at least one syntax element of the dependent image header based on at least one value of at least one corresponding syntax element of the at least one independent image header; and inferring at least one value of a non-existent syntax element of the dependent image header as equal to at least one value of an existing syntax element of the at least one independent image header.

[0017] In one embodiment, at least one processor of the device accesses the at least one memory and operates according to program code stored in the at least one memory to perform the method for reconstructing the encoded enhancement layer image as described in the embodiment.

[0018] According to an embodiment, an apparatus for reconstructing an encoded enhancement layer image is provided, comprising: first decoding code for decoding at least one independent image header of an encoded image of a reference layer, the encoded image of the reference layer being referenced by the encoded enhancement layer image; second decoding code for decoding a subordinate image header of the encoded enhancement layer image; and execution code for performing image header prediction, which includes one or both of the following: predicting at least one value of at least one syntax element of the subordinate image header based on at least one value of at least one corresponding syntax element of the at least one independent image header; and inferring at least one value of a non-existent syntax element of the subordinate image header as equal to at least one value of an existing syntax element of the at least one independent image header.

[0019] According to an embodiment, a non-volatile computer-readable storage medium is provided for storing instructions that, when executed by one or more processors, cause the one or more processors to: decode at least one independent image header of an encoded image of a reference layer, the encoded image of the reference layer being referenced by the encoded enhancement layer image; decode a dependent image header of the encoded enhancement layer image; and perform image header prediction, which includes one or both of the following: predicting at least one value of at least one syntax element of the dependent image header based on at least one value of at least one corresponding syntax element of the at least one independent image header; and inferring that at least one value of a non-existent syntax element of the dependent image header is equal to at least one value of a present syntax element of the at least one independent image header.

[0020] In an embodiment, when the instructions stored in the non-volatile computer-readable storage medium are executed by one or more processors, the one or more processors perform the method for reconstructing the encoded enhancement layer image described in the embodiment.

[0021] In the embodiments of this application, based on at least one value of at least one corresponding syntax element of at least one independent image header, at least one value of at least one syntax element of a dependent image header is predicted, and at least one value of a non-existent syntax element of a dependent image header is inferred to be equal to at least one value of an existing syntax element of the at least one independent image header. This eliminates the need to signal parameters referenced by one or more slices or VCL NAL units, thus improving encoding and decoding efficiency. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the NAL unit header based on H.264 and H.265.

[0023] Figure 2 This is a simplified block diagram of a communication system according to an embodiment.

[0024] Figure 3 This is a schematic diagram illustrating the placement of a video encoder and a video decoder in a streaming environment according to an embodiment.

[0025] Figure 4 This is a functional block diagram of a video decoder according to an embodiment.

[0026] Figure 5 This is a functional block diagram of a video encoder according to an embodiment.

[0027] Figure 6 This is a schematic diagram of the NAL unit header according to an embodiment.

[0028] Figure 7This is a schematic diagram of an image header having independent mode and subordinate mode according to an embodiment.

[0029] Figure 8A This is a schematic diagram of a NAL unit structure in which each PU in the AU has a PH VCL unit, according to an embodiment.

[0030] Figure 8B This is a schematic diagram of a NAL unit structure having an independent picture header (PH) VCL unit and one or more subordinate PH VCL units in the picture unit (PU) of the access unit (AU) according to an embodiment.

[0031] Figure 8C This is a schematic diagram of a NAL unit structure in which each PU in the AU has one or more PH VCL units, according to an embodiment.

[0032] Figure 8D This is a flowchart of a method for notifying subordinate image headers and independent image headers using signals according to an embodiment.

[0033] Figure 8E This is a block diagram of a device that uses signals to notify subordinate image heads and independent image heads according to an embodiment.

[0034] Figure 9 This is a schematic diagram of a computer system according to an embodiment. Detailed Implementation

[0035] Figure 2 This is a simplified block diagram of a communication system (200) according to an embodiment. The communication system (200) includes at least two terminal devices (210, 220) that can communicate with each other via a network (250). For one-way data transmission, the first terminal device (210) can encode video data locally for transmission to the second terminal device (220) via the network (250). The second terminal device (220) can receive the encoded video data from the other terminal from the network (250), decode the encoded video data to recover the video data, and display the recovered video data. One-way data transmission is common in applications such as media services.

[0036] Figure 2A second pair of terminal devices (230, 240) supporting bidirectional transmission of encoded video is shown, which may occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (230) and the fourth terminal device (240) can encode video data acquired at a local location for transmission over a network (250) to the other of the third terminal device (230) and the fourth terminal device (240). Each of the third terminal device (230) and the fourth terminal device (240) can also receive encoded video data transmitted by the other of the third terminal device (230) and the fourth terminal device (240), and can decode the encoded video data and display the recovered video data on a local display device.

[0037] exist Figure 2 In this embodiment, the first terminal device (210), the second terminal device (220), the third terminal device (230), and the fourth terminal device (240) may be servers, personal computers, and smartphones, but the principles of the embodiments are not limited thereto. The embodiments are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (250) refers to any number of networks that transmit encoded video data between the first terminal device (210), the second terminal device (220), the third terminal device (230), and the fourth terminal device (240), including, for example, wired and / or wireless communication networks. The communication network (250) may exchange data in circuit-switched and / or packet-switched channels. The network may include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this application, unless explained below, the architecture and topology of the network (250) may be irrelevant to the operation of the disclosure herein.

[0038] Figure 3 The illustration shows the placement of a video encoder and video decoder in a streaming environment according to an embodiment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0039] The streaming system may include an acquisition subsystem (313) that may include a video source (301) such as a digital camera, which creates, for example, an uncompressed video sample stream (302). The video sample stream (302) is depicted as a thicker line than the encoded video bitstream to emphasize that it is a higher-volume video sample stream. The video sample stream (302) may be processed by an encoder (303) coupled to the camera (301). The encoder (303) may include hardware, software, or a combination of hardware and software to implement or enforce aspects of the disclosed subject matter as described in more detail below. The encoded video bitstream (304) is depicted as a thinner line than the video sample stream (302) to emphasize the lower-volume encoded video bitstream, which may be stored on a streaming server (305) for future use. One or more streaming clients (306, 308) can access a streaming server (305) to retrieve copies (307) and (309) of the encoded video bitstream (304). The client (306) may include a video decoder (310). The video decoder (310) decodes the incoming copy (307) of the encoded video bitstream and produces an output video sample stream (311) that can be displayed on a display (312) or another presentation device (not shown). In some streaming systems, the video bitstream (304, 307, 309) may be encoded according to certain video codec / compression standards. Examples of these standards include ITU-T Recommendation H.265. The video codec standards under development are informally referred to as Multi-Functional Video Codec (VVC). The topics disclosed can be used in the context of VVC.

[0040] Figure 4 This is a functional block diagram of the video decoder (310) according to an embodiment.

[0041] The receiver (410) may receive one or more encoded video sequences to be decoded by the video decoder (310); in the same embodiment or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequences may be received from a channel (412), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (410) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective user entities (not indicated). The receiver (410) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (415) may be coupled between the receiver (410) and the entropy decoder / parser (420) (hereinafter referred to as the "parser"). The buffer memory (415) may not be required when the receiver (410) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, or the buffer memory may be made smaller. Of course, in order to be used on packet networks such as the Internet, a buffer memory (415) may also be required, which may be relatively large and have an adaptive size.

[0042] The video decoder (310) may include a parser (420) to reconstruct symbols (421) from an entropy-coded video sequence. These symbols may include information for managing the operation of the video decoder (310), and potential information for controlling a display device such as a display (312), which is not part of the decoder but may be coupled to it. Figure 4As shown in the diagram. The control information for the display device may be a parameter set fragment (not shown) of Supplementary Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (420) may parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (420) may extract a subgroup parameter set of at least one subgroup of pixels in the subgroup of pixels for use in the video decoder based on at least one parameter corresponding to a group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), etc. The entropy decoder / parser can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter (QP) values, motion vectors, and so on.

[0043] The parser (420) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (415) to create symbols (421). The parser (420) can receive encoded data and selectively decode specific symbols (421). In addition, the parser (420) can determine whether to provide specific symbols (421) to the motion compensation prediction unit (453), the scaler / inverse transform unit (451), the intra-frame prediction unit (452), or the loop filter unit (454).

[0044] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (421) may involve multiple different units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed by the parser (420) from the encoded video sequence. For brevity, the flow of such subgroup control information between the parser (420) and the various units described below is not described.

[0045] In addition to the functional blocks already mentioned, the video decoder (310) can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and can be integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0046] The first unit can be a scaler / inverse transform unit (451). The scaler / inverse transform unit (451) can receive quantization transform coefficients as symbols (421) and control information from the parser (420), including the transform method used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (451) can output a block including sample values, which can be input into the aggregator (455).

[0047] In some cases, the output samples of the scaler / inverse transform unit (451) may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-prediction unit (452). In some cases, the intra-prediction unit (452) uses reconstructed information extracted from the (partially reconstructed) current image (456) to generate surrounding blocks of the same size and shape as the block being reconstructed. In some cases, the aggregator (455) adds the predictive information generated by the intra-prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451) based on each sample.

[0048] In other cases, the output samples of the scaler / inverse transform unit (451) may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit (453) can access the reference image memory (457) to extract samples for prediction. After motion compensation of the extracted samples according to the symbols (421), these samples can be added by the aggregator (455) to the output of the scaler / inverse transform unit (451) (referred to in this case as residual samples or residual signals) to generate output sample information. The motion compensation prediction unit (453) can obtain the prediction samples from the address in the reference image memory (457) under motion vector control, and the motion vector is available to the motion compensation prediction unit (453) in the form of the symbols (421), which, for example, include X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (457) when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0049] The output samples of the aggregator (455) can be employed by various loop filtering techniques in the loop filter unit (454). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream, and these parameters can be used as symbols (421) from the parser (420) in the loop filter unit (454). However, in other embodiments, the video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0050] The output of the loop filter unit (454) can be a sample stream, which can be output to a display device (312) and stored in a reference image memory (457) for subsequent inter-frame image prediction.

[0051] Once fully reconstructed, some of the encoded images can be used as reference images for future predictions. Once the encoded images have been fully reconstructed and the encoded images (by, for example, the parser (420)) are identified as reference images, the current image (456) can become part of the reference image memory (457), and a new current image memory can be reallocated before the reconstruction of subsequent encoded images begins.

[0052] The video decoder (310) can perform decoding operations according to a predetermined video compression technique documented, for example, in the ITU-T H.265 standard. The encoded video sequence is considered to conform to the syntax specified by the video compression technique or standard used, in the sense that it follows the syntax of the video compression technique or standard as defined in the video compression technique literature or standard, particularly in the configuration file. For compliance, the complexity of the encoded video sequence is also required to be within the limits defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured, for example, megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0053] In this embodiment, the receiver (410) may receive additional (redundant) data along with the encoded video. The additional data may be a portion of the encoded video sequence. The additional data may be used by the video decoder (310) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0054] Figure 5 This is a functional block diagram of the video encoder (303) according to an embodiment.

[0055] The video encoder (303) can receive video samples from a video source (301) (not part of the decoder), which can capture video images that will be encoded by the video encoder (303).

[0056] A video source (301) can provide a sequence of source video samples encoded by a video encoder (303) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (301) can be a storage device storing previously prepared video. In a video conferencing system, the video source (301) can be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual pictures, which are given motion when viewed in sequence. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples depending on the sampling structure, color space, etc., used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0057] According to an embodiment, the video encoder (303) can encode and compress images of a source video sequence into an encoded video sequence (543) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (550). The controller (550) controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller (550) may include rate control related parameters (e.g., image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Other functions of the controller (550) will be readily recognized by those skilled in the art, as these functions relate to the video encoder (303) optimized for a particular system design.

[0058] Some video encoders operate within an “encoding loop” readily recognizable to those skilled in the art. In a simplified description, the encoding loop may include the encoding portion of an encoder (530) (hereinafter referred to as the “source coder,” responsible for creating symbols based on the input image to be encoded and the reference image), and a (local) decoder (533) embedded within the video encoder (303). The “local” decoder (533) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates the sample data (because in the video compression techniques considered in this application, any compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream is input to a reference image memory (534). Since the decoding of the symbol stream produces bit-precise results independent of the decoder location (local or remote), the contents of the reference image memory also correspond bit-precisely between the local encoder and the remote encoder. In other words, the reference image samples “seen” by the encoder’s prediction portion are exactly the same sample values ​​that the decoder will “see” during the prediction process. The basic principles of this reference image synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) are well known to those skilled in the art.

[0059] The operation of the “local” decoder (533) can be combined with, for example, the above-described method. Figure 4 The video decoder (310) is described in detail as the same as the "remote" decoder. However, a further brief reference is provided. Figure 5 When symbols are available and the entropy encoder (545) and parser (420) are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding part of the video decoder (310) (including the channel (412), receiver (410), buffer memory (415) and parser (420)) may not be fully implemented in the local decoder (533).

[0060] It can then be observed that any decoder technique other than parsing / entropy decoding, which exists in the decoder, must also exist in the corresponding encoder in essentially the same functional form. Descriptions of encoder techniques can be simplified, as they are inverses of fully described decoder techniques. More detailed descriptions are only required in certain areas, and are provided below.

[0061] As part of the operation, the source encoder (530) may perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence designated as "reference frames." In this manner, the encoding engine (532) encodes the differences between pixel blocks of the input frame and pixel blocks of the reference frame, which may be selected as a prediction reference for the input frame.

[0062] The local video decoder (533) can decode encoded video data of a frame that can be designated as a reference frame, based on symbols created by the source encoder (530). The operation of the encoding engine (532) can be a lossy process. When the encoded video data can be decoded by the video decoder (533), Figure 5 When the source video sequence (not shown) is decoded, the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (533) replicates the decoding process, which can be performed by the video decoder on the reference frame, and allows the reconstructed reference frame to be stored in a reference image memory (534). In this way, the encoder (303) can locally store a copy of the reconstructed reference frame that shares the same content (no transmission errors) as the reconstructed reference frame to be obtained by the remote video decoder.

[0063] The predictor (535) can perform a prediction search against the encoding engine (532). That is, for a new frame to be encoded, the predictor (535) can search in the reference image memory (534) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (535) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (535), it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory (534).

[0064] The controller (550) can manage the encoding operations of the source encoder (530), including, for example, setting parameters and subgroup parameters for encoding video data.

[0065] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (545). The entropy encoder (545) can perform lossless compression on the symbols generated by the various functional units according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

[0066] The transmitter (540) can buffer the encoded video sequence created by the entropy encoder (545) in preparation for transmission via a communication channel (560), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (540) can combine the encoded video data from the source encoder (530) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0067] The controller (550) manages the operation of the video encoder (303). During encoding, the controller (550) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types.

[0068] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other frames in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will understand variations of I-pictures and their corresponding applications and characteristics.

[0069] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0070] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0071] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined based on the coding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the blocks can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction.

[0072] The video encoder (303) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (303) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0073] In this embodiment, the transmitter (540) may transmit additional data and encoded video. The source encoder (530) may include such data as a portion of an encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant images and slices, SEI messages, VUI parameter set fragments, etc.

[0074] In one embodiment, PH may contain parameters (or syntax elements) that are applied as NAL units to all slices of the encoded image. For example, the PH NAL unit may contain information about the image type, image sequence count, and encoding / decoding mode, with values ​​identical to those for all slices of the encoded image.

[0075] Figure 6 This is a schematic diagram of the NAL unit header (601) according to an embodiment.

[0076] refer to Figure 6 The NAL unit may include a body (602) and a corresponding descriptor (603), the body including information about the image type, image sequence number and encoding / decoding mode, the values ​​of which are the same as all slices of the encoded image.

[0077] In the same or another embodiment, a PU is a set of NAL units that are associated with each other according to a specified classification rule, are consecutive in the decoding order, and contain exactly one encoded image. A PU may consist of a PH NAL unit, an encoded image including one or more VCL NAL units, and zero or more non-VCL NAL units.

[0078] In the same or another embodiment, an AU may consist of zero or one AU delimiter NAL unit and one or more PUs in ascending order of nuh_layer_id. The first AU in the bitstream may begin at the first NAL unit of the bitstream. An AU may contain at most one AU delimiter NAL unit.

[0079] In the same embodiment, the first VCL NAL unit of the image may be the first VCL NAL unit following the PHNAL unit in the decoding order of the image. When the VCL NAL unit is the first VCL NAL unit of the image and one or more conditions are true, the VCL NAL unit may be the first VCL NAL unit of the AU (and therefore, the image containing the first VCL NAL unit is the first image of the AU). The one or more conditions include:

[0080] The nuh_layer_id value of the -VCL NAL unit is less than the nuh_layer_id of the previous image in the decoding order.

[0081] The value of slice_pic_order_cnt_lsb in the -VCL NAL unit is different from the PicOrderCntVal of the previous image in the decoding order.

[0082] - The PicOrderCntVal exported for the VCL NAL unit is different from the PicOrderCntVal of the previous image in the decoding order.

[0083] Figure 8A This is a schematic diagram of a NAL unit structure in which each PU in the AU has a PH VCL unit, according to an embodiment.

[0084] refer to Figure 8A In the same or another embodiment, the AU may consist of one or more non-VCL NAL units and VCL NAL units, in the following order:

[0085] 1. Access the cell delimiter (AUD)NAL cell (805) (if it exists).

[0086] 2. DPS NAL unit (810) (if present),

[0087] 3. VPS NAL unit (815) (if present),

[0088] 4. SPS NAL unit (820) (if present),

[0089] 5. PPS NAL unit (825) (if present),

[0090] 6. Prefix APS NAL unit (830) (if present),

[0091] 7. PH NAL unit (835),

[0092] 8. Prefix SEI NAL unit (840) (if present),

[0093] 9. The NAL unit whose nal_unit_type is equal to RSV_NVCL_26 (if it exists).

[0094] 10. NAL units in the range UNSPEC28 to UNSPEC29 (if they exist), where nal_unit_type is not specified.

[0095] 11. One or more VCL NAL units (845),

[0096] 12. Suffix APS NAL unit (850) (if present),

[0097] 13. Suffix SEI NAL unit (855) (if present),

[0098] 14. End of sequence (EOS) NAL unit (860) (if present)

[0099] 15. End of stream (EOB) NAL unit (865) (if present)

[0100] When the bitstream consists of multiple layers, the NAL units from the DPS NAL unit to the EOS NAL unit (except for the VPS) in the above order will be repeated as many times as the number of layers (if any).

[0101] In the same embodiment, the order of the encoded images and non-VCL NAL units within the PU or AU can follow the following constraints:

[0102] - When an AU contains an AU delimiter NAL cell, it can be the first NAL cell of that AU.

[0103] - The PH NAL unit in the PU can be located before the first VCL NAL in that PU.

[0104] - When any DPS NAL unit, VPS NAL unit, SPS NAL unit, PPS NAL unit, prefix APS NAL unit, or prefix SEI NAL unit exists in the PU, they may not be after the last VCL NAL unit of the PU.

[0105] - When any DPS NAL unit, VPS NAL unit, SPS NAL unit or PPS NAL unit exists in the PU, they can be before the PH NAL unit of the PU.

[0106] The NAL unit of -nal_unit_type equal to SUFFIX_APS_NUT, SUFFIX_SEI_NUT or FD_NUT in the PU can be not before the first VCL NAL unit of the PU.

[0107] - When an EOS NAL cell exists in a PU, it can be the last NAL cell among all NAL cells in that PU except for the EOB NAL cell.

[0108] - When an EOB NAL element exists in an AU, it can be the last NAL element in that AU.

[0109] Figure 7 This is a schematic diagram of an image header having independent mode and subordinate mode according to an embodiment.

[0110] In one embodiment, PHs are classified into two types: independent PHs and dependent PHs. An independent layer's PH can be an independent PH. A dependent layer's PH can be either an independent PH or a dependent PH. A dependent PH of layer k (where k > 0) can reference the PH of layer l, which is a direct reference layer to layer k, where l... <k。

[0111] In this embodiment, the encoded enhancement layer image can be reconstructed in the decoder. Specifically, this includes: decoding at least one independent PH of the encoded image of reference layer 1, where the encoded image of reference layer 1 can be referenced by the encoded enhancement layer image; decoding the subordinate PH of the encoded enhancement layer image; and performing PH prediction. This PH prediction includes the processing steps described in the following embodiments.

[0112] In the same embodiment, when a subordinate PH of subordinate layer k references a PH of its direct reference layer l, one or more syntax elements that are normally present in an independent PH may not be signaled in the subordinate PH of subordinate layer k. The value of a syntax element that is skipped in the subordinate PH of subordinate layer k is inferred to be equal to the value of the same syntax element in the PH of the direct reference layer l. In one example, based on at least one value of at least one corresponding syntax element of at least one independent PH, at least one value of at least one syntax element of a subordinate PH can be predicted, and at least one value of a syntax element that is absent in the subordinate PH is inferred to be equal to at least one value of a syntax element that is present in the at least one independent PH.

[0113] In the same or another embodiment, when a subordinate PH of subordinate layer k references the PH of its direct reference layer l, one or more syntax elements typically present in an independent PH can signal in the subordinate PH of subordinate layer k. The value of the syntax element in the subordinate PH of subordinate layer k is predicted from the value of the same syntax element in the PH of the direct reference layer l.

[0114] In the same embodiment, when predicting the value of a syntax element in a dependent PH from the value of a corresponding syntax element in a reference PH, the incremental value between the predicted syntax element in the dependent PH and the reference syntax element in the reference PH can be signaled in the dependent PH. In one example, the incremental value between at least one value of at least one syntax element of at least one dependent picture header and at least one value of at least one corresponding syntax element of at least one independent picture header can be signaled.

[0115] In the same embodiment, when the subordinate layer k has multiple direct reference layers, the PH of the highest reference layer that immediately precedes the subordinate layer k in the decoding order can be the reference PH.

[0116] In the same embodiment, the reference pH can be an independent pH in an independent layer.

[0117] In the same embodiment, the reference pH can be an independent pH or a subordinate pH in a subordinate layer.

[0118] In the same embodiment, the subordinate PH and its reference PH may belong to the same access unit.

[0119] In the same embodiment, the dependent pH and its reference pH may belong to different layers and have a direct dependency.

[0120] refer to Figure 7 In the same embodiment, the flag `ph_dependent_flag` in the PH can indicate whether the current PH is dependent or independent. When the value of `ph_dependent_flag` is equal to 1, the following syntax elements may not exist in the current dependent PH. The value of skipped syntax elements can be inferred from the value of the same syntax element signaled in the reference PH. When the value of `ph_dependent_flag` is equal to 0, all of the following syntax elements may exist in the current independent PH.

[0121] In the same embodiment, the value of the PPS ID in the dependent PH, ph_pic_parameter_set_id, can differ from the value of the PPS ID in the reference PH. When the PPS ID values ​​are different, the values ​​of the parameters derived from the syntax elements in the PH may differ due to the different reference PPS.

[0122] In the same or another embodiment, when a subordinate layer references multiple reference layers, the index in the subordinate PH of the subordinate layer can indicate which reference layer's PH the current subordinate PH references.

[0123] In the same or another embodiment, NAL unit type ( Figure 6 The nal_unit_type in the table can indicate that the corresponding NAL unit contains a subordinate PH.

[0124] In the same or another embodiment, the NAL cell type may indicate that the corresponding NAL cell contains an independent pH.

[0125] In one embodiment, a VCL NAL unit following a PH NAL unit references a previous PH NAL unit according to the decoding order of the image. When there is only one PH NAL unit in the PU, any VCL NAL unit references the PH NAL unit preceding all VCL NAL units. When there are multiple PH NAL units in the PU, any VCL NAL unit references the PH NAL unit immediately preceding the current VCL NAL unit.

[0126] Figure 8B This is a schematic diagram of a NAL unit structure having one PH VCL unit and one or more dependent PH VCL units in the PU of AU according to an embodiment.

[0127] refer to Figure 8B In the same or another embodiment, the access unit may consist of one or more non-VCL NAL units and VCL NAL units, in the following order:

[0128] 1. AUD NAL unit (805) (if present),

[0129] 2. DPS NAL unit (810) (if present),

[0130] 3. VPS NAL unit (815) (if present),

[0131] 4. SPS NAL unit (820) (if present),

[0132] 5. PPS NAL unit (825) (if present),

[0133] 6. Prefix APS NAL unit (830) (if present),

[0134] 7. Prefix SEINAL unit (840) (if present),

[0135] 8. NAL units whose nal_unit_type equals RSV_NVCL_26 (if they exist).

[0136] 9. NAL units in the range UNSPEC28 to UNSPEC29 (if they exist), where nal_unit_type is not specified.

[0137] 10. PH NAL unit (835),

[0138] Option A: Independent PH NAL unit (835a)

[0139] Option B: Subordinate PH NAL unit (835b)

[0140] 11. One or more VCL NAL units (845),

[0141] 12. Suffix APS NAL unit (850) (if present),

[0142] 13. Suffix SEINAL unit (855) (if present),

[0143] 14. EOS NAL unit (860) (if present)

[0144] 15. EOB NAL unit (865) (if present)

[0145] When the bitstream can consist of multiple layers, the NAL units from the DPS NAL unit to the EOS NAL unit (except for the VPS) in the above order will be repeated as many times as the number of layers (if any). For each layer, each PU can have one PH VCL NAL unit. Depending on the value of the NAL unit type or a flag in the PH, the PH VCL NAL unit contains either an independent PH NAL unit or a dependent PH NAL unit. In one example, based on a flag indicating at least one independent image header, it can be determined that the encoded image includes that at least one independent image header; based on a flag indicating a dependent image header, it can be determined that the encoded enhancement layer image includes that dependent image header. In another example, based on the unit type of the encoded image, it can be determined that the encoded image includes at least one independent image header; based on the unit type of the encoded enhancement layer image, it can be determined that the encoded enhancement layer image includes a dependent image header.

[0146] Figure 8C This is a schematic diagram of a NAL unit structure in which each PU in the AU has one or more PH VCL units, according to an embodiment.

[0147] refer to Figure 8CFor each layer, the grouped PH NAL units (835) and VCL NAL units (845) can be repeated.

[0148] In the same or another embodiment, one or more NAL units containing the same pH may appear sequentially before one or more VCL NAL units.

[0149] In the same embodiment, the order of the encoded images and non-VCL NAL units within the PU or AU can follow the following constraints:

[0150] - When an AU contains an AU delimiter NAL cell, it can be the first NAL cell of that AU.

[0151] -The PH NAL unit in the PU can precede at least one VCL NAL in the PU.

[0152] - When any DPS NAL unit, VPS NAL unit, SPS NAL unit, PPS NAL unit, prefix APS NAL unit, or prefix SEINAL unit exists in the PU, they may not be after the last VCL NAL unit of the PU.

[0153] - When any DPS NAL unit, VPS NAL unit, SPS NAL unit or PPS NAL unit exists in the PU, they can be before the PH NAL unit of the PU.

[0154] The NAL unit of -nal_unit_type equal to SUFFIX_APS_NUT, SUFFIX_SEI_NUT or FD_NUT in the PU can be not before the first VCL NAL unit of the PU.

[0155] - When an EOS NAL cell exists in a PU, it can be the last NAL cell among all NAL cells in that PU except for the EOB NAL cell.

[0156] - When an EOB NAL element exists in an AU, it can be the last NAL element in that AU.

[0157] In the same or another embodiment, a PPS identifier (PPS_ID) specifying a reference PPS can be signaled in the PH. When a PPS_ID exists in the PH, any slice or any VCL NAL cell in the same PU or AU located after the PH can refer to the PPS_ID as the PPS signaled in the PH.

[0158] In the same embodiment, when a PPS_ID is signaled in the corresponding PH, the PPS_ID may not exist in the slice header, and the PH precedes the slices in the same PU or AU. In this case, all slices in the same PU can reference the same PPS without changing the syntax element values ​​in the PPS.

[0159] Figure 8D This is a flowchart of a method (870) for notifying subordinate and independent image heads by signals according to an embodiment. In some embodiments, Figure 8D One or more processing blocks can be executed by the decoder (310). In some implementations, Figure 8D One or more processing blocks may be performed by another device or a group of devices that are separate from or include the decoder (310), such as an encoder (303).

[0160] join Figure 8D In the first step (871), the method (870) includes decoding at least one independent image header of the encoded image of the reference layer, the encoded image of the reference layer being referenced by the encoded enhancement layer image.

[0161] In the second step (872), the method (870) includes decoding the dependent image header of the encoded enhancement layer image.

[0162] In the third step (873), the method (870) includes performing image header prediction, which includes one or both of the following: predicting at least one value of at least one syntax element of the dependent image header based on at least one value of at least one corresponding syntax element of the at least one independent image header; and inferring at least one value of a non-existent syntax element of the dependent image header as equal to at least one value of a present syntax element of the at least one independent image header.

[0163] The method (870) may further include signaling an incremental value between at least one value of at least one syntax element of the predicted dependent image header and at least one value of at least one corresponding syntax element of the at least one independent image header.

[0164] The method (870) may further include determining, based on a flag of the at least one independent image header, that the encoded image includes the at least one independent image header; and determining, based on a flag of the dependent image header, that the encoded enhancement layer image includes the dependent image header.

[0165] The method (870) may further include determining, based on the unit type of the encoded image, that the encoded image includes the at least one independent image header; and determining, based on the unit type of the encoded enhancement layer image, that the encoded enhancement layer image includes the dependent image header.

[0166] The at least one independent image header is located directly before the video codec layer in the encoded image.

[0167] The subordinate image header is located directly before the video codec layer in the encoded enhancement layer image.

[0168] Each of the encoded image and the encoded enhancement layer image includes multiple video codec layers and multiple image headers directly preceding the multiple video codec layers.

[0169] Although Figure 8D Example steps of method (870) are shown, but in some implementations, method (870) may include more than Figure 8D The steps shown may be more, fewer, different, or arranged differently. Alternatively, two or more steps of the method (870) may be performed in parallel.

[0170] Figure 8E This is a block diagram of a device (880) that signals subordinate image heads and independent image heads according to an embodiment.

[0171] See Figure 8E The device (880) includes a first decoding code (881), a second decoding code (882), and an execution code (883).

[0172] The first decoding code (881) is used to enable the at least one processor to decode at least one independent image header of the encoded image of the reference layer, the encoded image of the reference layer being referenced by the encoded enhancement layer image.

[0173] The second decoding code (882) is used to enable the at least one processor to decode the subordinate image header of the encoded enhancement layer image.

[0174] The execution code (883) is used to cause the at least one processor to perform image header prediction, which includes one or both of the following: predicting at least one value of at least one syntax element of the dependent image header based on at least one value of at least one corresponding syntax element of the at least one independent image header; and inferring at least one value of a non-existent syntax element of the dependent image header as equal to at least one value of a present syntax element of the at least one independent image header.

[0175] The device (880) may also include signaling codes for causing the at least one processor to signal the incremental value between at least one value of at least one syntax element of the predicted dependent picture header and at least one value of at least one corresponding syntax element of the at least one independent picture header.

[0176] The device (880) may further include first determining code for enabling the at least one processor to determine, based on a flag of the at least one independent image header, that the encoded image includes the at least one independent image header; and to determine, based on a flag of the dependent image header, that the encoded enhancement layer image includes the dependent image header.

[0177] The device (880) may further include second determining code for enabling the at least one processor to determine, based on the unit type of the encoded image, that the encoded image includes the at least one independent image header; and to determine, based on the unit type of the encoded enhancement layer image, that the encoded enhancement layer image includes the dependent image header.

[0178] The at least one independent image header is located directly before the video codec layer in the encoded image.

[0179] The subordinate image header is located directly before the video codec layer in the encoded enhancement layer image.

[0180] Each of the encoded image and the encoded enhancement layer image includes multiple video codec layers and multiple image headers directly preceding the multiple video codec layers.

[0181] The above-mentioned technology can be implemented as computer software through computer-readable instructions and physically stored in one or more computer-readable media. Figure 9 A schematic diagram of a computer system (900) according to an embodiment is shown.

[0182] The computer software can be encoded using any suitable machine code or computer language, and code including instructions can be created through mechanisms such as assembly, compilation, and linking. These instructions can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed through decoding, microcode, etc.

[0183] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0184] Figure 9The components shown for the computer system (900) are exemplary in nature and are not intended to limit the scope or functionality of the computer software used to implement the embodiments of this application. Nor should the configuration of the components be construed as having any dependency or requirement on any component or combination thereof shown in the embodiments of the computer system (900).

[0185] The computer system (900) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through tactile input (e.g., keyboard input, swiping, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-machine interface device may also be used to capture certain media, which need not be directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from still cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0186] The human-machine interface input device may include one or more of the following (only one is shown): keyboard (901), mouse (902), touchpad (903), touch screen (910), data glove, joystick (905), microphone (906), scanner (907), camera (908).

[0187] The computer system (900) may also include certain human-machine interface (HMI) output devices. Such HMI output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. These HMI output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (910), data gloves, or joystick (905), but may also include tactile feedback devices not used as input devices), audio output devices (e.g., speakers (909), headphones (not shown)), visual output devices (e.g., screens (910) including cathode ray tube (CRT) screens, liquid crystal (LCD) screens, plasma screens, organic light-emitting diode (OLED) screens, each with or without touchscreen input functionality, each with or without tactile feedback functionality—some of which may output two-dimensional or higher-dimensional visual outputs through means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown). A graphics adapter (950) generates images and outputs the images to the touchscreen (910).

[0188] The computer system (900) may also include human-accessible storage devices and related media, such as optical media including high-density read-only / rewritable optical discs (CD / DVD ROM / RW) (920) or similar media (921), thumb drives (922), removable hard disk drives or solid-state drives (923), conventional magnetic media such as magnetic tapes and floppy disks (not shown), dedicated devices based on ROM / ASIC / PLD such as security software protectors (not shown), and so on.

[0189] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0190] The computer system (900) may also include an interface to one or more communication networks (955). For example, the network (955) may be wireless, wired, or optical. The network (955) may also be a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a vehicular network, an industrial network, a real-time network, a latency-tolerant network, and so on. The network (955) also includes Ethernet, wireless LANs, cellular networks (Global System for Mobile Communications (GSM), 3G, 4G, 5G, LTE, etc.), cable or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicular and industrial networks (including CANbus), and so on. Some networks (955) typically require external network interface adapters for connection to certain general-purpose data ports or peripheral buses (949) (e.g., the Universal Serial Bus (USB) port of a computer system (900); other systems are typically integrated into the core of the computer system (900) via a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface (954) integrated into a smartphone computer system). By using any of these networks (955), the computer system (900) can communicate with other entities. This communication can be unidirectional, for receiving only (e.g., wireless television), unidirectional, for sending only (e.g., a CAN bus to certain CAN bus devices), or bidirectional, such as via a local area or wide area digital network to other computer systems. Each of the aforementioned networks (955) and network interfaces (954) may use certain protocols and protocol stacks.

[0191] The aforementioned human-computer interface device, human-accessible storage device, and network interface (954) can be connected to the core (940) of the computer system (900).

[0192] The core (940) may include one or more central processing units (CPU) (941), graphics processing units (GPUs) (942), dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) (943), task-specific hardware accelerators (944), etc. These devices, as well as read-only memory (ROM) (945), random access memory (RAM) (946), internal mass storage (e.g., internal non-user-accessible hard disk drives, solid-state drives (SSDs), etc.) (947), etc., can be connected via a system bus (948). In some computer systems, the system bus (948) may be accessed in the form of one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus (948) or connected via a peripheral bus (949). Peripheral bus architectures include peripheral bus interconnect (PCI), universal serial bus (USB), etc.

[0193] The CPU (941), GPU (942), FPGA (943), and accelerator (944) can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM (945) or RAM (946). Transient data can also be stored in RAM (946), while permanent data can be stored, for example, in internal mass storage (947). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (941), GPUs (942), mass storage (947), ROM (945), RAM (946), etc.

[0194] The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be specifically designed and constructed for the purposes of this embodiment, or they may be media and code well-known and available to those skilled in the art of computer software.

[0195] By way of example and not limitation, a computer system having an architecture (900), particularly a core (940), can provide functionality as a processor (including a CPU, GPU, FPGA, accelerator, etc.) to execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with the aforementioned user-accessible mass storage, as well as specific memory of the core (940) that is non-volatile, such as internal mass storage (947) or ROM (945). Software implementing various embodiments of this application can be stored in such a device and executed by the core (940). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the core (940), particularly the processor therein (including a CPU, GPU, FPGA, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (946) and modifying such data structures according to software-defined processes. Alternatively or as an alternative, the computer system may provide logic hardwired or otherwise incorporated into circuitry (e.g., an accelerator (944)) that may replace or operate with the software to perform the specific process or a specific portion of the specific process described herein. References to software may include logic, and vice versa, where appropriate. References to computer-readable media may include, where appropriate, circuitry storing the execution of software (such as an integrated circuit (IC)), circuitry containing the execution logic, or both. This application includes any suitable combination of hardware and software.

[0196] While this application has described several exemplary embodiments, various modifications, arrangements, and equivalent substitutions of the embodiments are all within the scope of this application. Therefore, it should be understood that those skilled in the art can design various systems and methods that, although not explicitly shown or described herein, embody the principles of this application and are thus within its spirit and scope.

Claims

1. A video decoding method, characterized in that, The method includes: Acquire the encoded bitstream including the access unit (AU); and Decoding at least one image from the encoded bitstream based on the AU, wherein the AU includes at least one image unit PU, the image unit comprising: Image Header Network Abstraction Layer (NAL) unit; and At least one video coding layer (VCL) NAL unit, the at least one VCL NAL unit being located after the picture header NAL unit within the AU.

2. The method according to claim 1, characterized in that, The AU delimiter NAL unit is the first NAL unit within the AU.

3. The method according to claim 1, characterized in that, The at least one PU includes at least one set of parameters, which is located before the last VCL NAL cell in the at least one VCL NAL cell within the AU.

4. The method according to claim 3, characterized in that, The at least one parameter set is located before the NAL unit of the image header of the PU within the AU.

5. The method according to claim 4, characterized in that, The at least one parameter set includes at least one of the following: decoder parameter set (DPS) NAL unit, video parameter set (VPS) NAL unit, sequence parameter set (IPS) NAL unit, picture parameter set (PPS) NAL unit, and prefix adaptive parameter set (APS) NAL unit.

6. The method according to claim 5, characterized in that, The at least one parameter set includes the DPS NAL unit, the VPS NAL unit, the IPS NAL unit, the PPS NAL unit, and the prefix APS NAL unit.

7. The method according to claim 3, characterized in that, The PU also includes at least one additional parameter set, which is located after the first VCLNAL cell in the at least one VCL NAL cell of the PU within the AU.

8. The method according to claim 7, characterized in that, The at least one additional parameter set includes the Suffix Adaptive Parameter Set (APS) NAL unit.

9. The method according to claim 8, characterized in that, The PU also includes a suffix supplemental enhancement information (SEI) NAL unit and a padding data NAL unit following the first VCL NAL unit in the at least one VCL NAL unit of the PU located within the AU.

10. The method according to claim 1, characterized in that, The PU from the at least one PU further includes an end-of-sequence (EOS) NAL unit, the EOS NAL unit being the last NAL unit within the PU, and The AU further includes a Bit Stream End (EOB) NAL unit, which is the last NAL unit within the AU.

11. A video encoding method applied to an encoder including a local decoder, characterized in that, The method includes: Acquire the encoded bitstream including the access unit (AU); and Decoding at least one image from the encoded bitstream based on the AU, wherein the AU includes at least one image unit PU, the image unit comprising: Image Header Network Abstraction Layer (NAL) unit; and At least one video coding layer (VCL) NAL unit, the at least one VCL NAL unit being located after the picture header NAL unit within the AU.

12. A video decoding device, characterized in that, The device includes: At least one memory is configured to store computer program code; and At least one processor is configured to access the at least one memory and operate according to the computer program code, the computer program code comprising: The first code is configured to cause the at least one processor to acquire an encoded bit stream including an access unit (AU); The second code is configured to cause the at least one processor to decode at least one image of the encoded bitstream based on the AU, wherein the AU includes at least one image unit PU, the image unit comprising: Image Header Network Abstraction Layer (NAL) unit; and At least one video coding layer (VCL) NAL unit, the at least one VCL NAL unit being located after the picture header NAL unit within the AU.

13. The apparatus according to claim 12, characterized in that, The AU delimiter NAL unit is the first NAL unit within the AU.

14. The apparatus according to claim 12, characterized in that, The at least one PU includes at least one set of parameters, which is located before the last VCLNAL cell in the at least one VCL NAL cell within the AU.

15. The apparatus according to claim 14, characterized in that, The at least one parameter set is located before the NAL unit of the image header of the PU within the AU.

16. The apparatus according to claim 15, characterized in that, The at least one parameter set includes at least one of the following: decoder parameter set (DPS) NAL unit, video parameter set (VPS) NAL unit, sequence parameter set (IPS) NAL unit, picture parameter set (PPS) NAL unit, and prefix adaptive parameter set (APS) NAL unit.

17. The apparatus according to claim 16, characterized in that, The at least one parameter set includes the DPS NAL unit, the VPS NAL unit, the IPS NAL unit, the PPS NAL unit, and the prefix APS NAL unit.

18. The apparatus according to claim 14, characterized in that, The PU also includes at least one additional parameter set, which is located after the first VCL NAL cell in the at least one VCL NAL cell of the PU within the AU.

19. The apparatus according to claim 18, characterized in that, The at least one additional parameter set includes the Suffix Adaptive Parameter Set (APS) NAL unit.

20. The apparatus according to claim 19, characterized in that, The PU also includes a suffix supplemental enhancement information (SEI) NAL unit and a padding data NAL unit following the first VCL NAL unit in the at least one VCL NAL unit of the PU located within the AU.

21. The apparatus according to claim 12, characterized in that, The PU from the at least one PU further includes an end-of-sequence (EOS) NAL unit, the EOS NAL unit being the last NAL unit within the PU, and The AU further includes a Bit Stream End (EOB) NAL unit, which is the last NAL unit within the AU.

22. A non-volatile computer-readable storage medium for storing instructions, characterized in that, The instruction causes at least one processor to: Acquire the encoded bit stream including access units (AU); as well as Decoding at least one image from the encoded bitstream based on the AU, wherein the AU includes at least one image unit PU, the image unit comprising: Image Header Network Abstraction Layer (NAL) unit; and At least one video coding layer (VCL) NAL unit, the at least one VCL NAL unit being located after the picture header NAL unit within the AU.

23. A method for storing video bitstreams, characterized in that, The video bitstream is generated by performing the video encoding method of claim 11; and the video bitstream is stored.

24. A method for transmitting video bitstreams, characterized in that, The video bitstream is generated by performing the video encoding method of claim 11; and the video bitstream is transmitted.

25. A non-volatile computer-readable storage medium, characterized in that, Used to store instructions and video bitstreams, wherein when the instructions are executed by one or more processors, the one or more processors, when executed, implement the steps of the video encoding method of claim 11 to generate the video bitstream.

Images