Video decoding method, device and readable storage medium

Abstract

The embodiments of the present application provide a video decoding method, an apparatus, and a non-volatile computer-readable storage medium. The video decoding method includes: receiving encoded information of a block encoded in a palette-based coding mode. The block is the beginning of a predefined coding area including at least one coding tree unit (CTU). The method further includes: determining the palette of the block independently of the palette information of at least one previously decoded block of the palette-based coding mode, and decoding the pixels of the block based on the encoded information of the block and the determined palette of the block.

Description

[0001] This application claims priority to U.S. Provisional Application No. 62 / 883,900, filed on August 7, 2019, entitled “PALETTEPREDICTOR LIST RESET” and U.S. Formal Application No. 169 / 40,842, filed on July 28, 2020, entitled “METHOD AND APPARATUS FOR VIDEO CODING”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to video encoding and decoding, and in particular to a video decoding method, device and readable storage medium. Background Art

[0003] The background description provided here is intended to present the context of the present application in general. To the extent that the work of the currently named inventors is described in this background section, as well as aspects of the description that may not be prior art at the time of filing this application, it is neither explicitly nor implicitly admitted to be prior art of the present application.

[0004] Video encoding and decoding can be performed using inter-frame prediction with motion compensation. An uncompressed digital video may include a series of pictures, each picture having a certain spatial dimension, such as 1920×1080 luminance samples and associated chrominance samples. The series of pictures can have a fixed or variable picture rate (also informally called a frame rate), for example, 60 pictures per second or 60 Hertz (Hz). Uncompressed video has specific requirements for bit rate. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at 60Hz frame rate) requires a bandwidth of nearly 1.5Gbit / s. Such a video requires more than 600GB of storage space for one hour.

[0005] One purpose of video encoding and decoding can be to reduce redundancy in the input video signal through compression. Compression can help reduce the above-mentioned bandwidth and / or storage space requirements, in some cases, by two or more orders of magnitude. Both lossless compression and lossy compression and combinations thereof can be used for video encoding and decoding. Lossless compression refers to a technique that can reconstruct an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be exactly the same as the original signal, but the distortion between the original signal and the reconstructed signal is small enough that the reconstructed signal can be used for the intended application. Lossy compression is widely used in video. The amount of distortion allowed by lossy compression depends on the application; for example, users of certain consumer streaming applications can tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect that higher allowable / tolerable distortion can produce higher compression ratios.

[0006] Video encoders and decoders may use several broad categories of techniques including, for example: motion compensation, transforms, quantization, and entropy coding.

[0007] Video codec techniques may include intra-coding techniques. In intra-coding techniques, sample values ​​are represented without reference to samples or other data in previously reconstructed reference pictures. In some video codec techniques, a picture is spatially divided into blocks of samples. When all sample blocks are encoded using intra-mode, the picture may be an intra-picture. Intra-pictures and their derivatives, such as pictures refreshed by independent decoders, may be used to reset the state of a decoder and may therefore be used as the first picture in an encoded video stream and video session, or as a still image. Samples of an intra-block may be transformed, and the transform coefficients may be quantized prior to entropy coding. Intra-prediction may be a technique for minimizing sample values ​​in a pre-transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficient, the fewer the number of bits required to represent the entropy coded block given a quantization step size.

[0008] Traditional intra-frame coding techniques, such as the known MPEG-2 coding techniques, do not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to use, for example, neighboring sample data and / or metadata obtained during the encoding / decoding of spatially adjacent and preceding data blocks in decoding order. Such techniques are therefore referred to as "intra-frame prediction" techniques. It should be noted that, at least in some cases, intra-frame prediction uses only reference data in the picture currently being reconstructed, and not reference data in reference pictures.

[0009] There are many different forms of intra-frame prediction. When more than one such technique is available for a given video coding technique, an intra-frame prediction mode can be used to encode the technique used. In some cases, some modes have sub-modes and / or parameters, which can be encoded separately or can be included in the mode codeword. Which codeword is used for a given mode / sub-mode / parameter combination affects the coding efficiency gain through intra-frame prediction, as well as the entropy coding technique used to translate the codeword into the bitstream.

[0010] A certain mode of intra prediction was introduced in the H.264 standard, improved in the H.265 standard, and further improved in newer coding techniques such as the Joint Exploitation Model (JEM), Versatile Video Coding (VVC), Benchmark Sets (BMS), etc. A prediction block can be formed using sample values ​​of neighboring samples that belong to samples that are already available. The sample values ​​of the neighboring samples are copied to the prediction block in one direction. The reference to the direction used can be encoded into the bitstream, or the reference to the direction used can itself be predicted.

[0011] refer to Figure 1 As shown, Figure 1 The lower right corner of the diagram depicts a subset of the 33 possible predictor directions (corresponding to the 33 angular modes of the 35 intra modes) that can be known from the H.265 standard, which has 9 predictor directions. The convergence point (101) of the arrows represents the sample being predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted based on at least one sample at the upper right that is at an angle of 45 degrees to the horizontal axis. Similarly, arrow (103) indicates that sample (101) is predicted based on at least one sample at the lower left of sample (101) that is at an angle of 22.5 degrees to the horizontal axis.

[0012] Refer to Figure 1 As shown, Figure 1 Depicted at the upper left of is a square block (104) having 4×4 samples (indicated by the bold dashed line). The square block (104) includes 16 samples, each of which is labeled with an "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (counting from top to bottom) and the first sample in the X dimension (counting from left to right). Similarly, sample S44 is the fourth sample in both the Y dimension and the X dimension in the block (104). Because the size of the block (104) is 4×4 samples, S44 is at its lower right corner. Figure 1Reference samples are further shown, following a similar numbering scheme. Reference samples are labeled with R, their Y position (e.g., row index) and X position (e.g., column index) relative to the block (104). In the H.264 standard and the H.265 standard, prediction samples are adjacent to the block being reconstructed; therefore, there is no need to use negative values.

[0013] Intra-picture prediction can be performed by appropriately copying reference sample values ​​of adjacent samples according to the prediction direction indicated by the signal. For example, assume that the encoded video bitstream contains signaling, and for the block, the signaling indicates a prediction direction consistent with arrow (102), that is, the samples in the block are predicted according to at least one prediction sample at an angle of 45 degrees to the horizontal direction in the upper right direction. In this case, samples S41, S32, S23, and S14 are predicted according to the same reference sample R05. Sample S44 is predicted according to reference sample R08.

[0014] In some cases, the values ​​of multiple reference samples may be combined together, for example, by interpolation, to calculate the reference sample value; in particular, when the direction is not divisible by 45 degrees.

[0015] As video coding technology develops, the number of possible directions is also increasing. In the H.264 standard (2003), 9 different directions can be represented. In the H.265 standard (2013), it increased to 33 directions. At the time of the invention of this application, JEM / VVC / BMS can support up to 65 directions. Some experiments have been conducted to identify the most likely directions, and some entropy coding techniques use a small number of bits to represent these most likely directions, accepting a certain cost for less likely directions. In addition, sometimes these directions themselves can be predicted based on the adjacent directions used by adjacent decoded blocks.

[0016] Figure 2 A schematic diagram (201) of 65 intra prediction directions depicted according to JEM is shown to illustrate that the number of prediction directions increases over time.

[0017] The mapping of the intra-prediction direction bits used to represent directions in the encoded video bitstream may vary from one video coding technique to another; for example, it may range from a simple direct mapping of the prediction direction of the intra-prediction mode to a codeword, to complex adaptive schemes involving most likely modes and similar techniques. Those skilled in the art are familiar with these techniques. However, in all of these cases, some directions are statistically less likely to appear in the video content than other directions. Since the goal of video compression is to reduce redundancy, in better performing video coding techniques, these less likely directions are represented with more bits than more likely directions.

[0018] In addition to the inter-frame prediction mode and the intra-frame prediction mode, a palette-based coding mode can also be used. The existing palette-based coding mode cannot truly achieve parallel processing, and the video encoding and decoding efficiency needs to be improved. Summary of the invention

[0019] Embodiments of the present application provide methods and devices for video encoding / decoding. In some examples, a video decoding device includes a receiving circuit and a processing circuit. The processing circuit receives encoded information of a block encoded in a palette-based encoding mode. The block is the beginning of a predefined coding area including at least one coding tree unit (CTU). The processing circuit further determines the palette of the block independently of the palette information of at least one previously decoded block of the palette-based encoding mode, and decodes the pixels of the block based on the encoded information of the block and the determined palette of the block.

[0020] In some embodiments, when the block is the beginning of the coding region including at least one CTU, the processing circuit resets a palette prediction sublist to a zero state, the palette prediction sublist storing palette information of the at least one previously decoded block.

[0021] In one embodiment, the block is the beginning of a CTU. In another example, the block is the beginning of a CTU row. In another example, the block is the beginning of a tile. In another example, the block is the beginning of a slice.

[0022] In one embodiment, the block is the beginning of a CTU row, and when wavefront parallel processing (WPP) is enabled, the processing circuit resets the palette prediction sublist to a zero state, wherein the palette prediction sublist stores the palette information of the at least one previously decoded block.

[0023] In another embodiment, the block is the beginning of a CTU row, and when WPP is enabled, the processing circuitry resets the palette predictor list according to the palette predictor initializer.

[0024] In another embodiment, the block is the beginning of a CTU and the processing circuitry resets the palette predictor list according to the palette predictor initializer when WPP is enabled.

[0025] In another embodiment, the block is the beginning of a first predefined coding region, and the processing circuit determines the palette of the block independently of the palette information of the block in a second predefined coding region, wherein the second predefined coding region is decoded in parallel with the first predefined coding region.

[0026] The present application provides a video decoding method, including:

[0027] Receiving encoded information of a block encoded in a palette-based coding mode, the block being the beginning of a predefined coding region including at least one coding tree unit (CTU);

[0028] determining a palette for the block independently of palette information of at least one previously decoded block of the palette based encoding mode; and

[0029] Pixels of the block are decoded based on the encoded information of the block and the determined palette of the block.

[0030] The embodiment of the present application provides a video decoding device, including:

[0031] A receiving module, configured to receive encoded information of a block encoded in a palette-based encoding mode, the block being the beginning of a predefined coding region including at least one coding tree unit CTU;

[0032] a palette determination module for determining a palette of the block independently of palette information of at least one previously decoded block of the palette-based encoding mode; and

[0033] A decoding module is used to decode pixels of the block based on the encoded information of the block and the determined palette of the block.

[0034] Each embodiment of the present application further provides a non-volatile computer-readable storage medium storing instructions, wherein when the instructions are executed by a computer for video decoding, the computer executes a method for video decoding.

[0035] The video decoding method, device and non-volatile computer-readable storage medium of the embodiments of the present application can realize true parallel processing in the palette-based encoding mode, thereby improving the decoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Other features, nature and various advantages of the subject matter of the present application will become more apparent from the following detailed description and accompanying drawings, in which:

[0037] Figure 1 is a schematic diagram of an exemplary subset of intra prediction modes.

[0038] Figure 2 is a diagram of exemplary intra prediction directions.

[0039] Figure 3 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment.

[0040] Figure 4 is a schematic diagram of a simplified block diagram of a communication system according to another embodiment.

[0041] Figure 5 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment.

[0042] Figure 6 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment.

[0043] Figure 7 A block diagram of an encoder according to another embodiment is shown.

[0044] Figure 8 A block diagram of a decoder according to another embodiment is shown.

[0045] Fig. 9 An example of segmentation in a video according to some embodiments of the present application is shown.

[0046] Fig.10 Another example of division in a video according to some embodiments of the present application is shown.

[0047] Fig.11 Another example of division in a video according to some embodiments of the present application is shown.

[0048] Fig.12 A flowchart outlining an example of a process according to some embodiments of the present application is shown.

[0049] Fig.13 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION

[0050] Figure 3 3 is a simplified block diagram of a communication system (300) according to an embodiment disclosed in the present application. The communication system (300) includes a plurality of terminal devices, which can communicate with each other through, for example, a network (350). For example, the communication system (300) includes a first terminal device (310) and a second terminal device (320) interconnected through the network (350). Figure 3In an embodiment of the present invention, the first terminal device (310) and the second terminal device (320) perform unidirectional data transmission. For example, the first terminal device (310) may encode video data (e.g., a video picture stream collected by the terminal device (310)) for transmission to the second terminal device (320) via the network (350). The encoded video data is transmitted in the form of at least one encoded video code stream. The second terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to restore the video data, and display the video picture based on the restored video data. Unidirectional data transmission is more common in applications such as media services.

[0051] In another embodiment, the communication system (300) includes a third terminal device (330) and a fourth terminal device (340) that perform bidirectional transmission of encoded video data, which can occur, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (330) and the fourth terminal device (340) can encode video data (e.g., a video picture stream collected by the terminal device) to transmit to the other terminal device of the third terminal device (330) and the fourth terminal device (340) through a network (350). Each of the third terminal device (330) and the fourth terminal device (340) can also receive the encoded video data transmitted by the other terminal device of the third terminal device (330) and the fourth terminal device (340), and can decode the encoded video data to restore the video data, and can display the video picture on an accessible display device based on the restored video data.

[0052] exist Figure 3 In the embodiment of the present invention, the first terminal device (310), the second terminal device (320), the third terminal device (330) and the fourth terminal device (340) may be servers, personal computers and smart phones, but the principles disclosed in the present application may not be limited thereto. The embodiments disclosed in the present application are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. The network (350) represents any number of networks that transmit encoded video data between the first terminal device (310), the second terminal device (320), the third terminal device (330) and the fourth terminal device (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) may exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purpose of the present application, unless explained below, the architecture and topology of the network (350) may be irrelevant to the operation disclosed in the present application.

[0053] As an example, Figure 4The video encoder and video decoder are shown in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0054] The streaming system may include an acquisition subsystem (413), which may include a video source (401) such as a digital camera, which creates an uncompressed video picture stream (402). In an embodiment, the video picture stream (402) includes samples taken by a digital camera. Compared to the encoded video data (404) (or the encoded video bitstream), the video picture stream (402) is depicted as a thick line to emphasize the high data volume of the video picture stream, and the video picture stream (402) can be processed by an electronic device (420), which includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video picture stream (402), the encoded video data (404) (or the encoded video bitstream (404)) is depicted as a thin line to emphasize the lower amount of data of the encoded video data (404) (or the encoded video bitstream (404)), which can be stored on the streaming server (405) for future use. At least one streaming client subsystem, such as Figure 4 The client subsystem (406) and the client subsystem (408) in the video transmission system can access the streaming server (405) to retrieve the copy (407) and the copy (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in the electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and generates an output video picture stream (411) that can be presented on a display (412) (e.g., a display screen) or another presentation device (not depicted). In some streaming transmission systems, the encoded video data (404), the video data (407), and the video data (409) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T H.265. In an embodiment, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and the present application can be used in the context of the VVC standard.

[0055] It should be noted that the electronic device (420) and the electronic device (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0056] Figure 5 is a block diagram of a video decoder (510) according to an embodiment disclosed in the present application. The video decoder (510) may be provided in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used to replace Figure 4 A video decoder (410) of an embodiment.

[0057] The receiver (531) may receive at least one encoded video sequence to be decoded by the video decoder (510); in the same or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data along with other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective consuming entities (not shown). The receiver (531) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, the buffer memory (515) may be provided external to the video decoder (510) (not shown). In other cases, a buffer memory (not shown) is provided outside the video decoder (510) to prevent network jitter, for example, and another buffer memory (515) may be configured inside the video decoder (510) to handle broadcast timing, for example. When the receiver (531) receives data from a storage / forwarding device with sufficient bandwidth and controllability or from an isochronous network, it may not be necessary to configure the buffer memory (515), or the buffer memory may be made smaller. Of course, in order to use on a service packet network such as the Internet, a buffer memory (515) may also be required, and the buffer memory may be relatively large and have an adaptive size, and may be at least partially implemented in an operating system or a similar element (not shown) outside the video decoder (510).

[0058] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The types of symbols include information used to manage the operation of the video decoder (510) and potential information used to control a display device such as a display device (512) (e.g., a display screen) that is not part of the electronic device (530) but can be coupled to the electronic device (530), such as Figure 5 As shown in . The control information for the display device may be a parameter set fragment (not indicated) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may parse / entropy decode the received coded video sequence. The encoding of the coded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (520) may extract a subgroup parameter set of at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), and the like. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0059] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515), thereby creating symbols (521).

[0060] Depending on the type of coded video picture or part of coded video picture (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of symbol (521) may involve multiple different units. Which units are involved and how they are involved may be controlled by subgroup control information parsed by parser (520) from the coded video sequence. For the sake of brevity, such subgroup control information flow between parser (520) and the multiple units below is not described.

[0061] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In a practical embodiment operating under commercial constraints, many of these units interact closely with each other and can be integrated with each other. However, for the purpose of describing the disclosed subject matter, the conceptual subdivision into the following functional units is appropriate.

[0062] The first unit is a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) receives quantized transform coefficients as symbols (521) from the parser (520) and control information, including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (551) can output a block including sample values, which can be input into an aggregator (555).

[0063] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates surrounding blocks of the same size and shape as the block being reconstructed using reconstructed information extracted from a current picture buffer (558). For example, the current picture buffer (558) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (555) adds the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0064] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (521), these samples may be added to the output of the scaler / inverse transform unit (551) (in this case referred to as residual samples or residual signals) by the aggregator (555) to generate output sample information. The acquisition of the predicted samples by the motion compensated prediction unit (553) from the address in the reference picture memory (557) may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (553) in the form of the symbols (521), for example, including X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory (557) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, and the like.

[0065] The output samples of the aggregator (555) may be employed by various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video bitstream) and available to the loop filter unit (556) as symbols (521) from the parser (520). However, in other embodiments, the video compression techniques may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, as well as to previously reconstructed and loop filtered sample values.

[0066] The output of the loop filter unit (556) may be a sample stream that may be output to a display device (512) and stored in a reference picture memory (557) for subsequent inter-picture prediction.

[0067] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557), and a new current picture buffer may be reallocated before starting to reconstruct a subsequent coded picture.

[0068] The video decoder (510) may perform decoding operations according to a predetermined video compression technique, such as in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the profile recorded in the video compression technique or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technique or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata of the HRD buffer management signaled in the encoded video sequence.

[0069] In an embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0070] Figure 6 6 is a block diagram of a video encoder (603) according to an embodiment disclosed in the present application. The video encoder (603) is provided in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used to replace Figure 4 A video encoder (403) in an embodiment.

[0071] The video encoder (603) can be used to obtain the video source (601) (not Figure 6 In another embodiment, the video source (601) is a part of the electronic device (620) to receive video samples, and the video source can collect video images to be encoded by the video encoder (603). In another embodiment, the video source (601) is a part of the electronic device (620).

[0072] The video source (601) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (603), wherein the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (601) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that are given motion when viewed sequentially. The pictures themselves may be constructed as a spatial pixel array, wherein each pixel may include at least one sample depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.

[0073] According to an embodiment, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to these units. For the sake of simplicity, coupling is not indicated in the figure. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be used to have other suitable functions that are related to the video encoder (603) optimized for a certain system design.

[0074] In some embodiments, the video encoder (603) operates in an encoding loop. As a simple description, in embodiments, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression technology considered in this application, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, eg due to channel errors) is also used in some related techniques.

[0075] The operation of the "local" decoder (633) can be combined with the above Figure 5 The "remote" decoder described in detail for the video decoder (510) is identical. However, additional brief reference is made to Figure 4 , when symbols are available and the entropy encoder (645) and parser (520) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0076] At this point, it can be observed that any decoder technology other than the parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the present application focuses on decoder operation. The description of encoder technology can be simplified because encoder technology is mutually inverse to the decoder technology described comprehensively. A more detailed description is only needed in certain areas and is provided below.

[0077] During operation, in some embodiments, the source encoder (630) may perform motion compensated predictive coding. The motion compensated predictive coding predictively encodes an input picture with reference to at least one previously encoded picture from a video sequence designated as a "reference picture." In this manner, the encoding engine (632) encodes the difference between a pixel block of the input picture and a pixel block of a reference picture that may be selected as a prediction reference for the input picture.

[0078] The local video decoder (633) may decode the encoded video data of the picture that may be designated as the reference picture based on the symbol created by the source encoder (630). The operation of the encoding engine (632) may be a lossy process. When the encoded video data is available at the video decoder ( Figure 6 When the video sequence is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may store a copy of the reconstructed reference picture locally that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.

[0079] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new picture. The predictor (635) may operate pixel-by-pixel based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (635), it may be determined that the input picture may have prediction references taken from a plurality of reference pictures stored in the reference picture memory (634).

[0080] The controller (650) may manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0081] The outputs of all the above functional units may be entropy encoded in an entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0082] The transmitter (640) may buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission over a communication channel (660), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) may combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0083] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following picture types:

[0084] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0085] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0086] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0087] The source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or the blocks may be predictively coded (spatial prediction or intra prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0088] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0089] In an embodiment, the transmitter (640) may transmit additional data when transmitting the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0090] The captured video may be taken as a plurality of source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often simplified to intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In an embodiment, a particular picture being encoded / decoded is divided into blocks, and the particular picture being encoded / decoded is referred to as the current picture. When a block in the current picture is similar to a reference block in a reference picture that was previously encoded in the video and is still buffered, the block in the current picture may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case where multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

[0091] In some embodiments, bidirectional prediction techniques may be used in inter-picture prediction. According to the bidirectional prediction technique, two reference pictures are used, for example, a first reference picture and a second reference picture that are both before the current picture in the video in decoding order (but may be in the past and future in display order, respectively). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Specifically, the block may be predicted by a combination of the first reference block and the second reference block.

[0092] In addition, merge mode technology can be used in inter-picture prediction to improve coding efficiency.

[0093] According to some embodiments disclosed in the present application, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a video picture sequence are divided into coding tree units (CTUs) for compression, and the CTUs in the pictures have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Furthermore, each CTU can be split into at least one coding unit (CU) using a quadtree. For example, a 64×64 pixel CTU can be split into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In an embodiment, each CU is analyzed to determine the prediction type for the CU, such as an inter-prediction type or an intra-prediction type. In addition, depending on temporal and / or spatial predictability, the CU is split into at least one prediction unit (PU). Typically, each PU includes a luminance prediction block (PB) and two chrominance PBs. In an embodiment, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Taking the luminance prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luminance values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and the like.

[0094] Figure 7 is a diagram of a video encoder (703) according to another embodiment disclosed in the present application. The video encoder (703) is used to receive a processed block (e.g., a prediction block) of sample values ​​in a current video picture in a video picture sequence, and encode the processed block into an encoded picture that is part of an encoded video sequence. In this embodiment, the video encoder (703) is used to replace Figure 4A video encoder (403) in an embodiment.

[0095] In an HEVC embodiment, a video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) uses, for example, rate-distortion (RD) optimization to determine whether to use intra mode, inter mode, or bidirectional prediction mode to encode the processing block. When encoding the processing block in intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded picture; and when encoding the processing block in inter mode or bidirectional prediction mode, the video encoder (703) may use inter prediction or bidirectional prediction techniques to encode the processing block into an encoded picture, respectively. In some video encoding techniques, the merge mode may be an inter-picture prediction submode, in which a motion vector is derived from at least one motion vector predictor without the aid of an encoded motion vector component external to the predictor. In some other video encoding techniques, there may be a motion vector component applicable to the subject block. In an embodiment, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0096] exist Figure 7 In an embodiment of the present invention, the video encoder (703) includes: Figure 7 Shown are an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together.

[0097] The inter-frame encoder (730) is used to receive samples of a current block (e.g., a processing block), compare the block with at least one reference block in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference picture is a decoded reference picture decoded based on the encoded video information.

[0098] The intra-frame encoder (722) is used to receive samples of a current block (e.g., a processing block), compare the block with an encoded block in the same picture in some cases, generate quantization coefficients after transformation, and in some cases also generate intra-frame prediction information (e.g., intra-frame prediction direction information according to at least one intra-frame coding technique). In an embodiment, the intra-frame encoder (722) also calculates an intra-frame prediction result (e.g., a predicted block) based on the intra-frame prediction information and a reference block in the same picture.

[0099] The general controller (721) is used to determine general control data and control other components of the video encoder (703) based on the general control data. In an embodiment, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and add the intra prediction information to the bitstream; and when the mode is inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select the inter prediction information and add the inter prediction information to the bitstream.

[0100] The residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra-frame encoder (722) or the inter-frame encoder (730). The residual encoder (724) is used to operate based on the residual data to encode the residual data to generate a transform coefficient. In an embodiment, the residual encoder (724) is used to convert the residual data from the spatial domain to the frequency domain and generate a transform coefficient. The transform coefficient is then processed by quantization to obtain a quantized transform coefficient. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra-frame encoder (722) and the inter-frame encoder (730). For example, the inter-frame encoder (730) can generate a decoded block based on the decoded residual data and the inter-frame prediction information, and the intra-frame encoder (722) can generate a decoded block based on the decoded residual data and the intra-frame prediction information. The decoded blocks are appropriately processed to generate a decoded picture, and in some embodiments, the decoded picture may be buffered in a memory circuit (not shown) and used as a reference picture.

[0101] The entropy encoder (725) is used to format the code stream to produce an encoded block. The entropy encoder (725) generates various information according to a suitable standard such as the HEVC standard. In an embodiment, the entropy encoder (725) is used to obtain general control data, selected prediction information (such as intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the code stream. It should be noted that according to the disclosed subject matter, when the block is encoded in the inter-frame mode or the merge sub-mode of the bidirectional prediction mode, there is no residual information.

[0102] Figure 8FIG. 8 is a diagram of a video decoder (810) according to another embodiment disclosed in the present application. The video decoder (810) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In an embodiment, the video decoder (810) is used to replace Figure 4 A video decoder (410) of an embodiment.

[0103] exist Figure 8 In one embodiment, the video decoder (810) includes: Figure 8 An entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874) and an intra-frame decoder (872) coupled together are shown in FIG.

[0104] The entropy decoder (871) can be used to reconstruct certain symbols from the encoded picture, which represent syntax elements that constitute the encoded picture. Such symbols may include, for example, a mode for encoding the block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, a merged sub-mode of the latter two, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can identify certain samples or metadata for prediction by the intra-frame decoder (872) or the inter-frame decoder (880), respectively, residual information in the form of, for example, quantized transform coefficients, etc. In an embodiment, when the prediction mode is inter-frame or bidirectional prediction mode, the inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is an intra-frame prediction type, the intra-frame prediction information is provided to the intra-frame decoder (872). The residual information can be inverse quantized and provided to the residual decoder (873).

[0105] The inter-frame decoder (880) is used to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0106] The intra-frame decoder (872) is used to receive intra-frame prediction information and generate a prediction result based on the intra-frame prediction information.

[0107] The residual decoder (873) is used to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require some control information (to obtain the quantizer parameter QP), and this information can be provided by the entropy decoder (871) (the data path is not indicated because this is only low-volume control information).

[0108] The reconstruction module (874) is used to combine the residual output by the residual decoder (873) with the prediction result (which can be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which can be part of a reconstructed picture, which in turn can be part of a reconstructed video. It should be noted that other suitable operations such as deblocking operations can be performed to improve visual quality.

[0109] It should be noted that the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using any suitable technology. In an embodiment, the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using at least one integrated circuit. In another embodiment, the video encoder (403), video encoder (603) and video encoder (703) and video decoder (410), video decoder (510) and video decoder (810) may be implemented using at least one processor that executes software instructions.

[0110] According to some embodiments of the present application, in addition to the inter prediction mode and the intra prediction mode, a coding mode called a palette-based coding mode may be used. The present application provides a technique for improving the coding and decoding efficiency of the palette-based coding mode.

[0111] In some embodiments, the palette is defined as a lookup table of entries that associate an index with a pixel value. In some examples, the pixel value includes at least one value corresponding to a color component. In one example, the pixel value may include three values ​​representing three color components, respectively. In another example, for a monochrome palette, the pixel value includes a value representing a monochrome component.

[0112] In some embodiments, the pixel block may be encoded based on a palette, and the encoding mode using the palette is called a palette-based encoding mode. In some video coding standards, such as VVC, a palette-based encoding mode is adopted.

[0113] According to some embodiments, the palette-based coding mode is an independent coding mode. In one embodiment, the palette-based coding mode is implemented as a coding unit (e.g., a palette-based encoder, a palette-based decoder, etc.) separated from the intra-frame encoder (722), the inter-frame encoder (730), the intra-frame decoder (872), and the inter-frame decoder (870). In one example, the palette-based coding mode is implemented as a hardware circuit separated from the intra-frame encoder (722), the inter-frame encoder (730), the intra-frame decoder (872), and the inter-frame decoder (870). In another example, the palette-based coding mode is implemented as a software module separated from the software modules for the intra-frame encoder (722), the inter-frame encoder (730), the intra-frame decoder (872), and the inter-frame decoder (870). The software module for the palette-based coding mode and other software modules can be executed by the same processor or different processors.

[0114] According to some embodiments of the present application, the palette-based encoding mode generally includes two parts: a first part encoding method for the palette and a second part encoding method for samples using the palette. The first part may include major color selection and palette encoding. The second part may include palette index encoding, runlength encoding, and escape pixel encoding.

[0115] In some examples, a palette-based encoder may encode a block of video data by: determining a palette for the block (e.g., explicitly encoding the palette, predicting the palette, a combination of signaling and predicting the palette, etc.); locating an entry in the palette to represent at least one of the pixel values; and encoding the block with an index value, wherein the index value is used to indicate an entry in the palette for representing the pixel value of the block. In some examples, the palette-based encoder may signal the palette and / or the index value in the encoded bitstream. In turn, the palette-based decoder may obtain the palette of a block and the index values ​​of each pixel of the block from the encoded bitstream. In other examples, the palette is predictable, the palette-based encoder may signal the index value in the encoded bitstream without signaling the palette, and the palette-based decoder may predict the palette and receive the index value in the encoded bitstream. The palette-based decoder can then associate the index value of each pixel with the palette entry that provides the pixel value. The palette-based decoder then reconstructs each pixel of the block based on the pixel value associated with the index value.

[0116] Some embodiments of the present application provide techniques used in a first part related to the palette encoding process.

[0117] According to some embodiments, the color palette may include the most dominant pixel values ​​in a given block. For example, the most dominant pixel value may include at least one pixel value that occurs most frequently within a given block. Additionally, in some examples, a threshold value (e.g., the number of pixels having a certain pixel value) may be used to determine whether to use the pixel value as one of the most dominant pixel values ​​in the block.

[0118] In some embodiments, the pixel values ​​of the pixels in a block are classified using a histogram-based algorithm to form a color palette. In some examples, the most significant L peaks in the histogram can be selected as the primary color (L is a positive integer). In one example, the pixel values ​​close to the primary color can be quantized as the primary color. In some examples, pixels that do not belong to any primary color are called escape pixels. Escape pixels can be quantized before encoding. In some embodiments, for lossless encoding, the quantization process is not performed.

[0119] Further, in some embodiments, a color index is assigned to each pixel to indicate which color the pixel belongs to. In some examples, if L primary colors are used, 0 to (L-1) can be used as the color index of the primary colors, and 0 to (L-1) can be called the primary color set. For those pixels whose primary color index does not exist, a special index (e.g., an index N other than 0 to (L-1)) is assigned to them, and these pixels are called "escape pixels".

[0120] Typically, a palette is implemented as a color lookup table. The color lookup table includes multiple entries. Each entry associates a color index with a color (e.g., a primary color). In some examples, a single palette is used to encode and decode both the luminance component and the chrominance component. Each entry can represent a specific RGB (YUV) color using three color components. For example, a palette can have entry 1 representing pure black (R, G, B) = (0, 0, 0), and can have entry 0 representing blue (R, G, B) = (2, 10, 200). It should be noted that when the video format is 420, the chrominance plane is upsampled to generate a color lookup table for the palette.

[0121] In some embodiments, palette-based encoding is performed on a CU basis. In one example, for the current CU, its color lookup table is derived, which includes the most dominant pixel values ​​in the current CU. In some examples, the size and elements of the color lookup table can be sent. In some other examples, the size and elements of the color lookup table of the CU can be predictively encoded using the size and / or elements of the color lookup table of other CUs.

[0122] According to some embodiments of the present application, palette prediction may be performed based on a palette prediction sublist. In one example, the palette prediction sublist may include multiple palettes used by other CUs. In order to encode the current palette using the palette prediction sublist as a reference, a binary vector may be used to indicate whether each palette in the palette prediction sublist is reused in the current palette.

[0123] In some examples (e.g., HEVC SCC), a palette prediction sublist stores previously encoded palette entries as a reference to predict the current palette. After encoding each CU in palette mode, the palette prediction sublist is updated. In one example, the palette prediction sublist works like a least recently used cache. The most recent palette can be inserted at the beginning of the palette prediction sublist, and if the palette prediction sublist size exceeds a threshold, the entry of the farthest CU in scan order is discarded. In one embodiment, an upper limit on the size of the palette prediction sublist is signaled in the SPS header (directly or indirectly). For example, the palette prediction sublist size can be roughly twice the palette size limit. In some examples, the palette prediction sublist is also referred to as a historical palette prediction sublist (HPPL).

[0124] In some embodiments, the entries in the palette prediction sublist may respectively have a reuse flag. When the reuse flag of an entry is "true" (true) (e.g., "1"), the entry is called a reuse entry. The color components of the reused entries in the palette prediction sublist can be copied to the color lookup table of the current palette. The reused entries are placed at the beginning of the current palette in a manner that maintains their order in the palette prediction sublist. After the reused entries in the current palette, there may be new palette entries that are not in the palette prediction sublist. The new palette entries may be signaled. Each new palette entry may include, for example, three color components.

[0125] In some embodiments, a palette predictor initializer with predefined entries may be used to initialize a palette predictor list. In some cases, using a palette predictor initializer may improve coding efficiency. The palette predictor initializer may be signaled, for example, in a picture parameter set (PPS), a sequence parameter set (SPS), or the like.

[0126] According to an embodiment of the present application, at the boundaries of slices and tiles, palette prediction based on adjacent CUs may not be very effective. Therefore, when the palette prediction sublist is reset at the boundaries of slices and tiles, the slices and tiles will cause considerable coding efficiency loss because it takes time to rebuild the palette prediction sublist after resetting the palette prediction sublist. In some examples, when a palette prediction sublist is initialized using a palette prediction subinitializer with predefined entries, the coding efficiency in this situation can be improved. The palette prediction subinitializer can be signaled in a picture parameter set (PPS) or a sequence parameter set (SPS). In one example, each of the three color components of an entry in the palette prediction subinitializer is signaled. In some examples, the palette prediction sublist is initialized by the palette prediction subinitializer at the beginning of a slice or tile. The entries of the palette prediction sublist also have three color components. In one example, each entry of the palette predictor list may be copied from the palette predictor initializer.

[0127] In some examples, in a palette-based coding mode, a palette prediction sublist is reset at the beginning of each stripe. When parallel processing is applied to a palette-based coding mode, multiple CTU rows can be encoded in parallel. The construction of the palette prediction sublist makes the encoding of a CTU row depend on the CTU row before it. Some embodiments of the present application can achieve true parallel processing of CTU rows, improving encoding and decoding efficiency.

[0128] The methods proposed in the embodiments of the present application may be used alone or in combination in any order. Further, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit (e.g., at least one processor or at least one integrated circuit). In one example, at least one processor executes a program stored in a non-volatile computer-readable storage medium. In the following, the term "block" may be understood as a prediction block, a coding block, or a coding unit (i.e., CU).

[0129] According to some embodiments of the present application, a picture (or frame) in a video may be divided into a plurality of coding regions and processed according to the coding regions. The coding regions may be defined (or predefined) according to any suitable technique. In some embodiments, the coding regions may be processed (e.g., encoded or decoded) independently. Therefore, in some examples, the coding regions may be processed in parallel for fast processing.

[0130] Fig. 9 An example of a picture (900) in a video according to some embodiments of the present application is shown. The picture (900) is divided into CTUs of the same square size for compression. In some examples, a CTU row is defined as a coding region. A CTU row includes multiple CTUs in the same row.

[0131] Fig.10 An example of a picture (1000) in a video according to some embodiments of the present application is shown. The picture (1000) is divided into CTUs of the same square size for ease of compression. Fig.10 In the example, a tile is defined as a coding area. A tile has a rectangular shape, and its width and height are defined by the number of CTUs. For example, Fig.10 The size of a tile in has a width of three CTUs and a height of two CTUs.

[0132] Fig.11 An example of a picture (1100) in a video according to some embodiments of the present application is shown. The picture (1100) is divided into CTUs of the same square size for ease of compression. Fig.11 In the example of FIG. 1 , a slice is defined as a coding area. The slice is defined according to a raster scan order. Each slice may include multiple CTUs arranged in a raster scan order.

[0133] According to some embodiments of the present application, the palette prediction sublist may be reset to a zero state or cleared at the beginning of the coding region. In one example, in order to reset the palette prediction sublist to a zero state, the size of the palette prediction sublist is set to zero. Further, the entries in the palette prediction sublist are also set to zero.

[0134] It should be noted that the coding region can be any suitable coding region. In one example, when the current CU is the beginning of a CTU, the palette prediction sublist can be reset to a zero state or cleared. In another example, when the current CU is the beginning of a tile, the palette prediction sublist can be reset to a zero state or cleared. In another example, when the current CU is the beginning of a CTU row, the palette prediction sublist can be reset to a zero state or cleared. In another example, when the current CU is the beginning of a slice, the palette prediction sublist can be reset to a zero state or cleared.

[0135] In some embodiments, when wavefront parallel processing (WPP) operation is enabled, CTU rows can be processed in parallel in a wavefront manner. In some examples, when WPP operation is enabled, at the beginning of each CTU row, the palette prediction sublist is reset to a zero state.

[0136] In some embodiments, when the palette predictor list is in the zero state, the palette of the block can be signaled in the encoded bitstream. In some other embodiments, the palette predictor list can be initialized according to a palette predictor initializer. The palette predictor initializer can be signaled in a high-level syntax (e.g., PPS or SPS) or can be predefined and known by the encoder and decoder.

[0137] In one embodiment, when the current CU is at the beginning of a CTU row, the palette predictor list may be initialized according to a palette predictor initializer.

[0138] In another embodiment, when WPP operation is enabled, when the current CU is at the beginning of a CTU row, the palette predictor list may be initialized according to the palette predictor initializer.

[0139] In another embodiment, when the current CU is at the beginning of a CTU, the palette predictor list may be initialized according to a palette predictor initializer.

[0140] In another embodiment, when the current CU is at the beginning of a tile, the palette predictor list may be initialized according to a palette predictor initializer.

[0141] In another embodiment, when the current CU is at the beginning of a slice, the palette predictor list may be initialized according to a palette predictor initializer.

[0142] Fig.12 A flowchart outlining a method (1200) according to an embodiment of the present application is shown. The method (1200) can be used for reconstruction of a block. In various embodiments, the method (1200) is performed by a processing circuit, such as a processing circuit in a terminal device (310), (320), (330) and (340), a processing circuit that performs the functions of a video encoder (403), a processing circuit that performs the functions of a video decoder (410), a processing circuit that performs the functions of a video decoder (510), a processing circuit that performs the functions of a video encoder (603), and the like. In some embodiments, the method (1200) is implemented using software instructions, so when the processing circuit executes the software instructions, the processing circuit performs the method (1200). The method starts at step (S1201) and proceeds to step (S1210).

[0143] At step (S1210), encoded information of a block is received from an encoded bitstream. The block is encoded in a palette-based encoding mode. The block is the beginning of a predefined coding region including at least one coding tree unit (CTU) within a current picture. In one example, the block is the beginning of a CTU. In another example, the block is the beginning of a CTU row. In another example, the block is the beginning of a tile. In another example, the block is the beginning of a slice.

[0144] At step (S1220), a palette of the block is determined independently of palette information of at least one previously decoded block of a palette-based encoding mode.

[0145] In some embodiments, when the block is the beginning of a coding region including at least one CTU, a palette prediction sublist is reset to a zero state, the palette prediction sublist storing palette information of at least one previously decoded block. In some examples, the palette of the block is represented by a signal in an encoded bitstream, and the palette of the block can be extracted from the encoded bitstream. In some examples, the palette prediction sublist is reset according to a palette predictor initializer, and then the palette of the block is determined based on the palette prediction sublist.

[0146] In one embodiment, the block is the beginning of a CTU row, and when wavefront parallel processing (WPP) is enabled, a palette prediction sublist is reset to a zero state, the palette prediction sublist storing palette information of at least one previously decoded block.

[0147] In another embodiment, the block is the start of a CTU row, and when wavefront parallel processing (WPP) is enabled, the palette predictor list is reset according to a palette predictor initializer. The palette predictor initializer may be signaled in a high-level syntax (e.g., PPS, SPS, etc.).

[0148] In another embodiment, the block is the beginning of a CTU, and when wavefront parallel processing (WPP) is enabled, the palette predictor list is reset according to the palette predictor initializer.

[0149] In some embodiments, the block is the beginning of a first predefined coding region, the palette of the block is determined independently of palette information of blocks in a second predefined coding region, and the second predefined coding region is decoded in parallel with the first predefined coding region.

[0150] At step (S1230), the pixels of the block are decoded based on the encoded information of the block and the determined palette of the block. In one example, the encoded information of the block includes a pixel index. The palette of the block includes a lookup table that associates the index with the pixel value (e.g., the value of the color component). Based on the lookup table and the index, the pixel value can be determined. Then, the method proceeds to step (S1299) and ends.

[0151] Accordingly, an embodiment of the present application provides a video decoding device, including:

[0152] A receiving module, configured to receive encoded information of a block encoded in a palette-based encoding mode, the block being the beginning of a predefined coding region including at least one coding tree unit CTU;

[0153] a palette determination module for determining a palette of the block independently of palette information of at least one previously decoded block of the palette-based encoding mode; and

[0154] A decoding module is used to decode pixels of the block based on the encoded information of the block and the determined palette of the block.

[0155] According to an embodiment of the present application, the device further includes: a reset module, used to reset the palette prediction sublist to a zero state when the block is the beginning of the coding area including at least one CTU, and the palette prediction sublist stores the palette information of the at least one previously decoded block.

[0156] The block is the beginning of at least one of a CTU, a CTU row, a tile, and a slice.

[0157] According to one embodiment of the present application, the block is the beginning of a CTU row, and the reset module is used to reset the palette prediction sub-list to a zero state when wavefront parallel processing WPP is enabled, and the palette prediction sub-list stores the palette information of the at least one previously decoded block.

[0158] According to an embodiment of the present application, the block is the beginning of a CTU row, and the reset module is used to reset the palette prediction sublist according to the palette prediction subinitializer when WPP is enabled.

[0159] According to an embodiment of the present application, the block is the beginning of a CTU, and the reset module is used to reset the palette prediction sublist according to the palette prediction subinitializer when WPP is enabled.

[0160] According to one embodiment of the present application, the block is the beginning of a first predefined coding area, and the reset module is used to determine the palette of the block independently of the palette information of the block in a second predefined coding area, wherein the second predefined coding area is decoded in parallel with the first predefined coding area.

[0161] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0162] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in at least one computer-readable storage medium. Fig.13 A computer system (1300) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0163] The computer software may be encoded using any suitable machine code or computer language, which may be subjected to assembly, compilation, linking or similar mechanisms to create code comprising instructions that may be executed directly or through interpretation, microcode execution, etc., by at least one computer central processing unit (CPU), graphics processing unit (GPU), etc.

[0164] The instructions may be executed on various types of computers or computer components, including, for example, personal computers, tablets, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0165] Fig.13 The components for the computer system (1300) shown in the example are exemplary in nature and are not intended to imply any limitation on the scope of use or functionality of the computer software implementing the embodiments of the application. Nor should the configuration of the components be interpreted as having any dependency or requirement on any one or combination of components shown in the exemplary embodiment of the computer system (1300).

[0166] The computer system (1300) may include certain human-machine interface input devices. Such human-machine interface input devices may be responsive to input from at least one human user, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, taps), visual input (e.g., gestures), olfactory input (not depicted). The human-machine interface devices may also be used to capture certain media that may not be directly related to a person's conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0167] The input human-machine interface device may include at least one of the following (only one of each is depicted): keyboard (1301), mouse (1302), track pad (1303), touch screen (1310), data gloves (not shown), joystick (1305), microphone (1306), scanner (1307), camera (1308).

[0168] The computer system (1300) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate at least one sense of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen (1310), a data glove (not shown), or a joystick (1305), but there may also be tactile feedback devices that do not serve as input devices), audio output devices (e.g., speakers (1309), headphones (not depicted)), visual output devices (e.g., screens (1310), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which are capable of outputting two-dimensional visual output or output greater than three dimensions through, for example, stereographic output; virtual reality glasses (not depicted), holographic displays, and smoke boxes (not depicted)), and printers (not depicted). Visual output devices (e.g., screens (1310)) may be connected to the system bus (1348) through a graphics adapter (1350).

[0169] The computer system (1300) may also include human-accessible storage devices and associated media for the storage devices, such as optical media, including CD / DVD ROM / RW (1320) with CD / DVD and other media (1321), thumb drives (1322), removable hard drives or solid-state drives (1323), legacy magnetic media such as tapes and floppy disks (not depicted), ROM / ASIC / PLD-based special-purpose devices, such as security protection devices (not depicted), and the like.

[0170] Those skilled in the art will also appreciate that the term "computer-readable storage media" used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transient signals.

[0171] The computer system (1300) may also include a network interface (1354) to at least one communication network (1355). The communication network (1355) may be, for example, wireless, wired, optical. The communication network (1355) may also be local, wide area, metropolitan, vehicle-mounted and industrial, real-time, delay-tolerant, etc. Examples of the communication network (1355) include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicle-mounted networks and industrial networks including CAN bus, etc. Some networks typically require an external network interface adapter attached to some general data port or peripheral bus (1349) (e.g., a USB port of the computer system (1300)); other networks are typically integrated into the core of the computer system (1300) by attaching to a system bus as described below (e.g., integrated into a PC computer system through an Ethernet interface, or integrated into a smartphone computer system through a cellular network interface). Using any of these networks, the computer system (1300) can communicate with other entities. Such communications can be one-way receive only (e.g., broadcast TV), one-way send only (e.g., a CAN bus connected to some CAN bus devices), or bidirectional, such as connecting to other computer systems using a local area digital network or a wide area digital network. Certain protocols and protocol stacks can be used on each of those networks and network interfaces as described above.

[0172] The above-mentioned human-machine interface device, human-accessible storage device, and network interface may be attached to the core (1340) of the computer system (1300).

[0173] The core (1340) may include at least one central processing unit (CPU) (1341), a graphics processing unit (GPU) (1342), a dedicated programmable processing unit in the form of a field programmable gate area (FPGA) (1343), hardware accelerators for certain tasks (1344), etc. These devices, along with read-only memory (ROM) (1345), random access memory (1346), and internal mass storage devices (1347) such as internal non-user accessible hard drives, SSDs, etc., may be connected via a system bus (1348). In some computer systems, the system bus (1348) may be accessible via at least one physical plug to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached to the core's system bus (1348) directly or via a peripheral bus (1349). Architectures for peripheral buses include PCI, USB, etc.

[0174] The CPU (1341), GPU (1342), FPGA (1343) and accelerator (1344) can execute certain instructions, which in combination can constitute the above-mentioned computer code. The computer code can be stored in ROM (1345) or RAM (1346). Transient data can also be stored in RAM (1346), while permanent data can be stored, for example, in an internal mass storage device (1347). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with at least one CPU (1341), GPU (1342), mass storage device (1347), ROM (1345), RAM (1346), etc.

[0175] The computer readable storage medium may have computer codes for performing various computer-implemented operations. The media and computer codes may be media and computer codes designed and constructed specifically for the purpose of this application, or may be of a type well known and available to technicians in the computer software field.

[0176] By way of example and not limitation, a computer system having the architecture (1300) and in particular the core (1340) may provide functionality resulting from the execution of software embodied in at least one tangible computer-readable storage medium by a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such computer-readable storage media may be media associated with the user-accessible mass storage device introduced above and certain storage devices of the non-volatile nature of the core (1340), such as the core internal mass storage device (1347) or ROM (1345). Software implementing various embodiments of the present application may be stored in such devices and executed by the core (1340). Depending on specific needs, the computer-readable storage medium may include at least one memory device or chip. The software may enable the core (1340) and, in particular, the processor therein (including a CPU, GPU, FPGA, etc.) to perform specific processes or specific portions of specific processes described herein, including defining data structures stored in the RAM (1346) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality produced by logic hardwired or otherwise embodied in circuits (e.g., accelerators (1344)) that may operate in place of or in conjunction with software to perform specific processes or specific portions of specific processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable storage media may encompass circuits (e.g., integrated circuits (ICs)) storing software for execution, circuits embodying logic for execution, or both, where appropriate. The present application encompasses any suitable combination of hardware and software.

[0177] Appendix A: Acronyms

[0178] JEM: Joint Exploration Model

[0179] VVC: Versatile Video Coding

[0180] BMS: Benchmark Set, benchmark set

[0181] MV: Motion Vector

[0182] HEVC: High Efficiency Video Coding

[0183] SEI: Supplementary Enhancement Information, auxiliary enhancement information

[0184] VUI: Video Usability Information, video availability information

[0185] GOP: Groups of Pictures, picture group

[0186] TU: Transform Unit

[0187] PU: Prediction Unit

[0188] CTU: Coding Tree Unit, coding tree unit

[0189] CTB: Coding Tree Block, coding tree block

[0190] PB: Prediction Block

[0191] HRD: Hypothetical Reference Decoder, hypothetical reference decoder

[0192] SNR: Signal Noise Ratio

[0193] CPU: Central Processing Unit

[0194] GPU: Graphics Processing Unit, Graphics Processing Unit

[0195] CRT: Cathode Ray Tube

[0196] LCD: Liquid-Crystal Display

[0197] OLED: Organic Light-Emitting Diode

[0198] CD: Compact Disc

[0199] DVD: Digital Video Disc

[0200] ROM:Read-Only Memory

[0201] RAM: Random Access Memory

[0202] ASIC: Application-Specific Integrated Circuit

[0203] PLD:Programmable Logic Device, programmable logic device

[0204] LAN: Local Area Network

[0205] GSM: Global System for Mobile communications

[0206] LTE: Long-Term Evolution

[0207] CANBus: Controller Area Network Bus, controller area network bus

[0208] USB: Universal Serial Bus

[0209] PCI: Peripheral Component Interconnect, peripheral device interconnect

[0210] FPGA: Field Programmable Gate Array

[0211] SSD: Solid-state drive

[0212] IC:Integrated Circuit

[0213] CU: Coding Unit

[0214] Although the present application describes several exemplary embodiments, various modifications, permutations and combinations, and various alternative equivalents are possible within the scope of the present application. Therefore, it should be understood that within the spirit and scope of the application, those skilled in the art can design various systems and methods that are not explicitly shown or described herein but can embody the principles of the present application.

Claims

1. A video decoding method, characterized in that: include: Receiving coded information of a block coded in a palette-based coding mode, the block being the beginning of a predefined coding region including at least one coding tree unit CTU; When the block is the beginning of the coding region including at least one CTU, setting the size of a palette prediction sublist storing palette information of at least one previously decoded block of the palette-based coding mode to zero to reset the palette prediction sublist to a zero state, the coding region including a CTU row, and wavefront parallel processing (WPP) is enabled; obtaining a palette of the block from an encoded video bitstream independently of palette information of at least one previously decoded block of the palette-based coding mode; as well as Pixels of the block are decoded based on the encoded information of the block and the determined palette of the block.

2. The method according to claim 1, characterized in that The block is the beginning of a first predefined coding region, and the method further comprises: The palette of the block is determined independently of palette information of a block in a second predefined coding region, wherein the second predefined coding region is decoded in parallel with the first predefined coding region.

3. A video decoding device, characterized in that: include: A receiving module, configured to receive encoded information of a block encoded in a palette-based encoding mode, the block being the beginning of a predefined coding region including at least one coding tree unit CTU; a palette determination module for obtaining a palette of the block from an encoded video bitstream independently of palette information of at least one previously decoded block of the palette-based encoding mode; as well as A decoding module, configured to decode pixels of the block based on the encoded information of the block and the determined color palette of the block; The device further comprises: A reset module is used to set the size of a palette prediction sublist storing palette information of at least one previously decoded block of a palette-based coding mode to zero to reset the palette prediction sublist to a zero state when the block is the beginning of the coding area including at least one CTU, the coding area including a CTU row, and wavefront parallel processing WPP is enabled.

4. The device according to claim 3, characterized in that The block is the beginning of a first predefined coding region, and the apparatus further comprises: A reset module is used to determine the palette of the block independently of the palette information of the block in a second predefined coding area, wherein the second predefined coding area is decoded in parallel with the first predefined coding area.

5. A video encoding method, characterized in that: include: Receiving pixel values ​​of a block in a current picture to be encoded in a palette-based coding mode, the block being the beginning of a predefined coding region including at least one coding tree unit CTU; When the block is the beginning of the coding region including at least one CTU, setting the size of a palette prediction sublist storing palette information of at least one previously encoded block in a palette-based coding mode to zero to reset the palette prediction sublist to a zero state, the coding region including a CTU row, and wavefront parallel processing (WPP) is enabled; determining a palette for the block independently of palette information of at least one previously encoded block of the palette-based encoding mode; Based on the determined color palette of the block, pixel values ​​of the block are encoded, and the color palette of the block is included in a video code stream.

6. A method for processing a video stream, characterized in that: The video code stream is decoded based on the decoding method according to claim 1 or 2, or is generated based on the encoding method according to claim 5.

7. An electronic device, characterized in that: The invention comprises a memory for storing computer-readable instructions; a processor for reading the computer-readable instructions and executing the method as claimed in any one of claims 1, 2, 5 and 6 according to the instructions of the computer-readable instructions.

8. A non-volatile computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1, 2, 5, and 6.

Images