Coding method and apparatus

Abstract

The embodiment of the application discloses a coding method and device, relates to the technical field of media, and can balance the subjective quality of an image after image padding. The method comprises the following steps: performing image padding on a to-be-encoded image to obtain a padded image. Then, a coding unit is obtained according to the padded image. Subsequently, coding information of the coding unit is determined according to full padding information of the coding unit. Then, the coding unit is encoded according to the coding information to generate a code stream. The full padding information is used to indicate whether all the samples in the coding unit are image padding samples, and the coding information comprises at least one of a padding bit number or a coding code length.

Description

[0001] This application is a divisional application. The original application has the application number 202310290271.4 and the original application date is March 13, 2023. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of media technology, and more particularly to encoding and decoding methods and apparatus. Background Technology

[0003] Media devices utilize display interfaces when transmitting media content. These interfaces can compress media content through encoding to reduce bandwidth usage during transmission. Upon receiving the compressed media content, the receiving end needs to decode it to restore the original media content.

[0004] In compression scenarios, the input image can be divided into multiple strips, and each strip is then encoded. The resolution of the input image and the number of strips are often determined based on product requirements; therefore, the image resolution may not be sufficient to meet the required number of strips. To ensure that the encoder and decoder function correctly under such input conditions, the input image needs to be padded before encoding to allow it to be divided into the required integer number of strips. The padded image is then encoded and transmitted, and the bitstream is decoded by the decoder to obtain the reconstructed image. Finally, the padded areas are cropped from the reconstructed image to restore the original resolution.

[0005] In related technologies, the subjective quality of different strips in the image after image filling may vary, resulting in uneven subjective quality of the filled image.

[0006] Therefore, how to achieve a balance in the subjective quality of the incomplete image is one of the problems that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] This application provides encoding / decoding methods and apparatus that enable subjective quality balance in incomplete images. To achieve the above objective, this application adopts the following technical solutions:

[0008] In a first aspect, embodiments of this application provide an encoding method, which includes: performing image padding on an image to be encoded to obtain a padded image; then obtaining encoding units based on the padded image; subsequently determining encoding information for the encoding units based on the full padding information; and then encoding the encoding units to generate a bitstream based on the encoding information. The full padding information is used to indicate whether all samples in the encoding unit are image-padded samples, and the encoding information includes at least one of the following: padding bit count or encoding code length.

[0009] In related technologies, when encoding and decoding the padded image, if there is too much padded content in the strip and the padded content is simple, the number of coding bits occupied by the fully padded coding unit in the strip is less. Consequently, the number of coding bits used for the incompletely padded coding unit in the strip will be more abundant than other strips with no padded content or less padded content. This results in a difference in the reconstruction quality of the non-padded content in the strip compared to the reconstruction quality of other strips, causing uneven subjective quality of the padded image.

[0010] In the method provided in this application embodiment, the encoding information of the encoding unit can be determined based on the full-padding information of the encoding unit, and then the encoding unit is encoded using the encoding information to generate a bitstream. This allows for the identification of fully-painted encoding units within a strip through the full-padding information. By adjusting the encoding parameters of the fully-painted encoding units to increase their bit count, the bit count of the incompletely-painted encoding units within the strip is reduced. This reduces the difference in reconstruction quality between the incompletely-painted content in the strip and other strips, avoiding subjective quality inconsistencies in the padded image and achieving a more balanced subjective quality. In other words, by adjusting the compression performance of the fully-painted encoding units, the subjective quality of the padded image is made more uniform.

[0011] In one possible implementation, the above-mentioned encoding unit can be encoded to generate a bitstream to be filled, and bits can be filled into the bitstream to be filled according to the above-mentioned number of filling bits to obtain the bitstream.

[0012] As can be seen, in the method provided in this application embodiment, the encoding information of the encoding unit can be determined based on the full padding information of the encoding unit, and then the encoding unit is encoded to generate a bitstream using the encoding information. In this way, the fully padding encoding units in the strip can be determined by the full padding information, and then bits are padded in the initial bitstream (i.e. the bitstream to be padded) generated by the fully padding encoding units to increase the number of encoded bits, thereby reducing the number of encoded bits of the non-fully padding encoding units in the strip. This reduces the difference between the reconstruction quality of the non-padding content in the strip and the reconstruction quality of other strips, avoids uneven subjective quality of the padded image, and thus makes the subjective quality of the padded image more balanced.

[0013] In one possible implementation, the above-mentioned coding unit can be encoded with a fixed length according to the above-mentioned coding code length to generate the above-mentioned bitstream.

[0014] As can be seen, in the method provided in this application embodiment, the encoding information of the encoding unit can be determined based on the full padding information of the encoding unit, and then the encoding unit is encoded using the encoding information to generate a bitstream. This allows for the determination of fully padding encoding units in a strip by the full padding information, and then fixed-length encoding of the encoding units to generate a bitstream based on the encoding code length, increasing the number of encoded bits. This reduces the number of encoded bits for incompletely padding encoding units within the strip, thereby reducing the difference in reconstruction quality between the non-padding content in the strip and other strips, avoiding subjective quality inconsistencies in the padded image, and ultimately achieving balanced subjective quality in the padded image.

[0015] In one possible implementation, when the above-mentioned coding unit is a target coding unit, the number of padding bits can be determined based on the actual number of bits of the above-mentioned coding unit and a first preset number of bits. The target coding unit is a coding unit in which all samples are image padding samples, and the actual number of bits is the number of bits in the bitstream to be filled corresponding to the above-mentioned coding unit.

[0016] Optionally, the number of padding bits, BitsGap, can satisfy: BitsGap = Max(X0 - BCU, 0).

[0017] Wherein, BCU is the bit of the bit stream to be filled corresponding to the coding unit (i.e., the actual number of coded bits obtained after the current coding block is encoded), and X0 is the first preset number of bits (i.e., the total number of bits of the bit stream corresponding to the coding unit when the complete filling flag is 1).

[0018] As can be seen, in the method provided by the embodiments of this application, the encoding information of the encoding unit can be determined according to the full padding information of the encoding unit, and then the encoding unit is encoded to generate a bitstream using the encoding information. In this way, the fully padding encoding units in the strip can be determined by the full padding information, and then the padding bit number is determined according to the actual bit number of the encoding unit and the first preset bit number. Then, bits are padded in the initial bitstream (i.e. the bitstream to be filled) generated by the fully padding encoding units according to the padding bit number, so that the number of encoded bits increases, thereby reducing the number of encoded bits of the non-fully padding encoding units in the strip. This reduces the difference between the reconstruction quality of the non-padding content in the strip and the reconstruction quality of other strips, avoids the subjective quality unevenness of the padded image, and thus makes the subjective quality of the padded image more balanced.

[0019] In one possible implementation, when the above-mentioned coding unit is a target coding unit, the coding length can be determined based on the header information overhead of the above-mentioned coding unit, the second preset number of bits, and the number of samples of the above-mentioned coding unit, wherein the target coding unit is a coding unit in which all samples are image filling samples.

[0020] Optional, code length Bpp pad It can satisfy: Bpp pad =(X1-X2) / Cu size .

[0021] Where X1 is the second preset number of bits (i.e., the total number of bits in the bitstream corresponding to the fully filled coding unit), X2 is the header information overhead of the current coding unit, and Cu size Number of samples for coding units

[0022] As can be seen, in the method provided in this application embodiment, the encoding information of the encoding unit can be determined based on the full-filling information of the encoding unit, and then the encoding unit is encoded to generate a bitstream using the encoding information. This allows for the determination of fully filled encoding units in a strip using the full-filling information. Then, the encoding code length is determined using the header information overhead of the encoding unit, the second preset number of bits, and the number of samples of the encoding unit. Finally, the encoding unit is encoded using a fixed-length encoding method to generate a bitstream, increasing the number of encoded bits. This reduces the number of encoded bits for incompletely filled encoding units within the strip, thereby reducing the difference in reconstruction quality between the non-filled content in the strip and other strips, avoiding uneven subjective quality in the filled image, and ultimately achieving balanced subjective quality in the filled image.

[0023] In one possible implementation, the aforementioned full-fill information is used to further indicate whether the aforementioned coding unit is located within the target strip of the aforementioned filled image, the target strip being the horizontal image filling strip.

[0024] In one possible implementation, the target coding unit is a coding unit whose samples are all image filling samples and located within the target strip of the filled image.

[0025] In one possible implementation, the image to be encoded can be divided into multiple strips, and then N columns of samples can be filled in at the right boundary of the last strip in the image to be encoded. N is a positive integer.

[0026] Understandably, instead of performing image filling on every single strip, performing image filling only on the last column of the image can avoid increasing the cost and power consumption of image filling hardware implementation.

[0027] Secondly, embodiments of this application also provide a decoding method, the method comprising: acquiring a bitstream; decoding the bitstream to obtain a reconstructed block; and generating a reconstructed image based on the reconstructed block. The bitstream is a bitstream generated by an encoding unit according to the encoding information of the encoding unit, the encoding information being determined based on the full padding information of the encoding unit, the full padding information being used to indicate whether all samples in the encoding unit are image padding samples, and the encoding information including at least one of the number of padding bits or the encoding code length.

[0028] In related technologies, when encoding and decoding the padded image, if there is too much padded content in the strip and the padded content is simple, the number of coding bits occupied by the fully padded coding unit in the strip is less. Consequently, the number of coding bits used for the incompletely padded coding unit in the strip will be more abundant than other strips with no padded content or less padded content. This results in a difference in the reconstruction quality of the non-padded content in the strip compared to the reconstruction quality of other strips, causing uneven subjective quality of the padded image.

[0029] In the method provided in this application embodiment, the encoding information of the encoding unit can be determined based on the full padding information of the encoding unit, and then the encoding unit is encoded using the encoding information to generate a bitstream. This allows for the identification of fully padded encoding units within a strip using the full padding information. By adjusting the encoding parameters of the fully padded encoding units to increase their bit count, the bit count of the incompletely padded encoding units within the strip is reduced. This reduces the difference in reconstruction quality between the non-padded content in the strip and other strips, avoiding subjective quality inconsistencies in the padded image and achieving a more balanced subjective quality in the padded image.

[0030] In one possible implementation, when the bitstream is the bitstream of the target coding unit, the bitstream can be decoded according to the coding code length to obtain the reconstructed block, where the target coding unit is a coding unit in which all samples are image filling samples.

[0031] In one possible implementation, the reconstructed block is obtained by decoding the bitstream based on the number of padding bits.

[0032] In one possible implementation, when the above-mentioned coding unit is a target coding unit, the number of padding bits is determined based on the actual number of bits of the above-mentioned coding unit and a first preset number of bits. The actual number of bits is the number of bits in the bitstream to be filled corresponding to the above-mentioned coding unit, and the target coding unit is a coding unit in which all samples are image padding samples.

[0033] In one possible implementation, the full-fill information is used to further indicate whether the coding unit is located within the target strip of the filled image, wherein the target strip is the strip of the image filled laterally.

[0034] In one possible implementation, the target coding unit is a coding unit whose samples are all image filling samples and located within the target strip of the filled image.

[0035] In one possible implementation, the image filling region in the reconstructed image can be cropped.

[0036] Thirdly, embodiments of this application also provide an encoding apparatus, comprising: an image filling unit, a division unit, a determination unit, and an encoding unit. The image filling unit is used to fill in an image to be encoded to obtain a filled image. The division unit is used to obtain an encoding unit based on the filled image. The determination unit is used to determine the encoding information of the encoding unit based on the full filling information of the encoding unit, wherein the full filling information indicates whether all samples in the encoding unit are image filling samples, and the encoding information includes at least one of the number of filling bits or the encoding code length. The encoding unit is used to encode the encoding unit according to the encoding information to generate a bitstream.

[0037] In one possible implementation, the above-mentioned encoding unit is specifically used to: encode the above-mentioned encoding unit to generate a bitstream to be filled; and fill the bitstream to be filled with bits according to the above-mentioned number of filling bits to obtain the bitstream.

[0038] In one possible implementation, the above-mentioned encoding unit is specifically used to: perform fixed-length encoding on the above-mentioned encoding unit according to the above-mentioned encoding code length to generate the above-mentioned bitstream.

[0039] In one possible implementation, the determining unit is specifically used to: when the coding unit is a target coding unit, determine the number of padding bits based on the actual number of bits of the coding unit and a first preset number of bits, wherein the target coding unit is a coding unit in which all samples are image padding samples, and the actual number of bits is the number of bits of the bitstream to be filled corresponding to the coding unit.

[0040] In one possible implementation, the determining unit is specifically used to: determine the coding length based on the header information overhead, the second preset number of bits, and the number of samples of the coding unit when the coding unit is a target coding unit, wherein the target coding unit is a coding unit in which all samples are image filling samples.

[0041] In one possible implementation, the aforementioned full-fill information is used to further indicate whether the aforementioned coding unit is located within the target strip of the aforementioned filled image, the target strip being the horizontal image filling strip.

[0042] In one possible implementation, the target coding unit is a coding unit whose samples are all image filling samples and located within the target strip of the filled image.

[0043] Fourthly, embodiments of this application also provide a decoding apparatus, comprising: a receiving unit, a decoding unit, and a reconstruction unit. The receiving unit is configured to acquire a bitstream, which is a bitstream generated by an encoding unit based on encoding information provided by the encoding unit. The encoding information is determined based on full-padding information provided by the encoding unit, which indicates whether all samples in the encoding unit are image-padding samples. The encoding information includes at least one of padding bit count or encoding code length. The decoding unit is configured to decode the bitstream to obtain a reconstructed block. The reconstruction unit is configured to generate a reconstructed image based on the reconstructed block.

[0044] In one possible implementation, the decoding unit is specifically used to: decode the bitstream according to the code length to obtain the reconstructed block when the bitstream is the bitstream of the target coding unit, wherein the target coding unit is a coding unit in which all samples are image filling samples.

[0045] In one possible implementation, the reconstructed block is obtained by decoding the bitstream based on the number of padding bits.

[0046] In one possible implementation, when the above-mentioned coding unit is a target coding unit, the number of padding bits is determined based on the actual number of bits of the above-mentioned coding unit and a first preset number of bits. The actual number of bits is the number of bits in the bitstream to be filled corresponding to the above-mentioned coding unit, and the target coding unit is a coding unit in which all samples are image padding samples.

[0047] In one possible implementation, the full-fill information is used to further indicate whether the coding unit is located within the target strip of the filled image, wherein the target strip is the strip of the image filled laterally.

[0048] In one possible implementation, the target coding unit is a coding unit whose samples are all image filling samples and located within the target strip of the filled image.

[0049] In one possible implementation, the reconstruction unit is further configured to: crop the image filling region in the reconstructed image.

[0050] Fifthly, embodiments of this application also provide an encoding device, the device comprising: at least one processor, which, when the at least one processor executes program code or instructions, implements the method described in the first aspect or any possible implementation thereof.

[0051] Optionally, the device may further include at least one memory for storing the program code or instructions.

[0052] In a sixth aspect, embodiments of this application also provide a decoding apparatus, the apparatus comprising: at least one processor, which, when the at least one processor executes program code or instructions, implements the method described in the second aspect above or any possible implementation thereof.

[0053] Optionally, the device may further include at least one memory for storing the program code or instructions.

[0054] In a seventh aspect, embodiments of this application also provide a chip, including: an input interface, an output interface, and at least one processor. Optionally, the chip further includes a memory. The at least one processor is used to execute code in the memory, and when the at least one processor executes the code, the chip implements the method described in the first aspect or any possible implementation thereof.

[0055] Alternatively, the chip described above can also be an integrated circuit.

[0056] Eighthly, embodiments of this application also provide a computer-readable storage medium for storing a computer program that includes methods for implementing the first aspect or any possible implementation thereof.

[0057] Ninthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to implement the method described in the first aspect or any possible implementation thereof.

[0058] The encoding / decoding device, computer storage medium, computer program product, and chip provided in this embodiment are all used to execute the encoding / decoding method provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the encoding / decoding method provided above, and will not be repeated here. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1a An exemplary block diagram of a decoding system provided in an embodiment of this application;

[0061] Figure 1b An exemplary block diagram of a video decoding system provided in an embodiment of this application;

[0062] Figure 2 An exemplary block diagram of a video encoder provided in an embodiment of this application;

[0063] Figure 3 An exemplary block diagram of a video decoder provided in an embodiment of this application;

[0064] Figure 4 An exemplary schematic diagram of a candidate image block provided in an embodiment of this application;

[0065] Figure 5 An exemplary block diagram of a video decoding device provided in an embodiment of this application;

[0066] Figure 6 An exemplary block diagram of the apparatus provided in the embodiments of this application;

[0067] Figure 7 This is a schematic diagram of the structure of an encoding system provided in an embodiment of this application;

[0068] Figure 8 This is a schematic diagram of another encoding system provided in an embodiment of this application;

[0069] Figure 9 A schematic diagram of an encoding method provided in an embodiment of this application.

[0070] Figure 10 A schematic diagram of an image filling process provided in an embodiment of this application.

[0071] Figure 11 This is a schematic diagram of the structure of a decoding system provided in an embodiment of this application;

[0072] Figure 12 A schematic diagram illustrating a decoding method provided in an embodiment of this application;

[0073] Figure 13 A schematic diagram of an encoding device provided in an embodiment of this application;

[0074] Figure 14 A schematic diagram of a decoding device provided in an embodiment of this application;

[0075] Figure 15 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0076] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the embodiments of this application.

[0077] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0078] The terms "first" and "second," etc., in the specification and drawings of the embodiments of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0079] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0080] It should be noted that in the description of the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0081] In the description of the embodiments of this application, unless otherwise stated, "a plurality of" means two or more.

[0082] First, the terms used in the embodiments of this application will be explained.

[0083] Image padding: The input image can be divided into strips of equal length and width. If the current image width is not an integer multiple of the strip width or the current image height is not an integer multiple of the strip height, image padding is required. The reconstructed or decoded image is cropped before output to finally output an image at its original resolution.

[0084] Fixed-length coding: A source coding method with a fixed codeword length, in which each codeword has the same length.

[0085] Header information overhead: The bit overhead of the current coding unit excluding the coding residual data.

[0086] Interface compression: Media devices use display interfaces when transmitting images and videos. Image bitstream: A binary stream generated after encoding image and video content through the display interface.

[0087] Bitstream: A binary stream generated after encoding image or video content.

[0088] Strips: The input image can be divided into one or more strips. Each strip is divided into one or more coding units for encoding.

[0089] Bitstream: A binary stream generated after encoding image or video content.

[0090] Rate control: The process of adjusting the output bit rate during encoding. Based on the analysis of information such as the current image content and bit stream buffer capacity, the output bit rate is adjusted by changing the quantization parameters and encoding mode.

[0091] Quantization Parameter (QP): During encoding, the residual values ​​generated by the prediction operation or the coefficients generated by the transform operation are quantized and written into the bitstream; during decoding, the syntax elements are dequantized to obtain the residual values ​​or coefficients. QP is the parameter used in this quantization process; generally, the larger the QP value, the more pronounced the quantization. Adjusting the QP value directly affects the length of the encoded bitstream and the quality of the decoded image.

[0092] Data encoding and decoding comprises two parts: data encoding and data decoding. Data encoding is performed on the source side (or commonly referred to as the encoder side) and typically involves processing (e.g., compressing) the raw data to reduce the amount of data required to represent it (thus enabling more efficient storage and / or transmission). Data decoding is performed on the destination side (or commonly referred to as the decoder side) and typically involves performing the inverse processing relative to the encoder side to reconstruct the original data. The "encoding and decoding" of data in the embodiments of this application should be understood as either "encoding" or "decoding" of the data. The encoding and decoding parts are also collectively referred to as encoding and decoding (CODEC).

[0093] In lossless data encoding, the original data can be reconstructed, meaning the reconstructed original data has the same quality as the original data (assuming no transmission loss or other data loss during storage or transmission). In lossy data encoding, further compression is performed through quantization to reduce the amount of data required to represent the original data, but the decoder cannot completely reconstruct the original data, meaning the quality of the reconstructed original data is lower or worse than the quality of the original data.

[0094] This application's embodiments can be applied to video data and other data requiring compression / decompression. The following description uses video data encoding (hereinafter referred to as video encoding) as an example to illustrate this application's embodiments. Other types of data (such as image data, audio data, integer data, and other data requiring compression / decompression) can be referred to the following description, and will not be repeated in this application's embodiments. It should be noted that, compared to video encoding, the encoding of audio data and integer data does not require dividing the data into blocks; instead, the data can be directly encoded.

[0095] Video coding generally refers to the processing of image sequences that form a video or video sequence. In the field of video coding, the terms "picture," "frame," or "image" can be used synonymously.

[0096] Several video coding standards fall under the category of "lossy hybrid video coding and decoding" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is typically segmented into a set of non-overlapping blocks, which are usually encoded at the block level. In other words, the encoder typically processes the video at the block (video block) level, for example, generating prediction blocks through spatial (intra-frame) prediction and temporal (inter-frame) prediction; subtracting the prediction blocks from the current block (the block currently being processed / to be processed) to obtain residual blocks; transforming and quantizing the residual blocks in the transform domain to reduce the amount of data to be transmitted (compressed), while the decoder applies the inverse processing relative to the encoder to the encoded or compressed blocks to reconstruct the current block for representation. Additionally, the encoder needs to repeat the decoder's processing steps so that the encoder and decoder generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstruct pixels for processing, i.e., encoding subsequent blocks.

[0097] In the following embodiment of the decoding system 10, the encoder 20 and decoder 30 are based on Figures 1a to 3 Describe it.

[0098] Figure 1a This is an exemplary block diagram of a decoding system 10 provided for an embodiment of this application, such as a video decoding system 10 (or simply decoding system 10) that can utilize the technology of the embodiments of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video decoding system 10 represent devices, etc., that can be used to perform various technologies according to the various examples described in the embodiments of this application.

[0099] like Figure 1a As shown, the decoding system 10 includes a source device 12, which provides encoded image data 21, such as encoded images, to a destination device 14 for decoding the encoded image data 21.

[0100] The source device 12 includes an encoder 20, and optionally may include an image source 16, an image preprocessor (or preprocessing unit) 18, and a communication interface (or communication unit) 22.

[0101] Image source 16 may include or may be any type of image capture device for capturing real-world images, and / or any type of image generation device, such as a computer graphics processor for generating computer animation images or any type of device for acquiring and / or providing real-world images, computer-generated images such as screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage device storing any of the aforementioned images.

[0102] To distinguish the processing performed by the preprocessor (or preprocessing unit) 18, the image (or image data) 17 may also be referred to as the raw image (or raw image data) 17.

[0103] The preprocessor 18 receives the raw image data 17 and preprocesses it to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component.

[0104] Video encoder (or encoder) 20 is used to receive preprocessed image data 19 and provide encoded image data 21 (hereinafter referred to as...) Figure 2 (and so on, for further description).

[0105] The communication interface 22 in the source device 12 can be used to: receive encoded image data 21 and send encoded image data 21 (or other arbitrarily processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0106] The target device 14 includes a decoder 30, and optionally may include a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32 and a display device 34.

[0107] The communication interface 28 in the destination device 14 is used to receive encoded image data 21 (or other processed versions) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0108] Communication interfaces 22 and 28 can be used to send or receive encoded image data (or encoded data 21) through a direct communication link between source device 12 and destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0109] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing, so as to transmit it on a communication link or communication network.

[0110] Communication interface 28 corresponds to communication interface 22. For example, it can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21.

[0111] Both communication interface 22 and communication interface 28 can be configured as follows: Figure 1a The arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 indicates a one-way communication interface or a two-way communication interface, and can be used to send and receive messages, etc., to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission, such as encoded image data transmission, etc.

[0112] Video decoder (or decoder) 30 is used to receive encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as...). Figure 3 (and so on, for further description).

[0113] The post-processor 32 is used to post-process the decoded image data 31 (also known as the reconstructed image data) to obtain post-processed image data 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, trimming or resampling, or any other processing to generate the decoded image data 31 for display by the display device 34, etc.

[0114] Display device 34 is used to receive post-processed image data 33 to display the image to a user or viewer. Display device 34 can be or includes any type of display for representing the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0115] The decoding system 10 also includes a training engine 25, which is used to train the encoder 20 (especially the entropy coding unit 270 in the encoder 20) or the decoder 30 (especially the entropy decoding unit 304 in the decoder 30) to entropy encode the coding unit to be encoded according to the estimated probability distribution obtained. For a detailed description of the training engine 25, please refer to the following method test example.

[0116] although Figure 1a The source device 12 and destination device 14 are shown as independent devices, but device embodiments may also include both source device 12 and destination device 14, or the functions of both source device 12 and destination device 14, that is, simultaneously including source device 12 or its corresponding functions and destination device 14 or its corresponding functions. In these embodiments, source device 12 or its corresponding functions and destination device 14 or its corresponding functions may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0117] According to the description, Figure 1a The presence and (accurate) division of different units or functions in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0118] Please refer to Figure 1b , Figure 1b This is an exemplary block diagram of a video decoding system 40 provided in an embodiment of this application. The encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both, can be transmitted via... Figure 1bThe processing circuitry in the video decoding system 40 shown is implemented using, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video encoding processors, or any combination thereof. Please refer to... Figure 2 and Figure 3 , Figure 2 This is an exemplary block diagram of a video encoder provided in an embodiment of this application. Figure 3 This is an exemplary block diagram of a video decoder provided in an embodiment of this application. The encoder 20 can be implemented using processing circuitry 46 to include reference... Figure 2 Encoder 20 refers to various modules discussed herein and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented via processing circuitry 46 to include references. Figure 3 Decoder 30 comprises various modules discussed herein and / or any other decoder system or subsystem described herein. The processing circuitry 46 can be used to perform various operations discussed below. Figure 5 As shown, if some of the technology is implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors, thereby performing the technology of the embodiments of this application. One of the video encoder 20 and video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as... Figure 1b As shown.

[0119] Source device 12 and destination device 14 can include any of a variety of devices, including any type of handheld or fixed device, such as laptops or tablets, mobile phones, smartphones, tablets or tablets, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (e.g., content service servers or content distribution servers), broadcast receiving devices, broadcast transmitting devices, and monitoring devices, etc., and may or may not use any type of operating system. Source device 12 and destination device 14 can also be devices in a cloud computing scenario, such as virtual machines in a cloud computing scenario. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0120] Source device 12 and destination device 14 can install virtual reality (VR), augmented reality (AR), or mixed reality (MR) applications, and can run VR, AR, or MR applications based on user actions (such as clicking, touching, swiping, shaking, voice control, etc.). Source device 12 and destination device 14 can acquire images / videos of any object in the environment through cameras and / or sensors, and then display virtual objects on the display device based on the acquired images / videos. These virtual objects can be virtual objects in VR, AR, or MR scenes (i.e., objects in the virtual environment).

[0121] It should be noted that, in this embodiment of the application, the virtual scene application in the source device 12 and the destination device 14 can be an application built into the source device 12 and the destination device 14 themselves, or it can be an application provided by a third-party service provider that is installed by the user. No specific limitation is made in this regard.

[0122] Furthermore, source device 12 and destination device 14 can be equipped with real-time video transmission applications, such as live streaming applications. Source device 12 and destination device 14 can capture images / videos via cameras and then display the captured images / videos on a display device.

[0123] In some cases, Figure 1a The video decoding system 10 shown is merely exemplary. The technology provided in this application embodiment can be applied to video encoding settings (e.g., video encoding or video decoding), which do not necessarily include any data communication between the encoding device and the decoding device. In other examples, data is retrieved from local memory, sent over a network, etc. The video encoding device may encode data and store it in memory, and / or the video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to memory and / or retrieve and decode data from memory.

[0124] Please refer to Figure 1b , Figure 1b An exemplary block diagram of the video decoding system 40 provided in this application embodiment is shown below. Figure 1b As shown, the video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memory storage devices 44, and / or a display device 45.

[0125] like Figure 1b As shown, the imaging device 41, antenna 42, processing circuitry 46, video encoder 20, video decoder 30, processor 43, memory storage 44, and / or display device 45 are capable of communicating with each other. In different instances, the video decoding system 40 may contain only the video encoder 20 or only the video decoder 30.

[0126] In some instances, antenna 42 can be used to transmit or receive encoded bitstreams of video data. Additionally, in some instances, display device 45 can be used to present video data. Processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Video decoding system 40 can also include an optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Furthermore, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory storage 44 can be implemented using high-speed cache memory. In other instances, processing circuitry 46 can include memory (e.g., cache, etc.) for implementing image buffers, etc.

[0127] In some instances, the video encoder 20 implemented via logic circuitry may include (e.g., implemented via processing circuitry 46 or memory storage 44) an image buffer and (e.g., implemented via processing circuitry 46) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video encoder 20 implemented via processing circuitry 46 to implement reference... Figure 2 And / or any other encoder system or subsystem described herein, including the various modules discussed. Logic circuits can be used to perform the various operations discussed herein.

[0128] In some instances, the video decoder 30 can be implemented in a similar manner via the processing circuitry 46 to implement the reference. Figure 3The video decoder 30 and / or any other decoder system or subsystem described herein are various modules discussed. In some instances, the logic circuit-implemented video decoder 30 may include (implemented via processing circuitry 46 or memory storage 44) an image buffer and (e.g., implemented via processing circuitry 46) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented via processing circuitry 46 to implement reference... Figure 3 And / or any other decoder system or subsystem described herein, including the various modules discussed.

[0129] In some instances, antenna 42 can be used to receive encoded bitstreams of video data. As discussed herein, the encoded bitstream may contain data related to encoded video frames, indicators, index values, mode selection data, etc., such as data related to code segmentation (e.g., transform coefficients or quantized transform coefficients, optional indicators, and / or data defining code segmentation). Video decoding system 40 may also include a video decoder 30 coupled to antenna 42 for decoding the encoded bitstream. Display device 45 is used to display the video frames.

[0130] It should be understood that, for the examples described with reference to video encoder 20 in this application embodiment, video decoder 30 can be used to perform the reverse process. Regarding signaling syntax elements, video decoder 30 can be used to receive and parse such syntax elements, and accordingly decode the associated video data. In some examples, video encoder 20 can entropy-encode syntax elements into an encoded video bitstream. In such instances, video decoder 30 can parse such syntax elements and accordingly decode the associated video data.

[0131] For ease of description, embodiments of this application are described with reference to the Universal Video Coding (VVC) reference software or the High-Efficiency Video Coding (HEVC) developed by the ITU-T Video Coding Experts Group (VCEG) and the Joint Collaboration Team on Video Coding (JCT-VC) of the ISO / IEC Moving Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of this application are not limited to HEVC or VVC.

[0132] Encoders and Encoding Methods

[0133] like Figure 2As shown, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The video encoder 20 shown can also be called a hybrid video encoder or a video encoder based on a hybrid video codec.

[0134] See Figure 2 Inter-frame prediction units are trained target models (also known as neural networks) that process input images, image regions, or image patches to generate predicted values ​​for those patches. For example, a neural network for inter-frame prediction receives an input image, image region, or image patch and generates predicted values ​​for that image, image region, or image patch.

[0135] The residual calculation unit 204, transform processing unit 206, quantization unit 208, and mode selection unit 260 constitute the forward signal path of encoder 20, while the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, buffer 216, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244, and intra-frame prediction unit 254 constitute the backward signal path of encoder 20. The backward signal path of encoder 20 corresponds to the signal path of decoder (see [link to decoder]). Figure 3 The decoder 30 in the video encoder 20 consists of an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded image buffer 230, an inter-frame prediction unit 244, and an intra-frame prediction unit 254.

[0136] Image and image segmentation (images and patches)

[0137] Encoder 20 can be used to receive images (or image data) 17 via input terminal 201, for example, images in an image sequence forming a video or video sequence. The received images or image data can also be pre-processed images (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 can also be referred to as the current image or the image to be encoded (especially in video encoding when the current image is distinguished from other images, such as those in the same video sequence, i.e., the video sequence that also includes the current image, previously encoded images, and / or decoded images).

[0138] A digital image is, or can be viewed as, a two-dimensional array or matrix of pixels with intensity values. Pixels in an array are also called pixels (short for image element). The number of pixels in the array or image along the horizontal and vertical directions (or axes) determines the image size and / or resolution. To represent color, three color components are typically used, meaning an image can be represented as or comprise an array of three pixels. In RBG format or color space, an image includes corresponding arrays of red, green, and blue pixels. However, in video coding, each pixel is typically represented in a luma / chroma format or color space, such as YCbCr, which includes the luma component indicated by Y (sometimes also represented by L) and two chroma components represented by Cb and Cr. The luma component Y represents the brightness or grayscale level intensity (e.g., both are the same in grayscale images), while the two chroma components Cb and Cr represent the chroma or color information components. Accordingly, a YCbCr format image consists of a luminance pixel array for the luminance pixel value (Y) and two chrominance pixel arrays for the chrominance values ​​(Cb and Cr). An RGB format image can be converted or transformed to YCbCr format, and vice versa; this process is also known as color conversion or transformation. If the image is black and white, it may only include the luminance pixel array. Accordingly, the image can be, for example, a monochrome format luminance pixel array or a 4:2:0, 4:2:2, and 4:4:4 color format luminance pixel array and two corresponding chrominance pixel arrays.

[0139] In one embodiment, the video encoder 20 may include an image segmentation unit ( Figure 2(Not shown in the image) is used to segment image 17 into multiple (typically non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC), coding tree blocks (CTBs), or coding tree units (CTUs) in the H.265 / HEVC and VVC standards. Segmentation units can be used to apply the same block size and a corresponding grid with defined block sizes to all images in a video sequence, or to vary the block size between images, subsets of images, or groups of images, segmenting each image into corresponding blocks.

[0140] In other embodiments, the video encoder may be used to directly receive blocks 203 of image 17, such as one, several, or all of the blocks that make up image 17. Image block 203 may also be referred to as the current image block or the encoding unit to be encoded.

[0141] Similar to image 17, image block 203 is also a two-dimensional array or matrix composed of pixels with intensity values ​​(pixel values), but image block 203 is smaller than that of image 17. In other words, block 203 may include a pixel array (e.g., a luminance array in the case of monochrome image 17 or a luminance or chrominance array in the case of a color image) or a three-pixel array (e.g., a luminance array and two chrominance arrays in the case of color image 17) or any other number and / or type of array depending on the color format used. The number of pixels in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Accordingly, the block may be an M×N (M columns × N rows) pixel array, or an M×N transform coefficient array, etc.

[0142] In one embodiment, Figure 2 The video encoder 20 shown is used to encode the image 17 block by block, for example, to perform encoding and prediction for each block 203.

[0143] In one embodiment, Figure 2 The video encoder 20 shown can also be used to segment and / or encode images using slices (also called video slices), where images can be segmented or encoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., coding tree units, CTUs) or one or more groups of blocks, such as coded tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard.

[0144] In one embodiment, Figure 2The video encoder 20 shown can also be used to segment and / or encode an image using slice / encoding block groups (also known as video encoding block groups) and / or encoding blocks (also known as video encoding blocks), wherein the image can be segmented or encoded using one or more slice / encoding block groups (typically non-overlapping), each slice / encoding block group may include one or more blocks (e.g., CTUs) or one or more encoding blocks, wherein each encoding block may be rectangular or the like, and may include one or more complete or partial blocks (e.g., CTUs).

[0145] Residual calculation

[0146] The residual calculation unit 204 is used to calculate the residual block 205 based on the image block (or original block) 203 and the prediction block 265 in the following manner (the prediction block 265 is described in detail later): for example, the residual block 205 in the pixel domain is obtained by subtracting the pixel value of the prediction block 265 from the pixel value of the image block 203 pixel by pixel.

[0147] Transformation

[0148] The transformation processing unit 206 performs discrete cosine transform (DCT) or discrete sine transform (DST) on the pixel values ​​of the residual block 205 to obtain the transformation coefficients 207 in the transform domain. The transformation coefficients 207 can also be called transformation residual coefficients, representing the residual block 205 in the transform domain.

[0149] Transform processing unit 206 can be used to apply an integer approximation of DCT / DST, such as the transform specified for H.265 / HEVC. This integer approximation is typically scaled by a certain factor compared to the orthogonal DCT transform. To maintain the norm of the residual block after both the forward and inverse transforms, other scaling factors are used as part of the transform process. These scaling factors are typically selected based on certain constraints, such as powers of 2 used for shift operations, the bit depth of the transform coefficients, and a trade-off between accuracy and implementation cost. For example, a specific scaling factor can be specified on the encoder 20 side via inverse transform processing unit 212 (and on the decoder 30 side via, for example, inverse transform processing unit 312) for the inverse transform, and correspondingly, a corresponding scaling factor can be specified on the encoder 20 side via transform processing unit 206 for the forward transform.

[0150] In one embodiment, the video encoder 20 (correspondingly, the transform processing unit 206) can be used to output transform parameters such as the type of one or more transforms, for example, directly outputting them or outputting them after being encoded or compressed by the entropy encoding unit 270, for example, so that the video decoder 30 can receive and use the transform parameters for decoding.

[0151] Quantification

[0152] Quantization unit 208 is used to quantize the transform coefficients 207 by, for example, scalar quantization or vector quantization, to obtain quantized transform coefficients 209. Quantized transform coefficients 209 can also be called quantized residual coefficients 209.

[0153] The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different scales can be applied to achieve finer or coarser quantization. Smaller quantization steps correspond to finer quantization, while larger quantization steps correspond to coarser quantization. The appropriate quantization step size can be indicated by the quantization parameter (QP). For example, the quantization parameter can be an index to a predefined set of appropriate quantization steps. For example, a smaller quantization parameter can correspond to fine quantization (smaller quantization step size), a larger quantization parameter can correspond to coarse quantization (larger quantization step size), and vice versa. Quantization may include division by the quantization step size, while corresponding or inverse dequantization performed by the dequantization unit 210, etc., may include multiplication by the quantization step size. Embodiments of some HEVC standards, for example, can be used to determine the quantization step size using the quantization parameter. In general, the quantization step size can be calculated using a fixed-point approximation of an equation involving division based on the quantization parameter. Additional scaling factors can be introduced for quantization and dequantization to recover the norm of the residual block, which may have been modified by the scaling used in the fixed-point approximation of the equations used for the quantization step size and quantization parameters. In one exemplary implementation, the scaling of the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where the loss increases with the quantization step size.

[0154] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) can be used to output the quantization parameter (QP), for example, directly outputting it or outputting it after being encoded or compressed by the entropy encoding unit 270, for example, so that the video decoder 30 can receive it and use the quantization parameter for decoding.

[0155] Inverse Quantization

[0156] The dequantization unit 210 is used to perform dequantization on the quantization coefficients by the quantization unit 208 to obtain the dequantization coefficients 211. For example, it performs a dequantization scheme based on or using the same quantization step size as the quantization unit 208 to perform the quantization scheme performed by the quantization unit 208. The dequantization coefficients 211 can also be called dequantization residual coefficients 211, corresponding to the transform coefficients 207. However, due to the loss caused by quantization, the dequantization coefficients 211 are usually not exactly the same as the transform coefficients.

[0157] Inverse Transformation

[0158] The inverse transform processing unit 212 is used to perform the inverse transform of the transform performed by the transform processing unit 206, such as the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST), to obtain the reconstructed residual block 213 (or the corresponding dequantization coefficients 213) in the pixel domain. The reconstructed residual block 213 may also be referred to as the transform block 213.

[0159] reconstruction

[0160] The reconstruction unit 214 (e.g., summer 214) is used to add the transform block 213 (i.e., the reconstruction residual block 213) to the prediction block 265 to obtain the reconstruction block 215 in the pixel domain, for example, by adding the pixel values ​​of the reconstruction residual block 213 and the pixel values ​​of the prediction block 265.

[0161] Filtering

[0162] Loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain the filtered block 221, or typically to filter the reconstructed pixels to obtain filtered pixel values. For example, the loop filter unit is used to smoothly perform pixel transformations or improve video quality. Loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, loop filter unit 220 may include a deblocking filter, a SAO filter, and an ALF filter. The filtering process may be performed in the order of deblocking filter, SAO filter, and ALF filter. As another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) may be added. This process is performed before deblocking. For example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 220 in... Figure 2 The loop filter is shown in the diagram, but in other configurations, the loop filter unit 220 can be implemented as a post-loop filter. The filter block 221 can also be called the filter reconstruction block 221.

[0163] In one embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) can be used to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, directly outputting or outputting after entropy encoding by the entropy encoding unit 270, for example, enabling the decoder 30 to receive and decode using the same or different loop filter parameters.

[0164] Decoding image buffer

[0165] The decoded picture buffer (DPB) 230 can be a reference picture memory that stores reference picture data for use by the video encoder 20 when encoding video data. The DPB 230 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer 230 can be used to store one or more filter blocks 221. The decoded picture buffer 230 can also be used to store other previous filter blocks of the same current image or different images, such as previously reconstructed images, such as previously reconstructed and filtered blocks 221, and can provide complete previously reconstructed i.e., decoded images (and corresponding reference blocks and pixels) and / or partially reconstructed current images (and corresponding reference blocks and pixels), for example, for inter-frame prediction. The decoded image buffer 230 can also be used to store one or more unfiltered reconstruction blocks 215, or generally store unfiltered reconstruction pixels, such as reconstruction blocks 215 that have not been filtered by the loop filter unit 220, or reconstruction blocks or reconstruction pixels that have not undergone any other processing.

[0166] Pattern selection (segmentation and prediction)

[0167] The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, for selecting data from the decoded image buffer 230 or other buffers (e.g., column buffers). Figure 2 (Not shown) receives or obtains raw image data such as raw block 203 (current block 203 of the current image 17) and reconstructed image data, for example, filtered and / or unfiltered reconstructed pixels or reconstructed blocks of the same (current) image and / or one or more previously decoded images. The reconstructed image data is used as reference image data for prediction such as inter-frame prediction or intra-frame prediction to obtain prediction block 265 or prediction value 265.

[0168] The mode selection unit 260 can be used to determine or select a segmentation for the current block (including no segmentation) and the prediction mode (e.g., intra-frame or inter-frame prediction mode) to generate the corresponding prediction block 265 for calculating the residual block 205 and reconstructing the reconstructed block 215.

[0169] In one embodiment, the mode selection unit 260 can be used to select a segmentation and prediction mode (e.g., from prediction modes supported or available by the mode selection unit 260), which provides the best match or minimum residual (minimum residual means better compression in transmission or storage), or provides minimum signaling overhead (minimum signaling overhead means better compression in transmission or storage), or considers or balances both. The mode selection unit 260 can be used to determine the segmentation and prediction mode based on rate distortion optimization (RDO), i.e., selecting the prediction mode that provides minimum RDO optimization. The terms "best," "lowest," and "optimal" in this document do not necessarily refer to "best," "lowest," or "optimal" overall, but can also refer to situations that meet termination or selection criteria. For example, values ​​exceeding or falling below a threshold or other limitations may lead to a "suboptimal choice," but reduce complexity and processing time.

[0170] In other words, segmentation unit 262 can be used to segment images in a video sequence into a sequence of coding tree units (CTUs), CTUs 203 can be further segmented into smaller block portions or sub-blocks (forming blocks again), for example, by iteratively using quad-tree partitioning (QT), binary-tree partitioning (BT), or triple-tree partitioning (TT), or any combination thereof, and is used to perform prediction, for example, on each of the block portions or sub-blocks, wherein mode selection includes selecting the tree structure of the segmented block 203 and selecting the prediction mode applied to each of the block portions or sub-blocks.

[0171] The segmentation (e.g., performed by segmentation unit 262) and prediction processing (e.g., performed by inter-frame prediction unit 244 and intra-frame prediction unit 254) performed by video encoder 20 will be described in detail below.

[0172] segmentation

[0173] Segmentation unit 262 can divide (or divide) an image block (or CTU) 203 into smaller parts, such as small blocks of square or rectangular shape. For an image with a three-pixel array, a CTU consists of N×N luma pixel blocks and two corresponding chroma pixel blocks. The maximum allowed size of the luma blocks in a CTU is specified as 128×128 in the developing Versatile Video Coding (VVC) standard, but may be specified as a value different from 128×128 in the future, such as 256×256. Image CTUs can be grouped / set into slices / coded block groups, coded blocks, or bricks. A coded block covers a rectangular area of ​​an image, and a coded block can be divided into one or more bricks. A brick consists of multiple rows of CTUs within a coded block. A coded block that is not divided into multiple bricks can be called a brick. However, a brick is a true subset of a coded block and therefore is not called a coded block. VVC supports two coded block group modes: raster scan slice / coded block group mode and rectangular slice mode. In raster scan coded block group mode, a slice / coded block group contains a sequence of coded blocks in a raster scan of an image. In rectangular slice mode, a slice contains multiple bricks of an image, which together form a rectangular region of the image. The bricks within a rectangular slice are arranged in the raster scan order of the slice's bricks. These smaller blocks (also called sub-blocks) can be further subdivided into even smaller parts. This is also called tree segmentation or hierarchical tree segmentation, where the root block at root tree level 0 (level 0, depth 0), etc., can be recursively segmented into two or more blocks at the next lower tree level, such as nodes at tree level 1 (level 1, depth 1). These blocks can then be further segmented into two or more blocks at the next lower level, such as tree level 2 (level 2, depth 2), etc., until the segmentation ends (because the termination criteria are met, such as reaching the maximum tree depth or minimum block size). Blocks that are not further segmented are also called leaf blocks or leaf nodes of the tree. A tree divided into two parts is called a binary tree (BT), a tree divided into three parts is called a ternary tree (TT), and a tree divided into four parts is called a quad tree (QT).

[0174] For example, a coding unit (CTU) can be or include a CTB of luma pixels, two corresponding CTBs of chroma pixels in an image with a three-pixel array, a CTB of pixels in a monochrome image, or a CTB of pixels in an image encoded using three independent color planes and a syntax structure (for encoding pixels). Correspondingly, a coding block (CTB) can be an N×N pixel block, where N can be set to a value such that the components are divided into CTBs; this is called segmentation. Similarly, a coding unit (CU) can be or include a coding block of luma pixels, two corresponding coding blocks of chroma pixels in an image with a three-pixel array, a coding block of pixels in a monochrome image, or a coding block of pixels in an image encoded using three independent color planes and a syntax structure (for encoding pixels). Correspondingly, a coding block (CB) can be an M×N pixel block, where M and N can be set to a value such that the CTB is divided into coding blocks; this is called segmentation.

[0175] For example, in one embodiment, HEVC can divide a Coding Tree Unit (CTU) into multiple Coding Units (CUs) using a quadtree structure represented as a coding tree. At the leaf CU level, a decision is made as to whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode image regions. Each leaf CU can be further divided into one, two, or four PUs based on the PU partitioning type. The same prediction process is used within a PU, and relevant information is transmitted to the decoder on a PU-by-PU basis. After obtaining residual blocks by applying the prediction process according to the PU partitioning type, the leaf CU can be segmented into Transform Units (TUs) according to another quadtree structure similar to the coding tree used for CUs.

[0176] For example, in one embodiment, according to the latest video coding standard currently under development (called Universal Video Coding (VVC), a combined quadtree of nested multi-type trees (e.g., binary and ternary trees) is used to partition the segmented structure for dividing the coding tree unit. Within the coding tree structure of a coding tree unit, the CU can be a square or a rectangle. For example, a coding tree unit (CTU) is first partitioned by a quadtree structure. The leaf nodes of the quadtree are further partitioned by multi-type tree structures. There are four partitioning types for the multi-type tree structure: vertical binary tree partitioning (SPLIT_BT_VER), horizontal binary tree partitioning (SPLIT_BT_HOR), vertical ternary tree partitioning, and vertical binary tree partitioning. SPLIT_TT_VER and horizontal ternary tree partitioning (SPLIT_TT_HOR). Multi-type leaf nodes are called coding units (CUs). Unless the CU is too large for the maximum transform length, such segmentation is used for prediction and transform processing without any further partitioning. In most cases, this means that the CU, PU, ​​and TU have the same block size in the quadtree-nested multi-type tree coding block structure. This anomaly occurs when the maximum supported transform length is less than the width or height of the color component of the CU. VVC specifies a unique signaling mechanism for partitioning information in a coding structure with quadtree-nested multi-type trees. In the signaling mechanism, coding... The tree unit (CTU), serving as the root of the quadtree, is first segmented by the quadtree structure. Then, each quadtree leaf node (if large enough) is further segmented into a multi-type tree structure. In the multi-type tree structure, a first flag (mtt_split_cu_flag) indicates whether a node is further segmented. When a node is further segmented, a second flag (mtt_split_cu_vertical_flag) indicates the direction of the segmentation, and a third flag (mtt_split_cu_binary_flag) indicates whether the segmentation is a binary or ternary tree. This is determined by mtt_split_c... The values ​​of `u_vertical_flag` and `mtt_split_cu_binary_flag` allow the decoder to deduce the multi-type tree partitioning mode (MttSplitMode) of the CU based on predefined rules or tables. It's important to note that for certain designs, such as a 64×64 luma block and a 32×32 chroma pipeline design in a VVC hardware decoder, TT partitioning is not allowed when the width or height of the luma encoding block exceeds 64. Similarly, TT partitioning is not allowed when the width or height of the chroma encoding block exceeds 32. The pipeline design divides the image into multiple virtual pipeline data units (VPDUs), each defined as a non-overlapping unit in the image. In the hardware decoder, consecutive VPDUs are processed simultaneously in multiple pipeline stages. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, thus requiring small VPDU sizes.In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, the partitioning of the ternary tree (TT) and binary tree (BT) can increase the VPDU size.

[0177] Additionally, it should be noted that when a portion of a tree node block extends beyond the bottom or right edge of the image, the tree node block is forcibly divided until all pixels of each encoded CU are within the image boundary.

[0178] For example, the intra-sub-partitions (ISP) tool can divide the luminance intra-prediction block vertically or horizontally into two or four sub-parts based on the block size.

[0179] In one example, the mode selection unit 260 of the video encoder 20 can be used to perform any combination of the segmentation techniques described above.

[0180] As described above, the video encoder 20 is used to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, intra-frame prediction modes and / or inter-frame prediction modes.

[0181] Intra-frame prediction

[0182] The intra-prediction mode set can include 35 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in HEVC, or it can include 67 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in VVC. For example, several conventional angular intra-prediction modes are adaptively replaced with wide-angle intra-prediction modes for non-square blocks as defined in VVC. As another example, to avoid division operations in DC prediction, only the longer side is used to calculate the average value of non-square blocks. Furthermore, the intra-prediction results of planar mode can be modified using the position-dependent intra-prediction combination (PDPC) method.

[0183] Intra-prediction unit 254 is used to generate intra-prediction block 265 using reconstructed pixels of adjacent blocks of the same current image according to the intra-prediction mode in the intra-prediction mode set.

[0184] Intra-prediction unit 254 (or typically mode selection unit 260) is also used to output intra-prediction parameters (or typically information indicating the selected intra-prediction mode of the block) to entropy coding unit 270 in the form of syntax element 266 to be included in encoded image data 21, so that video decoder 30 can perform operations such as receiving and using the prediction parameters for decoding.

[0185] HEVC's intra-frame prediction modes include DC prediction mode, planar prediction mode, and 33 angle prediction modes, totaling 35 candidate prediction modes. The current block can use pixels from the reconstructed image blocks to its left and above as references for intra-frame prediction. The image blocks in the surrounding region used for intra-frame prediction of the current block are called reference blocks, and the pixels in the reference blocks are called reference pixels. Among the 35 candidate prediction modes, DC prediction mode is suitable for regions with flat textures in the current block, where all pixels use the average value of the reference pixels in the reference block as the prediction; planar prediction mode is suitable for image blocks with smoothly varying textures, where the current block uses bilinear interpolation of the reference pixels in the reference block to predict all pixels in the current block; angle prediction mode utilizes the high correlation between the texture of the current block and the textures of adjacent reconstructed image blocks, copying the values ​​of the reference pixels in the corresponding reference block along a certain angle as the prediction for all pixels in the current block.

[0186] The HEVC encoder selects an optimal intra-prediction mode from 35 candidate prediction modes for the current block and writes this optimal intra-prediction mode into the video stream. To improve the coding efficiency of intra-prediction, the encoder / decoder derives three most probable modes from the optimal intra-prediction modes of the reconstructed image blocks in the surrounding regions that use intra-prediction. If the optimal intra-prediction mode selected for the current block is one of these three most probable modes, a first index is encoded to indicate that the selected optimal intra-prediction mode is one of these three most probable modes; if the selected optimal intra-prediction mode is not one of these three most probable modes, a second index is encoded to indicate that the selected optimal intra-prediction mode is one of the other 32 modes (other modes from the 35 candidate prediction modes besides the aforementioned three most probable modes). The HEVC standard uses a 5-bit fixed-length code as the aforementioned second index.

[0187] The HEVC encoder derives the three most likely modes by: selecting the best intra-prediction modes of the left and top adjacent image blocks of the current block and adding them to the set; if these two best intra-prediction modes are the same, only one needs to be kept in the set. If these two best intra-prediction modes are the same and both are angle prediction modes, then two angle prediction modes adjacent to that angle direction are selected and added to the set; otherwise, the planar prediction mode, DC mode, and vertical prediction mode are selected and added to the set in sequence until the number of modes in the set reaches three.

[0188] After performing entropy decoding on the bitstream, the HEVC decoder obtains the mode information of the current block. This mode information includes an indicator that indicates whether the optimal intra-prediction mode of the current block is among the three most likely modes, as well as the index of the optimal intra-prediction mode of the current block among the three most likely modes or the index of the optimal intra-prediction mode of the current block among the other 32 modes.

[0189] Inter-frame prediction

[0190] In a possible implementation, the set of inter-frame prediction modes depends on the available reference image (i.e., at least part of the previously decoded image stored in the DBP 230 as described above) and other inter-frame prediction parameters, such as whether to use the entire reference image or only a part of the reference image, such as a search window region near the current block, to search for the best matching reference block, and / or, for example, whether to perform pixel interpolation of half-pixel, quarter-pixel, and / or 1 / 16th interpolation.

[0191] In addition to the prediction modes mentioned above, skip mode and / or direct mode can also be used.

[0192] For example, in extended merge prediction, the merge candidate list for this mode consists of five candidate types in sequence: spatial MVP from spatially adjacent CUs, temporal MVP from co-located CUs, history-based MVP from a FIFO table, pairwise averaged MVP, and zero MV. Decoder-side motion vector refinement (DMVR) based on bilateral matching can be used to increase the accuracy of the merge mode's MV. Mergemode with MVD (MMVD) comes from merge modes with motion vector differences. The MMVD flag is sent immediately after the skip flag and merge flag to specify whether the CU uses MMVD mode. The CU-level adaptive motion vector resolution (AMVR) scheme can be used. AMVR supports encoding the CU's MVD with different precisions. The MVD of the current CU is adaptively selected based on its prediction mode. When the CU is encoding in merge mode, the combined inter / intra prediction (CIIP) mode can be applied to the current CU. The CIIP prediction is obtained by weighted averaging of the inter and intra prediction signals. For affine motion compensation prediction, the affine motion field of the block is described using motion information from motion vectors at 2 control points (4 parameters) or 3 control points (6 parameters). Subblock-based temporal motion vector prediction (SbTMVP) is similar to temporal motion vector prediction (TMVP) in HEVC, but it predicts the motion vectors of sub-CUs within the current CU. Bidirectional optical flow (BDOF), formerly known as BIO, is a simplified version that reduces computation, particularly in terms of the number of multiplications and the size of the multipliers. In the triangular partitioning mode, the CU is uniformly divided into two triangular parts using both diagonal and anti-diagonal partitioning. Furthermore, the bidirectional prediction mode extends the simple averaging method to support weighted averaging of the two predicted signals.

[0193] Inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both in... Figure 2(Not shown in the image). The motion estimation unit can be used to receive or acquire image block 203 (current image block 203 of current image 17) and decoded image 231, or at least one or more previously reconstructed blocks, such as one or more other / different previously decoded image blocks 231, to perform motion estimation. For example, the video sequence may include the current image and the previously decoded image 231, or in other words, the current image and the previously decoded image 231 may be part of or form the image sequence that forms the video sequence.

[0194] For example, encoder 20 can be used to select a reference block from multiple reference blocks of the same or different images in multiple other images, and provide the offset (spatial offset) between the position (x, y coordinates) of the reference image (or reference image index) and / or the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. This offset is also called a motion vector (MV).

[0195] The motion compensation unit is used to acquire, for example, receive, inter-frame prediction parameters, and perform inter-frame prediction based on or using these parameters to obtain inter-frame prediction blocks 246. Motion compensation performed by the motion compensation unit may include extracting or generating prediction blocks based on motion / block vectors determined by motion estimation, and may also include performing interpolation with sub-pixel precision. Interpolation filtering can generate pixels of other pixels from pixels of known pixels, thereby potentially increasing the number of candidate prediction blocks available for encoding image blocks. Once the motion vector corresponding to the PU of the current image block is received, the motion compensation unit can locate the prediction block pointed to by the motion vector in one of the reference image lists.

[0196] The motion compensation unit can also generate syntax elements associated with blocks and video slices for use by the video decoder 30 when decoding image blocks of the video slices. Alternatively, or as an alternative to slices and corresponding syntax elements, coded block groups and / or coded blocks and their corresponding syntax elements can be generated or used.

[0197] In the process of obtaining the candidate motion vector list in the advanced motion vector prediction (AMVP) mode, the motion vectors (MVs) that can be added to the candidate motion vector list as candidates include the MVs of spatially and temporally adjacent image blocks of the current block. The MVs of spatially adjacent image blocks can further include the MVs of the candidate image blocks to the left of the current block and the MVs of the candidate image blocks above the current block. For an example, please refer to... Figure 4 , Figure 4 An exemplary schematic diagram of a candidate image block provided in an embodiment of this application, such as... Figure 4As shown, the set of candidate image blocks to the left includes {A0, A1}, the set of candidate image blocks above includes {B0, B1, B2}, and the set of temporally adjacent candidate image blocks includes {C, T}. All three sets can be added to the candidate motion vector list as candidates. However, according to existing coding standards, the maximum length of the candidate motion vector list for AMVP is 2. Therefore, it is necessary to determine the MV (Motion Vector Model) to be added to the candidate motion vector list from the three sets according to a specified order. This order can be: first, consider the set of candidate image blocks to the left of the current block {A0, A1} (consider A0 first, then A1 if A0 is unavailable); second, consider the set of candidate image blocks above the current block {B0, B1, B2} (consider B0 first, then B1 if B0 is unavailable, then B2 if B1 is unavailable); finally, consider the set of temporally adjacent candidate image blocks {C, T} of the current block (consider T first, then C if T is unavailable).

[0198] After obtaining the candidate motion vector list, the optimal motion vector prediction (MV) is determined from the list using the rate distortion cost (RDcost). The candidate motion vector with the lowest RDcost is then used as the motion vector predictor (MVP) for the current block. The rate distortion cost is calculated using the following formula:

[0199] J = SAD + λR

[0200] Where J represents the RD cost, SAD is the sum of absolute differences (SAD) between the pixel values ​​of the predicted block obtained after motion estimation using candidate motion vectors and the pixel values ​​of the current block, R represents the bit rate, and λ represents the Lagrange multiplier.

[0201] The encoder passes the index of the determined MVP in the candidate motion vector list to the decoder. Further, a motion search can be performed within the neighborhood centered on the MVP to obtain the actual motion vector of the current block. The encoder calculates the motion vector difference (MVD) between the MVP and the actual motion vector and also passes the MVD to the decoder. The decoder parses the index, finds the corresponding MVP in the candidate motion vector list based on the index, parses the MVD, and adds the MVD to the MVP to obtain the actual motion vector of the current block.

[0202] During the process of obtaining the candidate motion information list in the merging mode, the motion information that can be added to the candidate motion information list includes the motion information of spatially adjacent or temporally adjacent image blocks of the current block. The spatially adjacent and temporally adjacent image blocks can be referenced... Figure 4 The candidate motion information list corresponds to spatially adjacent blocks (A0, A1, B0, B1, and B2) in the spatial domain. If a spatially adjacent block is unavailable or is intra-frame prediction, its motion information is not added to the candidate motion information list. The temporal candidate motion information for the current block is obtained by scaling the MV of the corresponding block in the reference frame based on the picture order count (POC) of the reference frame and the current frame. First, it is determined whether the block at position T in the reference frame is available; if not, the block at position C is selected. After obtaining the above candidate motion information list, the optimal motion information is determined from the candidate motion information list using the RD cost as the motion information for the current block. The encoder transmits the index value (denoted as mergeindex) of the optimal motion information in the candidate motion information list to the decoder.

[0203] Entropy coding

[0204] Entropy coding unit 270 is used to apply entropy coding algorithms or schemes (e.g., variable length coding (VLC), context adaptive VLC (CALVC), arithmetic coding schemes, binarization algorithms, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to quantization residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements, to obtain encoded image data 21 that can be output as an encoded bitstream 21 through output terminal 272, so that video decoder 30 and the like can receive and use the parameters for decoding. The encoded bitstream 21 can be transmitted to video decoder 30, or stored in memory for later transmission or retrieval by video decoder 30.

[0205] Other architectural variations of the video encoder 20 can be used to encode the video stream. For example, a non-transform-based encoder 20 can directly quantize the residual signal in certain blocks or frames without the transform processing unit 206. In another implementation, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0206] Decoder and Decoding Method

[0207] like Figure 3 As shown, the video decoder 30 is used to receive, for example, encoded image data 21 (e.g., encoded bitstream 21) encoded by the encoder 20, to obtain a decoded image 331. The encoded image data or bitstream includes information for decoding the encoded image data, such as data representing image blocks (and / or groups or blocks of encoded video segments) and associated syntax elements.

[0208] exist Figure 3 In the example, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded image buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. The inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same functions as the referenced unit. Figure 2 The video encoder 100 describes the encoding process as the opposite of the decoding process.

[0209] As described in encoder 20, the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded image buffer DPB 230, inter-frame prediction unit 344, and intra-frame prediction unit 354 also constitute the "built-in decoder" of video encoder 20. Correspondingly, inverse quantization unit 310 can be functionally identical to inverse quantization unit 110, inverse transform processing unit 312 can be functionally identical to inverse transform processing unit 122, reconstruction unit 314 can be functionally identical to reconstruction unit 214, loop filter 320 can be functionally identical to loop filter 220, and decoded image buffer 330 can be functionally identical to decoded image buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 is correspondingly applicable to the corresponding units and functions of video decoder 30.

[0210] Entropy Decoding

[0211] Entropy decoding unit 304 is used to parse bitstream 21 (or generally encoded image data 21) and perform entropy decoding on encoded image data 21 to obtain quantization coefficients 309 and / or decoded encoded parameters. Figure 3(Not shown in the image) Examples of parameters include inter-frame prediction parameters (e.g., reference image index and motion vector), intra-frame prediction parameters (e.g., intra-frame prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 can be used to apply the decoding algorithm or scheme corresponding to the encoding scheme of the entropy coding unit 270 of the encoder 20. The entropy decoding unit 304 can also be used to provide inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 can receive syntax elements at the video slice and / or video block level. Furthermore, or as an alternative to slices and corresponding syntax elements, it can receive or use coded block groups and / or coded blocks and corresponding syntax elements.

[0212] Inverse Quantization

[0213] The dequantization unit 310 can be used to receive quantization parameters (QP) (or generally information related to dequantization) and quantization coefficients from encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and dequantize the decoded quantization coefficients 309 based on the quantization parameters to obtain dequantization coefficients 311, which may also be referred to as transform coefficients 311. The dequantization process may include using the quantization parameters calculated by the video encoder 20 for each video block in the video slice to determine the degree of quantization, and also to determine the degree of dequantization to be performed.

[0214] Inverse Transformation

[0215] The inverse transform processing unit 312 can be used to receive the dequantized coefficients 311, also known as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain the reconstructed residual block 213 in the pixel domain. The reconstructed residual block 213 can also be called transform block 313. The transform can be an inverse transform, such as inverse DCT, inverse DST, inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 can also be used to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304) to determine the transform applied to the dequantized coefficients 311.

[0216] reconstruction

[0217] The reconstruction unit 314 (e.g., summer 314) is used to add the reconstruction residual block 313 to the prediction block 365 to obtain the reconstruction block 315 in the pixel domain, for example, by adding the pixel values ​​of the reconstruction residual block 313 and the pixel values ​​of the prediction block 365.

[0218] Filtering

[0219] Loop filter unit 320 (in or after the encoding loop) is used to filter the reconstructed block 315 to obtain filtered block 321, thereby facilitating pixel transformation or improving video quality. Loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, loop filter unit 320 may include a deblocking filter, a SAO filter, and an ALF filter. The filtering process may be performed in the order of deblocking filter, SAO filter, and ALF filter. As another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) may be added. This process is performed before deblocking. For example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 320 in... Figure 3 The loop filter is shown in the diagram, but in other configurations, the loop filter unit 320 can be implemented as a post-loop filter.

[0220] Decoding image buffer

[0221] The decoded video block 321 in one image is then stored in the decoded image buffer 330, which stores the decoded image 331 as a reference image. The reference image is used for subsequent motion compensation for other images and / or output displays respectively.

[0222] The decoder 30 is used to output the decoded image 311 through the output terminal 312, etc., for display to the user or for the user to view.

[0223] predict

[0224] Inter-frame prediction unit 344 is functionally identical to inter-frame prediction unit 244 (especially motion compensation unit), and intra-frame prediction unit 354 is functionally identical to inter-frame prediction unit 254. It determines segmentation or partitioning and performs prediction based on segmentation and / or prediction parameters or corresponding information received from coded image data 21 (e.g., parsed and / or decoded by entropy decoding unit 304). Pattern application unit 360 can be used to perform prediction (intra-frame or inter-frame prediction) for each block based on the reconstructed image, block, or corresponding pixel (filtered or unfiltered), resulting in prediction block 365.

[0225] When a video slice is encoded as an intra-coded (I) slice, the intra-prediction unit 354 in the mode application unit 360 generates a prediction block 365 for the current video slice based on the indicated intra-prediction mode and data from the previous decoded block of the current image. When a video image is encoded as an inter-coded (i.e., B or P) slice, the inter-prediction unit 344 (e.g., a motion compensation unit) in the mode application unit 360 generates a prediction block 365 for the current video slice based on motion vectors and other syntax elements received from the entropy decoding unit 304. For inter-prediction, these prediction blocks can be generated from one of the reference images in one of the reference image lists. The video decoder 30 can construct reference frame lists 0 and 1 using a default construction technique based on the reference images stored in the DPB 330. In addition to slices (e.g., video slices) or as a substitute for slices, the same or similar processes can be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), such as video can be encoded using I, P, or B coding block groups and / or coding blocks.

[0226] The pattern application unit 360 is used to determine prediction information for video blocks in the current video slice by parsing motion vectors and other syntax elements, and to generate prediction blocks for the current video slice being decoded using the prediction information. For example, the pattern application unit 360 uses some received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction), inter-frame prediction slice type (e.g., B-slice, P-slice, or GPB-slice), construction information for one or more reference image lists for the slice, motion vectors for each inter-frame coded video block in the slice, inter-frame prediction state for each inter-frame coded video block in the slice, and other information to decode video blocks within the current video slice. In addition to slices (e.g., video slices) or as alternatives to slices, the same or similar process can be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), for example, videos can be encoded using I, P, or B coding block groups and / or coding blocks.

[0227] In one embodiment, Figure 3The video encoder 30 can also be used to segment and / or decode images using slices (also called video slices), where images can be segmented or decoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., coded blocks in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0228] In one embodiment, Figure 3 The video decoder 30 shown can also be used to segment and / or decode an image using slice / coded block groups (also known as video coded block groups) and / or coded blocks (also known as video coded blocks), wherein the image can be segmented or decoded using one or more slice / coded block groups (typically non-overlapping), each slice / coded block group may include one or more blocks (e.g., CTUs) or one or more coded blocks, wherein each coded block may be rectangular or the like, and may include one or more complete or partial blocks (e.g., CTUs).

[0229] Other variations of the video decoder 30 can be used to decode the encoded image data 21. For example, the decoder 30 can generate an output video stream without the loop filter unit 320. For example, the non-transform-based decoder 30 can directly dequantize the residual signal in certain blocks or frames without the inverse transform processing unit 312. In another implementation, the video decoder 30 may have a dequantization unit 310 and an inverse transform processing unit 312 combined into a single unit.

[0230] It should be understood that in encoder 20 and decoder 30, the processing result of the current step can be further processed and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, further operations can be performed on the processing result of interpolation filtering, motion vector derivation, or loop filtering, such as clipping or shifting operations.

[0231] It should be noted that further calculations can be performed on the derived motion vector of the current block (including but not limited to control point motion vectors in affine mode, affine, planar, sub-block motion vectors in ATMVP mode, time motion vectors, etc.). For example, the value of the motion vector can be restricted to a predefined range based on the representation bits of the motion vector. If the representation bits of the motion vector are bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents exponentiation. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of four 4×4 sub-blocks in an 8×8 block) is restricted such that the maximum difference between the integer parts of the MV of the four 4×4 sub-blocks does not exceed N pixels, for example, not more than 1 pixel. Two methods for restricting motion vectors based on bitDepth are provided here.

[0232] Although the above embodiments primarily describe video encoding and decoding, it should be noted that embodiments of the decoding system 10, encoder 20, and decoder 30, as well as other embodiments described herein, can also be used for still image processing or encoding and decoding, i.e., the processing or encoding and decoding of a single image independent of any previous or consecutive images in video encoding and decoding. Generally, if image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and inter-frame prediction unit 344 (decoder) may not be available. All other functions (also referred to as tools or techniques) of the video encoder 20 and video decoder 30 can also be used for still image processing, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy coding 270, and entropy decoding 304.

[0233] Please refer to Figure 5 , Figure 5 This is an exemplary block diagram of a video decoding device 500 provided for embodiments of this application. The video decoding device 500 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 500 may be a decoder, such as... Figure 1a The video decoder 30 in the text can also be an encoder, for example... Figure 1a The video encoder 20 in the middle.

[0234] The video decoding device 500 includes: an input port 510 (or input port 510) and a receiver unit (Rx) 520 for receiving data; a processor, logic unit, or central processing unit (CPU) 530 for processing data; for example, the processor 530 may be a neural network processor 530; a transmitter unit (Tx) 540 and an output port 550 (or output port 550) for transmitting data; and a memory 560 for storing data. The video decoding device 500 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the input port 510, receiver unit 520, transmitter unit 540, and output port 550 for the entry or exit of optical or electrical signals.

[0235] Processor 530 is implemented in both hardware and software. Processor 530 may be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 530 communicates with ingress port 510, receiver unit 520, transmitter unit 540, egress port 550, and memory 560. Processor 530 includes a decoding module 570 (e.g., a neural network-based decoding module 570). Decoding module 570 implements the embodiments disclosed above. For example, decoding module 570 performs, processes, prepares, or provides various encoding operations. Therefore, decoding module 570 provides a substantial improvement to the functionality of video decoding device 500 and affects the switching of video decoding device 500 to different states. Alternatively, decoding module 570 may be implemented with instructions stored in memory 560 and executed by processor 530.

[0236] Memory 560 includes one or more disks, tape drives, and solid-state drives, which can be used as overflow data storage devices to store such programs when an executable program is selected, and to store instructions and data read during program execution. Memory 560 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0237] Please refer to Figure 6 , Figure 6This is an exemplary block diagram of the device 600 provided in the embodiments of this application. The device 600 can be used as... Figure 1a The source device 12 and the destination device 14 are either or both of them.

[0238] The processor 602 in device 600 may be a central processing unit. Alternatively, processor 602 may be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. While the disclosed implementation may be implemented using a single processor, such as processor 602 as shown, using more than one processor is faster and more efficient.

[0239] In one implementation, the memory 604 in device 600 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 604. Memory 604 may include code and data 606 accessed by processor 602 via bus 612. Memory 604 may also include an operating system 608 and an application program 610, which includes at least one program that allows processor 602 to perform the methods described herein. For example, application program 610 may include applications 1 to N, and also includes a video decoding application that performs the methods described herein.

[0240] The device 600 may also include one or more output devices, such as a display 618. In one example, the display 618 may be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 618 may be coupled to the processor 602 via a bus 612.

[0241] Although bus 612 in device 600 is described herein as a single bus, bus 612 may include multiple buses. Furthermore, auxiliary storage may be directly coupled to other components of device 600 or accessed via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, device 600 can have a wide variety of configurations.

[0242] In compression scenarios, the input image can be divided into multiple strips, and each strip is then encoded. The resolution of the input image and the number of strips are often determined based on product requirements; therefore, the image resolution may not be sufficient to meet the required number of strips. To ensure that the encoder and decoder function correctly under such input conditions, the input image needs to be padded before encoding to allow it to be divided into the required integer number of strips. The padded image is then encoded and transmitted, and the bitstream is decoded by the decoder to obtain the reconstructed image. Finally, the padded areas are cropped from the reconstructed image to restore the original resolution.

[0243] In related technologies, the subjective quality of different stripes in the filled image may vary, resulting in uneven subjective quality of the filled image.

[0244] Therefore, embodiments of this application provide an encoding method that enables the subjective quality of the infilled image to be balanced. This encoding method is applicable to encoding systems. Figure 7 This illustrates one possible form of the encoding system.

[0245] like Figure 7 The encoding system shown includes: an image filling module 701, a segmentation module 702, a prediction module 703, a quantization module 704, an entropy coding module 705, and a bit rate control module 706.

[0246] The image to be encoded input into the encoding system undergoes image inpainting. The inpainted image is then divided into encoding units (also called encoding blocks or image blocks). After being input into the encoder, the image passes through encoding modules such as prediction, quantization, and entropy coding. Finally, the bit stream corresponding to the encoding unit is output. The bit streams of all encoding units are concatenated to obtain the bit stream corresponding to the entire image.

[0247] Image filling module 701 is used to fill in the input image to be encoded.

[0248] The partitioning module 702 is used to divide the padded image into coding units.

[0249] The prediction module 703 is used to predict the coding unit.

[0250] The quantization module 704 is used to quantize the coding unit according to the quantization parameters.

[0251] The entropy coding module 705 is used to perform entropy coding on the coding unit. In addition, the entropy coding module 705 can also obtain the number of coded bits based on the coding unit.

[0252] In one possible implementation, the entropy coding module 705 can operate according to the set code length Bpp. pad Fixed-length encoding is performed on the fully filled coding units.

[0253] The bitrate control module 706 is used to adjust the output bitrate. For example, the bitrate control module 706 can adjust the output bitrate based on the number of encoded bits, the number of padding bits, and the image content.

[0254] like Figure 8 As shown, the above encoding system may further include: an adjustment unit 707.

[0255] The adjustment unit 707 is used to adjust the compression performance of the coding unit. For example, the adjustment unit 707 can adjust the compression performance of the coding unit by adjusting the bit stuffing number.

[0256] Figure 9 An encoding method provided by an embodiment of this application is illustrated, such as... Figure 9 The method shown includes:

[0257] S901. Perform image filling on the image to be encoded to obtain the filled image.

[0258] In one possible implementation, the image to be encoded can first be divided into multiple strips, and then N columns of samples can be filled in at the right boundary of the last strip to obtain a filled image. N is a positive integer. Here, a sample can be understood as a pixel of the image, and a strip is composed of multiple samples.

[0259] Understandably, compared to performing image filling on every single strip, performing image filling only on the last column of the image can reduce the hardware implementation cost and power consumption of image filling.

[0260] In one possible implementation, N columns of samples can be padded at the right boundary of the last column of the image to be encoded, and M columns of samples can be padded at the bottom boundary of the last row of the image to obtain the padded image. Here, N and M are both positive integers.

[0261] For example, such as Figure 10 As shown, the image to be encoded can first be divided into 12 strips. Assuming the total number of columns to be filled horizontally is N and the total number of columns to be filled vertically is M, then N columns of samples can be filled at the right boundary of the last strip to obtain the filled image. The filling method is to copy the most recent valid sample column from the left. Besides copying the most recent valid sample column, the filling method can also be default value filling (a pre-agreed value) filling, or copying valid sample columns from other positions. Then, M columns of samples are filled at the lower boundary of the last row of strips, using default value filling. Besides default value filling, the filling method can also be copying the valid sample row above.

[0262] S902. Obtain the coding unit based on the above-mentioned filled image.

[0263] The specific implementation of obtaining the coding unit based on the above-described incomplete image can be any method that can be conceived by those skilled in the art, and the embodiments of this application do not limit it. For example, each strip obtained from the incomplete image can be divided into one or more coding units for encoding.

[0264] S903. Determine the encoding information of the above encoding unit based on the full fill information of the above encoding unit.

[0265] The aforementioned full-filling information is used to indicate whether all samples in the aforementioned coding unit are image-filled samples. The aforementioned coding information includes at least one of the number of fill bits or the coding code length. Image-filled samples are samples obtained through image filling.

[0266] In one possible implementation, the aforementioned full-fill information is used to further indicate whether the aforementioned coding unit is located within the target strip of the aforementioned filled image, the target strip being the horizontal image filling strip.

[0267] In one possible implementation, the above full-fill information can be represented by fully filling the PadFlag flag.

[0268] For example, if all samples in the current coding unit are obtained through padding, the padding flag of the current coding unit is 1; otherwise, the padding flag of the current coding unit is 0.

[0269] That is, a complete filling flag of the coding unit being 1 indicates that all samples in the coding unit are image filling samples; a complete filling flag of the coding unit being 0 indicates that not all samples in the coding unit are image filling samples.

[0270] For example, if all samples in the current coding unit are obtained through padding, and the current coding unit is located in a strip where horizontal padding is performed, then the complete padding flag of the current coding unit is 1; otherwise, the complete padding flag of the current coding unit is 0.

[0271] That is, a fully filled flag of the coding unit being 1 indicates that all samples in the coding unit are image-filled samples and the coding unit is located within the horizontally filled image stripe; a fully filled flag of the coding unit being 0 indicates that not all samples in the coding unit are image-filled samples and / or the coding unit is not located within the horizontally filled image stripe.

[0272] In one possible implementation, when the above-mentioned coding unit is a target coding unit, the number of padding bits can be determined based on the actual number of bits of the above-mentioned coding unit and a first preset number of bits. The target coding unit is a coding unit in which all samples are image padding samples, and the actual number of bits is the number of bits in the bitstream to be filled corresponding to the above-mentioned coding unit.

[0273] For example, when the full padding information of the coding unit indicates whether all samples in the coding unit are image padding samples (i.e., the full padding flag of the coding unit is 1), the number of padding bits can be determined based on the actual number of bits of the coding unit and the first preset number of bits.

[0274] Optionally, the number of padding bits, BitsGap, can satisfy: BitsGap = Max(X0 - BCU, 0).

[0275] Wherein, BCU represents the bits of the bitstream to be filled corresponding to the coding unit (i.e., the actual number of coded bits obtained after encoding the current coding block), and X0 represents the first preset number of bits (i.e., the total number of bits of the bitstream corresponding to the coding unit when the full fill flag is 1). The value of X0 can be obtained by multiplying the original number of bits of the input coding unit by the target compression ratio, or it can be an adjustment of the value obtained above based on the header information overhead and coding margin.

[0276] As can be seen, in the method provided by the embodiments of this application, the encoding information of the encoding unit can be determined according to the full padding information of the encoding unit, and then the encoding unit is encoded to generate a bitstream using the encoding information. In this way, the fully padding encoding units in the strip can be determined by the full padding information, and then the padding bit number is determined according to the actual bit number of the encoding unit and the first preset bit number. Then, bits are padded in the initial bitstream (i.e. the bitstream to be filled) generated by the fully padding encoding units according to the padding bit number, so that the number of encoded bits increases, thereby reducing the number of encoded bits of the non-fully padding encoding units in the strip. This reduces the difference between the reconstruction quality of the non-padding content in the strip and the reconstruction quality of other strips, avoids the subjective quality unevenness of the padded image, and thus makes the subjective quality of the padded image more balanced.

[0277] It is worth mentioning that when the aforementioned coding unit is a non-target coding unit, the number of padding bits can be determined to be 0, that is, no padding is applied to the bitstream of the non-target coding unit. Other methods can also be used to determine the number of padding bits, and this application does not limit this approach.

[0278] In one possible implementation, when the above-mentioned coding unit is a target coding unit, the coding length can be determined based on the header information overhead of the above-mentioned coding unit, the second preset number of bits, and the number of samples of the above-mentioned coding unit, wherein the target coding unit is a coding unit in which all samples are image filling samples.

[0279] For example, when the full padding information of the coding unit indicates whether all samples in the coding unit are image padding samples (i.e., the full padding flag of the coding unit is 1), the coding code length can be determined based on the header information overhead of the coding unit, the second preset number of bits, and the number of samples in the coding unit.

[0280] Optional, code length Bpp pad It can satisfy: Bpp pad =(X1-X2) / Cu size .

[0281] Where X1 is the second preset number of bits (i.e., the total number of bits in the bitstream corresponding to the fully filled coding unit), X2 is the header information overhead of the current coding unit, and Cu size Number of samples for coding units

[0282] As can be seen, in the method provided in this application embodiment, the encoding information of the encoding unit can be determined based on the full-filling information of the encoding unit, and then the encoding unit is encoded to generate a bitstream using the encoding information. This allows for the determination of fully filled encoding units in a strip using the full-filling information. Then, the encoding code length is determined using the header information overhead of the encoding unit, the second preset number of bits, and the number of samples of the encoding unit. Finally, the encoding unit is encoded using a fixed-length encoding method to generate a bitstream, increasing the number of encoded bits. This reduces the number of encoded bits for incompletely filled encoding units within the strip, thereby reducing the difference in reconstruction quality between the non-filled content in the strip and other strips, avoiding uneven subjective quality in the filled image, and ultimately achieving balanced subjective quality in the filled image.

[0283] In one possible implementation, the target coding unit is a coding unit whose samples are all image filling samples and located within the target strip of the filled image.

[0284] It is worth mentioning that when the aforementioned coding unit is not a target coding unit, the coding length may be uncertain. Other methods can also be used to determine the coding length, and this application does not limit this approach.

[0285] S904. Encode the above-mentioned encoding units according to the above-mentioned encoding information to generate a code stream.

[0286] In one possible implementation, the above-mentioned encoding unit can be encoded to generate a bitstream to be filled, and bits can be filled into the bitstream to be filled according to the above-mentioned number of filling bits to obtain the bitstream.

[0287] For example, if the full padding information of the coding unit indicates whether all samples in the coding unit are image padding samples (i.e., the full padding flag of the coding unit is 1), then after the coding unit encodes the bitstream to be filled, additional padding bits BitsGap can be added to the bitstream to be filled. The padding bit content can be BitsGap zeros, BitsGap 1s, or other bit content; this embodiment does not limit this.

[0288] As can be seen, in the method provided in this application embodiment, the encoding information of the encoding unit can be determined based on the full padding information of the encoding unit, and then the encoding unit is encoded to generate a bitstream using the encoding information. In this way, the fully padding encoding units in the strip can be determined by the full padding information, and then bits are padded in the initial bitstream (i.e. the bitstream to be padded) generated by the fully padding encoding units to increase the number of encoded bits, thereby reducing the number of encoded bits of the non-fully padding encoding units in the strip. This reduces the difference between the reconstruction quality of the non-padding content in the strip and the reconstruction quality of other strips, avoids uneven subjective quality of the padded image, and thus makes the subjective quality of the padded image more balanced.

[0289] It is worth mentioning that when the above-mentioned coding unit is a non-target coding unit (i.e. the complete padding flag of the coding unit is 0), it is not necessary to pad the bit stream generated by the coding unit with extra bits, or other methods can be used to pad the bits. This application embodiment does not limit this.

[0290] In one possible implementation, the above-mentioned coding unit can be encoded with a fixed length according to the above-mentioned coding code length to generate the above-mentioned bitstream.

[0291] For example, if the full filling information of the coding unit indicates whether all samples in the coding unit are image filling samples (i.e., the full filling flag of the coding unit is 1), the coding unit can be encoded with a fixed length according to the coding code length Bpppad.

[0292] As can be seen, in the method provided in this application embodiment, the encoding information of the encoding unit can be determined based on the full padding information of the encoding unit, and then the encoding unit is encoded using the encoding information to generate a bitstream. This allows for the determination of fully padding encoding units in a strip by the full padding information, and then fixed-length encoding of the encoding units to generate a bitstream based on the encoding code length, increasing the number of encoded bits. This reduces the number of encoded bits for incompletely padding encoding units within the strip, thereby reducing the difference in reconstruction quality between the non-padding content in the strip and other strips, avoiding subjective quality inconsistencies in the padded image, and ultimately achieving balanced subjective quality in the padded image.

[0293] It is worth mentioning that when the above-mentioned coding unit is a non-target coding unit (i.e., the complete padding flag of the coding unit is 0), fixed-length coding can be used, or other coding methods can be used to encode it. This application embodiment does not limit this.

[0294] In related technologies, when encoding and decoding the padded image, if there is too much padded content in the strip and the padded content is simple, the number of coding bits occupied by the fully padded coding unit in the strip is less. Consequently, the number of coding bits used for the incompletely padded coding unit in the strip will be more abundant than other strips with no padded content or less padded content. This results in a difference in the reconstruction quality of the non-padded content in the strip compared to the reconstruction quality of other strips, causing uneven subjective quality of the padded image.

[0295] In the method provided in this application embodiment, the encoding information of the encoding unit can be determined based on the full padding information of the encoding unit, and then the encoding unit is encoded using the encoding information to generate a bitstream. This allows for the identification of fully padded encoding units within a strip using the full padding information. By adjusting the encoding parameters of the fully padded encoding units to increase their bit count, the bit count of the incompletely padded encoding units within the strip is reduced. This reduces the difference in reconstruction quality between the non-padded content in the strip and other strips, avoiding subjective quality inconsistencies in the padded image and achieving a more balanced subjective quality in the padded image.

[0296] This application provides a decoding method that enables the subjective quality of the filled-in image to be balanced. This decoding method is applicable to decoding systems. Figure 11 This illustrates one possible form of the decoding system.

[0297] like Figure 11 The decoding system shown includes: an entropy decoding module 1101, an inverse quantization module 1102, a prediction module 1103, an image cropping module 1104, and a bitrate control module 1105.

[0298] The bitstream corresponding to each coding unit is input into the decoder of the decoding system and undergoes decoding processes such as entropy decoding, dequantization, and prediction. The fully filled coding unit needs to be decoded according to the aforementioned coding code length (Bpppad). This yields the reconstructed image corresponding to that coding unit. The reconstructed image needs to be cropped before output to finally output an image at its original resolution. Cropping involves removing the filled region from the reconstructed image.

[0299] The entropy decoding module 1101 is used to perform entropy encoding on the bit stream corresponding to the encoding unit. Additionally, the image complexity can be obtained from the bit stream corresponding to the encoding unit.

[0300] The dequantization module 1102 is used to dequantize the bit stream (bit stream) corresponding to the encoding unit according to the quantization parameters.

[0301] The prediction module 1103 is used to predict the bit stream corresponding to the coding unit.

[0302] Image cropping module 1104 is used to crop the filled area in the reconstructed image.

[0303] The bitrate control module 1105 is used to adjust the output bitrate according to the number of encoded bits and / or image complexity.

[0304] Figure 12 This application illustrates a decoding method provided by an embodiment, such as... Figure 12 The method shown includes:

[0305] S1201, Obtain the bitstream.

[0306] The bitstream is generated by the encoding unit based on the encoding information of the encoding unit. The encoding information is determined based on the full padding information of the encoding unit. The full padding information is used to indicate whether all samples in the encoding unit are image padding samples. The encoding information includes at least one of the padding bit count or the encoding code length.

[0307] In one possible implementation, the full-fill information is used to further indicate whether the coding unit is located within the target strip of the filled image, wherein the target strip is the strip of the image filled laterally.

[0308] S1202. Decode the above bitstream to obtain the reconstructed block.

[0309] In one possible implementation, when the bitstream is the bitstream of the target coding unit, the bitstream can be decoded according to the coding code length to obtain the reconstructed block, where the target coding unit is a coding unit in which all samples are image filling samples.

[0310] In one possible implementation, the target coding unit is a coding unit whose samples are all image filling samples and located within the target strip of the filled image.

[0311] S1203. Generate a reconstructed image based on the above reconstruction blocks.

[0312] In related technologies, when encoding and decoding the padded image, if there is too much padded content in the strip and the padded content is simple, the number of coding bits occupied by the fully padded coding unit in the strip is less. Consequently, the number of coding bits used for the incompletely padded coding unit in the strip will be more abundant than other strips with no padded content or less padded content. This results in a difference in the reconstruction quality of the non-padded content in the strip compared to the reconstruction quality of other strips, causing uneven subjective quality of the padded image.

[0313] In the method provided in this application embodiment, the encoding information of the encoding unit can be determined based on the full padding information of the encoding unit, and then the encoding unit is encoded using the encoding information to generate a bitstream. This allows for the identification of fully padded encoding units within a strip using the full padding information. By adjusting the encoding parameters of the fully padded encoding units to increase their bit count, the bit count of the incompletely padded encoding units within the strip is reduced. This reduces the difference in reconstruction quality between the non-padded content in the strip and other strips, avoiding subjective quality inconsistencies in the padded image and achieving a more balanced subjective quality in the padded image.

[0314] The following will combine Figure 13 An encoding apparatus for performing the above encoding method is introduced.

[0315] It is understood that, in order to achieve the above-mentioned functions, the encoding device includes hardware and / or software modules corresponding to the execution of each function. Based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0316] This application embodiment can divide the encoding device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0317] When dividing each function into modules according to its corresponding function. Figure 13 A schematic diagram of a possible composition of the encoding device involved in the above embodiments is shown, such as... Figure 13 As shown, the encoding device 1300 may include: an image filling unit 1301, a division unit 1302, a determination unit 1303, and an encoding unit 1304.

[0318] The image filling unit 1301 described above is used to fill in the image to be encoded to obtain a filled image.

[0319] The aforementioned division unit 1302 is used to obtain the encoding unit based on the aforementioned filled image.

[0320] The determining unit 1303 is used to determine the encoding information of the encoding unit based on the full padding information of the encoding unit. The full padding information is used to indicate whether all samples in the encoding unit are image padding samples. The encoding information includes at least one of padding bit count or encoding code length.

[0321] The aforementioned encoding unit 1304 is used to encode the aforementioned encoding unit according to the aforementioned encoding information to generate a bitstream.

[0322] In one possible implementation, the encoding unit 1304 is specifically used to: encode the encoding unit to generate a code stream to be filled; and fill the code stream to be filled with bits according to the number of filling bits to obtain the code stream.

[0323] In one possible implementation, the encoding unit 1304 is specifically used to: perform fixed-length encoding on the encoding unit according to the encoding code length to generate the bitstream.

[0324] In one possible implementation, the determining unit 1303 is specifically used to: when the coding unit is a target coding unit, determine the number of padding bits based on the actual number of bits of the coding unit and a first preset number of bits, wherein the target coding unit is a coding unit in which all samples are image padding samples, and the actual number of bits is the number of bits of the bitstream to be filled corresponding to the coding unit.

[0325] In one possible implementation, the determining unit 1303 is specifically used to: determine the coding length based on the header information overhead, the second preset number of bits, and the number of samples of the coding unit when the coding unit is a target coding unit, wherein the target coding unit is a coding unit in which all samples are image filling samples.

[0326] In one possible implementation, the aforementioned full-fill information is used to further indicate whether the aforementioned coding unit is located within the target strip of the aforementioned filled image, the target strip being the horizontal image filling strip.

[0327] In one possible implementation, the target coding unit is a coding unit whose samples are all image filling samples and located within the target strip of the filled image.

[0328] The following will combine Figure 14 A decoding apparatus for performing the above decoding method is introduced.

[0329] It is understood that, in order to achieve the above-mentioned functions, the decoding device includes hardware and / or software modules that perform the respective functions. Based on the algorithm steps of the examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0330] This application embodiment can divide the decoding device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0331] When dividing each function into modules according to its corresponding function. Figure 14 A schematic diagram of a possible composition of the decoding device involved in the above embodiments is shown, such as... Figure 14 As shown, the decoding device 1400 may include a receiving unit 1401, a decoding unit 1402, and a reconstruction unit 1403.

[0332] The receiving unit 1401 is used to acquire a bitstream, which is a bitstream generated by the encoding unit according to the encoding information of the encoding unit. The encoding information is determined according to the full padding information of the encoding unit. The full padding information is used to indicate whether all samples in the encoding unit are image padding samples. The encoding information includes at least one of the padding bit count or the encoding code length.

[0333] The aforementioned decoding unit 1402 is used to decode the aforementioned bitstream to obtain a reconstructed block.

[0334] The reconstruction unit 1403 is used to generate a reconstructed image based on the reconstruction blocks.

[0335] In one possible implementation, the decoding unit 1402 is specifically used to: decode the bitstream according to the coding code length to obtain the reconstructed block when the bitstream is the bitstream of the target coding unit, wherein the target coding unit is a coding unit in which all samples are image filling samples.

[0336] In one possible implementation, the full-fill information is used to further indicate whether the coding unit is located within the target strip of the filled image, wherein the target strip is the strip of the image filled laterally.

[0337] In one possible implementation, the target coding unit is a coding unit whose samples are all image filling samples and located within the target strip of the filled image.

[0338] In one possible implementation, the reconstruction unit 1403 is further configured to: crop the image filling region in the reconstructed image.

[0339] This application also provides an encoding device, which includes at least one processor. When the at least one processor executes program code or instructions, it implements the above-described related method steps to achieve the encoding method in the above embodiments.

[0340] Optionally, the device may further include at least one memory for storing the program code or instructions.

[0341] This application also provides a decoding device, which includes at least one processor. When the at least one processor executes program code or instructions, it implements the above-described related method steps to achieve the decoding method in the above embodiments.

[0342] Optionally, the device may further include at least one memory for storing the program code or instructions.

[0343] This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an encoding device, the encoding device performs the aforementioned related method steps to implement the encoding and decoding methods in the above embodiments.

[0344] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the encoding / decoding method described in the above embodiments.

[0345] This application also provides an encoding / decoding apparatus, which may specifically be a chip, integrated circuit, component, or module. Specifically, the apparatus may include a connected processor and a memory for storing instructions, or the apparatus may include at least one processor for fetching instructions from external memory. When the apparatus is running, the processor can execute instructions to cause the chip to perform the encoding / decoding methods in the above-described method embodiments.

[0346] Figure 15A schematic diagram of a chip 1500 is shown. The chip 1500 includes one or more processors 1501 and interface circuitry 1502. Optionally, the chip 1500 may also include a bus 1503.

[0347] Processor 1501 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above encoding / decoding method can be completed through integrated logic circuits in the hardware of processor 1501 or through software instructions.

[0348] Optionally, the processor 1501 described above may be a general-purpose processor, a digital signal processing (DSP) processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods and steps disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0349] The interface circuit 1502 can be used to send or receive data, instructions or information. The processor 1501 can use the data, instructions or other information received by the interface circuit 1502 to process the data, instructions or other information, and can send the processed information out through the interface circuit 1502.

[0350] Optionally, the chip may also include memory, which may include read-only memory and random access memory, providing operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0351] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0352] Optionally, the chip can be used in the encoding device or DOP involved in the embodiments of this application. Optionally, the interface circuit 1502 can be used to output the execution result of the processor 1501. For the encoding and decoding methods provided in one or more embodiments of this application, please refer to the foregoing embodiments, which will not be repeated here.

[0353] It should be noted that the functions of processor 1501 and interface circuit 1502 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0354] In this embodiment, the device, computer storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0355] It should be understood that in various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0356] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0357] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0358] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0359] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0360] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0361] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this application, essentially, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0362] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be included within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. An encoding method, characterized in that, include: Image filling is performed on the image to be encoded to obtain the filled image; The encoding unit is obtained based on the filled image; The encoding information of the encoding unit is determined based on the full padding information of the encoding unit. The full padding information is used to indicate that all samples in the encoding unit are image padding samples. The encoding information includes at least one of padding bit count or encoding code length. The full padding information is also used to indicate that the encoding unit is located within the target strip of the padded image. The target strip is a horizontal image padding strip. The encoding unit is encoded according to the encoding information to generate a bitstream.

2. The method according to claim 1, characterized in that, The encoding information includes the number of padding bits, and the step of encoding the encoding unit according to the encoding information to generate a bitstream includes: The encoding unit is encoded to generate a bitstream to be filled; The bitstream is obtained by filling bits into the bitstream to be filled according to the number of filling bits.

3. The method according to claim 1 or 2, characterized in that, The encoding information includes the number of padding bits, and determining the encoding information of the encoding unit based on the full padding information of the encoding unit includes: When the encoding unit is a target encoding unit, the number of padding bits is determined based on the actual number of bits of the encoding unit and a first preset number of bits. The target encoding unit is an encoding unit in which all samples are image padding samples, and the actual number of bits is the number of bits in the bitstream to be filled corresponding to the encoding unit.

4. A decoding method, characterized in that, The method includes: A bitstream is obtained, wherein the bitstream is a bitstream generated by an encoding unit according to the encoding information of the encoding unit. The encoding information is determined according to the full padding information of the encoding unit. The full padding information is used to indicate that all samples in the encoding unit are image padding samples. The encoding information includes at least one of padding bit count or encoding code length. The full padding information is also used to indicate that the encoding unit is located within the target strip of the image to be filled. The target strip is a horizontally filled strip of the image. The bitstream is decoded to obtain the reconstructed blocks; A reconstructed image is generated based on the reconstruction blocks.

5. The method according to claim 4, characterized in that, When the encoding information includes the number of padding bits, the reconstructed block is obtained by decoding the bitstream based on the number of padding bits.

6. The method according to claim 4 or 5, characterized in that, When the coding unit is a target coding unit, the number of padding bits is determined based on the actual number of bits of the coding unit and a first preset number of bits. The actual number of bits is the number of bits in the bitstream to be filled corresponding to the coding unit. The target coding unit is a coding unit in which all samples are image padding samples.

7. The method according to claim 4, characterized in that, The method further includes: The image filling region in the reconstructed image is cropped.

8. An encoding device, characterized in that, The device includes: an image filling unit, a segmentation unit, a determination unit, and an encoding unit; The image filling unit is used to fill in the image to be encoded to obtain a filled image; The segmentation unit is used to obtain the encoding unit based on the filled image; The determining unit is configured to determine the encoding information of the encoding unit based on the full filling information of the encoding unit. The full filling information is used to indicate that all samples in the encoding unit are image filling samples. The encoding information includes at least one of the number of filling bits or the encoding code length. The full filling information is also used to indicate that the encoding unit is located within the target strip of the filled image. The target strip is a horizontal image filling strip. The encoding unit is used to encode the encoding unit according to the encoding information to generate a bitstream.

9. The apparatus according to claim 8, characterized in that, The encoded information includes the number of padding bits, and the encoding unit is specifically used for: The encoding unit is encoded to generate a bitstream to be filled; The bitstream is obtained by filling bits into the bitstream to be filled according to the number of filling bits.

10. The apparatus according to claim 8 or 9, characterized in that, The encoded information includes the number of padding bits, and the determining unit is specifically used for: When the encoding unit is a target encoding unit, the number of padding bits is determined based on the actual number of bits of the encoding unit and a first preset number of bits. The target encoding unit is an encoding unit in which all samples are image padding samples, and the actual number of bits is the number of bits in the bitstream to be filled corresponding to the encoding unit.

11. A decoding device, characterized in that, The device includes: a receiving unit, a decoding unit, and a reconstruction unit; The receiving unit is used to acquire a bitstream, which is a bitstream generated by the encoding unit according to the encoding information of the encoding unit. The encoding information is determined according to the full padding information of the encoding unit. The full padding information is used to indicate that all samples in the encoding unit are image padding samples. The encoding information includes at least one of padding bit count or encoding code length. The full padding information is also used to indicate that the encoding unit is located within the target strip of the image to be filled. The target strip is a horizontal image padding strip. The decoding unit is used to decode the bitstream to obtain the reconstructed block; The reconstruction unit is used to generate a reconstructed image based on the reconstruction block.

12. The apparatus according to claim 11, characterized in that, When the encoding information includes the number of padding bits, the reconstructed block is obtained by decoding the bitstream based on the number of padding bits.

13. The apparatus according to claim 11 or 12, characterized in that, When the coding unit is a target coding unit, the number of padding bits is determined based on the actual number of bits of the coding unit and a first preset number of bits. The actual number of bits is the number of bits in the bitstream to be filled corresponding to the coding unit. The target coding unit is a coding unit in which all samples are image padding samples.

14. The apparatus according to claim 12, characterized in that, The reconstruction unit is also used for: The image filling region in the reconstructed image is cropped.

15. An encoding device, comprising at least one processor and a memory, characterized in that, The at least one processor executes a program or instructions stored in the memory to cause the encoding device to implement the method of any one of claims 1 to 3.

16. A decoding device, comprising at least one processor and a memory, characterized in that, The at least one processor executes a program or instructions stored in the memory to cause the decoding device to implement the method of any one of claims 4 to 7.

17. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is run on a computer or processor, it causes the computer or processor to perform the method of any one of claims 1 to 3 or any one of claims 4 to 7.

18. A computer program product, the computer program product comprising instructions, characterized in that, When the instructions are executed on a computer or processor, the computer or processor performs the method of any one of claims 1 to 3 or any one of claims 4 to 7.

Images