Entropy coding / decoding method and device

Abstract

The present application relates to the technical field of artificial intelligence (AI)-based image or audio coding / decoding, and provides an entropy coding / decoding method and device. The entropy coding method comprises: acquiring a plurality of symbols of an object to be processed; grouping the plurality of symbols, wherein the number of symbols of each group is N, and 1<N; acquiring M symbols from a current group, wherein 1≤M≤N; and coding the M symbols to obtain a code stream. The present application can reduce the memory read-write frequency, improving coding efficiency.

Description

Entropy encoding / decoding method and device Technical Field

[0001] The present application relates to video or image compression technology based on artificial intelligence (AI), and in particular to an entropy encoding / decoding method and device. Background Art

[0002] Video coding (video encoding and decoding) is widely used in digital video applications such as broadcast digital television, video transmission over the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, Digital Versatile Disc (DVD) and Blu-ray Disc, video content acquisition and editing systems, and security applications in camcorders.

[0003] Even in the case of relatively short films, large amounts of video data are required to describe them, which can create difficulties when the data is to be sent or otherwise transmitted across networks with limited bandwidth capacity. Therefore, video data is often compressed before being transmitted across modern telecommunications networks. Since memory resources may be limited, the size of the video can also be an issue when storing the video on a storage device. Video compression equipment typically uses software and / or hardware on the source side to encode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received by a video decompression device on the destination side. With limited network resources and the growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase the compression ratio with little impact on image quality.

[0004] In recent years, the application of deep learning in audio, video, and image encoding and decoding has become a growing trend. For example, Google has organized the Challenge on Learned Image Compression (CLIC) competition at the Institute of Electrical and Electronics Engineers (IEEE) International Conference on Computer Vision and Pattern Recognition (CVPR) for several consecutive years. CLIC focuses on using deep neural networks to improve image compression efficiency. Based on performance evaluations of the competition proposals, the overall compression efficiency of current deep learning-based image encoding and decoding solutions is comparable to that of the latest generation video image encoding and decoding standard, Versatile Video Coding (VVC), and they also have unique advantages in improving user-perceived quality.

[0005] When image features are encoded as information in related technologies, relevant information before and after encoding / decoding is stored in memory, which causes serious time consumption in memory reading and writing, thereby reducing encoding / decoding efficiency.

[0006] Summary of the Invention

[0007] The present application provides an entropy encoding / decoding method and device to reduce the number of memory read and write times and improve encoding efficiency.

[0008] In a first aspect, the present application provides an entropy coding method, comprising: obtaining multiple symbols of an object to be processed; grouping the multiple symbols, with the number of symbols in each group being N, 1<N; obtaining M symbols from the current group, 1≤M≤N; and encoding the M symbols to obtain a code stream.

[0009] In this embodiment, by encoding N symbols in groups (obtained by grouping the symbols of the objects to be processed) (including using the first encoding method and the second encoding method), the number of memory read and write times can be reduced and the encoding efficiency can be improved.

[0010] In this application, the objects to be processed may include the following forms:

[0011] (1) A file type or storage standard. For example, images, videos, audio, etc.

[0012] (2) A data transmission protocol. For example, online video, live broadcast, end-cloud collaborative content, etc.

[0013] (3) A compression software or storage device, such as audio and video compression software.

[0014] It should be noted that in addition to the above-mentioned forms, the objects to be processed in this application may also include other forms, the common point of which is that they involve entropy coding / decoding technology. Therefore, this application does not make specific limitations on the objects to be processed.

[0015] The object to be processed can be composed of symbols. For example, a video consists of a sequence of frames, each frame consisting of multiple pixels, so the symbols of the video can be pixels. Another example is that a plain text file consists of multiple characters (which can be Chinese characters, letters, numbers, etc.), so the symbols of the plain text file can be characters. And so on. Symbols are the basic units that make up information (data to be encoded / decoded).

[0016] The object to be processed can include many symbols. If these symbols are encoded together, the encoding process will be limited by register space, and the relevant data can only be stored in and read from memory. However, multiple memory reads and writes increase the time consumption. Therefore, to reduce the time consumption and improve the encoding efficiency, these symbols can be grouped, so that the number of symbols in a group is small, and the subsequent steps are processed in groups.

[0017] In this application, a group includes N symbols, where N is an integer greater than 1, for example, N = 2, 4, 5, 8, etc. These symbols can be grouped into the same group in order of N symbols based on their index. If the code stream is in first-in, first-out mode, the grouping can be done in ascending order of index; if the code stream is in first-in, last-out mode, the grouping can be done in descending order of index. Optionally, all symbols can be obtained at once before grouping, or one or more symbols can be grouped as soon as they are obtained. This application does not specifically limit this.

[0018] The current group can be the group currently being processed by the encoder during the encoding process. The current group is the period from the first symbol of a group to the Nth symbol of the group. As described in step 602, the current group includes N symbols.

[0019] In a possible implementation, the M symbols are symbols for which encoding operations are not skipped among the N symbols of the current group.

[0020] Based on this, the M symbols may include the following situations:

[0021] (1) 1≤M<N, indicating that some (one or more) symbols in the current group do not skip the encoding operation, so M symbols are some (one or more) symbols in the current group; correspondingly, NM symbols in the current group except the aforementioned M symbols all skip the encoding operation.

[0022] (2) M=N, indicating that all (N) symbols in the current group do not skip the encoding operation, so M symbols are all symbols in the current group; accordingly, there is no symbol in the current group that skips the encoding operation.

[0023] (3) M=0, indicating that all (N) symbols in the current group are skipped for encoding, and therefore there are no M symbols. In this case, the encoding process for the current group is completed, and there is no need to execute steps 603 and 604.

[0024] Optionally, the encoder can determine whether to skip encoding operations for each of the N symbols in the current group based on a premise (ie, data known during encoding / decoding, such as syntax elements). This operation is performed at the symbol level.

[0025] The first symbol may be a symbol that meets a preset condition among multiple symbols of the object to be processed. The preset condition may include a low probability and a large number. For example, if the probability of a certain symbol is less than a probability threshold and the proportion of the symbol in all symbols exceeds a proportion threshold, then the symbol can be determined to be the first symbol. The aforementioned probability may refer to the probability that the symbol takes a specific value (e.g., 0 or 1), and reference may be made to the technical description of entropy coding above.

[0026] The code stream includes two situations: first-in-last-out and first-in-first-out. At the decoding end, decoding is usually performed first using the first decoding method and then using the second decoding method. Therefore, in order to ensure that the decoding end decodes correctly, the encoding end can determine the encoding order according to the two situations of the aforementioned code stream, that is, when first-in-first-out, the encoding end first encodes using the first encoding method and then encodes using the second encoding method; when first-in-last-out, the encoding end first encodes using the second encoding method and then encodes using the first encoding method. Optionally, the aforementioned first encoding can be the INBOUND encoding described above, and the aforementioned second encoding can be the OUTBOUND encoding described above. It should be noted that in the OUTBOUND encoding, there is no specific limitation on the encoder of the first symbol, and in the INBOUND encoding, there is no specific limitation on the encoder of the second symbol (the symbol that does not meet the preset conditions among the multiple symbols of the object to be processed) and / or the preset value. The encoders of the two can be the same or different.

[0027] In one possible implementation, the M symbols include one or more first symbols that meet a preset condition. In this case, the first symbols are replaced with preset values ​​in the M symbols to obtain M processed symbols; the M processed symbols are encoded using a first encoding method to obtain a first code stream; the one or more first symbols are encoded using a second encoding method to obtain a second code stream; and the first code stream and the second code stream form a code stream.

[0028] In this scenario, for a first-in, first-out (FIFO) code stream, the first symbol of the M symbols can be replaced with a preset value (e.g., 2) to obtain M processed symbols. In this way, the second symbol of the M processed symbols retains its original value, while the first symbol becomes the preset value. If there are multiple first symbols in the M symbols, the encoder can replace all of them with the preset value.

[0029] The M processed symbols are then encoded using the first encoding method to obtain a first code stream. The encoding end can encode the M processed symbols together using the first encoding method. In this way, the first code stream can be written to the register at once, avoiding wasting register space. If the register space is insufficient, it is written to the memory (FLUSH).

[0030] The first symbol is then encoded using the second encoding method to obtain a second code stream. If there are multiple first symbols among the M symbols, the encoder can encode these multiple first symbols together using the second encoding method. In this way, the second code stream can be written to the register at once, avoiding wasting register space. If the register space is insufficient, it is written to the memory (FLUSH).

[0031] In one possible implementation, the M symbols include one or more first symbols that meet a preset condition. In this case, the one or more first symbols are encoded using the second encoding method to obtain a third code stream; among the M symbols, the first symbols are replaced with preset values ​​to obtain M processed symbols; the M processed symbols are encoded using the first encoding method to obtain a fourth code stream; and the third code stream and the fourth code stream form a code stream.

[0032] This scenario corresponds to a first-in, last-out code stream. For the first symbol among the M symbols, the second encoding method can be used to encode the first symbol first to obtain the third code stream. If there are multiple first symbols among the M symbols, the encoder can encode all of them together using the second encoding method. This allows the third code stream to be written to the register all at once, avoiding wasted register space. If register space is insufficient, the code stream is written to the memory (FLUSH).

[0033] Then, the aforementioned first symbol is replaced with a preset value (eg, 2) to obtain M processed symbols, so that the second symbol in the M processed symbols still retains the original value, and the first symbol becomes the preset value.

[0034] The M processed symbols are then encoded using the first encoding method to obtain a fourth code stream. The encoding end can encode the M processed symbols together using the first encoding method. In this way, the fourth code stream can be written to the register at once, avoiding wasting register space. If the register space is insufficient, it is written to the memory (FLUSH).

[0035] For example, in the register, the first code can write a maximum of 17 bits of code stream, and the second code can write a maximum of 8 bits of code stream. If the register is 32 bits, encoding one character requires a flush; if the register is 64 bits, encoding two characters requires a flush, which increases the number of memory read operations and increases the time consumption. When N=4, if the register is 32 bits, the four symbols are encoded together with the first and second codes, which is equivalent to only flushing once for four times of the second code and flushing once for one time of the first code. When the probability of the first symbol is small (for example, less than 1%), the average number of flush executions for the four symbols is 1.04, which is significantly less than the aforementioned four flushes, and the number of memory operations is significantly reduced.

[0036] Furthermore, when the first symbol is not included in the M symbols, the M symbols are encoded using the first encoding method to obtain a code stream. That is, if the first symbol is not included in the M symbols, then there is no need to replace the first symbol with a preset value, and the M symbols can be directly encoded using the first encoding method.

[0037] In a second aspect, the present application provides an entropy decoding method, including: obtaining a code stream, the code stream corresponding to multiple symbols of an object to be reconstructed, the multiple symbols being divided into multiple groups, each group including N symbols, among the N symbols, there are M symbols for which encoding operations are not skipped and D symbols for which encoding operations are skipped, 1<N, 1≤M≤N, M+D=N; decoding the code stream to obtain the M symbols; and setting the D symbols to a first preset value.

[0038] In this embodiment, by decoding the code stream of N symbols in groups (obtained by grouping the symbols of the objects to be processed) (including using the first decoding method and the second decoding method), the number of memory read and write times can be reduced and the encoding efficiency can be improved.

[0039] The decoder can decode after receiving the entire bitstream, or it can decode while receiving the bitstream, without specific restrictions. Using the prerequisites in the bitstream (i.e., data known during encoding and decoding, such as syntax elements), the decoder knows how many bits of the bitstream correspond to each symbol, and can therefore obtain the bitstream corresponding to N symbols for subsequent processing.

[0040] Corresponding to the encoding end, the decoding end may also have the concept of a group, where a group includes N symbols, where N is an integer greater than 1, for example, N=2, 4, 5, 8, etc.

[0041] The decoder can determine which of the N symbols in the current group have their coding operations skipped and which have their coding operations not skipped based on a pre-defined set of data (i.e., data known at both encoding and decoding time, such as syntax elements). This operation is performed at the symbol level. Based on this operation, the decoder determines that, among the N symbols in the current group, there are M symbols for which coding operations are not skipped and D symbols for which coding operations are skipped, where 1≤M≤N and M+D=N.

[0042] In the present application, the code stream can be decoded using the first decoding method to obtain M pre-processing symbols; when the M pre-processing symbols include M' second preset values, the code stream can be decoded using the second decoding method to obtain M' symbols (corresponding to the first symbol, satisfying the preset condition), 1≤M'≤M; in the M pre-processing symbols, the M' second preset values ​​are replaced with M' symbols to obtain M post-processing symbols (including the first symbol and the second symbol (not satisfying the preset condition)); the M post-processing symbols are used as M symbols; when the M pre-processing symbols do not include M' second preset values, the M pre-processing symbols are used as M symbols.

[0043] Optionally, the first decoding method may be the inbound decoding described above, and the second decoding method may be the outbound decoding described above. It should be noted that there is no specific restriction on the decoder of the bitstream in outbound decoding, and there is no specific restriction on the decoder of the bitstream in inbound decoding; the decoders of the two methods may be the same or different.

[0044] The above-mentioned preset conditions may include low probability and large quantity, for example, the probability of a certain symbol is less than a probability threshold, and the proportion of the symbol in all symbols exceeds a proportion threshold.

[0045] The decoding end decodes the bitstream using the first decoding method to obtain M pre-processed symbols. The M pre-processed symbols may include only the second symbol. In this case, the M pre-processed symbols are the aforementioned M symbols. Step 703 ends at this point, and step 704 proceeds. The decoding end may decode the bitstream of the M pre-processed symbols simultaneously using the first decoding method. In this way, the bitstream can be written to the register at once, avoiding wasted register space. If register space is insufficient, the bitstream is written to the memory (FLUSH).

[0046] The M pre-processed symbols may also include a second symbol and M' second preset values. In this case, the decoder decodes the code stream using the second decoding method to obtain M' symbols. The M' symbols are the original values ​​of the first symbol in the original M symbols. The decoder replaces the M' second preset values ​​with the M' symbols to obtain the aforementioned M symbols. The decoder can decode the code stream of M' symbols together using the second decoding method. In this way, the code stream can be written to the register at once, avoiding wasted register space. If register space is insufficient, the code stream is written to the memory (FLUSH).

[0047] Depending on the values ​​of M and D, the following situations can be included:

[0048] 1) M=N, D=0, indicating that the encoding operation is not skipped for the above N symbols, and therefore the M (ie, N) symbols can all be obtained according to the code stream. In this case, step 704 is not executed.

[0049] 2) 1≤M<N, D>0, indicating that some of the N symbols do not skip the encoding operation, while another part of the symbols skip the encoding operation. Therefore, the M symbols can be obtained according to the code stream, and the remaining D symbols can be obtained through step 704.

[0050] 3) M=0, D=N, indicating that the encoding operation is skipped for the above N symbols. The N symbols do not need to be obtained according to the code stream. In this case, step 703 is skipped and step 704 is directly executed.

[0051] Based on this, D>0 indicates that there are symbols in the current group whose encoding operation is skipped (ie, the D symbols mentioned above), so the D symbols can be set to a first preset value (eg, 0, -1, etc.) to reconstruct all symbols of the current group.

[0052] In a third aspect, the present application provides an entropy coding device, comprising: an acquisition module for acquiring multiple symbols of an object to be processed; a grouping module for grouping the multiple symbols, with the number of symbols in each group being N, 1<N; a skipping module for acquiring M symbols from the current group, 1≤M≤N; and an encoding module for encoding the M symbols to obtain a code stream.

[0053] In one possible implementation, the M symbols include one or more first symbols that meet preset conditions, and the encoding module is specifically configured to replace the first symbols in the M symbols with preset values ​​to obtain M processed symbols; encode the M processed symbols using a first encoding method to obtain a first code stream; encode the one or more first symbols using a second encoding method to obtain a second code stream; and the first code stream and the second code stream form the code stream.

[0054] In one possible implementation, the M symbols include one or more first symbols that meet a preset condition, and the encoding module is specifically configured to encode the one or more first symbols using a second encoding method to obtain a third code stream; replace the first symbols in the M symbols with preset values ​​to obtain M processed symbols; encode the M processed symbols using the first encoding method to obtain a fourth code stream; and the third code stream and the fourth code stream form the code stream.

[0055] In a possible implementation, the M symbols do not include a first symbol that meets a preset condition, and the encoding module is specifically configured to encode the M symbols using a first encoding method to obtain the code stream.

[0056] In a possible implementation, the preset condition includes that the probability of the first symbol is less than a probability threshold, and the proportion of the first symbol in the multiple symbols exceeds a proportion threshold.

[0057] In a possible implementation, the M symbols are symbols for which encoding operations are not skipped among the N symbols of the current group.

[0058] In one possible implementation, N=4.

[0059] In a fourth aspect, the present application provides an entropy decoding device, comprising: an acquisition module for acquiring a code stream, the code stream corresponding to multiple symbols of an object to be reconstructed, the multiple symbols being divided into multiple groups, each group including N symbols, and among the N symbols, there are M symbols for which encoding operations are not skipped and D symbols for which encoding operations are skipped, 1<N, 1≤M≤N, M+D=N; a decoding module for decoding the code stream to obtain the M symbols; and a setting module for setting the D symbols to a first preset value.

[0060] In one possible implementation, the decoding module is specifically configured to decode the code stream using a first decoding method to obtain M pre-processed symbols; when the M pre-processed symbols include M' second preset values, decode the code stream using a second decoding method to obtain M' symbols, 1≤M'≤M; replace the M' second preset values ​​in the M pre-processed symbols with the M' symbols to obtain M post-processed symbols; use the M post-processed symbols as the M symbols; and when the M pre-processed symbols do not include M' second preset values, use the M pre-processed symbols as the M symbols.

[0061] In one possible implementation, N=4.

[0062] In a fifth aspect, the present application provides an encoder comprising a processing circuit for executing the method according to any one of claims 1-7.

[0063] In a sixth aspect, the present application provides a decoder comprising a processing circuit for executing the method according to any one of claims 8 to 10.

[0064] In a seventh aspect, the present application provides a computer program product, comprising a computer program code, which enables the computer to execute the method according to any one of claims 1 to 10 when the computer program code is run on the computer.

[0065] In an eighth aspect, the present application provides a non-transitory computer-readable storage medium, comprising a code stream obtained by the entropy coding method according to any one of claims 1-7.

[0066] In a ninth aspect, the present application provides an encoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processor and storing a program judged by the processor, wherein the program, when judged by the processor, causes the encoder to execute a method according to any one of claims 1-7.

[0067] In the tenth aspect, the present application provides a decoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processor and storing a program judged by the processor, wherein the program, when judged by the processor, causes the decoder to execute the method according to any one of claims 8-10.

[0068] In an eleventh aspect, the present application provides an image or audio processor, comprising a processing circuit for executing the method according to any one of claims 1-10. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] FIG1a is an exemplary block diagram of a decoding system 10 according to an embodiment of the present application;

[0070] FIG1 b is an exemplary block diagram of a video decoding system 40 according to an embodiment of the present application;

[0071] FIG2 is an exemplary block diagram of a video encoder 20 according to an embodiment of the present application;

[0072] FIG3 is an exemplary block diagram of a video decoder 30 according to an embodiment of the present application;

[0073] FIG4 is an exemplary block diagram of a video decoding device 400 according to an embodiment of the present application;

[0074] FIG5 is an exemplary block diagram of an apparatus 500 according to an embodiment of the present application;

[0075] FIG6 is a flowchart of a process 600 of an entropy coding method provided in an embodiment of the present application;

[0076] FIG7 is a flowchart of a process 700 of an entropy decoding method provided in an embodiment of the present application;

[0077] FIG8 is a flowchart of the present application after skipping the execution of encoding operations and encoding fusion;

[0078] FIG9 is a flowchart of the present application after skipping the encoding operation and decoding fusion;

[0079] FIG10 is a schematic structural diagram of an entropy coding apparatus 1000 of the present application;

[0080] FIG11 is a schematic structural diagram of an entropy decoding device 1100 of the present application. DETAILED DESCRIPTION

[0081] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0082] The terms "first," "second," and the like in the description, embodiments, claims, and drawings of this application are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or order. Furthermore, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions, such as, for example, inclusion of a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0083] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0084] Video coding generally refers to the processing of a sequence of images to form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" can be used as synonyms. Video coding (or commonly referred to as coding) includes two parts: video encoding and video decoding. Video coding is performed on the source side and generally includes processing (for example, compressing) the original video image to reduce the amount of data required to represent the video image (thereby making it more efficient to store and / or transmit). Video decoding is performed on the destination side and generally includes inverse processing relative to the encoder to reconstruct the video image. The "encoding" of the video image (or commonly referred to as image) involved in the embodiment should be understood as the "encoding" or "decoding" of the video image or video sequence. The encoding part and the decoding part are also collectively referred to as codec (encoding and decoding, CODEC).

[0085] In the case of lossless video coding, the original video image can be reconstructed, that is, the reconstructed video image has the same quality as the original video image (assuming there is no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed through quantization, etc. to reduce the amount of data required to represent the video image, but the decoder side cannot fully reconstruct the video image, that is, the quality of the reconstructed video image is lower or worse than the quality of the original video image.

[0086] Several video coding standards fall under the category of "lossy hybrid video codecs" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is typically divided into a set of non-overlapping blocks, which are typically coded at the block level. In other words, the encoder typically processes, i.e., encodes, the video at the block (video block) level, for example, by generating a prediction block through spatial (intra-frame) and temporal (inter-frame) prediction; subtracting the prediction block from the current block (currently processed / to-be-processed block) to obtain a residual block; transforming and quantizing the residual block in the transform domain to reduce the amount of data to be transmitted (compressed), while the decoder applies the inverse of the encoder's processing to the coded or compressed block to reconstruct the current block for representation. Furthermore, the encoder needs to repeat the decoder's processing steps so that the encoder and decoder generate the same predictions (e.g., intra-frame predictions and inter-frame predictions) and / or reconstructed pixels for processing, i.e., encoding, the subsequent block.

[0087] In the following embodiment of the decoding system 10 , the encoder 20 and the decoder 30 are described with reference to FIG. 1 a to FIG. 3 .

[0088] FIG1a is an exemplary block diagram of a decoding system 10 according to an embodiment of the present application, such as a video decoding system 10 (or simply, decoding system 10) that can utilize the techniques of the present application. The video encoder 20 (or simply, encoder 20) and the video decoder 30 (or simply, decoder 30) in the video decoding system 10 represent devices that can be used to perform various techniques according to various examples described herein.

[0089] As shown in FIG. 1 a , a decoding system 10 includes a source device 12 for providing encoded image data 21 such as an encoded image to a destination device 14 for decoding the encoded image data 21 .

[0090] The source device 12 includes an encoder 20 , and optionally, may include an image source 16 , a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22 .

[0091] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0092] In order to distinguish the processing performed by the pre-processor (or pre-processing unit) 18 , the image (or image data) 17 may also be referred to as a raw image (or raw image data) 17 .

[0093] The preprocessor 18 is configured to receive raw image data 17 and preprocess the raw image data 17 to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color grading, or denoising. It will be appreciated that the preprocessor 18 may be an optional component.

[0094] The video encoder (or encoder) 20 is used to receive the pre-processed image data 19 and provide encoded image data 21 (which will be further described below with reference to FIG. 2 and the like).

[0095] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device through the communication channel 13 for storage or direct reconstruction.

[0096] The destination device 14 includes a decoder 30 and, in addition or alternatively, may include a communication interface (or communication unit) 28 , a post-processor (or post-processing unit) 32 , and a display device 34 .

[0097] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is a encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0098] The communication interface 22 and the communication interface 28 can be used to send or receive encoded image data (or encoded data) 21 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0099] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0100] The communication interface 28 corresponds to the communication interface 22 , and can be used, for example, to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21 .

[0101] Both the communication interface 22 and the communication interface 28 can be configured as a unidirectional communication interface as indicated by the arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 in Figure 1a, or a bidirectional communication interface, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0102] The video decoder (or decoder) 30 is used to receive the encoded image data 21 and provide decoded image data (or decoded image data) 31 (which will be further described below with reference to FIG. 3 and the like).

[0103] The post-processor 32 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image, to obtain post-processed image data 33, such as the post-processed image. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, cropping, or resampling, or any other processing for generating the decoded image data 31 for display on a display device 34, etc.

[0104] The display device 34 is configured to receive the post-processed image data 33 and display the image to a user or viewer. The display device 34 may be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen.

[0105] The decoding system 10 also includes a training engine 25, which is used to train the encoder 20 or the decoder 30 to process images, image regions, or image blocks. The training data in the embodiment of the present application can be stored in a database (not shown), and the training engine 25 trains the target model based on the training data. It should be noted that the embodiment of the present application does not limit the source of the training data. For example, the training data can be obtained from the cloud or other places for model training.

[0106] The target model trained by the training engine 25 can be applied to the decoding system 10, for example, to the source device 12 (e.g., encoder 20) or the destination device 14 (e.g., decoder 30) shown in FIG1a. The training engine 25 can train the target model in the cloud, and then the decoding system 10 downloads and uses the target model from the cloud; alternatively, the training engine 25 can train the target model in the cloud and use the target model, and the decoding system 10 directly obtains the processing results from the cloud.

[0107] Although FIG1a shows source device 12 and destination device 14 as separate devices, device embodiments may also include both source device 12 and destination device 14 or the functions of both source device 12 and destination device 14, that is, both source device 12 or the corresponding functions and destination device 14 or the corresponding functions. In these embodiments, source device 12 or the corresponding functions and destination device 14 or the corresponding functions may be implemented using the same hardware and / or software or through separate hardware and / or software or any combination thereof.

[0108] According to the description, the existence and (accurate) division of different units or functions in the source device 12 and / or the destination device 14 shown in FIG. 1 a may vary depending on actual devices and applications, which is obvious to those skilled in the art.

[0109] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both, may be implemented using processing circuitry as shown in FIG1b, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video encoding processors, or any combination thereof. The encoder 20 may be implemented using processing circuitry 46 to include the various modules discussed with reference to the encoder 20 in FIG2 and / or any other encoder systems or subsystems described herein. The decoder 30 may be implemented using processing circuitry 46 to include the various modules discussed with reference to the decoder 30 in FIG3 and / or any other decoder systems or subsystems described herein. The processing circuitry 46 may be used to perform the various operations discussed below. As shown in FIG5, if portions of the technology are implemented in software, the device may store the software's instructions in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the technology. One of the video encoder 20 and the video decoder 30 may be integrated into a single device as part of a combined encoder / decoder (CODEC), as shown in FIG. 1 b .

[0110] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook computer or laptop, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or content distribution server), a broadcast receiving device, a broadcast transmitting device, etc., and may not use or use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0111] In some cases, the video decoding system 10 shown in FIG1a is merely exemplary, and the techniques provided herein may be applicable to video encoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data is retrieved from a local memory, sent over a network, and so on. A video encoding device may encode data and store the data in a memory, and / or a video decoding device may retrieve data from a memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data to a memory and / or retrieve and decode data from a memory.

[0112] Figure 1b is an exemplary block diagram of a video decoding system 40 of an embodiment of the present application. As shown in Figure 1b, the video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by a processing circuit 46), an antenna 42, one or more processors 43, one or more memory storages 44 and / or a display device 45.

[0113] 1b, imaging device 41, antenna 42, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory 44, and / or display device 45 are capable of communicating with one another. In different embodiments, video decoding system 40 may include only video encoder 20 or only video decoder 30.

[0114] In some instances, antenna 42 can be used to transmit or receive an encoded bitstream of video data. Additionally, in some instances, display device 45 can be used to present the video data. Processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Video decoding system 40 can also include an optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Furthermore, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory). In a non-limiting example, memory storage 44 can be implemented as cache memory. In other instances, processing circuitry 46 can include memory (e.g., cache memory, etc.) for implementing an image buffer, etc.

[0115] In some examples, video encoder 20 implemented by logic circuitry may include an image buffer (e.g., implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented by processing circuitry 46 to implement the various modules discussed with reference to FIG. 2 and / or any other encoder systems or subsystems described herein. Logic circuitry may be used to perform the various operations discussed herein.

[0116] In some examples, the video decoder 30 may be implemented by processing circuitry 46 in a similar manner to implement the various modules discussed with reference to the video decoder 30 of FIG. 3 and / or any other decoder systems or subsystems described herein. In some examples, the logic circuitry-implemented video decoder 30 may include an image buffer (implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented by processing circuitry 46 to implement the various modules discussed with reference to FIG. 3 and / or any other decoder systems or subsystems described herein.

[0117] In some examples, antenna 42 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data related to the encoded video frames, indicators, index values, mode selection data, etc., as discussed herein, such as data related to encoding partitions (e.g., transform coefficients or quantized transform coefficients, optional indicators (as discussed), and / or data defining encoding partitions). Video decoding system 40 may also include video decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0118] It should be understood that for the examples described herein with reference to video encoder 20, video decoder 30 can be configured to perform the reverse process. With respect to signaling syntax elements, video decoder 30 can be configured to receive and parse such syntax elements and decode the associated video data accordingly. In some examples, video encoder 20 can entropy encode the syntax elements into an encoded video bitstream. In such examples, video decoder 30 can parse such syntax elements and decode the associated video data accordingly.

[0119] For ease of description, the embodiments of the present application are described with reference to the universal video coding (VVC) reference software or the high-efficiency video coding (HEVC) developed by the joint collaboration team on video coding (JCT-VC) of the ITU-T video coding experts group (VCEG) and the ISO / IEC motion picture experts group (MPEG). Those skilled in the art will understand that the embodiments of the present application are not limited to HEVC or VVC.

[0120] Encoders and encoding methods

[0121] FIG2 is an exemplary block diagram of a video encoder 20 according to an embodiment of the present application. As shown in FIG2 , the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 shown in FIG2 may also be referred to as a hybrid video encoder or a video encoder based on a hybrid video codec.

[0122] The residual calculation unit 204, the transform processing unit 206, the quantization unit 208, and the mode selection unit 260 constitute the forward signal path of the encoder 20, while the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-frame prediction unit 244, and the intra-frame prediction unit 254 constitute the backward signal path of the encoder, where the backward signal path of the encoder 20 corresponds to the signal path of the decoder (see decoder 30 in Figure 3). The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer 230, the inter-frame prediction unit 244, and the intra-frame prediction unit 254 also constitute the "internal decoder" of the video encoder 20.

[0123] Images and image segmentation (images and blocks)

[0124] Encoder 20 is operable to receive, via input 201 or the like, an image (or image data) 17, e.g., an image from a sequence of images forming a video or video sequence. The received image or image data may also be a pre-processed image (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 may also be referred to as a current image or image to be encoded (particularly when distinguishing the current image from other images in video encoding, such as previously encoded and / or decoded images in the same video sequence, i.e., a video sequence that also includes the current image).

[0125] A (digital) image is, or can be considered to be, a two-dimensional array or matrix of pixels with intensity values. The pixels in the array are also referred to as pixels (or pels, short for picture elements). The number of pixels in the array or image in the horizontal and vertical directions (or axes) determines the image size and / or resolution. To represent color, three color components are typically used, meaning that an image can be represented as or include three pixel arrays. In the RBG format or color space, an image includes corresponding arrays of red, green, and blue pixels. However, in video coding, each pixel is typically represented in a luma / chroma format or color space, such as YCbCr, which includes a luma component indicated by Y (sometimes also indicated by L) and two chroma components, indicated by Cb and Cr. The luma component Y represents the brightness or grayscale level intensity (for example, in grayscale images, both are the same), while the two chroma components (abbreviated as chroma) Cb and Cr represent the chroma or color information components. Accordingly, an image in YCbCr format includes a luma pixel array of luma pixel values ​​(Y) and two chroma pixel arrays of chroma values ​​(Cb and Cr). An image in RGB format can be converted or transformed into YCbCr format, and vice versa, a process also known as color conversion or transformation. If the image is black and white, the image may include only a luma pixel array. Accordingly, the image may be, for example, a luma pixel array in monochrome format or a luma pixel array and two corresponding chroma pixel arrays in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0126] In one embodiment, an embodiment of the video encoder 20 may include an image segmentation unit (not shown in FIG. 2 ) for segmenting the image 17 into a plurality of (typically non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC) or coding tree blocks (CTBs), or coding tree units (CTUs) in the H.265 / HEVC and VVC standards. The segmentation unit may be used to use the same block size for all images in a video sequence and a corresponding grid of defined block sizes, or to vary the block size between images or subsets or groups of images, and to segment each image into corresponding blocks.

[0127] In other embodiments, the video encoder may be configured to directly receive a block 203 of the image 17, for example, one, several or all blocks constituting the image 17. The image block 203 may also be referred to as a current image block or an image block to be encoded.

[0128] Like image 17, image block 203 is also or can be considered to be a two-dimensional array or matrix of pixels having intensity values ​​(pixel values), but image block 203 is smaller than image 17. In other words, block 203 may include one pixel array (e.g., a luminance array in the case of monochrome image 17, or a luminance array or chrominance array in the case of a color image), or three pixel arrays (e.g., one luminance array and two chrominance arrays in the case of a color image 17), or any other number and / or type of arrays depending on the color format used. The number of pixels in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Accordingly, a block may be an M×N (M columns×N rows) pixel array, or an M×N transform coefficient array, etc.

[0129] In one embodiment, the video encoder 20 shown in FIG. 2 is configured to encode the image 17 block by block, for example, performing encoding and prediction on each block 203 .

[0130] In one embodiment, the video encoder 20 shown in FIG2 may also be configured to partition and / or encode an image using slices (also referred to as video slices), where an image may be partitioned or encoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., coding tree units (CTUs)) or one or more groups of blocks (e.g., tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0131] In one embodiment, the video encoder 20 shown in Figure 2 can also be used to segment and / or encode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), where the image can be segmented or encoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTUs) or one or more coding blocks, etc., where each coding block can be in a shape such as a rectangle and may include one or more complete or partial blocks (e.g., CTUs).

[0132] Residual calculation

[0133] The residual calculation unit 204 is used to calculate the residual block 205 (the prediction block 265 is described in detail later) based on the image block (or original block) 203 and the prediction block 265 in the following manner: for example, the pixel value of the prediction block 265 is subtracted from the pixel value of the image block 203 pixel by pixel (pixel by pixel) to obtain the residual block 205 in the pixel domain.

[0134] Transform

[0135] The transform processing unit 206 is configured to perform a discrete cosine transform (DCT) or a discrete sine transform (DST) on the pixel values ​​of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 may also be referred to as transform residual coefficients, representing the residual block 205 in the transform domain.

[0136] The transform processing unit 206 may be used to apply an integerized approximation of the DCT / DST, such as the transform specified for H.265 / HEVC. This integerized approximation is typically scaled by a factor compared to the orthogonal DCT transform. In order to maintain the norm of the residual block after the forward and inverse transforms, other scaling factors are used as part of the transform process. The scaling factor is typically selected based on certain constraints, such as whether the scaling factor is a power of 2 for the shift operation, the bit depth of the transform coefficients, a trade-off between accuracy and implementation cost, etc. For example, a specific scaling factor is specified for the inverse transform on the encoder 20 side by the inverse transform processing unit 212 (and for the corresponding inverse transform on the decoder 30 side by, for example, the inverse transform processing unit 312), and correspondingly, a corresponding scaling factor may be specified for the forward transform on the encoder 20 side by the transform processing unit 206.

[0137] In one embodiment, the video encoder 20 (correspondingly, the transform processing unit 206) may be configured to output transform parameters such as one or more transform types, for example, directly output or output after being encoded or compressed by the entropy coding unit 270, such that the video decoder 30 may receive and use the transform parameters for decoding.

[0138] Quantification

[0139] The quantization unit 208 is configured to quantize the transform coefficients 207 by, for example, scalar quantization or vector quantization to obtain quantized transform coefficients 209 . The quantized transform coefficients 209 may also be referred to as quantized residual coefficients 209 .

[0140] The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, during quantization, an n-bit transform coefficient may be rounded down to an m-bit transform coefficient, where n is greater than m. The degree of quantization may be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, varying degrees of scaling may be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to finer quantization, while a larger quantization step size corresponds to coarser quantization. The appropriate quantization step size may be indicated by a quantization parameter (QP). For example, the quantization parameter may be an index into a predefined set of appropriate quantization step sizes. For example, a smaller quantization parameter may correspond to fine quantization (a smaller quantization step size), while a larger quantization parameter may correspond to coarse quantization (a larger quantization step size), or vice versa. Quantization may include dividing by the quantization step size, while the corresponding or inverse dequantization performed by the inverse quantization unit 210, etc., may include multiplying by the quantization step size. Embodiments according to some standards, such as HEVC, may be used to determine the quantization step size using the quantization parameter. Generally, the quantization step size may be calculated based on the quantization parameter using a fixed-point approximation of an equation involving division. Other scaling factors can be introduced for quantization and dequantization to restore the norm of the residual block that may have been modified by the scaling used in the fixed-point approximation of the equations for the quantization step size and the quantization parameter. In one exemplary implementation, the scaling of the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where larger quantization step sizes result in greater losses.

[0141] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) may be configured to output a quantization parameter (QP), for example, directly or after being encoded or compressed by the entropy coding unit 270, such that the video decoder 30 may receive and use the quantization parameter for decoding.

[0142] Dequantization

[0143] The inverse quantization unit 210 is configured to perform inverse quantization performed by the quantization unit 208 on the quantized coefficients to obtain dequantized coefficients 211. For example, the inverse quantization scheme performed by the quantization unit 208 may be performed according to or using the same quantization step size as the quantization unit 208. The dequantized coefficients 211 may also be referred to as dequantized residual coefficients 211, and correspond to the transform coefficients 207. However, due to the loss caused by quantization, the dequantized coefficients 211 are generally not identical to the transform coefficients.

[0144] Inverse transform

[0145] The inverse transform processing unit 212 is configured to perform the inverse transform of the transform performed by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), to obtain a reconstructed residual block 213 (or corresponding dequantized coefficients 213) in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 213.

[0146] reconstruction

[0147] The reconstruction unit 214 (e.g., the summer 214) is used to add the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265 to obtain the reconstructed block 215 in the pixel domain, for example, by adding the pixel point values ​​of the reconstructed residual block 213 and the pixel point values ​​of the prediction block 265.

[0148] Filtering

[0149] The loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain a filter block 221, or generally to filter the reconstructed pixels to obtain filtered pixel values. For example, the loop filter unit is used to smoothly perform pixel conversion or improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process can be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop reshaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 220 is shown as a loop filter in FIG2 , in other configurations, the loop filter unit 220 can be implemented as a post-loop filter. The filter block 221 can also be referred to as a filter reconstruction block 221.

[0150] In one embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) may be configured to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, directly or after being entropy-encoded by the entropy coding unit 270, such that the decoder 30 may receive and use the same or different loop filter parameters for decoding.

[0151] Decoded Image Buffer

[0152] The decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference picture data for use by the video encoder 20 when encoding video data. The DPB 230 may be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer 230 may be used to store one or more filter blocks 221. The decoded picture buffer 230 may also be used to store other previously filtered blocks, such as previously reconstructed and filtered blocks 221, for the same current picture or a different picture, such as a previously reconstructed picture, and may provide a complete previously reconstructed, or decoded, picture (and corresponding reference blocks and pixels) and / or a partially reconstructed current picture (and corresponding reference blocks and pixels), for example, for inter-frame prediction. The decoded image buffer 230 may also be used to store one or more unfiltered reconstructed blocks 215, or generally to store unfiltered reconstructed pixels, for example, reconstructed blocks 215 that have not been filtered by the loop filtering unit 220, or reconstructed blocks or reconstructed pixels that have not undergone any other processing.

[0153] Mode selection (segmentation and prediction)

[0154] The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, and is configured to receive or obtain original image data, such as an original block 203 (current block 203 of the current image 17) and reconstructed image data, such as filtered and / or unfiltered reconstructed pixels or reconstructed blocks of the same (current) image and / or one or more previously decoded images, from the decoded image buffer 230 or other buffer (e.g., a column buffer, not shown). The reconstructed image data is used as reference image data for prediction, such as inter-frame prediction or intra-frame prediction, to obtain a prediction block 265 or a prediction value 265.

[0155] The mode selection unit 260 may be used to determine or select a partitioning for the current block (including no partitioning) and prediction mode (eg, intra-frame or inter-frame prediction mode), generate a corresponding prediction block 265 , and calculate the residual block 205 and reconstruct the reconstruction block 215 .

[0156] In one embodiment, the mode selection unit 260 may be configured to select a segmentation and prediction mode (e.g., from prediction modes supported or available by the mode selection unit 260) that provides the best match or minimum residual (minimum residual means better compression during transmission or storage), or provides minimum signaling overhead (minimum signaling overhead means better compression during transmission or storage), or simultaneously considers or balances both. The mode selection unit 260 may be configured to determine the segmentation and prediction mode based on rate distortion optimization (RDO), i.e., select the prediction mode that provides minimum rate distortion optimization. Terms such as "best," "lowest," and "optimal" herein do not necessarily refer to "best," "lowest," or "optimal" overall, but may also refer to situations where termination or selection criteria are met, e.g., values ​​exceeding or falling below a threshold or other limit may result in a "suboptimal selection" but reduce complexity and processing time.

[0157] In other words, the partitioning unit 262 may be configured to partition an image in a video sequence into a sequence of coding tree units (CTUs), the CTU 203 being further partitioned into smaller block portions or sub-blocks (again forming blocks), e.g., by iteratively using quad-tree partitioning (QT), binary-tree partitioning (BT), or triple-tree partitioning (TT), or any combination thereof, and for, e.g., performing prediction on each of the block portions or sub-blocks, wherein the mode selection comprises selecting a tree structure for partitioning the block 203 and selecting a prediction mode to be applied to each of the block portions or sub-blocks.

[0158] The segmentation (eg, performed by segmentation unit 262) and prediction processes (eg, performed by inter-prediction unit 244 and intra-prediction unit 254) performed by video encoder 20 are described in detail below.

[0159] segmentation

[0160] The partitioning unit 262 can partition (or divide) an image block (or CTU) 203 into smaller parts, such as square or rectangular blocks. For an image with three pixel arrays, a CTU consists of N×N luminance pixel blocks and two corresponding chrominance pixel blocks. The maximum allowed size of a luminance block in a CTU is specified as 128×128 in the developing Universal Video Coding (VVC) standard, but may be specified to a value other than 128×128, such as 256×256, in the future. The CTUs of an image can be grouped / collected into slices / coding block groups, coding blocks, or bricks. A coding block covers a rectangular area of ​​an image and can be divided into one or more bricks. A brick consists of multiple CTU rows within a coding block. A coding block that is not partitioned into multiple bricks can be called a brick. However, a brick is a true subset of a coding block and is therefore not called a coding block. VVC supports two coding block group modes: raster scan slice / coding block group mode and rectangular slice mode. In raster scan CBG mode, a slice / CBG contains a sequence of CBs from a raster scan of the CBs of an image. In rectangular slice mode, a slice contains multiple bricks of an image that together form a rectangular region of the image. The bricks within a rectangular slice are arranged in the slice's brick raster scan order. These smaller blocks (also called sub-blocks) can be further split into smaller parts. This is also known as tree partitioning or hierarchical tree partitioning, where a root block at, for example, root tree level 0 (hierarchy level 0, depth 0) can be recursively split into two or more blocks at the next lower tree level, such as nodes at tree level 1 (hierarchy level 1, depth 1). These blocks can in turn be split into two or more blocks at the next lower level, such as tree level 2 (hierarchy level 2, depth 2), and so on, until the partitioning is complete (because an end criterion is met, such as reaching a maximum tree depth or minimum block size). Blocks that are not further split are also called leaf blocks or leaf nodes of the tree. A tree divided into two parts is called a binary tree (BT), a tree divided into three parts is called a ternary tree (TT), and a tree divided into four parts is called a quadtree (QT).

[0161] For example, a coding tree unit (CTU) may be or include a CTB for luma pixels, two corresponding CTBs for chroma pixels of an image with a three-pixel array, or a CTB for pixels of a monochrome image, or a CTB for pixels of an image encoded using three independent color planes and syntax structures for encoding pixels. Accordingly, a coding tree block (CTB) may be an N×N block of pixels, where N may be set to a value such that a component is divided into CTBs, which is known as partitioning. A coding unit (CU) may be or include a coding block of luma pixels, two corresponding coding blocks for chroma pixels of an image with a three-pixel array, or a coding block of pixels of a monochrome image, or a coding block of pixels of an image encoded using three independent color planes and syntax structures for encoding pixels. Accordingly, a coding block (CB) may be an M×N block of pixels, where M and N may be set to a value such that a CTB is divided into coding blocks, which is known as partitioning.

[0162] For example, in an embodiment, according to HEVC, a coding tree unit (CTU) can be divided into multiple CUs using a quadtree structure represented as a coding tree. A decision is made at the leaf-CU level whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode an image area. Each leaf-CU can be further divided into one, two, or four PUs according to the PU partition type. The same prediction process is used within a PU, and relevant information is transmitted to the decoder in units of PUs. After applying the prediction process according to the PU partition type to obtain a residual block, the leaf-CU can be divided into transform units (TUs) according to other quadtree structures similar to the coding tree for the CU.

[0163] For example, in an embodiment, according to the latest video coding standard currently under development (called Versatile Video Coding (VVC), a combined quadtree of nested multi-type trees (such as binary trees and ternary trees) is used to divide the segment structure for partitioning the coding tree unit. In the coding tree structure within the coding tree unit, the CU can be square or rectangular. For example, the coding tree unit (CTU) is first partitioned by the quadtree structure. The quadtree leaf nodes are further partitioned by the multi-type tree structure. The multi-type tree structure has four partition types: vertical binary tree partition (SPLIT_BT_VER), horizontal binary tree partition (SPLIT_BT_HOR), vertical ternary tree partition The multi-type tree leaf nodes are called coding units (CUs), unless the CU is too large for the maximum transform length, such segmentation is used for prediction and transform processing without any other splitting. In most cases, this means that the block sizes of CUs, PUs, and TUs in the coding block structure of the quadtree nested multi-type tree are the same. This exception occurs when the maximum supported transform length is less than the width or height of the color components of the CU. VVC has developed a unique signaling mechanism for the split partitioning information in the coding structure with quadtree nested multi-type trees. In the signaling mechanism, the coding The tree unit (CTU) as the root of the quadtree is first split by the quadtree structure. Then each quadtree leaf node (when large enough) is further split into a multi-type tree structure. In the multi-type tree structure, the first flag (mtt_split_cu_flag) is used to indicate whether the node is further split. When the node is further split, the second flag (mtt_split_cu_vertical_flag) is used to indicate the division direction, and the third flag (mtt_split_cu_binary_flag) is used to indicate whether the division is a binary tree division or a ternary tree division. According to mtt_split_c The values ​​of u_vertical_flag and mtt_split_cu_binary_flag allow the decoder to derive the multi-type tree splitting mode (MttSplitMode) of the CU based on predefined rules or tables. It should be noted that for certain designs, such as the 64×64 luminance block and 32×32 chroma pipeline design in the VVC hardware decoder, TT splitting is not allowed when the width or height of the luminance coding block is greater than 64. TT splitting is also not allowed when the width or height of the chroma coding block is greater than 32. The pipeline design divides the image into multiple virtual pipeline data units (VPDUs), and each VPDU is defined as a non-overlapping unit in the image. In the hardware decoder, consecutive VPDUs are processed simultaneously in multiple pipeline stages.In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so it is desirable to keep the VPDU small. In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, the ternary tree (TT) and binary tree (BT) partitioning may increase the VPDU size.

[0164] In addition, it should be noted that when a part of the tree node block exceeds the bottom or the right boundary of the image, the tree node block is forcibly divided until all pixels of each coding CU are located within the image boundary.

[0165] For example, the intra sub-partitions (ISP) tool may divide the luma intra prediction block into two or four sub-partitions vertically or horizontally according to the block size.

[0166] In one example, mode select unit 260 of video encoder 20 may be used to perform any combination of the segmentation techniques described above.

[0167] As described above, the video encoder 20 is configured to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, intra-frame prediction modes and / or inter-frame prediction modes.

[0168] Intra-frame prediction

[0169] The intra prediction mode set may include 35 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in HEVC, or may include 67 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in VVC. For example, several traditional angle intra prediction modes are adaptively replaced by wide-angle intra prediction modes for non-square blocks defined in VVC. For another example, in order to avoid the division operation of DC prediction, only the longer side is used to calculate the average value of the non-square block. In addition, the intra prediction result of the planar mode can also be modified using the position dependent intra prediction combination (PDPC) method.

[0170] The intra prediction unit 254 is configured to generate an intra prediction block 265 using reconstructed pixels of adjacent blocks of the same current image according to an intra prediction mode in the intra prediction mode set.

[0171] The intra-frame prediction unit 254 (or generally the mode selection unit 260) is also used to output intra-frame prediction parameters (or generally information indicating the selected intra-frame prediction mode for the block) in the form of syntax elements 266 to the entropy coding unit 270 for inclusion in the encoded image data 21, so that the video decoder 30 can perform operations, such as receiving and using the prediction parameters for decoding.

[0172] The intra prediction modes in HEVC include DC prediction mode, plane prediction mode and 33 angular prediction modes, with a total of 35 candidate prediction modes. The current block can use the pixels of the reconstructed image blocks on the left and above as references for intra prediction. The image blocks in the surrounding area of ​​the current block used for intra prediction of the current block are called reference blocks, and the pixels in the reference blocks are called reference pixels. Among the 35 candidate prediction modes, the DC prediction mode is applicable to areas with flat textures in the current block. All pixels in this area use the average value of the reference pixels in the reference block as prediction; the plane prediction mode is applicable to image blocks with smoothly changing textures. The current block that meets this condition uses the reference pixels in the reference block for bilinear interpolation as the prediction of all pixels in the current block; the angular prediction mode uses the characteristic that the texture of the current block is highly correlated with the texture of the adjacent reconstructed image blocks, and copies the values ​​of the reference pixels in the corresponding reference block along a certain angle as the prediction of all pixels in the current block.

[0173] The HEVC encoder selects an optimal intra-frame prediction mode for the current block from 35 candidate prediction modes and writes this optimal intra-frame prediction mode into the video bitstream. To improve the coding efficiency of intra-frame prediction, the encoder / decoder derives three most probable modes from the optimal intra-frame prediction modes of the reconstructed image blocks in the surrounding area using intra-frame prediction. If the optimal intra-frame prediction mode selected for the current block is one of these three most probable modes, a first index is encoded to indicate that the selected optimal intra-frame prediction mode is one of these three most probable modes; if the selected optimal intra-frame prediction mode is not one of these three most probable modes, a second index is encoded to indicate that the selected optimal intra-frame prediction mode is one of the other 32 modes (other than the aforementioned three most probable modes among the 35 candidate prediction modes). The HEVC standard uses a 5-bit fixed-length code as the aforementioned second index.

[0174] The HEVC encoder derives the three most probable modes by selecting the optimal intra-frame prediction modes of the left and upper neighboring image blocks of the current block and adding them to a set. If these two optimal intra-frame prediction modes are the same, only one is retained in the set. If these two optimal intra-frame prediction modes are the same and both are angular prediction modes, two angular prediction modes adjacent to the angular direction are selected and added to the set. Otherwise, the planar prediction mode, the DC mode, and the vertical prediction mode are selected and added to the set in sequence until the number of modes in the set reaches three.

[0175] After the HEVC decoder performs entropy decoding on the bitstream, it obtains the mode information of the current block, which includes an indicator indicating whether the optimal intra-frame prediction mode of the current block is among the three most probable modes, and the index of the optimal intra-frame prediction mode of the current block among the three most probable modes or the index of the optimal intra-frame prediction mode of the current block among the other 32 modes.

[0176] Inter-frame prediction

[0177] In a possible implementation, the set of inter-prediction modes depends on the available reference picture (i.e., at least part of the previously decoded picture stored in the DBP 230 as mentioned above) and other inter-prediction parameters, e.g., on whether the entire reference picture is used or only a part of the reference picture is used, e.g., a search window area around the area of ​​the current block, to search for the best matching reference block, and / or on whether pixel interpolation such as half-pixel, quarter-pixel and / or 1 / 16 interpolation is performed, for example.

[0178] In addition to the above prediction modes, skip mode and / or direct mode may also be employed.

[0179] For example, in extended merge prediction, the merge candidate list of this mode consists of the following five candidate types in order: spatial MVP from spatially neighboring CUs, temporal MVP from collocated CUs, history-based MVP from the FIFO table, pairwise average MVP, and zero MV. Decoder-side motion vector refinement (DMVR) based on bilateral matching can be used to increase the accuracy of the MV in merge mode. Merge mode with MVD (MMVD) is derived from merge mode with motion vector difference. The MMVD flag is sent immediately after the skip flag and merge flag to specify whether the CU uses MMVD mode. A CU-level adaptive motion vector resolution (AMVR) scheme can be used. AMVR supports encoding the CU's MVD with different precisions. The MVD of the current CU is adaptively selected based on the prediction mode of the current CU. When the CU is encoded in merge mode, the combined inter / intra prediction (CIIP) mode can be applied to the current CU. The CIIP prediction is obtained by weighted averaging the inter and intra prediction signals. For affine motion compensation prediction, the affine motion field of the block is described by the motion information of 2 control points (4 parameters) or 3 control points (6 parameters) motion vectors. Subblock-based temporal motion vector prediction (SbTMVP) is similar to the temporal motion vector prediction (TMVP) in HEVC, but it predicts the motion vector of the sub-CU within the current CU. Bidirectional optical flow (BDOF), formerly known as BIO, is a simplified version that reduces calculations, especially in terms of the number of multiplications and the size of the multipliers. In the triangle partitioning mode, the CU is evenly divided into two triangular parts using diagonal partitioning and anti-diagonal partitioning. In addition, the bidirectional prediction mode is extended based on the simple average to support the weighted average of the two prediction signals.

[0180] The inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both not shown in FIG. 2 ). The motion estimation unit may be configured to receive or obtain an image block 203 (the current image block 203 of the current image 17 ) and a decoded image 231 , or at least one or more previously reconstructed blocks, e.g., reconstructed blocks of one or more other / different previously decoded images 231 , to perform motion estimation. For example, a video sequence may include the current image and the previously decoded image 231 , or in other words, the current image and the previously decoded image 231 may be part of or form a sequence of images forming the video sequence.

[0181] For example, the encoder 20 may be configured to select a reference block from a plurality of reference blocks of the same or different images in a plurality of other images, and provide the reference image (or reference image index) and / or the offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. This offset is also referred to as a motion vector (MV).

[0182] The motion compensation unit is configured to obtain, for example, receive, inter-frame prediction parameters and perform inter-frame prediction based on or using the inter-frame prediction parameters to obtain an inter-frame prediction block 246. The motion compensation performed by the motion compensation unit may include extracting or generating a prediction block based on a motion / block vector determined by motion estimation, and may also include performing interpolation with sub-pixel accuracy. Interpolation filtering can generate pixel points of other pixels from pixel points of known pixels, thereby potentially increasing the number of candidate prediction blocks that can be used to encode the image block. Upon receiving a motion vector corresponding to a PU of the current image block, the motion compensation unit may locate the prediction block pointed to by the motion vector in one of the reference picture lists.

[0183] The motion compensation unit may also generate syntax elements associated with blocks and video slices for use by video decoder 30 when decoding image blocks of a video slice. In addition to, or in lieu of, slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be generated or used.

[0184] In the process of obtaining a candidate motion vector list in an advanced motion vector prediction (AMVP) mode, motion vectors (MVs) that can be added to the candidate motion vector list as alternatives include MVs of spatially adjacent and temporally adjacent image blocks of the current block, wherein the MVs of the spatially adjacent image blocks may include the MV of a left candidate image block located on the left side of the current block and the MV of an upper candidate image block located above the current block.

[0185] After obtaining the candidate motion vector list, the optimal MV is determined from the candidate motion vector list using the rate distortion cost (RD cost). The candidate motion vector with the smallest RD cost is used as the motion vector predictor (MVP) for the current block. The rate distortion cost is calculated using the following formula: J = SAD + λR

[0186] Wherein, J represents RD cost, SAD is the sum of absolute differences (SAD) between the pixel values ​​of the predicted block obtained after motion estimation using the candidate motion vector and the pixel values ​​of the current block, R represents the bit rate, and λ represents the Lagrange multiplier.

[0187] The encoder passes the index of the determined MVP in the candidate motion vector list to the decoder. Furthermore, a motion search can be performed within a neighborhood centered on the MVP to obtain the actual motion vector of the current block. The encoder calculates the motion vector difference (MVD) between the MVP and the actual motion vector and also passes the MVD to the decoder. The decoder parses the index, finds the corresponding MVP in the candidate motion vector list based on the index, parses the MVD, and adds the MVD to the MVP to obtain the actual motion vector of the current block.

[0188] When obtaining the candidate motion information list in Merge mode, the motion information that can be added to the candidate motion information list includes the motion information of spatially or temporally adjacent image blocks to the current block. The spatially adjacent candidate motion information in the candidate motion information list comes from five spatially adjacent blocks (A0, A1, B0, B1, and B2). If the spatially adjacent blocks are unavailable or are intra-predicted, their motion information is not added to the candidate motion information list. The temporal candidate motion information for the current block is obtained by scaling the MV of the corresponding block in the reference frame based on the picture order count (POC) of the reference frame and the current frame. The block at position T in the reference frame is first determined to be available. If not, the block at position C is selected. After obtaining the candidate motion information list, the optimal motion information from the candidate motion information list is determined using the RD cost as the motion information for the current block. The encoder transmits the index of the optimal motion information in the candidate motion information list (denoted as the merge index) to the decoder.

[0189] Entropy Coding

[0190] The entropy coding unit 270 is configured to apply an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC (CALVC) scheme, an arithmetic coding scheme, a binarization algorithm, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to the quantized residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements, resulting in coded image data 21 that can be output via an output terminal 272 in the form of a coded bitstream 21, etc., so that the video decoder 30, etc. can receive and use the parameters for decoding. The coded bitstream 21 can be transmitted to the video decoder 30 or stored in a memory for later transmission or retrieval by the video decoder 30.

[0191] Other structural variations of the video encoder 20 may be used to encode the video stream. For example, a non-transform based encoder 20 may directly quantize the residual signal without a transform processing unit 206 for certain blocks or frames. In another implementation, the encoder 20 may have the quantization unit 208 and the inverse quantization unit 210 combined into a single unit.

[0192] Decoder and decoding method

[0193] FIG3 is an exemplary block diagram of a video decoder 30 according to an embodiment of the present application. The video decoder 30 is configured to receive coded image data 21 (e.g., coded bitstream 21), for example, encoded by the encoder 20, and generate a decoded image 331. The coded image data or bitstream includes information for decoding the coded image data, such as data representing image blocks of a coded video slice (and / or coding block group or coding block) and associated syntax elements.

[0194] 3 , decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., summer 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an inter-prediction unit 344, and an intra-prediction unit 354. Inter-prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform a decoding process that is generally the reverse of the encoding process described with reference to video encoder 100 of FIG. 2 .

[0195] As described with respect to encoder 20, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer DPB 230, inter-prediction unit 344, and intra-prediction unit 354 also constitute the "built-in decoder" of video encoder 20. Accordingly, inverse quantization unit 310 may be functionally identical to inverse quantization unit 110, inverse transform processing unit 312 may be functionally identical to inverse transform processing unit 122, reconstruction unit 314 may be functionally identical to reconstruction unit 214, loop filter 320 may be functionally identical to loop filter 220, and decoded picture buffer 330 may be functionally identical to decoded picture buffer 230. Therefore, the explanations of the corresponding units and functions of video encoder 20 apply accordingly to the corresponding units and functions of video decoder 30.

[0196] Entropy decoding

[0197] Entropy decoding unit 304 is configured to parse bitstream 21 (or generally, coded image data 21) and perform entropy decoding on the coded image data 21 to obtain quantization coefficients 309 and / or decoded coding parameters (not shown in FIG. 3 ), such as any or all of inter-frame prediction parameters (e.g., reference image indexes and motion vectors), intra-frame prediction parameters (e.g., intra-frame prediction modes or indices), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. Entropy decoding unit 304 may be configured to apply a decoding algorithm or scheme corresponding to the coding scheme of entropy coding unit 270 of encoder 20. Entropy decoding unit 304 may also be configured to provide inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to mode application unit 360, as well as to provide other parameters to other units of decoder 30. Video decoder 30 may receive syntax elements at the video slice and / or video block level. In addition to, or in lieu of, slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may also be received or used.

[0198] Dequantization

[0199] The inverse quantization unit 310 may be configured to receive a quantization parameter (QP) (or generally information related to inverse quantization) and quantization coefficients from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and inverse quantize the decoded quantization coefficients 309 based on the quantization parameter to obtain inverse quantization coefficients 311, which may also be referred to as transform coefficients 311. The inverse quantization process may include using the quantization parameter calculated by the video encoder 20 for each video block in the video slice to determine a degree of quantization, and thus a degree of inverse quantization to be performed.

[0200] Inverse transform

[0201] The inverse transform processing unit 312 may be configured to receive the dequantized coefficients 311, also referred to as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain a reconstructed residual block 213 in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 313. The transform may be an inverse transform, such as an inverse DCT, an inverse DST, an inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 may also be configured to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304) to determine the transform to be applied to the dequantized coefficients 311.

[0202] reconstruction

[0203] The reconstruction unit 314 (eg, summer 314 ) is configured to add the reconstructed residual block 313 to the prediction block 365 to obtain the reconstructed block 315 in the pixel domain, eg, by adding the pixel values ​​of the reconstructed residual block 313 and the pixel values ​​of the prediction block 365 .

[0204] Filtering

[0205] The loop filter unit 320 (in or after the encoding loop) is used to filter the reconstructed block 315 to obtain a filter block 321, thereby smoothly performing pixel conversion or improving video quality. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process can be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop reshaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 320 is shown as a loop filter in Figure 3, in other configurations, the loop filter unit 320 can be implemented as a post-loop filter.

[0206] Decoded Image Buffer

[0207] The decoded video blocks 321 of one picture are then stored in a decoded picture buffer 330, which stores the decoded picture 331 as a reference picture for subsequent motion compensation of other pictures and / or for respective output displays.

[0208] The decoder 30 is used to output the decoded image 311 through the output terminal 312, etc., for display to the user or for the user to view.

[0209] predict

[0210] The inter-frame prediction unit 344 may be functionally identical to the inter-frame prediction unit 244 (particularly the motion compensation unit), and the intra-frame prediction unit 354 may be functionally identical to the inter-frame prediction unit 254 and may determine the partitioning or segmentation and perform prediction based on the segmentation and / or prediction parameters or corresponding information received from the coded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304). The mode application unit 360 may be configured to perform prediction (intra-frame or inter-frame prediction) for each block based on the reconstructed image, block, or corresponding pixel point (filtered or unfiltered), resulting in a prediction block 365.

[0211] When the video slice is encoded as an intra-coded (I) slice, the intra-prediction unit 354 in the mode application unit 360 is configured to generate a prediction block 365 for the image block of the current video slice based on the indicated intra-prediction mode and data from previously decoded blocks of the current image. When the video image is encoded as an inter-coded (i.e., B or P) slice, the inter-prediction unit 344 (e.g., a motion compensation unit) in the mode application unit 360 is configured to generate a prediction block 365 for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 304. For inter-prediction, these prediction blocks may be generated from one of the reference pictures in one of the reference picture lists. Video decoder 30 may construct reference frame list 0 and list 1 using a default construction technique based on the reference pictures stored in DPB 330. The same or similar processes may be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks) in addition to or instead of slices (e.g., video slices), e.g., video may be encoded using I, P, or B coding block groups and / or coding blocks.

[0212] Mode application unit 360 is configured to determine prediction information for video blocks of a current video slice by parsing motion vectors and other syntax elements, and to use the prediction information to generate a prediction block for the current video block being decoded. For example, mode application unit 360 uses received syntax elements to determine the prediction mode (e.g., intra prediction or inter prediction) used to encode the video blocks of the video slice, the inter prediction slice type (e.g., B slice, P slice, or GPB slice), construction information for one or more reference picture lists for the slice, motion vectors for each inter-coded video block in the slice, inter prediction status for each inter-coded video block in the slice, and other information to decode the video blocks within the current video slice. In addition to or in lieu of slices (e.g., video slices), the same or similar processes may be applied to embodiments involving coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), for example, where video may be encoded using I, P, or B coding block groups and / or coding blocks.

[0213] In one embodiment, the video encoder 30 of FIG3 may also be configured to partition and / or decode an image using slices (also referred to as video slices), where an image may be partitioned or decoded using one or more (typically non-overlapping) slices. Each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., coding blocks in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0214] In one embodiment, the video decoder 30 shown in Figure 3 can also be used to segment and / or decode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), where the image can be segmented or decoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTUs) or one or more coding blocks, etc., where each coding block can be in a shape such as a rectangle and may include one or more complete or partial blocks (e.g., CTUs).

[0215] Other variations of the video decoder 30 may be used to decode the encoded image data 21. For example, the decoder 30 may generate an output video stream without the loop filter unit 320. For example, a non-transform-based decoder 30 may directly inverse quantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another implementation, the video decoder 30 may have the inverse quantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0216] It should be understood that the processing result of the current step can be further processed in the encoder 20 and the decoder 30 and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, the processing result of interpolation filtering, motion vector derivation, or loop filtering can be further operated on, such as clipping or shifting operations.

[0217] It should be noted that further operations can be performed on the derived motion vector of the current block (including but not limited to the control point motion vector of the affine mode, the sub-block motion vector of the affine, planar, ATMVP mode, the temporal motion vector, etc.). For example, the value of the motion vector is limited to a predefined range based on the representation bit of the motion vector. If the representation bit of the motion vector is bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents a power. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of four 4×4 sub-blocks in an 8×8 block) is limited so that the maximum difference between the integer parts of the four 4×4 sub-block MVs does not exceed N pixels, for example, not more than 1 pixel. Two methods of limiting the motion vector based on bitDepth are provided here.

[0218] Although the above embodiments primarily describe video coding, it should be noted that embodiments of the decoding system 10, encoder 20, and decoder 30, as well as other embodiments described herein, may also be used for still image processing or coding, i.e., processing or coding a single image in a video codec that is independent of any previous or subsequent images. In general, if image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and the inter-frame prediction unit 344 (decoder) may not be available. All other functionalities (also referred to as tools or techniques) of the video encoder 20 and video decoder 30 are also applicable to still image processing, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354, and / or loop filtering 220 / 320, entropy coding 270, and entropy decoding 304.

[0219] FIG4 is an exemplary block diagram of a video decoding device 400 according to an embodiment of the present application. Video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, video decoding device 400 may be a decoder, such as video decoder 30 in FIG1a , or an encoder, such as video encoder 20 in FIG1a .

[0220] Video decoding device 400 includes: an input port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; for example, processor 430 may be a neural network processor 430; a transmitter unit (Tx) 440 and an output port 450 (or output port 450) for transmitting data; and a memory 460 for storing data. Video decoding device 400 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to input port 410, receiver unit 420, transmitter unit 440, and output port 450 for outputting or transmitting optical or electrical signals.

[0221] The processor 430 is implemented in hardware and software. The processor 430 can be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the input port 410, the receiving unit 420, the transmitting unit 440, the output port 450, and the memory 460. The processor 430 includes a decoding module 470 (e.g., a neural network-based decoding module 470). The decoding module 470 implements the embodiments disclosed above. For example, the decoding module 470 performs, processes, prepares, or provides various encoding operations. Therefore, the decoding module 470 provides substantial improvements to the functionality of the video decoding device 400 and affects the switching of the video decoding device 400 to different states. Alternatively, the decoding module 470 is implemented by instructions stored in the memory 460 and executed by the processor 430.

[0222] Memory 460 includes one or more disks, tape drives, and solid-state drives and can be used as overflow data storage for storing programs when such programs are selected for execution, and for storing instructions and data read during program execution. Memory 460 can be volatile and / or non-volatile and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0223] FIG5 is an exemplary block diagram of an apparatus 500 according to an embodiment of the present application. The apparatus 500 may be used as either or both of the source device 12 and the destination device 14 in FIG1a.

[0224] The processor 502 in the apparatus 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices, now available or developed in the future, capable of manipulating or processing information. While the disclosed implementations may be implemented using a single processor, such as the processor 502 shown, using more than one processor may provide greater speed and efficiency.

[0225] In one implementation, the memory 504 in the apparatus 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 that are accessed by the processor 502 via a bus 512. The memory 504 may also include an operating system 508 and application programs 510, which include at least one program that allows the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1 through N, as well as a video decoding application that performs the methods described herein.

[0226] The apparatus 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with touch-sensitive elements that can be used to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0227] Although bus 512 in device 500 is described herein as a single bus, bus 512 may include multiple buses. Furthermore, secondary storage may be directly coupled to other components of device 500 or accessed via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, device 500 may have a variety of configurations.

[0228] Since the embodiments of the present application involve the application of neural networks, in order to facilitate understanding, some nouns or terms used in the embodiments of the present application are explained below, and these nouns or terms are also considered part of the content of the invention.

[0229] (1) Neural Network

[0230] A neural network (NN) is a machine learning model. A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0231] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0232] (2) Deep Neural Networks

[0233] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with many hidden layers. The "many" here does not have a specific metric. Based on the position of different layers in a DNN, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0234] (3) Convolutional Neural Networks

[0235] A convolutional neural network (CNN) is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture involves multiple levels of learning at different levels of abstraction using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network in which individual neurons respond to input images. A CNN consists of a feature extractor consisting of convolutional and pooling layers. The feature extractor can be thought of as a filter, and the convolution process can be thought of as convolving an input image or feature map with a trainable filter.

[0236] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution on the input signal. A convolutional layer can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter that extracts specific information from the input image matrix. A convolution operator is essentially a weight matrix, which is usually predefined. During the convolution operation, the weight matrix is ​​typically applied horizontally to the input image, pixel by pixel (or two pixels by two pixels, depending on the stride), to extract specific features from the image. The size of the weight matrix should be proportional to the image size. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during the convolution operation, the weight matrix extends across the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolved output with a single depth dimension. However, in most cases, multiple weight matrices of the same size (row × column) are applied instead. This is known as multiple homogeneous matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. The dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors in the image, and yet another weight matrix is ​​used to blur unwanted noise in the image. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these multiple weight matrices of the same size are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation. In practical applications, the weight values ​​in these weight matrices require extensive training. The weight matrices formed by the trained weight values ​​can be used to extract information from the input image, allowing the convolutional neural network to make accurate predictions. When a convolutional neural network has multiple convolutional layers, the initial convolutional layers often extract more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by subsequent convolutional layers become increasingly complex, such as high-level semantic features. Features with higher semantics are more applicable to the problem being solved.

[0237] Because it's often necessary to reduce the number of trainable parameters, pooling layers are often periodically introduced after convolutional layers. This can be done in a single convolutional layer followed by a pooling layer, or in a multi-layered system followed by one or more pooling layers. In image processing, the sole purpose of a pooling layer is to reduce the spatial size of an image. Pooling layers can include average pooling and / or max pooling operators, which are used to downsample the input image to produce a smaller image. The average pooling operator calculates the average value of pixel values ​​within a specific range, producing the average pooling result. The max pooling operator takes the pixel with the largest value within a specific range as the max pooling result. Furthermore, just as the size of the weight matrix used in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The output image size after processing by a pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding subregion of the image input to the pooling layer.

[0238] After being processed by the convolution layer / pooling layer, the convolutional neural network is still not sufficient to output the required output information. Because as mentioned above, the convolution layer / pooling layer only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network needs to use the neural network layer to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers, and the parameters contained in the multiple hidden layers can be pre-trained based on relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0239] Optionally, after the multiple hidden layers in the neural network layer, an output layer of the entire convolutional neural network is also included. The output layer has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the backpropagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0240] (4) Recurrent Neural Network

[0241] Recurrent neural networks (RNNs) are designed to process sequential data. In traditional neural network models, layers are fully connected, from the input layer to the hidden layer to the output layer, while nodes within each layer are disconnected. While these conventional neural networks have solved many difficult problems, they are still inadequate for many others. For example, to predict the next word in a sentence, you generally need to use the previous words, as the previous and next words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is dependent on the previous output. Specifically, the network memorizes previous information and applies it to the calculation of the current output. This means that nodes within the hidden layer are no longer disconnected but connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the previous hidden layer. In theory, RNNs can process sequence data of any length. Training an RNN is similar to training a traditional CNN or DNN. This approach also uses the backpropagation algorithm, but with one key difference: if the RNN is expanded, its parameters, such as W, are shared; this is not the case with traditional neural networks, as in the example above. Furthermore, when using gradient descent, the output of each step depends not only on the state of the network at the current step but also on the state of the network at several previous steps. This learning algorithm is called backpropagation through time (BPTT).

[0242] Since we already have convolutional neural networks, why do we still need recurrent neural networks? The reason is simple. Convolutional neural networks assume that elements are independent of each other, and that inputs and outputs are also independent, such as cats and dogs. However, in the real world, many elements are interconnected, such as the changes in stock prices over time. Or, for example, someone says, "I love traveling, and my favorite place is Yunnan. I must visit it someday." Humans should know to fill in the blank with "Yunnan." This is because humans make inferences based on context, but how can machines do the same? That's where RNNs come in. RNNs aim to give machines the ability to remember, like humans do. Therefore, the output of an RNN depends on both the current input and historical memory.

[0243] (5) Loss function

[0244] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0245] (6) Backpropagation algorithm

[0246] Convolutional neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during training, reducing the reconstruction error loss of the super-resolution model. Specifically, the forward propagation of the input signal to the output generates an error loss. This error loss information is then backpropagated to update the parameters of the initial super-resolution model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by the error loss, aiming to obtain the optimal super-resolution model parameters, such as the weight matrix.

[0247] (7) Generative Adversarial Networks

[0248] Generative adversarial networks (GANs) are a type of deep learning model. They consist of at least two modules: a generative model and a discriminative model. These two modules learn from each other through interaction to produce better outputs. Both the generative and discriminative models can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of a GAN is as follows: For example, consider a GAN that generates images. Suppose there are two networks, G (Generator) and D (Discriminator). G is the image generator network, which receives random noise z and generates an image from it, denoted as G(z). D is the discriminator network, which determines whether an image is "real." Its input parameter is x, representing an image. Its output, D(x), represents the probability that x is real. A value of 1 indicates a 100% probability of authenticity, while a value of 0 indicates a high probability of non-authenticity. During the training of this generative adversarial network, the goal of the generative network G is to generate realistic images as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the images generated by G from real images as much as possible. This creates a dynamic "game" between G and D, which is the "game" in "generative adversarial network." Ultimately, under ideal conditions, G can generate images G(z) that are sufficiently realistic, while D has difficulty determining whether the images generated by G are real, i.e., D(G(z)) = 0.5. This results in an excellent generative model G that can be used to generate images.

[0249] Before describing the technical solution of this application, the relevant terms and technologies involved in this application are first explained.

[0250] 1. Terminology:

[0251] Information: data to be encoded / decoded, such as a picture, a video, an audio clip, a plain text file, etc.

[0252] Symbol: The basic unit of information, such as a pixel in an image or a character in a plain text file.

[0253] Encoding: The process of converting information into a string of 0s and 1s.

[0254] Code stream: The 0 / 1 string obtained by encoding information.

[0255] Decoding: The process of restoring the code stream to information, which is the inverse process of encoding.

[0256] Prerequisites: Data that is known during encoding / decoding. Using these prerequisites during encoding / decoding can reduce the bitstream length.

[0257] Entropy coding: This method uses the Shannon entropy principle to avoid information loss during the encoding process. Using probabilistic modeling, the code stream length is kept close to the theoretical minimum (Shannon entropy). Entropy coding can also reduce the code stream length by taking advantage of the underlying assumptions.

[0258] Entropy decoding: A method of decoding using probabilistic modeling, which is the inverse of entropy coding. If the premise is used during entropy coding, the same premise must be used during entropy decoding.

[0259] Throughput: The number of symbols that can be encoded / decoded per second by entropy encoding / decoding.

[0260] Register: A memory area used by the processor to temporarily store instructions, data, and so on. It has a small capacity but fast read and write speeds. Current computing devices typically have 8-32 16-bit, 32-bit, or 64-bit registers.

[0261] Memory: The primary storage unit in a computing device. Its capacity is much larger than registers, but its read and write speed is typically more than 10 times that of registers.

[0262] 2. Entropy Coding Method

[0263] 1) Huffman code: Discovered in 1952, this is the oldest and most widely used entropy encoder. Each symbol is represented by a string of 0s and 1s, with the string being shorter as the symbol probability increases. Encoding each symbol requires at least one bit. The latest compression algorithms, such as those for images and video, often require less than one bit per symbol, so Huffman coding is used in newer algorithms. The bitstream is ordered in a first-in-first-out or last-in-last-out manner.

[0264] 2) Arithmetic Coding (AC): Discovered in 1976, it solves the 1-bit bottleneck problem by using interval allocation. Its codeword length can be infinitely close to Shannon entropy, but the encoding / decoding computational complexity is high. The code stream is in first-in, first-out order.

[0265] 3) Asymmetric Digital System (ANS): Discovered in 2013, this is another encoder with a codeword length that approaches Shannon entropy. Unlike arithmetic coding, ANS primarily utilizes integer state transitions for encoding and decoding. ANS has two main implementations: rANS and tANS. These typically achieve compression rates close to AC with less computational effort. The code stream is in first-in, last-out order.

[0266] 4) Skipping Symbols (SKIP) Encoding / Decoding: During encoding, a probability estimate is made based on the preconditions to determine whether each symbol should be encoded or decoded. If the probability of a symbol taking a certain value exceeds a threshold, the symbol is assumed to have that value and is not encoded or decoded. SKIP can reduce the code length and increase the speed, but it may cause a small number of symbols in the decoded information to be inconsistent with the original information. Therefore, it is only suitable for lossy compression.

[0267] 5) Two-stage encoding / decoding method for long-tail data (INBOUND+OUTBOUND)

[0268] When encoding symbols, there are two approaches:

[0269] A. Stream first-in, first-out

[0270] i. If the symbol is the first symbol, replace it with a specific tag value (for example, 2). Long-tail data usually has a small probability (the probability can refer to the probability of taking a specific value (for example, 0 or 1)), but a large data range (also referred to as the amount of data);

[0271] ii. INBOUND encoding: The non-long-tail symbols are probabilistically modeled together with the tag value of the first symbol, and then encoded;

[0272] iii. OUTBOUND encoding: For the first symbol, encode it using the long-tail encoding method.

[0273] B. Stream first in, last out

[0274] i. If the symbol is the first symbol, OUTBOUND encoding: for the first symbol, encode it using the long-tail encoding method and replace the first symbol with a specific tag value (for example, 2);

[0275] ii. INBOUND encoding: The non-long-tail symbols are probability modeled together with the tag value of the first symbol, and then encoded.

[0276] When decoding the code stream, first perform INBOUND encoding, then OUTBOUND encoding

[0277] i. INBOUND decoding: Probabilistic modeling is performed on the non-long-tail symbols together with the tag value of the first symbol, and then decoding is performed;

[0278] ii. OUTBOUND decoding: For the first symbol, the long-tail decoding method is used to decode the code stream.

[0279] This method can effectively reduce the code length.

[0280] 3. Entropy Coding

[0281] A commonly used entropy coding technique. The encoding process mainly includes two steps: (1) for each symbol, the probability model of the symbol is determined based on the context information, usually called the context model; (2) the symbol is encoded according to the probability value in the context model. Correspondingly, the decoding process mainly includes three steps: (1) for each symbol, the context model of the symbol is determined based on the context information; (2) the code stream of the symbol is decoded based on the probability value in the context model.

[0282] Statistically, the closer a symbol's distribution is to uniformity, the more bits are required to encode it; the closer it is to a constant, the fewer bits are required to encode it. By selecting a probability model group from multiple probability models based on context information as a context model, the total probability of a symbol being 1 can be split into conditional probabilities of the symbol being 1 under various circumstances. Each conditional probability can be closer to 0 or 1 than the total probability, thereby reducing the number of encoded bits and improving the efficiency of entropy coding.

[0283] Based on this, the present application provides an entropy encoding / decoding method and device, which can reduce the time consumption of reading and writing memory, thereby improving encoding / decoding efficiency.

[0284] FIG6 is a flow chart of process 600 of the entropy coding method provided in an embodiment of the present application. Process 600 can be executed by an electronic device such as a computer or a server as an encoding end, or by a chip in the aforementioned electronic device. For example, it can be executed by the video encoder 20 described above. Process 600 is described as a series of steps or operations. It should be understood that process 600 can be executed in various orders and / or occur simultaneously, and is not limited to the execution order shown in FIG6. Process 600 may include:

[0285] Step 601: Acquire multiple symbols of the object to be processed.

[0286] In this application, the objects to be processed may include the following forms:

[0287] (1) A file type or storage standard. For example, images, videos, audio, etc.

[0288] (2) A data transmission protocol. For example, online video, live broadcast, end-cloud collaborative content, etc.

[0289] (3) A compression software or storage device, such as audio and video compression software.

[0290] It should be noted that in addition to the above-mentioned forms, the objects to be processed in this application may also include other forms, the common point of which is that they involve entropy coding / decoding technology. Therefore, this application does not make specific limitations on the objects to be processed.

[0291] The object to be processed can be composed of symbols. For example, a video consists of a sequence of frames, each frame consisting of multiple pixels, so the symbols of the video can be pixels. Another example is that a plain text file consists of multiple characters (which can be Chinese characters, letters, numbers, etc.), so the symbols of the plain text file can be characters. And so on. Symbols are the basic units that make up information (data to be encoded / decoded).

[0292] Step 602: Group multiple symbols, with each group containing N symbols.

[0293] The object to be processed can include many symbols. If these symbols are encoded together, the encoding process will be limited by register space, and the relevant data can only be stored in and read from memory. However, multiple memory reads and writes increase the time consumption. Therefore, to reduce the time consumption and improve the encoding efficiency, these symbols can be grouped, so that the number of symbols in a group is small, and the subsequent steps are processed in groups.

[0294] In this application, a group includes N symbols, where N is an integer greater than 1, for example, N = 2, 4, 5, 8, etc. These symbols can be grouped into the same group in order of N symbols based on their index. If the code stream is in first-in, first-out mode, the grouping can be done in ascending order of index; if the code stream is in first-in, last-out mode, the grouping can be done in descending order of index. Optionally, all symbols can be obtained at once before grouping, or one or more symbols can be grouped as soon as they are obtained. This application does not specifically limit this.

[0295] Step 603: Obtain M symbols from the current group.

[0296] The current group can be the group currently being processed by the encoder during the encoding process. The current group is the period from the first symbol of a group to the Nth symbol of the group. As described in step 602, the current group includes N symbols.

[0297] In a possible implementation, the M symbols are symbols for which encoding operations are not skipped among the N symbols of the current group.

[0298] Based on this, the M symbols may include the following situations:

[0299] (1) 1≤M<N, indicating that some (one or more) symbols in the current group do not skip the encoding operation, so M symbols are some (one or more) symbols in the current group; correspondingly, NM symbols in the current group except the aforementioned M symbols all skip the encoding operation.

[0300] (2) M=N, indicating that all (N) symbols in the current group do not skip the encoding operation, so M symbols are all symbols in the current group; accordingly, there is no symbol in the current group that skips the encoding operation.

[0301] (3) M=0, indicating that all (N) symbols in the current group are skipped for encoding, and therefore there are no M symbols. In this case, the encoding process for the current group is completed, and there is no need to execute steps 603 and 604.

[0302] Optionally, the encoder can determine whether to skip encoding operations for each of the N symbols in the current group based on a premise (ie, data known during encoding / decoding, such as syntax elements). This operation is performed at the symbol level.

[0303] Step 604: Encode the M symbols to obtain a code stream.

[0304] The first symbol may be a symbol that meets a preset condition among multiple symbols of the object to be processed. The preset condition may include a low probability and a large number. For example, if the probability of a certain symbol is less than a probability threshold and the proportion of the symbol in all symbols exceeds a proportion threshold, then the symbol can be determined to be the first symbol. The aforementioned probability may refer to the probability that the symbol takes a specific value (e.g., 0 or 1), and reference may be made to the technical description of entropy coding above.

[0305] The code stream includes two situations: first-in-last-out and first-in-first-out. At the decoding end, decoding is usually performed first using the first decoding method and then using the second decoding method. Therefore, in order to ensure that the decoding end decodes correctly, the encoding end can determine the encoding order according to the two situations of the aforementioned code stream, that is, when first-in-first-out, the encoding end first encodes using the first encoding method and then encodes using the second encoding method; when first-in-last-out, the encoding end first encodes using the second encoding method and then encodes using the first encoding method. Optionally, the aforementioned first encoding can be the INBOUND encoding described above, and the aforementioned second encoding can be the OUTBOUND encoding described above. It should be noted that in the OUTBOUND encoding, there is no specific limitation on the encoder of the first symbol, and in the INBOUND encoding, there is no specific limitation on the encoder of the second symbol (the symbol that does not meet the preset conditions among the multiple symbols of the object to be processed) and / or the preset value. The encoders of the two can be the same or different.

[0306] In one possible implementation, the M symbols include one or more first symbols that meet a preset condition. In this case, the first symbols are replaced with preset values ​​in the M symbols to obtain M processed symbols; the M processed symbols are encoded using a first encoding method to obtain a first code stream; the one or more first symbols are encoded using a second encoding method to obtain a second code stream; and the first code stream and the second code stream form a code stream.

[0307] In this scenario, for a first-in, first-out (FIFO) code stream, the first symbol of the M symbols can be replaced with a preset value (e.g., 2) to obtain M processed symbols. In this way, the second symbol of the M processed symbols retains its original value, while the first symbol becomes the preset value. If there are multiple first symbols in the M symbols, the encoder can replace all of them with the preset value.

[0308] The M processed symbols are then encoded using the first encoding method to obtain a first code stream. The encoding end can encode the M processed symbols together using the first encoding method. In this way, the first code stream can be written to the register at once, avoiding wasting register space. If the register space is insufficient, it is written to the memory (FLUSH).

[0309] The first symbol is then encoded using the second encoding method to obtain a second code stream. If there are multiple first symbols among the M symbols, the encoder can encode these multiple first symbols together using the second encoding method. In this way, the second code stream can be written to the register at once, avoiding wasting register space. If the register space is insufficient, it is written to the memory (FLUSH).

[0310] In one possible implementation, the M symbols include one or more first symbols that meet a preset condition. In this case, the one or more first symbols are encoded using the second encoding method to obtain a third code stream; among the M symbols, the first symbols are replaced with preset values ​​to obtain M processed symbols; the M processed symbols are encoded using the first encoding method to obtain a fourth code stream; and the third code stream and the fourth code stream form a code stream.

[0311] This scenario corresponds to a first-in, last-out code stream. For the first symbol among the M symbols, the second encoding method can be used to encode the first symbol first to obtain the third code stream. If there are multiple first symbols among the M symbols, the encoder can encode all of them together using the second encoding method. This allows the third code stream to be written to the register all at once, avoiding wasted register space. If register space is insufficient, the code stream is written to the memory (FLUSH).

[0312] Then, the aforementioned first symbol is replaced with a preset value (eg, 2) to obtain M processed symbols, so that the second symbol in the M processed symbols still retains the original value, and the first symbol becomes the preset value.

[0313] The M processed symbols are then encoded using the first encoding method to obtain a fourth code stream. The encoding end can encode the M processed symbols together using the first encoding method. In this way, the fourth code stream can be written to the register at once, avoiding wasting register space. If the register space is insufficient, it is written to the memory (FLUSH).

[0314] For example, in the register, the first code can write a maximum of 17 bits of code stream, and the second code can write a maximum of 8 bits of code stream. If the register is 32 bits, encoding one character requires a flush; if the register is 64 bits, encoding two characters requires a flush, which increases the number of memory read operations and increases the time consumption. When N=4, if the register is 32 bits, the four symbols are encoded together with the first and second codes, which is equivalent to only flushing once for four times of the second code and flushing once for one time of the first code. When the probability of the first symbol is small (for example, less than 1%), the average number of flush executions for the four symbols is 1.04, which is significantly less than the aforementioned four flushes, and the number of memory operations is significantly reduced.

[0315] Furthermore, when the first symbol is not included in the M symbols, the M symbols are encoded using the first encoding method to obtain a code stream. That is, if the first symbol is not included in the M symbols, then there is no need to replace the first symbol with a preset value, and the M symbols can be directly encoded using the first encoding method.

[0316] In this embodiment, by encoding N symbols in groups (obtained by grouping the symbols of the objects to be processed) (including using the first encoding method and the second encoding method), the number of memory read and write times can be reduced and the encoding efficiency can be improved.

[0317] FIG7 is a flow chart of process 700 of the entropy decoding method provided in an embodiment of the present application. Process 700 can be executed by a computer, terminal, or other electronic device serving as a decoding end, or by a chip in the aforementioned electronic device. For example, it can be executed by the video decoder 30 described above. Process 700 is described as a series of steps or operations. It should be understood that process 700 can be executed in various orders and / or occur simultaneously, and is not limited to the execution order shown in FIG7. Process 700 may include:

[0318] Step 701: Obtain a code stream, where the code stream corresponds to multiple symbols of an object to be reconstructed. The multiple symbols are divided into multiple groups, each group includes N symbols, and among the N symbols, there are M symbols for which encoding operations are not skipped and D symbols for which encoding operations are skipped.

[0319] The decoder can decode after receiving the entire bitstream, or it can decode while receiving the bitstream, without specific restrictions. Using the prerequisites in the bitstream (i.e., data known during encoding and decoding, such as syntax elements), the decoder knows how many bits of the bitstream correspond to each symbol, and can therefore obtain the bitstream corresponding to N symbols for subsequent processing.

[0320] Corresponding to the encoding end, the decoding end may also have the concept of a group, where a group includes N symbols, where N is an integer greater than 1, for example, N=2, 4, 5, 8, etc.

[0321] The decoder can determine which of the N symbols in each group have their coding operations skipped and which have their coding operations not skipped based on a precondition (i.e., data known at both encoding and decoding time, such as syntax elements). This operation is performed at the symbol level. Based on this operation, the decoder determines that there are M symbols in the N symbols for which coding operations are not skipped and D symbols for which coding operations are skipped, where 1≤M≤N and M+D=N.

[0322] Step 702: Decode the code stream to obtain M symbols.

[0323] In the present application, the code stream can be decoded using the first decoding method to obtain M pre-processing symbols; when the M pre-processing symbols include M' second preset values, the code stream can be decoded using the second decoding method to obtain M' symbols (corresponding to the first symbol, satisfying the preset condition), 1≤M'≤M; in the M pre-processing symbols, the M' second preset values ​​are replaced with M' symbols to obtain M post-processing symbols (including the first symbol and the second symbol (not satisfying the preset condition)); the M post-processing symbols are used as M symbols; when the M pre-processing symbols do not include M' second preset values, the M pre-processing symbols are used as M symbols.

[0324] Optionally, the first decoding method may be the inbound decoding described above, and the second decoding method may be the outbound decoding described above. It should be noted that there is no specific restriction on the decoder of the bitstream in outbound decoding, and there is no specific restriction on the decoder of the bitstream in inbound decoding; the decoders of the two methods may be the same or different.

[0325] The above-mentioned preset conditions may include low probability and large quantity, for example, the probability of a certain symbol is less than a probability threshold, and the proportion of the symbol in all symbols exceeds a proportion threshold.

[0326] The decoding end decodes the bitstream using the first decoding method to obtain M pre-processed symbols. The M pre-processed symbols may include only the second symbol. In this case, the M pre-processed symbols are the aforementioned M symbols. Step 702 ends at this point, and step 703 proceeds. The decoding end may decode the bitstream of the M pre-processed symbols simultaneously using the first decoding method. In this way, the bitstream can be written to the register at once, avoiding wasted register space. If register space is insufficient, the bitstream is written to the memory (FLUSH).

[0327] The M pre-processed symbols may also include a second symbol and M' second preset values. In this case, the decoder decodes the code stream using the second decoding method to obtain M' symbols. The M' symbols are the original values ​​of the first symbol in the original M symbols. The decoder replaces the M' second preset values ​​with the M' symbols to obtain the aforementioned M symbols. The decoder can decode the code stream of M' symbols together using the second decoding method. In this way, the code stream can be written to the register at once, avoiding wasted register space. If register space is insufficient, the code stream is written to the memory (FLUSH).

[0328] Step 703: Set D symbols to a first preset value.

[0329] Depending on the values ​​of M and D, the following situations can be included:

[0330] 1) M=N, D=0, indicating that the encoding operation is not skipped for the N symbols, and therefore the M (ie, N) symbols can all be obtained according to the code stream. In this case, step 703 is not executed.

[0331] 2) 1≤M<N, D>0, indicating that some of the N symbols do not skip the encoding operation, while another part of the symbols skip the encoding operation. Therefore, the M symbols can be obtained according to the code stream, and the remaining D symbols can be obtained through step 703.

[0332] 3) M=0, D=N, indicating that the encoding operation is skipped for the N symbols. The N symbols do not need to be obtained according to the code stream. In this case, step 702 is skipped and step 703 is directly executed.

[0333] Based on this, D>0 indicates that there are symbols in the current group whose encoding operation is skipped (ie, the D symbols mentioned above), so the D symbols can be set to a first preset value (eg, 0, -1, etc.) to reconstruct all symbols of the current group.

[0334] In this embodiment, by decoding the code stream of N symbols in groups (obtained by grouping the symbols of the objects to be processed) (including using the first decoding method and the second decoding method), the number of memory read and write times can be reduced and the encoding efficiency can be improved.

[0335] The following uses several specific embodiments to explain in detail the technical solutions of the method embodiments shown in FIG6 and FIG7 .

[0336] FIG8 is a flowchart of the present application after skipping the encoding operation and encoding fusion. As shown in FIG8 , the encoding end performs the following steps:

[0337] 1. Before encoding, the premise (data known during encoding and decoding) and the original information (data to be encoded) are included. For example, Table 1 exemplarily lists the premise and the original information, as well as the corresponding relationship between the two:

[0338] Table 1

[0339] As shown in Table 1, the premise includes 1, skip, 3, skip, 2, 1, skip, skip, and the original information includes the symbols -1, 0, 1, 0, 0, 2, 0, 0. The premise in each column corresponds to the original information. When the premise is skip, it means that the corresponding original information skips the encoding operation. For the convenience of description, the serial number row is added. This serial number row does not belong to the information to be encoded. For example, if the serial number is 2 and the premise is skip, then the corresponding original information (0) skips the encoding operation; if the serial number is 4 and the premise is skip, then the corresponding original information (0) skips the encoding operation; if the serial number is 7 and the premise is skip, then the corresponding original information (0) skips the encoding operation; if the serial number is 8 and the premise is skip, then the corresponding original information (0) skips the encoding operation.

[0340] 2. In this embodiment, the current group includes 8 (N=8) symbols (-1, 0, 1, 0, 0, 2, 0, 0). According to Table 1, the encoding operation (SKIP) is skipped for the symbols with sequence numbers 2, 4, 7, and 8 among the 8 symbols, and 4 (M=4) symbols (-1, 1, 0, 2) are obtained.

[0341] 3. Encode the above four symbols to obtain a code stream.

[0342] Assuming that 2 is the first symbol and -1, 1, and 0 are the second symbols among the four symbols, one encoding process is as follows:

[0343] 1) Replace 2 with 3 (the default value) to obtain 4 processed symbols (-1, 1, 0, 3);

[0344] 2) Encoding the above four processed symbols using the INBOUND encoding method to obtain a first code stream;

[0345] 3) Encode 2 using OUTBOUND encoding to obtain a second bit stream;

[0346] 4) The first code stream and the second code stream form a code stream (101…01).

[0347] Another encoding process is as follows:

[0348] 1) Encode 2 using OUTBOUND encoding to obtain a third bit stream;

[0349] 2) Replace 2 with 3 (the default value) to obtain 4 processed symbols (-1, 1, 0, 3);

[0350] 3) Encoding the above four processed symbols using the INBOUND encoding method to obtain a fourth code stream;

[0351] 4) The third code stream and the fourth code stream form a code stream (101…01).

[0352] 4. Write the code stream into memory.

[0353] In this embodiment, the results of steps 1-3 do not need to be stored, and the encoding operation (SKIP) and encoding can be skipped directly. Compared with related technologies, the premise and original information need to be read from the memory once, and the code stream needs to be written into the memory once.

[0354] FIG9 is a flowchart of the present application after skipping the encoding operation and decoding fusion. As shown in FIG9 , the decoding end performs the following steps:

[0355] 1. Before decoding, the premise (data known during encoding and decoding) and the code stream (101...01) are included. The premise includes 1, skip, 3, skip, 2, 1, skip, skip. The premise and code stream are consistent with the embodiment shown in Figure 8.

[0356] 2. Decode the code stream to obtain the 8 symbols of the current group:

[0357] 1) Decode the code stream using the INBOUND decoding method to obtain four pre-processing symbols (-1, 1, 0, 3);

[0358] 2) The four pre-processed symbols include the preset value 3, and the code stream is decoded using the OUTBOUND decoding method to obtain 2;

[0359] 3) Replace 3 of the above 4 pre-processed symbols with 2 to obtain 4 symbols (-1, 1, 0, 2);

[0360] 4) According to the premise, the current group has 4 symbols (serial numbers 2, 4, 7, and 8), and the encoding operation is skipped. The 4 symbols are set (i.e., assigned) to the preset value 0, and 8 symbols are obtained (-1, 0, 1, 0, 0, 2, 0, 0), which is consistent with the original information in the embodiment shown in Figure 8.

[0361] In this embodiment, the results of steps 1-3 do not need to be stored and can be directly decoded and reverse SKIPed. Compared with related technologies, the code stream needs to be read from the memory once, and the premise and original information need to be written into the memory once each.

[0362] Table 2 compares the throughput of the Intel i7-1260P CPU in related technologies and this application. It can be seen that the encoding / decoding throughput of this application is improved by an average of 1.5 times, especially when the SKIP probability is small, the improvement can be as high as 2 times.

[0363] Table 2

[0364] Encoding side:

[0365] If the code stream order is FIFO, divide every 4 symbols into a group and perform the following operations respectively:

[0366] For each symbol, if it is determined according to the premise that this symbol does not skip the encoding operation and this symbol is the first symbol, then this symbol (and other similar symbols in the four symbols) are OUTBOUND encoded and this symbol is changed to a tag value;

[0367] For each symbol, if it is determined according to the premise that this symbol does not skip the encoding operation and this symbol is the second symbol, then this symbol (as well as other similar symbols and tag values ​​in the 4 symbols) will be INBOUND encoded.

[0368] If the code stream order is FIFO, divide every 4 symbols into a group and perform the following operations respectively:

[0369] For each symbol, if it is determined according to the premise that this symbol does not skip the encoding operation and this symbol is the first symbol, then change this symbol to a tag value and perform inbound encoding on the tag value (as well as other similar tag values ​​and the second symbol in the four symbols);

[0370] For each symbol, if it is determined based on the premise that the encoding operation for this symbol will not be skipped and this symbol is a long tail, then this symbol (and other similar symbols among the 4 symbols) will be OUTBOUND encoded.

[0371] Optionally, the number of symbols in each group may take different constants depending on the number of registers, the bit width, and the number of OUTBOUND / INBOUND bits.

[0372] Decoding end:

[0373] In the information to be decoded, every 4 symbols form a group, and the following operations are performed respectively:

[0374] For each symbol, if it is determined according to the premise that the encoding operation of this symbol is skipped, then this symbol is set (i.e., assigned a value) to a constant, otherwise this symbol is decoded inbound;

[0375] For each symbol, if it is determined based on the premise that this symbol does not skip the encoding operation, and the value of this symbol obtained by the previous decoding step is the tag value, then this symbol is OUTBOUND decoded and the tag value is replaced.

[0376] Optionally, the number of symbols in each group may take different constants depending on the number of registers, the bit width, and the number of OUTBOUND / INBOUND bits.

[0377] FIG10 is a schematic diagram of the structure of the entropy coding device 1000 of the present application. As shown in FIG10 , the entropy coding device 1000 of this embodiment can be applied to the electronic device at the encoding end. The entropy coding device 1000 may include: an acquisition module 1001, a grouping module 1002, a skipping module 1003 and an encoding module 1004.

[0378] An acquisition module 1001 is used to acquire multiple symbols of an object to be processed; a grouping module 1002 is used to group the multiple symbols, where the number of symbols in each group is N, 1<N; a skipping module 1003 is used to acquire M symbols from the current group, 1≤M≤N; and an encoding module 1004 is used to encode the M symbols to obtain a code stream.

[0379] In one possible implementation, the M symbols include one or more first symbols that meet a preset condition, and the encoding module 1004 is specifically configured to replace the first symbols in the M symbols with preset values ​​to obtain M processed symbols; encode the M processed symbols using a first encoding method to obtain a first code stream; encode the one or more first symbols using a second encoding method to obtain a second code stream; and the first code stream and the second code stream form the code stream.

[0380] In one possible implementation, the M symbols include one or more first symbols that meet a preset condition, and the encoding module 1004 is specifically configured to encode the one or more first symbols using a second encoding method to obtain a third code stream; replace the first symbols in the M symbols with preset values ​​to obtain M processed symbols; encode the M processed symbols using the first encoding method to obtain a fourth code stream; and the third code stream and the fourth code stream form the code stream.

[0381] In a possible implementation, the M symbols do not include a first symbol that meets a preset condition, and the encoding module 1004 is specifically configured to encode the M symbols using a first encoding method to obtain the code stream.

[0382] In a possible implementation, the preset condition includes that the probability of the first symbol is less than a probability threshold, and the proportion of the first symbol in the multiple symbols exceeds a proportion threshold.

[0383] In a possible implementation, the M symbols are symbols for which encoding operations are not skipped among the N symbols of the current group.

[0384] In one possible implementation, N=4.

[0385] The device of this embodiment can be used to execute the technical solution of the method embodiment shown in Figure 6. Its implementation principle and technical effects are similar and will not be repeated here.

[0386] FIG11 is a schematic diagram of the structure of the entropy decoding device 1100 of the present application. As shown in FIG11 , the entropy decoding device 1100 of this embodiment can be applied to the electronic device at the decoding end mentioned above. The entropy decoding device 1100 may include: an acquisition module 1101, a decoding module 1102, and a setting module 1103.

[0387] An acquisition module 1101 is configured to acquire a code stream, where the code stream corresponds to multiple symbols of an object to be reconstructed, the multiple symbols being divided into multiple groups, each group including N symbols, wherein M symbols are not skipped for encoding and D symbols are skipped for encoding, where 1<N, 1≤M≤N, and M+D=N. A decoding module 1102 is configured to decode the code stream to obtain the M symbols. A setting module 1103 is configured to set the D symbols to a first preset value.

[0388] In one possible implementation, the decoding module 1102 is specifically configured to decode the code stream using a first decoding method to obtain M pre-processed symbols; when the M pre-processed symbols include M' second preset values, decode the code stream using a second decoding method to obtain M' symbols, 1≤M'≤M; replace the M' second preset values ​​in the M pre-processed symbols with the M' symbols to obtain M post-processed symbols; use the M post-processed symbols as the M symbols; and when the M pre-processed symbols do not include M' second preset values, use the M pre-processed symbols as the M symbols.

[0389] In one possible implementation, N=4.

[0390] The device of this embodiment can be used to execute the technical solution of the method embodiment shown in Figure 7. Its implementation principle and technical effects are similar and will not be repeated here.

[0391] During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of the hardware in the processor or an instruction in the form of software. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware coding processor, or can be executed by a combination of hardware and software modules in the coding processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0392] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0393] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0394] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0395] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0396] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0397] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0398] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0399] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An entropy coding method, characterized in that: include: Get multiple symbols of the object to be processed; Grouping the multiple symbols, the number of symbols in each group is N, 1<N; Get M symbols from the current group, 1≤M≤N; The M symbols are encoded to obtain a code stream.

2. The method according to claim 1, characterized in that The M symbols include one or more first symbols that meet a preset condition, and encoding the M symbols to obtain a code stream includes: Among the M symbols, replacing the first symbol with a preset value to obtain M processed symbols; Encoding the M processed symbols using a first encoding method to obtain a first code stream; Encoding the one or more first symbols using a second encoding method to obtain a second code stream; The first code stream and the second code stream form the code stream.

3. The method according to claim 1, characterized in that The M symbols include one or more first symbols that meet a preset condition, and encoding the M symbols to obtain a code stream includes: Encoding the one or more first symbols using a second encoding method to obtain a third code stream; Among the M symbols, replacing the first symbol with a preset value to obtain M processed symbols; Encoding the M processed symbols using a first encoding method to obtain a fourth code stream; The third code stream and the fourth code stream form the code stream.

4. The method according to claim 1, characterized in that: The M symbols do not include a first symbol that meets a preset condition, and encoding the M symbols to obtain a code stream includes: The M symbols are encoded using a first encoding method to obtain the code stream.

5. The method according to any one of claims 2 to 4, characterized in that: The preset condition includes that the probability of the first symbol is less than a probability threshold, and the proportion of the first symbol in the multiple symbols exceeds a proportion threshold.

6. The method according to any one of claims 1 to 5, characterized in that The M symbols are symbols for which encoding operations are not skipped among the N symbols of the current group.

7. The method according to any one of claims 1 to 6, characterized in that N＝4。 8. An entropy decoding method, characterized in that: include: Acquire a code stream, the code stream corresponding to a plurality of symbols of a to-be-reconstructed object, the plurality of symbols being divided into a plurality of groups, each group including N symbols, the N symbols including M symbols for which encoding operations are not skipped and D symbols for which encoding operations are skipped, 1<N, 1≤M≤N, M+D=N; Decoding the code stream to obtain the M symbols; The D symbols are set to first preset values.

9. The method according to claim 8, characterized in that The decoding the code stream to obtain the M symbols includes: Decoding the code stream using a first decoding method to obtain M pre-processing symbols; When the M pre-processed symbols include M' second preset values, decoding the code stream using a second decoding method to obtain M' symbols, 1≤M'≤M; In the M pre-processed symbols, the M' second preset values ​​are replaced by the M' symbols to obtain M post-processed symbols; Using the M processed symbols as the M symbols; When the M pre-processed symbols do not include M′ second preset values, the M pre-processed symbols are used as the M symbols.

10. The method according to claim 8 or 9, characterized in that: N＝4。 11. An entropy coding device, characterized in that: include: An acquisition module, used for acquiring multiple symbols of the object to be processed; A grouping module, used for grouping the multiple symbols, the number of symbols in each group is N, 1<N; A skip module is used to obtain M symbols from the current group, 1≤M≤N; The encoding module is used to encode the M symbols to obtain a code stream.

12. The device according to claim 11, characterized in that The M symbols include one or more first symbols that meet preset conditions, and the encoding module is specifically used to replace the first symbols with preset values ​​in the M symbols to obtain M processed symbols; encode the M processed symbols using a first encoding method to obtain a first code stream; encode the one or more first symbols using a second encoding method to obtain a second code stream; the first code stream and the second code stream form the code stream.

13. The device according to claim 11, characterized in that The M symbols include one or more first symbols that meet preset conditions, and the encoding module is specifically used to encode the one or more first symbols using a second encoding method to obtain a third code stream; among the M symbols, the first symbol is replaced with a preset value to obtain M processed symbols; the M processed symbols are encoded using the first encoding method to obtain a fourth code stream; the third code stream and the fourth code stream form the code stream.

14. The device according to claim 11, characterized in that The M symbols do not include a first symbol that meets a preset condition, and the encoding module is specifically configured to encode the M symbols using a first encoding method to obtain the code stream.

15. The device according to any one of claims 12 to 14, characterized in that The preset condition includes that the probability of the first symbol is less than a probability threshold, and the proportion of the first symbol in the multiple symbols exceeds a proportion threshold.

16. The device according to any one of claims 11 to 15, characterized in that The M symbols are symbols for which encoding operations are not skipped among the N symbols of the current group.

17. The device according to any one of claims 11 to 16, characterized in that N＝4。 18. An entropy decoding device, characterized in that: include: An acquisition module, used for acquiring a code stream, the code stream corresponding to a plurality of symbols of an object to be reconstructed, the plurality of symbols being divided into a plurality of groups, each group including N symbols, the N symbols including M symbols for which encoding operations are not skipped and D symbols for which encoding operations are skipped, 1<N, 1≤M≤N, M+D=N; A decoding module, used for decoding the code stream to obtain the M symbols; A setting module is used to set the D symbols to a first preset value.

19. The device according to claim 18, characterized in that The decoding module is specifically used to decode the code stream using a first decoding method to obtain M pre-processing symbols; when the M pre-processing symbols include M' second preset values, decode the code stream using a second decoding method to obtain M' symbols, 1≤M'≤M; among the M pre-processing symbols, replace the M' second preset values ​​with the M' symbols to obtain M post-processing symbols; use the M post-processing symbols as the M symbols; when the M pre-processing symbols do not include M' second preset values, use the M pre-processing symbols as the M symbols.

20. The device according to claim 18 or 19, characterized in that N＝4。 21. An encoder, characterized in that: The method comprises a processing circuit for executing the method of any one of claims 1-7.

22. A decoder, characterized in that: Comprising a processing circuit for executing the method of any one of claims 8-10.

23. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 10.

24. A non-transitory computer-readable storage medium, characterized in that: The invention comprises a code stream obtained by the entropy coding method according to any one of claims 1 to 7.

25. An encoder, characterized in that: include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program to be judged by the processor, wherein the program, when judged by the processor, causes the encoder to execute the method according to any one of claims 1-7.

26. A decoder, characterized in that include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program judged by the processor, wherein the program, when judged by the processor, causes the decoder to execute the method according to any one of claims 8-10.

27. An image or audio processor, characterized in that: Comprising a processing circuit for performing a method according to any one of claims 1-10.

Images