Coding method and apparatus

Abstract

The application discloses a coding method and device, relates to the technical field of artificial intelligence, and can improve the rate distortion performance of a data coding method. The method comprises the following steps: first, obtaining to-be-coded data; then, inputting the to-be-coded data into a first coding network to obtain target parameters; then, constructing a second coding network according to the target parameters; then, inputting the to-be-coded data into the second coding network to obtain first features; finally, coding the first features to obtain a coded bitstream.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to encoding and decoding methods and apparatus. Background Technology

[0002] With the advancement of technology and the continuous increase in the number and resolution of images, large numbers of images not only require larger storage media, but also wider transmission bandwidth and longer transmission time, which has become a fundamental problem in image processing. In order to improve the storage and transmission efficiency of images, it is necessary to encode large images to compress them.

[0003] Image compression based on neural networks can improve the efficiency of image compression. Existing neural network-based image compression methods are mainly divided into neural network image compression methods that require online training (referred to as Method 1) and neural network image compression methods that do not require online training (referred to as Method 2). Method 1 has better rate-distortion performance, but the image compression speed is slow because it requires online training. Method 2 has poorer rate-distortion performance but faster image compression speed. Summary of the Invention

[0004] This application provides an encoding / decoding method and apparatus that can improve the rate-distortion performance of data encoding / decoding methods without requiring online training. To achieve the above objectives, this application adopts the following technical solution:

[0005] In a first aspect, this application provides an encoding method, which includes: firstly acquiring data to be encoded, then inputting the data to be encoded into a first encoding network to obtain target parameters; then constructing a second encoding network based on the target parameters; next, inputting the data to be encoded into the second encoding network to obtain a first feature; and finally encoding the first feature to obtain an encoded bitstream.

[0006] In existing encoding methods, the encoding network (i.e., the second encoding network) extracts the content features (i.e., the first feature) of the data to be encoded using fixed parameter weights. These content features are then encoded into a bitstream (i.e., the encoded bitstream) and sent to the decoding end. The decoding end then decodes and reconstructs the bitstream to obtain the decoded data. It can be seen that the parameter weights of the encoding network in existing technologies are unrelated to the data to be encoded. However, in the encoding method provided in this application, the data to be encoded is first input into the first encoding network. The first encoding network then generates parameter weights for the second encoding network based on the data to be encoded. These weights are then dynamically adjusted based on the obtained weights, making them relevant to the data to be encoded. This increases the expressive power of the second encoding network, allowing the decoded data reconstructed from the bitstream obtained by the decoding end through the encoding of the first feature to be more closely approximated to the data to be encoded, thereby improving the rate-distortion performance of the encoding and decoding network.

[0007] Optionally, the target parameters are the parameter weights of all or part of the convolutions and nonlinear activations of the second encoding network.

[0008] In one possible implementation, encoding the first feature to obtain an encoded bitstream includes: rounding the first feature to obtain a rounded value of the first feature; performing probability estimation on the rounded value of the first feature to obtain an estimated probability distribution of the rounded value of the first feature; and performing entropy encoding on the rounded value of the first feature based on the estimated probability distribution of the rounded value of the first feature to obtain the encoded bitstream.

[0009] By estimating the probability distribution of the rounded value of the first feature, entropy encoding is performed on the rounded value of the first feature to form a bitstream. This can reduce the coding redundancy of the output first feature and further reduce the amount of data transmitted during the data encoding and decoding (compression) process.

[0010] In one possible implementation, the step of probabilistically estimating the rounded value of the first feature to obtain an estimated probability distribution of the rounded value of the first feature includes: probabilistically estimating the rounded value of the first feature based on first information to obtain an estimated probability distribution of the rounded value of the first feature, wherein the first information includes at least one of context information and edge information.

[0011] By estimating the probability distribution using contextual and edge information, the accuracy of the estimated probability distribution can be improved, thereby reducing the code rate in the entropy coding process and reducing the entropy coding overhead.

[0012] Secondly, this application provides a decoding method, which includes: firstly acquiring a bitstream to be decoded; then decoding the bitstream to obtain a rounded value of a first feature and a rounded value of a second feature; then inputting the rounded value of the second feature into a first decoding network to obtain a target parameter; next, constructing a second decoding network based on the target parameter; and finally inputting the rounded value of the first feature into the second decoding network to obtain decoded data. Wherein, the rounded value of the first feature is used to obtain the decoded data, and the rounded value of the second feature is used to obtain the target parameter.

[0013] In existing decoding methods, the decoding network (i.e., the second decoding network) uses fixed parameter weights to decode and reconstruct the content value features (i.e., the rounded value of the first feature) of the data to be encoded. It can be seen that the parameter weights of the decoding network in the prior art are unrelated to the data to be decoded. In this application, the content features and model features (i.e., the first and second features) of the data to be decoded are encoded into a code stream to be decoded. The decoding end then obtains the rounded value of the second feature by decoding the code stream. This rounded value is input into the first decoding network to obtain the parameter weights of the second decoding network. The parameter weights of the second decoding network are then dynamically adjusted based on these weights, making them relevant to the data to be decoded. This improves the expressive power of the second decoding network and makes the decoded data obtained by the second decoding network more closely resemble the data to be encoded, thereby improving the rate-distortion performance of the encoding / decoding network.

[0014] Optionally, the target parameters are the parameter weights of all or part of the convolutions and nonlinear activations of the second decoding network.

[0015] Optionally, the bitstream to be decoded may include a first bitstream to be decoded and a second bitstream to be decoded.

[0016] In one possible implementation, decoding the bitstream to be decoded to obtain the rounded value of the first feature and the rounded value of the second feature includes: decoding the first bitstream to be decoded to obtain the rounded value of the first feature; and decoding the second bitstream to be decoded to obtain the rounded value of the second feature.

[0017] In one possible implementation, decoding the first bitstream to be decoded to obtain the rounded value of the first feature includes: performing probability estimation on the rounded value of the first feature in the first bitstream to be decoded to obtain an estimated probability distribution of the rounded value of the first feature; and performing entropy decoding on the first bitstream to be decoded based on the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

[0018] In one possible implementation, the step of probabilistically estimating the rounded value of the first feature in the first bitstream to be decoded to obtain an estimated probability distribution of the rounded value of the first feature includes: probabilistically estimating the rounded value of the first feature in the first bitstream to be decoded based on first information to obtain an estimated probability distribution of the rounded value of the first feature, wherein the first information includes at least one of context information and side information.

[0019] In one possible implementation, decoding the second bitstream to be decoded to obtain the rounded value of the second feature includes: performing probability estimation on the rounded value of the second feature in the second bitstream to be decoded to obtain an estimated probability distribution of the rounded value of the second feature; and performing entropy decoding on the second bitstream to be decoded based on the estimated probability distribution of the rounded value of the second feature to obtain the rounded value of the second feature.

[0020] In one possible implementation, the step of probabilistically estimating the rounded value of the second feature in the second bitstream to be decoded to obtain an estimated probability distribution of the rounded value of the second feature includes: probabilistically estimating the rounded value of the second feature in the second bitstream to be decoded based on first information to obtain an estimated probability distribution of the rounded value of the second feature, wherein the first information includes at least one of context information and side information.

[0021] Thirdly, this application provides a decoding method, which includes: firstly acquiring a bitstream to be decoded; then decoding the bitstream to obtain a rounded value of a first feature; subsequently inputting the rounded value of the first feature into a third decoding network to obtain target parameters; next, constructing a second decoding network based on the target parameters; and finally inputting the rounded value of the first feature into the second decoding network to obtain decoded data. The first feature is used to obtain the decoded data and the target parameters.

[0022] In existing decoding methods, the decoding network (i.e., the second decoding network) uses fixed parameter weights to decode and reconstruct the decoded data based on the content value features of the data to be encoded (i.e., the rounded value of the first feature). It can be seen that the parameter weights of the decoding network in existing technologies are unrelated to the data to be decoded. However, in this application, the rounded value of the first feature is obtained by decoding the features of the data to be decoded (i.e., the first feature) into a decoded bitstream. This rounded value is then input into the first decoding network to obtain the parameter weights of the second decoding network. The parameter weights of the second decoding network are then dynamically adjusted based on these weights, making them relevant to the data to be decoded. This improves the expressive power of the second decoding network and makes the decoded data reconstructed by the second decoding network closer to the data to be encoded, thereby improving the rate-distortion performance of the encoding / decoding network.

[0023] Optionally, the target parameters are the parameter weights of all or part of the convolutions and nonlinear activations of the second decoding network.

[0024] In one possible implementation, decoding the bitstream to be decoded to obtain the rounded value of the first feature includes: performing probability estimation on the rounded value of the first feature in the bitstream to be decoded to obtain an estimated probability distribution of the rounded value of the first feature; and performing entropy decoding on the bitstream to be decoded based on the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

[0025] In one possible implementation, the step of probabilistically estimating the rounded value of the first feature in the bitstream to be decoded to obtain an estimated probability distribution of the rounded value of the first feature includes: probabilistically estimating the rounded value of the first feature in the bitstream to be decoded based on first information to obtain an estimated probability distribution of the rounded value of the first feature, wherein the first information includes at least one of context information and side information.

[0026] Fourthly, this application provides an encoding apparatus, which includes a processing circuit for: acquiring data to be encoded; inputting the data to be encoded into a first encoding network to obtain target parameters; constructing a second encoding network based on the target parameters; inputting the data to be encoded into the second encoding network to obtain a first feature; and encoding the first feature to obtain an encoded bitstream.

[0027] Optionally, the target parameters are the parameter weights of all or part of the convolutions and nonlinear activations of the second encoding network.

[0028] In one possible implementation, the processing circuit is specifically configured to: round the first feature to obtain a rounded value of the first feature; perform probability estimation on the rounded value of the first feature to obtain an estimated probability distribution of the rounded value of the first feature; and perform entropy encoding on the rounded value of the first feature according to the estimated probability distribution of the rounded value of the first feature to obtain the encoded bitstream.

[0029] In one possible implementation, the processing circuit is specifically used to: perform probability estimation on the rounded value of the first feature based on first information to obtain an estimated probability distribution of the rounded value of the first feature, wherein the first information includes at least one of context information and edge information.

[0030] Fifthly, this application provides a decoding apparatus, which includes a processing circuit. The processing circuit is configured to: acquire a bitstream to be decoded; decode the bitstream to obtain a rounded value of a first feature and a rounded value of a second feature, wherein the rounded value of the first feature is used to obtain decoded data and the rounded value of the second feature is used to obtain a target parameter; input the rounded value of the second feature into a first decoding network to obtain the target parameter; construct a second decoding network based on the target parameter; and input the rounded value of the first feature into the second decoding network to obtain decoded data.

[0031] Optionally, the target parameters are the parameter weights of all or part of the convolutions and nonlinear activations of the second decoding network.

[0032] Optionally, the bitstream to be decoded includes a first bitstream to be decoded and a second bitstream to be decoded.

[0033] In one possible implementation, the processing circuit is specifically used to: decode the first bitstream to be decoded to obtain the rounded value of the first feature; and decode the second bitstream to be decoded to obtain the rounded value of the second feature.

[0034] In one possible implementation, the processing circuit is specifically configured to: perform probability estimation on the rounded value of a first feature in the first bitstream to be decoded to obtain an estimated probability distribution of the rounded value of the first feature; and perform entropy decoding on the first bitstream to be decoded based on the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

[0035] In one possible implementation, the processing circuit is specifically used to: perform probability estimation on the rounded value of a first feature in the first bitstream to be decoded based on first information to obtain an estimated probability distribution of the rounded value of the first feature, wherein the first information includes at least one of context information and side information.

[0036] In one possible implementation, the processing circuit is specifically used to: perform probability estimation on the rounded value of the second feature in the second bitstream to be decoded to obtain an estimated probability distribution of the rounded value of the second feature; and perform entropy decoding on the second bitstream to be decoded based on the estimated probability distribution of the rounded value of the second feature to obtain the rounded value of the second feature.

[0037] In one possible implementation, the processing circuit is specifically used to: perform probability estimation on the rounded value of the second feature in the second bitstream to be decoded based on first information to obtain an estimated probability distribution of the rounded value of the second feature, wherein the first information includes at least one of context information and side information.

[0038] In a sixth aspect, this application provides a decoding apparatus, which includes a processing circuit for: acquiring a bitstream to be decoded; decoding the bitstream to obtain a rounded value of a first feature, the rounded value of the first feature being used to obtain decoded data and a target parameter; inputting the rounded value of the first feature into a first decoding network to obtain the target parameter; constructing a second decoding network based on the target parameter; and inputting the rounded value of the first feature into the second decoding network to obtain decoded data.

[0039] Optionally, the target parameters are the parameter weights of all or part of the convolutions and nonlinear activations of the second decoding network.

[0040] In one possible implementation, the processing circuit is specifically used to: perform probability estimation on the rounded value of the first feature in the bitstream to be decoded to obtain an estimated probability distribution of the rounded value of the first feature; and perform entropy decoding on the bitstream to be decoded based on the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

[0041] In one possible implementation, the processing circuit is specifically used to: perform probability estimation on the rounded value of a first feature in the bitstream to be decoded based on first information to obtain an estimated probability distribution of the rounded value of the first feature, wherein the first information includes at least one of context information and side information.

[0042] In a seventh aspect, embodiments of this application also provide an encoder, the encoder comprising: at least one processor, which, when the at least one processor executes program code or instructions, implements the method described in the first aspect or any possible implementation thereof.

[0043] Optionally, the encoder may also include at least one memory for storing the program code or instructions.

[0044] Eighthly, embodiments of this application also provide a decoder, the decoder comprising: at least one processor, which, when the at least one processor executes program code or instructions, implements the method described in the second aspect above or any possible implementation thereof.

[0045] Optionally, the decoder may also include at least one memory for storing the program code or instructions.

[0046] In a ninth aspect, embodiments of this application also provide a chip, including: an input interface, an output interface, and at least one processor. Optionally, the chip further includes a memory. The at least one processor is used to execute code in the memory, and when the at least one processor executes the code, the chip implements the method described in the first aspect or any possible implementation thereof.

[0047] Alternatively, the chip described above can also be an integrated circuit.

[0048] In a tenth aspect, embodiments of this application also provide a terminal, which includes the above-described encoding device, decoding device, encoder, decoder, or chip.

[0049] In one aspect, this application also provides a computer-readable storage medium for storing a computer program that includes methods for implementing the first aspect or any possible implementation thereof.

[0050] In a twelfth aspect, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to implement the method described in the first aspect or any possible implementation thereof.

[0051] The encoding device, decoding device, encoder, decoder, computer storage medium, computer program product, and chip provided in this embodiment are all used to execute the method provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the method provided above, and will not be repeated here. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1a An exemplary block diagram of a decoding system provided in an embodiment of this application;

[0054] Figure 1b An exemplary block diagram of a video decoding system provided in an embodiment of this application;

[0055] Figure 2 An exemplary block diagram of a video encoder provided in an embodiment of this application;

[0056] Figure 3 An exemplary block diagram of a video decoder provided in an embodiment of this application;

[0057] Figure 4 An exemplary schematic diagram of a candidate image block provided in an embodiment of this application;

[0058] Figure 5 An exemplary block diagram of a video decoding device provided in an embodiment of this application;

[0059] Figure 6 An exemplary block diagram of the apparatus provided in the embodiments of this application;

[0060] Figure 7a A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0061] Figure 7b A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0062] Figure 8 A flowchart illustrating an encoding / decoding method provided in an embodiment of this application;

[0063] Figure 9 This is a schematic diagram of the structure of an encoding / decoding system provided in an embodiment of this application;

[0064] Figure 10 A flowchart illustrating another encoding / decoding method provided in an embodiment of this application;

[0065] Figure 11 This is a schematic diagram of another encoding / decoding system provided in an embodiment of this application;

[0066] Figure 12 This is a schematic diagram of the structure of another encoding / decoding system provided in the embodiments of this application;

[0067] Figure 13 A flowchart illustrating another encoding / decoding method provided in an embodiment of this application;

[0068] Figure 14 This is a schematic diagram of the structure of another encoding / decoding system provided in the embodiments of this application;

[0069] Figure 15 A schematic diagram illustrating the performance of the encoding / decoding method provided in the embodiments of this application;

[0070] Figure 16 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0071] Figure 17 A schematic diagram illustrating another application scenario provided by an embodiment of this application;

[0072] Figure 18 This is a schematic diagram of the structure of an encoding / decoding device provided in an embodiment of this application;

[0073] Figure 19 This is a schematic diagram of another encoding / decoding device provided in an embodiment of this application;

[0074] Figure 20 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0075] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0076] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0077] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0078] Furthermore, the terms “comprising” and “having”, and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0079] It should be noted that in the description of the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0080] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0081] This application provides an AI-based data compression / decompression technology, particularly a neural network-based data compression / decompression technology, specifically an encoding / decoding technology to improve traditional hybrid data encoding / decoding systems.

[0082] Data encoding and decoding comprises two parts: data encoding and data decoding. Data encoding is performed on the source side (or commonly referred to as the encoder side) and typically involves processing (e.g., compressing) the raw data to reduce the amount of data required to represent it (thus enabling more efficient storage and / or transmission). Data decoding is performed on the destination side (or commonly referred to as the decoder side) and typically involves performing the inverse processing relative to the encoder side to reconstruct the original data. The "encoding and decoding" of data in the embodiments of this application should be understood as either "encoding" or "decoding" of the data. The encoding and decoding parts are also collectively referred to as encoding and decoding (CODEC).

[0083] In lossless data encoding, the original data can be reconstructed, meaning the reconstructed original data has the same quality as the original data (assuming no transmission loss or other data loss during storage or transmission). In lossy data encoding, further compression is performed through quantization to reduce the amount of data required to represent the original data, but the decoder cannot completely reconstruct the original data, meaning the quality of the reconstructed original data is lower or worse than the quality of the original data.

[0084] This application's embodiments can be applied to video data, image data, audio data, integer data, and other data requiring compression / decompression. The following description uses video data encoding (hereinafter referred to as video encoding) as an example to illustrate this application's embodiments. Other types of data (such as image data, audio data, integer data, and other data requiring compression / decompression) can be referred to the following description, and will not be repeated in this application's embodiments. It should be noted that, compared to video encoding, the encoding of audio data and integer data does not require dividing the data into blocks; instead, the data can be directly encoded.

[0085] Video coding generally refers to the processing of image sequences that form a video or video sequence. In the field of video coding, the terms "picture," "frame," or "image" can be used synonymously.

[0086] Several video coding standards fall under the category of "lossy hybrid video coding and decoding" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is typically segmented into a set of non-overlapping blocks, which are usually encoded at the block level. In other words, the encoder typically processes the video at the block (video block) level, for example, generating prediction blocks through spatial (intra-frame) prediction and temporal (inter-frame) prediction; subtracting the prediction blocks from the current block (the block currently being processed / to be processed) to obtain residual blocks; transforming and quantizing the residual blocks in the transform domain to reduce the amount of data to be transmitted (compressed), while the decoder applies the inverse processing relative to the encoder to the encoded or compressed blocks to reconstruct the current block for representation. Additionally, the encoder needs to repeat the decoder's processing steps so that the encoder and decoder generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstruct pixels for processing, i.e., encoding subsequent blocks.

[0087] In the following embodiment of the decoding system 10, the encoder 20 and decoder 30 are based on Figures 1a to 3 Describe it.

[0088] Figure 1a This is an exemplary block diagram of a decoding system 10 provided for an embodiment of this application, such as a video decoding system 10 (or simply decoding system 10) that can utilize the technology of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video decoding system 10 represent devices, etc., that can be used to perform various technologies according to the various examples described in this application.

[0089] like Figure 1a As shown, the decoding system 10 includes a source device 12, which provides encoded image data 21, such as encoded images, to a destination device 14 for decoding the encoded image data 21.

[0090] The source device 12 includes an encoder 20, and optionally may include an image source 16, a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22.

[0091] Image source 16 may include or may be any type of image capture device for capturing real-world images, and / or any type of image generation device, such as a computer graphics processor for generating computer animation images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage device storing any of the images described above.

[0092] To distinguish the processing performed by the preprocessor (or preprocessing unit) 18, the image (or image data) 17 may also be referred to as the raw image (or raw image data) 17.

[0093] The preprocessor 18 receives the raw image data 17 and preprocesses it to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component.

[0094] Video encoder (or encoder) 20 is used to receive preprocessed image data 19 and provide encoded image data 21 (hereinafter referred to as...) Figure 2 (and so on, for further description).

[0095] The communication interface 22 in the source device 12 can be used to: receive encoded image data 21 and send encoded image data 21 (or other arbitrarily processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0096] The target device 14 includes a decoder 30, and optionally may include a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32 and a display device 34.

[0097] The communication interface 28 in the destination device 14 is used to receive encoded image data 21 (or other processed versions) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0098] Communication interfaces 22 and 28 can be used to send or receive encoded image data (or encoded data 21) through a direct communication link between source device 12 and destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0099] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing, so as to transmit it on a communication link or communication network.

[0100] Communication interface 28 corresponds to communication interface 22. For example, it can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21.

[0101] Both communication interface 22 and communication interface 28 can be configured as follows: Figure 1a The arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 indicates a one-way or two-way communication interface, which can be used to send and receive messages, establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission, such as encoded image data transmission, etc.

[0102] Video decoder (or decoder) 30 is used to receive encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as...). Figure 3 (and so on, for further description).

[0103] The post-processor 32 is used to post-process the decoded image data 31 (also known as the reconstructed image data) to obtain post-processed image data 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, trimming or resampling, or any other processing to generate the decoded image data 31 for display by the display device 34, etc.

[0104] Display device 34 is used to receive post-processed image data 33 to display the image to a user or viewer. Display device 34 can be or includes any type of display for representing the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0105] The decoding system 10 also includes a training engine 25, which is used to train the encoder 20 (especially the entropy coding unit 270 in the encoder 20) or the decoder 30 (especially the entropy decoding unit 304 in the decoder 30) to entropy encode the image block to be encoded according to the estimated probability distribution. For a detailed description of the training engine 25, please refer to the following method implementation examples.

[0106] although Figure 1a The source device 12 and destination device 14 are shown as independent devices, but device embodiments may also include both source device 12 and destination device 14, or the functions of both source device 12 and destination device 14, that is, simultaneously including source device 12 or its corresponding functions and destination device 14 or its corresponding functions. In these embodiments, source device 12 or its corresponding functions and destination device 14 or its corresponding functions may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0107] According to the description, Figure 1a The presence and (accurate) division of different units or functions in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0108] Please refer to Figure 1b , Figure 1b This is an exemplary block diagram of a video decoding system 40 provided in an embodiment of this application. The encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both, can be transmitted via... Figure 1b The processing circuitry in the video decoding system 40 shown is implemented using, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video encoding processors, or any combination thereof. Please refer to... Figure 2 and Figure 3 , Figure 2 This is an exemplary block diagram of a video encoder provided in an embodiment of this application. Figure 3 This is an exemplary block diagram of a video decoder provided in an embodiment of this application. The encoder 20 can be implemented using processing circuitry 46 to include reference... Figure 2 Encoder 20 refers to various modules discussed herein and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented via processing circuitry 46 to include references. Figure 3 Decoder 30 may include various modules discussed herein and / or any other decoder system or subsystem described herein. The processing circuitry 46 may be used to perform various operations discussed below. Figure 5As shown, if some of the technology is implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors, thereby performing the technology of this application. One of the video encoder 20 and video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as... Figure 1b As shown.

[0109] Source device 12 and destination device 14 can include any of a variety of devices, including any type of handheld or fixed device, such as laptops or tablets, mobile phones, smartphones, tablets or tablets, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (e.g., content service servers or content distribution servers), broadcast receiving devices, broadcast transmitting devices, and monitoring devices, etc., and may or may not use any type of operating system. Source device 12 and destination device 14 can also be devices in a cloud computing scenario, such as virtual machines in a cloud computing scenario. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0110] Source device 12 and destination device 14 can install virtual reality (VR), augmented reality (AR), or mixed reality (MR) applications, and can run VR, AR, or MR applications based on user actions (such as clicking, touching, swiping, shaking, voice control, etc.). Source device 12 and destination device 14 can acquire images / videos of any object in the environment through cameras and / or sensors, and then display virtual objects on the display device based on the acquired images / videos. These virtual objects can be virtual objects in VR, AR, or MR scenes (i.e., objects in the virtual environment).

[0111] It should be noted that, in this embodiment of the application, the virtual scene application in the source device 12 and the destination device 14 can be an application built into the source device 12 and the destination device 14 themselves, or it can be an application provided by a third-party service provider that is installed by the user. No specific limitation is made in this regard.

[0112] Furthermore, source device 12 and destination device 14 can be equipped with real-time video transmission applications, such as live streaming applications. Source device 12 and destination device 14 can capture images / videos via cameras and then display the captured images / videos on a display device.

[0113] In some cases, Figure 1a The video decoding system 10 shown is merely exemplary, and the technology provided in this application can be applied to video encoding setups (e.g., video encoding or video decoding) that do not necessarily include any data communication between the encoding and decoding devices. In other examples, data is retrieved from local memory, sent over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data into memory and / or retrieve and decode data from memory.

[0114] Please refer to Figure 1b , Figure 1b An exemplary block diagram of the video decoding system 40 provided in this application embodiment is shown below. Figure 1b As shown, the video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memory storage devices 44, and / or a display device 45.

[0115] like Figure 1b As shown, the imaging device 41, antenna 42, processing circuitry 46, video encoder 20, video decoder 30, processor 43, memory storage 44, and / or display device 45 are capable of communicating with each other. In different instances, the video decoding system 40 may contain only the video encoder 20 or only the video decoder 30.

[0116] In some instances, antenna 42 can be used to transmit or receive encoded bitstreams of video data. Additionally, in some instances, display device 45 can be used to present video data. Processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Video decoding system 40 can also include an optional processor 43, which similarly can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Furthermore, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory storage 44 can be implemented using high-speed cache memory. In other instances, processing circuitry 46 can include memory (e.g., cache, etc.) for implementing image buffers, etc.

[0117] In some instances, the video encoder 20 implemented via logic circuitry may include (e.g., implemented via processing circuitry 46 or memory storage 44) an image buffer and (e.g., implemented via processing circuitry 46) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video encoder 20 implemented via processing circuitry 46 to implement reference... Figure 2 And / or any other encoder system or subsystem described herein, and the various modules discussed herein. Logic circuits may be used to perform the various operations discussed herein.

[0118] In some instances, the video decoder 30 can be implemented in a similar manner via the processing circuitry 46 to implement the reference. Figure 3 The video decoder 30 and / or any other decoder system or subsystem described herein are various modules discussed. In some instances, the logic circuit-implemented video decoder 30 may include (implemented via processing circuitry 46 or memory storage 44) an image buffer and (e.g., implemented via processing circuitry 46) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented via processing circuitry 46 to implement reference... Figure 3 And / or the various modules discussed in any other decoder system or subsystem described herein.

[0119] In some instances, antenna 42 can be used to receive encoded bitstreams of video data. As discussed herein, the encoded bitstream may contain data related to encoded video frames, indicators, index values, mode selection data, etc., such as data related to code segmentation (e.g., transform coefficients or quantized transform coefficients, optional indicators, and / or data defining code segmentation). Video decoding system 40 may also include a video decoder 30 coupled to antenna 42 for decoding the encoded bitstream. Display device 45 is used to display the video frames.

[0120] It should be understood that, for the examples described with reference to video encoder 20 in this application embodiment, video decoder 30 can be used to perform the reverse process. Regarding signaling syntax elements, video decoder 30 can be used to receive and parse such syntax elements, and accordingly decode the associated video data. In some examples, video encoder 20 can entropy-encode syntax elements into an encoded video bitstream. In such instances, video decoder 30 can parse such syntax elements and accordingly decode the associated video data.

[0121] For ease of description, embodiments of this application are described with reference to the Universal Video Coding (VVC) reference software or the High-Efficiency Video Coding (HEVC) developed by the ITU-T Video Coding Experts Group (VCEG) and the Joint Collaboration Team on Video Coding (JCT-VC) of the ISO / IEC Moving Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of this application are not limited to HEVC or VVC.

[0122] Encoders and Encoding Methods

[0123] like Figure 2 As shown, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The video encoder 20 shown can also be called a hybrid video encoder or a video encoder based on a hybrid video codec.

[0124] See Figure 2 Inter-frame prediction units are trained target models (also known as neural networks) that process input images, image regions, or image patches to generate predicted values ​​for those patches. For example, a neural network for inter-frame prediction receives an input image, image region, or image patch and generates predicted values ​​for that image, image region, or image patch.

[0125] The residual calculation unit 204, transform processing unit 206, quantization unit 208, and mode selection unit 260 constitute the forward signal path of encoder 20, while the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, buffer 216, loop filter 220, decoded picture buffer (DPB) 230, inter-frame prediction unit 244, and intra-frame prediction unit 254 constitute the backward signal path of encoder 20. The backward signal path of encoder 20 corresponds to the signal path of decoder (see [link to decoder]). Figure 3 The decoder 30 in the video encoder 20 consists of an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded image buffer 230, an inter-frame prediction unit 244, and an intra-frame prediction unit 254.

[0126] Image and image segmentation (images and patches)

[0127] Encoder 20 can be used to receive images (or image data) 17 via input terminal 201, for example, images in an image sequence forming a video or video sequence. The received images or image data can also be pre-processed images (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 can also be referred to as the current image or the image to be encoded (especially in video encoding when distinguishing the current image from other images, such as those in the same video sequence, i.e., the video sequence that also includes the current image, previously encoded images, and / or decoded images).

[0128] A digital image is, or can be viewed as, a two-dimensional array or matrix of pixels with intensity values. Pixels in an array are also called pixels (short for image element). The number of pixels in the array or image along the horizontal and vertical directions (or axes) determines the image size and / or resolution. To represent color, three color components are typically used, meaning an image can be represented as or comprise an array of three pixels. In RBG format or color space, an image includes corresponding arrays of red, green, and blue pixels. However, in video coding, each pixel is typically represented in a luma / chroma format or color space, such as YCbCr, which includes the luma component indicated by Y (sometimes also represented by L) and two chroma components represented by Cb and Cr. The luma component Y represents the brightness or grayscale level intensity (e.g., both are the same in grayscale images), while the two chroma components Cb and Cr represent the chroma or color information components. Accordingly, a YCbCr format image consists of a luminance pixel array for the luminance pixel value (Y) and two chrominance pixel arrays for the chrominance values ​​(Cb and Cr). An RGB format image can be converted or transformed to YCbCr format, and vice versa; this process is also known as color conversion or transformation. If the image is black and white, it may only include the luminance pixel array. Accordingly, the image can be, for example, a monochrome format luminance pixel array or a 4:2:0, 4:2:2, and 4:4:4 color format luminance pixel array and two corresponding chrominance pixel arrays.

[0129] In one embodiment, the video encoder 20 may include an image segmentation unit ( Figure 2 (Not shown in the image) is used to segment image 17 into multiple (typically non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC), coding tree blocks (CTBs), or coding tree units (CTUs) in the H.265 / HEVC and VVC standards. Segmentation units can be used to apply the same block size and a corresponding grid with defined block sizes to all images in a video sequence, or to vary the block size between images, subsets of images, or groups of images, segmenting each image into corresponding blocks.

[0130] In other embodiments, the video encoder may be used to directly receive blocks 203 of image 17, such as one, several, or all of the blocks that make up image 17. Image block 203 may also be referred to as the current image block or the image block to be encoded.

[0131] Similar to image 17, image block 203 is also a two-dimensional array or matrix composed of pixels with intensity values ​​(pixel values), but image block 203 is smaller than that of image 17. In other words, block 203 may include a pixel array (e.g., a luminance array in the case of monochrome image 17 or a luminance or chrominance array in the case of a color image) or a three-pixel array (e.g., a luminance array and two chrominance arrays in the case of color image 17) or any other number and / or type of array depending on the color format used. The number of pixels in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Accordingly, the block may be an M×N (M columns × N rows) pixel array, or an M×N transform coefficient array, etc.

[0132] In one embodiment, Figure 2 The video encoder 20 shown is used to encode the image 17 block by block, for example, to perform encoding and prediction for each block 203.

[0133] In one embodiment, Figure 2 The video encoder 20 shown can also be used to segment and / or encode images using slices (also called video slices), where images can be segmented or encoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., coding tree units, CTUs) or one or more groups of blocks (e.g., coded tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0134] In one embodiment, Figure 2 The video encoder 20 shown can also be used to segment and / or encode an image using slice / encoding block groups (also known as video encoding block groups) and / or encoding blocks (also known as video encoding blocks), wherein the image can be segmented or encoded using one or more slice / encoding block groups (typically non-overlapping), each slice / encoding block group may include one or more blocks (e.g., CTUs) or one or more encoding blocks, wherein each encoding block may be rectangular or the like, and may include one or more complete or partial blocks (e.g., CTUs).

[0135] Residual calculation

[0136] The residual calculation unit 204 is used to calculate the residual block 205 based on the image block (or original block) 203 and the prediction block 265 in the following manner (the prediction block 265 is described in detail later): for example, the residual block 205 in the pixel domain is obtained by subtracting the pixel value of the prediction block 265 from the pixel value of the image block 203 pixel by pixel.

[0137] Transformation

[0138] The transformation processing unit 206 performs discrete cosine transform (DCT) or discrete sine transform (DST) on the pixel values ​​of the residual block 205 to obtain the transformation coefficients 207 in the transform domain. The transformation coefficients 207 can also be called transformation residual coefficients, representing the residual block 205 in the transform domain.

[0139] Transform processing unit 206 can be used to apply an integer approximation of DCT / DST, such as the transform specified for H.265 / HEVC. This integer approximation is typically scaled by a certain factor compared to the orthogonal DCT transform. To maintain the norm of the residual block after both the forward and inverse transforms, other scaling factors are used as part of the transform process. These scaling factors are typically selected based on certain constraints, such as powers of 2 used for shift operations, the bit depth of the transform coefficients, and a trade-off between accuracy and implementation cost. For example, a specific scaling factor can be specified on the encoder 20 side via inverse transform processing unit 212 (and on the decoder 30 side via, for example, inverse transform processing unit 312) for the inverse transform, and correspondingly, a corresponding scaling factor can be specified on the encoder 20 side via transform processing unit 206 for the forward transform.

[0140] In one embodiment, the video encoder 20 (correspondingly, the transform processing unit 206) can be used to output transform parameters such as the type of one or more transforms, for example, directly outputting them or outputting them after being encoded or compressed by the entropy encoding unit 270, for example, so that the video decoder 30 can receive and use the transform parameters for decoding.

[0141] Quantification

[0142] Quantization unit 208 is used to quantize the transform coefficients 207 by, for example, scalar quantization or vector quantization, to obtain quantized transform coefficients 209. Quantized transform coefficients 209 can also be called quantized residual coefficients 209.

[0143] The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different scales can be applied to achieve finer or coarser quantization. Smaller quantization steps correspond to finer quantization, while larger quantization steps correspond to coarser quantization. The appropriate quantization step size can be indicated by the quantization parameter (QP). For example, the quantization parameter can be an index to a predefined set of appropriate quantization steps. For example, a smaller quantization parameter can correspond to fine quantization (smaller quantization step size), a larger quantization parameter can correspond to coarse quantization (larger quantization step size), and vice versa. Quantization may include division by the quantization step size, while corresponding or inverse dequantization performed by the dequantization unit 210, etc., may include multiplication by the quantization step size. Embodiments of some HEVC standards, for example, can be used to determine the quantization step size using the quantization parameter. In general, the quantization step size can be calculated using a fixed-point approximation of an equation involving division based on the quantization parameter. Additional scaling factors can be introduced for quantization and dequantization to recover the norm of the residual block, which may have been modified by the scaling used in the fixed-point approximation of the equations used for the quantization step size and quantization parameters. In one exemplary implementation, the scaling of the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where the loss increases with the quantization step size.

[0144] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) can be used to output the quantization parameter (QP), for example, directly outputting it or outputting it after being encoded or compressed by the entropy encoding unit 270, for example, so that the video decoder 30 can receive it and use it for decoding.

[0145] Inverse Quantization

[0146] The dequantization unit 210 is used to perform dequantization on the quantization coefficients by the quantization unit 208 to obtain the dequantization coefficients 211. For example, it performs a dequantization scheme based on or using the same quantization step size as the quantization unit 208 to perform the quantization scheme performed by the quantization unit 208. The dequantization coefficients 211 can also be called dequantization residual coefficients 211, corresponding to the transform coefficients 207. However, due to the loss caused by quantization, the dequantization coefficients 211 are usually not exactly the same as the transform coefficients.

[0147] Inverse Transformation

[0148] The inverse transform processing unit 212 is used to perform the inverse transform of the transform performed by the transform processing unit 206, such as the inverse discrete cosine transform (DCT) or the inverse discrete sine transform (DST), to obtain the reconstructed residual block 213 (or the corresponding dequantization coefficients 213) in the pixel domain. The reconstructed residual block 213 may also be referred to as the transform block 213.

[0149] reconstruction

[0150] The reconstruction unit 214 (e.g., summer 214) is used to add the transform block 213 (i.e., the reconstruction residual block 213) to the prediction block 265 to obtain the reconstruction block 215 in the pixel domain, for example, by adding the pixel values ​​of the reconstruction residual block 213 and the pixel values ​​of the prediction block 265.

[0151] Filtering

[0152] Loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain the filtered block 221, or typically to filter the reconstructed pixels to obtain filtered pixel values. For example, the loop filter unit is used to smoothly perform pixel transformations or improve video quality. Loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, loop filter unit 220 may include a deblocking filter, a SAO filter, and an ALF filter. The filtering process may be performed in the order of deblocking filter, SAO filter, and ALF filter. As another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) may be added. This process is performed before deblocking. For example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 220 in... Figure 2 The loop filter is shown in the diagram, but in other configurations, the loop filter unit 220 can be implemented as a post-loop filter. The filter block 221 can also be called the filter reconstruction block 221.

[0153] In one embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) can be used to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, directly outputting or outputting after entropy encoding by the entropy encoding unit 270, for example, enabling the decoder 30 to receive and decode using the same or different loop filter parameters.

[0154] Decoding image buffer

[0155] The decoded picture buffer (DPB) 230 can be a reference picture memory that stores reference picture data for use by the video encoder 20 when encoding video data. The DPB 230 can be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer 230 can be used to store one or more filter blocks 221. The decoded picture buffer 230 can also be used to store other previous filter blocks of the same current image or different images, such as previously reconstructed images, such as previously reconstructed and filtered blocks 221, and can provide complete previously reconstructed i.e., decoded images (and corresponding reference blocks and pixels) and / or partially reconstructed current images (and corresponding reference blocks and pixels), for example, for inter-frame prediction. The decoded image buffer 230 can also be used to store one or more unfiltered reconstruction blocks 215, or generally store unfiltered reconstruction pixels, such as reconstruction blocks 215 that have not been filtered by the loop filter unit 220, or reconstruction blocks or reconstruction pixels that have not undergone any other processing.

[0156] Pattern selection (segmentation and prediction)

[0157] The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, for selecting data from the decoded image buffer 230 or other buffers (e.g., column buffers). Figure 2 (Not shown) receives or obtains raw image data such as raw block 203 (current block 203 of the current image 17) and reconstructed image data, for example, filtered and / or unfiltered reconstructed pixels or reconstructed blocks of the same (current) image and / or one or more previously decoded images. The reconstructed image data is used as reference image data for prediction such as inter-frame prediction or intra-frame prediction to obtain prediction block 265 or prediction value 265.

[0158] The mode selection unit 260 can be used to determine or select a segmentation for the current block (including no segmentation) and the prediction mode (e.g., intra-frame or inter-frame prediction mode) to generate the corresponding prediction block 265 for calculating the residual block 205 and reconstructing the reconstructed block 215.

[0159] In one embodiment, the mode selection unit 260 can be used to select a segmentation and prediction mode (e.g., from prediction modes supported or available by the mode selection unit 260), which provides the best match or minimum residual (minimum residual refers to better compression in transmission or storage), or provides minimum signaling overhead (minimum signaling overhead refers to better compression in transmission or storage), or considers or balances both. The mode selection unit 260 can be used to determine the segmentation and prediction mode based on rate distortion optimization (RDO), i.e., selecting the prediction mode that provides minimum RDO optimization. The terms "best," "lowest," and "optimal" in this document do not necessarily refer to "best," "lowest," or "optimal" overall, but can also refer to situations that meet termination or selection criteria. For example, values ​​exceeding or falling below a threshold or other limitations may lead to a "suboptimal choice," but reduce complexity and processing time.

[0160] In other words, segmentation unit 262 can be used to segment images in a video sequence into a sequence of coding tree units (CTUs), CTUs 203 can be further segmented into smaller block portions or sub-blocks (forming blocks again), for example, by iteratively using quad-tree partitioning (QT), binary-tree partitioning (BT), or triple-tree partitioning (TT), or any combination thereof, and is used to perform prediction, for example, on each of the block portions or sub-blocks, wherein mode selection includes selecting the tree structure of the segmented block 203 and selecting the prediction mode applied to each of the block portions or sub-blocks.

[0161] The segmentation (e.g., performed by segmentation unit 262) and prediction processing (e.g., performed by inter-frame prediction unit 244 and intra-frame prediction unit 254) performed by video encoder 20 will be described in detail below.

[0162] segmentation

[0163] Segmentation unit 262 can divide (or divide) an image block (or CTU) 203 into smaller parts, such as small blocks of square or rectangular shape. For an image with a three-pixel array, a CTU consists of N×N luma pixel blocks and two corresponding chroma pixel blocks. The maximum allowed size of the luma blocks in a CTU is specified as 128×128 in the developing Versatile Video Coding (VVC) standard, but may be specified as a value different from 128×128 in the future, such as 256×256. Image CTUs can be grouped / set into slices / coded block groups, coded blocks, or bricks. A coded block covers a rectangular area of ​​an image, and a coded block can be divided into one or more bricks. A brick consists of multiple rows of CTUs within a coded block. A coded block that is not divided into multiple bricks can be called a brick. However, a brick is a true subset of a coded block and therefore is not called a coded block. VVC supports two coded block group modes: raster scan slice / coded block group mode and rectangular slice mode. In raster scan coded block group mode, a slice / coded block group contains a sequence of coded blocks in a raster scan of an image. In rectangular slice mode, a slice contains multiple bricks of an image, which together form a rectangular region of the image. The bricks within a rectangular slice are arranged in the raster scan order of the slice's bricks. These smaller blocks (also called sub-blocks) can be further subdivided into even smaller parts. This is also called tree segmentation or hierarchical tree segmentation, where the root block at root tree level 0 (level 0, depth 0), etc., can be recursively segmented into two or more blocks at the next lower tree level, such as nodes at tree level 1 (level 1, depth 1). These blocks can then be further segmented into two or more blocks at the next lower level, such as tree level 2 (level 2, depth 2), etc., until the segmentation ends (because the termination criteria are met, such as reaching the maximum tree depth or minimum block size). Blocks that are not further segmented are also called leaf blocks or leaf nodes of the tree. A tree divided into two parts is called a binary tree (BT), a tree divided into three parts is called a ternary tree (TT), and a tree divided into four parts is called a quad tree (QT).

[0164] For example, a coding unit (CTU) can be or include a CTB of luma pixels, two corresponding CTBs of chroma pixels in an image with a three-pixel array, a CTB of pixels in a monochrome image, or a CTB of pixels in an image encoded using three independent color planes and a syntax structure (for encoding pixels). Correspondingly, a coding block (CTB) can be an N×N pixel block, where N can be set to a value such that the components are divided into CTBs; this is called segmentation. Similarly, a coding unit (CU) can be or include a coding block of luma pixels, two corresponding coding blocks of chroma pixels in an image with a three-pixel array, a coding block of pixels in a monochrome image, or a coding block of pixels in an image encoded using three independent color planes and a syntax structure (for encoding pixels). Correspondingly, a coding block (CB) can be an M×N pixel block, where M and N can be set to a value such that the CTB is divided into coding blocks; this is called segmentation.

[0165] For example, in one embodiment, HEVC can divide a Coding Tree Unit (CTU) into multiple Coding Units (CUs) using a quadtree structure represented as a coding tree. At the leaf CU level, a decision is made as to whether to use inter-frame (temporal) prediction or intra-frame (spatial) prediction to encode image regions. Each leaf CU can be further divided into one, two, or four PUs based on the PU partitioning type. The same prediction process is used within a PU, and relevant information is transmitted to the decoder on a PU-by-PU basis. After obtaining residual blocks by applying the prediction process according to the PU partitioning type, the leaf CU can be divided into Transform Units (TUs) according to other quadtree structures similar to those used for the coding tree of the CU.

[0166] For example, in one embodiment, according to the latest video coding standard currently under development (called Universal Video Coding (VVC), a combined quadtree of nested multi-type trees (e.g., binary and ternary trees) is used to partition the segmented structure for dividing the coding tree unit. Within the coding tree structure of a coding tree unit, the CU can be a square or a rectangle. For example, a coding tree unit (CTU) is first partitioned by a quadtree structure. The leaf nodes of the quadtree are further partitioned by multi-type tree structures. There are four partitioning types for the multi-type tree structure: vertical binary tree partitioning (SPLIT_BT_VER), horizontal binary tree partitioning (SPLIT_BT_HOR), vertical ternary tree partitioning, and vertical binary tree partitioning. SPLIT_TT_VER and horizontal ternary tree partitioning (SPLIT_TT_HOR). Multi-type leaf nodes are called coding units (CUs). Unless the CU is too large for the maximum transform length, such segmentation is used for prediction and transform processing without any other partitioning. In most cases, this means that the CU, PU, ​​and TU have the same block size in the quadtree-nested multi-type tree coding block structure. This anomaly occurs when the maximum supported transform length is less than the width or height of the color component of the CU. VVC specifies a unique signaling mechanism for partitioning information in a quadtree-nested multi-type tree coding structure. In the signaling mechanism, coding... The tree unit (CTU), serving as the root of the quadtree, is first segmented by the quadtree structure. Then, each quadtree leaf node (if large enough) is further segmented into a multi-type tree structure. In the multi-type tree structure, a first flag (mtt_split_cu_flag) indicates whether a node is further segmented. When a node is further segmented, a second flag (mtt_split_cu_vertical_flag) indicates the direction of the segmentation, and a third flag (mtt_split_cu_binary_flag) indicates whether the segmentation is a binary or ternary tree. This is determined by mtt_split_c... The values ​​of `u_vertical_flag` and `mtt_split_cu_binary_flag` allow the decoder to deduce the multi-type tree partitioning mode (MttSplitMode) of the CU based on predefined rules or tables. It's important to note that for certain designs, such as a 64×64 luma block and a 32×32 chroma pipeline design in a VVC hardware decoder, TT partitioning is not allowed when the width or height of the luma encoding block exceeds 64. Similarly, TT partitioning is not allowed when the width or height of the chroma encoding block exceeds 32. The pipeline design divides the image into multiple virtual pipeline data units (VPDUs), each defined as a non-overlapping unit in the image. In the hardware decoder, consecutive VPDUs are processed simultaneously in multiple pipeline stages. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, thus requiring small VPDU sizes.In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, the partitioning of the ternary tree (TT) and binary tree (BT) can increase the VPDU size.

[0167] Additionally, it should be noted that when a portion of a tree node block extends beyond the bottom or right edge of the image, the tree node block is forcibly divided until all pixels of each encoded CU are within the image boundary.

[0168] For example, the intra-sub-partitions (ISP) tool can divide the luminance intra-prediction block vertically or horizontally into two or four sub-parts based on the block size.

[0169] In one example, the mode selection unit 260 of the video encoder 20 can be used to perform any combination of the segmentation techniques described above.

[0170] As described above, the video encoder 20 is used to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, intra-frame prediction modes and / or inter-frame prediction modes.

[0171] Intra-frame prediction

[0172] The intra-prediction mode set can include 35 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in HEVC, or it can include 67 different intra-prediction modes, such as non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in VVC. For example, several conventional angular intra-prediction modes are adaptively replaced with wide-angle intra-prediction modes for non-square blocks as defined in VVC. As another example, to avoid division operations in DC prediction, only the longer side is used to calculate the average value of non-square blocks. Furthermore, the intra-prediction results of planar mode can be modified using the position-dependent intra-prediction combination (PDPC) method.

[0173] Intra-prediction unit 254 is used to generate intra-prediction block 265 using reconstructed pixels of adjacent blocks of the same current image according to the intra-prediction mode in the intra-prediction mode set.

[0174] Intra-prediction unit 254 (or typically mode selection unit 260) is also used to output intra-prediction parameters (or typically information indicating the selected intra-prediction mode of the block) to entropy coding unit 270 in the form of syntax element 266 to be included in encoded image data 21, so that video decoder 30 can perform operations such as receiving and using the prediction parameters for decoding.

[0175] HEVC's intra-frame prediction modes include DC prediction mode, planar prediction mode, and 33 angle prediction modes, totaling 35 candidate prediction modes. The current block can use pixels from the reconstructed image blocks to its left and above as references for intra-frame prediction. The image blocks in the surrounding region used for intra-frame prediction of the current block are called reference blocks, and the pixels in the reference blocks are called reference pixels. Among the 35 candidate prediction modes, DC prediction mode is suitable for regions with flat textures in the current block, where all pixels use the average value of the reference pixels in the reference block as the prediction; planar prediction mode is suitable for image blocks with smoothly varying textures, where the current block uses bilinear interpolation of the reference pixels in the reference block to predict all pixels in the current block; angle prediction mode utilizes the high correlation between the texture of the current block and the textures of adjacent reconstructed image blocks, copying the values ​​of the reference pixels in the corresponding reference block along a certain angle as the prediction for all pixels in the current block.

[0176] The HEVC encoder selects an optimal intra-prediction mode from 35 candidate prediction modes for the current block and writes this optimal intra-prediction mode into the video stream. To improve the coding efficiency of intra-prediction, the encoder / decoder derives three most probable modes from the optimal intra-prediction modes of the reconstructed image blocks in the surrounding regions that use intra-prediction. If the optimal intra-prediction mode selected for the current block is one of these three most probable modes, a first index is encoded to indicate that the selected optimal intra-prediction mode is one of these three most probable modes; if the selected optimal intra-prediction mode is not one of these three most probable modes, a second index is encoded to indicate that the selected optimal intra-prediction mode is one of the other 32 modes (other modes from the 35 candidate prediction modes besides the aforementioned three most probable modes). The HEVC standard uses a 5-bit fixed-length code as the aforementioned second index.

[0177] The HEVC encoder derives the three most likely modes by: selecting the best intra-prediction modes of the left and top adjacent image blocks of the current block and adding them to the set; if these two best intra-prediction modes are the same, only one needs to be kept in the set. If these two best intra-prediction modes are the same and both are angle prediction modes, then two angle prediction modes adjacent to that angle direction are selected and added to the set; otherwise, the planar prediction mode, DC mode, and vertical prediction mode are selected and added to the set in sequence, until the number of modes in the set reaches three.

[0178] After performing entropy decoding on the bitstream, the HEVC decoder obtains the mode information of the current block. This mode information includes an indicator that indicates whether the optimal intra-prediction mode of the current block is among the three most likely modes, as well as the index of the optimal intra-prediction mode of the current block among the three most likely modes or the index of the optimal intra-prediction mode of the current block among the other 32 modes.

[0179] Inter-frame prediction

[0180] In a possible implementation, the set of inter-frame prediction modes depends on the available reference image (i.e., at least a portion of the previously decoded image stored in the DBP230 as described above) and other inter-frame prediction parameters, such as whether to use the entire reference image or only a portion of the reference image, such as a search window region near the current block, to search for the best matching reference block, and / or, for example, whether to perform pixel interpolation of half-pixel, quarter-pixel, and / or 1 / 16th interpolation.

[0181] In addition to the prediction modes mentioned above, skip mode and / or direct mode can also be used.

[0182] For example, in extended merge prediction, the merge candidate list for this mode consists of five candidate types in sequence: spatial MVP from spatially adjacent CUs, temporal MVP from co-located CUs, history-based MVP from a FIFO table, pairwise averaged MVP, and zero MV. Decoder-side motion vector refinement (DMVR) based on bilateral matching can be used to increase the accuracy of the merge mode's MV. Mergemode with MVD (MMVD) comes from merge modes with motion vector differences. The MMVD flag is sent immediately after the skip flag and merge flag to specify whether the CU uses MMVD mode. The CU-level adaptive motion vector resolution (AMVR) scheme can be used. AMVR supports encoding the CU's MVD with different precisions. The MVD of the current CU is adaptively selected based on its prediction mode. When the CU is encoding in merge mode, the combined inter / intra prediction (CIIP) mode can be applied to the current CU. The CIIP prediction is obtained by weighted averaging of the inter and intra prediction signals. For affine motion compensation prediction, the affine motion field of the block is described using motion information from motion vectors at 2 control points (4 parameters) or 3 control points (6 parameters). Subblock-based temporal motion vector prediction (SbTMVP) is similar to temporal motion vector prediction (TMVP) in HEVC, but it predicts the motion vectors of sub-CUs within the current CU. Bidirectional optical flow (BDOF), formerly known as BIO, is a simplified version that reduces computation, particularly in terms of the number of multiplications and the size of the multipliers. In the triangular partitioning mode, the CU is uniformly divided into two triangular parts using both diagonal and anti-diagonal partitioning. Furthermore, the bidirectional prediction mode extends the simple averaging method to support weighted averaging of the two predicted signals.

[0183] Inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both in... Figure 2(Not shown in the image). The motion estimation unit can be used to receive or acquire image block 203 (current image block 203 of current image 17) and decoded image 231, or at least one or more previously reconstructed blocks, such as one or more other / different previously decoded image blocks 231, to perform motion estimation. For example, the video sequence may include the current image and the previously decoded image 231, or in other words, the current image and the previously decoded image 231 may be part of or form the image sequence that forms the video sequence.

[0184] For example, encoder 20 can be used to select a reference block from multiple reference blocks of the same or different images in multiple other images, and provide the offset (spatial offset) between the position (x, y coordinates) of the reference image (or reference image index) and / or the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. This offset is also called a motion vector (MV).

[0185] The motion compensation unit is used to acquire, for example, receive, inter-frame prediction parameters, and perform inter-frame prediction based on or using these parameters to obtain inter-frame prediction blocks 246. Motion compensation performed by the motion compensation unit may include extracting or generating prediction blocks based on motion / block vectors determined by motion estimation, and may also include performing interpolation with sub-pixel precision. Interpolation filtering can generate pixels of other pixels from pixels of known pixels, thereby potentially increasing the number of candidate prediction blocks available for encoding image blocks. Once the motion vector corresponding to the PU of the current image block is received, the motion compensation unit can locate the prediction block pointed to by the motion vector in one of the reference image lists.

[0186] The motion compensation unit can also generate syntax elements associated with blocks and video slices for use by the video decoder 30 when decoding image blocks of the video slices. Alternatively, or as an alternative to slices and corresponding syntax elements, coded block groups and / or coded blocks and their corresponding syntax elements can be generated or used.

[0187] In the process of obtaining the candidate motion vector list in the advanced motion vector prediction (AMVP) mode, the motion vectors (MVs) that can be added to the candidate motion vector list as candidates include the MVs of spatially and temporally adjacent image blocks of the current block. The MVs of spatially adjacent image blocks can further include the MVs of the candidate image blocks to the left of the current block and the MVs of the candidate image blocks above the current block. For an example, please refer to... Figure 4 , Figure 4 An exemplary schematic diagram of a candidate image block provided in an embodiment of this application, such as... Figure 4As shown, the set of candidate image blocks to the left includes {A0, A1}, the set of candidate image blocks above includes {B0, B1, B2}, and the set of temporally adjacent candidate image blocks includes {C, T}. All three sets can be added to the candidate motion vector list as candidates. However, according to existing coding standards, the maximum length of the candidate motion vector list for AMVP is 2. Therefore, it is necessary to determine the MV (Motion Vector Model) to be added to the candidate motion vector list from the three sets according to a prescribed order. This order can be: first, consider the set of candidate image blocks to the left of the current block {A0, A1} (consider A0 first, then A1 if A0 is unavailable); second, consider the set of candidate image blocks above the current block {B0, B1, B2} (consider B0 first, then B1 if B0 is unavailable, then B2 if B1 is unavailable); finally, consider the set of temporally adjacent candidate image blocks {C, T} of the current block (consider T first, then C if T is unavailable).

[0188] After obtaining the above list of candidate motion vectors, the optimal motion vector prediction (MV) is determined from the list using the rate distortion cost (RDcost). The candidate motion vector with the lowest RDcost is then used as the motion vector predictor (MVP) for the current block. The rate distortion cost is calculated using the following formula:

[0189] J = SAD + λR

[0190] Where J represents the RD cost, SAD is the sum of absolute differences (SAD) between the pixel values ​​of the predicted block obtained after motion estimation using candidate motion vectors and the pixel values ​​of the current block, R represents the bit rate, and λ represents the Lagrange multiplier.

[0191] The encoder passes the index of the determined MVP in the candidate motion vector list to the decoder. Further, a motion search can be performed within the neighborhood centered on the MVP to obtain the actual motion vector of the current block. The encoder calculates the motion vector difference (MVD) between the MVP and the actual motion vector and also passes the MVD to the decoder. The decoder parses the index, finds the corresponding MVP in the candidate motion vector list based on the index, parses the MVD, and adds the MVD to the MVP to obtain the actual motion vector of the current block.

[0192] During the process of obtaining the candidate motion information list in the merging mode, the motion information that can be added to the candidate motion information list includes the motion information of spatially adjacent or temporally adjacent image blocks of the current block. The spatially adjacent and temporally adjacent image blocks can be referenced... Figure 4 The candidate motion information list corresponds to spatially adjacent blocks (A0, A1, B0, B1, and B2) in the spatial domain. If a spatially adjacent block is unavailable or is intra-frame prediction, its motion information is not added to the candidate motion information list. The temporal candidate motion information for the current block is obtained by scaling the MV of the corresponding block in the reference frame based on the picture order count (POC) of the reference frame and the current frame. First, it is determined whether the block at position T in the reference frame is available; if not, the block at position C is selected. After obtaining the above candidate motion information list, the optimal motion information is determined from the candidate motion information list using the RD cost as the motion information for the current block. The encoder transmits the index value (denoted as mergeindex) of the optimal motion information in the candidate motion information list to the decoder.

[0193] Entropy coding

[0194] Entropy coding unit 270 is used to apply entropy coding algorithms or schemes (e.g., variable length coding (VLC), context adaptive VLC (CALVC), arithmetic coding schemes, binarization algorithms, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to quantization residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements, to obtain encoded image data 21 that can be output as an encoded bitstream 21 through output terminal 272, so that video decoder 30 and the like can receive and use the parameters for decoding. The encoded bitstream 21 can be transmitted to video decoder 30, or stored in memory for later transmission or retrieval by video decoder 30.

[0195] Other architectural variations of the video encoder 20 can be used to encode the video stream. For example, a non-transform-based encoder 20 can directly quantize the residual signal in certain blocks or frames without the transform processing unit 206. In another implementation, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0196] Decoder and Decoding Method

[0197] like Figure 3 As shown, the video decoder 30 is used to receive, for example, encoded image data 21 (e.g., encoded bitstream 21) encoded by the encoder 20, to obtain a decoded image 331. The encoded image data or bitstream includes information for decoding the encoded image data, such as data representing image blocks (and / or groups or blocks of encoded video segments) and associated syntax elements.

[0198] exist Figure 3 In the example, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded image buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. The inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same functions as the referenced unit. Figure 2 The video encoder 100 describes the encoding process as the opposite of the decoding process.

[0199] As described in encoder 20, the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded image buffer DPB 230, inter-frame prediction unit 344, and intra-frame prediction unit 354 also constitute the "built-in decoder" of video encoder 20. Correspondingly, inverse quantization unit 310 can be functionally identical to inverse quantization unit 110, inverse transform processing unit 312 can be functionally identical to inverse transform processing unit 122, reconstruction unit 314 can be functionally identical to reconstruction unit 214, loop filter 320 can be functionally identical to loop filter 220, and decoded image buffer 330 can be functionally identical to decoded image buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 is correspondingly applicable to the corresponding units and functions of video decoder 30.

[0200] Entropy Decoding

[0201] Entropy decoding unit 304 is used to parse bitstream 21 (or generally encoded image data 21) and perform entropy decoding on encoded image data 21 to obtain quantization coefficients 309 and / or decoded encoded parameters. Figure 3(Not shown in the image) Examples of parameters include inter-frame prediction parameters (e.g., reference image index and motion vector), intra-frame prediction parameters (e.g., intra-frame prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 can be used to apply the decoding algorithm or scheme corresponding to the encoding scheme of the entropy coding unit 270 of the encoder 20. The entropy decoding unit 304 can also be used to provide inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 can receive syntax elements at the video slice and / or video block level. Furthermore, or as an alternative to slices and corresponding syntax elements, it can receive or use coded block groups and / or coded blocks and corresponding syntax elements.

[0202] Inverse Quantization

[0203] The dequantization unit 310 can be used to receive quantization parameters (QP) (or generally information related to dequantization) and quantization coefficients from encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and dequantize the decoded quantization coefficients 309 based on the quantization parameters to obtain dequantization coefficients 311, which may also be referred to as transform coefficients 311. The dequantization process may include using the quantization parameters calculated by the video encoder 20 for each video block in the video slice to determine the degree of quantization, and also to determine the degree of dequantization to be performed.

[0204] Inverse Transformation

[0205] The inverse transform processing unit 312 can be used to receive the dequantized coefficients 311, also known as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain the reconstructed residual block 213 in the pixel domain. The reconstructed residual block 213 can also be called transform block 313. The transform can be an inverse transform, such as inverse DCT, inverse DST, inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 can also be used to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304) to determine the transform applied to the dequantized coefficients 311.

[0206] reconstruction

[0207] The reconstruction unit 314 (e.g., summer 314) is used to add the reconstruction residual block 313 to the prediction block 365 to obtain the reconstruction block 315 in the pixel domain, for example, by adding the pixel values ​​of the reconstruction residual block 313 and the pixel values ​​of the prediction block 365.

[0208] Filtering

[0209] Loop filter unit 320 (in or after the encoding loop) is used to filter the reconstructed block 315 to obtain filtered block 321, thereby facilitating pixel transformation or improving video quality. Loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, loop filter unit 320 may include a deblocking filter, a SAO filter, and an ALF filter. The filtering process may be performed in the order of deblocking filter, SAO filter, and ALF filter. As another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) may be added. This process is performed before deblocking. For example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 320 in... Figure 3 The loop filter is shown in the diagram, but in other configurations, the loop filter unit 320 can be implemented as a post-loop filter.

[0210] Decoding image buffer

[0211] The decoded video block 321 in one image is then stored in the decoded image buffer 330, which stores the decoded image 331 as a reference image. The reference image is used for subsequent motion compensation for other images and / or output displays respectively.

[0212] The decoder 30 is used to output the decoded image 311 through the output terminal 312, etc., for display to the user or for the user to view.

[0213] predict

[0214] Inter-frame prediction unit 344 is functionally identical to inter-frame prediction unit 244 (especially motion compensation unit), and intra-frame prediction unit 354 is functionally identical to inter-frame prediction unit 254. It determines segmentation or partitioning and performs prediction based on segmentation and / or prediction parameters or corresponding information received from coded image data 21 (e.g., parsed and / or decoded by entropy decoding unit 304). Pattern application unit 360 can be used to perform prediction (intra-frame or inter-frame prediction) for each block based on the reconstructed image, block, or corresponding pixel (filtered or unfiltered), resulting in prediction block 365.

[0215] When a video slice is encoded as an intra-coded (I) slice, the intra-prediction unit 354 in the mode application unit 360 generates a prediction block 365 for the current video slice based on the indicated intra-prediction mode and data from the previous decoded block of the current image. When a video image is encoded as an inter-coded (i.e., B or P) slice, the inter-prediction unit 344 (e.g., a motion compensation unit) in the mode application unit 360 generates a prediction block 365 for the current video slice based on motion vectors and other syntax elements received from the entropy decoding unit 304. For inter-prediction, these prediction blocks can be generated from one of the reference images in one of the reference image lists. The video decoder 30 can construct reference frame lists 0 and 1 using the default construction technique based on the reference images stored in the DPB 330. In addition to slices (e.g., video slices) or as a substitute for slices, the same or similar processes can be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), such as video can be encoded using I, P, or B coding block groups and / or coding blocks.

[0216] The pattern application unit 360 is used to determine prediction information for video blocks in the current video slice by parsing motion vectors and other syntax elements, and to generate prediction blocks for the current video slice being decoded using the prediction information. For example, the pattern application unit 360 uses some received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction), inter-frame prediction slice type (e.g., B-slice, P-slice, or GPB-slice), construction information for one or more reference image lists for the slice, motion vectors for each inter-frame coded video block in the slice, inter-frame prediction state for each inter-frame coded video block in the slice, and other information to decode video blocks within the current video slice. In addition to slices (e.g., video slices) or as alternatives to slices, the same or similar process can be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), for example, where video can be encoded using I, P, or B coding block groups and / or coding blocks.

[0217] In one embodiment, Figure 3The video encoder 30 can also be used to segment and / or decode images using slices (also called video slices), where images can be segmented or decoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., coded blocks in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0218] In one embodiment, Figure 3 The video decoder 30 shown can also be used to segment and / or decode an image using slice / coded block groups (also known as video coded block groups) and / or coded blocks (also known as video coded blocks), wherein the image can be segmented or decoded using one or more slice / coded block groups (typically non-overlapping), each slice / coded block group may include one or more blocks (e.g., CTUs) or one or more coded blocks, wherein each coded block may be rectangular or the like, and may include one or more complete or partial blocks (e.g., CTUs).

[0219] Other variations of the video decoder 30 can be used to decode the encoded image data 21. For example, the decoder 30 can generate an output video stream without the loop filter unit 320. For example, the non-transform-based decoder 30 can directly dequantize the residual signal in certain blocks or frames without the inverse transform processing unit 312. In another implementation, the video decoder 30 may have a dequantization unit 310 and an inverse transform processing unit 312 combined into a single unit.

[0220] It should be understood that in encoder 20 and decoder 30, the processing result of the current step can be further processed and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, further operations can be performed on the processing result of interpolation filtering, motion vector derivation, or loop filtering, such as clipping or shifting operations.

[0221] It should be noted that further calculations can be performed on the derived motion vector of the current block (including but not limited to control point motion vectors in affine mode, affine, planar, sub-block motion vectors in ATMVP mode, time motion vectors, etc.). For example, the value of the motion vector can be restricted to a predefined range based on the representation bits of the motion vector. If the representation bits of the motion vector are bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents exponentiation. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of four 4×4 sub-blocks in an 8×8 block) is restricted such that the maximum difference between the integer parts of the MV of the four 4×4 sub-blocks does not exceed N pixels, for example, not more than 1 pixel. Two methods for restricting motion vectors based on bitDepth are provided here.

[0222] Although the above embodiments primarily describe video encoding and decoding, it should be noted that embodiments of the decoding system 10, encoder 20, and decoder 30, as well as other embodiments described herein, can also be used for still image processing or encoding and decoding, i.e., the processing or encoding and decoding of a single image independent of any previous or consecutive images in video encoding and decoding. Generally, if image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and inter-frame prediction unit 344 (decoder) may not be available. All other functions (also referred to as tools or techniques) of the video encoder 20 and video decoder 30 can also be used for still image processing, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy coding 270, and entropy decoding 304.

[0223] Please refer to Figure 5 , Figure 5 This is an exemplary block diagram of a video decoding device 500 provided for embodiments of this application. The video decoding device 500 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 500 may be a decoder, such as... Figure 1a The video decoder 30 in the text can also be an encoder, for example... Figure 1a The video encoder 20 in the middle.

[0224] The video decoding device 500 includes: an input port 510 (or input port 510) and a receiver unit (Rx) 520 for receiving data; a processor, logic unit, or central processing unit (CPU) 530 for processing data; for example, the processor 530 may be a neural network processor 530; a transmitter unit (Tx) 540 and an output port 550 (or output port 550) for transmitting data; and a memory 560 for storing data. The video decoding device 500 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the input port 510, receiver unit 520, transmitter unit 540, and output port 550 for the entry or exit of optical or electrical signals.

[0225] Processor 530 is implemented in both hardware and software. Processor 530 may be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 530 communicates with ingress port 510, receiver unit 520, transmitter unit 540, egress port 550, and memory 560. Processor 530 includes a decoding module 570 (e.g., a neural network-based decoding module 570). Decoding module 570 implements the embodiments disclosed above. For example, decoding module 570 performs, processes, prepares, or provides various encoding operations. Therefore, decoding module 570 provides a substantial improvement to the functionality of video decoding device 500 and affects the switching of video decoding device 500 to different states. Alternatively, decoding module 570 may be implemented with instructions stored in memory 560 and executed by processor 530.

[0226] Memory 560 includes one or more disks, tape drives, and solid-state drives, which can be used as overflow data storage devices to store such programs when an executable program is selected, and to store instructions and data read during program execution. Memory 560 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0227] Please refer to Figure 6 , Figure 6This is an exemplary block diagram of the device 600 provided in the embodiments of this application. The device 600 can be used as... Figure 1a The source device 12 and the destination device 14 are either or both of them.

[0228] The processor 602 in device 600 may be a central processing unit. Alternatively, processor 602 may be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. While the disclosed implementation may be implemented using a single processor, such as processor 602 as shown, using more than one processor is faster and more efficient.

[0229] In one implementation, the memory 604 in device 600 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 604. Memory 604 may include code and data 606 accessed by processor 602 via bus 612. Memory 604 may also include an operating system 608 and an application program 610, which includes at least one program that allows processor 602 to perform the methods described herein. For example, application program 610 may include applications 1 to N, and also includes a video decoding application that performs the methods described herein.

[0230] The device 600 may also include one or more output devices, such as a display 618. In one example, the display 618 may be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 618 may be coupled to the processor 602 via a bus 612.

[0231] Although bus 612 in device 600 is described herein as a single bus, bus 612 may include multiple buses. Furthermore, auxiliary storage may be directly coupled to other components of device 600 or accessed via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, device 600 can have a wide variety of configurations.

[0232] Since the embodiments of this application involve the application of neural networks, for ease of understanding, some nouns or terms used in the embodiments of this application will be explained below, and these nouns or terms are also part of the content of the invention.

[0233] (1) Neural Network

[0234] A neural network (NN) is a machine learning model. A neural network can be composed of neural units, which are computational units that take xs and an intercept of 1 as input. The output of this computational unit can be:

[0235]

[0236] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0237] (2) Deep Neural Networks

[0238] Deep neural networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0239] (3) Convolutional Neural Network

[0240] A convolutional neural network (CNN) is a deep neural network with convolutional structures. It is a deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, CNN is a feedforward artificial neural network, where each neuron responds to an input image. A CNN contains a feature extractor consisting of convolutional layers and pooling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as performing convolution with an input image or a convolutional feature map using a trainable filter.

[0241] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution processing on the input signal. A convolutional layer can contain multiple convolution operators, also called kernels. In image processing, a convolution operator acts as a filter to extract specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during the convolution operation, the weight matrix extends to the entire depth of the input image. Therefore, convolving with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same size (rows × columns) are used instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. This dimension can be understood as being determined by the "multiple" factors mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix can be used to extract edge information, another to extract specific colors, and yet another to blur unwanted noise. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these weight matrices also have the same size. These extracted feature maps are then merged to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by these trained weight values ​​can be used to extract information from the input image, enabling the convolutional neural network to make correct predictions. When a convolutional neural network has multiple convolutional layers, the initial convolutional layers often extract more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by later convolutional layers become increasingly complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem being solved.

[0242] Because it's often necessary to reduce the number of training parameters, pooling layers are frequently introduced periodically after convolutional layers. This can be a single convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range as the result of average pooling. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0243] After processing by convolutional / pooling layers, a convolutional neural network (CNN) is still insufficient to output the required information. As mentioned earlier, convolutional / pooling layers only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the CNN needs to utilize neural network layers to generate one or a set of desired class numbers of output. Therefore, the neural network can include multiple hidden layers, the parameters of which can be pre-trained based on training data relevant to a specific task type, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0244] Optionally, after the multiple hidden layers in the neural network, there is also an output layer of the entire convolutional neural network. This output layer has a loss function similar to the classification cross-entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the backpropagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0245] (4) Recurrent Neural Network

[0246] Recurrent neural networks (RNNs) are used to process sequential data. In traditional neural network models, the layers from the input layer to the hidden layer and then to the output layer are fully connected, but the nodes within each layer are unconnected. While this type of neural network has solved many difficult problems, it remains inadequate for many others. For example, predicting the next word in a sentence generally requires using the preceding words because words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is related to the outputs of previous sequences. Specifically, the network memorizes previous information and applies it to the calculation of the current output; that is, nodes within the same hidden layer are no longer unconnected but connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous time step. Theoretically, RNNs can process sequential data of any length. Training an RNN is similar to training a traditional CNN or DNN. This algorithm also uses the backpropagation algorithm, but with one key difference: when an RNN is expanded, its parameters, such as W, are shared; however, this is not the case with traditional neural networks as illustrated above. Furthermore, in gradient descent, the output at each step depends not only on the network at the current step but also on the states of the network in previous steps. This learning algorithm is called Backpropagation Through Time (BPTT).

[0247] Since we already have convolutional neural networks (CNNs), why do we need recurrent neural networks (RNNs)? The reason is simple. CNNs rely on the fundamental assumption that elements are independent of each other, and that input and output are also independent—like a cat and a dog. However, in the real world, many elements are interconnected. For example, stock prices fluctuate over time. Or, imagine someone saying, "I love traveling, and my favorite place is Yunnan. I definitely want to go there someday." Humans know the answer to this question is "Yunnan." Humans can infer from context, but how can machines do the same? This is where RNNs come in. RNNs aim to give machines the ability to remember, just like humans. Therefore, the output of an RNN depends on both the current input information and historical memory information.

[0248] (5) Loss Function

[0249] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0250] (6) Backpropagation algorithm

[0251] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0252] (7) Generative Adversarial Networks

[0253] Generative adversarial networks (GANs) are a type of deep learning model. This model comprises at least two modules: a generative model and a discriminative model. These two modules learn from each other through a game-like interaction, resulting in better outputs. Both the generative and discriminative models can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GANs is as follows: Taking an image-generating GAN as an example, suppose there are two networks, G (Generator) and D (Discriminator). G is a network that generates images by receiving random noise z and using this noise, denoted as G(z). D is a discriminative network used to determine whether an image is "real." Its input parameter is x, representing an image, and its output D(x) represents the probability that x is a real image. A value of 1 indicates that the image is 100% real, while a value of 0 indicates that the image is impossible to be real. During the training of this generative adversarial network (GAN), the goal of the generative network G is to generate realistic images to deceive the discriminator network D, while the goal of the discriminator network D is to distinguish the images generated by G from real images as much as possible. Thus, G and D constitute a dynamic "game," which is the "adversarial" aspect of the GAN. Ideally, the game will result in G generating images G(z) that are sufficiently realistic, while D struggles to determine whether the images generated by G are real or not, i.e., D(G(z)) = 0.5. This yields a superior generative model G that can be used to generate images.

[0254] Figure 7a This is a schematic diagram illustrating an application scenario provided in an embodiment of this application, such as... Figure 7a As shown, this application scenario involves a device acquiring data, compressing the acquired data, and then storing the compressed data. This device can integrate the functions of both the aforementioned source and destination devices.

[0255] 1. The device acquires data.

[0256] 2. The equipment compresses the data to obtain compressed data.

[0257] 3. The device stores compressed data.

[0258] It should be understood that the device compresses the data to save storage space. Optionally, the device can store the compressed data in a photo album or a cloud photo album.

[0259] 4. The device decompresses the compressed data to obtain the data.

[0260] Figure 7b This is a schematic diagram illustrating an application scenario provided in an embodiment of this application, such as... Figure 7b As shown, this application scenario involves the source device acquiring data, compressing the acquired data to obtain compressed data, and then sending the compressed data to the destination device.

[0261] In this embodiment of the application, the source device can compress the acquired data before transmitting it to the destination device, which can reduce the transmission bandwidth.

[0262] 1. Source device acquires data.

[0263] 2. The source device compresses the data to obtain compressed data.

[0264] 3. The source device sends compressed data to the destination device.

[0265] Compressing data before transmission by the source device can reduce transmission bandwidth and improve transmission efficiency.

[0266] 4. The target device decompresses the compressed data to obtain the data.

[0267] Please refer to Figure 8 , Figure 8 This is a flowchart illustrating an encoding / decoding method 800 provided in an embodiment of this application. The encoding / decoding method 800 can be executed by an encoder and a decoder. The encoding / decoding method 800 is described as a series of steps or operations; it should be understood that the encoding / decoding method 800 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 8 The execution order is shown. Figure 8 As shown, the encoding / decoding method 800 may include:

[0268] Step 801: The encoder acquires the data to be encoded.

[0269] For example, the encoder acquires the data x to be encoded.

[0270] Step 802: The encoder inputs the data to be encoded into the first encoding network to obtain the target parameters.

[0271] The target parameters can be the weights of all or part of the convolutions and nonlinear activations of the second encoding network.

[0272] Optionally, the first encoding network may include a convolution kernel generator (convolutional or fully connected group) for generating target parameters based on the data to be encoded.

[0273] For example, the encoder inputs the data to be encoded x into the first encoding network to obtain the target parameter θ. g .

[0274] Step 803: The encoder constructs a second encoding network based on the target parameters.

[0275] For example, the encoder is based on the target parameter θ g Constructing the second coding network g a (x;θ g ).

[0276] Step 804: The encoder inputs the data to be encoded into the second encoding network to obtain the first feature.

[0277] The first feature is used to reconstruct the data to be encoded; it can also be called a content feature. For example, the first feature can be a three-dimensional feature map of the data to be encoded, x.

[0278] For example, the encoder inputs the data to be encoded x into the second encoding network g. a (x;θ g To obtain the first feature y, where y satisfies y = g a (x;θ g ).

[0279] Step 805: The encoder encodes the first feature to obtain the encoded bitstream (i.e. the bitstream to be decoded).

[0280] In one possible implementation, the encoder encodes the first feature to obtain an encoded bitstream, which may include: the encoder first rounding the first feature to obtain a rounded value; then, the encoder performs probability estimation on the rounded value of the first feature to obtain an estimated probability distribution of the rounded value; and finally, the encoder performs entropy encoding on the rounded value of the first feature based on the estimated probability distribution to obtain the encoded bitstream. Here, the rounded value of the first feature can be referred to as the first value feature or the content rounding feature.

[0281] For example, the encoder first rounds the first feature y to obtain the rounded value of the first feature. Then the encoder takes the integer value of the first feature. Perform probability estimation to obtain the estimated probability distribution of the rounded values ​​of the first feature. The encoder then estimates the probability distribution of the rounded value of the first feature. The integer value of the first feature Entropy encoding is performed to obtain the encoded bitstream.

[0282] in, satisfy round means rounding down to the nearest integer.

[0283] satisfy For entropy estimation networks.

[0284] Optionally, the encoder performs probability estimation on the rounded value of the first feature to obtain an estimated probability distribution of the rounded value of the first feature, which may include: the encoder performing probability estimation on the rounded value of the first feature based on first information to obtain an estimated probability distribution of the rounded value of the first feature. The first information includes at least one of context information and edge information.

[0285] It is worth mentioning that estimating the probability distribution using contextual and edge information can improve the accuracy of the estimated probability distribution, thereby reducing the code rate in the entropy coding process and reducing the entropy coding overhead.

[0286] Step 806: The encoder sends the encoded bitstream to the decoder.

[0287] Step 807: The decoder decodes the encoded bitstream to obtain the rounded value of the first feature.

[0288] In one possible implementation, the decoder decodes the encoded bitstream to obtain the rounded value of the first feature, which may include: the decoder first performs probability estimation on the rounded value of the first feature in the encoded bitstream to obtain an estimated probability distribution of the rounded value of the first feature; then the decoder performs entropy decoding on the encoded bitstream based on the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

[0289] Step 808: The decoder inputs the rounded value of the first feature into the decoding network to obtain the decoded data.

[0290] For example, the decoder will round up the value of the first feature. Input decoding network Obtain decoded data Among them, decoded data satisfy The parameter weights are for all or part of the convolutions and non-linear activations of the decoding network.

[0291] In existing encoding methods, the encoding network (i.e., the second encoding network) uses fixed parameter weights to extract the content features (i.e., the first feature) of the data to be encoded, and then encodes the content features into a bitstream (i.e., the encoded bitstream) and sends it to the decoding end. The decoding end then decodes and reconstructs the bitstream to obtain the decoded data. It can be seen that the parameter weights of the encoding network in the prior art are unrelated to the data to be encoded. However, in the encoding method provided in this application, the data to be encoded is first input into the first encoding network, then the first encoding network generates parameter weights for the second encoding network based on the data to be encoded, and then dynamically adjusts the parameter weights of the second encoding network according to the obtained weights. This makes the parameter weights of the second encoding network related to the data to be encoded, increasing the expressive power of the second encoding network. This allows the decoded data obtained by the decoding end through the bitstream encoded with the first feature to be more closely similar to the data to be encoded, thereby improving the rate-distortion performance of the encoding and decoding network.

[0292] The encoding / decoding method 800 provided in this application embodiment is applicable to... Figure 9 The encoding / decoding system shown. For example... Figure 9 As shown, the encoding and decoding system includes a first encoding network 901, a second encoding network 902, a rounding module 903, an entropy estimation network 904, an entropy encoding module 905, an entropy decoding module 906, and a decoding network 907.

[0293] like Figure 9 As shown, the data to be encoded is first input into the first encoding network 901 to obtain the target parameters, and then the parameters of the second encoding network 902 are adjusted by the target parameters (that is, the parameter weights of all or part of the convolution and nonlinear activation of the second encoding network 902 are adjusted by the target parameters).

[0294] The data to be encoded is input into the second encoding network 902 to obtain the first feature.

[0295] The rounding module 903 rounds the first feature to obtain the rounded value of the first feature.

[0296] The entropy estimation network 904 performs probability estimation on the rounded value of the first feature to obtain the estimated probability distribution of the rounded value of the first feature.

[0297] The entropy coding module 905 performs entropy coding on the rounded value of the first feature based on the estimated probability distribution of the rounded value of the first feature to obtain the coded bitstream.

[0298] The entropy decoding module 906 performs entropy decoding on the encoded bitstream based on the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

[0299] The rounded value of the first feature is input into the decoding network 907 to obtain the decoded data.

[0300] Please refer to Figure 10 , Figure 10 This is a flowchart illustrating an encoding / decoding method 1000 provided in an embodiment of this application. The encoding / decoding method 1000 can be executed by an encoder and a decoder. The encoding / decoding method 1000 is described as a series of steps or operations; it should be understood that the encoding / decoding method 1000 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 10 The execution order is shown. Figure 10 As shown, the encoding / decoding method 1000 may include:

[0301] Step 1001: The encoder acquires the data to be encoded.

[0302] Step 1002: The encoder inputs the data to be encoded into the second encoding network to obtain the first feature.

[0303] The first feature is used to reconstruct the data to be encoded.

[0304] Step 1003: The encoder inputs the data to be encoded into the first encoding network to obtain the second feature.

[0305] The second feature is used to reconstruct the target parameters. The second feature can also be called the model feature. The target parameters are the weights of all or part of the convolutions and non-linear activations of the second decoding network.

[0306] In one possible implementation, the encoder can first divide the first feature into two parts (first sub-feature and second sub-feature) along the channel dimension. One part is used to reconstruct the data to be encoded (first sub-feature), and the other part is used to reconstruct the target parameters (second sub-feature). The encoder then inputs the second sub-feature into the first encoding network to obtain the second feature.

[0307] Optionally, to enable the second feature to be compressed with a smaller bitstream, it can be transformed using convolutional and fully connected networks before being input into the third encoding network. The second feature before transformation can be called the initial model feature, and the second feature after transformation can be called the model feature.

[0308] Step 1004: The encoder encodes the first feature to obtain the first code stream to be decoded.

[0309] Step 1005: The encoder encodes the second feature to obtain the second code stream to be decoded.

[0310] In one possible implementation, the encoder can encode the first feature and the second feature to obtain the bitstream to be decoded.

[0311] Step 1006: The encoder sends the first and second code streams to be decoded to the decoder.

[0312] Step 1007: The decoder decodes the first bitstream to be decoded to obtain the rounded value of the first feature.

[0313] In one possible implementation, the decoder decoding the first bitstream to be decoded to obtain the rounded value of the first feature may include: the decoder performing probability estimation on the rounded value of the first feature in the first bitstream to be decoded to obtain an estimated probability distribution of the rounded value of the first feature; and performing entropy decoding on the bitstream to be decoded according to the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

[0314] In one possible implementation, the above-mentioned probability estimation of the rounded value of the first feature in the first bitstream to be decoded to obtain the estimated probability distribution of the rounded value of the first feature includes: performing probability estimation of the rounded value of the first feature in the first bitstream to be decoded based on first information to obtain the estimated probability distribution of the rounded value of the first feature, wherein the first information includes at least one of context information and side information.

[0315] Step 1008: The decoder decodes the second bitstream to be decoded to obtain the rounded value of the second feature.

[0316] The rounded value of the second feature can also be called the model rounding feature.

[0317] In one possible implementation, the decoder decodes the second bitstream to be decoded to obtain the rounded value of the second feature, including: the decoder performs probability estimation on the rounded value of the second feature in the second bitstream to be decoded to obtain an estimated probability distribution of the rounded value of the second feature; and performs entropy decoding on the second bitstream to be decoded based on the estimated probability distribution of the rounded value of the second feature to obtain the rounded value of the second feature.

[0318] In one possible implementation, the above-mentioned probability estimation of the rounded value of the second feature in the second bitstream to be decoded to obtain the estimated probability distribution of the rounded value of the second feature includes: performing probability estimation of the rounded value of the second feature in the second bitstream to be decoded based on first information to obtain the estimated probability distribution of the rounded value of the second feature, wherein the first information includes at least one of context information and side information.

[0319] Step 1009: The decoder inputs the rounded value of the second feature into the first decoding network to obtain the target parameters.

[0320] Step 1010: The decoder constructs a second decoding network based on the target parameters.

[0321] Step 1011: The decoder inputs the rounded value of the first feature into the second decoding network to obtain the decoded data.

[0322] In existing decoding methods, the decoding network (i.e., the second decoding network) uses fixed parameter weights to decode and reconstruct the content value features (i.e., the rounded value of the first feature) of the data to be encoded. It can be seen that the parameter weights of the decoding network in the prior art are unrelated to the data to be decoded. However, in this embodiment, the content features and model features (i.e., the first and second features) of the data to be decoded are encoded into a code stream to be decoded. The decoding end then decodes the code stream to obtain the rounded value of the second feature. This rounded value is input into the first decoding network to obtain the parameter weights of the second decoding network. The parameter weights of the second decoding network are then dynamically adjusted based on these weights, making them relevant to the data to be decoded. This improves the expressive power of the second decoding network and makes the decoded data obtained by the second decoding network more closely resemble the data to be encoded, thereby improving the rate-distortion performance of the encoding / decoding network.

[0323] The encoding / decoding method 1000 provided in this application embodiment is applicable to... Figure 11 The aforementioned encoding / decoding system. For example... Figure 11 As shown, the encoding and decoding system includes a first encoding network 1101, a second encoding network 1102, a first rounding module 1103, a second rounding module 1104, an entropy estimation network 1105, a first entropy encoding module 1106, a second entropy encoding module 1107, a first entropy decoding module 1108, a second entropy decoding module 1109, a first decoding network 1110, and a second decoding network 1111.

[0324] As shown in 11, the data to be encoded is first input into the second encoding network 1102 to obtain the first feature, and then the data to be encoded is input into the first encoding network 1101 to obtain the second feature.

[0325] The first rounding module 1103 rounds the first feature to obtain the rounded value of the first feature.

[0326] The second rounding module 1104 rounds the second feature to obtain the rounded value of the second feature.

[0327] The entropy estimation network 1105 first performs probability estimation on the rounded value of the first feature to obtain the estimated probability distribution of the rounded value of the first feature, and then performs probability estimation on the rounded value of the second feature to obtain the estimated probability distribution of the rounded value of the second feature.

[0328] The first entropy encoding module 1106 entropy-encodes the rounded value of the first feature according to the estimated probability distribution of the rounded value of the first feature to obtain the first code stream to be decoded.

[0329] The second entropy encoding module 1107 entropy-encodes the rounded value of the second feature according to the estimated probability distribution of the rounded value of the second feature to obtain the second code stream to be decoded.

[0330] The first entropy decoding module 1108 performs entropy decoding on the first bitstream to be decoded based on the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

[0331] The second entropy decoding module 1109 performs entropy decoding on the second bitstream to be decoded based on the estimated probability distribution of the rounded value of the second feature to obtain the rounded value of the second feature.

[0332] The rounded value of the second feature is first input into the first decoding network 1110 to obtain the target parameter, and then the parameters of the second decoding network 1111 are adjusted according to the target parameter (that is, the parameter weights of all or part of the convolution and non-linear activation of the second decoding network 1111 are adjusted by the target parameter).

[0333] The rounded value of the first feature is input into the second decoding network 1111 to obtain the decoded data.

[0334] The encoding / decoding method 1000 provided in this application embodiment can also be applied to... Figure 12 The aforementioned encoding / decoding system. For example... Figure 12 As shown, the encoding and decoding system includes a first encoding network 1201, a second encoding network 1202, a channel-splitting module 1203, a first rounding module 1204, a second rounding module 1205, an entropy estimation network 1206, a first entropy encoding module 1207, a second entropy encoding module 1208, a first entropy decoding module 1209, a second entropy decoding module 1210, a first decoding network 1211, and a second decoding network 1212.

[0335] like Figure 12 As shown, the data to be encoded is first input into the second encoding network 1102 to obtain the first feature.

[0336] The first feature is input into the channel sub-module 1203 and is divided into a first sub-feature and a second sub-feature in the channel dimension.

[0337] The second sub-feature is input into the first coding network 1201 to obtain the second feature.

[0338] The first rounding module 1204 rounds the first sub-feature to obtain the rounded value of the first feature.

[0339] The second rounding module 1205 rounds the second feature to obtain the rounded value of the second feature.

[0340] The entropy estimation network 1206 first performs probability estimation on the rounded value of the first feature to obtain the estimated probability distribution of the rounded value of the first feature, and then performs probability estimation on the rounded value of the second feature to obtain the estimated probability distribution of the rounded value of the second feature.

[0341] The first entropy encoding module 1207 performs entropy encoding on the rounded value of the first feature according to the estimated probability distribution of the rounded value of the first feature to obtain the first code stream to be decoded.

[0342] The second entropy encoding module 1208 performs entropy encoding on the rounded value of the second feature according to the estimated probability distribution of the rounded value of the second feature to obtain the second code stream to be decoded.

[0343] The first entropy decoding module 1209 performs entropy decoding on the first bitstream to be decoded based on the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

[0344] The second entropy decoding module 1210 performs entropy decoding on the second bitstream to be decoded based on the estimated probability distribution of the rounded value of the second feature to obtain the rounded value of the second feature.

[0345] The rounded value of the second feature is first input into the first decoding network 1211 to obtain the target parameter, and then the parameters of the second decoding network 1212 are adjusted according to the target parameter (that is, the parameter weights of all or part of the convolution and non-linear activation of the second decoding network 1212 are adjusted by the target parameter).

[0346] The rounded value of the first feature is input into the second decoding network 1212 to obtain the decoded data.

[0347] Please refer to Figure 13 , Figure 13 This is a flowchart of an encoding / decoding method 1300 provided in an embodiment of this application. The encoding / decoding method 1300 can be executed by an encoder and a decoder. The encoding / decoding method 1300 is described as a series of steps or operations; it should be understood that the encoding / decoding method 1300 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 13 The execution order is shown. Figure 13 As shown, the encoding / decoding method 1300 may include:

[0348] Step 1301: The encoder acquires the data to be encoded.

[0349] Step 1302: The encoder inputs the data to be encoded into the encoding network to obtain the first feature.

[0350] Step 1303: The encoder encodes the first feature to obtain the code stream to be decoded.

[0351] Step 1304: The encoder sends the bitstream to be decoded to the decoder.

[0352] Step 1305: The decoder decodes the bitstream to be decoded to obtain the rounded value of the first feature.

[0353] In one possible implementation, the decoder decoding the bitstream to be decoded to obtain the rounded value of the first feature may include: the decoder performing probability estimation on the rounded value of the first feature in the bitstream to be decoded to obtain an estimated probability distribution of the rounded value of the first feature; and performing entropy decoding on the bitstream to be decoded according to the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

[0354] In one possible implementation, probabilistic estimation of the rounded value of a first feature in the bitstream to be decoded to obtain an estimated probability distribution of the rounded value of the first feature includes: probabilistic estimation of the rounded value of a first feature in the bitstream to be decoded based on first information to obtain an estimated probability distribution of the rounded value of the first feature, wherein the first information includes at least one of context information and side information.

[0355] Step 1306: The decoder inputs the rounded value of the first feature into the first decoding network to obtain the target parameters.

[0356] Optionally, the first decoding network may include a convolutional kernel generator (convolutional or fully connected group) for generating target parameters based on the rounded value of a first feature of the data to be encoded.

[0357] Step 1307: The decoder constructs a second decoding network based on the target parameters.

[0358] Step 1308: The decoder inputs the rounded value of the first feature into the second decoding network to obtain the decoded data.

[0359] In existing decoding methods, the decoding network (i.e., the second decoding network) uses fixed parameter weights to decode and reconstruct the content value features (i.e., the rounded value of the first feature) of the data to be encoded. It can be seen that the parameter weights of the decoding network in the prior art are unrelated to the data to be decoded. However, in this embodiment, the rounded value of the first feature is obtained by decoding the features (i.e., the first feature) of the data to be decoded into a code stream. This rounded value is then input into the first decoding network to obtain the parameter weights of the second decoding network. The parameter weights of the second decoding network are then dynamically adjusted based on these weights, making them relevant to the data to be decoded. This improves the expressive power of the second decoding network and makes the decoded data reconstructed by the second decoding network closer to the data to be encoded, thereby improving the rate-distortion performance of the encoding / decoding network.

[0360] The encoding / decoding method 1300 provided in this application embodiment is applicable to... Figure 14 The aforementioned encoding / decoding system. For example... Figure 14As shown, the encoding and decoding system includes an encoding network 1401, a rounding module 1402, an entropy estimation network 1403, an entropy encoding module 1404, an entropy decoding module 1405, a first decoding network 1406, and a second decoding network 1407.

[0361] like Figure 14 As shown, the data to be encoded is first input into the encoding network 1401 to obtain the first feature.

[0362] The rounding module 1402 rounds the first feature to obtain the rounded value of the first feature.

[0363] The entropy estimation network 1403 performs probability estimation on the rounded value of the first feature to obtain the estimated probability distribution of the rounded value of the first feature.

[0364] The entropy coding module 1404 performs entropy coding on the rounded value of the first feature according to the estimated probability distribution of the rounded value of the first feature to obtain the code stream to be decoded.

[0365] The entropy decoding module 1405 performs entropy decoding on the bitstream to be decoded based on the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

[0366] The rounded value of the first feature is input into the first decoding network 1406 to obtain the target parameters. Then, the parameters of the second decoding network 1406 are adjusted according to the target parameters (that is, the parameter weights of all or part of the convolution and non-linear activation of the second decoding network 1407 are adjusted by the target parameters).

[0367] The rounded value of the first feature is input into the second decoding network 1407 to obtain the decoded data.

[0368] Please refer to Figure 15 , Figure 15 This is a schematic diagram illustrating the performance of the encoding / decoding method provided in the embodiments of this application. Figure 15 The coordinate system in the diagram illustrates the performance of encoding and decoding the test set using the embodiments of this application and related technologies, respectively, under the Peak Signal-to-Noise Ratio (PSNR) metric. The test set is the Kodak test set, which consists of 24 images in Portable Network Graphics (PNG) format. The resolution of these 24 images can be either 768×512 or 512×768. Figure 15 In the coordinate system, the horizontal axis represents bit rate (bits per pixel, BPP), and the vertical axis represents PSNR. BPP is the number of bits used to store each pixel; the smaller the BPP, the lower the compression bit rate. PSNR is an objective standard for evaluating images; the higher the PSNR, the better the image quality.

[0369] Figure 15 In the coordinate system shown, line segment A represents an embodiment of this application, and line segment B represents related technology. Figure 15 It can be seen that, under the same BPP index, the PSNR index of the embodiments of this application is higher than that of related technologies. Under the same image compression quality (i.e., the same PSNR index), the BPP index of the embodiments of this application is lower than that of related technologies. Therefore, it can be concluded that the rate-distortion performance of the embodiments of this application is higher than that of related technologies. The embodiments of this application can improve the rate-distortion performance of the data encoding and decoding method.

[0370] The encoding and decoding methods provided in this application can be applied to scenarios including but not limited to: all services involving the collection, storage, and transmission of image, video, and voice data in electronic devices, cloud services, and video surveillance (such as electronic devices taking photos, videos, and voice recordings, photo albums, cloud photo albums, video surveillance, video conferencing, model compression, etc.).

[0371] Figure 16 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application, such as... Figure 16 As shown, in this application scenario, after the electronic device captures image data (data to be compressed), it inputs the captured image data (such as RAW, YUV, or RGB format image data) into the AI ​​encoding unit of the electronic device. The AI ​​encoding unit of the electronic device calls the first encoding network, the second encoding network, and the entropy estimation network to transform the image data into output features with lower redundancy and generate an estimated probability distribution of the output features. The arithmetic encoding unit of the electronic device calls the entropy encoding module to encode the output features into a data file based on the estimated probability distribution of the output features. The file saving unit saves the data file generated by the entropy encoding module to the corresponding storage location of the electronic device. When the electronic device needs to use the data file, it loads the data file from the corresponding storage location of the electronic device through the file loading unit and inputs it into the arithmetic decoding unit. The arithmetic decoding unit calls the entropy decoding module to decode the data file to obtain the output features and inputs the output features into the AI ​​decoding unit. The AI ​​decoding unit calls the first decoding network, the second decoding network, and the third decoding network to perform an inverse transformation on the output features, parsing the output features into image data (such as RGB image data), i.e., compressed data. The AI ​​encoding and decoding units can be deployed in the neural network processing unit (NPU) or graphics processing unit (GPU) of the electronic device. The arithmetic encoding unit, file saving unit, file loading unit, and arithmetic decoding unit can be deployed in the CPU of the electronic device.

[0372] Figure 17 A schematic diagram illustrating another application scenario provided by an embodiment of this application, such as... Figure 17 As shown, in this application scenario, the electronic device directly uploads or encodes (e.g., JPEG encoding) the stored image data (data to be compressed) to the cloud (e.g., a server). The cloud side directly inputs or decodes (e.g., JPEG decoding) the received image data into its AI encoding unit. The cloud-side AI encoding unit calls the first encoding network, the second encoding network, and the entropy estimation network to transform the image data into output features with lower redundancy and generate an estimated probability distribution of the output features. The cloud-side arithmetic encoding unit calls the entropy encoding module to encode the output features into a data file based on the estimated probability distribution of the output features. The file saving unit saves the data file generated by the entropy encoding module to the corresponding storage location on the cloud side. When an electronic device needs to use a data file, it sends a download request to the cloud. Upon receiving the download request, the cloud loads the data file from the corresponding storage location using the file loading unit and inputs it into the arithmetic decoding unit. The arithmetic decoding unit calls the entropy decoding module to decode the data file and obtain the output features. These output features are then input into the AI ​​decoding unit. The AI ​​decoding unit calls the first, second, and third decoding networks to perform an inverse transform on the output features, parsing them into image data. The cloud then sends this image data (compressed data) directly or after encoding to the electronic device.

[0373] The following will combine Figure 18 and Figure 19 An encoding / decoding apparatus for performing the above encoding / decoding method is introduced.

[0374] It is understood that, in order to achieve the above-mentioned functions, the encoding and decoding device includes hardware and / or software modules that perform the respective functions. Based on the algorithmic steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0375] This application embodiment can divide the encoding / decoding device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0376] When dividing each function into modules according to its corresponding function. Figure 18A schematic diagram of a possible composition of the encoding / decoding apparatus involved in the above embodiments is shown, such as... Figure 18 As shown, the device 1800 may include a transceiver unit 1801 and a processing unit 1802. The processing unit 1802 may implement the methods performed by the encoding device, decoding device, encoder or decoder in the above method embodiments, and / or other processes used in the techniques described herein.

[0377] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0378] When using integrated units, device 1800 may include a processing unit, a storage unit, and a communication unit. The processing unit can be used to control and manage the operation of device 1800, for example, to support device 1800 in executing the steps performed by the aforementioned units. The storage unit can be used to support device 1800 in executing stored program code and / or data. The communication unit can be used to support communication between device 1800 and other devices.

[0379] The processing unit can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination of functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage unit can be a memory. The communication unit can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or other devices that interact with other electronic devices.

[0380] In one possible implementation, the encoding / decoding apparatus involved in the embodiments of this application can be a device having... Figure 19 The device 1900 shown includes a processor 1901 and a transceiver 1902. Figure 18 The functions implemented by the transceiver unit 1801 and the processing unit 1802 can be implemented by the processor 1901.

[0381] Optionally, the device 1900 may also include a memory 1903, and the processor 1901 and the memory 1903 communicate with each other via an internal connection path. Figure 18 The relevant functions implemented by the storage unit in the memory can be implemented by memory 1903.

[0382] This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the encoding / decoding method in the above embodiments.

[0383] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the encoding / decoding method described in the above embodiments.

[0384] This application also provides an encoding / decoding apparatus, which may specifically be a chip, integrated circuit, component, or module. Specifically, the apparatus may include a connected processor and a memory for storing instructions, or the apparatus may include at least one processor for fetching instructions from external memory. When the apparatus is running, the processor can execute instructions to cause the chip to perform the encoding / decoding methods in the above-described method embodiments.

[0385] Figure 20 A schematic diagram of a chip 2000 is shown. The chip 2000 includes one or more processors 2001 and interface circuits 2002. Optionally, the chip 2000 may also include a bus 2003.

[0386] Processor 2001 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above encoding method can be completed through integrated logic circuits in the processor 2001 or through software instructions.

[0387] Optionally, the processor 2001 described above may be a general-purpose processor, a digital signal processing (DSP) processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods and steps disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0388] The interface circuit 2002 can be used to send or receive data, instructions or information. The processor 2001 can use the data, instructions or other information received by the interface circuit 2002 to process the data, instructions or other information, and can send the processed information out through the interface circuit 2002.

[0389] Optionally, the chip may also include memory, which may include read-only memory and random access memory, and provide operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0390] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0391] Optionally, the chip can be used in the electronic device or DOP involved in the embodiments of this application. Optionally, the interface circuit 2002 can be used to output the execution result of the processor 2001. For the encoding methods provided in one or more embodiments of this application, please refer to the foregoing embodiments, which will not be repeated here.

[0392] It should be noted that the functions of the processor 2001 and the interface circuit 2002 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0393] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0394] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0395] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0396] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0397] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0398] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0399] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0400] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0401] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An encoding method, characterized in that, The method includes: Obtain the data to be encoded; The data to be encoded is input into the first encoding network to obtain the target parameters; Construct a second encoding network based on the target parameters; The data to be encoded is input into the second encoding network to obtain the first feature; The first feature is encoded to obtain an encoded bitstream.

2. The method according to claim 1, characterized in that, The target parameters are the weights of all or part of the convolutions and nonlinear activations of the second encoding network.

3. The method according to claim 1 or 2, characterized in that, The process of encoding the first feature to obtain an encoded bitstream includes: The first feature is rounded down to obtain the rounded value of the first feature; The probability of the rounded value of the first feature is estimated to obtain the estimated probability distribution of the rounded value of the first feature; The rounded value of the first feature is entropy encoded based on the estimated probability distribution of the rounded value of the first feature to obtain the encoded bitstream.

4. The method according to claim 3, characterized in that, The step of probabilistically estimating the rounded value of the first feature to obtain the estimated probability distribution of the rounded value of the first feature includes: The probability of the rounded value of the first feature is estimated based on the first information to obtain the estimated probability distribution of the rounded value of the first feature. The first information includes at least one of context information and edge information.

5. A decoding method, characterized in that, The method includes: Obtain the bitstream to be decoded; The bitstream to be decoded is decoded to obtain the rounded value of the first feature and the rounded value of the second feature. The rounded value of the first feature is used to obtain the decoded data, and the rounded value of the second feature is used to obtain the target parameter. The rounded value of the second feature is input into the first decoding network to obtain the target parameter; Construct a second decoding network based on the target parameters; The rounded value of the first feature is input into the second decoding network to obtain the decoded data.

6. The method according to claim 5, characterized in that, The target parameters are the weights of all or part of the convolutions and nonlinear activations of the second decoding network.

7. The method according to claim 5 or 6, characterized in that, The bitstream to be decoded includes a first bitstream to be decoded and a second bitstream to be decoded. Decoding the bitstream to be decoded to obtain the rounded values ​​of the first feature and the second feature includes: The first bitstream to be decoded is decoded to obtain the rounded value of the first feature; The second bitstream to be decoded is decoded to obtain the rounded value of the second feature.

8. The method according to claim 7, characterized in that, Decoding the first bitstream to be decoded to obtain the rounded value of the first feature includes: The probability of the rounded value of the first feature in the first bitstream to be decoded is estimated to obtain the estimated probability distribution of the rounded value of the first feature; The first bitstream to be decoded is entropy decoded based on the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

9. The method according to claim 8, characterized in that, The step of probabilistically estimating the rounded value of the first feature in the first bitstream to be decoded to obtain the estimated probability distribution of the rounded value of the first feature includes: The probability of the rounded value of the first feature in the first bitstream to be decoded is estimated based on the first information to obtain the estimated probability distribution of the rounded value of the first feature. The first information includes at least one of context information and side information.

10. The method according to any one of claims 7 to 9, characterized in that, Decoding the second bitstream to be decoded to obtain the rounded value of the second feature includes: The probability of the rounded value of the second feature in the second bitstream to be decoded is estimated to obtain the estimated probability distribution of the rounded value of the second feature; The second bitstream to be decoded is entropy decoded based on the estimated probability distribution of the rounded value of the second feature to obtain the rounded value of the second feature.

11. The method according to claim 10, characterized in that, The step of probabilistically estimating the rounded value of the second feature in the second bitstream to be decoded to obtain the estimated probability distribution of the rounded value of the second feature includes: The probability of the rounded value of the second feature in the second bitstream to be decoded is estimated based on the first information to obtain the estimated probability distribution of the rounded value of the second feature. The first information includes at least one of context information and side information.

12. A decoding method, characterized in that, The method includes: Obtain the bitstream to be decoded; The bitstream to be decoded is decoded to obtain the rounded value of the first feature, which is used to obtain the decoded data and the target parameters. The rounded value of the first feature is input into the first decoding network to obtain the target parameters; Construct a second decoding network based on the target parameters; The rounded value of the first feature is input into the second decoding network to obtain decoded data.

13. The method according to claim 12, characterized in that, The target parameters are the weights of all or part of the convolutions and nonlinear activations of the second decoding network.

14. The method according to claim 12 or 13, characterized in that, Decoding the bitstream to be decoded to obtain the rounded value of the first feature includes: The probability of the rounded value of the first feature in the bitstream to be decoded is estimated to obtain the estimated probability distribution of the rounded value of the first feature; The bitstream to be decoded is entropy decoded based on the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

15. The method according to claim 14, characterized in that, The step of probabilistically estimating the rounded value of the first feature in the bitstream to be decoded to obtain the estimated probability distribution of the rounded value of the first feature includes: The probability of the rounded value of the first feature in the bitstream to be decoded is estimated based on the first information to obtain the estimated probability distribution of the rounded value of the first feature. The first information includes at least one of context information and side information.

16. An encoding device, characterized in that, Includes a processing circuit, the processing circuit being used for: Obtain the data to be encoded; The data to be encoded is input into the first encoding network to obtain the target parameters; Construct a second encoding network based on the target parameters; The data to be encoded is input into the second encoding network to obtain the first feature; The first feature is encoded to obtain an encoded bitstream.

17. The apparatus according to claim 16, characterized in that, The target parameters are the weights of all or part of the convolutions and nonlinear activations of the second encoding network.

18. The apparatus according to claim 16 or 17, characterized in that, The processing circuit is specifically used for: The first feature is rounded down to obtain the rounded value of the first feature; The probability of the rounded value of the first feature is estimated to obtain the estimated probability distribution of the rounded value of the first feature; The rounded value of the first feature is entropy encoded based on the estimated probability distribution of the rounded value of the first feature to obtain the encoded bitstream.

19. The apparatus according to claim 18, characterized in that, The processing circuit is specifically used for: The probability of the rounded value of the first feature is estimated based on the first information to obtain the estimated probability distribution of the rounded value of the first feature. The first information includes at least one of context information and edge information.

20. A decoding device, characterized in that, Includes a processing circuit, the processing circuit being used for: Obtain the bitstream to be decoded; The bitstream to be decoded is decoded to obtain the rounded value of the first feature and the rounded value of the second feature. The rounded value of the first feature is used to obtain the decoded data, and the rounded value of the second feature is used to obtain the target parameter. The rounded value of the second feature is input into the first decoding network to obtain the target parameter; Construct a second decoding network based on the target parameters; The rounded value of the first feature is input into the second decoding network to obtain the decoded data.

21. The apparatus according to claim 20, characterized in that, The target parameters are the weights of all or part of the convolutions and nonlinear activations of the second decoding network.

22. The apparatus according to claim 20 or 21, characterized in that, The code stream to be decoded includes a first code stream to be decoded and a second code stream to be decoded. The processing circuit is specifically used for: The first bitstream to be decoded is decoded to obtain the rounded value of the first feature; The second bitstream to be decoded is decoded to obtain the rounded value of the second feature.

23. The apparatus according to claim 22, characterized in that, The processing circuit is specifically used for: The probability of the rounded value of the first feature in the first bitstream to be decoded is estimated to obtain the estimated probability distribution of the rounded value of the first feature; The first bitstream to be decoded is entropy decoded based on the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

24. The apparatus according to claim 23, characterized in that, The processing circuit is specifically used for: The probability of the rounded value of the first feature in the first bitstream to be decoded is estimated based on the first information to obtain the estimated probability distribution of the rounded value of the first feature. The first information includes at least one of context information and side information.

25. The apparatus according to any one of claims 22 to 24, characterized in that, The processing circuit is specifically used for: The probability of the rounded value of the second feature in the second bitstream to be decoded is estimated to obtain the estimated probability distribution of the rounded value of the second feature; The second bitstream to be decoded is entropy decoded based on the estimated probability distribution of the rounded value of the second feature to obtain the rounded value of the second feature.

26. The apparatus according to claim 25, characterized in that, The processing circuit is specifically used for: The probability of the rounded value of the second feature in the second bitstream to be decoded is estimated based on the first information to obtain the estimated probability distribution of the rounded value of the second feature. The first information includes at least one of context information and side information.

27. A decoding device, characterized in that, Includes a processing circuit, the processing circuit being used for: Obtain the bitstream to be decoded; The bitstream to be decoded is decoded to obtain the rounded value of the first feature, which is used to obtain the decoded data and the target parameters. The rounded value of the first feature is input into the first decoding network to obtain the target parameters; Construct a second decoding network based on the target parameters; The rounded value of the first feature is input into the second decoding network to obtain decoded data.

28. The apparatus according to claim 27, characterized in that, The target parameters are the weights of all or part of the convolutions and nonlinear activations of the second decoding network.

29. The apparatus according to claim 27 or 28, characterized in that, The processing circuit is specifically used for: The probability of the rounded value of the first feature in the bitstream to be decoded is estimated to obtain the estimated probability distribution of the rounded value of the first feature; The bitstream to be decoded is entropy decoded based on the estimated probability distribution of the rounded value of the first feature to obtain the rounded value of the first feature.

30. The apparatus according to claim 29, characterized in that, The processing circuit is specifically used for: The probability of the rounded value of the first feature in the bitstream to be decoded is estimated based on the first information to obtain the estimated probability distribution of the rounded value of the first feature. The first information includes at least one of context information and side information.

31. An encoder, characterized in that, include: One or more processors; A non-transitory computer-readable storage medium coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the encoder to perform the method according to any one of claims 1 to 4.

32. A decoder, characterized in that, include: One or more processors; A non-transitory computer-readable storage medium coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method according to any one of claims 5 to 11 or 12 to 15.

33. A computer-readable storage medium, characterized in that, Includes a computer program, which, when executed on a computer, causes the computer to perform the method of any one of claims 1 to 4, 5 to 11, or 12 to 15.

34. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the method of any one of claims 1 to 4, 5 to 11, or 12 to 15.

Images