Encoding method and device, and decoding method and device

Abstract

Embodiments of the present application relate to the technical field of media, and disclose an encoding method and device, and a decoding method and device, for improving the image quality of AI image encoding. The method comprises: performing AI encoding on an input image so as to obtain a feature map; performing entropy encoding on the feature map so as to obtain a bitstream; performing a transformation operation on a target area of the input image so as to obtain a transformation coefficient; encoding the transformation coefficient to the bitstream; and encoding position information to the bitstream, wherein the position information is used for indicating the position of the target area in the input image.

Description

Encoding and decoding methods and apparatus

[0001] This application claims priority to Chinese Patent Application No. 202410077119.2, filed on January 18, 2024, entitled "Encoding Method", and to Chinese Patent Application No. 202410166019.7, filed on February 5, 2024, entitled "Encoding / Decoding Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of media technology, and more particularly to encoding and decoding methods and apparatus. Background Technology

[0003] Currently, significant breakthroughs have been made in Artificial Intelligence (AI) image coding, which significantly improves compression efficiency compared to commonly used image coding standards while maintaining the same subjective quality. AI image coding is widely used in various fields, such as cloud storage, visual surveillance, autonomous vehicles and equipment, image acquisition, storage and management, real-time monitoring of visual data, and media distribution.

[0004] Under lossy coding conditions at low to medium bit rates, AI image coding schemes significantly outperform traditional image coding schemes in terms of compression ratio for the same quality. However, under extremely high bit rates or near-visual lossless conditions, the image quality of AI image coding is poor due to factors such as network capacity and network generalization.

[0005] Summary of the Invention

[0006] This application provides an encoding method to improve the image quality of AI image encoding. To achieve the above objective, this application adopts the following technical solution:

[0007] In a first aspect, embodiments of this application provide an encoding method, which includes: performing AI encoding on an input image to obtain a feature map; performing entropy encoding on the feature map to obtain a bitstream; performing a transformation operation on a target region of the input image to obtain transformation coefficients; encoding the transformation coefficients into the bitstream; and encoding position information into the bitstream. The position information is used to indicate the position of the target region in the input image.

[0008] The method provided in this application additionally employs a traditional encoding scheme to enhance the quality of local image regions, compensating for the shortcomings of AI encoding networks, such as large local image quality distortion due to limited capacity and generalization. Compared to AI encoding only of the input image, additionally employing a traditional encoding scheme to enhance the quality of local image regions can improve the overall image quality of AI image encoding.

[0009] In one possible implementation, the target region can be divided into multiple image blocks; the transformation operation described above can be performed on the multiple image blocks to obtain the transformation coefficients.

[0010] The method provided in this application additionally divides a local region of the input image into multiple image blocks and performs a transformation operation on the local region of the input image, thereby improving the image quality of the local region of the input image. Compared to only performing AI encoding on the input image, the additional transformation operation on the local region of the input image can improve the image quality of AI image encoding.

[0011] In one possible implementation, the aforementioned location information can be entropy encoded into the aforementioned bitstream.

[0012] It can be seen that the location information indicating the location of the target region in the input image can be encoded into the bitstream using entropy coding.

[0013] In one possible implementation, the above transform coefficients can be entropy encoded into the above bitstream.

[0014] It can be seen that the transformation coefficients used to perform the transformation operation on the target region can be encoded into the bitstream using entropy coding.

[0015] In one possible implementation, the transform coefficients are quantized; the quantized transform coefficients are then encoded into the bitstream. Quantization can reduce the bit depth associated with the transform coefficients.

[0016] In one possible implementation, the transform parameters can be encoded (entropy encoding) into the aforementioned bitstream. Here, the transform parameters are those used in the aforementioned transform operation.

[0017] In one possible implementation, the transform operation may include at least one of the following: discrete cosine transform (DCT), discrete fourier transform (DFT), or discrete wavelet transform (DWT).

[0018] The method provided in this application additionally performs DCT, DFT, or DWT transformation operations on local regions of the input image, thereby improving the image quality of local regions of the input image. Compared to only performing AI encoding on the input image, the additional transformation operation on local regions of the input image can improve the image quality of AI image encoding.

[0019] In one possible implementation, the aforementioned location information is mask image information, which includes a first region used to indicate the target region. For example, the mask image can consist of 0s and 1s, with the region containing 1s being the target region.

[0020] It can be seen that the location information indicating the location of the target region in the input image can be saved in the form of a mask image.

[0021] In one possible implementation, the transformation coefficients mentioned above can be residuals.

[0022] Secondly, embodiments of this application provide a decoding method, which includes: performing entropy decoding on a bitstream to obtain a feature map; performing AI decoding on the feature map to obtain a reconstructed image; performing entropy decoding on the bitstream to obtain transform coefficients; performing an inverse transform operation on the transform coefficients to obtain reconstructed pixel values ​​of a target region; performing entropy decoding on the bitstream to obtain location information; and determining a fused image based on the location information, the reconstructed pixel values ​​of the target region, and the reconstructed image. The location information is used to indicate the location of the target region in the reconstructed image.

[0023] The method provided in this application additionally employs a traditional decoding scheme to enhance the quality of local image regions, compensating for the shortcomings of AI decoding networks, such as limited capacity and generalization, which lead to significant local image quality distortion. Compared to simply performing AI encoding on the bitstream to obtain a reconstructed image, additionally employing a traditional decoding scheme to enhance the quality of local image regions can improve the overall image quality of AI image decoding.

[0024] In one possible implementation, the target region in the reconstructed image can be determined based on the aforementioned location information; the pixel values ​​of the target region in the reconstructed image can be updated based on the aforementioned reconstructed pixel values ​​to obtain the aforementioned fused image.

[0025] In one possible implementation, entropy decoding is performed on the above bitstream to obtain quantized transform coefficients; the quantized transform coefficients are then dequantized to obtain the above transform coefficients.

[0026] In one possible implementation, the above transformation coefficients can be dequantized.

[0027] Thirdly, embodiments of this application provide an encoding apparatus, comprising: an encoding unit and a transformation unit. The encoding unit is configured to perform AI encoding on an input image to obtain a feature map. The encoding unit is further configured to perform entropy encoding on the feature map to obtain a bitstream. The transformation unit is configured to perform a transformation operation on a target region of the input image to obtain transformation coefficients. The encoding unit is further configured to encode the transformation coefficients into the bitstream. The encoding unit is further configured to encode position information into the bitstream, the position information indicating the position of the target region in the input image.

[0028] In one possible implementation, the transformation unit is specifically used to: divide the target region into multiple image blocks; and perform the transformation operation on the multiple image blocks to obtain transformation coefficients.

[0029] In one possible implementation, the above transform coefficients are quantized; the quantized transform coefficients are then encoded into the above bitstream.

[0030] Fourthly, embodiments of this application provide a decoding apparatus, comprising: a decoding unit, a transformation unit, and a fusion unit. The decoding unit is configured to perform entropy decoding on a bitstream to obtain a feature map. The decoding unit is further configured to perform AI decoding on the feature map to obtain a reconstructed image. The decoding unit is further configured to perform entropy decoding on the bitstream to obtain transformation coefficients. The transformation unit is configured to perform an inverse transformation operation on the transformation coefficients to obtain reconstructed pixel values ​​of a target region. The decoding unit is further configured to perform entropy decoding on the bitstream to obtain location information, the location information indicating the location of the target region. The fusion unit is configured to determine a fused image based on the location information, the reconstructed pixel values ​​of the target region, and the reconstructed image.

[0031] In one possible implementation, the fusion unit is specifically used to: determine the target region in the reconstructed image based on the location information; and update the pixel values ​​of the target region in the reconstructed image based on the reconstructed pixel values ​​to obtain the fused image.

[0032] In one possible implementation, the decoding unit is specifically used to: perform entropy decoding on the bitstream to obtain quantized transform coefficients; and perform dequantization on the quantized transform coefficients to obtain the transform coefficients.

[0033] Fifthly, embodiments of this application also provide an encoding device, which includes: at least one processor, which, when executing program code or instructions, implements the method described in the first aspect or any possible implementation thereof.

[0034] Optionally, the device may further include at least one memory for storing the program code or instructions.

[0035] In a sixth aspect, embodiments of this application also provide a decoding apparatus, which includes: at least one processor, which, when the at least one processor executes program code or instructions, implements the method described in the second aspect above or any possible implementation thereof.

[0036] In a seventh aspect, embodiments of this application also provide a bitstream storage method, the method comprising: acquiring and storing the bitstream obtained by the method described in the first aspect or any possible implementation thereof.

[0037] Eighthly, embodiments of this application also provide a bitstream storage device, which is used to acquire and store the bitstream obtained by the method described in the first aspect or any possible implementation thereof.

[0038] In a ninth aspect, embodiments of this application also provide a code stream transmission method, the method comprising: acquiring and transmitting a code stream obtained by the method described in the first aspect or any possible implementation thereof.

[0039] In a tenth aspect, embodiments of this application also provide a bitstream transmission apparatus for acquiring and transmitting a bitstream obtained by the method described in the first aspect or any possible implementation thereof.

[0040] Eleventhly, embodiments of this application also provide a computer-readable storage medium storing a bitstream obtained by the method described in the first aspect or any possible implementation thereof.

[0041] In a twelfth aspect, embodiments of this application also provide a chip, including: an input interface, an output interface, and at least one processor. Optionally, the chip further includes a memory. The at least one processor is used to execute code in the memory, and when the at least one processor executes the code, the chip implements the method described in the first aspect or any possible implementation thereof.

[0042] Alternatively, the chip described above can also be an integrated circuit.

[0043] In a thirteenth aspect, embodiments of this application also provide a computer-readable storage medium for storing a computer program that includes methods for implementing the first aspect or any possible implementation thereof.

[0044] In a fourteenth aspect, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to implement the method described in the first aspect or any possible implementation thereof.

[0045] The encoding / decoding device, computer storage medium, computer program product, and chip provided in this embodiment are all used to execute the encoding / decoding method provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the encoding / decoding method provided above, and will not be repeated here. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1a is an exemplary block diagram of a decoding system provided in an embodiment of this application;

[0048] Figure 1b is an exemplary block diagram of a video decoding system provided in an embodiment of this application;

[0049] Figure 2 is an exemplary block diagram of a video encoder provided in an embodiment of this application;

[0050] Figure 3 is an exemplary block diagram of a video decoder provided in an embodiment of this application;

[0051] Figure 4 is an exemplary block diagram of a video decoding device provided in an embodiment of this application;

[0052] Figure 5 is an exemplary block diagram of the apparatus provided in an embodiment of this application;

[0053] Figure 6 is a schematic diagram of an image compression method based on a neural network provided in an embodiment of this application;

[0054] Figure 7 is a schematic diagram of an end-to-end image coding framework provided in an embodiment of this application;

[0055] Figure 8 is a schematic diagram of a neural network structure provided in an embodiment of this application;

[0056] Figure 9 is a schematic diagram of a video communication system provided in an embodiment of this application;

[0057] Figure 10 is a flowchart illustrating an encoding method provided in an embodiment of this application;

[0058] Figure 11 is a flowchart illustrating a decoding method provided in an embodiment of this application;

[0059] Figure 12 is a schematic diagram of an encoding device provided in an embodiment of this application;

[0060] Figure 13 is a schematic diagram of a decoding device provided in an embodiment of this application;

[0061] Figure 14 is a schematic diagram of the structure of a chip provided in an embodiment of this application;

[0062] Figure 15 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0063] Figure 16 is a schematic diagram of another electronic device provided in an embodiment of this application;

[0064] Figure 17 is a schematic diagram of a neural network structure provided in an embodiment of this application;

[0065] Figure 18 is a schematic diagram of the structure of a machine video encoding system provided in an embodiment of this application;

[0066] Figure 19 is a schematic diagram of a bitstream structure provided in an embodiment of this application;

[0067] Figure 20 is a schematic diagram of a hierarchical coding scheme provided in an embodiment of this application;

[0068] Figure 21 is a schematic diagram of an end-to-end image encoding based on a super-prior structure provided in an embodiment of this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the embodiments of this application.

[0070] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone.

[0071] The terms "first" and "second," etc., in the specification and drawings of the embodiments of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0072] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0073] It should be noted that in the description of the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0074] First, the terms used in the embodiments of this application will be explained.

[0075] The Joint Photographic Experts Group (JPEG) uses artificial intelligence (AI) to create a learning-based image coding standard that provides a single-stream, compact compression domain representation, significantly improving compression efficiency compared to commonly used image coding standards while maintaining the same subjective quality. JPEG AI has wide applications across various fields. For example, it can be used in cloud storage, visual surveillance, autonomous vehicles and equipment, image acquisition, storage and management, real-time monitoring of visual data, and media distribution.

[0076] Convolutional Neural Networks (CNNs): These are neural networks that contain convolutional layers. They may also include activation layers (such as ReLU, PReLU, etc.), pooling layers, batch normalization layers (BN layers), and fully connected layers. Typical CNNs include LeNet, AlexNet, VGGNet, and ResNet. A basic CNN consists of a backbone network and a head network; complex CNNs consist of a backbone, neck, and head network.

[0077] Feature map: The three-dimensional data output by convolutional layers, activation layers, pooling layers, batch normalization layers, etc. in a convolutional neural network. The three dimensions are called width, height, and channel, respectively.

[0078] The backbone network is the first part of a convolutional neural network. Its function is to extract feature maps at multiple scales from the input image. It typically consists of convolutional layers, pooling layers, and activation layers, but does not contain fully connected layers. Generally, the feature maps output by layers closer to the input image in the backbone network have higher resolution (width and height) but fewer channels. Typical backbone networks include VGG-16, ResNet-50, and ResNeXt-101.

[0079] Head network: The last part of a convolutional neural network. Its function is to process feature maps to obtain the prediction results of the neural network output. Common head networks include fully connected layers, softmax modules, etc.

[0080] Neck network: The middle part of a convolutional neural network, its function is to further integrate and process the feature maps generated by the backbone to obtain new feature maps. A common example is the Feature Pyramid Network (FPN) in Faster R-CNN.

[0081] Bottleneck structure: A multi-layer network structure in which the input data first passes through one or more neural network layers to obtain intermediate data, and the intermediate data then passes through one or more neural network layers to obtain output data. The amount of intermediate data (i.e., the product of width, height and number of channels) is less than the amount of input data and output data.

[0082] Data encoding and decoding comprises two parts: data encoding and data decoding. Data encoding is performed on the source side (or commonly referred to as the encoder side) and typically involves processing (e.g., compressing) the raw data to reduce the amount of data required to represent it (thus enabling more efficient storage and / or transmission). Data decoding is performed on the destination side (or commonly referred to as the decoder side) and typically involves performing the inverse processing relative to the encoder side to reconstruct the original data. The "encoding and decoding" of data in the embodiments of this application should be understood as either "encoding" or "decoding" of the data. The encoding and decoding parts are also collectively referred to as encoding and decoding (CODEC).

[0083] In lossless data encoding, the original data can be reconstructed, meaning the reconstructed original data has the same quality as the original data (assuming no transmission loss or other data loss during storage or transmission). In lossy data encoding, further compression is performed through quantization to reduce the amount of data required to represent the original data, but the decoder cannot completely reconstruct the original data, meaning the quality of the reconstructed original data is lower or worse than the quality of the original data.

[0084] This application's embodiments can be applied to video data and other data requiring compression / decompression. The following description uses video data encoding (hereinafter referred to as video encoding) as an example to illustrate this application's embodiments. Other types of data (such as image data, audio data, integer data, and other data requiring compression / decompression) can be referred to the following description, and will not be repeated in this application's embodiments. It should be noted that, compared to video encoding, the encoding of audio data and integer data does not require dividing the data into blocks; instead, the data can be directly encoded.

[0085] Video coding generally refers to the processing of image sequences that form a video or video sequence. In the field of video coding, the terms "picture," "frame," or "image" can be used synonymously.

[0086] Several video coding standards fall under the category of "lossy hybrid video coding and decoding" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is typically segmented into a set of non-overlapping blocks, which are usually encoded at the block level. In other words, the encoder typically processes the video at the block (video block) level, for example, generating prediction blocks through spatial (intra-frame) prediction and temporal (inter-frame) prediction; subtracting the prediction blocks from the current block (the block currently being processed / to be processed) to obtain residual blocks; transforming and quantizing the residual blocks in the transform domain to reduce the amount of data to be transmitted (compressed), while the decoder applies the inverse processing relative to the encoder to the encoded or compressed blocks to reconstruct the current block for representation. Additionally, the encoder needs to repeat the decoder's processing steps so that the encoder and decoder generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstruct pixels for processing, i.e., encoding subsequent blocks.

[0087] In the following embodiments of the decoding system 10, the encoder 20 and the decoder 30 are described with reference to Figures 1a to 3.

[0088] Figure 1a is an exemplary block diagram of a decoding system 10 provided in an embodiment of this application, such as a video decoding system 10 (or simply decoding system 10) that can utilize the technology of the embodiments of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video decoding system 10 represent devices, etc., that can be used to perform various technologies according to the various examples described in the embodiments of this application.

[0089] As shown in Figure 1a, the decoding system 10 includes a source device 12, which provides encoded image data 21, such as encoded images, to a destination device 14 for decoding the encoded image data 21.

[0090] The source device 12 includes an encoder 20, and optionally may include an image source 16, a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22.

[0091] Image source 16 may include or may be any type of image capture device for capturing real-world images, and / or any type of image generation device, such as a computer graphics processor for generating computer animation images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage device storing any of the images described above.

[0092] To distinguish the processing performed by the preprocessor (or preprocessing unit) 18, the image (or image data) 17 may also be referred to as the raw image (or raw image data) 17.

[0093] The preprocessor 18 receives the raw image data 17 and preprocesses it to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component.

[0094] The video encoder (or encoder) 20 is used to receive preprocessed image data 19 and provide encoded image data 21 (which will be further described below with reference to Figure 2, etc.).

[0095] The communication interface 22 in the source device 12 can be used to: receive encoded image data 21 and send encoded image data 21 (or other arbitrarily processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0096] The target device 14 includes a decoder 30, and optionally may include a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32 and a display device 34.

[0097] The communication interface 28 in the destination device 14 is used to receive encoded image data 21 (or other processed versions) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0098] Communication interfaces 22 and 28 can be used to send or receive encoded image data (or encoded data 21) through a direct communication link between source device 12 and destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0099] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing, so as to transmit it on a communication link or communication network.

[0100] Communication interface 28 corresponds to communication interface 22. For example, it can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21.

[0101] Both communication interface 22 and communication interface 28 can be configured as a one-way communication interface or a two-way communication interface as indicated by the arrow pointing from the source device 12 to the destination device 14 in FIG1a, and can be used to send and receive messages, etc., to establish a connection, acknowledge and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0102] The video decoder (or decoder) 30 is used to receive encoded image data 21 and provide decoded image data (or decoded image data) 31 (which will be further described below with reference to Figure 3, etc.).

[0103] The post-processor 32 is used to post-process the decoded image data 31 (also known as the reconstructed image data) to obtain post-processed image data 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, trimming or resampling, or any other processing to generate the decoded image data 31 for display by the display device 34, etc.

[0104] Display device 34 is used to receive post-processed image data 33 to display the image to a user or viewer. Display device 34 can be or includes any type of display for representing the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0105] The decoding system 10 also includes a training engine 25, which is used to train the encoder 20 (especially the entropy coding unit 270 in the encoder 20) or the decoder 30 (especially the entropy decoding unit 304 in the decoder 30) to entropy encode the image block to be encoded according to the estimated probability distribution obtained. For a detailed description of the training engine 25, please refer to the following method test example.

[0106] Although Figure 1a shows source device 12 and destination device 14 as independent devices, device embodiments may also include both source device 12 and destination device 14, or the functions of both source device 12 and destination device 14, that is, simultaneously including source device 12 or its corresponding functions and destination device 14 or its corresponding functions. In these embodiments, source device 12 or its corresponding functions and destination device 14 or its corresponding functions may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0107] As described, the presence and (accurate) division of different units or functions in the source device 12 and / or destination device 14 shown in Figure 1a may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0108] Please refer to Figure 1b, which is an exemplary block diagram of a video decoding system 40 provided in an embodiment of this application. An encoder 20 (e.g., a video encoder 20) or a decoder 30 (e.g., a video decoder 30), or both, can be implemented through processing circuitry in the video decoding system 40 shown in Figure 1b. This processing circuitry can include, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video encoding processors, or any combination thereof. Please refer to Figures 2 and 3, where Figure 2 is an exemplary block diagram of a video encoder provided in an embodiment of this application, and Figure 3 is an exemplary block diagram of a video decoder provided in an embodiment of this application. The encoder 20 can be implemented through processing circuitry 46 to include various modules discussed with reference to the encoder 20 in Figure 2 and / or any other encoder system or subsystem described herein. The decoder 30 can be implemented through processing circuitry 46 to include various modules discussed with reference to the decoder 30 in Figure 3 and / or any other decoder system or subsystem described herein. The processing circuitry 46 can be used to perform various operations discussed below. As shown in Figure 4, if some of the technology is implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors, thereby performing the technology of the embodiments of this application. One of the video encoder 20 and the video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), as shown in Figure 1b.

[0109] Source device 12 and destination device 14 can include any of a variety of devices, including any type of handheld or fixed device, such as laptops or tablets, mobile phones, smartphones, tablets or tablets, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (e.g., content service servers or content distribution servers), broadcast receiving devices, broadcast transmitting devices, and monitoring devices, etc., and may or may not use any type of operating system. Source device 12 and destination device 14 can also be devices in a cloud computing scenario, such as virtual machines in a cloud computing scenario. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0110] Source device 12 and destination device 14 can install virtual reality (VR), augmented reality (AR), or mixed reality (MR) applications, and can run VR, AR, or MR applications based on user actions (such as clicking, touching, swiping, shaking, voice control, etc.). Source device 12 and destination device 14 can acquire images / videos of any object in the environment through cameras and / or sensors, and then display virtual objects on the display device based on the acquired images / videos. These virtual objects can be virtual objects in VR, AR, or MR scenes (i.e., objects in the virtual environment).

[0111] It should be noted that, in this embodiment of the application, the virtual scene application in the source device 12 and the destination device 14 can be an application built into the source device 12 and the destination device 14 themselves, or it can be an application provided by a third-party service provider that is installed by the user. No specific limitation is made in this regard.

[0112] Furthermore, source device 12 and destination device 14 can be equipped with real-time video transmission applications, such as live streaming applications. Source device 12 and destination device 14 can capture images / videos via cameras and then display the captured images / videos on a display device.

[0113] In some cases, the video decoding system 10 shown in Figure 1a is merely exemplary. The technology provided in this application embodiment can be applied to video encoding settings (e.g., video encoding or video decoding), which do not necessarily include any data communication between the encoding device and the decoding device. In other examples, data is retrieved from local memory, sent over a network, etc. The video encoding device can encode data and store it in memory, and / or the video decoding device can retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data into memory and / or retrieve and decode data from memory.

[0114] Please refer to Figure 1b, which is an exemplary block diagram of a video decoding system 40 provided in an embodiment of this application. As shown in Figure 1b, the video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by a processing circuit 46), an antenna 42, one or more processors 43, one or more memory storage devices 44, and / or a display device 45.

[0115] As shown in Figure 1b, the imaging device 41, antenna 42, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory storage 44, and / or display device 45 are capable of communicating with each other. In different instances, the video decoding system 40 may contain only the video encoder 20 or only the video decoder 30.

[0116] In some instances, antenna 42 can be used to transmit or receive encoded bitstreams of video data. Additionally, in some instances, display device 45 can be used to present video data. Processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Video decoding system 40 can also include an optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, etc. Furthermore, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory storage 44 can be implemented using high-speed cache memory. In other instances, processing circuitry 46 can include memory (e.g., cache, etc.) for implementing image buffers, etc.

[0117] In some instances, the video encoder 20 implemented via logic circuitry may include (e.g., implemented via processing circuitry 46 or memory storage 44) an image buffer and (e.g., implemented via processing circuitry 46) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video encoder 20 implemented via processing circuitry 46 to implement various modules discussed with reference to FIG2 and / or any other encoder system or subsystem described herein. The logic circuitry may be used to perform various operations discussed herein.

[0118] In some instances, the video decoder 30 may be implemented in a similar manner via processing circuitry 46 to implement the various modules discussed in the video decoder 30 with reference to FIG3 and / or any other decoder system or subsystem described herein. In some instances, the logic circuit-implemented video decoder 30 may include (implemented via processing circuitry 46 or memory storage 44) an image buffer and (e.g., implemented via processing circuitry 46) a graphics processing unit. The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented via processing circuitry 46 to implement the various modules discussed in FIG3 and / or any other decoder system or subsystem described herein.

[0119] In some instances, antenna 42 can be used to receive encoded bitstreams of video data. As discussed herein, the encoded bitstream may contain data related to encoded video frames, indicators, index values, mode selection data, etc., such as data related to code segmentation (e.g., transform coefficients or quantized transform coefficients, optional indicators, and / or data defining code segmentation). Video decoding system 40 may also include a video decoder 30 coupled to antenna 42 for decoding the encoded bitstream. Display device 45 is used to display the video frames.

[0120] It should be understood that, for the examples described with reference to video encoder 20 in this application embodiment, video decoder 30 can be used to perform the reverse process. Regarding signaling syntax elements, video decoder 30 can be used to receive and parse such syntax elements, and accordingly decode the associated video data. In some examples, video encoder 20 can entropy-encode syntax elements into an encoded video bitstream. In such instances, video decoder 30 can parse such syntax elements and accordingly decode the associated video data.

[0121] For ease of description, embodiments of this application are described with reference to the Universal Video Coding (VVC) reference software or the High-Efficiency Video Coding (HEVC) developed by the ITU-T Video Coding Experts Group (VCEG) and the Joint Collaboration Team on Video Coding (JCT-VC) of the ISO / IEC Moving Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of this application are not limited to HEVC or VVC.

[0122] Encoders and Encoding Methods

[0123] As shown in Figure 2, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 shown in Figure 2 can also be called a hybrid video encoder or a video encoder based on a hybrid video codec.

[0124] Image and image segmentation (images and patches)

[0125] Encoder 20 can be used to receive images (or image data) 17 via input terminal 201, for example, images in an image sequence forming a video or video sequence. The received images or image data can also be pre-processed images (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 can also be referred to as the current image or the image to be encoded (especially in video encoding when the current image is distinguished from other images, such as those in the same video sequence, i.e., the video sequence that also includes the current image, previously encoded images, and / or decoded images).

[0126] A digital image is, or can be viewed as, a two-dimensional array or matrix of pixels with intensity values. Pixels in an array are also called pixels (short for image element). The number of pixels in the array or image along the horizontal and vertical directions (or axes) determines the image size and / or resolution. To represent color, three color components are typically used, meaning an image can be represented as or comprise an array of three pixels. In RBG format or color space, an image includes corresponding arrays of red, green, and blue pixels. However, in video coding, each pixel is typically represented in a luma / chroma format or color space, such as YCbCr, which includes the luma component indicated by Y (sometimes also represented by L) and two chroma components represented by Cb and Cr. The luma component Y represents the brightness or grayscale level intensity (e.g., both are the same in grayscale images), while the two chroma components Cb and Cr represent the chroma or color information components. Accordingly, a YCbCr format image consists of a luminance pixel array for the luminance pixel value (Y) and two chrominance pixel arrays for the chrominance values ​​(Cb and Cr). An RGB format image can be converted or transformed to YCbCr format, and vice versa; this process is also known as color conversion or transformation. If the image is black and white, it may only include the luminance pixel array. Accordingly, the image can be, for example, a monochrome format luminance pixel array or a 4:2:0, 4:2:2, and 4:4:4 color format luminance pixel array and two corresponding chrominance pixel arrays.

[0127] In one embodiment, the video encoder 20 may include an image segmentation unit (not shown in FIG. 2) for segmenting image 17 into multiple (typically non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC), coding tree blocks (CTBs), or coding tree units (CTUs) in the H.265 / HEVC and VVC standards. The segmentation unit may be used to apply the same block size and a corresponding grid with defined block sizes to all images in a video sequence, or to vary the block size between images, subsets of images, or groups of images, segmenting each image into corresponding blocks.

[0128] In other embodiments, the video encoder may be used to directly receive blocks 203 of image 17, such as one, several, or all of the blocks that make up image 17. Image block 203 may also be referred to as the current image block or the image block to be encoded.

[0129] Similar to image 17, image block 203 is also a two-dimensional array or matrix composed of pixels with intensity values ​​(pixel values), but image block 203 is smaller than that of image 17. In other words, block 203 may include a pixel array (e.g., a luminance array in the case of monochrome image 17 or a luminance or chrominance array in the case of a color image) or a three-pixel array (e.g., a luminance array and two chrominance arrays in the case of color image 17) or any other number and / or type of array depending on the color format used. The number of pixels in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Accordingly, the block may be an M×N (M columns × N rows) pixel array, or an M×N transform coefficient array, etc.

[0130] In one embodiment, the video encoder 20 shown in FIG2 is used to encode the image 17 block by block, for example, to perform encoding and prediction for each block 203.

[0131] In one embodiment, the video encoder 20 shown in FIG2 can also be used to segment and / or encode images using slices (also called video slices), wherein the images can be segmented or encoded using one or more slices (typically non-overlapping). Each slice may include one or more blocks (e.g., coding tree units, CTUs) or one or more groups of blocks (e.g., coded tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0132] In one embodiment, the video encoder 20 shown in FIG2 can also be used to segment and / or encode an image using slice / encoding block groups (also referred to as video encoding block groups) and / or encoding blocks (also referred to as video encoding blocks), wherein the image can be segmented or encoded using one or more slice / encoding block groups (typically non-overlapping), each slice / encoding block group may include one or more blocks (e.g., CTUs) or one or more encoding blocks, wherein each encoding block may be rectangular or the like, and may include one or more complete or partial blocks (e.g., CTUs).

[0133] Residual calculation

[0134] The residual calculation unit 204 is used to calculate the residual block 205 based on the image block (or original block) 203 and the prediction block 265 in the following manner (the prediction block 265 is described in detail later): for example, the residual block 205 in the pixel domain is obtained by subtracting the pixel value of the prediction block 265 from the pixel value of the image block 203 pixel by pixel.

[0135] Quantification

[0136] Quantization unit 208 is used to quantize the transform coefficients 207 by, for example, scalar quantization or vector quantization, to obtain quantized transform coefficients 209. Quantized transform coefficients 209 can also be called quantized residual coefficients 209.

[0137] The quantization process can reduce the bit depth associated with some or all of the transform coefficients 207. For example, n-bit transform coefficients can be rounded down to m-bit transform coefficients during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different scales can be applied to achieve finer or coarser quantization. Smaller quantization steps correspond to finer quantization, while larger quantization steps correspond to coarser quantization. The appropriate quantization step size can be indicated by the quantization parameter (QP). For example, the quantization parameter can be an index to a predefined set of appropriate quantization steps. For example, a smaller quantization parameter can correspond to fine quantization (smaller quantization step size), a larger quantization parameter can correspond to coarse quantization (larger quantization step size), and vice versa. Quantization may include division by the quantization step size, while corresponding or inverse dequantization performed by the dequantization unit 210, etc., may include multiplication by the quantization step size. Embodiments of some HEVC standards, for example, can be used to determine the quantization step size using the quantization parameter. In general, the quantization step size can be calculated using a fixed-point approximation of an equation involving division based on the quantization parameter. Additional scaling factors can be introduced for quantization and dequantization to recover the norm of the residual block, which may have been modified by the scaling used in the fixed-point approximation of the equations used for the quantization step size and quantization parameters. In one exemplary implementation, the scaling of the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where the loss increases with the quantization step size.

[0138] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) can be used to output the quantization parameter (QP), for example, directly outputting it or outputting it after being encoded or compressed by the entropy encoding unit 270, for example, so that the video decoder 30 can receive it and use it for decoding.

[0139] Inverse Quantization

[0140] The dequantization unit 210 is used to perform dequantization on the quantization coefficients by the quantization unit 208 to obtain the dequantization coefficients 211. For example, it performs a dequantization scheme based on or using the same quantization step size as the quantization unit 208 to perform the quantization scheme performed by the quantization unit 208. The dequantization coefficients 211 can also be called dequantization residual coefficients 211, corresponding to the transform coefficients 207. However, due to the loss caused by quantization, the dequantization coefficients 211 are usually not exactly the same as the transform coefficients.

[0141] reconstruction

[0142] The reconstruction unit 214 (e.g., summer 214) is used to add the transform block 213 (i.e., the reconstruction residual block 213) to the prediction block 265 to obtain the reconstruction block 215 in the pixel domain, for example, by adding the pixel values ​​of the reconstruction residual block 213 and the pixel values ​​of the prediction block 265.

[0143] segmentation

[0144] Segmentation unit 262 can divide (or divide) an image block (or CTU) 203 into smaller parts, such as small blocks of square or rectangular shape. For an image with a three-pixel array, a CTU consists of N×N luma pixel blocks and two corresponding chroma pixel blocks. The maximum allowed size of the luma blocks in a CTU is specified as 128×128 in the developing Versatile Video Coding (VVC) standard, but may be specified as a value different from 128×128 in the future, such as 256×256. Image CTUs can be grouped / set into slices / coded block groups, coded blocks, or bricks. A coded block covers a rectangular area of ​​an image, and a coded block can be divided into one or more bricks. A brick consists of multiple rows of CTUs within a coded block. A coded block that is not divided into multiple bricks can be called a brick. However, a brick is a true subset of a coded block and therefore is not called a coded block. VVC supports two coded block group modes: raster scan slice / coded block group mode and rectangular slice mode. In raster scan coded block group mode, a slice / coded block group contains a sequence of coded blocks in a raster scan of an image. In rectangular slice mode, a slice contains multiple bricks of an image, which together form a rectangular region of the image. The bricks within a rectangular slice are arranged in the raster scan order of the slice's bricks. These smaller blocks (also called sub-blocks) can be further subdivided into even smaller parts. This is also called tree segmentation or hierarchical tree segmentation, where the root block at root tree level 0 (level 0, depth 0), etc., can be recursively segmented into two or more blocks at the next lower tree level, such as nodes at tree level 1 (level 1, depth 1). These blocks can then be further segmented into two or more blocks at the next lower level, such as tree level 2 (level 2, depth 2), etc., until the segmentation ends (because the termination criteria are met, such as reaching the maximum tree depth or minimum block size). Blocks that are not further segmented are also called leaf blocks or leaf nodes of the tree. A tree divided into two parts is called a binary tree (BT), a tree divided into three parts is called a ternary tree (TT), and a tree divided into four parts is called a quad tree (QT).

[0145] Entropy coding

[0146] Entropy coding unit 270 is used to apply entropy coding algorithms or schemes (e.g., variable length coding (VLC), context adaptive VLC (CALVC), arithmetic coding schemes, binarization algorithms, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to quantization residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements, to obtain encoded image data 21 that can be output as an encoded bitstream 21 through output terminal 272, so that video decoder 30 and the like can receive and use the parameters for decoding. The encoded bitstream 21 can be transmitted to video decoder 30, or stored in memory for later transmission or retrieval by video decoder 30.

[0147] Other architectural variations of the video encoder 20 can be used to encode the video stream. For example, a non-transform-based encoder 20 can directly quantize the residual signal in certain blocks or frames without the transform processing unit 206. In another implementation, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0148] Decoder and Decoding Method

[0149] As shown in Figure 3, the video decoder 30 is used to receive, for example, encoded image data 21 (e.g., encoded bitstream 21) encoded by the encoder 20, to obtain a decoded image 331. The encoded image data or bitstream includes information for decoding the encoded image data, such as data representing image blocks (and / or groups or blocks of encoded video segments) and associated syntax elements.

[0150] In the example of Figure 3, the decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded image buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. The inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, the video decoder 30 may perform a decoding process that is generally the reverse of the encoding process described with reference to the video encoder 100 in Figure 2.

[0151] As described in encoder 20, the inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded image buffer DPB 230, inter-frame prediction unit 344, and intra-frame prediction unit 354 also constitute the "built-in decoder" of video encoder 20. Correspondingly, inverse quantization unit 310 can be functionally identical to inverse quantization unit 110, inverse transform processing unit 312 can be functionally identical to inverse transform processing unit 122, reconstruction unit 314 can be functionally identical to reconstruction unit 214, loop filter 320 can be functionally identical to loop filter 220, and decoded image buffer 330 can be functionally identical to decoded image buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 is correspondingly applicable to the corresponding units and functions of video decoder 30.

[0152] Entropy Decoding

[0153] The entropy decoding unit 304 is used to parse the bitstream 21 (or generally coded image data 21) and perform entropy decoding on the coded image data 21 to obtain quantization coefficients 309 and / or decoded coding parameters (not shown in Figure 3), such as any one or all of inter-frame prediction parameters (e.g., reference image index and motion vector), intra-frame prediction parameters (e.g., intra-frame prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 can be used to apply the decoding algorithm or scheme corresponding to the coding scheme of the entropy coding unit 270 of the encoder 20. The entropy decoding unit 304 can also be used to provide inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 can receive video slices and / or video block-level syntax elements. Furthermore, or as an alternative to slices and corresponding syntax elements, it can receive or use coded block groups and / or coded blocks and corresponding syntax elements.

[0154] Inverse Quantization

[0155] The dequantization unit 310 can be used to receive quantization parameters (QP) (or generally information related to dequantization) and quantization coefficients from encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and dequantize the decoded quantization coefficients 309 based on the quantization parameters to obtain dequantization coefficients 311, which may also be referred to as transform coefficients 311. The dequantization process may include using the quantization parameters calculated by the video encoder 20 for each video block in the video slice to determine the degree of quantization, and also to determine the degree of dequantization to be performed.

[0156] reconstruction

[0157] The reconstruction unit 314 (e.g., summer 314) is used to add the reconstruction residual block 313 to the prediction block 365 to obtain the reconstruction block 315 in the pixel domain, for example, by adding the pixel values ​​of the reconstruction residual block 313 and the pixel values ​​of the prediction block 365.

[0158] Other variations of the video decoder 30 can be used to decode the encoded image data 21. For example, the decoder 30 can generate an output video stream without the loop filter unit 320. For example, the non-transform-based decoder 30 can directly dequantize the residual signal in certain blocks or frames without the inverse transform processing unit 312. In another implementation, the video decoder 30 may have a dequantization unit 310 and an inverse transform processing unit 312 combined into a single unit.

[0159] It should be understood that in encoder 20 and decoder 30, the processing result of the current step can be further processed and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, further operations can be performed on the processing result of interpolation filtering, motion vector derivation, or loop filtering, such as clipping or shifting operations.

[0160] It should be noted that further calculations can be performed on the derived motion vector of the current block (including but not limited to control point motion vectors in affine mode, affine, planar, sub-block motion vectors in ATMVP mode, time motion vectors, etc.). For example, the value of the motion vector can be restricted to a predefined range based on the representation bits of the motion vector. If the representation bits of the motion vector are bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents exponentiation. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of four 4×4 sub-blocks in an 8×8 block) is restricted such that the maximum difference between the integer parts of the MV of the four 4×4 sub-blocks does not exceed N pixels, for example, not more than 1 pixel. Two methods for restricting motion vectors based on bitDepth are provided here.

[0161] Although the above embodiments primarily describe video encoding and decoding, it should be noted that embodiments of the decoding system 10, encoder 20, and decoder 30, as well as other embodiments described herein, can also be used for still image processing or encoding and decoding, i.e., the processing or encoding and decoding of a single image independent of any previous or consecutive images in video encoding and decoding. Generally, if image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and inter-frame prediction unit 344 (decoder) may not be available. All other functions (also referred to as tools or techniques) of the video encoder 20 and video decoder 30 can also be used for still image processing, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy coding 270, and entropy decoding 304.

[0162] Please refer to Figure 4, which is an exemplary block diagram of a video decoding device 400 provided in an embodiment of this application. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be a decoder, such as the video decoder 30 in Figure 1a, or an encoder, such as the video encoder 20 in Figure 1a.

[0163] The video decoding device 400 includes: an input port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; for example, the processor 430 may be a neural network processor 430; a transmitter unit (Tx) 440 and an output port 450 (or output port 550) for transmitting data; and a memory 460 for storing data. The video decoding device 400 may also include optical-to-electrical (OE) components and electro-optical (EO) components coupled to the input port 410, receiver unit 420, transmitter unit 440, and output port 450 for the entry or exit of optical or electrical signals.

[0164] Processor 430 is implemented in both hardware and software. Processor 430 may be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 430 communicates with ingress port 410, receiver unit 420, transmitter unit 440, egress port 450, and memory 460. Processor 430 includes a neural network-based codec 470. The neural network-based codec 470 implements the embodiments disclosed above. For example, the neural network-based codec 470 performs, processes, prepares, or provides various encoding operations. Therefore, the neural network-based codec 470 provides substantial improvements to the functionality of the video decoding device 400 and affects the switching of the video decoding device 400 to different states. Alternatively, the neural network-based codec 470 may be implemented with instructions stored in memory 460 and executed by processor 430.

[0165] Memory 460 includes one or more disks, tape drives, and solid-state drives, which can be used as overflow data storage devices to store such programs when an executable program is selected, and to store instructions and data read during program execution. Memory 460 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0166] Please refer to FIG. 5. FIG. 5 is an exemplary block diagram of a device 500 provided by an embodiment of the present application. The device 500 can be used as any one or both of the source device 12 and the destination device 14 in FIG. 1a.

[0167] The processor 502 in the device 500 can be a central processing unit. Alternatively, the processor 502 can be any other type of device or multiple devices that can manipulate or process information, existing or to be developed in the future. Although a single processor such as the processor 502 shown in the figure can be used to implement the disclosed implementations, using more than one processor is faster and more efficient.

[0168] In one implementation, the memory 504 in the device 500 can be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device can be used as the memory 504. The memory 504 can include code and data 506 that the processor 502 accesses through the bus 512. The memory 504 can also include an operating system 508 and application programs 510. The application programs 510 include at least one program that allows the processor 502 to execute the methods described herein. For example, the application programs 510 can include Applications 1 to N, and also include a video decoding application that executes the methods described herein.

[0169] The device 500 can also include one or more output devices, such as a display 518. In one example, the display 518 can be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch inputs. The display 518 can be coupled to the processor 502 through the bus 512.

[0170] Although the bus 512 in the device 500 is described herein as a single bus, the bus 512 can include multiple buses. In addition, the auxiliary storage can be directly coupled to other components of the device 500 or accessed through a network, and can include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, the device 500 can have various configurations.

[0171] Image Coding:

[0172] Today, multimedia data occupies most of the Internet traffic. The compression of image data plays an important role in the storage and efficient transmission of multimedia data. Therefore, image coding technology is a very practical technology. It should be noted that the Chinese translations of the English terms "image coding" and "image encoding" are usually "图像编码". Image coding is broad and includes the process of encoding an image into a bitstream and the process of decoding (decoding) the bitstream into an image.

[0173] Image coding refers specifically to the process of encoding images into bitstreams. Research on image coding has a long history. Researchers have proposed numerous methods and developed I-frame coding methods for various image and video coding standards, including JPEG, JPEG2000, JPEG-XL, JPEG-XX, WebP, H.264 / AVC, H.264 / HEVC, H.26 / VVC, AVS3, and AV1. Most of these methods are based on transform, prediction, and entropy coding techniques. Although these methods are currently widely used, the increasing volume of image data and the emergence of new media types necessitate coding methods with higher compression efficiency.

[0174] Image encoding based on deep learning:

[0175] In recent years, researchers have studied image coding methods based on deep learning. Some researchers have achieved good results. For example, Balle et al. proposed an end-to-end optimized image coding method that outperforms existing best image coding methods and even the best existing traditional coding standard H.265 / HEVC.

[0176] Image coding based on deep learning is based on deep neural networks, typically convolutional neural networks. Some research has proposed an image coding method based on Transformer networks. The structure of a deep neural network can be designed manually or obtained through neural architecture search (NAS). The parameters of the deep neural network are obtained using a loss function and a backpropagation algorithm.

[0177] Figure 6 illustrates a typical deep learning-based image compression method, also known as a neural network-based image compression method. Generally, neural network-based image compression methods include the following components: a feature extraction module, a feature quantization module, an entropy encoding module, an entropy decoding module, a feature dequantization module, and a feature decoding module. On the encoder side, the feature extraction module uses a nonlinear mapping activation function to obtain the extracted 3D feature map through multiple convolutional layers. The feature quantization module quantizes the floating-point feature values ​​using eigenvalue quantization to obtain quantized feature values. Lossless entropy encoding is then performed on the quantized feature values ​​to obtain the encoded bitstream. When the entropy-encoded bitstream is received, the decoder performs lossless entropy decoding to obtain the 3D quantized feature values. The feature decoding module decodes the features into a reconstructed image to achieve decoding.

[0178] After the image to be compressed passes through the feature extraction module and the feature quantization module, a three-dimensional feature quantization map is obtained. When processing each feature value in the three-dimensional feature quantization map, the entropy coding module can estimate and obtain the probability distribution of the feature value by using the feature value in the processed neighborhood as context, and perform subsequent coding based on the probability distribution to obtain the encoded bitstream.

[0179] With the outstanding performance of deep learning in various fields, researchers have proposed an end-to-end image encoding solution based on deep learning. Figure 7 illustrates the encoding framework. The specific technical solution is as follows: On the encoder side, the original image is input to the feature extraction module, and a feature map is output. The feature map is processed by the side information extraction module, which outputs side information. On the encoder and decoder sides, the image is input to the probability estimation module, and the probability distribution of each feature element is output to obtain the value of the feature element to be encoded. Furthermore, the feature map is input to the quantization module to obtain a quantized feature map. The entropy encoding module performs entropy encoding on each feature element based on the probability distribution of each feature element in the quantized feature map to obtain the encoded bitstream.

[0180] On the decoder side, the decoder parses the bitstream and outputs the probability distribution of the symbols to be encoded. Based on the accompanying information, the values ​​of the feature elements to be decoded are obtained. The k-entropy decoding module performs arithmetic decoding on each feature element in the quantized feature map, based on the probability distribution of each feature element, to obtain the value of the feature element. The feature map is input to the image reconstruction module, and the reconstructed image is output.

[0181] Neural Networks

[0182] A neural network can include neurons. A neuron can be a computational unit that takes xs and an intercept of 1 as input. The output of the computational unit can be:

[0183] Here, s = s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, b is the neuron's bias, and f is the neuron's activation function (activation function), used to introduce nonlinear characteristics into the neural network to convert the input signal in the neuron into an output signal. The output signal of the activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network constructed by connecting multiple individual neurons together. Specifically, the output of one neuron can be the input to another neuron. The input of each neuron can be connected to the local receptive field of the previous layer to extract features from the local receptive field. A local receptive field can be a region containing several neurons.

[0184] Convolutional Neural Networks:

[0185] A Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. A CNN includes a feature extractor, which comprises convolutional layers and subsampling layers. The feature extractor can be thought of as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to a subset of neurons in adjacent layers. A convolutional layer typically includes several feature planes, each of which may contain neurons arranged in a rectangle. Neurons on the same feature plane share a weight; this shared weight is the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size. During the training of a CNN, appropriate weights can be learned for the convolutional kernel. Furthermore, shared weights directly reduce the connections between layers in the CNN and reduce the risk of overfitting.

[0186] Figure 8 schematically illustrates the general concept of processing by neural networks such as CNNs. A convolutional neural network consists of input layers, output layers, and multiple hidden layers. The input layer is the layer that provides input (e.g., a portion of the image shown in Figure 8) for processing. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve with multiplication or other dot products. The result of a layer is one or more feature maps, sometimes called channels. Subsampling may be involved in some or all of the layers. Thus, as shown in Figure 8, the feature maps can become much smaller. The activation function in a CNN is typically a ReLU (Rectified Linear Unit) layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. Although these layers are colloquially referred to as convolutions, this is just convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is important for the indices in the matrix because it affects how weights are determined at a particular index point.

[0187] When programming a CNN for image processing, as shown in Figure 8, the input is a tensor with shape (number of images) x (image width) x (image height) x (image depth). Then, after passing through convolutional layers, the image is abstracted into a feature map with shape (number of images) x (feature map width) x (feature map height) x (feature map channels). The convolutional layers in a neural network should have the following properties: A convolutional kernel (hyperparameter) defined by width and height. The number of input and output channels (hyperparameters). The depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0188] Traditional multilayer perceptron (MLP) models have been used for image recognition. However, due to the full connectivity between nodes, they have high dimensionality and do not scale well with higher-resolution images. A 1000×1000 pixel image with RGB color channels has 3 million weights, which is too high to be efficiently processed at scale with full connectivity. Furthermore, this network architecture does not consider the spatial structure of the data, treating distant input pixels in the same way as nearby pixels. This ignores the locality of references in image data both computationally and semantically. Therefore, the full connectivity of neurons is a waste for purposes such as image recognition dominated by spatially local input patterns.

[0189] Convolutional Neural Networks (CNNs) are biologically inspired variants of multilayer perceptrons (MLPs) specifically designed to mimic the behavior of the visual cortex. These models alleviate the challenges of MLP architectures by leveraging the strong spatial local correlations present in natural images. Convolutional layers are the core building blocks of CNNs. The parameters of this layer consist of a set of learnable filters (the kernels mentioned above) with small receptive fields that extend across the entire depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, the dot product between the filter entry and the input is computed, and a two-dimensional activation map of that filter is produced. Thus, the network learns the filters to activate when it detects a specific type of feature at a certain spatial location in the input.

[0190] The activation maps of all filters stacked along the depth dimension form the complete output volume of the convolutional layer. Therefore, each entry in the output volume can also be interpreted as the output of a neuron that observes a small region of the input and shares parameters with neurons in the same activation map. A feature map, or activation map, is the output activation of a given filter. Feature maps and activations have the same meaning. In some papers, it is called an activation map because it is a mapping corresponding to the activations of different parts of an image, and also a feature map because it is a mapping that discovers a certain feature in the image. High activation means that a certain feature has been found.

[0191] Another important concept in cellular neural networks is pooling, a form of non-linear downsampling. Several non-linear functions can be used to implement pooling, with max pooling being the most common. It divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such sub-region.

[0192] Intuitively, the exact location of a feature is less important than its approximate location compared to other features. This is the idea behind using pooling in convolutional neural networks. Pooling layers are used to gradually reduce the spatial size of the representation, decreasing the number of parameters, memory footprint, and computational cost in the network, thus also controlling overfitting. In CNN architectures, it is common to periodically insert pooling layers between consecutive convolutional layers. Pooling operations provide another form of translation invariance.

[0193] Pooling layers operate independently on each depth slice of the input and spatially resize themselves. The most common form is a pooling layer with a 2×2 filter that applies a 2-sampling stride along both width and height on each depth slice of the input, discarding 75% of the activations. In this case, each maximum operation exceeds 4 digits. The depth dimension remains constant.

[0194] In addition to the maximum pool, pool units can also use other functions, such as average pooling or... 2-norm pooling. Average pooling was historically frequently used, but has recently fallen out of favor compared to max pooling, which performs better in practice. Due to the significant reduction in representation size, there's a recent trend towards using smaller filters or discarding pooling layers entirely. Region of Interest (ROI) pooling is a variant of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is a crucial component of convolutional neural networks used for object detection based on the Fast R-CNN architecture.

[0195] The ReLU above stands for Rectified Linear Unit, which applies a non-saturating activation function. By setting negative values ​​to zero, it effectively removes negative values ​​from the activation map. It increases the non-linearity of the decision function and the entire network without affecting the receptive field of the convolutional layers. Other functions are also used to increase non-linearity, such as the saturating hyperbolic tangent and sigmoid functions. ReLU is generally more popular than other functions because it trains neural networks several times faster without significantly impacting generalization accuracy.

[0196] After several convolutional and max-pooling layers, high-level inference in a neural network is accomplished through fully connected layers. Neurons in a fully connected layer are connected to all activations in the previous layer, as seen in regular (non-convolutional) artificial neural networks. Therefore, their activations can be computed as affine transformations, and the matrix multiplication yields a bias shift (a vector sum of learned or fixed bias terms).

[0197] The "loss layer" specifies how training penalizes the deviation between the predicted (output) and the true label, and is typically the last layer of the neural network. Various loss functions suitable for different tasks can be used. Softmax loss is used to predict a single class among K mutually exclusive classes. Sigmoid cross-entropy loss is used to predict K independent probability values ​​in [0, 1]. Euclidean loss is used to regress to real-valued labels.

[0198] In summary, Figure 8 illustrates the data flow in a typical convolutional neural network. First, the input image passes through a convolutional layer and is abstracted into a feature map comprising several channels, corresponding to the number of filters in a set of learnable filters for that layer (e.g., one channel per filter). The feature map is then subsampled using, for example, a pooling layer, which reduces the dimensionality of each channel in the feature map. Subsequent data enters another convolutional layer, which may have a different number of output channels, resulting in a different number of channels in the feature map. As mentioned above, the number of input and output channels are hyperparameters of that layer. To establish network connectivity, these parameters need to be synchronized between two connected layers; for example, the number of input channels in the current layer should equal the number of output channels in the previous layer. For the first layer processing input data (e.g., an image), the number of input channels is typically equal to the number of channels in the data representation, such as 3 channels for an RGB or YUV representation of an image or video, or 1 channel for a grayscale image or video representation.

[0199] The method provided in this application is applied to video codecs, such as those in video communication systems, as shown in Figure 9: After video is captured using a video acquisition device, it undergoes a series of preprocessing steps, and then the processed video is compressed and encoded to obtain an encoded bitstream. The bitstream is transmitted to the receiving module via a transmission network using a sending module, and after being decoded by a decoder, it can be rendered and displayed. Alternatively, the encoded video bitstream can also be directly stored.

[0200] This application describes a neural network-based encoding and decoding scheme for enhancement layers, which can be applied to hierarchical encoding schemes.

[0201] The method provided in this application can be used in devices or products containing video encoders and / or decoders, such as video processing hardware and software products, like chips. It can also be used in products or devices containing such chips, such as media products like mobile phones.

[0202] Figure 10 shows a flowchart of an encoding method provided in an embodiment of this application. As shown in Figure 10, the encoding method provided in this embodiment includes:

[0203] S1001. Perform AI encoding on the input image to obtain a feature map.

[0204] The AI ​​encoding mentioned above can be any AI encoder.

[0205] For example, the input image can be fed into a JPEG AI encoder for AI encoding to obtain a feature map.

[0206] S1002. Perform entropy encoding on the feature map to obtain the bitstream.

[0207] For example, the feature map can be entropy encoded using an asymmetric numeral system (ANS) algorithm to obtain the bitstream.

[0208] S1003. Perform a transformation operation on the target region of the input image to obtain the transformation coefficient.

[0209] In one possible implementation, the aforementioned variation coefficient can be the residual.

[0210] In one possible implementation, one or more regions of the input image can be selected as the target region.

[0211] For example, one or more regions of the input image can be selected as target regions based on the image distortion (image distortion degree) of the region.

[0212] For example, the region in the input image where the image distortion (image distortion degree) is greater than the image distortion threshold can be designated as the target region.

[0213] Alternatively, one or more regions of interest in the human eye of the input image can be selected as the target region.

[0214] Alternatively, one or more regions of the input image can be randomly selected as the target region.

[0215] In one possible implementation, the above transformation operation includes at least one of DCT, DFT, or DWT.

[0216] For example, DCT transformation can be performed on the target region of the input image.

[0217] In one possible implementation, the target region can be divided into multiple image blocks. The aforementioned transformation operation is then performed on these multiple image blocks.

[0218] For example, the target region can be divided into multiple image blocks with a granularity of HxW. The above transformation operation is then performed on these multiple image blocks. Here, H is a positive integer, W is a positive integer, and H and W can be the same.

[0219] For example, the target region can be divided into multiple image blocks with an 8x8 granularity based on a binary mask image. DCT operations can then be performed on these multiple image blocks.

[0220] In this context, the binary mask image represents all 8x8 image blocks in the original image that need to undergo DCT transformation, and the 8x8 blocks corresponding to the points marked as 1 in the mask image represent the image blocks that need to undergo DCT transformation.

[0221] S1004. Encode the transform coefficients into the bitstream.

[0222] In one possible implementation, the above-mentioned transform coefficients can be quantized; the quantized transform coefficients are then encoded into the above-mentioned bitstream.

[0223] In one possible implementation, the above-mentioned transform coefficients can be entropy encoded into the above-mentioned bitstream.

[0224] For example, the DC conversion coefficients and AC conversion coefficients of the input image can be encoded separately. Specifically, transforming an NxN block yields NxN conversion coefficients, with the conversion coefficient at position (0,0) being the DC conversion coefficient and the conversion coefficients at other positions being the AC conversion coefficients.

[0225] For example, differential encoding can be used for the DC conversion coefficients of the input image, that is, encoding the difference between the DC component and the previous block. The difference is divided into multiple groups according to its magnitude, and the group number of each group is encoded using the tabled asymmetric numeral systems (tANS) entropy encoding algorithm, while the magnitude within each group is encoded using a fixed-length code.

[0226] For example, the DC-DC conversion coefficients of the input image can be differentially encoded according to the table shown in Table 1. Table 1 shows the group number, the corresponding numerical range for each group, and the number of bits required for the numerical value. When using tANS encoding for the group number, the probabilities P0 to P11 of each group number are first calculated, and the probability values ​​are represented by 8-bit unsigned integers. The probabilities are written into the bitstream. At the same time, the group number is entropy encoded according to the probabilities P0 to P11 and written into the bitstream.

[0227] Table 1

[0228] For example, run-length encoding can be used for the AC transform coefficients of the input image. The encoding scheme for non-zero coefficients can be the same as the encoding scheme for the differential amplitude of the DC transform coefficients.

[0229] Run-length encoding: Encodes the number of zeros R between the current non-zero coefficient and the previous non-zero coefficient, with a maximum value of 15, and then encodes the current non-zero coefficient S; if the number of zeros is greater than 15, then encode (15,0). When R is 16, it indicates that all AC transform coefficients from the current position to the last one are zero. Entropy encoding is performed using tANS on the number of consecutive zeros R and the group number of coefficient S: The probabilities Pr of R being different values ​​from 0 to 16 are calculated, and a 1x17 probability vector is represented by 17 8-bit unsigned integers and written into the bitstream; the probabilities Ps of group number S being different values ​​from 0 to 11 are calculated, and a 1x12 probability vector is represented by 12 8-bit unsigned integers and written into the bitstream; based on Pr and Ps, ANS is used to entropy encode R and the group number of coefficient S. The magnitude of coefficient S is represented by a fixed-length code using the number of bits in its group and written into the bitstream.

[0230] For example, the AC transform coefficients of the input image can be run-length encoded according to the table shown in Table 2. Table 2 shows the group number, the corresponding numerical range for each group, and the number of bits required for the numerical value.

[0231] Table 2

[0232] S1005. Encode the location information into the bitstream.

[0233] The aforementioned location information is used to indicate the location of the target region in the aforementioned input image.

[0234] In one possible implementation, the location information can be mask image information (such as a binary mask image). This mask image information includes a first region, which is used to indicate the target region. For example, the mask image can consist of 0s and 1s, with the region containing 1s being the target region.

[0235] In one possible implementation, the aforementioned location information can be entropy encoded into the aforementioned bitstream.

[0236] For example, for an HxW image, dividing it into 8x8 blocks results in a binary mask image with a height of ((H+7)>>3) and a width of ((W+7)>>3). The mask image is then converted into a 1-dimensional 01 vector by raster scanning, and the 01 vector generated from the mask image is entropy encoded to be written into the aforementioned bitstream.

[0237] For example, run-length coding can be used to entropy encode the 01 vector generated by the mask image. An unsigned 8-bit number represents the number of zeros preceding a non-zero 1; a 0 / 1 string organized into 8 bits for each zero is used to generate the bitstream. (When the number of zeros is greater than 255, 255 is binarized and written into the bitstream, and the 256th zero is also written into the bitstream).

[0238] For example, ANS can be used to entropy encode the aforementioned 01 vector. The frequency P of 0 in the 01 vector is counted, and this frequency is taken as the probability of 0. This probability P is represented by an 8-bit integer A and written into the bitstream. The probability of 1 is (256-A) / 256. Based on the probability P, ANS is used to entropy encode the 01 vector.

[0239] The method provided in this application additionally employs a traditional encoding scheme to enhance the quality of local image regions, compensating for the shortcomings of AI encoding networks, such as large local image quality distortion due to limited capacity and generalization. Compared to AI encoding only of the input image, additionally employing a traditional encoding scheme to enhance the quality of local image regions can improve the overall image quality of AI image encoding.

[0240] In one possible implementation, the transform parameters can be encoded (entropy encoding) into the aforementioned bitstream. Here, the transform parameters are those used in the aforementioned transform operation.

[0241] Figure 11 shows a flowchart of an encoding method provided in an embodiment of this application. As shown in Figure 11, the encoding method provided in this embodiment includes:

[0242] S1101. Perform entropy decoding on the bitstream to obtain the feature map.

[0243] For example, the feature map can be obtained by performing ANS entropy decoding on the bitstream.

[0244] S1102. Perform AI decoding on the feature map to obtain the reconstructed image.

[0245] The AI ​​decoding mentioned above can be any AI decoder.

[0246] For example, the feature map can be input into the JPEGAI decoder for AI decoding to obtain the reconstructed image.

[0247] S1103. Perform entropy decoding on the bitstream to obtain the transform coefficients.

[0248] For example, the DC conversion coefficients and AC conversion coefficients in the bitstream can be entropy decoded separately.

[0249] For example, for DC coefficient decoding, the probabilities P0 to P11 of the DC differential value in each group can be decoded from the bitstream, as shown in Table 1. Based on probabilities P0 to P11, tANS entropy decoding is applied to the group numbers to obtain the group number of the current DC differential value from the bitstream. According to the number of bits required for the value within that group number, the corresponding bits are read from the bitstream to obtain the differential value. The sum of this differential value and the DC component of the previous block yields the current DC coefficient value. This process is used to obtain the DC coefficients of all DCT blocks.

[0250] For example, in decoding AC coefficients, the probability Pr of the number R between adjacent non-zero coefficients taking values ​​from 0 to 16 can be decoded from the bitstream. This 1x17 probability vector is represented by 17 8-bit unsigned integers. The probability Ps of the AC coefficient S value being different values ​​in group numbers from 0 to 11 can be decoded from the bitstream. This 1x12 probability vector is represented by 12 8-bit unsigned integers. Based on Pr, R is obtained from the bitstream using entropy decoding with ANS. When R is 16, the AC coefficients from the current position to the 63rd position are all 0; otherwise, based on Ps, the group number of coefficient S is obtained from the bitstream using entropy decoding with ANS. Based on the group where coefficient S belongs, the number of bits required for the amplitude of coefficient S is obtained. The corresponding bits are read from the bitstream to obtain the value of AC coefficient S, and the first R positions of the current coefficient S are filled with 0. By fully implementing this process, the AC coefficients of all DCT blocks are obtained.

[0251] In one possible implementation, the bitstream can be entropy-decoded to obtain the quantized transform coefficients; the quantized transform coefficients can then be dequantized.

[0252] S1104. Perform an inverse transformation operation on the transformation coefficients to obtain the reconstructed pixel values ​​of the target region.

[0253] S1105. Perform entropy decoding on the bitstream to obtain location information.

[0254] The location information is used to indicate the location of the target area, and the transformation coefficient is the coefficient used to perform the transformation operation on the target area.

[0255] In one possible implementation, the bitstream can be entropy decoded to obtain a binary mask graph representing location information.

[0256] For example, for an HxW image, dividing it into 8x8 blocks results in a binary mask image with a height of ((H+7)>>3) and a width of ((W+7)>>3). The mask image is then converted into a 1-dimensional 01 vector by raster scanning.

[0257] For example, the above 01 vector can be decoded using run-length coding. The number of zeros preceding the non-zero value 1 is represented by an unsigned 8-bit number; the number of zeros and the 0 / 1 value are generated from the 01 string in the bitstream in the order of (8-bit zero count, 1-bit 0 / 1), thus obtaining the entire binary mask image.

[0258] For example, we can use ANS to perform entropy decoding on the aforementioned 01 vector. The frequency P of 0 in the 01 vector is obtained from the decoded bitstream, and this probability P is represented by an 8-bit integer A. The probability of 1 is (256-A) / 256. Based on probability P, we use ANS to perform entropy decoding on the 01 vector to obtain a binary mask image.

[0259] S1106. Determine the fused image based on the location information, the reconstructed pixel values ​​of the target area, and the reconstructed image.

[0260] In one possible implementation, the target region in the reconstructed image can be determined based on the aforementioned location information; the pixel values ​​of the target region in the reconstructed image can be updated based on the aforementioned reconstructed pixel values ​​to obtain the aforementioned fused image.

[0261] For example, the region corresponding to the enhancement layer image patch in the reconstructed image can be determined based on the binary mask image, and the pixel value of the region corresponding to the enhancement layer image patch in the reconstructed image can be set as the reconstructed pixel value of the target region of the reconstructed image.

[0262] The encoding device used to perform the above encoding method will now be described with reference to Figure 12.

[0263] It is understood that, in order to achieve the above-mentioned functions, the encoding device includes hardware and / or software modules corresponding to the execution of each function. Based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0264] This application embodiment can divide the encoding device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0265] Figure 12 illustrates a possible configuration of the encoding device involved in the above embodiments, where each functional module is divided according to its corresponding function. This device can be an electronic device, a module applied to an electronic device (e.g., a processor, chip, or chip system), or a logic node, logic module, or software capable of implementing all or part of the functions of an electronic device. As shown in Figure 12, the encoding device 1200 may include an encoding unit 1201 and a transformation unit 1202.

[0266] The aforementioned encoding unit 1201 is used to perform AI encoding on the input image to obtain a feature map.

[0267] The aforementioned encoding unit 1201 is also used to perform entropy encoding on the aforementioned feature map to obtain a bitstream.

[0268] The aforementioned transformation unit 1202 is used to perform a transformation operation on the target region of the aforementioned input image to obtain transformation coefficients.

[0269] The aforementioned encoding unit 1201 is also used to encode the aforementioned transform coefficients into the aforementioned bitstream.

[0270] The aforementioned encoding unit 1201 is further configured to encode location information into the aforementioned bitstream, wherein the location information is used to indicate the number of positions of the aforementioned target region in the aforementioned input image.

[0271] In one possible implementation, the transformation unit 1202 is specifically used to: divide the target region into multiple image blocks; and perform the transformation operation on the multiple image blocks to obtain transformation coefficients.

[0272] In one possible implementation, the encoding unit 1201 is specifically used to: quantize the transform coefficients; and encode the quantized transform coefficients into the bitstream.

[0273] The decoding device used to perform the above decoding method will be described below with reference to Figure 13.

[0274] It is understood that, in order to achieve the above-mentioned functions, the decoding device includes hardware and / or software modules that perform the respective functions. Based on the algorithm steps of the examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0275] This application embodiment can divide the decoding device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0276] With each functional module divided according to its corresponding function, Figure 13 shows a possible schematic diagram of the decoding device involved in the above embodiments. This device can be an electronic device, a module applied to an electronic device (such as a processor, chip, or chip system), or a logic node, logic module, or software that can implement all or part of the functions of an electronic device. As shown in Figure 13, the decoding device 1300 may include: a decoding unit 1301, a transformation unit 1302, and a fusion unit 1303.

[0277] The aforementioned decoding unit 1301 is used to perform entropy decoding on the bitstream to obtain a feature map.

[0278] The aforementioned decoding unit 1301 is also used to perform AI decoding on the aforementioned feature map to obtain a reconstructed image.

[0279] The aforementioned decoding unit 1301 is also used to perform entropy decoding on the aforementioned bitstream to obtain transform coefficients.

[0280] The aforementioned transformation unit 1302 is used to perform an inverse transformation operation on the aforementioned transformation coefficients to obtain the reconstructed pixel values ​​of the target region.

[0281] The aforementioned decoding unit 1301 is also used to perform entropy decoding on the aforementioned bitstream to obtain location information, the aforementioned location information being used to indicate the location of the aforementioned target area.

[0282] The aforementioned fusion unit 1303 is used to determine a fused image based on the aforementioned location information, the reconstructed pixel values ​​of the aforementioned target region, and the aforementioned reconstructed image.

[0283] In one possible implementation, the fusion unit 1303 is specifically used to: determine the target region in the reconstructed image based on the location information; and update the pixel values ​​of the target region in the reconstructed image based on the reconstructed pixel values ​​to obtain the fused image.

[0284] In one possible implementation, the decoding unit 1301 is specifically used to: perform entropy decoding on the bitstream to obtain quantized transform coefficients; and perform dequantization on the quantized transform coefficients to obtain the transform coefficients.

[0285] This application also provides a chip, which can be the chip of the encoding or decoding device described above. Figure 14 shows a schematic diagram of the structure of a chip 1400. The chip 1400 includes one or more processors 1401 and interface circuits 1402. Optionally, the chip 1400 may also include a bus 1403.

[0286] Processor 1401 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above encoding / decoding method can be completed through integrated logic circuits in the hardware of processor 1401 or through software instructions.

[0287] Optionally, the processor 1401 described above may be a general-purpose processor, a digital signal processing (DSP) processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods and steps disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0288] The interface circuit 1402 can be used to send or receive data, instructions or information. The processor 1401 can use the data, instructions or other information received by the interface circuit 1402 to process the data, instructions or other information, and can send the processed information out through the interface circuit 1402.

[0289] Optionally, the chip may also include memory, which may include read-only memory and random access memory, providing operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0290] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0291] Optionally, the chip can be used in the encoding or decoding apparatus involved in the embodiments of this application. Optionally, the interface circuit 1402 can be used to output the execution result of the processor 1401. For the encoding and decoding methods provided in one or more embodiments of this application, please refer to the foregoing embodiments, which will not be repeated here.

[0292] It should be noted that the functions of the processor 1401 and the interface circuit 1402 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0293] Figure 15 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device can be an encoding device or a decoding device, a chip or functional module in the encoding device, or a chip or functional module in the decoding device. As shown in Figure 15, the electronic device 1500 includes a processor 1501, a transceiver 1502, and a communication line 1503.

[0294] The processor 1501 is used to execute any step of the encoding and decoding method provided in the embodiments of this application, and in the process of executing any step of the encoding and decoding method provided in the embodiments of this application, it may choose to call the transceiver 1502 and the communication line 1503 to complete the corresponding operation.

[0295] Furthermore, the electronic device 1500 may also include a memory 1504. The processor 1501, the memory 1504, and the transceiver 1502 can be connected via a communication line 1503.

[0296] The processor 1501 can be a processor, a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 1501 can also be other devices with processing capabilities, such as circuits, devices, or software modules, without limitation.

[0297] Transceiver 1502 is used to communicate with other devices or other communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. Transceiver 1502 can be a module, circuit, transceiver, or any device capable of enabling communication.

[0298] The transceiver 1502 is mainly used for sending and receiving commands and information, and may include a transmitter and a receiver to send and receive commands and information, respectively; operations other than sending and receiving commands and information are implemented by the processor.

[0299] Communication line 1503 is used to transmit information between the various components included in electronic device 1500.

[0300] In one design, the processor can be viewed as a logic circuit, and the transceiver as an interface circuit.

[0301] Memory 1504 is used to store instructions. These instructions can be computer programs.

[0302] The memory 1504 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM). Memory 1504 can also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices. It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0303] It should be noted that the memory 1504 can exist independently of the processor 1501, or it can be integrated with the processor 1501. The memory 1504 can be used to store instructions, program code, or some data, etc. The memory 1504 can be located inside or outside the electronic device 1500, without limitation. The processor 1501 is used to execute the instructions stored in the memory 1504 to implement the methods provided in the above embodiments of this application.

[0304] In one example, processor 1501 may include one or more processors, such as processor 0 and processor 1 in Figure 15.

[0305] As an alternative implementation, the electronic device 1500 may include multiple processors, for example, in addition to processor 1501 in FIG15, it may also include processor 1509.

[0306] As an optional implementation, the electronic device 1500 also includes an output device 1505 and an input device 1506. For example, the input device 1506 is a device such as a keyboard, mouse, microphone, or joystick, and the output device 1505 is a device such as a display screen or speaker.

[0307] It should be noted that the electronic device 1500 can be a chip system or a device with a similar structure to that shown in Figure 15. The chip system can be composed of chips or may include chips and other discrete devices. Actions, terms, etc., involved in the various embodiments of this application can be referred to mutually without limitation. The message names or parameter names in the messages used for interaction between devices in the embodiments of this application are merely examples; other names may be used in specific implementations without limitation. Furthermore, the composition structure shown in Figure 15 does not constitute a limitation on the electronic device 1500. In addition to the components shown in Figure 15, the electronic device 1500 may include more or fewer components than those shown in Figure 15, or combine certain components, or have different component arrangements.

[0308] The processor and transceiver described in this application can be implemented on integrated circuits (ICs), analog ICs, radio frequency integrated circuits, mixed-signal ICs, application-specific integrated circuits (ASICs), printed circuit boards (PCBs), electronic devices, etc. The processor and transceiver can also be manufactured using various IC process technologies, such as complementary metal oxide semiconductors (CMOS), n-metal-oxide-semiconductor (NMOS), p-type metal oxide semiconductors (PMOS), bipolar junction transistors (BJTs), bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.

[0309] Figure 16 is a schematic diagram of another electronic device provided in an embodiment of this application. This electronic device can be an encoding device or a decoding device, a chip or functional module in the encoding device, or a chip or functional module in the decoding device. For ease of explanation, Figure 16 only shows the main components of the electronic device, including a processor 1601, a memory 1602, a control circuit 1603, and an input / output device 1604. The processor 1601 is mainly used to process communication protocols and communication data, execute software programs, and process the data of the software programs. The memory 1602 is mainly used to store software programs and data. The control circuit 1603 is mainly used for power supply and the transmission of various electrical signals. The input / output device 1604 is mainly used to receive user input data and output data to the user.

[0310] When the electronic device is a processor 1601, the control circuit 1603 can be a motherboard, and the memory 1602 includes storage media such as hard disks, RAM, and ROM. The processor 1601 can include a baseband processor 1601 and a central processing unit (CPU). The baseband processor is mainly used to process communication protocols and communication data, while the CPU is mainly used to control the entire electronic device, execute software programs, and process data from the software programs. Input / output devices 1604 include displays, keyboards, and mice. The control circuit 1603 can further include or be connected to transceiver circuits or transceivers, such as network cable interfaces, for sending or receiving data or signals, such as for data transmission and communication with other devices. Furthermore, it can also include an antenna for transmitting and receiving wireless signals for data / signal transmission with other devices.

[0311] This application also provides an encoding device, which includes at least one processor. When the at least one processor executes program code or instructions, it implements the above-mentioned related method steps to implement the encoding method in the above embodiments.

[0312] Optionally, the device may further include at least one memory for storing the program code or instructions.

[0313] This application also provides a decoding device, which includes at least one processor. When the at least one processor executes program code or instructions, it implements the above-mentioned related method steps to achieve the decoding method in the above embodiments.

[0314] Optionally, the device may further include at least one memory for storing the program code or instructions.

[0315] This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on a communication device, the communication device performs the aforementioned method steps to implement the encoding / decoding method in the above embodiments.

[0316] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the encoding / decoding method described in the above embodiments.

[0317] This application also provides an encoding apparatus, which may specifically be a chip, integrated circuit, component, or module. Specifically, the apparatus may include a connected processor and a memory for storing instructions, or the apparatus may include at least one processor for fetching instructions from external memory. When the apparatus is running, the processor can execute the instructions to cause the chip to perform the encoding methods in the above-described method embodiments.

[0318] This application also provides a decoding device, which may specifically be a chip, integrated circuit, component, or module. Specifically, the device may include a connected processor and a memory for storing instructions, or the device may include at least one processor for fetching instructions from external memory. When the device is running, the processor can execute instructions to cause the chip to perform the decoding methods in the above-described method embodiments.

[0319] This application also provides a bitstream storage method, which includes: acquiring and storing the bitstream obtained by the above encoding and decoding method.

[0320] This application also provides a bitstream storage device for acquiring and storing bitstreams obtained by the above-described encoding and decoding methods.

[0321] This application also provides a bitstream transmission method, which includes: acquiring and transmitting the bitstream obtained by the above encoding and decoding method.

[0322] This application also provides a bitstream transmission device for acquiring and transmitting bitstreams obtained by the above-described encoding and decoding methods.

[0323] This application also provides a computer-readable storage medium storing the bitstream obtained by the above-described encoding / decoding method.

[0324] Please refer to Figure 17, which schematically illustrates the general concepts of processing via neural networks such as convolutional neural networks (CNNs). A CNN consists of input layers, output layers, and multiple hidden layers. The input layer is the layer that provides the input (a portion of the input image shown in Figure 17) for processing. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve with multiplication or other dot products. The result of a layer is one or more feature maps (represented by empty solid-line rectangles), sometimes also called channels. Resampling (such as subsampling) may be involved in some or all layers. Therefore, the feature maps may become smaller, as shown in Figure 17. Note that convolution with stride can also reduce the size of the input feature map (resampling). The activation function in a CNN is typically a ReLU (Rectified Linear Unit) layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, called hidden layers, because their inputs and outputs are masked by the activation function and the final convolution. Although these layers are colloquially referred to as convolutions, this is just convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is important for the indices in the matrix because it affects how weights are determined at specific index points.

[0325] When programming a CNN for image processing, as shown in Figure 17, the input is a tensor with shape (number of images) x (image width) x (image height) x (image depth). It should be understood that image depth can be composed of the image's channels. After passing through convolutional layers, the image is abstracted into a feature map with shape (number of images) x (feature map width) x (feature map height) x (feature map channels). Convolutional layers in a neural network should have the following properties: A convolutional kernel (hyperparameter) defined by its width and height. The number of input and output channels (hyperparameters). The depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0326] Machine video coding (VCM) is another popular area of ​​computer science today. The main idea behind this approach is to transmit coded representations of image or video information for further processing by computer vision (CV) algorithms, such as object segmentation, detection, and recognition. Unlike traditional image and video coding that targets human perception, VCM focuses on performance characteristics for computer vision tasks, such as object detection accuracy, rather than reconstruction quality. (See Figure 18.)

[0327] Machine video coding, also known as collaborative intelligence, is a relatively new paradigm for efficiently deploying deep neural networks in mobile cloud infrastructure. By partitioning the network between the mobile side 1810 and the cloud side 1890 (e.g., cloud servers), the computational workload can be distributed, minimizing the system's total energy and / or latency. Typically, collaborative intelligence is a paradigm where the processing of a neural network is distributed across two or more distinct computing nodes; for example, devices, but generally any functionally defined node. Here, the term "node" does not refer to the aforementioned neural network node. Instead, a (computing) node here refers to (physically or at least logically) independent devices / modules that implement parts of the neural network. Such devices can be different servers, different end-user devices, a mixture of servers and / or user devices and / or cloud and / or processors, etc. In other words, computing nodes can be considered as nodes belonging to the same neural network and communicating with each other to transmit encoded data within / for the neural network. For example, to perform complex computations, one or more layers can be executed on a first device (such as a device on the mobile side 1810), and one or more layers can be executed on another device (such as a cloud server on the cloud side 1890). However, the distribution can be more granular, and a single layer can be executed on multiple devices. In this disclosure, the term "multiple" refers to two or more. In some existing solutions, a portion of the neural network functionality is executed on one or more devices (such as user devices or edge devices), and the output (feature maps) is then passed to the cloud. The cloud is a collection of processing or computing systems located outside the device that is operating a portion of the neural network. The concept of collaborative intelligence has also been extended to model training. In this case, data flows bidirectionally: from the cloud to the mobile device during backpropagation of training, and from the mobile device to the cloud during forward propagation and inference of training (as shown in Figure 18).

[0328] Several works have proposed semantic image compression by encoding deep features and then reconstructing the input image from them. One example demonstrates compression based on uniform quantization, followed by context-based adaptive arithmetic coding (CABAC) from H.264. In some scenarios, it may be more efficient to send the output of the hidden layer (deep feature map) from the mobile side 1810 to the cloud 1890 than to send compressed natural image data to the cloud and perform object detection using the reconstructed image. Therefore, compressing the data (features) generated by the mobile side 1810 may be advantageous, and the mobile side 1810 can include a quantization layer 1820 for this purpose. Correspondingly, the cloud side 1890 can include an inverse quantization layer 1860. Efficient compression of feature maps is beneficial for image and video compression and reconstruction in both human perception and machine vision. Entropy coding methods, such as arithmetic coding, are a popular approach for compressing deep features (i.e., feature maps).

[0329] Please refer to Figure 19, which shows a bitstream structure provided in an embodiment of this application. As shown in Figure 19, the bitstream includes: Start of Image, File Header, Entropy Encoded Data, and End of Image.

[0330] In one possible implementation, the aforementioned location information, the aforementioned transformation coefficients, the Gaussian distribution parameter index number, the Gaussian distribution parameter, the target probability distribution, and the target probability distribution index number can be stored in the file header.

[0331] In one possible implementation, the target quality residual matrix or target quality matrix can be stored in entropy-encoded data.

[0332] Please refer to Figure 20, which shows a schematic diagram of a layered coding scheme provided in an embodiment of this application.

[0333] In video network transmission, especially in real-time multi-user communication over the internet, there are differences in network bandwidth and device processing capabilities among users. This differentiation in network bandwidth resources necessitates adaptive bitrate adjustment for different users. Layered coding proposes the concept of layered temporal, spatial, and quality domains, adding enhancement layers on top of the base layer to obtain higher frame rates, resolutions, and quality video content. Different users can choose whether or not to require the enhancement layer bitstream to match their respective electronic device processing capabilities and network bandwidth.

[0334] Figure 20 is a schematic diagram of a layered coding scheme provided in an embodiment of this application. As shown in Figure 20, the layered coding scheme includes: performing AI coding on the input image to obtain a feature map; performing entropy coding on the feature map to obtain a bitstream; writing the location information of the target region of the input image into the bitstream; performing a transformation operation on the target region of the input image; and writing the transformation coefficients used in the transformation operation into the bitstream.

[0335] Please refer to Figure 21, which illustrates an end-to-end image coding based on a super-prior structure provided in an embodiment of this application.

[0336] With the outstanding performance of deep learning in various fields, researchers have proposed end-to-end image coding schemes based on deep learning. A typical coding framework is shown in Figure 21. The specific scheme is as follows:

[0337] At the encoding end, the original image is input to the feature extraction module Ga, which outputs a feature map. This feature map is then input to the quantization module to obtain a quantized feature map. On one hand, the feature map is processed by the edge information extraction module Ha, which outputs edge information z. This edge information z is then input to the quantization module for quantization, entropy-encoded, and written into the bitstream. The encoding end performs entropy decoding to obtain the decoded bitstream, which is input to the probability estimation module Hs, which outputs the probability distribution of each feature element in the feature map (entropy encoding is performed before decoding to ensure encoding and decoding synchronization). On the other hand, the entropy encoding module entropy-encodes each feature element in the feature map based on its probability distribution, resulting in a compressed bitstream. The edge information is also a type of feature information, represented as a three-dimensional feature map, containing fewer feature elements than the feature map itself. Both the feature map and the feature graph are MxWxH, where M is the number of channels (the same as the number of channels in the last convolutional layer), and WxH is related to the width and height of the input image and the downsampling factor (stride) set for each convolutional operation in the network. In this diagram, the width and height of the input image are downsampled four times at a factor of 2.

[0338] At the decoding end, the edge information is first decoded and then input into the Hs module, which outputs the probability distribution of the symbol to be decoded. The entropy decoding module performs arithmetic decoding on each feature element in the feature map according to the probability distribution of each feature element, obtaining the value of each feature element in the feature map, thus obtaining the decoded feature map. The feature map is then input into the image reconstruction module Gs module to output the reconstructed image.

[0339] It is worth noting that in the scheme shown in Figure 2, the input image contains three RGB channels, and the R, G, and B components have the same W and H values. Image data of size 3xWxH can be input into the network for encoding and decoding. When the input image is in YUV444 format, the network shown in Figure 21 can also be used for encoding and decoding.

[0340] In this embodiment, the device, computer storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0341] It should be understood that in various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0342] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0343] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0344] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0345] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0346] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0347] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this application, essentially, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0348] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be included within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. An encoding method, characterized in that, include: Artificial intelligence (AI) encoding is performed on the input image to obtain a feature map; The feature map is entropy encoded to obtain a bitstream; Transformation coefficients are obtained by performing a transformation operation on the target region of the input image; The transform coefficients are encoded into the bitstream; Location information is encoded into the bitstream, and the location information is used to indicate the location of the target region in the input image.

2. The method according to claim 1, characterized in that, The transformation operation on the target region of the input image to obtain the transformation coefficients includes: The target region is divided into multiple image blocks; The transformation operation is performed on the plurality of image blocks to obtain the transformation coefficients.

3. The method according to claim 1 or 2, characterized in that, Encoding the transform coefficients into the bitstream includes: The transformation coefficients are quantized; The quantized transform coefficients are encoded into the bitstream.

4. The method according to any one of claims 1 to 3, characterized in that, The transformation operation includes at least one of Discrete Cosine Transform (DCT), Discrete Fourier Transform (DFT), or Discrete Wavelet Transform (DWT).

5. The method according to any one of claims 1 to 4, characterized in that, The location information is mask image information, which includes a first region used to indicate the target region.

6. A decoding method, characterized in that, include: Entropy decoding of the bitstream yields a feature map; The reconstructed image is obtained by performing AI decoding on the feature map; The transform coefficients are obtained by entropy decoding of the bitstream; The reconstructed pixel values ​​of the target region are obtained by performing an inverse transform operation on the transform coefficients; Entropy decoding is performed on the bitstream to obtain location information, which is used to indicate the location of the target area; The fused image is determined based on the location information, the reconstructed pixel values ​​of the target region, and the reconstructed image.

7. The method according to claim 6, characterized in that, The step of determining the fused image based on the location information, the reconstructed pixel values ​​of the target region, and the reconstructed image includes: The target region in the reconstructed image is determined based on the location information; The fused image is obtained by updating the pixel values ​​of the target region in the reconstructed image based on the reconstructed pixel values.

8. The method according to claim 6 or 7, characterized in that, The step of entropy decoding the bitstream to obtain transform coefficients includes: Entropy decoding is performed on the bitstream to obtain the quantized transform coefficients; The quantized transform coefficients are dequantized to obtain the transform coefficients.

9. An encoding device, characterized in that, Includes encoding units and transformation units; The encoding unit is used to perform AI encoding on the input image to obtain a feature map; The encoding unit is also used to perform entropy encoding on the feature map to obtain a bitstream; The transformation unit is used to perform a transformation operation on the target region of the input image to obtain transformation coefficients; The encoding unit is further configured to encode the transform coefficients into the bitstream; The encoding unit is further configured to encode location information into the bitstream, the location information being used to indicate the location of the target region in the input image.

10. The apparatus according to claim 9, characterized in that, The transformation unit is specifically used for: The target region is divided into multiple image blocks; The transformation operation is performed on the plurality of image blocks to obtain the transformation coefficients.

11. The apparatus according to claim 9 or 10, characterized in that, The encoding unit is specifically used for: The transformation coefficients are quantized; The quantized transform coefficients are encoded into the bitstream.

12. A decoding device, characterized in that, It includes a decoding unit, a transformation unit, and a fusion unit; The decoding unit is used to perform entropy decoding on the bitstream to obtain a feature map; The decoding unit is also used to perform AI decoding on the feature map to obtain a reconstructed image; The decoding unit is also used to perform entropy decoding on the bitstream to obtain transform coefficients; The transformation unit is used to perform an inverse transformation operation on the transformation coefficients to obtain the reconstructed pixel values ​​of the target region; The decoding unit is further configured to perform entropy decoding on the bitstream to obtain location information, the location information being used to indicate the location of the target area; The fusion unit is used to determine a fused image based on the location information, the reconstructed pixel values ​​of the target region, and the reconstructed image.

13. The apparatus according to claim 12, characterized in that, The fusion unit is specifically used for: The target region in the reconstructed image is determined based on the location information; The fused image is obtained by updating the pixel values ​​of the target region in the reconstructed image based on the reconstructed pixel values.

14. The apparatus according to claim 12 or 13, characterized in that, The decoding unit is specifically used for: Entropy decoding is performed on the bitstream to obtain the quantized transform coefficients; The quantized transform coefficients are dequantized to obtain the transform coefficients.

15. An encoding device, characterized in that, It includes at least one processor and a memory, wherein the at least one processor executes a program or instructions stored in the memory to cause the encoding device to implement the method of any one of claims 1 to 5.

16. A decoding device, characterized in that, It includes at least one processor and a memory, wherein the at least one processor executes a program or instructions stored in the memory to cause the decoding device to implement the method of any one of claims 6 to 8.

17. A method for storing a bitstream, characterized in that, include: Acquire and store the bitstream obtained by the method of any one of claims 1 to 8.

18. A bitstream storage device, characterized in that, The apparatus is used to acquire and store the bitstream obtained by the method of any one of claims 1 to 8.

19. A method for transmitting a code stream, characterized in that, include: Acquire and transmit the bitstream obtained by the method of any one of claims 1 to 8.

20. A code stream transmission device, characterized in that, The apparatus is used to acquire and transmit the bitstream obtained by the method of any one of claims 1 to 8.

21. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores the bitstream obtained by the method as described in any one of claims 1 to 8.

22. A computer-readable storage medium, characterized in that, Used to store a computer program that, when the computer program is run on a computer or processor, causes the computer or processor to perform the method of any one of claims 1 to 8.

23. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer or processor, cause the computer or processor to perform the method of any one of claims 1 to 8.

Images