Content-adaptive online training method, apparatus, and computer program for post-filtering

Abstract

Aspects of the present disclosure provide a method, an apparatus, and a non-transitory computer-readable storage medium for video decoding. The apparatus may include a processing circuit. The processing circuit is configured to receive an image or video including one or more blocks. The processing circuit may decode first post-filtering parameters in the image or video corresponding to the one or more blocks to be restored. The first post-filtering parameters are applied to at least one of the one or more blocks and are updated by a post-filtering module in a post-filtering neural network (NN) trained based on a training dataset. The processing circuit may determine a post-filtering NN in the video decoder corresponding to the one or more blocks based on the first post-filtering parameters. The processing circuit may decode the one or more blocks based on the determined post-filtering NN corresponding to the one or more blocks.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Patent Application No. 17 / 749,641, "CONTENT-ADAPTIVE ONLINE TRAINING METHOD AND APPARATUS FOR POST-FILTERING," filed on May 20, 2022, which in turn claims priority to U.S. Provisional Application No. 63 / 194,057, "Content-Adaptive Online Training Method for Post-Filtering," filed on May 27, 2021. The disclosures of the prior applications are incorporated herein by reference in their entirety.

[0002] This disclosure generally describes embodiments related to video coding. [Background technology]

[0003] The background art descriptions provided herein are intended to provide a general overview of the context of this disclosure. The work of the inventors named herein is not, to the extent that their work is described in this background art section, nor are any descriptions that may otherwise not qualify as prior art at the time of filing, expressly or implicitly recognized as prior art to this disclosure.

[0004] Video coding and decoding can be performed using interpicture prediction with motion compensation. Uncompressed digital images and / or videos can contain a series of pictures, each picture having spatial dimensions of, for example, 1920×1080 luminance samples and associated saturation samples. A series of pictures can have a fixed or variable picture rate (also informally known as frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed images and / or videos have specific bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (1920×1080 luminance sample resolution at a frame rate of 60 Hz) requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 GB of storage space.

[0005] One purpose of video coding and decoding may be to reduce redundancy in the input image and / or video signal through compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements by two orders of magnitude or more, in some cases. While the description herein uses video coding / decoding as an example for illustrative purposes, the same techniques can be applied to image coding / decoding in a similar manner without departing from the spirit of this disclosure. Both lossless and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to a technique that allows an exact copy of the original signal to be restored from the compressed original signal. With lossy compression, the restored signal may not be identical to the original signal, but the distortion between the original and restored signals is small enough to make the restored signal useful for its intended purpose. In the case of video, lossy compression is widely employed. The amount of distortion that can be tolerated depends on the application; for example, users of a particular consumer streaming application may tolerate higher distortion than users of a television distribution application. The feasible compression ratio can reflect that a higher tolerance / tolerance for distortion can lead to a higher compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transformation, quantization, and entropy coding.

[0007] Video codec techniques may include a technique known as intra coding. In intra coding, sample values ​​are represented without referencing samples or other data from a previously restored reference picture. In some video codecs, the picture is spatially subdivided into blocks of samples. A picture can be an intra picture when all blocks of samples are coded in intra mode. Intra pictures, and their derivatives such as independent decoder refresh pictures, can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and video session, or as a still image. Samples in intra blocks may be subjected to transformations, and the transformation coefficients can be quantized before entropy coding. Intra prediction may be a technique to minimize the sample values ​​in the pre-transformation region. In some cases, the smaller the post-transformation DC value and the smaller the AC coefficients, the fewer bits are required at a given quantization step size to represent the post-entropy-coded block.

[0008] Traditional intra-coding, such as that known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include, for example, techniques that try from surrounding sample data and / or metadata acquired during the encoding / decoding of spatially adjacent and preceding blocks of data in the decoding order. Such techniques will hereafter be referred to as “intra-prediction” techniques. It should be noted that, at least in some cases, intra-prediction uses only reference data from the current picture being restored, and not from the reference picture.

[0009] Intra-prediction can take many different forms. When two or more of these techniques can be used in a given video coding technique, the techniques in use can be coded in intra-prediction mode. In certain cases, a mode may have submodes and / or parameters, which may be coded individually or included in a mode codeword. The choice of codeword for a given mode / submode / parameter combination may affect the entropy coding technique used to convert the codeword into a bitstream, as this can impact the improvement in coding efficiency by intra-prediction.

[0010] Certain modes of intra-prediction were introduced in H.264, improved in H.265, and further refined in newer coding techniques such as Joint Search Models (JEM), Versatile Video Coding (VVC), and Benchmark Sets (BMS). Predictor blocks can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of neighboring samples are copied to the predictor block according to direction. References to the direction in use may be coded within the bitstream or may be predicted themselves.

[0011] Referring to Figure 1A, the lower right shows a subset of nine predictor directions known from the 33 possible predictor directions of H.265 (corresponding to 33 of the 35 intra-modes). The point where the arrows converge (101) represents the predicted sample. The arrows indicate the direction from which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples located to the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples located to the lower left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] Referring further to Figure 1A, a 4x4 sample square block (104) is depicted in the upper left (indicated by a thick dashed line). The square block (104) contains 16 samples, each labeled with "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in both the Y and X dimensions within block (104). Since the block is 4x4 samples in size, S44 is in the lower right. Further reference samples are shown following a similar numbering scheme. The reference samples are labeled with R relative to block (104), its Y position (e.g., row index), and its X position (column index). In both H.264 and H.265, the predicted samples are adjacent to the block being reconstructed, so negative values ​​do not need to be used.

[0013] Intra-picture prediction can function by copying reference sample values ​​from adjacent samples so that they are assigned by the signaled prediction direction. For example, suppose a coded video bitstream includes signaling that indicates a prediction direction corresponding to arrow (102) with respect to this block, i.e., a sample is predicted from one or more prediction samples that are to the upper right at an angle of 45 degrees from the horizontal. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, to calculate the reference sample, especially when the direction is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example, by interpolation.

[0015] The number of possible directions has been increasing as video coding technology develops. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013), and at the time of this disclosure, JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, and certain techniques of entropy coding are used to represent those likely directions with fewer bits, accepting a certain penalty for less likely directions. Further, the directions themselves can sometimes be predicted from the neighboring directions used in already decoded adjacent blocks.

[0016] FIG. 1B shows a schematic diagram (110) depicting 65 intra prediction directions by JEM to show an increasing number of prediction directions over time.

[0017] The mapping of intra prediction direction bits in a coded video bitstream representing directions may vary for each video coding technology and can range from a simple direct mapping of prediction directions to intra prediction modes to complex adaptive schemes including codewords, the most likely modes, and similar techniques. However, in all cases, there may be certain directions that statistically occur less likely than certain other directions within the video content. Since the purpose of video compression is redundancy reduction, those less likely directions are represented with more bits than the more likely directions in well-functioning video coding technologies.

[0018] Motion compensation can be an irreversible compression technique and can relate to a technique in which blocks of sample data from a previously restored picture or a part thereof (reference picture) are spatially shifted in a direction indicated by a motion vector (hereinafter, MV) and then used for prediction of a newly restored picture or a part of a picture. In some cases, the reference picture can be the same as the picture currently being restored. The MV can have two dimensions X and Y, or three dimensions, the third dimension being an indication of the reference picture in use (the latter can be indirectly the time dimension).

[0019] In some video compression techniques, the MV applicable to a particular region of sample data can be predicted from other MVs, for example, from an MV associated with another region of sample data that is spatially adjacent to the region being restored and that precedes that MV in decoding order. By doing so, the amount of data required for coding the MV can be significantly reduced, thereby eliminating redundancy and increasing the compression ratio. For example, when coding an input video signal derived from a camera (known as natural video), MV prediction can function effectively because there is a statistical likelihood that regions larger than a region to which a single MV is applicable move in a similar direction, and thus, in some cases, a similar motion vector derived from the MVs of adjacent regions can be used for prediction. As a result, the MV detected for a given region is similar or identical to the MV predicted from surrounding MVs and can be represented with fewer bits than the number of bits used when directly coding the MV after entropy coding. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself can be lossy, for example, due to rounding errors when calculating a predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec.H.265, "High Efficiency Video Coding," December 2016). Of the many MV prediction mechanisms provided by H.265, the technique described herein is hereafter referred to as "spatial merging."

[0021] Referring to Figure 2, the current block (201) contains samples that the encoder found to be predictable from a previous block of the same size, but spatially shifted, during the motion search process. Instead of directly coding its MV, the MV can be derived from metadata associated with one or more reference pictures, for example, from the most recent reference picture (in decoding order), using the MV associated with one of the five surrounding samples, denoted as A0, A1, and B0, B1, B2 (202-206, respectively). In H.265, the MV prediction can use a predictor from the same reference picture used by adjacent blocks. [Overview of the Initiative] [Means for solving the problem]

[0022] Aspects of this disclosure provide methods and apparatus for video coding and video decoding. In some examples, the apparatus for video decoding includes a processing circuit. The processing circuit is configured to receive an image or video containing one or more blocks. The processing circuit can decode a first post-filtering parameter in the image or video corresponding to one or more blocks to be restored. The first post-filtering parameter is applied to at least one of the one or more blocks, and the first post-filtering parameter is updated by a post-filtering module in a post-filtering neural network (NN) trained on a training dataset. Based on the first post-filtering parameter, the processing circuit can determine a post-filtering NN in a video decoder corresponding to one or more blocks. Based on the determined post-filtering NN corresponding to one or more blocks, the processing circuit can decode one or more blocks.

[0023] In one embodiment, a processing circuit decodes a second post-filtering parameter in an image or video corresponding to one or more blocks. The processing circuit can further determine a post-filtering NN based on the second post-filtering parameter. The second post-filtering parameter is applied to a second block within one or more blocks, which is different from at least one of the one or more blocks. The second post-filtering parameter is updated by a post-filtering module within the post-filtering NN.

[0024] In one embodiment, the first post-filtering parameter corresponds to the second image to be restored. The processing circuit can decode the second image based on the determined post-filtering neural network.

[0025] In one embodiment, the first post-filtering parameter is different from the second post-filtering parameter. The first post-filtering parameter can be applied to the content of a first block among one or more blocks, and the second post-filtering parameter can be applied to the content of a second block.

[0026] In one embodiment, the first post-filtering parameter is updated based on a bias term or weighting coefficient in the post-filtering neural network.

[0027] In one embodiment, the post-filtering neural network (NN) is composed of initial parameters. The processing circuit updates at least one of the initial parameters with a first post-filtering parameter.

[0028] In one embodiment, coding information corresponding to one or more blocks indicates the difference between a first post-filtering parameter and one of the initial parameters. The processing circuit can determine the first post-filtering parameter according to the sum of the difference and one of the initial parameters.

[0029] In one embodiment, the first post-filtering parameter is updated in (i) a single layer of the post-filtering NN, (ii) multiple layers of the post-filtering NN, or (iii) all layers of the post-filtering NN.

[0030] For example, the number of layers in a post-filtering neural network depends on the step size or number of steps corresponding to different blocks within one or more blocks.

[0031] Aspects of the present disclosure also provide a non-temporary computer-readable storage medium for storing a program executable by at least one processor to perform a method for video decoding.

[0032] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the detailed description and accompanying drawings below. [Brief explanation of the drawing]

[0033] [Figure 1A] This is a schematic diagram of an exemplary subset of intra-predictive modes. [Figure 1B] This is an example diagram of the intra-prediction direction. [Figure 2] This figure shows the current block (201) and surrounding sample according to one embodiment. [Figure 3] This is a schematic diagram of a simplified block diagram of a communication system (300) according to one embodiment. [Figure 4] This is a schematic diagram of a simplified block diagram of a communication system (400) according to one embodiment. [Figure 5] This is a schematic diagram of a simplified block diagram of a decoder according to one embodiment. [Figure 6] This is a schematic diagram of a simplified block diagram of an encoder according to one embodiment. [Figure 7] This is a block diagram of an encoder according to another embodiment. [Figure 8] This is a block diagram of a decoder according to another embodiment. [Figure 9A] This figure shows an example of block-based image coding according to one embodiment of the present disclosure. [Figure 9B] This figure shows an exemplary NIC framework according to one embodiment of the present disclosure. [Figure 10] This figure shows an exemplary convolutional neural network (CNN) of the main encoder network according to one embodiment of the present disclosure. [Figure 11] This figure shows an exemplary CNN of the main decoder network according to one embodiment of the present disclosure. [Figure 12] This figure shows an exemplary CNN of a hyperencoder according to one embodiment of the present disclosure. [Figure 13]This figure shows an exemplary CNN of a hyperdecoder according to one embodiment of the present disclosure. [Figure 14] This figure shows an exemplary CNN of a context model network according to one embodiment of the present disclosure. [Figure 15] This figure shows an exemplary CNN of an entropy parameter network according to one embodiment of the present disclosure. [Figure 16A] This figure shows an exemplary video encoder according to one embodiment of the present disclosure. [Figure 16B] This figure shows an exemplary video decoder according to one embodiment of the present disclosure. [Figure 17] This figure shows an exemplary video encoder according to one embodiment of the present disclosure. [Figure 18] This figure shows an exemplary video decoder according to one embodiment of the present disclosure. [Figure 19A] This figure shows an example of block-based image filtering according to one embodiment of the present disclosure. [Figure 19B] This figure shows an exemplary post-filtering module and NIC framework according to one embodiment of the present disclosure. [Figure 20A] This figure shows an exemplary video decoder according to one embodiment of the present disclosure. [Figure 20B] This figure shows an exemplary video decoder according to one embodiment of the present disclosure. [Figure 21A] This figure shows an exemplary deblocking process according to one embodiment of the present disclosure. [Figure 21B] This figure shows an exemplary deblocking process according to one embodiment of the present disclosure. [Figure 21C] This figure shows an exemplary deblocking process according to one embodiment of the present disclosure. [Figure 22] This figure shows an example of a boundary region containing three or more block samples according to an embodiment of the present disclosure. [Figure 23]This figure shows an exemplary deblocking process based on multiple deblocking models according to one embodiment of the present disclosure. [Figure 24] This figure shows an exemplary extension process according to one embodiment of the present disclosure. [Figure 25] This figure shows an exemplary extension process according to one embodiment of the present disclosure. [Figure 26] This figure shows an exemplary image level augmentation process according to one embodiment of the present disclosure. [Figure 27] This is a flowchart outlining the encoding process according to one embodiment of the present disclosure. [Figure 28] This is a flowchart outlining the decryption process according to one embodiment of the present disclosure. [Figure 29] This is a schematic diagram of a computer system according to one embodiment. [Modes for carrying out the invention]

[0034] Figure 3 shows a simplified block diagram of a communication system (300) according to one embodiment of the present disclosure. The communication system (300) includes, for example, a plurality of terminal devices that can communicate with each other over a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected over the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) perform unidirectional transmission of data. For example, terminal device (310) can encode video data (e.g., a stream of video pictures captured by terminal device (310)) for transmission to another terminal device (320) over the network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. Terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to restore the video pictures, and display the video pictures according to the restored video data. Unidirectional data transmission can be common in media serving applications, for example.

[0035] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of coded video data, which may occur, for example, during a video conference. In the case of bidirectional transmission of data, in one example, each terminal device of terminal devices (330) and (340) can code video data (e.g., a stream of video pictures captured by a terminal device) for transmission to the other terminal device of terminal devices (330) and (340) via the network (350). Each terminal device of terminal devices (330) and (340) can also receive coded video data transmitted by the other terminal device of terminal devices (330) and (340), decode the coded video data to restore the video pictures, and display the video pictures on an accessible display device according to the restored video data.

[0036] In the example in Figure 3, terminal devices (310), (320), (330), and (340) may be represented as a server, a personal computer, and a smartphone, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure find applications using laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (350) represents any number of networks that transmit coded video data between terminal devices (310), (320), (330), and (340), including, for example, wired and / or wireless communication networks. The communication network (350) can replace data with circuit-switched channels and / or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this description, the architecture and topology of network (350) may not be important to the operation of this disclosure unless described below herein.

[0037] Figure 4 shows an example of the arrangement of a video encoder and video decoder in a streaming environment as an application for the disclosed subject matter. The disclosed subject matter may be equally applicable to other video-enabled applications, such as video conferencing, digital television, and the storage of compressed video on digital media including CDs, DVDs, and memory sticks.

[0038] The streaming system may include, for example, a capture subsystem (413) which may include a video source (401), such as a digital camera, that creates a stream (402) of uncompressed video pictures. In one example, the stream (402) of video pictures includes a sample captured by the digital camera. The stream (402) of video pictures, depicted as a thick line to emphasize the large amount of data compared to encoded video data (404) (or encoded video bitstream), can be processed by an electronic device (420) which includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject, as will be described in more detail below. The encoded video data (404) (or encoded video bitstream), depicted as a thin line to emphasize the small amount of data compared to the stream (402), can be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) in Figure 4, can access a streaming server (405) to retrieve copies (407) and (409) of encoded video data (404). Client subsystem (406) may include, for example, a video decoder (410) within an electronic device (430). The video decoder (410) decodes the input copy (407) of the encoded video data (404) to create an output stream (411) of a video picture that can be rendered on a display (412) (e.g., a display screen) or other rendering device (without rendering). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstreams) can be encoded according to specific video coding / compression standards. Examples of such standards include ITU-T Recommendation H.265.For example, a video coding standard under development is informally known as Multipurpose Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0039] It should be noted that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

[0040] Figure 5 shows a block diagram of a video decoder (510) according to one embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) can be used in place of the video decoder (410) in the example of Figure 4.

[0041] The receiver (531) can receive one or more coded video sequences to be decoded by the video decoder (510), in the same or different embodiments, one coded video sequence at a time, and the decoding of each coded video sequence is independent of other coded video sequences. Coded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device that stores coded video data. The receiver (531) can receive coded video data together with other data that may be transferred to their respective usage entities (not described), e.g., coded audio data and / or auxiliary data streams. The receiver (531) can isolate coded video sequences from other data. To counteract network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter, "Parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). In other applications, it may be located outside the video decoder (510) (not described). In yet other applications, for example, to counteract network jitter, there may be a buffer memory outside the video decoder (510) (not described), and in addition, for example, to handle playout timing, there may be another buffer memory (515) inside the video decoder (510). When the receiver (531) is receiving data from a store / forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may not be needed or may be small. For use in best-effort packet networks such as the Internet, the buffer memory (515) may be needed, may be relatively large, may be advantageously adaptively sized, and may be at least partially implemented in the operating system or a similar element outside the video decoder (510) (not described).

[0042] The video decoder (510) may include a parser (520) to recover symbols (521) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (510), and potentially information for controlling rendering devices, such as a rendering device (512) (e.g., a display screen) that is not an integral part of the electronic device (530) but can be coupled to the electronic device (530), as shown in Figure 5. The control information for rendering devices may be in the form of supplemental extension information (SEI messages) or parameter set fragments (not depicted) of video usability information (VUI). The parser (520) can parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding, etc., with or without context sensitivity. The parser (520) can extract a set of subgroup parameters from the coded video sequence for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to a group. Subgroups can include picture groups (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), predictive units (PU), and the like. The parser (520) can also extract information from the coded video sequence such as transform coefficients, quantizer parameter values, and motion vectors.

[0043] The parser (520) can perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (515) in order to create a symbol (521).

[0044] The reconstruction of symbol (521) may involve multiple different units, depending on the type of coded video picture or part thereof (such as between and within pictures, between and within blocks), as well as other factors. How each unit is involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following multiple units is not depicted for clarity.

[0045] In addition to the functional blocks already described, the video decoder (510) can be conceptually subdivided into several functional units, as described below. In actual implementations operating under commercial constraints, many of these units can interact closely with each other and integrate with each other at least partially. However, the following conceptual subdivision into functional units is appropriate for describing the disclosed subject matter.

[0046] The first unit is the scaler / inverse unit (551). The scaler / inverse unit (551) receives control information from the parser (520) as symbols (521), including the quantization conversion coefficients, which conversion to use, block size, quantization coefficients, and quantization scaling matrix. The scaler / inverse unit (551) can output a block containing sample values ​​that can be input to the aggregator (555).

[0047] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use predictive information from previously restored pictures but can use predictive information from previously restored portions of the current picture. Such predictive information can be provided by the intra-picture predictive unit (552). In some cases, the intra-picture predictive unit (552) generates a block of the same size and shape as the block being restored, using the surrounding already restored information fetched from the current picture buffer (558). The current picture buffer (558) buffers, for example, partially restored current pictures and / or fully restored current pictures. The aggregator (555) may, in some cases, add the predictive information generated by the intra-predictive unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a sample-by-sample basis.

[0048] In other cases, the output samples of the scaler / inverse unit (551) may be associated with an intercoded and potentially motion-compensated block. In such cases, the motion-compensated prediction unit (553) can access the reference picture memory (557) to fetch samples to be used for prediction. After motion-compensating the fetched samples according to the symbols (521) associated with the block, these samples can be added to the output of the scaler / inverse unit (551) by the aggregator (555) to generate output sample information (in this case, called residual samples or residual signals). The address in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches the prediction samples can be controlled by motion vectors available to the motion-compensated prediction unit (553) in the form of symbols (521), which may have X, Y, and reference picture components, for example. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (557) when the exact motion vectors of the subsamples are used, motion vector prediction mechanisms, etc.

[0049] The output samples of the aggregator (555) can undergo various loop filtering techniques in the loop filter unit (556). The video compression technique may include in-loop filtering techniques that are controlled by parameters contained in the coded video sequence (also called coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but can also respond to previously restored and loop-filtered sample values, as well as metadata obtained during decoding of earlier parts (in decoding order) of the coded picture or coded video sequence.

[0050] The output of the loop filter unit (556) can be a sample stream that can be output to the rendering device (512) as well as stored in reference picture memory (557) for use in future interpicture prediction.

[0051] A particular coded picture, once fully restored, can be used as a reference picture for future predictions. For example, once the coded picture corresponding to the current picture is fully restored and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and any unused current picture buffer can be reallocated before starting the restoration of the next coded picture.

[0052] The video decoder (510) can perform decoding operations according to a specified video compression technique in a standard such as ITU-T Rec.H.265. The coded video sequence can conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence adheres to both the syntax of the video compression technique or standard and the documented profile within the video compression technique. Specifically, a profile can select several tools from all the tools available in the video compression technique or standard as the only tools available for use under that profile. Furthermore, compliance may require that the complexity of the coded video sequence be within the range defined by the level of the video compression technique or standard. In some cases, the level may limit the maximum picture size, maximum frame rate, maximum recovery sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limitations set by the level may, in some cases, be further restricted by the virtual reference decoder (HRD) specification and the metadata for HRD buffer management signaled within the coded video sequence.

[0053] In one embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately restore the original video data. The additional data may take the form of, for example, extension layers of time, space, or signal-to-noise ratio (SNR), redundant slices, redundant pictures, forward error correction codes, etc.

[0054] Figure 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of Figure 4.

[0055] The video encoder (603) can capture video images that are encoded by the video encoder (603) and can receive video samples from a video source (601) which is not part of the electronic device (620) in the example in Figure 6. In another example, the video source (601) is part of the electronic device (620).

[0056] The video source (601) can provide a source video sequence encoded by a video encoder (603) in the form of a digital video sample stream, which can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a series of separate pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each pixel may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description will focus on samples.

[0057] According to one embodiment, the video encoder (603) can encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Enforcing an appropriate coding speed is one function of the controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units described below. For simplicity, the coupling is not depicted. Parameters set by the controller (650) may include rate control-related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique, ...), picture size, picture group (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions related to the video encoder (603) optimized for a particular system design.

[0058] In some embodiments, the video encoder (603) is configured to operate in a coding loop. For the most simplified explanation, in one example, the coding loop may include a source coder (630) (which is involved in creating symbols, such as a symbol stream, based on, for example, an input picture to be coded and a reference picture), as well as a (local) decoder (633) built into the video encoder (603). The decoder (633) deconstructs the symbols to create sample data in a similar manner to how a (remote) decoder also creates them (since any compression between the symbols and the coded video bitstream is reversible in the video compression techniques considered in the disclosed subject). The deconstructed sample stream (sample data) is fed into the reference picture memory (634). Since decoding the symbol stream leads to bit-accurate results regardless of the decoder's location (local or remote), the contents in the reference picture memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the predictive part of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder "sees" when using the predictions during decoding. This fundamental principle of reference picture synchronization (and the resulting drift when synchronization cannot be maintained, for example, due to channel errors) is also used in several related techniques.

[0059] The operation of the “local” decoder (633) may be the same as that of a “remote” decoder, such as the video decoder (510), which has already been described in detail above with reference to Figure 5. However, referring again briefly to Figure 5, since symbols are available and the encoding / decoding of symbols to the coded video sequence by the entropy coder (645) and parser (520) may be reversible, the entropy decoding portion of the video decoder (510), including the buffer memory (515), and the parser (520) may not be fully implemented in the local decoder (633).

[0060] In one embodiment, the decoder techniques, excluding parsing / entropy decoding present within the decoder, exist in the corresponding encoder in the same or substantially identical functional form. Therefore, the subject matter disclosed focuses on the operation of the decoder. A description of the encoder techniques can be omitted, as it is the inverse of the comprehensively described decoder techniques. More detailed descriptions are provided below in specific areas.

[0061] During operation, in some examples, the source coder (630) can perform motion-compensated predictive coding, predictively coding the input picture by referencing one or more previously coded pictures from a video sequence designated as “reference pictures”. In this way, the coding engine (632) codes the difference between the pixel blocks of the input picture and the pixel blocks of the reference picture that may be selected as a predictive reference to the input picture.

[0062] The local video decoder (633) can decode the coded video data of a picture that may be designated as a reference picture based on symbols created by the source coder (630). The operation of the coding engine (632) may, advantageously, be a lossy process. When coded video data can be decoded by a video decoder (not shown in Figure 6), the restored video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (633) can replicate the decoding process that may be performed by the video decoder on the reference picture so that the restored reference picture is stored in the reference picture memory (634). In this way, the video encoder (603) can locally store a copy of the restored reference picture that has common content as the restored reference picture acquired by the far-end video decoder (without transmission errors).

[0063] The predictor (635) can perform predictive searches for the coding engine (632). That is, for a new picture to be coded, the predictor (635) can search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or reference picture motion vectors, block shapes, and other specific metadata that can serve as appropriate predictive references for the new picture. The predictor (635) can operate on sample blocks pixel by pixel to find appropriate predictive references. In some cases, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory (634), as determined by the search results obtained by the predictor (635).

[0064] The controller (650) can manage the coding operations of the source coder (630), including, for example, setting parameters and subgroup parameters used to encode video data.

[0065] The outputs of all the aforementioned functional units can undergo entropy coding within the entropy coder (645). The entropy coder (645) converts the symbols generated by the various functional units into coded video sequences by losslessly compressing the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0066] The transmitter (640) can buffer the coded video sequence created by the entropy coder (645) and prepare it for transmission over the communication channel (660), which may be a hardware / software link to a storage device that stores coded video data. The transmitter (640) can merge the coded video data from the video encoder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0067] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a specific coded picture type to each coded picture, which may affect the coding technique that can be applied to each picture. For example, a picture may often be assigned as one of the following picture types:

[0068] An intra-picture (I-picture) can be a picture that can be coded and decoded without using any other pictures in the sequence as a source of prediction. Some video codecs enable various types of intra-pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of their variations of I-pictures, as well as their respective uses and characteristics.

[0069] A predictive picture (P-picture) can be a picture that can be coded and decoded using intra-prediction or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0070] A bidirectional predictive picture (B-picture) can be a picture that can be coded and decoded using intra-prediction or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use three or more reference pictures and associated metadata to reconstruct a single block.

[0071] A source picture is typically spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block may be coded. Blocks may be predictively coded by referencing other (already coded) blocks, as determined by the coding assignment applied to each picture in the block. For example, blocks in picture I may be coded non-predictively, or they may be coded predictively by referencing already coded blocks in the same picture (spatial prediction or intra-prediction). Pixel blocks in picture P may be coded predictively by referencing one previously coded reference picture via spatial prediction or temporal prediction. Blocks in picture B may be coded predictively by referencing one or two previously coded reference pictures via spatial prediction or temporal prediction.

[0072] The video encoder (603) can perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec.H.265. In this operation, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data can conform to the syntax specified by the video coding technique or standard being used.

[0073] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the encoded video sequence. The additional data may include time / space / SNR extension layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and the like.

[0074] Video may be captured as multiple source pictures (video pictures) in a time series. Intra-picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given picture, while inter-picture prediction utilizes (time or other) correlations between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. When a block in the current picture is analogous to a reference block in a reference picture that has been previously coded and is still buffered within the video, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension to identify the reference picture if multiple reference pictures are used.

[0075] In some embodiments, a dual prediction technique can be used in interpicture prediction. According to the dual prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture, both of which are earlier in the decoding order than the current picture in the video (but may be past and future in the display order, respectively). A block in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture, and a second motion vector pointing to a second reference block in the second reference picture. A block can be predicted by a combination of the first and second reference blocks.

[0076] Furthermore, to improve coding efficiency, merge mode techniques can be used in interpicture prediction.

[0077] According to some embodiments of this disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU contains three coding tree blocks (CTBs), which are one luminance CTB and two saturation CTBs. Each CTU can be recursively quadtree-divided into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, or four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of the CU, such as inter-prediction type or intra-prediction type. The CU is divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Generally, each PU includes one luminance prediction block (PB) and two saturation PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. Using a luminance prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luminance values) such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0078] Figure 7 shows a diagram of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values ​​in the current video picture within a sequence of video pictures and to encode the processing block into a coded picture which is part of a coded video sequence. In one example, the video encoder (703) is used instead of the video encoder (403) in the example of Figure 4.

[0079] In the HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as an 8x8 sample prediction block. The video encoder (703) determines whether the processing block is best coded using intra-mode, inter-mode, or bi-prediction mode, for example, using rate-distortion optimization. When the processing block is coded in intra-mode, the video encoder (703) can encode the processing block into a coded picture using the intra-prediction technique; when the processing block is coded in inter-mode or bi-prediction mode, the video encoder (703) can encode the processing block into a coded picture using the inter-prediction technique or the bi-prediction technique, respectively. In certain video coding techniques, the merge mode may be an inter-picture prediction submode where the motion vectors are derived from one or more motion vector predictors, without the advantage of coded motion vector components outside the predictors. In certain other video coding techniques, there may be motion vector components applicable to the target block. In one example, the video encoder (703) includes other components such as a mode determination module (not shown) to determine the mode of the processing block.

[0080] In the example shown in Figure 7, the video encoder (703) includes an interencoder (730), an intraencoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general-purpose controller (721), and an entropy encoder (725), all coupled together as shown in Figure 7.

[0081] The interencoder (730) is configured to receive a sample of the current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in the previous and subsequent pictures), generate interprediction information (e.g., a description of redundant information, motion vectors, and merge mode information by the intercoding technique), and compute an interprediction result (e.g., a predicted block) based on the interprediction information using any appropriate technique. In some examples, the reference picture is a decoded reference picture decoded based on encoded video information.

[0082] The intra encoder (722) is configured to receive a sample of the current block (e.g., a processing block), and optionally compare the block to an already coded block in the same picture, generate quantization coefficients after the transformation, and optionally also generate intra prediction information (e.g., intra prediction direction information by one or more intra coding techniques). In one example, the intra encoder (722) also computes an intra prediction result (e.g., a prediction block) based on the intra prediction information and reference block in the same picture.

[0083] The general-purpose controller (721) is configured to determine general-purpose control data and to control other components of the video encoder (703) based on the general-purpose control data. For example, the general-purpose controller (721) determines the mode of a block and provides control signals to the switch (726) based on the mode. For example, when the mode is intra-mode, the general-purpose controller (721) controls the switch (726) to select intra-mode results for use by the residual calculator (723), controls the entropy encoder (725) to select intra-prediction information, and includes the intra-prediction information in the bitstream. When the mode is inter-mode, the general-purpose controller (721) controls the switch (726) to select inter-prediction results for use by the residual calculator (723), controls the entropy encoder (725) to select inter-prediction information, and includes the inter-prediction information in the bitstream.

[0084] The residual calculator (723) is configured to calculate the difference (residual data) between the receiving block and the prediction result selected from the intra-encoder (722) or inter-encoder (730). The residual encoder (724) is configured to operate on the residual data to encode the residual data and generate conversion coefficients. In one example, the residual encoder (724) is configured to convert the residual data from the spatial domain to the frequency domain and generate conversion coefficients. The conversion coefficients are then quantized to obtain quantized conversion coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be appropriately used by the intra-encoder (722) and inter-encoder (730). For example, an interencoder (730) can generate a decoded block based on decoded residual data and interprediction information, and an intraencoder (722) can generate a decoded block based on decoded residual data and intraprediction information. The decoded block is appropriately processed to generate a decoded picture, which is buffered in a memory circuit (not shown) and can be used as a reference picture in some examples.

[0085] The entropy encoder (725) is configured to format the bitstream to include the encoded blocks. The entropy encoder (725) is configured to include various information according to an appropriate standard such as the HEVC standard. For example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information within the bitstream. Note that, according to the disclosed subject, residual information is not present when coding blocks in either inter-mode or bi-prediction mode merge submodes.

[0086] Figure 8 shows a diagram of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive an encoded picture which is part of an encoded video sequence, and to decode the encoded picture to produce a restored picture. In one example, the video decoder (810) is used instead of the video decoder (410) in the example of Figure 4.

[0087] In the example shown in Figure 8, the video decoder (810) includes an entropy decoder (871), an interdecoder (880), a residual decoder (873), a restoration module (874), and an intradecoder (872) coupled together as shown in Figure 8.

[0088] The entropy decoder (871) can be configured to recover specific symbols from the coded picture that represent the syntactic elements that make up the coded picture. Such symbols can identify the mode in which the block is coded (e.g., intra-mode, inter-mode, bi-prediction mode, merge sub-mode, or the latter two of another sub-mode), each of which can identify specific samples or metadata used for prediction by the intra decoder (872) or inter-decoder (880), and may include prediction information (e.g., intra-prediction information or inter-prediction information), such as residual information in the form of quantization transformation coefficients. For example, when the prediction mode is inter-mode or bi-prediction mode, inter-prediction information is provided to the inter-decoder (880), and when the prediction type is intra-prediction type, intra-prediction information is provided to the intra-decoder (872). The residual information can undergo inverse quantization and be provided to the residual decoder (873).

[0089] The interdecoder (880) is configured to receive interprediction information and generate interprediction results based on the interprediction information.

[0090] The intra decoder (872) is configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0091] The residual decoder (873) is configured to perform inverse quantization to extract inverse quantization conversion coefficients, process these coefficients, and convert the residuals from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to include quantizer parameters (QP)), which may be provided by the entropy decoder (871) (the data path is not depicted as this may only contain a small amount of control information).

[0092] The restoration module (874) is configured to combine the residual output by the residual decoder (873) and the prediction results (possibly output by the inter-prediction module or intra-prediction module) in the spatial domain to form a restored block that may be part of the restored picture, which may be part of the restored video. Note that other appropriate operations, such as deblocking operations, may be performed to improve the appearance.

[0093] It should be noted that the video encoders (403), (603), and (703), as well as the video decoders (410), (510), and (810), can be implemented using any suitable technique. In one embodiment, the video encoders (403), (603), and (703), as well as the video decoders (410), (510), and (810), can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603), as well as the video decoders (410), (510), and (810), can be implemented using one or more processors that execute software instructions.

[0094] This disclosure describes video coding techniques related to neural image compression techniques and / or neural video compression techniques, such as artificial intelligence (AI)-based neural image compression (NIC). Aspects of this disclosure include content-adaptive online training in NICs, such as a content-adaptive online training NIC method with post-filtering for an end-to-end (E2E) optimized image coding framework based on a neural network. The neural network (NN) may include artificial neural networks (ANNs), such as deep neural networks (DNNs) and convolutional neural networks (CNNs).

[0095] In one embodiment, the associated hybrid video codec is difficult to optimize as a whole. For example, an improvement in a single module (e.g., the encoder) in a hybrid video codec may not result in a coding gain in overall performance. In a NN-based video coding framework, different modules can be optimized together from input to output to improve the end goal (e.g., rate distortion performance such as the rate distortion loss L described herein) by performing a learning or training process (e.g., a machine learning process), thus resulting in an E2E optimized NIC.

[0096] An exemplary NIC framework or system can be described as follows: The NIC framework uses input block x as input to a neural network encoder (e.g., a neural network-based encoder such as a DNN) to create a compressed representation (e.g., a compact representation) that can be made compact for purposes such as storage and transmission.

number

number

number

number

number

number

[0097] In some examples, the NIC framework can use a variational autoencoder (VAE) structure. In a VAE structure, the neural network encoder can directly use the entire input block x as input to the neural network encoder. The entire input block x acts as a black box and is compressed into a representation.

number

number

number

number

number

number

number

number

number

number

[0098] A neural network (e.g., an ANN) can learn to perform tasks from examples without task-specific programming. An ANN can consist of connected nodes or artificial neurons. The connections between nodes allow signals to be transmitted from a first node to a second node (e.g., a receiving node), and the signals can be modified by weights, which can be represented by weight coefficients for the connections. The receiving node can process signals from nodes that transmit signals to it (i.e., input signals for the receiving node) and then generate an output signal by applying a function to the input signals. The function can be a linear function. In one example, the output signal is a weighted sum of the input signals. In another example, the output signal is further modified by a bias, which can be represented by a bias term, and therefore the output signal is the sum of the bias and the weighted sum of the input signals. The function can include, for example, a weighted sum of the input signals or a nonlinear operation on the bias and the sum of the weighted sums. The output signal can be transmitted to nodes connected to the receiving node (downstream nodes). An ANN can be represented or constructed by parameters (e.g., connection weights and / or biases). Weights and / or biases can be obtained by training an ANN with examples, and these weights and / or biases can be adjusted iteratively. A trained ANN, composed of the determined weights and / or biases, can then be used to perform a task.

[0099] Nodes within an ANN can be organized in any suitable architecture. In various embodiments, nodes within an ANN are organized in layers that include an input layer that receives input signals to the ANN and an output layer that outputs output signals from the ANN. In one embodiment, the ANN further includes layers such as a hidden layer between the input and output layers. Different layers can perform different types of transformations on the respective inputs of different layers. Signals can move from the input layer to the output layer.

[0100] An ANN having multiple layers between the input and output layers can be called a DNN. In one embodiment, a DNN is a feedforward network where data flows from the input layer to the output layer without loopback. In one example, a DNN is a fully connected network where each node in one layer is connected to all nodes in the next layer. In one embodiment, a DNN is a recurrent neural network (RNN) where data can flow in any direction. In one embodiment, a DNN is a CNN.

[0101] A CNN can include an input layer, an output layer, and a hidden layer between the input and output layers. The hidden layer can include a convolutional layer that performs convolutions, such as a two-dimensional (2D) convolution (used, for example, in an encoder). In one embodiment, the 2D convolution performed in the convolutional layer is between a convolutional kernel (also called a filter or channel, such as a 5x5 matrix) and an input signal to the convolutional layer (e.g., a 2D block, a 256x256 matrix, or a 2D matrix). In various examples, the dimension of the convolutional kernel (e.g., 5x5) is smaller than the dimension of the input signal (e.g., 256x256). Therefore, a portion of the input signal (e.g., a 256x256 matrix) covered by the convolutional kernel (e.g., a 5x5 region) is smaller than the region of the input signal (e.g., a 256x256 region) and can therefore be called the receptive field at each node in the next layer.

[0102] During convolution, the inner product of the convolution kernel and the corresponding receptive fields in the input signal is calculated. Therefore, each element of the convolution kernel is a weight applied to the corresponding sample in the receptive field, and thus the convolution kernel contains weights. For example, a convolution kernel represented by a 5x5 matrix has 25 weights. In some examples, a bias is applied to the output signal of the convolutional layer, and the output signal is based on the sum of the inner product and the bias.

[0103] A convolutional kernel can be shifted along an input signal (e.g., a 2D matrix) by a size called the stride. Thus, the convolutional operation generates a feature map or activation map (e.g., another 2D matrix), which then contributes to the input of the next layer in the CNN. For example, if the input signal is a 2D block with 256 × 256 samples and the stride is 2 samples (e.g., a stride of 2), then the convolutional kernel shifts by 2 samples along the X direction (e.g., horizontal) and / or the Y direction (e.g., vertical).

[0104] Multiple convolutional kernels can be applied to an input signal within the same convolutional layer to generate multiple feature maps, each representing a specific feature of the input signal. Generally, a convolutional layer with N channels (i.e., N convolutional kernels) where each convolutional kernel has M×M samples, and a stride S, can be specified as Conv:M×M cN sS. For example, a convolutional layer with 192 channels where each convolutional kernel has 5×5 samples, and a stride of 2, is specified as Conv:5×5 c192 s2. Hidden layers can include deconvolutional layers that perform deconvolution, such as 2D deconvolution (used, for example, within a decoder). Deconvolution is the opposite of convolution. A deconvolutional layer with 192 channels where each deconvolutional kernel has 5×5 samples, and a stride of 2, is specified as DeConv:5×5 c192 s2.

[0105] In various embodiments, CNNs offer the following advantages: The number of learnable parameters (i.e., parameters to be trained) in a CNN can be significantly smaller than the number of learnable parameters in a DNN, such as a feedforward DNN. In a CNN, a relatively large number of nodes can share the same filter (e.g., the same weights) and the same bias (if bias is used), and therefore a single bias and a single weight vector can be used across all receptive fields sharing the same filter, thus reducing the memory footprint. For example, for an input signal with 100 × 100 samples, a convolutional layer with a convolutional kernel having 5 × 5 samples has 25 learnable parameters (e.g., weights). If bias is used, one channel uses 26 learnable parameters (e.g., 25 weights and 1 bias). If the convolutional layer has N channels, the total number of learnable parameters is 26 × N. In contrast, for a fully connected layer in a DNN, 100 × 100 (i.e., 10000) weights are used for each node in the next layer. If the next layer has L nodes, the total number of learnable parameters is 10,000 × L.

[0106] A CNN can further include one or more other layers, such as pooling layers, fully connected layers that can connect all nodes in one layer to all nodes in another layer, and normalization layers. The layers within a CNN can be arranged in any appropriate order and in any appropriate architecture (e.g., feedforward architecture, recurrent architecture). In one example, the convolutional layer is followed by other layers such as pooling layers, fully connected layers, and normalization layers.

[0107] A pooling layer can be used to reduce the dimensionality of data by combining the outputs from multiple nodes in one layer into a single node in the next layer. The pooling operation for a pooling layer with a feature map as input is described below. The description can be appropriately adapted to other input signals. The feature map can be divided into sub-regions (e.g., rectangular sub-regions), and the features within each sub-region can be downsampled (or pooled) individually to a single value, for example, by taking the mean value in mean pooling or the maximum value in max pooling.

[0108] Pooling layers can perform various types of pooling, including local pooling, global pooling, max pooling, and average pooling. Pooling is a form of nonlinear downsampling. Local pooling combines a small number of nodes in a feature map (e.g., local clusters of nodes such as 2x2 nodes). Global pooling can combine all nodes in a feature map, for example.

[0109] Pooling layers can reduce the size of representations, and therefore reduce the number of parameters, memory footprint, and computational complexity in a CNN. In one example, a pooling layer is inserted between consecutive convolutional layers in a CNN. In another example, an activation function such as a rectified linear unit (ReLU) layer follows the pooling layer. In yet another example, a pooling layer is omitted between consecutive convolutional layers in a CNN.

[0110] The normalization layer can be ReLU, Leaky ReLU, Generalized Partition Normalization (GDN), or Inverse GDN (IGDN). ReLU can remove negative values ​​from an input signal, such as a feature map, by applying a non-saturated activation function to set negative values ​​to 0. Leaky ReLU can have a small gradient (e.g., 0.01) for negative values ​​instead of a flat gradient (e.g., 0). Therefore, if the value x is greater than 0, the output from Leaky ReLU is x. Otherwise, the output from Leaky ReLU is the value x multiplied by the small gradient (e.g., 0.01). In one example, the gradient is determined before training and therefore not learned during training.

[0111] In NN-based image compression methods, such as DNN-based or CNN-based image compression methods, instead of directly encoding the entire image, block-based or block-unit coding mechanisms can be effective for compressing images in DNN-based video coding standards such as FVC. The entire image can be divided into blocks of the same (or varying) size, and the blocks can be compressed individually. In one embodiment, the image may be divided into blocks having equal or unequal sizes. The divided blocks can be compressed instead of the image. Figure 9A shows an example of block-unit image coding according to one embodiment of the present disclosure. Image (980) can be divided into blocks, for example, blocks (981) to (996). Blocks (981) to (996) can be compressed, for example, in scan order. In the example shown in Figure 9A, blocks (981) to (989) have already been compressed, and blocks (990) to (996) are compressed.

[0112] Images can be treated as blocks. In one embodiment, images are compressed without being divided into blocks. The entire image can be the input to an E2E NIC framework.

[0113] In the following explanation, for the sake of brevity, signals are used to refer to images or blocks. Therefore, an input signal can refer to an input image or input block, an encoded signal can refer to an encoded image or encoded block, and a reconstructed signal can refer to a reconstructed image or reconstructed block.

[0114] Figure 9B shows an exemplary NIC framework (900) (e.g., a NIC system) according to one embodiment of the present disclosure. The NIC framework (900) may be based on a neural network such as a DNN and / or CNN. The NIC framework (900) can be used to compress (e.g., encode) a signal (e.g., blocks and / or images) and decompress (e.g., decode or restore) a compressed signal (e.g., blocks and / or images) (e.g., encoded blocks and / or images). The NIC framework (900) may include two subneural networks, a first subNN (951) and a second subNN (952), which are implemented using a neural network.

[0115] The first sub-NN(951) can be similar to an autoencoder, taking an input signal (e.g., an input block or input image) x and a compressed signal (e.g., a compressed block or compressed image).

number

number

number

[0116] The latent representation y is the quantized latent image.

number

number

number

number

[0117] The encoded signal (931) can be decoded (e.g., entropy decoding) by the entropy decoder (914) to produce an output. The entropy decoder (914) can use entropy coding techniques such as Huffman coding and arithmetic coding, which correspond to the entropy coding techniques used in the entropy encoder (913). In one example, the entropy decoder (914) uses arithmetic decoding and is an arithmetic decoder. In one example, lossless compression is used in the entropy encoder (913) and lossless decoded in the entropy decoder (914), noise due to the transmission of the encoded signal (931) can be omitted, and the output from the entropy decoder (914) is a quantized latent image.

number

[0118] The main decoder network (915) is quantized latent image

number

number

number

number

number

number

number

[0119] The second sub-NN(952) is the quantized latent image used for entropy coding.

number

number

[0120] Referring to Figure 9B, on the encoder side, the quantized latent image from the quantizer (912)

number

number

number

number

Number

Number

Number

[0121] The output o from the context model NN(916) cm,i and the output o from the hyper-decoder (925) hc are supplied to the entropy parameter NN(917) to generate the output o ep . The entropy parameter NN(917) can be implemented using a neural network such as a CNN. The output o of the entropy parameter NN(917) ep and the input (e.g., o cm,i and o hc ) can be described using Equation 5. o ep =f4(o cm,i ,o hc ;θ4) Equation 5 Here, the parameter θ4 represents parameters such as the weights used in the convolutional kernels within the entropy parameter NN(917) and the bias (if a bias is used within the entropy parameter NN(917)). The output o of the entropy parameter NN(917) epThis can be used when determining (e.g., conditioning) the entropy model, and therefore the conditional entropy model is, for example, the output from the hyperdecoder (925) o hc It can depend on the input signal x via . For example, output o ep This includes parameters such as mean parameters and scale parameters used to condition the entropy model (e.g., GMM). Referring to Figure 9B, the entropy model (e.g., conditional entropy model) can be used by the entropy encoder (913) and entropy decoder (914) in entropy coding and entropy decoding, respectively.

[0122] The second sub-NN(952) can be described later. The latent image y can be supplied to the hyperencoder(921) to generate the hyperlatent image z. In one example, the hyperencoder(921) is implemented using a neural network such as a CNN. The relationship between the hyperlatent image z and the latent image y can be described using equation 6. z = f5(y;θ5) Equation 6 Here, parameter θ5 represents parameters such as the weights used in the convolution kernel within the hyperencoder(921) and the bias (if a bias is used within the hyperencoder(921)).

[0123] The hyperlatent image z is quantized by the quantizer (922) to form the quantized latent image.

number

number

[0124] Side information such as encoded bits (932) can be decompressed (e.g., entropy decoding) by the entropy decoder (924) to generate an output. The entropy decoder (924) can use entropy coding techniques such as Huffman coding and arithmetic coding. In one example, the entropy decoder (924) uses arithmetic decoding and is an arithmetic decoder. In one example, lossless compression is used in the entropy encoder (923) and lossless decompression is used in the entropy decoder (924), noise due to the transmission of side information can be omitted, and the output from the entropy decoder (924) is a quantized latent image.

number

number

number

number

[0125] As described above, compression bits or encoding bits (932) can be added to the encoded bitstream as side information, thereby enabling the entropy decoder (914) to use a conditional entropy model. Thus, the entropy model can be signal-dependent (e.g., block-dependent or image-dependent) and spatially adaptable, and therefore more accurate than a fixed entropy model.

[0126] The NIC framework (900) can be appropriately adapted to omit, modify, and / or include one or more components shown in Figure 9B, for example. For example, a NIC framework using a fixed-entropy model includes a first sub-NN (951) but does not include a second sub-NN (952). For example, the NIC framework includes components within the NIC framework (900) except for the entropy encoder (923) and entropy decoder (924).

[0127] In one embodiment, one or more components within the NIC framework (900) shown in Figure 9B are implemented using a neural network such as a CNN. Each NN-based component within the NIC framework (e.g., NIC framework (900)) (e.g., main encoder network (911), main decoder network (915), context model NN (916), entropy parameter NN (917), hyperencoder (921), or hyperdecoder (925)) may include any suitable architecture (e.g., having any suitable combination of layers), any suitable type of parameters (e.g., weights, biases, combinations of weights and biases, etc.), and may include any suitable number of parameters.

[0128] In one embodiment, the main encoder network (911), main decoder network (915), context model NN (916), entropy parameter NN (917), hyperencoder (921), and hyperdecoder (925) are implemented using their respective CNNs.

[0129] Figure 10 shows an exemplary CNN of a main encoder network (911) according to one embodiment of the present disclosure. For example, the main encoder network (911) includes four sets of layers, each set of layers including a 5×5 c192 s2 convolutional layer followed by a GDN layer. One or more layers shown in Figure 10 can be modified and / or omitted. Further layers can be added to the main encoder network (911).

[0130] Figure 11 shows an exemplary CNN of a main decoder network (915) according to one embodiment of the present disclosure. For example, the main decoder network (915) includes three sets of layers, each set of layers including an IGDN layer followed by a 5×5 c192 s2 deconvolutional layer. In addition, the three sets of layers are followed by a 5×5 c3 s2 deconvolutional layer, followed by an IGDN layer. One or more layers shown in Figure 11 can be modified and / or omitted. Further layers can be added to the main decoder network (915).

[0131] Figure 12 shows an exemplary CNN of a hyperencoder (921) according to one embodiment of the present disclosure. For example, the hyperencoder (921) includes a convolutional layer 3×3 c192 s1 followed by a leaked ReLU, a convolutional layer 5×5 c192 s2 followed by a leaked ReLU, and another convolutional layer 5×5 c192 s2. One or more layers shown in Figure 12 can be modified and / or omitted. Further layers can be added to the hyperencoder (921).

[0132] Figure 13 shows an exemplary CNN of a hyperdecoder (925) according to one embodiment of the present disclosure. For example, the hyperdecoder (925) includes a 5×5 c192 s2 inverse convolutional layer followed by a leaky ReLU, a 5×5 c288 s2 inverse convolutional layer followed by a leaky ReLU, and a 3×3 c384 s1 inverse convolutional layer. One or more layers shown in Figure 13 can be modified and / or omitted. Further layers can be added to the hyperdecoder (925).

[0133] Figure 14 shows an exemplary CNN of the context model NN(916) according to one embodiment of the present disclosure. For example, the context model NN(916) includes a masked convolution 5×5 c384 s1 for context prediction, and thus the context of Equation 4

number

[0134] Figure 15 shows an exemplary CNN of entropy-parameterized NN(917) according to one embodiment of the present disclosure. For example, entropy-parameterized NN(917) includes a convolutional layer 1×1 c640 s1 followed by a leaky ReLU, a convolutional layer 1×1 c512 s1 followed by a leaky ReLU, and a convolutional layer 1×1 c384 s1. One or more layers shown in Figure 15 can be modified and / or omitted. Further layers can be added to entropy-parameterized NN(917).

[0135] The NIC framework (900) can be implemented using a CNN, as described with reference to Figures 10–15. The NIC framework (900) can be appropriately adapted so that one or more components within the NIC framework (900) (e.g., (911), (915), (916), (917), (921), and / or (925)) are implemented using any appropriate type of neural network (e.g., a CNN or a non-CNN-based neural network). One or more other components of the NIC framework (900) can be implemented using a neural network.

[0136] The NIC framework (900), which includes neural networks (e.g., CNNs), can be trained to learn the parameters used in the neural network. For example, when a CNN is used, the parameters represented by θ1 to θ6, such as the weights used in the convolutional kernel within the main encoder network (911) and the bias (if bias is used within the main encoder network (911)), the weights used in the convolutional kernel within the main decoder network (915) and the bias (if bias is used within the main decoder network (915)), the weights used in the convolutional kernel within the hyperencoder (921) and the bias (if bias is used within the hyperencoder (921)), the weights used in the convolutional kernel within the hyperdecoder (925) and the bias (if bias is used within the hyperdecoder (925)), the weights used in the convolutional kernel within the context model NN (916) and the bias (if bias is used within the context model NN (916)), and the weights used in the convolutional kernel within the entropy parameter NN (917) and the bias (if bias is used within the entropy parameter NN (917)), can be learned during the training process.

[0137] In one example, referring to Figure 10, the main encoder network (911) includes four convolutional layers, each with a 5x5 convolutional kernel and 192 channels. Therefore, the number of weights used in the convolutional kernel within the main encoder network (911) is 19200 (i.e., 4x5x5x192). The parameters used in the main encoder network (911) include the 19200 weights and an optional bias. Additional parameters may be included when biases and / or further neural networks are used in the main encoder network (911).

[0138] Referring to Figure 9B, the NIC framework (900) includes at least one component or module built on top of a neural network. The at least one component may include one or more of the following: the main encoder network (911), the main decoder network (915), the hyperencoder (921), the hyperdecoder (925), the context model NN (916), and the entropy parameter NN (917). At least one component can be trained individually. In one example, the training process is used to learn the parameters of each component separately. At least one component can be trained together as a group. In one example, the training process is used to learn the parameters of a subset of at least one component together. In one example, the training process is used to learn all the parameters of at least one component, and is therefore called E2E optimization.

[0139] In the training process for one or more components within the NIC framework (900), the weights (or weight coefficients) of one or more components can be initialized. For example, the weights are initialized based on a pre-trained corresponding neural network model (e.g., a DNN model, a CNN model). For example, the weights are initialized by setting them to random numbers.

[0140] For example, after the weights have been initialized, a set of training blocks can be used to train one or more components. A set of training blocks can contain any suitable blocks of any suitable size. In some examples, a set of training blocks may include raw images, natural images, computer-generated images, etc., in the spatial domain. In some examples, a set of training blocks may include residual blocks in the spatial domain or blocks from residual images that have residual data. Residual data can be computed by a residual calculator (e.g., residual calculator (723)). In some examples, raw images and / or residual images containing residual data can be used directly to train a neural network in a NIC framework, such as NIC framework (900). Thus, raw images, residual images, blocks from raw images, and / or blocks from residual images can be used to train a neural network in a NIC framework.

[0141] For brevity, the following training process will be explained using a training block as an example. The explanation can be appropriately adapted to the training block. A training block t of a set of training blocks can be passed through the encoding process in Figure 9B to generate a compressed representation (e.g., encoded information into a bitstream). The encoded information is then passed through the decoding process shown in Figure 9B to the decompression block.

number

[0142] In the case of the NIC framework (900), two competing goals are balanced, e.g., restoration quality and bit consumption. Quality loss function (e.g., distortion or distortion loss)

number

number

[0143] In the case of neural image compression, differentiable approximations of quantization can be used in E2E optimization. In various examples, noise injection is used in the training process of neural network-based image compression to simulate quantization; therefore, quantization is simulated by noise injection instead of being performed by a quantizer (e.g., quantizer (912)). Thus, training with noise injection can variationally approximate the quantization error. A bit-per-pixel (BPP) estimator can be used to simulate the entropy coder; therefore, entropy coding is simulated by the BPP estimator instead of being performed by an entropy encoder (e.g., (913)) and an entropy decoder (e.g., (914)). Thus, the rate loss R in the loss function L shown in Equation 1 during the training process can be estimated based on noise injection and the BPP estimator, for example. In general, a higher rate R can allow for lower distortion D, and a lower rate R can result in higher distortion D. Therefore, the trade-off hyperparameter λ in Equation 1 can be used to optimize the junction RD loss L, and L can be optimized as the sum of λD and R. The training process can be used to tune the parameters of one or more components (e.g., (911), (915)) within the NIC framework (900) so that the junction RD loss L is minimized or optimized. In some examples, the trade-off hyperparameter λ is,

number

[0144] Various models can be used to determine the strain loss D and rate loss R, and therefore the junction RD loss L in Equation 1. In one example, the strain loss

number

[0145] In one example, the goal of the training process is to train an encoding neural network (e.g., encoding DNN) used on the encoder side, such as a video encoder, and a decoding neural network (e.g., decoding DNN) used on the decoder side, such as a video decoder. In one example, referring to Figure 9B, the encoding neural network may include a main encoder network (911), a hyperencoder (921), a hyperdecoder (925), a context model NN (916), and an entropy parameter NN (917). The decoding neural network may include a main decoder network (915), a hyperdecoder (925), a context model NN (916), and an entropy parameter NN (917). The video encoder and / or video decoder may include other components that are based on and / or not based on NNs.

[0146] NIC frameworks (e.g., NIC framework(900)) can be trained in an end-to-end (E2E) manner. For example, the coding neural network and the decoding neural network are updated together in an E2E training process based on backpropagation gradients.

[0147] After the parameters of the neural network within the NIC framework (900) have been trained, one or more components within the NIC framework (900) can be used to encode and / or decode blocks. In one embodiment, on the encoder side, the video encoder is configured to encode the input signal x into an encoded signal (931) transmitted in the bitstream. The video encoder can include multiple components within the NIC framework (900). In one embodiment, on the decoder side, the corresponding video decoder decodes the encoded signal (931) in the bitstream into a decoded signal

number

[0148] For example, when content-adaptive online training is employed, the video encoder includes all components within the NIC framework (900).

[0149] Figure 16A shows an exemplary video encoder (1600A) according to one embodiment of the present disclosure. The video encoder (1600A) includes, for example, a main encoder network (911), a quantizer (912), an entropy encoder (913), and a second sub-NN (952) as described with reference to Figure 9B. Figure 16B shows an exemplary video decoder (1600B) according to one embodiment of the present disclosure. The video decoder (1600B) may correspond to the video encoder (1600A). The video decoder (1600B) may include a main decoder network (915), an entropy decoder (914), a context model NN (916), an entropy parameter NN (917), an entropy decoder (924), and a hyperdecoder (925). Referring to Figures 16A and 16B, on the encoder side, the video encoder (1600A) can generate the encoded signal (931) and encoded bits (932) to be transmitted within the bitstream. On the decoder side, the video decoder (1600B) can receive and decode the encoded signal (931) and encoded bits (932).

[0150] Figures 17 and 18 show exemplary video encoders (1700) and corresponding video decoders (1800), respectively, according to embodiments of the present disclosure. Referring to Figure 17, the encoder (1700) includes a main encoder network (911), a quantizer (912), and an entropy encoder (913). Examples of the main encoder network (911), quantizer (912), and entropy encoder (913) are described with reference to Figure 9B. Referring to Figure 18, the video decoder (1800) includes a main decoder network (915) and an entropy decoder (914). Examples of the main decoder network (915) and entropy decoder (914) are described with reference to Figure 9B. Referring to Figures 17 and 18, the video encoder (1700) can generate an encoded signal (931) contained in a bitstream. The video decoder (1800) can receive and decode the encoded signal (931).

[0151] Post-filtering or post-processing can be applied to a restored signal, such as a restored block or restored image, to determine a post-filtered signal (e.g., a post-filtered image or post-filtered block). Post-filtering can be applied to reduce the distortion loss of the post-filtered signal; for example, the distortion loss of the post-filtered signal can be made smaller than the distortion loss of the corresponding restored signal.

[0152] Figure 19A shows an example of block-based image filtering according to one embodiment of the present disclosure. Instead of directly post-filtering the entire image (1980), a block-based filtering mechanism can be applied to post-filtering. The entire image (1980) is divided into blocks (1981) to (1996) of the same size (e.g., W × H samples) or of various sizes, and blocks (1981) to (1996) are post-filtered individually. In one embodiment, the post-filtering module is applied to blocks (1981) to (1996) instead of the image (1980).

[0153] In one embodiment, the image (e.g., a reconstructed image) is post-filtered without being divided into blocks, and therefore the entire image is the input to the post-filtering module.

[0154] Figure 19B shows an exemplary post-filtering module (1910) and NIC framework (e.g., NIC framework (900)) according to one embodiment of the present disclosure. The NIC framework (900) is described with reference to Figure 9B. Restored signal from the main decoder network (915)

number

Number

Number

Number

Number

Number

Number

Number

Number

Number

[0155] The post-filtering module (1910) can be implemented using any suitable filtering method that can reduce distortion loss and improve appearance. According to one embodiment of the present disclosure, the post-filtering module (1910) is NN-based and is implemented using an NN (e.g., a post-filtering NN) such as a post-filtering DNN or a post-filtering CNN.

[0156] A post-filtering neural network may include any suitable architecture (e.g., having any suitable combination of layers), any suitable type of parameters (e.g., weights, biases, combinations of weights and biases, etc.), and any suitable number of parameters, as described in this disclosure.

[0157] In one example, a post-filtering NN is implemented using a CNN that includes one or more convolutional layers. The post-filtering NN may include additional layers described in this disclosure, such as pooling layers, fully connected layers, and normalization layers. The layers within the post-filtering NN can be arranged in any suitable order and in any suitable architecture (e.g., a feedforward architecture, a recurrent architecture). In one example, a convolutional layer is followed by other layers such as a pooling layer, a fully connected layer, and a normalization layer. Each of the convolutional layers within the post-filtering NN may include any suitable number of channels as well as any suitable convolutional kernel and stride.

[0158] A post-filtering NN can be trained on a training signal (e.g., a training image and / or training block) similar to or identical to the training signal described above with reference to Figure 9B. In some examples, the compressed (e.g., encoded) and / or transmitted input signal (e.g., an input image or input block) has significantly different characteristics from the training signal. Therefore, post-filtering a restored signal corresponding to the input signal using a post-filtering NN trained on the training signal can result in a relatively low RD loss L (e.g., relatively large distortion). Accordingly, aspects of this disclosure describe a content-adaptive online training method for post-filtering. In a content-adaptive online training method for post-filtering, one or more parameters of the post-filtering NN can be determined based on the compressed (e.g., encoded) and / or transmitted one or more input signals. The input signals may include raw data or residual data.

[0159] In one embodiment, one or more input signals are one or more blocks in an input image. One or more blocks can be used to determine one or more parameters in a post-filtering neural network by optimizing rate distortion performance, and the content-adaptive online training method for post-filtering can be called a block-based content-adaptive online training method for post-filtering.

[0160] In one embodiment, one or more input signals are one or more input images. The one or more input images can be used to determine one or more parameters in the post-filtering neural network by optimizing rate-distortion performance, and the content-adaptive online training method for post-filtering can be called an image-based content-adaptive online training method for post-filtering.

[0161] Post-filtering information indicating one or more determined parameters can be encoded within the video bitstream along with one or more encoded signals (e.g., one or more encoded blocks or one or more encoded images). On the decoder side, the post-filtering neural network can be determined based on the one or more determined parameters, post-filter the recovered one or more signals, and achieve better compression performance using the determined parameters. Each of the recovered one or more signals can be restored by the video decoder based on one or more encoded signals.

[0162] According to aspects of this disclosure, content-adaptive online training can be applied in post-filtering, such as NN-based post-filtering. NN-based post-filtering (e.g., DNN-based post-filtering or CNN-based post-filtering) can be implemented in an NN-based image coding framework (e.g., NIC framework (900)) or other image compression methods.

[0163] The post-filtering module (1910) can be implemented using a NN (e.g., DNN or CNN) and the restored signal

number

number

number

number

number

number

number

Number

Number

Number

[0164] For simplicity, the content-adaptive online training of the post-filtering module (1910) can be called post-filtering training. The block-based content-adaptive online training method for post-filtering can be called block-based post-filtering training. The image-based content-adaptive online training method for post-filtering can be called image-based post-filtering training.

[0165] In one embodiment, the goal of post-filtering training is, as described in Equation 1, to reduce (e.g., minimize) the R-D loss L p where the restored signal

Number

Number

Number

number

number

[0166] According to one embodiment of the present disclosure, one or more parameters in a post-filtering NN can be determined in post-filtering training, for example, based on one or more input signals. Referring to Figure 19B, input signal x is supplied to the NIC framework (900), and the encoded signal in the encoding process

number

number

number

number

number

number

[0167] A post-filtering neural network (NN) can be configured, for example, with initial parameters (e.g., initial weights and / or initial biases) before post-filtering training. In one embodiment, the post-filtering NN is pre-trained on a training dataset containing training blocks and / or training images, and the initial parameters include pre-trained parameters (e.g., pre-trained weights and / or pre-trained biases). In one embodiment, the post-filtering NN is not pre-trained. The initial parameters may be random parameters.

[0168] In one embodiment, one or more of the initial parameters in the post-filtering NN are determined based on the restored signal x.

number

[0169] Post-filtering training can be used as a preprocessing step (e.g., a pre-encoding step) to improve the compression performance of any image compression method.

[0170] The determined one or more parameters are used in the encoded signal.

number

number

number

[0171] By using one or more determined parameters (e.g., one or more substitution parameters) within the post-filtering NN, the post-filtering module (1910) restores the signal.

number

[0172] The above description can be appropriately applied when multiple input signals are used to determine one or more parameters, and the determined one or more parameters are shared by the multiple input signals.

[0173] Referring to Figure 19B, on the encoder side, one or more parameters of the post-filtering NN in the post-filtering module (1910) are used to restore the post-filtering signal during post-filtering training.

number

number

number

number

[0174] Figure 20A shows an exemplary video decoder (2000A) according to one embodiment of the present disclosure. The video decoder (2000A) may include components of the video decoder (1600B) described with reference to Figure 16B and a post-filtering module (1910) described with reference to Figure 19B.

[0175] Figure 20B shows an exemplary video decoder (2000B) according to one embodiment of the present disclosure. The video decoder (2000B) may include the components of the video decoder (1800) described with reference to Figure 18 and the post-filtering module (1910) described with reference to Figure 19B.

[0176] On the decoder side, referring to Figures 19B, 20A, and 20B, the post-filtering information can be decoded by a video decoder (e.g., video decoder (2000A) or (2000B)). One or more parameters of the post-filtering NN in the post-filtering module (1910) can be obtained based on the post-filtering information. In one embodiment, the post-filtering NN consists of initial parameters, one or more of which are one or more replacement parameters. One or more of the initial parameters can each be updated with one or more replacement parameters.

[0177] In one example, post-filtering information shows the difference between one or more parameters and each of the one or more initial parameters. The one or more parameters can be determined according to the sum of the difference and each of the one or more initial parameters.

[0178] The post-filtering neural network (NN) in the post-filtering module (1910) within the video decoder (e.g., video decoder (2000A) or (2000B)) can be determined based on post-filtering information. In one embodiment, one or more initial parameters in the post-filtering NN are updated based on one or more parameters, for example, one or more initial parameters are replaced by one or more parameters.

[0179] Referring to Figures 19B, 20A, and 20B, the encoded signal

number

number

number

number

number

[0180] In some examples, one or more parameters indicated by post-filtering information are decompressed and then used to update the post-filtering neural network.

[0181] According to one embodiment of the present disclosure, post-filtering training can be called a fine-tuning process, in which one or more initial parameters (e.g., one or more pre-trained parameters) of the initial parameters in the post-filtering NN can be updated (e.g., fine-tuned) based on the encoded and / or transmitted input signal. The input signal may be different from the training signal used to obtain the initial parameters. Thus, the post-filtering NN can be adapted to target the content of the input signal.

[0182] In one example, the input signal includes one or more input signals that are encoded and / or transmitted. In another example, the one or more input signals include a single input signal (e.g., a single input image or a single input block), and post-filtering training is performed using the single input signal. The post-filtering NN is determined (e.g., trained or updated) based on the single input signal. The determined post-filtering NN on the decoder side can be used to post-filter the restored signal (e.g., a restored image or a restored block) corresponding to the single input signal, and optionally other restored signals corresponding to other input signals. The post-filtering information can be encoded within the video bitstream together with the encoded single input signal (e.g., an encoded image or an encoded block).

[0183] In one embodiment, one or more input signals comprise multiple input signals, and post-filtering training is performed using multiple input signals. The post-filtering NN is determined (e.g., trained or updated) based on the multiple input signals. The determined post-filtering NN on the decoder side can be used to post-filter the reconstructed signals corresponding to the multiple input signals, and optionally other reconstructed signals corresponding to other input signals. The post-filtering information can be encoded within the bitstream together with the encoded multiple input signals.

[0184] Rate loss R p This can increase with the signaling of post-filtering information within the video bitstream. When the input signal contains a single input signal, post-filtering information is signaled for each encoded signal, and the rate loss R p The first increase to is due to the rate loss R caused by signaling of post-filtering information per signal. p It is used to indicate an increase to. When the input signal contains multiple input signals, the post-filtering information is signaled for the multiple input signals and shared by the multiple input signals, resulting in a rate loss R pThe second increase is due to the rate loss R caused by signaling of post-filtering information per signal. p It is used to indicate an increase to . Post-filtering information is shared by multiple input signals, so the rate loss R p The second increase to is the rate loss R p The increase to the first value may be smaller. Therefore, in some cases, it may be advantageous to use multiple input signals to determine (e.g., train or update) the post-filtering NN.

[0185] Post-filtering neural networks (NNs) can be specified by different types of parameters, such as weights and biases. Post-filtering NNs can consist of appropriate initial parameters, such as weights, biases, or a combination of weights and biases. When a CNN is used, the weights can include elements within the convolutional kernel. As mentioned above, a subset or the entire set of initial parameters can be updated through post-filtering training.

[0186] In one embodiment, the one or more initial parameters to be updated are located within a single layer of the post-filtering NN (e.g., a single convolutional layer). In another embodiment, the one or more initial parameters to be updated are located within multiple or all layers of the post-filtering NN (e.g., multiple or all convolutional layers).

[0187] One or more types of parameters can be used to specify a post-filtering neural network. In one embodiment, the one or more initial parameters to be updated are bias terms, and only the bias terms are replaced by one or more determined parameters. In one embodiment, the one or more initial parameters to be updated are weights, and only the weights are replaced by one or more determined parameters. In one embodiment, the one or more initial parameters to be updated include weights and bias terms, and the weights and bias terms are replaced by one or more determined parameters. In one example, the entire set of initial parameters is replaced by one or more determined parameters.

[0188] Post-filtering training can involve multiple epochs (e.g., iterations), with one or more initial parameters being updated during the iterative process. Post-filtering training can be stopped when the training loss has flattened or is about to flatten. In one example, post-filtering training may be stopped when the training loss (e.g., RD ​​loss L) is flattened. p The process stops when the difference between two consecutive training losses falls below the first threshold. In one example, post-filtering training stops when the difference between two consecutive training losses falls below the second threshold.

[0189] Two hyperparameters (e.g., step size and maximum number of steps) determine the loss function (e.g., RD ​​loss L). p It can be used in post-filtering training along with ). The maximum number of iterations can be used as a threshold for the maximum number of iterations to terminate post-filtering training. In one example, post-filtering training stops when the number of iterations reaches the maximum number of iterations.

[0190] The step size can indicate the learning rate of an online training process (e.g., post-filtering training). The step size can be used in the backpropagation calculation performed in gradient descent algorithms or post-filtering training. The step size can be determined using any suitable method.

[0191] In block-based post-filtering training, the step size for each block in an image may vary. In one embodiment, different step sizes can be assigned to the image to achieve better compression results (e.g., better RD loss Lp).

[0192] In image-based post-filtering training, the step size may vary per image. In one embodiment, different step sizes can be assigned to different input images to achieve better compression results (e.g., better RD loss Lp).

[0193] In one embodiment, different step sizes are used for signals with different types of content (e.g., blocks or images) to achieve optimal results. The signal consists of an input signal (e.g., input signal x) and an encoded signal (e.g., encoded signal x).

number

number

number

number

[0194] In one embodiment, the step size is selected based on characteristics of the signal (e.g., block or image), such as the RGB dispersion of the signal. In another embodiment, the step size is selected based on the RD performance (e.g., RD ​​loss L) of the signal (e.g., block or image). Multiple sets of parameters (e.g., multiple sets of substitution parameters) can be generated based on different step sizes, and a set having better compression performance (e.g., lower RD loss) can be selected.

[0195] In one embodiment, a first step size can be used to perform a specific number of iterations (e.g., 100). Then, a second step size (e.g., the first step size plus or minus a size increment) can be used to perform a specific number of iterations. To determine the step size to be used, the results from the first step size and the second step size can be compared. Three or more step sizes may be tested to determine the optimal step size.

[0196] The step size can change during post-filtering training. The step size may have an initial value at the start of post-filtering training, and this initial value can be reduced (e.g., halved) after a certain number of iterations in later stages of post-filtering training, for example, to achieve finer adjustments. The step size or learning rate can be modified by the scheduler during iterative post-filtering training. The scheduler may include a parameter tuning method used to adjust the step size. The scheduler can determine a value for the step size such that the step size may increase, decrease, or remain constant at several intervals. In one example, the learning rate is changed by the scheduler at each step. A single scheduler or multiple different schedulers can be used for different blocks or different images. Thus, multiple sets of parameters can be generated based on multiple schedulers, resulting in better compression performance (e.g., smaller RD loss L). p One of several sets of parameters having ) can be selected.

[0197] In one embodiment, multiple learning rate schedules are assigned to different signals (e.g., different blocks or different images) to achieve better compression results. In one example, all blocks in an image share the same learning rate schedule. In another example, a set of images share the same learning rate schedule. In one embodiment, the selection of the learning rate schedule is based on characteristics of the signal (e.g., blocks or images), such as the RGB variance of the signal. In one embodiment, the selection of the learning rate schedule is based on the RD performance of the signal.

[0198] In one embodiment, different signals (e.g., different blocks or different images) can be used to determine (e.g., update) different types of parameters (e.g., biases or weights) in a post-filtering neural network. For example, a first signal (e.g., a first block) is used to update at least one bias in the post-filtering neural network, and a second signal (e.g., a second block) is used to update at least one weight in the post-filtering neural network.

[0199] In one embodiment, multiple signals (e.g., multiple blocks within one image or multiple images) are used to determine (e.g., update) the same one or more parameters. In one example, all blocks within an image are used to determine (e.g., update) the same one or more parameters.

[0200] In one embodiment, one or more initial parameters to be updated are selected based on characteristics of the signal (e.g., block or image), such as the RGB dispersion of the signal. In another embodiment, one or more initial parameters to be updated are selected based on the RD performance of the signal.

[0201] At the end of post-filtering training, one or more updated parameters can be calculated for each of the determined one or more parameters (e.g., each of the one or more substitution parameters). In one embodiment, the one or more updated parameters are calculated as the difference between the determined one or more parameters (e.g., each of the one or more substitution parameters) and the corresponding one or more initial parameters (e.g., one or more pre-trained parameters). In one embodiment, each of the one or more updated parameters is one or more of the determined parameters.

[0202] In one embodiment, a method for obtaining one or more updated parameters from each of the determined one or more parameters is a signal used in post-filtering training (e.g., input signal x, restored signal x).

number

[0203] In one embodiment, different blocks have different relationships between one or more updated parameters and one or more replacement parameters. For example, in the first block, one or more updated parameters are calculated as the difference between one or more replacement parameters and one or more corresponding pre-trained parameters. In the second block, one or more updated parameters are each one or more replacement parameters.

[0204] In one embodiment, a method for obtaining one or more updated parameters from each of the determined one or more parameters is a signal used in post-filtering training (e.g., input signal x, restored signal x).

number

[0205] In one embodiment, the relationship between one or more updated parameters and one or more replacement parameters is selected based on characteristics of the signal (e.g., block or image), such as the RGB dispersion of the signal. In one embodiment, the relationship between one or more updated parameters and one or more replacement parameters is selected based on the RD performance of the signal.

[0206] In one embodiment, one or more updated parameters can be generated from one or more determined parameters (e.g., one or more substitution parameters) using, for example, a specific linear or nonlinear transformation, and the one or more updated parameters are representative parameters generated based on the one or more determined parameters. The one or more determined parameters are transformed into one or more updated parameters for better compression.

[0207] In one example, one or more updated parameters can be compressed using, for example, the LZMA2 algorithm, a variation of the Lempel-Ziv-Markov chain algorithm (LZMA), or the bzip2 algorithm. In one example, compression is omitted for one or more updated parameters. In some embodiments, one or more updated parameters can be encoded in the bitstream as post-filtering information, which may indicate one or more determined parameters (e.g., one or more substitution parameters).

[0208] In one embodiment, the compression method for one or more updated parameters differs for different signals (e.g., different blocks or different images). For example, LZMA2 is used to compress one or more updated parameters for a first block, and bzip2 is used to compress one or more updated parameters for a second block. In one embodiment, the same compression method is used to compress one or more updated parameters for multiple blocks in an image (e.g., all blocks). In one embodiment, the compression method is selected based on characteristics of the signal (e.g., blocks or images), such as the RGB dispersion of the signal. In one embodiment, the compression method is selected based on the RD performance of the signal.

[0209] In one embodiment, the structure (e.g., architecture) of the post-filtering NN is identical for each signal. The structure of the post-filtering NN can include the number of layers, how different nodes and / or layers are organized and connected, and whether it is a feedforward architecture, recurrent architecture, DNN, CNN, etc. In one example, the structure refers to the number of convolutional layers, and different blocks have the same number of convolutional layers.

[0210] In one embodiment, different structures of the post-filtering NN correspond to different signals. In one example, the post-filtering NNs used to post-filter different blocks have different numbers of convolutional layers.

[0211] In one embodiment, whether a post-filtering module is used to post-filter a signal (e.g., a block or an image) depends on the RD loss L p It is determined based on a comparison of the RD loss L. In one embodiment, the best RD performance (e.g., minimum RD loss L) is achieved. p Post-filtering modules with different RD loss L values ​​for different post-filtering modules ps The selection is made based on comparison.

[0212] According to one embodiment of the present disclosure, each signal (e.g., each block or each image) corresponds to a post-filtering NN determined in post-filtering training based on the signal. A post-filtering NN can be updated independently of another post-filtering NN. For example, a post-filtering NN corresponding to a first signal is updated independently of a post-filtering NN corresponding to a second signal.

[0213] According to one embodiment of the present disclosure, a post-filtering NN corresponding to a signal (e.g., a second signal) can be updated based on a post-filtering NN corresponding to another signal (e.g., a first signal). In one example, first post-filtering information in a coded bitstream corresponding to the first signal is decoded. The first post-filtering information may indicate first parameters of a post-filtering NN in a video decoder. The post-filtering NN in the video decoder corresponding to the first signal can be determined based on the first parameters indicated by the first post-filtering information. In one example, the first signal includes a first coded block to be restored. The first signal can be decoded based on the determined post-filtering NN corresponding to the first signal. For example, the first signal is restored, and the restored first signal is post-filtered based on the determined post-filtering NN.

[0214] For example, parameters in a post-filtering NN corresponding to a first signal can be used to update a post-filtering NN corresponding to a second signal. For example, the pixel distribution within blocks of the same image may be similar, and therefore, the updated parameters of the post-filtering NN corresponding to different blocks may be reduced. According to one embodiment of the present disclosure, second post-filtering information in an encoded bitstream corresponding to a second signal can be decoded. The second post-filtering information may indicate a second parameter. The second signal is different from the first signal. The post-filtering NN corresponding to the first signal can be updated based on the second parameter. The updated post-filtering NN corresponds to the second signal and consists of the first and second parameters. The second signal can be decoded (e.g., post-filtered) based on the updated post-filtering NN corresponding to the second signal.

[0215] In one embodiment, different post-filtering neural networks (NNs) can be applied to signals of different sizes (e.g., blocks or images). Generally, the number of parameters in a post-filtering NN may increase with the size (e.g., width, height, or area) of the signal (e.g., block or image).

[0216] In one embodiment, different post-filtering neural networks are applied to the same signal (e.g., an image or a block) corresponding to different compression quality targets.

[0217] The NIC framework and post-filtering NN can include any type of neural network and can use any neural network-based image compression method, such as the context-hyperpre-encoder-decoder framework, the scale-hyperpre-encoder-decoder framework, the Gaussian mixture likelihood framework and variations of the Gaussian mixture likelihood framework, the RNN-based recurrent compression method and variations of the RNN-based recurrent compression method.

[0218] Compared with related end-to-end image compression methods, the post-filtering training method and apparatus of this disclosure may have the following advantages: The adaptive online training mechanism is utilized to improve coding efficiency. The use of a flexible and general framework can be adapted to various types of pre-trained frameworks and quality metrics. For example, certain pre-trained parameters in a post-filtering NN can be replaced by using online training with encoded and transmitted blocks or images.

[0219] Video coding techniques may include filtering operations performed on the reconstructed samples to reduce artifacts arising from lossy compression, such as quantization. One such filtering operation is a deblocking filter process, in which block boundaries between two adjacent blocks (e.g., boundary regions) can be filtered to enable a smoother transition of sample values ​​from one block to the other.

[0220] In some related examples (e.g., HEVC), a deblocking filter process can be applied to samples adjacent to block boundaries. The deblocking filter process can be performed per CU in the same order as the decoding process. For example, the deblocking filter process can be performed by first horizontal filtering on the vertical boundaries of the image, followed by vertical filtering on the horizontal boundaries of the image. Filtering can be applied to the boundaries of 8x8 blocks that are determined to be filtered, for both luminance and saturation components. In one example, 4x4 block boundaries are not processed to reduce complexity.

[0221] A boundary strength (BS) can be used to indicate the degree or intensity of the deblocking filtering process. In one embodiment, a value of 2 for BS indicates strong filtering, 1 indicates weak filtering, and 0 indicates no deblocking filtering.

[0222] Blocks of an image can be reconstructed from an encoded video bitstream using any suitable method, such as those found in embodiments of the present disclosure. For example, a block can be reconstructed using a video decoder (e.g., (1600B), (1800), (2000A), or (2000B)) which includes, for example, a neural network (e.g., CNN). A block can also be reconstructed using a video decoder that is not based on an NN. According to some embodiments of the present disclosure, post-filtering or post-processing can be performed on one of a plurality of regions of a first pair of adjacent reconstructed blocks of the reconstructed block using at least one post-processing NN. The first pair of adjacent reconstructed blocks may have a first shared boundary and may include a boundary region having samples on both sides of the first shared boundary. The plurality of regions of the first pair of adjacent reconstructed blocks may include a boundary region and an unbounded region outside the boundary region. One of the plurality of regions can be replaced with one of the post-processed regions of the plurality of regions of the first pair of adjacent reconstructed blocks. The post-processing performed may include deblocking of the bound region, extending one or more of the non-boundary regions, or a combination of deblocking and extension.

[0223] One or more deblocking methods can be used to reduce artifacts between blocks (e.g., reconstructed blocks in an image). One or more NN-based deblocking models can be used to reduce interblock artifacts, such as artifacts in boundary regions. NN-based deblocking models may include DNN-based deblocking models, CNN-based deblocking models, etc. NN-based deblocking models can be implemented using NNs such as DNNs and CNNs.

[0224] In one embodiment, one of the multiple regions is a boundary region. At least one post-processing NN includes at least one deblocking NN, and deblocking can be performed on the boundary region using at least one deblocking NN. The boundary region can be replaced with the deblocked boundary region. Examples of deblocking are shown in Figures 21A-21C, 22, 23, and 26.

[0225] Figures 21A to 21C illustrate an exemplary deblocking process (2100) according to one embodiment of the present disclosure. Referring to Figure 21A, an image (2101) can be divided into several blocks (2111) to (2114). For brevity, Figure 21A shows four blocks (2111) to (2114) of equal size. In general, an image can be divided into any appropriate number of blocks, and the blocks may be of different or identical sizes, and the explanation can be appropriately adapted. In some examples, for example, areas containing artifacts resulting from dividing the image into blocks can be processed by deblocking.

[0226] In one example, blocks (2111) to (2114) are reconstructed blocks from the main decoder network (915). The first two adjacent reconstructed blocks among (2111) to (2114) may include blocks (2111) and (2113) separated by a first shared boundary (2141). Blocks (2111) and (2113) may include boundary region A, which has samples on both sides of the first shared boundary (2141). Referring to Figures 21A and 21B, boundary region A may include sub-boundary regions A1 and A2 located in blocks (2111) and (2113), respectively.

[0227] Two adjacent reconstructed blocks from reconstructed blocks (2111) to (2114) may include blocks (2112) and (2114) separated by a second shared boundary (2142). Blocks (2112) and (2114) may include boundary region B, which has samples on both sides of the second shared boundary (2142). Referring to Figures 21A and 21B, boundary region B may include sub-boundary regions B1 and B2 located in blocks (2112) and (2114), respectively.

[0228] Two adjacent reconstructed blocks from reconstructed blocks (2111) to (2114) may include blocks (2111) and (2112) separated by a shared boundary (2143). Blocks (2111) and (2112) may include boundary regions C with samples on both sides of the shared boundary (2143). Referring to Figures 21A and 21B, boundary region C may include sub-boundary regions C1 and C2 located in blocks (2111) and (2112), respectively.

[0229] Two adjacent reconstructed blocks from reconstructed blocks (2111) to (2114) may include blocks (2113) and (2114) separated by a shared boundary (2144). Blocks (2113) and (2114) may include boundary regions D with samples on both sides of the shared boundary (2144). Referring to Figures 21A and 21B, boundary region D may include sub-boundary regions D1 and D2 located in blocks (2113) and (2114), respectively.

[0230] Subboundary regions A1-D1 and A2-D2 (and boundary regions A-D) can have any suitable size (e.g., width and / or height). In the embodiment shown in Figure 21A, subboundary regions A1, A2, B1, and B2 have the same size of m × n, where n is the width of block (2111)-(2114) and m is the height of subboundary regions A1, A2, B1, and B2. Both m and n are positive integers. In one example, m is 4 pixels or 4 samples. Thus, boundary regions A and B have the same size of 2m × n. Subboundary regions C1, C2, D1, and D2 have the same size of n × m, where n is the height of block (2111)-(2114) and m is the width of subboundary regions C1, C2, D1, and D2. Thus, boundary regions C and D have the same size of n × 2m. As described above, subboundary regions and boundary regions can have different sizes, such as different widths and different heights. For example, subboundary regions A1 and A2 can have different heights. In one example, subboundary regions C1 and C2 can have different widths. Boundary regions A and B can have different widths. Boundary regions C and D can have different heights.

[0231] Referring to Figures 21A-21B, boundary region A includes m samples (e.g., m rows of samples) within block (2111) from the first shared boundary (2141) and m samples (e.g., m rows of samples) within block (2113) from the first shared boundary (2141). Boundary region C includes m samples (e.g., m columns of samples) within block (2111) from the shared boundary (2143) and m samples (e.g., m columns of samples) within block (2112) from the shared boundary (2143).

[0232] Deblocking can be performed on one or more of the boundary regions A to D using at least one deblocking NN, such as a DNN, CNN, or a deblocking NN based on any suitable NN. In one example, the at least one deblocking NN includes deblocking NN(2130). In one example, deblocking NN(2130) is implemented using a CNN that includes one or more convolutional layers. Deblocking NN(2130) may include additional layers described in this disclosure, such as pooling layers, fully connected layers, and normalization layers. The layers within deblocking NN(2130) can be arranged in any suitable order and in any suitable architecture (e.g., feedforward architecture, recurrent architecture). In one example, the convolutional layers are followed by other layers such as pooling layers, fully connected layers, and normalization layers.

[0233] Deblocking can be performed on boundary regions A-D using the deblocking NN(2130). One or more of the boundary regions A-D contain artifacts. Artifacts may be induced by their respective adjacent blocks. One or more of the boundary regions A-D can be sent to the deblocking NN(2130) to reduce artifacts. Therefore, the input to the deblocking NN(2130) includes one or more of the boundary regions A-D, and the output from the deblocking NN(2130) includes one or more of the deblocked boundary regions A-D.

[0234] Referring to Figure 21B, boundary regions A-D contain artifacts induced by their respective adjacent blocks. Boundary regions A-D can be sent to a deblocking NN(2130) to reduce artifacts. The output from the deblocking NN(2130) includes the deblocked boundary regions A'-D'. In one example, the artifacts in the deblocked boundary regions A'-D' are reduced compared to the artifacts in boundary regions A-D.

[0235] Referring to Figures 21B and 21C, boundary regions A-D in image (2101) are updated, for example, by being replaced by deblocked boundary regions A'-D'. Thus, image (2150) is generated, which includes the deblocked boundary regions A'-D' and non-boundary regions (2121)-(2124).

[0236] One or more samples may reside within multiple boundary regions. When multiple boundary regions are replaced by corresponding deblocked boundary regions, any suitable method can be used to determine the value of one of the one or more shared samples.

[0237] Referring to Figure 21A, sample S is located within boundary regions A and C. After obtaining boundary regions A' and C', the following method can be used to obtain the value of sample S. In one example, boundary region A is replaced by the deblocked boundary region A', and then boundary region C is replaced by the deblocked boundary region C'. Therefore, the value of sample S is determined by the value of sample S within the deblocked boundary region C'.

[0238] In one example, boundary region C is replaced by a deblocked boundary region C', and then boundary region A is replaced by a deblocked boundary region A'. Therefore, the value of sample S is determined by the value of sample S within the deblocked boundary region A'.

[0239] In one example, the value of sample S is determined by the average (e.g., weighted average) of the values ​​of sample S in the deblocked boundary region A' and the values ​​of sample S in the deblocked boundary region C'.

[0240] A boundary region can include samples of three or more blocks. Figure 22 shows an example of a boundary region containing samples of three or more blocks according to an embodiment of the present disclosure. A single boundary region AB can include boundary regions A and B. Boundary region AB may include samples on both sides of the shared boundary (2141) between two adjacent reconstructed blocks (2111) and (2113), and samples on both sides of the shared boundary (2142) between two adjacent reconstructed blocks (2112) and (2114). A single boundary region CD may include boundary regions C and D. The boundary region CD may include samples on both sides of the shared boundary (2143) between two adjacent reconstructed blocks (2111) and (2112), and may also include samples on both sides of the shared boundary (2144) between two adjacent reconstructed blocks (2113) and (2114).

[0241] A deblocking neural network (NN) such as deblockingNN(2130) can perform deblocking on one or more of the boundary regions AB and CD to generate one or more deblocked boundary regions. Referring to Figure 22, boundary regions AB and CD are sent to the deblockingNN(2130), and deblocked boundary regions AB' and CD' are generated. The deblocked boundary regions AB' and CD' can replace boundary regions AB and CD in image (2101), thus generating image (2250). Image (2250) may contain the deblocked boundary regions AB'~CD' and non-boundary regions (2121)~(2124).

[0242] According to embodiments of this disclosure, a multi-model deblocking method can be used. Different deblocking models can be applied to boundary regions of different types or categories to remove artifacts. A classification module can be applied to classify boundary regions into different categories. Any classification module can be applied. In one example, the classification module is based on a neural network. In another example, the classification module is not based on a neural network. Boundary regions can be sent to different deblocking models according to their respective categories.

[0243] In one embodiment, at least one deblocking NN includes multiple deblocking NNs, each implemented based on a different deblocking model. It is possible to decide which of the multiple deblocking NNs to apply to the boundary region. Deblocking can then be performed on the boundary region using the determined deblocking NNs. In one example, the decision of which of the multiple deblocking NNs to apply is determined by a classification module based on a NN (also known as a classification NN), such as a DNN or CNN.

[0244] Figure 23 shows an exemplary deblocking process (2300) based on multiple deblocking models according to one embodiment of the present disclosure. A classification module (2310) can classify boundary regions A to D into one or more categories. For example, boundary regions C to D are classified into a first category, boundary region B into a second category, and boundary region A into a third category. Different deblocking models can be applied to boundary regions of different categories. In Figure 23, a deblocking NN (2330) can be used to perform deblocking such as multi-model deblocking based on multiple deblocking models (e.g., deblocking models 1 to L). L is a positive integer. When L is 1, the deblocking NN (2330) includes a single deblocking model. When L is greater than 1, the deblocking NN (2330) includes multiple deblocking models.

[0245] In one example, deblocking model 1 is applied to boundary regions (e.g., C and D) within a first category, generating deblocked boundary regions (e.g., C'' and D''). Deblocking model 2 is applied to boundary regions (e.g., B) within a second category, generating a deblocked boundary region (e.g., B''). Deblocking model 3 is applied to boundary regions (e.g., A) within a third category, generating a deblocked boundary region (e.g., A''). The deblocked boundary regions A''~D'' can replace the corresponding boundary regions A~D in image (2101), thus generating image (2350). Image (2350) may contain the deblocked boundary regions A''~D'' and unboundary regions (2121)~(2124).

[0246] To classify or categorize boundary regions, any appropriate metric can be applied. For example, boundary regions are classified according to their content. For instance, boundary regions with high-frequency content (e.g., content with relatively large variance) and boundary regions with low-frequency content (e.g., content with relatively small variance) are classified into different categories corresponding to different deblocking models. The intensity of artifacts within a boundary region can be used to classify boundary regions. The multi-model deblocking method can be applied to any appropriate boundary region, such as boundary regions between two or more blocks (e.g., A, B, C, D, AB, and / or CD). The frequency of a boundary region may be determined based on the maximum difference of samples within the boundary region. For example, a first difference of samples near a first edge on the first side of a shared boundary is determined. For example, a second difference of samples near a second edge on the second side of a shared boundary is determined. For example, the first and second differences are determined.

[0247] To remove artifacts between blocks, a deblocking neural network (e.g., deblocking NN(2130) in Figure 21B or deblocking NN(2330) in Figure 23) can be applied. In one example, samples or pixels closer to the shared boundary can be deblocked more than samples (or pixels) further away from the shared boundary. Returning to Figure 21A, sample S is closer to the shared boundary (2141) than sample F, and therefore sample S can be deblocked more than sample F.

[0248] The deblocking model in a deblocking NN (e.g., the deblocking NN (2130) in Figure 21B or the deblocking NN (2330) in Figure 23) may include one or more convolutional layers. For example, a CNN-based attention mechanism (e.g., a nonlocal attention, squeeze, and excitation network (SENet), or a residual neural network (ResNet) (e.g., including a set of CNNs or convnets and activation functions) may be used. For example, a DNN used in image super-resolution can be used by, for example, modifying the output size to be the same as the input size. Image super-resolution can extend the resolution of an image from low resolution to high resolution.

[0249] The above describes how to perform deblocking on boundary regions using neural networks (NNs) or other learning-based methods. In some examples, video encoders and / or video decoders can choose between NN-based and non-NN-based deblocking methods. The choice can be made at various levels, such as slice level, picture level, picture group level, and sequence level. The choice can be signaled using flags. The choice can be inferred from the content of the boundary region.

[0250] A video encoder and / or video decoder may apply various levels of boundary intensity (BS) in addition to the methods and embodiments described herein, for example, when the NN derivation adjustment for a pixel or sample is at the default level of boundary intensity (BS). Different levels of BS may be assigned to modify (e.g., enlarge or reduce) the default adjustment by analyzing boundary conditions and block coding features.

[0251] According to one embodiment of the present disclosure, at least one post-processing NN may include at least one extension NN. One or more unbounded regions of adjacent restored blocks may be extended by at least one extension NN. One or more unbounded regions may be replaced by one or more extended unbounded regions.

[0252] The reconstructed image (e.g., image (2101) in Figure 21A) can be sent to an augmentation module to generate an augmented image (e.g., the final reconstructed image). In some embodiments where deblocking is performed, the reconstructed image can be sent to the augmentation module after artifacts have been reduced by using a deblocking NN. To improve image quality, NN-based post-augmentation models (e.g., post-augmentation models based on DNNs or CNNs) can be used in post-augmentation modules such as the post-augmentation NN (2430) in Figure 24.

[0253] Figure 24 shows an exemplary extension process (2400) according to one embodiment of the present disclosure. In some examples, non-boundary regions (2121)~(2124) in image (2101) (e.g., the remaining region other than boundary regions A~D) are not sent to the deblocking module (e.g., deblocking NN(2130)). In one example, the non-boundary region (e.g., non-boundary region (2121)) is from a restored block (e.g., (2111)) in image (e.g., (2101)), and the size of the boundary region may be (nm) × (nm). As described with reference to Figure 21A, n is the side length (e.g., width and / or height) of the restored block (e.g., (2111)), and m is the side length of the sub-boundary region (e.g., A1) for deblocking. One or more of the non-boundary regions (2121)~(2124) can be sent to an augmentation module to further improve the quality of one or more of the non-boundary regions (2121)~(2124). One or more of the augmented non-boundary regions can replace one or more of the non-boundary regions (2121)~(2124) in the image. Referring to Figure 24, the non-boundary regions (2121)~(2124) are fed to the post-augmentation NN (2430) to generate the augmented non-boundary regions (2121')~(2124'). The augmented non-boundary regions (2121')~(2124') can replace the augmented non-boundary regions (2121)~(2124) to generate the augmented image (2450).

[0254] In one example, an unbounded region overlaps with a bounded region such that a portion of the unbounded region lies within the bounded region. In one example, the unbounded region is an entire coding block. Referring to Figure 24, block (2111) can be an unbounded region, and therefore the unbounded region (2111) is adjacent to other neighboring blocks such as (2112)~(2113).

[0255] In some embodiments, at least one augmented NN is based on multiple augmented models (e.g., post-augmented models). Which of the multiple augmented models to apply to the non-boundary region can be determined, for example, by a classification module. The non-boundary region can be augmented using the determined augmented model. In one example, which of the multiple augmented models to apply is determined by a classification module based on an NN (e.g., also called a classification NN), such as a DNN or CNN. The classification module (e.g., classification module (2510) used in the post-augmented process (e.g., (2500))) may be the same as or different from the classification module (e.g., classification module (2310)) used in the deblocking process (e.g., (2300)). The classification module (e.g., classification module (2510) used in the post-augmented process) may include an NN (e.g., a DNN or CNN). In one example, the classification module (e.g., classification module (2510) used in the post-augmented process) does not include an NN.

[0256] Figure 25 shows an exemplary extension process (2500), such as a multi-model post-extension module, according to one embodiment of the present disclosure.

[0257] The classification module (2510) can classify unbounded regions (2121) to (2124) into one or more categories. For example, unbounded regions (2122) to (2123) are classified into the first category, and unbounded regions (2121) and (2124) are classified into the second category. Different extension models (e.g., post-extension models) can be applied to unbounded regions in different categories. In Figure 25, the extension NN (2530) can be used to perform extensions such as multi-model extensions based on multiple extension models (e.g., extension models 1 to J). J is a positive integer. When J is 1, the extension NN (2530) contains a single extension model. When J is greater than 1, the extension NN (2530) contains multiple extension models.

[0258] In one example, extension model 1 is applied to the non-boundary region of the first category (e.g., (2122)~(2123)), generating the extended non-boundary region (e.g., (2122'')~(2123'')). Extension model 2 is applied to the non-boundary region of the second category (e.g., (2121) and (2124)), generating the extended non-boundary region (e.g., (2121'') and (2124'')). The extended non-boundary region (2121'')~(2124'') can replace the corresponding non-boundary region (2121)~(2124), and the extended image (2550) includes the extended non-boundary region (2121'')~(2124'') and boundary regions A~D.

[0259] Any appropriate metric can be applied to classify or categorize non-boundary regions. For example, non-boundary regions can be classified according to their content. For instance, non-boundary regions with high-frequency content (e.g., content with relatively large variance) and non-boundary regions with low-frequency content (e.g., content with relatively small variance) can be classified into different categories corresponding to different extension models.

[0260] Images can be augmented at the block level, as described with reference to Figures 21–25. Augmentation models (e.g., post-augmentation models) can augment the entire image. Figure 26 shows an exemplary image-level augmentation process (2600) for augmenting the entire image according to one embodiment of the present disclosure. Image (2101) includes non-boundary regions (2121)–(2124) and boundary regions A–D, as described in Figure 21A. In one example, image (2101) is a reconstructed image including reconstructed blocks (2111)–(2114) as described above. Artifacts within boundary regions can be reduced, and non-boundary regions can be augmented with improved appearance.

[0261] Referring to Figure 26, an image (2101) containing boundary regions A-D and non-boundary regions (2121)-(2124) can be supplied to the extension module (2630). The extension module (2630) can generate extended boundary regions E-H corresponding to boundary regions A-D, for example, by deblocking boundary regions A-D. The extension module (2630) can also generate extended non-boundary regions (2621)-(2624) corresponding to non-boundary regions (2121)-(2124), respectively. The extended boundary regions E-H can replace boundary regions A-D, respectively, and the extended non-boundary regions (2621)-(2624) can replace non-boundary regions (2121)-(2124), respectively, and thus the extended image (2650) is generated based on the reconstructed image (2101).

[0262] For example, the image-based extension module (2630) includes an extension NN that can perform both deblocking and extension. For example, the image-based extension module (2630) includes an extension NN that can perform extension and a deblocking NN that can perform deblocking.

[0263] The extension modules described with reference to Figures 24-26 (e.g., (2430), (2530), and (2630)) can enhance image quality. These extension modules (e.g., (2430), (2530), and (2630)) may include one or more convolutional layers. CNN-based attention mechanisms (e.g., nonlocal attention, SENet), ResNet (e.g., including a set of CNNs or convnets and activation functions), etc., may be used. For example, a DNN used in image super-resolution can be used, for example, by modifying the output size to be the same as the input size.

[0264] Boundary and non-boundary regions within an image can be processed by extended neural networks and deblocking neural networks in any appropriate order, such as sequentially or simultaneously. In one example, boundary regions are deblocked by the deblocking neural network, and then non-boundary regions are processed by the extended neural network. In another example, non-boundary regions are processed by the extended neural network, and then boundary regions are deblocked by the deblocking neural network.

[0265] According to one embodiment of the present disclosure, an extended NN (e.g., (2430), (2530), or (2630)), a deblocking NN (e.g., (2130) or (2330)), and / or a classification NN (e.g., (2310) or (2510)) may include any neural network architecture as described in the present disclosure, may include any number of layers, may include one or more subneural networks, and may be trained on any suitable training image or training block. The training image may include a raw image or an image containing residual data. The training block may be a raw image or an image containing residual data.

[0266] As described in this disclosure, content-adaptive online training can be applied to update one or more pre-trained parameters in one of the following: an extended NN (e.g., (2430), (2530), or (2630)), a deblocking NN (e.g., (2130) or (2330)), and / or a classification NN (e.g., (2310) or (2510)).

[0267] Extended NNs (e.g., (2430), (2530), or (2630)), deblocking NNs (e.g., (2130) or (2330)), and / or classification NNs (e.g., (2310) or (2510)) can be trained separately; for example, a single deblocking NN is trained to determine pre-trained parameters in a deblocking NN. Extended NNs (e.g., (2430), (2530), or (2630)), deblocking NNs (e.g., (2130) or (2330)), and / or classification NNs (e.g., (2310) or (2510)) can be trained (pre-trained or trained online) as components within the NIC framework. For example, the NIC framework (900) and at least one of the following can be trained together: an extended NN (e.g., (2430), (2530), or (2630)), a deblocking NN (e.g., (2130) or (2330)), and / or a classification NN (e.g., (2310) or (2510)).

[0268] In one embodiment, the post-filtering NN within the post-filtering module (1910) may include one or more of the following: extension NNs (e.g., (2430), (2530), or (2630)), deblocking NNs (e.g., (2130) or (2330)), and classification NNs (e.g., (2310) or (2510)). The extension NNs (e.g., (2430), (2530), or (2630)) and deblocking NNs (e.g., (2130) or (2330)) can be applied to the reconstructed image or the reconstructed block.

[0269] An extension process (e.g., (2400), (2500), or (2600)) can be called a post-extension process because, for example, the extension process is executed after a block has been restored. For the same reason, an extension module (e.g., (2430), (2530), or (2630)) can be called a post-extension module or post-extension NN.

[0270] In some examples, the reconstructed image (2101) includes residual data.

[0271] Figure 27 shows a flowchart outlining an encoding process (2700) according to one embodiment of the present disclosure. Process (2700) can be used when encoding an input signal (e.g., an input block or an input image). In various embodiments, process (2700) is performed by processing circuits, such as processing circuits within terminal devices (310), (320), (330), and (340), and processing circuits that perform the functions of a video encoder (e.g., (403), (603), (703), (1600A), or (1700)). In one example, the processing circuit performs a combination of functions, such as (i) one of the video encoders (403), (603), and (703), and (ii) one of the video encoders (1600A) or (1700). In some embodiments, process (2700) is implemented by software instructions, and therefore, when the processing circuit executes a software instruction, the processing circuit executes process (2700). The process starts at (S2701) and proceeds to (S2710).

[0272] In (S2710), the encoded signal (for example,

number

number

[0273] In (S2720), one or more parameters of the post-filtering NN in the post-filtering module (e.g., (1910)) are used in the post-filtering training to restore the signal

number

number

number

[0274] In (S2730), post-filtering information indicating one or more determined parameters of the post-filtering NN can be encoded. The post-filtering information is encoded into the encoded signal

number

number

[0275] Process (2700) can be appropriately adapted to various scenarios, and the steps within Process (2700) can be adjusted accordingly. One or more steps within Process (2700) can be adapted, omitted, repeated, and / or combined. Any appropriate order can be used to carry out Process (2700). Further steps can be added.

[0276] Figure 28 shows a flowchart outlining a decoding process (2800) according to one embodiment of the present disclosure. Process (2800) can be used in the restoration and post-filtering of an encoded signal (e.g., an encoded block or an encoded image). In various embodiments, process (2800) is performed by processing circuits such as processing circuits within terminal devices (310), (320), (330), and (340), processing circuits that perform the functions of a video decoder (1600B), and processing circuits that perform the functions of a video decoder (e.g., (2000A) or (2000B)). In one example, the processing circuit performs a combination of functions such as (i) one of video decoders (410), video decoder (510), and video decoder (810), and (ii) one of video decoders (2000A) or video decoder (2000B). In some embodiments, process (2800) is implemented with software instructions, and therefore, when the processing circuit executes the software instructions, the processing circuit executes process (2800). The process starts at (S2801) and proceeds to (S2810).

[0277] In (S2810), first post-filtering information in the encoded bitstream corresponding to the first encoded signal can be decoded. The first encoded signal may be an encoded block or an encoded image. The first post-filtering information may indicate first parameters (e.g., first post-filtering parameters) of a post-filtering NN in the video decoder. The post-filtering NN can be implemented by a post-filtering module (e.g., (1910)) in the video decoder (e.g., (2000A) or (2000B)).

[0278] In one example, the first encoded signal includes the first encoded image to be restored, and the first post-filtering information corresponds to the first encoded image.

[0279] In one example, the first post-filtering parameter further corresponds to the second image being restored.

[0280] The first parameter (for example, the first post-filtering parameter) can be a bias term or weight coefficient in the post-filtering neural network. In one example, the first post-filtering parameter is updated based on the bias term or weight coefficient in the post-filtering neural network.

[0281] The first post-filtering information can represent one or more parameters of the post-filtering NN in various ways. In one example, the post-filtering NN consists of initial parameters. The first post-filtering information represents the difference between the first parameter and one of the initial parameters, and the first parameter can be determined according to the sum of the difference and one of the initial parameters. In another example, the first post-filtering information directly represents one or more parameters.

[0282] For example, the number of layers in a post-filtering neural network depends on the size of the first encoded signal (e.g., width, height, or area).

[0283] For example, the number of layers in a post-filtering neural network depends on the step size or number of steps corresponding to different blocks within one or more blocks.

[0284] In one example, an image or video containing one or more blocks is received. A first post-filtering parameter in the image or video corresponding to the one or more blocks to be restored is decoded. The first post-filtering parameter can be applied to at least one of the one or more blocks. The first post-filtering parameter is updated by a post-filtering module in a post-filtering neural network trained on a training dataset.

[0285] In (S2820), the post-filtering NN in the video decoder corresponding to the first encoded signal can be determined, for example, based on a first parameter (e.g., a first post-filtering parameter) indicated by the first post-filtering information. For example, at least one of the initial parameters configured for the post-filtering NN is updated (e.g., replaced) with the first parameter (e.g., a first post-filtering parameter).

[0286] In (S2830), the first encoded signal can be decoded based on the determined post-filtering NN corresponding to the first encoded signal. The first encoded signal can be restored to generate a restored signal. The restored signal can be post-filtered based on the determined post-filtering NN to generate a post-filtered signal.

[0287] In one example, the first encoded signal includes encoded blocks, which can be reconstructed. The reconstructed blocks can be post-filtered to generate post-filtered blocks based on a determined post-filtering neural network.

[0288] In one example, the first encoded signal includes the first encoded image, which is decoded based on a determined post-filtering neural network (NN) corresponding to the first encoded image.

[0289] In one example, the second encoded image is decoded based on a determined post-filtering neural network corresponding to the first encoded image.

[0290] Process (2800) proceeds to (S2899) and terminates.

[0291] Process (2800) can be appropriately adapted to various scenarios, and the steps within Process (2800) can be adjusted accordingly. One or more steps within Process (2800) can be adapted, omitted, repeated, and / or combined. Any appropriate order can be used to carry out Process (2800). Further steps can be added.

[0292] In one example, a second post-filtering parameter in an image or video corresponding to one or more blocks is decoded. A post-filtering neural network (NN) can be determined based on the second post-filtering parameter. The second post-filtering parameter can be applied to a second block within one or more blocks. The second block may be different from at least one of the one or more blocks. The second post-filtering parameter is updated by a post-filtering module within the post-filtering NN.

[0293] The first post-filtering parameter may differ from the second post-filtering parameter, and the first post-filtering parameter can be applied to the content of the first block among one or more blocks, while the second post-filtering parameter can be applied to the content of the second block. The first block may differ from the second block.

[0294] The first post-filtering parameter can be determined based on the content of the first block among one or more blocks, and the second post-filtering parameter can be determined based on the content of the second block.

[0295] In one example, the first post-filtering parameter corresponds to the second image to be restored, and the second image is decoded based on the determined post-filtering neural network.

[0296] In one example, second post-filtering information in the coded bitstream corresponding to the second coded signal is decoded. The second post-filtering information indicates the second parameter. The second coded signal is different from the first coded signal. The post-filtering NN corresponding to the first coded signal can be updated based on the second parameter. The updated post-filtering NN corresponds to the second coded signal and can consist of the first and second parameters. The first and second parameters can include (i) two different weight coefficients, (ii) two different biases, or (iii) weight coefficients and biases of the updated post-filtering NN. The second coded signal can be decoded based on the updated post-filtering NN corresponding to the second coded signal.

[0297] In one example, the first post-filtering parameter is updated in (i) a single layer of the post-filtering NN, (ii) multiple layers of the post-filtering NN, or (iii) all layers of the post-filtering NN.

[0298] Embodiments of this disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), encoder, and decoder may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-temporary computer-readable medium.

[0299] This disclosure does not impose any restrictions on methods used in encoders such as neural network-based encoders, or decoders such as neural network-based decoders. The neural network used in encoders, decoders, etc., may be any suitable type of neural network, such as DNNs or CNNs.

[0300] Therefore, the content-adaptive online training method of this disclosure can be adapted to different types of NIC frameworks, such as different types of encoding DNNs, decoding DNNs, encoding CNNs, decoding CNNs, etc.

[0301] The techniques described above can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 29 shows a computer system (2900) suitable for implementing some embodiments of the disclosed subject matter.

[0302] Computer software can be coded using any suitable machine language or computer language that can undergo assembly, compilation, linking, or similar mechanisms to create code that contains instructions that can be executed directly or via interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0303] Instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things devices.

[0304] The components shown in Figure 29 with respect to the computer system (2900) are essentially illustrative and do not imply any limitation on the scope of use or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should not be construed as having any dependence or requirement on any one or combination of components shown in the exemplary embodiments of the computer system (2900).

[0305] The computer system (2900) may include certain human interface input devices. Such human interface input devices can respond to input from one or more human users, for example, via tactile input (such as keystrokes, swipes, or data glove movements), audio input (such as voices or clapping), visual input (such as gestures), or olfactory input (not depicted). Human interface devices can also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (such as voices, music, or ambient sounds), images (such as scanned images or photographic images taken from a still camera), or video (such as two-dimensional video or three-dimensional video, including stereoscopic video).

[0306] Input human interface devices may include one or more of the following: keyboard (2901), mouse (2902), trackpad (2903), touchscreen (2910), data glove (not shown), joystick (2905), microphone (2906), scanner (2907), and camera (2908) (only one of each is depicted).

[0307] The computer system (2900) may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (2910), data glove (not shown), or joystick (2905), although there may also be tactile feedback devices that do not function as input devices), audio output devices (such as speakers (2909), headphones (not depicted)), visual output devices (such as screens (2910), including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capabilities, each with or without tactile feedback capabilities, some of which may be capable of outputting two-dimensional visual output or three-dimensional or more output via means such as stereographic output, virtual reality glasses (not depicted), holographic displays, and smoke tanks (not depicted)), and printers (not depicted).

[0308] Computer systems (2900) may also include human-accessible storage devices and associated media such as optical media including CD / DVD ROM / RW (2920) with CD / DVD or similar media (2921), thumb drives (2922), removable hard drives or solid-state drives (2923), legacy magnetic media such as tapes and floppy disks (not described), and specialized ROM / ASIC / PLD-based devices such as security dongles (not described).

[0309] Those skilled in the art should also understand that the term “computer-readable medium” as used in relation to the subject matter currently disclosed does not include transmission media, carrier waves, or other transient signals.

[0310] A computer system (2900) may also include an interface (2954) to one or more communication networks (2955). These networks may be, for example, wireless, wired, or optical. Networks may further be local, wide-area, metropolitan, automotive, and industrial, real-time, or latency-tolerant. Examples of networks include local area networks such as Ethernet and wireless LANs; cellular networks such as GSM, 3G, 4G, 5G, and LTE; wired or wireless wide-area digital networks for television, including cable TV, satellite TV, and terrestrial broadcast TV; and automotive and industrial networks such as CANBus. Certain networks typically require an external network interface adapter attached to a specific general-purpose data port or peripheral bus (2949) (e.g., a USB port on the computer system (2900)), while other networks are typically integrated into the core of the computer system (2900) by being attached to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, a computer system (2900) can communicate with other entities. Such communication may be unidirectional (e.g., broadcast TV), unidirectional (e.g., CANbus to a specific CANbus device), or bidirectional (e.g., with other computer systems using local or wide-area digital networks). Specific protocols and protocol stacks may be used with each of these networks and network interfaces described above.

[0311] The aforementioned human interface devices, human-accessible storage devices, and network interfaces can be attached to the core (2940) of the computer system (2900).

[0312] The core (2940) may include one or more specialized programmable processing units, such as a central processing unit (CPU) (2941), a graphics processing unit (GPU) (2942), a field-programmable gate area (FPGA) (2943), a hardware accelerator for a specific task (2944), or a graphics adapter (2950). These devices may be connected via a system bus (2948) along with read-only memory (ROM) (2945), random access memory (2946), and internal mass storage (2947) such as an internal hard drive or SSD that is not accessible to the user. In some computer systems, the system bus (2948) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals may be connected directly to the core's system bus (2948) or via a peripheral bus (2949). For example, a display (2910) may be connected to a graphics adapter (2950). Peripheral bus architectures include PCI, USB, and others.

[0313] The CPU (2941), GPU (2942), FPGA (2943), and accelerator (2944) can, in combination, execute certain instructions that constitute the aforementioned computer code. This computer code can be stored in ROM (2945) or RAM (2946). Transition data can also be stored in RAM (2946), while persistent data can be stored, for example, in internal mass storage (2947). High-speed storage and retrieval of any of the memory devices can be made possible using cache memory, which can be closely associated with one or more CPUs (2941), GPUs (2942), mass storage (2947), ROM (2945), RAM (2946), etc.

[0314] Computer-readable media may contain computer code for performing various computer implementation operations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to persons skilled in computer software technology.

[0315] As an example, and not as an limitation, a computer system having an architecture (2900), specifically a core (2940), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as described above, as well as media associated with specific storage of the core (2940) of a non-transient nature, such as core internal mass storage (2947) or ROM (2945). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2940). The computer-readable media may include one or more memory devices or chips, depending on the specific needs. The software can cause the core (2940), and specifically its processor (including a CPU, GPU, FPGA, etc.), to execute specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (2946) and modifying such data structures according to processes defined by the software. In addition, or as an alternative, a computer system may provide functionality as a result of logic wired or otherwise embodied within a circuit (e.g., an accelerator (2944)) which can operate in place of or in conjunction with software to perform a particular process or a particular part of a particular process described herein. If necessary, a reference to software may encompass logic and vice versa. If necessary, a reference to a computer-readable medium may encompass a circuit (such as an integrated circuit (IC)) that stores software for execution, a circuit that embodies logic for execution, or both. This disclosure encompasses any suitable combination of hardware and software. Note A: Acronym JEM Collaborative Search Model VVC Multipurpose Video Coding BMS Benchmark Set MV motion vector HEVC High Efficiency Video Coding SEI Supplemental and Extended Information VUI (Video User Interface) Video Usability Information GOPs Picture Group TUs Conversion Unit PUs (Prediction Units) CTUs Coding Tree Units CTBs (Coding Tree Blocks) PBs prediction block HRD Virtual Reference Decoder SNR (Signal-to-Noise Ratio) CPUs (Central Processing Units) GPUs (Graphics Processing Units) CRT cathode ray tube LCD liquid crystal display OLED (Organic Light-Emitting Diode) CD Compact Disc DVD Digital Video Disc ROM (Read-Only Memory) RAM (Random Access Memory) ASIC (Application-Specific Integrated Circuit) PLD (Programmable Logical Device) LAN (Local Area Network) GSM (Global System for Mobile Communications) LTE Long-Term Evolution CANBus Controller Area Network Bus USB Universal Serial Bus PCI peripheral component interconnection FPGA Field Programmable Gate Area SSD Solid State Drive IC Integrated Circuit CU Coding Unit NIC Neural Image Compression RD Rate Distortion E2E (End-to-End) ANN (Artificial Neural Network) DNN (Deep Neural Network) CNN (Convolutional Neural Network)

[0316] While this disclosure describes several exemplary embodiments, there are many variations, substitutions, and alternative equivalents that fall within the scope of this disclosure. Those skilled in the art will understand that, as such, numerous systems and methods can be devised that embody the principles of this disclosure and thus fall within its spirit and scope, although these are not expressly illustrated or described herein. [Explanation of Symbols]

[0317] 101 samples 102 Arrow 103 Arrow 104 square blocks 110 Schematic Diagram 201 Currently blocked 202 Surrounding sample 203 Surrounding sample 204 Ambient Sample 205 Surrounding sample 206 Surrounding sample 300 Communication Systems 310 Terminal devices 320 terminal devices 330 terminal devices 340 terminal devices 350 Communication Networks 400 Communication Systems 401 Video Source 402 Video Picture Stream 403 Video Encoder 404 encoded video data 405 Streaming Server 406 Client Subsystem 407 Copy, Encode video data 408 Client Subsystem 409 copies, encoded video data 410 Video Decoder 411 Video Picture Output Stream 412 displays 413 Capture Subsystem 420 Electronic Devices 430 Electronic Devices 501 Channel 510 Video Decoder 512 Rendering Devices 515 buffer memory 520 Parser 521 Symbols 530 Electronic Devices 531 Receiver 551 Scaler / Inverse Unit 552 Intrapicture Prediction Units 553 Motion Compensation Prediction Unit 555 Aggregator 556 Loop Filter Unit 557 Reference Picture Memory 558 Current picture buffer 601 Video Sources 603 Video Encoder 620 Electronic Devices 630 Source Coder 632 Coding Engine 633 Local Decoder 634 Reference Picture Memory 635 Predictor 640 Transmitter 643 coded video sequences 645 Entropy Coder 650 Controller 660 communication channels 703 Video Encoder 721 General-purpose controller 722 Intra Encoders 723 Residual Calculator 724 Residual Encoder 725 Entropy Encoder 726 switches 728 Residual Decoder 730 Interencoder 810 Video Decoder 871 Entropy Decoder 872 Intra Decoder 873 Residual Decoder 874 Recovery Module 880 Interdecoder 900 NIC framework 911 Main Encoder Network 912 Quantizer 913 Entropy Encoder 914 Entropy Decoder 915 Main Decoder Network 916 Context Model NN 917 Entropy parameter NN 921 Hyper Encoder 922 Quantizer 923 Entropy Encoder 924 Entropy Decoder 925 Hyper Decoder 931 encoded signal 932 encoding bits 951 First sub-NN 952 Second sub-NN 980 images 981 blocks 982 blocks 983 blocks 984 blocks 985 blocks 986 blocks 987 blocks 988 blocks 989 blocks 990 blocks 991 blocks 992 blocks 993 blocks 994 blocks 995 blocks 996 blocks 1600A Video Encoder 1600B Video Decoder 1700 video encoders 1800 Video Decoder 1910 Post-filtering module 1980 Image 1981 Block 1982 Block 1983 Block 1984 Block 1985 Block 1986 Block 1987 Block 1988 Block 1989 Block 1990 Block 1991 Block 1992 Block 1993 Block 1994 Block 1995 Block 1996 Block 2000A Video Decoder 2000B Video Decoder 2100 Deblocking Process 2101 Images 2111 blocks 2112 blocks 2113 blocks 2114 blocks 2121 Non-boundary area 2121' Non-boundary area 2121'' non-boundary area 2122 Non-boundary area 2122' Non-boundary area 2122'' non-boundary area 2123 Non-boundary area 2123' Non-boundary area 2123'' non-boundary area 2124 Non-boundary area 2124' non-boundary area 2124'' non-boundary area 2130 Deblocking NN 2141 Shared Boundary 2142 Shared Boundary 2143 Shared Boundary 2144 Shared Boundary 2150 images 2250 images 2300 Deblocking Process 2310 Classification Module 2330 Deblocking NN 2350 images 2400 extended processes 2430 Extended NN 2450 images 2500 expansion processes 2510 Classification Module 2530 Extended NN 2550 images 2600 Expansion Processes 2621 Extended non-boundary region 2622 Extended non-boundary region 2623 Extended non-boundary region 2624 Extended non-boundary regions 2630 Expansion Module 2650 expanded images 2700 processes 2800 processes 2900 Computer Systems 2901 Keyboard 2902 Mouse 2903 Trackpad 2905 Joystick 2906 Microphone 2907 Scanner 2908 Camera 2909 Speaker 2910 Touchscreen 2920 CD / DVD ROM / RW 2921 CD / DVD or similar media 2922 Thumb Drive 2923 Removable hard drive or solid state drive 2940 cores 2941 Central Processing Unit (CPU) 2942 Graphics Processing Unit (GPU) 2943 Field-Programmable Gate Area (FPGA) 2944 Hardware Accelerator 2945 Read-only memory (ROM) 2946 Random Access Memory (RAM) 2947 Internal large-capacity storage 2948 System Bus 2949 Local buses 2950 Graphics Adapter 2954 Interface 2955 Communication Network

Claims

1. A method for video decoding performed by a video decoder, The steps include receiving encoded video data which includes an image or video containing one or more blocks, A step of decoding a first post-filtering parameter corresponding to a first block among the one or more blocks, wherein the first post-filtering parameter is a parameter of a post-filtering NN trained to reduce rate distortion loss determined based on the distortion loss of the post-filtering signal, the rate loss of the encoded signal, and the rate loss of the first post-filtering parameter, and the first post-filtering parameter is updated before encoding the encoded video data by a post-filtering module implemented using a post-filtering neural network (NN) trained on a training dataset. A step of determining the post-filtering NN in the video decoder corresponding to the first block based on the first post-filtering parameters, The steps include decoding the first block based on the determined post-filtering NN and The steps include decoding a second post-filtering parameter corresponding to a second block among the one or more blocks, The steps include determining the post-filtering NN based on the second post-filtering parameter and Includes, The second post-filtering parameter is applied to the second block, The second block differs from the first block in that, The second post-filtering parameter is updated before encoding the encoded video data by the post-filtering module implemented using the post-filtering NN, The first post-filtering parameter and the second post-filtering parameter are encoded by different compression methods selected based on the rate distortion loss of the first post-filtering parameter and the second post-filtering parameter. The first post-filtering parameter and the second post-filtering parameter are decompressed based on a decompression method selected based on the different compression method. method.

2. The first post-filtering parameter corresponds to a second image different from the first image, The method further includes the step of decoding the second image based on the determined post-filtering NN, The method according to claim 1.

3. The method according to claim 1, wherein the first post-filtering parameter is different from the second post-filtering parameter, the first post-filtering parameter is adaptable to the content of the first block, and the second post-filtering parameter is adaptable to the content of the second block.

4. The method according to claim 1, wherein the first post-filtering parameter is updated before encoding the encoded video data based on a bias term or weight coefficient in the post-filtering neural network.

5. The post-filtering neural network is configured with initial parameters, The step of determining the post-filtering NN includes updating at least one of the initial parameters with the first post-filtering parameter, The method according to claim 1.

6. The coding information corresponding to the first block indicates the difference between the first post-filtering parameter and at least one of the initial parameters. The method further includes the step of determining the first post-filtering parameter according to the sum of the difference and at least one of the initial parameters, The method according to claim 5.

7. The method according to claim 1, wherein the first post-filtering parameter is updated before encoding the encoded video data in (i) a single layer of the post-filtering NN, (ii) multiple layers of the post-filtering NN, or (iii) all layers of the post-filtering NN.

8. The method according to claim 1, wherein the number of layers in the post-filtering NN depends on the step size or number of steps corresponding to one or more blocks, the step size indicates the learning rate of the post-filtering training process, the number of steps indicates the maximum number of iterations, and the post-filtering training process terminates when the number of iterations reaches the maximum number of iterations.

9. An apparatus configured to perform the method described in any one of claims 1 to 8.

10. A computer program that causes at least one processor to perform the method described in any one of claims 1 to 8.

Images