Method and device for video decoding, and storage medium

Abstract

Aspects of the present disclosure provide a method, an apparatus, and a non-transient computer-readable storage medium for video decoding. The apparatus may include a processing circuit. The processing circuit is configured to decode first neural network update information in a coded bitstream for a first neural network in a video decoder. The first neural network consists of first pre-trained parameters. The first neural network update information corresponds to a first block in an image to be reconstructed and indicates a first alternative parameter corresponding to the first pre-trained parameter among the first pre-trained parameters. The processing circuit is configured to update the first neural network in the video decoder based on the first alternative parameter. The processing circuit may decode the first block based on the updated first neural network for the first block.

Description

Technology Field

[0001] The present application claims the benefit of priority to U.S. Provisional Application No. 63 / 182,366, "Block-wise Content-Adaptive Online Training in Neural Image Compression," filed on April 30, 2021, and the benefit of priority to U.S. Patent Application No. 17 / 729,978, "BLOCK-WISE CONTENT-ADAPTIVE ONLINE TRAINING IN NEURAL IMAGE," filed on April 26, 2022.

[0002] The present disclosure describes embodiments generally related to video coding. Background Technology

[0003] The background description provided in this specification is intended to provide a general context for the present disclosure. The research of the currently registered inventors and any modes of description described in the background section that may not otherwise be considered prior art at the time of filing are not recognized as prior art with respect to the present disclosure, either explicitly or implicitly.

[0004] Video coding and decoding can be performed using inter-picture prediction accompanied by motion compensation. Uncompressed digital images and / or videos may contain a series of pictures, each having, for example, a spatial dimension of 1920x1080 luminance samples and associated chrominance samples. This series of pictures may have a fixed or variable picture rate (informally also known as the frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed images and / or videos have specific bitrate requirements. For example, 1080p60 4:2:0 video at 8 bits per sample (1920x1080 luminance sample resolution at a 60 Hz frame rate) requires bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 gigabytes of storage space.

[0005] One objective of video coding and decoding may be to reduce redundancy in input image and / or video signals through compression. Compression can help reduce the previously described bandwidth and / or storage requirements by more than two orders of magnitude in some cases. Although the descriptions in this specification use video encoding / decoding as exemplary examples, the same techniques may be applied to image encoding / decoding in a similar manner without departing from the spirit of this disclosure. Both lossless compression and lossy compression, as well as combinations thereof, may be employed. Lossless compression refers to techniques in which an exact copy of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough that the reconstructed signal is useful for the intended application. For video, lossy compression is widely employed. The amount of acceptable distortion depends on the application; For example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio may reflect the fact that higher acceptable distortion can yield a higher compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transformation, quantization, and entropy coding.

[0007] Video codec technology may include a technique known as intra-coding. In intra-coding, sample values ​​are represented without referencing samples from previously reconstructed reference pictures or other data. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture may be an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as a first picture or a still image in a coded video bitstream and video session. Samples in an intra-block may be exposed to transformation, and transformation coefficients may be quantized before entropy coding. Intra-prediction may be a technique that minimizes sample values ​​in a pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficients, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0008] For example, traditional intra-coding, such as that known from MPEG-2 generation coding technology, does not use intra-prediction. However, some newer video compression techniques include methods that attempt to use surrounding sample data and / or metadata acquired during the encoding / decoding of spatially adjacent and preceding blocks of data in the decoding order, for example. These techniques are hereinafter referred to as "intra-prediction" techniques. Note that in at least some cases, intra-prediction uses only reference data from the current picture being reconstructed, rather than from reference pictures.

[0009] There may be many different forms of intra-prediction. If two or more of these techniques can be used in a given video coding technique, the technique in use can be coded in an intra-prediction mode. In certain cases, modes may have submodes and / or parameters, which can be coded individually or included in a mode codeword. The codeword used for a given combination of mode, submode, and / or parameter can affect the coding efficiency gain through intra-prediction and can therefore be used as an entropy coding technique to convert codewords into a bitstream.

[0010] Specific modes of intra-prediction were introduced with H.264, improved in H.265, and further enhanced in newer coding techniques such as JEM (joint exploration model), VVC (versatile video coding), and BMS (benchmark set). Predictor blocks can be formed using neighbor sample values ​​belonging to already available samples. The sample values ​​of neighbor samples are copied into the predictor block according to the direction. A reference to the direction in use can be encoded in the bitstream or predicted itself.

[0011] Referring to FIG. 1a, a subset of 9 known predictor directions from the 33 possible predictor directions of H.265 (corresponding to 33 angle modes of 35 intra modes) is depicted in the lower right. The point where the arrows converge (101) indicates the predicted sample. The arrows indicate the direction in which the sample is being predicted. For example, arrow (102) indicates that the sample (101) is predicted to the upper right from the sample or samples at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that the sample (101) is predicted to the lower left from the sample or samples at an angle of 22.5 degrees from the horizontal.

[0012] Referring again to FIG. 1a, a square block (104) of 4x4 samples (indicated by a thick dashed line) is depicted in the upper left. The square block (104) contains 16 samples, each labeled "S," with their positions in the Y dimension (e.g., row index) and their positions in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in both the Y and X dimensions of the block (104). Since the block is 4x4 samples in size, S44 is located in the lower right. Additional reference samples following a similar numbering scheme are illustrated. The reference samples are labeled R relative to the block (104), with their Y position (e.g., row index) and X position (column index). In both H.264 and H.265, the predicted samples are adjacent to the block being reconstructed; therefore, negative values ​​do not need to be used.

[0013] Intra-picture prediction can be performed by appropriately copying reference sample values ​​from neighboring samples by the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating a prediction direction corresponding to the arrow (102) for this block—that is, samples are predicted from the prediction sample or samples at an angle of 45 degrees from the horizontal, upward to the right. In that case, samples S41, S32, S23 and S14 are predicted from the same reference sample R05. Subsequently, sample S44 is predicted from reference sample R08.

[0014] In certain cases, particularly when directions cannot be divided uniformly by 45 degrees; the values ​​of multiple reference samples can be combined, for example, through interpolation to calculate the reference samples.

[0015] As video coding technology has developed, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013), and at the time of this disclosure, JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, and specific techniques in entropy coding are used to represent these likely directions with a small number of bits, allowing for specific penalties for less likely directions. Additionally, sometimes the directions themselves can be predicted from neighboring directions used in already decoded blocks.

[0016] FIG. 1b shows a schematic diagram (110) depicting 65 intra-predicted directions according to JEM to illustrate a number of predicted directions that increase over time.

[0017] The mapping of intra-predicted direction bits in a coded video bitstream representing direction can vary depending on the video coding technique; for example, it can range from simple direct mappings of predicted directions to intra-predicted modes to complex adaptive methods involving codewords, most probable modes, and similar techniques. However, in all cases, there may be certain directions that are statistically less likely to occur in the video content than other specific directions. Since the goal of video compression is to reduce redundancy, in a well-functioning video coding technique, these less likely directions will be represented by a greater number of bits than the more likely directions.

[0018] Motion compensation may be a lossy compression technique and may relate to techniques used for predicting a newly reconstructed picture or part of a picture, after a block of sample data from a previously reconstructed picture or part thereof (reference picture) has been spatially shifted in the direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture may be identical to the picture currently being reconstructed. MVs may have two dimensions X and Y, or three dimensions, the third being an indication of the reference picture in use (the latter may, indirectly, be a time dimension).

[0019] In some video compression techniques, applicable MVs for a specific region of sample data can be predicted from other MVs, for example, from other regions of sample data spatially adjacent to the region being reconstructed and preceding that MV in the decoding order. By doing so, the amount of data required to code the MV can be substantially reduced, thereby eliminating redundancy and increasing compression. MV prediction can work effectively, for example, when coding an input video signal derived from a camera (known as natural video), because there is a statistical probability that regions larger than the region where a single MV is applicable move in similar directions, and thus, in some cases, can be predicted using similar motion vectors derived from MVs of neighboring regions. As a result, the MV found for a given region becomes similar or identical to the MV predicted from surrounding MVs, which can ultimately be represented after entropy coding using fewer bits than is used when directly coding the MV. In some cases, MV prediction can be an example of lossless compression of signals (i.e., MVs) derived from the original signal (i.e., sample stream). In other cases, MV prediction itself can be lossy, for example, due to rounding errors when calculating the predictor from some surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms proposed by H.265, a technique referred to as "spatial merge" below is described here.

[0021] Referring to FIG. 2, the current block (201) contains samples discovered by the encoder during the motion search process, which are predictable from a previous block of the same size that is spatially shifted. Instead of directly coding the MV, the MV may be derived from metadata associated with one or more reference pictures, for example, from the most recent (in decoding order) reference picture, using an MV associated with any one of five surrounding samples represented as A0, A1, and B0, B1, B2 (202 to 206, respectively). In H.265, the MV prediction may use predictors from the same reference picture used by neighboring blocks. means of solving the problem

[0022] Aspects of the present disclosure provide methods and apparatuses for video encoding / decoding. In some examples, the apparatus for video decoding includes a processing circuit. The processing circuit is configured to decode first neural network update information in a coded bitstream for a first neural network in a video decoder. The first neural network consists of first pre-trained parameters. The first neural network update information corresponds to a first block in an image to be reconstructed and indicates a first alternative parameter corresponding to the first pre-trained parameter among the first pre-trained parameters. The processing circuit can update the first neural network in the video decoder based on the first alternative parameter and decode the first block based on the updated first neural network for the first block.

[0023] In an embodiment, the first neural network update information additionally indicates one or more alternative parameters for one or more remaining neural networks in a video decoder. The processing circuit can update one or more remaining neural networks based on one or more alternative parameters.

[0024] In an embodiment, the processing circuit decodes second neural network update information in a coded bitstream for a second neural network in a video decoder. The second neural network consists of second pre-trained parameters. The second neural network update information corresponds to a second block in the image to be reconstructed and indicates a second alternative parameter corresponding to the second pre-trained parameter among the second pre-trained parameters. In the example, the second neural network is different from the first neural network. The processing circuit can update the second neural network in the video decoder based on the second alternative parameter and decode the second block based on the updated second neural network for the second block.

[0025] In the embodiment, the first pre-trained parameter is one of the pre-trained weight coefficients and the pre-trained bias term.

[0026] In the embodiment, the second pre-trained parameter is the other of the pre-trained weight coefficient and the pre-trained bias term.

[0027] In an embodiment, the processing circuit decodes a second block in a bitstream coded based on an updated first neural network for a first block.

[0028] In an embodiment, the first neural network update information indicates the difference between the first surrogate parameter and the first pre-trained parameter. The processing circuit determines the first surrogate parameter based on the sum of the difference and the first pre-trained parameter.

[0029] In an embodiment, the processing circuit decodes the first neural network update information based on one of the Lempel-Ziv-Markov chain algorithm (LZMA2) and the bzip2 algorithm.

[0030] In the example, the processing circuit decodes the second neural network update information based on the other of the LZMA2 and bzip2 algorithms.

[0031] Aspects of the present disclosure also provide a non-transient computer-readable storage medium storing a program executable by at least one processor for performing methods for video decoding. Brief explanation of the drawing

[0032] Additional features, nature, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. Figure 1a is a schematic example of an exemplary subset of intra prediction modes. Figure 1b is an example of exemplary intra-predicted directions. FIG. 2 illustrates the current block (201) and surrounding samples according to an embodiment. FIG. 3 is a schematic example of a simplified block diagram of a communication system (300) according to an embodiment. FIG. 4 is a schematic example of a simplified block diagram of a communication system (400) according to an embodiment. FIG. 5 is a schematic example of a simplified block diagram of a decoder according to an embodiment. FIG. 6 is a schematic example of a simplified block diagram of an encoder according to an embodiment. FIG. 7 illustrates a block diagram of an encoder according to another embodiment. FIG. 8 illustrates a block diagram of a decoder according to another embodiment. FIG. 9a illustrates an example of block-based image coding according to an embodiment of the present disclosure. FIG. 9b illustrates an exemplary NIC framework according to an embodiment of the present disclosure. FIG. 10 illustrates an exemplary convolutional neural network (CNN) of a main encoder network according to an embodiment of the present disclosure. FIG. 11 illustrates an exemplary CNN of a main decoder network according to an embodiment of the present disclosure. FIG. 12 illustrates an exemplary CNN of a hyperencoder according to an embodiment of the present disclosure. FIG. 13 illustrates an exemplary CNN of a hyperdecoder according to an embodiment of the present disclosure. FIG. 14 illustrates an exemplary CNN of a context model network according to an embodiment of the present disclosure. FIG. 15 illustrates an exemplary CNN of an entropy parameter network according to an embodiment of the present disclosure. FIG. 16a illustrates an exemplary video encoder according to an embodiment of the present disclosure. FIG. 16b illustrates an exemplary video decoder according to an embodiment of the present disclosure. FIG. 17 illustrates an exemplary video encoder according to an embodiment of the present disclosure. FIG. 18 illustrates an exemplary video decoder according to an embodiment of the present disclosure. FIG. 19 illustrates a flowchart outlining a process according to an embodiment of the present disclosure. FIG. 20 illustrates a flowchart outlining a process according to an embodiment of the present disclosure. FIG. 21 is a schematic example of a computer system according to an embodiment. Specific details for implementing the invention

[0033] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices capable of communicating with each other, for example, through a network (350). For example, the communication system (300) includes a first pair of terminal devices (310 and 320) interconnected through the network (350). In the example of FIG. 3, the first pair of terminal devices (310 and 320) perform unidirectional transmission of data. For example, a terminal device (310) may encode video data (for example, a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) through the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) receives coded video data from the network (350), decodes the coded video data to recover video pictures, and can display the video pictures according to the recovered video data. Unidirectional data transmission may be common in media serving applications, etc.

[0034] In another example, the communication system (300) includes a second pair of terminal devices (330 and 340) that perform bidirectional transmission of coded video data that may occur, for example, during videoconferencing. For bidirectional transmission of data, in one example, each terminal device among the terminal devices (330 and 340) may code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to another terminal device among the terminal devices (330 and 340) via a network (350). Each terminal device among the terminal devices (330 and 340) may also receive coded video data transmitted by another terminal device among the terminal devices (330 and 340), decode the coded video data to recover video pictures, and display the video pictures on an accessible display device according to the recovered video data.

[0035] In the example of FIG. 3, the terminal devices (310, 320, 330, and 340) may be exemplified as servers, personal computers, and smartphones, but the principles of the present disclosure are not so limited. Embodiments of the present disclosure find applications using laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (350) represents any number of networks that transmit coded video data between the terminal devices (310, 320, 330, and 340), including, for example, wireline and / or wireless communication networks. The communication network (350) may exchange data on circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) may not be important to the operation of the present disclosure unless otherwise described in the specification below.

[0036] FIG. 4 illustrates the arrangement of a video encoder and a video decoder in a streaming environment as an example of an application for the disclosed subject. The disclosed subject may be equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0037] A streaming system may include a capture subsystem (413) that may include, for example, a digital camera, a video source (401) that generates a stream (402) of uncompressed video pictures. In one example, the stream (402) of video pictures includes samples captured by the digital camera. The stream (402) of video pictures, depicted in bold lines to emphasize the large data volume compared to encoded video data (404) (or coded video bitstream), may be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement the embodiments of the disclosed subject matter as described in more detail below. Encoded video data (404) (or encoded video bitstream (404)), depicted as a thin line to emphasize the small data volume compared to the stream (402) of video pictures, may be stored on a streaming server (405) for future use. One or more streaming client subsystems, such as the client subsystems (406 and 408) in FIG. 4, may access the streaming server (405) to retrieve copies (407 and 409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) within an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and generates an outgoing stream (411) of video pictures that can be rendered on a display (412) (e.g., a display screen) or another rendering device (not depicted). In some streaming systems, encoded video data (404, 407, and 409) (e.g., video bitstreams) may be encoded according to specific video coding / compression standards.Examples of such standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject may be used in the context of VVC.

[0038] Note that the electronic devices (420 and 430) may include other components (not shown). For example, the electronic device (420) may also include a video decoder (not shown) and the electronic device (430) may also include a video encoder (not shown).

[0039] FIG. 5 illustrates a block diagram of a video decoder (510) according to an embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used instead of the video decoder (410) in the example of FIG. 4.

[0040] A receiver (531) may receive one or more coded video sequences to be decoded by a video decoder (510); in the same or other embodiments, may receive one coded video sequence at a time, wherein the decoding of each coded video sequence is independent of other coded video sequences. A coded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (531) may receive the encoded video data along with other data, e.g., coded audio data and / or auxiliary data streams, which may be forwarded to their respective use entities (not described). The receiver (531) may separate the coded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be combined between the receiver (531) and the entropy decoder / parser (520) (hereinafter "parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). In others, it may be outside the video decoder (510) (not described). In yet others, for example, to prevent network jitter, there may be a buffer memory (not described) outside the video decoder (510), and additionally, for example, to handle playout timing, there may be another buffer memory (515) inside the video decoder (510). When the receiver (531) is receiving data from a storage / forward device or from an isosynchronous network that has sufficient bandwidth and controllability, the buffer memory (515) may not be needed or may be small.For use on best-effort packet networks such as the Internet, a buffer memory (515) may be required, may be relatively large, advantageously adaptive in size, and may be implemented at least partially in an operating system or similar elements (not shown) outside the video decoder (510).

[0041] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from a coded video sequence. The categories of said symbols include information used to manage the operation of the video decoder (510), and potentially, as illustrated in FIG. 5, information for controlling a rendering device (512) (e.g., a display screen) that is not an integral part of the electronic device (530) but can be coupled to the electronic device (530). The control information for the rendering device(s) may be in the form of Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow various principles including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser (520) may extract a set of subgroup parameters for at least one of the subgroups of pixels in a video decoder from the coded video sequence based on at least one parameter corresponding to a group. The subgroups may include Group of Pictures (GOPs), pictures, tiles, slices, macroblocks, Coding Units (CUs), blocks, Transform Units (TUs), Prediction Units (PUs), etc. The parser (520) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the coded video sequence.

[0042] The parser (520) can generate symbols (521) by performing an entropy decoding / parsing operation on a video sequence received from a buffer memory (515).

[0043] The reconstruction of the symbols (521) may involve a number of different units depending on the type of the coded video picture or its parts (e.g., inter- and intra-picture, inter- and intra-block), and other factors. Which units are involved, and the manner in which, may be controlled by subgroup control information parsed from the video sequence coded by the parser (520). The flow of this subgroup control information between the parser (520) and the number of units below is not described for clarity.

[0044] In addition to the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into a number of functional units as described below. In actual implementations operating under commercial constraints, many of these units may interact closely with one another and be at least partially integrated with one another. However, for the purpose of illustrating the disclosed subject, the conceptual subdivision into the functional units below is appropriate.

[0045] The first unit is a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives quantized transform coefficients as well as control information as symbol(s) (521) from the parser (520), and the control information includes which transform to use, block size, quantization factor, quantization scaling matrices, etc. The scaler / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0046] In some cases, the output samples of the scaler / inverse transform (551) may relate to an intra-coded block; that is, a block that uses prediction information from previously reconstructed parts of the current picture rather than prediction information from previously reconstructed pictures. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses surrounding already reconstructed information fetched from the current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (555) adds the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0047] In other cases, the output samples of the scaler / inverse unit (551) may be intercoded and potentially associated with a motion-compensated block. In this case, the motion compensation prediction unit (553) may access the reference picture memory (557) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (521) associated with the block, these samples may be added by the aggregator (555) to the output of the scaler / inverse unit (551) (in this case, called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) where the motion compensation prediction unit (553) fetches prediction samples may be controlled by motion vectors available to the motion compensation prediction unit (553) in the form of symbols (521) that may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from reference picture memory (557) when subsample accurate motion vectors are in use, motion vector prediction mechanisms, etc.

[0048] Various loop filtering techniques within the loop filter unit (556) may be performed on the output samples of the aggregator (555). Video compression techniques may include in-loop filter techniques that are made available to the loop filter unit (556) as symbols (521) from the parser (520) and are controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream), but which respond not only to meta-information acquired during the decoding of the previous (in decoding order) parts of the coded picture or coded video sequence, but also to previously reconstructed and loop-filtered sample values.

[0049] The output of the loop filter unit (556) may be a sample stream that is not only output to the rendering device (512) but may also be stored in the reference picture memory (557) for use in future inter-picture predictions.

[0050] Certain coded pictures, when fully reconstructed, can be used as reference pictures for future prediction. For example, when a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before starting the reconstruction of the next coded picture.

[0051] A video decoder (510) can perform decoding operations according to a predetermined video compression technique in a standard such as ITU-T Rec. H.265. In that the coded video sequence adheres to both the syntax of the video compression technique or standard, or the profiles documented in the video compression technique or standard, the coded video sequence may comply with the syntax specified by the video compression technique or standard in use. Specifically, the profile may select specific tools as dedicated tools available for use under that profile from all tools available in the video compression technique or standard. Additionally, for compliance, it may be necessary for the complexity of the coded video sequence to be within the boundaries defined by the levels of the video compression technique or standard. In some cases, the levels limit the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the levels may, in some cases, be further restricted through HRD (Hypothetical Reference Decoder) specifications and metadata for managing HRD buffers signaled in the coded video sequence.

[0052] In an embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. This additional data may be included as part of the encoded video sequence(s). This additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0053] FIG. 6 illustrates a block diagram of a video encoder (603) according to an embodiment of the present disclosure. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) may be used instead of the video encoder (403) in the example of FIG. 4.

[0054] The video encoder (603) can receive video samples from a video source (601) (which is not part of the electronic device (620) in the example of FIG. 6) capable of capturing video image(s) to be encoded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0055] A video source (601) may provide a source video sequence to be coded by a video encoder (603) in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that impart motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, wherein each pixel may contain one or more samples depending on the sampling structure, color space, etc. being used. A person skilled in the art can easily understand the relationship between pixels and samples. The following explanation focuses on the samples.

[0056] According to an embodiment, the video encoder (603) can code and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other time constraints required by the application. Enforcing an appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls other functional units and is functionally coupled to other functional units as described below. The coupling is not described for clarity. Parameters set by the controller (650) may include rate control-related parameters (picture skip, quantizer, lambda values ​​of rate-distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions related to the video encoder (603) optimized for a specific system design.

[0057] In some embodiments, the video encoder (603) is configured to operate in a coding loop. For the sake of oversimplification, in one example, the coding loop may include a source coder (630) (responsible for generating symbols, such as a symbol stream, based on, for example, the input picture to be coded and reference picture(s)), and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in a manner similar to that which the (remote) decoder also generates (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques considered in the disclosed subject). The reconstructed sample stream (sample data) is input into the reference picture memory (634). Because the decoding of the symbol stream produces bit-exact results independently of the decoder location (local or remote), the content within the reference picture memory (634) is also bit-exact between the local encoder and the remote encoder. That is, the prediction part of the encoder “knows” as reference picture samples exactly the same as what the decoder “knows” when using prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift, e.g., when synchronization cannot be maintained due to channel errors) is also used in some related technologies.

[0058] The operation of the “local” decoder (633) may be the same as that of the “remote” decoder, such as the video decoder (510) already described in detail above in relation to FIG. 5. However, referring also briefly to FIG. 5, since symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and the parser (520) may be lossless, the entropy decoding parts of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0059] In the embodiments, the decoder technology, excluding the parsing / entropy decoding present in the decoder, exists in the corresponding encoder in the same or substantially the same functional form. Accordingly, the subject matter of the disclosure focuses on decoder operations. Since the encoder technologies are the inverse of the comprehensively described decoder technologies, their description may be abbreviated. More detailed descriptions in specific areas are provided below.

[0060] During operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes the input picture by referring to one or more previously coded pictures from a video sequence designated as "reference pictures." In this way, the coding engine (632) codes the differences between pixel blocks of the input picture and pixel blocks of the reference picture(s) that can be selected as predictive reference(s) for the input picture.

[0061] The local video decoder (633) can decode the coded video data of pictures that may be designated as reference pictures based on the symbols generated by the source coder (630). The operations of the coding engine (632) may advantageously be lossy processes. If the coded video data can be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may be a replica of the source video sequence, typically having some errors. The local video decoder (633) can replicate the decoding processes that can be performed by the video decoder on the reference pictures and allow the reconstructed reference pictures to be stored in the reference picture cache (634). In this way, the video encoder (603) can locally store copies of reconstructed reference pictures having common content as reconstructed reference pictures to be acquired by the far-end video decoder (without transmission errors).

[0062] The predictor (635) can perform prediction searches for the coding engine (632). That is, for a new picture to be coded, the predictor (635) can search the reference picture memory (634) for specific metadata or sample data (as candidate reference pixel blocks), such as reference picture motion vectors, block shapes, etc., which can serve as appropriate prediction references for the new pictures. The predictor (635) can operate on a sample block-by-pixel block basis to find appropriate prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from a number of reference pictures stored in the reference picture memory (634).

[0063] The controller (650) can manage the coding operation of the source coder (630), including, for example, the settings of parameters and subgroup parameters used to encode video data.

[0064] The outputs of all the aforementioned function units may undergo entropy coding in an entropy coder (645). The entropy coder (645) converts the symbols generated by the various function units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0065] The transmitter (640) may buffer the coded video sequence(s) generated by the entropy coder (645) to prepare for transmission over a communication channel (660), which may be a hardware / software link to a storage device for storing the encoded video data. The transmitter (640) may merge the coded video data from the video coder (603) with other data to be transmitted, e.g., coded audio data and / or auxiliary data streams (sources not shown).

[0066] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a specific coded picture type to each coded picture, which can influence the coding techniques that can be applied to each picture. For example, pictures can often be assigned as one of the following picture types:

[0067] An intra-picture (I-picture) may be one that can be coded and decoded without using any other picture within the sequence as a source of prediction. Some video codecs allow different types of intra-pictures, including, for example, "IDR (Independent Decoder Refresh)" pictures. A person skilled in the art recognizes the corresponding variations of I-pictures and their respective applications and features.

[0068] The predictive picture (P picture) may be coded and decoded using intra prediction or inter prediction using at most one motion vector and reference index to predict sample values ​​of each block.

[0069] A bi-directionally predictive picture (Picture B) may be coded and decoded using intra-prediction or inter-prediction, which uses up to two motion vectors and reference indices to predict sample values ​​for each block. Similarly, multi-prediction pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0070] Source pictures are generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and can be coded on a block-by-block basis. Blocks can be predictively coded by referencing other (already coded) blocks determined by the coding assignment applied to each picture in the blocks. For example, blocks of pictures I can be non-predictively coded, or they can be predictively coded by referencing already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of picture P can be predictively coded via spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks of pictures B can be predictively coded via spatial prediction or temporal prediction by referencing one or two previously coded reference pictures.

[0071] The video encoder (603) can perform coding operations according to a predetermined video coding technology or standard, such as ITU-T Rec. H.265. In these operations, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data can comply with the syntax specified by the video coding technology or standard in use.

[0072] In an embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include this data as part of the encoded video sequence. The additional data may include other forms of redundant data such as time / space / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0073] Video can be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra prediction) uses spatial correlation within a given picture, while inter-picture prediction uses (temporal or other) correlation between pictures. In one example, a specific picture being encoded / decoded, referred to as the current picture, is divided into blocks. When a block within the current picture is similar to a reference block within a previously encoded and still buffered reference picture in the video, the block within the current picture can be encoded by a vector referred to as a motion vector. The motion vector points to the reference block within the reference picture and, if multiple reference pictures are in use, may have a third dimension identifying the reference picture.

[0074] In some embodiments, a bi-prediction technique may be used in inter-picture prediction. According to the bi-prediction technique, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which have a decoding order preceding the current picture in the video (however, in the display order, they may be in the past and future, respectively). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture, and a second motion vector pointing to a second reference block in the second reference picture. A block may be predicted by a combination of the first reference block and the second reference block.

[0075] In addition, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0076] According to some embodiments of the present disclosure, predictions such as inter-picture predictions and intra-picture predictions are performed on a block basis. For example, according to the HEVC standard, a picture within a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs within the picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU contains three coding tree blocks (CTBs), which are one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree split into one or more coding units (CUs). For example, a CTU of 64x64 pixels can be divided into one CU of 64x64 pixels, four CUs of 32x32 pixels, or sixteen CUs of 16x16 pixels. In one example, each CU is analyzed to determine a prediction type for the CU, such as an inter-prediction type or an intra-prediction type. The CU is divided into one or more prediction units (PUs) based on temporal and / or spatial predictability. Typically, each PU includes a luminal prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in coding (encoding / decoding) is performed on a unit of prediction blocks. Using a luminal prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luminal values) for pixels, such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0077] FIG. 7 illustrates a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures, and to encode the processing block into a coded picture that is part of a coded video sequence. In one example, the video encoder (703) is used instead of the video encoder (403) in the example of FIG. 4.

[0078] In the HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block, such as 8x8 samples. The video encoder (703) determines whether the processing block is best coded using an intra mode, an inter mode, or a bidirectional prediction mode, for example, using rate-distortion optimization. When the processing block is coded in intra mode, the video encoder (703) may encode the processing block into a coded picture using an intra prediction technique; and when the processing block is coded in inter mode or bidirectional prediction mode, the video encoder (703) may encode the processing block into a coded picture using an inter prediction or bidirectional prediction technique, respectively. In certain video coding techniques, the merge mode may be an inter-picture prediction submode in which motion vectors are derived from one or more motion vector predictors without the benefit of a coded motion vector component outside the predictors. In certain other video coding techniques, there may be a motion vector component applicable to the target block. In one example, the video encoder (703) includes other components such as a mode determination module (not shown) for determining the mode of the processing blocks.

[0079] In the example of FIG. 7, the video encoder (703) includes an inter encoder (730), an intra encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) combined together as shown in FIG. 7.

[0080] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks within reference pictures (e.g., blocks within previous and later pictures), generate inter-prediction information (e.g., description of redundancy information according to the inter-encoding technique, motion vectors, merge mode information), and calculate inter-prediction results (e.g., predicted blocks) based on the inter-prediction information using any appropriate technique. In some examples, the reference pictures are decoded reference pictures that are decoded based on encoded video information.

[0081] The intra-encoder (722) is configured to receive samples of the current block (e.g., processing block), compare the block with already coded blocks within the same picture in some cases, generate quantized coefficients after transformation, and in some cases also receive intra-prediction information (e.g., intra-prediction direction information according to one or more intra-encoding techniques). In one example, the intra-encoder (722) also calculates intra-prediction results (e.g., prediction blocks) based on reference blocks within the same picture and intra-prediction information.

[0082] A general controller (721) is configured to determine general control data and to control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream; and when the mode is inter mode, the general controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0083] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction results selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) is configured to operate based on the residual data to encode the residual data and generate transformation coefficients. In one example, the residual encoder (724) is configured to convert the residual data from the spatial domain to the frequency domain and generate transformation coefficients. Then, quantization processing is performed on the transformation coefficients to obtain quantized transformation coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transformation and generate decoded residual data. The decoded residual data can be appropriately used by the intra-encoder (722) and the inter-encoder (730). For example, the inter encoder (730) can generate decoded blocks based on decoded residual data and inter prediction information, and the intra encoder (722) can generate decoded blocks based on decoded residual data and intra prediction information. The decoded blocks are appropriately processed to generate decoded pictures, and the decoded pictures are buffered in a memory circuit (not shown) and can be used as reference pictures in some examples.

[0084] The entropy encoder (725) is configured to format the bitstream to include the encoded block. The entropy encoder (725) is configured to include various information according to a suitable standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other appropriate information within the bitstream. According to the subject matter disclosed, it is noted that when coding a block in a merge submode of either the inter mode or the bidirectional prediction mode, residual information is not present.

[0085] FIG. 8 illustrates a drawing of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and to decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) is used instead of the video decoder (410) in the example of FIG. 4.

[0086] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) combined together as shown in FIG. 8.

[0087] The entropy decoder (871) may be configured to reconstruct specific symbols representing syntax elements constituting the coded picture from the coded picture. These symbols may include, for example, prediction information (e.g., intra prediction information or inter prediction information) capable of identifying specific samples or metadata used for prediction by each of the mode in which the block is coded (e.g., intra mode, inter mode, bidirectional prediction mode, the latter two being in a merged submode or other submode), the intra decoder (872), or the inter decoder (880), for example, residual information in the form of quantized transformation coefficients. In one example, when the prediction mode is an inter or bidirectional prediction mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). Inverse quantization may be performed on the residual information and provided to the residual decoder (873).

[0088] The inter decoder (880) is configured to receive inter prediction information and generate inter prediction results based on the inter prediction information.

[0089] The intra decoder (872) is configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0090] The residual decoder (873) is configured to perform inverse quantization to extract dequantized transform coefficients and to process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require specific control information (including quantizer parameters (QP)), and that information may be provided by the entropy decoder (871) (since this may only be low-capacity control information, the data path is not shown).

[0091] The reconstruction module (874) is configured to form a reconstructed block by combining the residuals output by the residual decoder (873) and the prediction results (which, in some cases, are output by the inter or intra prediction module) in the spatial domain, and the reconstructed block may be part of a reconstructed picture, and the reconstructed picture may eventually be part of a reconstructed video. Note that other appropriate actions, such as deblocking, may be performed to improve visual quality.

[0092] It should be noted that the video encoders (403, 603, and 703), and video decoders (410, 510, and 810) can be implemented using any suitable technique. In an embodiment, the video encoders (403, 603, and 703), and video decoders (410, 510, and 810) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403, 603, and 603), and video decoders (410, 510, and 810) can be implemented using one or more processors that execute software instructions.

[0093] The present disclosure describes video coding techniques related to neural image compression techniques and / or neural video compression techniques, such as artificial intelligence (AI)-based neural image compression (NIC). Aspects of the present disclosure include content-adaptive online training in NICs, such as block-based content-adaptive online training NIC methods for end-to-end (E2E) optimized image coding frameworks based on neural networks. Neural networks (NN) may include artificial neural networks (ANNs), such as deep neural networks (DNN), convolutional neural networks (CNN), etc.

[0094] In the embodiments, the related hybrid video codec is difficult to optimize as a whole. For example, an improvement in a single module (e.g., an encoder) in the hybrid video codec may not lead to coding gains in overall performance. In an NN-based video coding framework, different modules can be jointly optimized from input to output to improve a final objective (e.g., rate-distortion performance such as the rate-distortion loss L described in this disclosure) by performing a learning process or a training process (e.g., a machine learning process), thus leading to an E2E optimized NIC.

[0095] An exemplary NIC framework or system may be described as follows. The NIC framework uses an input block x as an input to a neural network encoder (e.g., a neural network-based encoder such as a DNN) to obtain, for example, a compressed representation (e.g., a compact representation) that can be compact for storage and transmission purposes. It can calculate. A neural network decoder (e.g., a decoder based on a neural network such as a DNN) is a compressed representation Using as input, the output block (also referred to as the reconstructed block) It can be reconfigured. In various embodiments, the input block x and the reconfigured block is in the spatial domain and is a compressed representation is in a domain different from the spatial domain. In some examples, the compressed representation It is quantized and entropy-coded.

[0096] In some examples, the NIC framework can use a Variable Autoencoder (VAE) architecture. In a VAE architecture, the neural network encoder can directly use the entire input block x as input to the neural network encoder. The entire input block x is a compressed representation To calculate, it can pass through a set of neural network layers acting as a black box. Compressed representation is the output of the neural network encoder. The neural network decoder is the fully compressed representation It can take as input. Compressed expression is a reconstructed block To calculate, it can pass through a different set of neural network layers acting as another black box. Rate-distortion (RD) loss L( ) is a reconstructed block distortion loss ) and compact expression The trade-off between bit consumption R can be optimized to be achieved using the trade-off hyperparameter λ.

[0097] L( ) = ) + R( ) Mathematical formula 1

[0098] Neural networks (e.g., ANNs) can be trained to perform tasks from examples without task-specific programming. An ANN may consist of connected nodes or artificial neurons. Connections between nodes can transmit signals from a first node to a second node (e.g., a receiving node), and the signals may be modified by weights, which can be represented by weighting coefficients for the connections. The receiving node can generate an output signal by processing the signal(s) from the node(s) transmitting the signal(s) to the receiving node (i.e., input signal(s) for the receiving node) and then applying a function to the input signals. The function may be a linear function. In the example, the output signal is the weighted sum of the input signal(s). In the example, the output signal is further modified by a bias, which can be represented by a bias term, and thus the output signal is the sum of the bias and the weighted sum of the input signal(s). The function may include, for example, a weighted sum of input signal(s) or a non-linear operation on the sum of a weighted sum with a bias. The output signal may be transmitted to node(s) (downstream node(s)) connected to the receiving node. The ANN may be represented or configured by parameters (e.g., weights and / or biases of connections). The weights and / or biases may be obtained by training the ANN with examples in which the weights and / or biases can be iteratively adjusted. A trained ANN composed of determined weights and / or determined biases may be used to perform tasks.

[0099] Nodes within an ANN can be organized into any suitable architecture. In various embodiments, nodes within an ANN are organized into layers including an input layer that receives input signal(s) for the ANN and an output layer that outputs output signal(s) from the ANN. In an embodiment, the ANN further includes layer(s), such as hidden layer(s) between the input layer and the output layer. Different layers may perform different types of transformations on the respective inputs of the different layers. Signals may travel from the input layer to the output layer.

[0100] An ANN having multiple layers between an input layer and an output layer may be referred to as a DNN. In an embodiment, the DNN is a feedforward network in which data flows from the input layer to the output layer without loopback. In an example, the DNN is a fully connected network in which each node in one layer is connected to all nodes in the next layer. In an embodiment, the DNN is a recurrent neural network (RNN) in which data can flow in any direction. In an embodiment, the DNN is a CNN.

[0101] A CNN may include an input layer, an output layer, and hidden layer(s) between the input layer and the output layer. The hidden layer(s) may include convolution layer(s) (e.g., used in an encoder) that perform convolutions such as two-dimensional (2D) convolutions. In an embodiment, the 2D convolution performed in the convolution layer is between a convolution kernel (also referred to as a filter or channel, such as a 5x5 matrix) and an input signal to the convolution layer (e.g., a 2D block, a 2D matrix, such as a 256x256 matrix). In various examples, the dimensions of the convolution kernel (e.g., 5x5) are smaller than the dimensions of the input signal (e.g., 256x256). Therefore, a portion (e.g., a 5x5 area) within an input signal (e.g., a 256x256 matrix) covered by a convolution kernel is smaller than the area of ​​the input signal (e.g., a 256x256 area) and can therefore be referred to as the receiving field of each node in the next layer.

[0102] During convolution, the dot product of the convolution kernel and the corresponding recept field in the input signal is calculated. Thus, each element of the convolution kernel is a weight applied to the corresponding sample in the recept field, and therefore the convolution kernel contains weights. For example, a convolution kernel represented by a 5x5 matrix has 25 weights. In some examples, a bias is applied to the output signal of the convolution layer, and the output signal is based on the sum of the dot product and the bias.

[0103] A convolution kernel can shift along the input signal (e.g., a 2D matrix) by a size referred to as the stride, and thus the convolution operation generates a feature map or activation map (e.g., another 2D matrix), which in turn contributes to the input of the next layer in the CNN. For example, the input signal is a 2D block with 256x256 samples, and the stride is 2 samples (e.g., a stride of 2). For a stride of 2, the convolution kernel shifts by 2 samples along the X direction (e.g., horizontal direction) and / or the Y direction (e.g., vertical direction).

[0104] Multiple convolution kernels can be applied to an input signal in the same convolution layer to generate multiple feature maps, each of which can represent a specific feature of the input signal. Generally, a convolution layer with N channels (i.e., N convolution kernels)—each convolution kernel having MxM samples—and a stride S can be designated as Conv: MxM cN sS. For example, a convolution layer with 192 channels—each convolution kernel having 5x5 samples—and a stride of 2 is designated as Conv: 5x5 c192 s2. The hidden layer(s) may include deconvolution layer(s) that perform deconvolutions such as 2D deconvolution (e.g., used in a decoder). Deconvolution is the inverse of convolution. A deconvolution layer with 192 channels—each deconvolution kernel has 5x5 samples—and a stride of 2 is specified as DeConv: 5x5 c192 s2.

[0105] In various embodiments, CNNs have the following advantages. The number of learnable parameters (i.e., parameters to be trained) within a CNN can be significantly smaller than the number of learnable parameters within a DNN such as a feedforward DNN. In a CNN, a relatively large number of nodes can share the same filter (e.g., the same weights) and the same bias (if a bias is used), and thus the memory footprint can be reduced because a single vector of a single bias and weights can be used across all recept fields that share the same filter. For example, for an input signal with 100x100 samples, a convolution layer with a convolution kernel having 5x5 samples has 25 learnable parameters (e.g., weights). If a bias is used, one channel uses 26 learnable parameters (e.g., 25 weights and one bias). If the convolution layer has N channels, the total number of learnable parameters is 26xN. Meanwhile, for a single fully connected layer within the DNN, 100x100 (i.e., 10,000) weights are used for each node in the next layer. If the next layer has L nodes, the total number of learnable parameters is 10,000xL.

[0106] CNNs may additionally include one or more other layer(s), such as pooling layer(s), fully connected layer(s) capable of connecting all nodes in one layer to all nodes in another layer, and normalization layer(s). The layers within a CNN can be arranged in any appropriate order and any appropriate architecture (e.g., feed-forward architecture, recurrent architecture). In the example, other layer(s), such as pooling layer(s), fully connected layer(s), and normalization layer(s), follow the convolution layer.

[0107] Pooling layers can be used to reduce the dimensions of data by combining the outputs from multiple nodes within one layer into a single node within the next layer. Pooling operations for a pooling layer having a feature map as input are described below. The description can be adapted to suit other input signals. The feature map can be divided into sub-regions (e.g., rectangular sub-regions), and the features in each sub-region can be independently down-sampled (or pooled) to a single value by, for example, taking the average value in average pooling or the maximum value in maximum pooling.

[0108] The pooling layer can perform pooling such as local pooling, global pooling, max pooling, and mean pooling. Pooling is a form of non-linear down-sampling. Local pooling combines a small number of nodes from a feature map (e.g., local clusters of nodes such as 2x2 nodes). Global pooling can combine all nodes from a feature map, for example.

[0109] Pooling layers can reduce the size of the representation and thus reduce the number of parameters, memory footprint, and computational load in CNNs. In the example, the pooling layer is inserted between successive convolutional layers within the CNN. In the example, the pooling layer is followed by an activation function such as a ReLU (rectified linear unit) layer. In the example, the pooling layer is omitted between successive convolutional layers within the CNN.

[0110] The normalization layer can be ReLU, leaky ReLU, GDN (generalized divisive normalization), or inverse GDN (IGDN). ReLU can remove negative values ​​from input signals, such as feature maps, by applying a non-saturating activation function and setting negative values ​​to zero. Leaky ReLU can have a small gradient (e.g., 0.01) for negative values ​​instead of a flat gradient (e.g., 0). Therefore, if the value x is greater than 0, the output from leaky ReLU is x. Otherwise, the output from leaky ReLU is x multiplied by the small gradient (e.g., 0.01). In the example, the gradient is determined before training and is therefore not learned during training.

[0111] In NN-based image compression methods, such as DNN-based or CNN-based image compression methods, instead of directly encoding the entire image, a block-based or block-style coding mechanism can be effective for compressing images in DNN-based video coding standards such as FVC. The entire image may be divided into blocks of the same (or varying) size, and the blocks may be compressed individually. In an embodiment, the image may be divided into blocks of the same or different sizes. Instead of the image, the divided blocks may be compressed. FIG. 9a illustrates an example of block-style image coding according to an embodiment of the present disclosure. An image (980) may be divided into blocks, for example, blocks (981)–(996). Blocks (981)–(996) may be compressed, for example, according to the scanning order. In the example illustrated in FIG. 9a, blocks (981)-(989) are already compressed, and blocks (990)-(996) will be compressed.

[0112] Images can be treated as blocks. In an embodiment, images are compressed without being divided into blocks. The entire image can be an input to an E2E NIC framework.

[0113] FIG. 9b illustrates an exemplary NIC framework (900) (e.g., NIC system) according to an embodiment of the present disclosure. The NIC framework (900) may be based on a neural network such as a DNN and / or a CNN. The NIC framework (900) may be used to compress blocks (e.g., encoding) and to decompress (e.g., decoding or reconstructing) the compressed blocks (e.g., encoded blocks). The NIC framework (900) may include two sub-neural networks, a first sub-NN (951) and a second sub-NN (952), implemented using neural networks.

[0114] The first sub-NN (951) may be similar to an autoencoder, and the compressed block of input block x Generates and compresses the block Block reconstructed by decompressing It can be trained to acquire. The first sub-NN (951) may include a plurality of components (or modules), such as a main encoder neural network (or main encoder network) (911), a quantizer (912), an entropy encoder (913), an entropy decoder (914), and a main decoder neural network (or main encoder network) (915). Referring to FIG. 9, the main encoder network (911) can generate a latent or latent representation y from an input block x (e.g., a block to be compressed or encoded). In the example, the main encoder network (911) is implemented using a CNN. The relationship between the latent representation y and the input block x can be described using Equation 2.

[0115] Mathematical formula 2

[0116] Here, the parameter represents parameters such as weights and biases used in convolution kernels within the main encoder network (911) (where biases are used in the main encoder network (911)).

[0117] The potential representation y is the quantized potential Quantized potential can be quantized using a quantizer (912) to generate. For example, the compressed representation of input block x compressed block (e.g., encoded block) To generate (931), it can be compressed using lossless compression by an entropy encoder (913). The entropy encoder (913) can use entropy coding techniques such as Huffman coding, arithmetic coding, etc. In the example, the entropy encoder (913) uses arithmetic encoding and is an arithmetic encoder. In the example, the encoded block (931) is transmitted as a coded bitstream.

[0118] The encoded block (931) can be decompressed (e.g., entropy decoding) by the entropy decoder (914) to generate an output. The entropy decoder (914) may use entropy coding techniques such as Huffman coding and arithmetic coding that correspond to the entropy encoding techniques used in the entropy encoder (913). In the example, the entropy decoder (914) uses arithmetic decoding and is an arithmetic decoder. In the example, lossless compression is used in the entropy encoder (913) and lossless decompression is used in the entropy decoder (914), noise caused by the transmission of the encoded block (931) can be omitted, and the output from the entropy decoder (914) is a quantized potential am.

[0119] The main decoder network (915) is a reconstructed block Quantized potential to generate It can be decoded. In the example, the main decoder network (915) is implemented using a CNN. Reconstructed block (i.e., the output of the main decoder network (915)) and the quantized potential The relationship between (i.e., the input of the main decoder network (915)) can be described using Equation 3.

[0120] Mathematical formula 3

[0121] Here, the parameter represents parameters such as weights and biases (where biases are used in the main decoder network (915)) used in convolution kernels within the main decoder network (915). Accordingly, the first sub-NN (951) compresses (e.g., encodes) the input block x to obtain an encoded block (931) and decompresses (e.g., decodes) the encoded block (931) to reconstruct a block You can obtain the reconstructed block It may differ from the input block x due to the quantization loss introduced by the quantizer (912).

[0122] The second sub-NN (952) is the quantized potential used in entropy coding An entropy model (e.g., a prior probability model) can be trained on it. Thus, the entropy model may be a conditioned entropy model, e.g., a Gaussian mixture model (GMM), or a Gaussian scale model (GSM) dependent on the input block x. The second sub-NN (952) may include a context model NN (916), an entropy parameter NN (917), a hyperencoder (921), a quantizer (922), an entropy encoder (923), an entropy decoder (924), and a hyperdecoder (925). The entropy model used in the context model NN (916) is an autoregressive model for potentiality (e.g., quantized potentiality ...may be. In the example, the hyperencoder (921), quantizer (922), entropy encoder (923), entropy decoder (924), and hyperdecoder (925) form a hyperneural network (e.g., a hyperprior NN). The hyperneural network can represent information useful for correcting context-based predictions. Data from the context model NN (916) and the hyperneural network can be combined by an entropy parameter NN (917). The entropy parameter NN (917) can generate parameters such as mean and scale parameters for an entropy model, such as a conditional Gaussian entropy model (e.g., GMM).

[0123] Referring to FIG. 9b, at the encoder side, the quantized potential from the quantizer (912) The context model NN (916) is supplied. On the decoder side, the quantized potential from the entropy decoder (914) is fed to the context model NN (916). The context model NN (916) can be implemented using a neural network such as a CNN. The context model NN (916) has quantized potentials available to the context model NN (916). In context Output o based on cm,i Can create. Context may include a quantized potential previously quantized on the encoder side or a quantized potential previously entropy-decoded on the decoder side. Output o cm,i and the input of the context model NN (916) (e.g., The relationship between ) can be described using mathematical formula 4.

[0124] Mathematical formula 4

[0125] Here, the parameter represents parameters such as weights and biases used in convolution kernels within the context model NN (916) (where biases are used in the context model NN (916)).

[0126] Output o from context model NN(916) cm,i and output o from hyperdecoder (925) hc is supplied to the entropy parameter NN(917) and output o ep Generates. The entropy parameter NN (917) can be implemented using a neural network such as a CNN. The output o of the entropy parameter NN (917) ep and inputs (e.g., o cm,i and o hc The relationship between ) can be described using mathematical formula 5.

[0127] Mathematical formula 5

[0128] Here, the parameter represents parameters such as weights and biases used in convolution kernels within the entropy parameter NN (917) (where biases are used in the entropy parameter NN (917)). The output o of the entropy parameter NN (917) ep can be used to determine (e.g., conditioning) the entropy model, and thus the conditioned entropy model is, for example, the output o from the hyperdecoder (925). hc It can depend on the input block x through. In the example, the output o ep It includes parameters such as mean and scale parameters used to condition an entropy model (e.g., GMM). Referring to FIG. 9b, the entropy model (e.g., conditioned entropy model) can be employed by an entropy encoder (913) and an entropy decoder (914) in entropy coding and entropy decoding, respectively.

[0129] The second sub-NN (952) can be described below. Potential y can be fed to a hyperencoder (921) to generate hyperpotential z. In the example, the hyperencoder (921) is implemented using a neural network such as a CNN. The relationship between hyperpotential z and potential y can be described using Equation 6.

[0130] Mathematical formula 6

[0131] Here, the parameter represents parameters such as weights and biases used in convolution kernels within the hyperencoder (921) (where biases are used in the hyperencoder (921)).

[0132] Hyperpotential z is a quantized potential It is quantized by a quantizer (922) to generate quantized potential. For example, additional information such as encoded bits (932) from a hyper-neural network can be generated by compressing using lossless compression by an entropy encoder (923). The entropy encoder (923) can use entropy coding techniques such as Huffman coding, arithmetic coding, etc. In the example, the entropy encoder (923) uses arithmetic encoding and is an arithmetic encoder. In the example, additional information such as encoded bits (932) can be transmitted as a coded bitstream together with, for example, an encoded block (931).

[0133] Additional information, such as encoded bits (932), can be decompressed (e.g., entropy decoding) by an entropy decoder (924) to produce an output. The entropy decoder (924) may use entropy coding techniques such as Huffman coding, arithmetic coding, etc. In an example, the entropy decoder (924) uses arithmetic decoding and is an arithmetic decoder. In an example, lossless compression is used in the entropy encoder (923) and lossless decompression is used in the entropy decoder (924), noise caused by the transmission of additional information, etc., can be omitted, and the output from the entropy decoder (924) is a quantized potential It can be. The hyperdecoder (925) is a quantized potential Decode and output o hc Can generate. Output o hc and quantized potential The relationship between them can be described using mathematical formula 7.

[0134] Mathematical formula 7

[0135] Here, the parameter represents parameters such as weights and biases (where biases are used in the hyperdecoder (925)) used in the convolution kernels within the hyperdecoder (925).

[0136] As described above, the compressed or encoded bits (932) can be added to the coded bitstream as additional information, which enables the entropy decoder (914) to use a conditional entropy model. Thus, the entropy model can be block-dependent and spatially adaptive, and therefore can be more accurate than a fixed entropy model.

[0137] The NIC framework (900) may be suitably adapted, for example, to omit one or more components illustrated in FIG. 9, to modify one or more components illustrated in FIG. 9, and / or to include one or more components not illustrated in FIG. 9. In an example, the NIC framework using a fixed entropy model includes a first sub-NN (951) and does not include a second sub-NN (952). In an example, the NIC framework includes components within the NIC framework (900) excluding the entropy encoder (923) and the entropy decoder (924).

[0138] In an embodiment, one or more components within the NIC framework (900) illustrated in FIG. 9 are implemented using neural network(s) such as CNN(s). Each NN-based component within the NIC framework (e.g., NIC framework (900)) (e.g., main encoder network (911), main decoder network (915), context model NN (916), entropy parameter NN (917), hyperencoder (921), or hyperdecoder (925)) may include any suitable architecture (e.g., may have any suitable combination of layers), may include any suitable type of parameters (e.g., weights, biases, combinations of weights and biases, and / or other), and may include any suitable number of parameters.

[0139] In the embodiment, the main encoder network (911), main decoder network (915), context model NN (916), entropy parameter NN (917), hyperencoder (921) and hyperdecoder (925) are implemented using their respective CNNs.

[0140] FIG. 10 illustrates an exemplary CNN of a main encoder network (911) according to an embodiment of the present disclosure. For example, the main encoder network (911) includes four sets of layers, each set of layers comprising a convolutional layer 5x5 c192 s2 followed by a GDN layer. One or more layers illustrated in FIG. 10 may be modified and / or omitted. Additional layer(s) may be added to the main encoder network (911).

[0141] FIG. 11 illustrates an exemplary CNN of a main decoder network (915) according to an embodiment of the present disclosure. For example, the main decoder network (915) comprises three sets of layers, wherein each set of layers comprises a deconvolution layer 5x5 c192 s2 followed by an IGDN layer. Additionally, a deconvolution layer 5x5 c3 s2 follows the three sets of layers, followed by an IGDN layer. One or more layers illustrated in FIG. 11 may be modified and / or omitted. Additional layer(s) may be added to the main decoder network (915).

[0142] FIG. 12 illustrates an exemplary CNN of a hyperencoder (921) according to an embodiment of the present disclosure. For example, the hyperencoder (921) includes a convolutional layer 3x3 c192 s1 followed by a Leaky ReLU, a convolutional layer 5x5 c192 s2 followed by a Leaky ReLU, and a convolutional layer 5x5 c192 s2. One or more layers illustrated in FIG. 12 may be modified and / or omitted. Additional layer(s) may be added to the hyperencoder (921).

[0143] FIG. 13 illustrates an exemplary CNN of a hyperdecoder (925) according to an embodiment of the present disclosure. For example, the hyperdecoder (925) includes a deconvolution layer 5x5 c192 s2 followed by a Leaky ReLU, a deconvolution layer 5x5 c288 s2 followed by a Leaky ReLU, and a deconvolution layer 3x3 c384 s1. One or more layers illustrated in FIG. 13 may be modified and / or omitted. Additional layer(s) may be added to the hyperencoder (925).

[0144] FIG. 14 illustrates an exemplary CNN of a context model NN (916) according to an embodiment of the present disclosure. For example, since the context model NN (916) includes a masked convolution 5x5 c384 s1 for context prediction, the context in Equation 4 It includes a restricted context (e.g., a 5x5 convolution kernel). The convolution layer of FIG. 14 can be modified. Additional layer(s) can be added to the context model NN (916).

[0145] FIG. 15 illustrates an exemplary CNN of an entropy parameter NN (917) according to an embodiment of the present disclosure. For example, the entropy parameter NN (917) includes a convolutional layer 1x1 c640 s1 followed by a Leaky ReLU, a convolutional layer 1x1 c512 s1 followed by a Leaky ReLU, and a convolutional layer 1x1 c384 s1. One or more layers illustrated in FIG. 15 may be modified and / or omitted. Additional layer(s) may be added to the entropy parameter NN (917).

[0146] The NIC framework (900) may be implemented using a CNN as described with reference to FIGS. 10 through 15. The NIC framework (900) may be appropriately adapted so that one or more components within the NIC framework (900) (e.g., (911), (915), (916), (917), (921), and / or (925)) are implemented using any appropriate type of neural network (e.g., CNNs or non-CNN-based neural networks). One or more other components of the NIC framework (900) may be implemented using neural network(s).

[0147] A NIC framework (900) including neural networks (e.g., CNNs) can be trained to learn the parameters used in the neural network. For example, when CNNs are used, weights and biases used in convolution kernels within the main encoder network (911) (where biases are used in the main encoder network (911)), weights and biases used in convolution kernels within the main decoder network (915) (where biases are used in the main decoder network (915)), weights and biases used in convolution kernels within the hyper-encoder (921) (where biases are used in the hyper-encoder (921)), weights and biases used in convolution kernels within the hyper-decoder (925) (where biases are used in the hyper-decoder (925)), weights and biases used in convolution kernel(s) within the context model NN (916) (where biases are used in the context model NN (916)), and weights and biases used in convolution kernels within the entropy parameter NN (917) (where biases Entropy parameter (when used in NN(917)) such as - The parameters represented by can each be learned in the training process.

[0148] In the example, referring to FIG. 10, the main encoder network (911) includes four convolution layers, each having a 5x5 and 192-channel convolution kernel. Thus, the number of weights used in the convolution kernels within the main encoder network (911) is 19200 (i.e., 4x5x5x192). The parameters used in the main encoder network (911) include 19200 weights and optional biases. Additional parameter(s) may be included when biases and / or additional NN(s) are used in the main encoder network (911).

[0149] Referring to FIG. 9b, the NIC framework (900) includes at least one component or module built on the neural network(s). The at least one component may include one or more of a main encoder network (911), a main decoder network (915), a hyper-encoder (921), a hyper-decoder (925), a context model NN (916), and an entropy parameter NN (917). At least one component may be trained individually. In an example, the training process is used to learn parameters for each component individually. At least one component may be trained jointly as a group. In an example, the training process is used to learn parameters for a subset of at least one component jointly. In an example, the training process is used to learn parameters for all of at least one component, and is therefore referred to as E2E optimization.

[0150] In the training process for one or more components within the NIC framework (900), the weights (or weight coefficients) of one or more components may be initialized. In the example, the weights are initialized based on the corresponding neural network model(s) that have been pre-trained (e.g., DNN models, CNN models). In the example, the weights are initialized by setting the weights to random numbers.

[0151] A set of training blocks may be employed to train one or more components, for example, after the weights have been initialized. A set of training blocks may include any suitable blocks having any suitable size(s). In some examples, a set of training blocks includes blocks from raw images, natural images, computer-generated images, etc., in the spatial domain. In some examples, a set of training blocks includes blocks from residual blocks or residual images containing residual data in the spatial domain. Residual data may be computed by a residual calculator (e.g., residual calculator (723)). In some examples, raw images and / or residual images containing residual data may be used directly to train neural networks within the NIC framework. Thus, raw images, residual images, blocks from raw images, and / or blocks from residual images may be used to train neural networks within the NIC framework.

[0152] For brevity, the following training process is described using a training block as an example. The description can be adapted to suit the training image. A training block t of a set of training blocks may pass through the encoding process of FIG. 9b to generate a compressed representation (e.g., encoded information, e.g., a bitstream). The encoded information is a reconstructed block To calculate and reconstruct it, it can pass through the decoding process described in Fig. 9b.

[0153] In the case of the NIC framework (900), two competing targets, e.g., reconstruction quality and bit consumption, are balanced. Quality loss function (e.g., distortion or distortion loss) ) is reconstruction (e.g., reconstructed block It can be used to indicate reconstruction quality, such as the difference between ) and the original block (e.g., training block t). The rate (or rate loss) R can be used to indicate the bit consumption of the compressed representation. In the example, the rate loss R additionally includes additional information used, for example, to determine the context model.

[0154] In the case of neural image compression, a differentiable approximation of quantization can be used in E2E optimization. In various examples, during the training process of neural network-based image compression, noise injection is used to simulate quantization, so quantization is simulated by noise injection instead of being performed by a quantizer (e.g., quantizer (912)). Thus, training by noise injection can variably approximate the quantization error. A bits per pixel (BPP) estimator can be used to simulate an entropy coder, so entropy coding is simulated by a BPP estimator instead of being performed by an entropy encoder (e.g., (913)) and an entropy decoder (e.g., (914)). Thus, during the training process, the rate loss R in the loss function L shown in Equation 1 can be estimated, for example, based on noise injection and a BPP estimator. Generally, a higher rate R may allow for lower distortion D, and a lower rate R may lead to higher distortion D. Therefore, the trade-off hyperparameter λ in Equation 1 can be used to optimize the joint RD loss L, where L can be optimized as the sum of λD and R. The training process can be used to adjust the parameters of one or more components (e.g., (911) (915)) within the NIC framework (900) so that the joint RD loss L is minimized or optimized. In the example, the trade-off hyperparameter λ can be used to optimize the joint rate-distortion (RD) loss as follows:

[0155] L( ) = λ ) + R + Mathematical formula 8

[0156] Here, E measures the distortion of decoded block residuals compared to the original block residuals before encoding, acting as a normalization loss for the residual encoding / decoding DNNs and the encoding / decoding DNNs. is a hyperparameter used to balance the importance of normalization loss.

[0157] Various models can be used to determine the distortion loss D and rate loss R, and thus to determine the joint RD loss L in Equation 1. In the example, the distortion loss ) is expressed as the peak signal-to-noise ratio (PSNR), a metric based on mean squared error, MS-SSIM (multiscale structural similarity) quality index, and a weighted combination of PSNR and MS-SSIM.

[0158] In the example, the target of the training process is to train an encoding neural network (e.g., an encoding DNN), such as a video encoder to be used on the encoder side, and a decoding neural network (e.g., a decoding DNN), such as a video decoder to be used on the decoder side. In the example, referring to FIG. 9b, the encoding neural network may include a main encoder network (911), a hyper-encoder (921), a hyper-decoder (925), a context model NN (916), and an entropy parameter NN (917). The decoding neural network may include a main decoder network (915), a hyper-decoder (925), a context model NN (916), and an entropy parameter NN (917). The video encoder and / or video decoder may include other component(s) based on the NN(s) and / or not based on the NN(s).

[0159] The NIC framework (e.g., NIC framework (900)) can be trained in an E2E manner. In the example, the encoding neural network and the decoding neural network are jointly updated in the training process based on gradients backpropagated in an E2E manner.

[0160] After the parameters of the neural networks within the NIC framework (900) are trained, one or more components within the NIC framework (900) may be used to encode and / or decode the block. In an embodiment, on the encoder side, the video encoder is configured to encode the input block x into an encoded block (931) to be transmitted as a bitstream. The video encoder may include a plurality of components within the NIC framework (900). In an embodiment, on the decoder side, the corresponding video decoder converts the encoded block (931) in the bitstream into a reconstructed block It is configured to decode. The video decoder may include multiple components within the NIC framework (900).

[0161] In the example, the video encoder includes all components within the NIC framework (900), for example, when content-adaptive online training is employed.

[0162] FIG. 16a illustrates an exemplary video encoder (1600A) according to an embodiment of the present disclosure. The video encoder (1600A) includes a main encoder network (911), a quantizer (912), an entropy encoder (913), and a second sub-NN (952) described with reference to FIG. 9b, and detailed descriptions are omitted for brevity. FIG. 16b illustrates an exemplary video decoder (1600B) according to an embodiment of the present disclosure. The video decoder (1600B) may correspond to the video encoder (1600A). The video decoder (1600B) may include a main decoder network (915), an entropy decoder (914), a context model NN (916), an entropy parameter NN (917), an entropy decoder (924), and a hyper decoder (925). Referring to FIGS. 16a and 16b, at the encoder side, the video encoder (1600A) can generate an encoded block (931) and encoded bits (932) to be transmitted in a bitstream. At the decoder side, the video decoder (1600B) can receive and decode the encoded block (931) and encoded bits (932).

[0163] FIGS. 17 and FIGS. 18 respectively illustrate an exemplary video encoder (1700) and a corresponding video decoder (1800) according to embodiments of the present disclosure. Referring to FIG. 17, the encoder (1700) includes a main encoder network (911), a quantizer (912), and an entropy encoder (913). Examples of the main encoder network (911), the quantizer (912), and the entropy encoder (913) are described with reference to FIG. 9b. Referring to FIG. 18, the video decoder (1800) includes a main decoder network (915) and an entropy decoder (914). Examples of the main decoder network (915) and the entropy decoder (914) are described with reference to FIG. 9b. Referring to FIGS. 17 and 18, a video encoder (1700) can generate an encoded block (931) to be transmitted as a bitstream. A video decoder (1800) can receive and decode the encoded block (931).

[0164] As described above, a NIC framework (900) including a video encoder and a video decoder may be trained based on images and / or blocks within a set of training images. In some examples, one or more blocks to be compressed (e.g., encoded) and / or transmitted have properties that are significantly different from the set of training blocks. Accordingly, encoding and decoding one or more blocks, respectively, using a video encoder and a video decoder trained based on a set of training blocks may result in relatively poor RD loss L (e.g., relatively large distortion and / or relatively large bit rate). Accordingly, aspects of the present disclosure describe a content-adaptive online training method for a NIC, such as a block-based content-adaptive online training method for a NIC.

[0165] In a block-based content-adaptive online training method, input images may be divided into blocks, and one or more of the blocks may be used to update one or more parameters within a pre-trained NIC framework to become one or more surrogate parameters by optimizing rate-distortion performance. Neural network update information representing one or more surrogate parameters or a subset of one or more surrogate parameters may be encoded into a bitstream along with one or more encoded blocks among the blocks. On the decoder side, a video decoder may decode one or more encoded blocks among the blocks and achieve better compression performance using one or more surrogate parameters or a subset of one or more surrogate parameters. The block-based content-adaptive online training method may be used as a preprocessing step (e.g., a pre-encoding step) to boost the compression performance of a pre-trained E2E NIC compression method.

[0166] To distinguish between a training process based on a set of training blocks and a content-adaptive online training process based on one or more blocks to be compressed (e.g., encoded) and / or transmitted, the NIC framework (900), video encoder, and video decoder trained by the set of training blocks are each referred to as the pre-trained NIC framework (900), pre-trained video encoder, and pre-trained video decoder. The parameters in the pre-trained NIC framework (900), pre-trained video encoder, or pre-trained video decoder are each referred to as NIC pre-trained parameters, encoder pre-trained parameters, and decoder pre-trained parameters. In the example, the NIC pre-trained parameters include encoder pre-trained parameters and decoder pre-trained parameters. In the example, the encoder pre-trained parameters and decoder pre-trained parameters do not overlap if none of the encoder pre-trained parameters are included in the decoder pre-trained parameters. For example, the encoder pre-trained parameters in (1700) (e.g., pre-trained parameters in the main encoder network (911)) and the decoder pre-trained parameters in (1800) (e.g., pre-trained parameters in the main decoder network (915)) do not overlap. In the example, the encoder pre-trained parameters and the decoder pre-trained parameters overlap, wherein at least one of the encoder pre-trained parameters is included in the decoder pre-trained parameters. For example, the encoder pre-trained parameters in (1600A) (e.g., pre-trained parameters in the context model NN (916)) and the decoder pre-trained parameters in (1600B) (e.g., pre-trained parameters in the context model NN (916)) overlap.NIC pre-trained parameters can be obtained based on blocks and / or images within a set of training blocks.

[0167] The content-adaptive online training process may be referred to as a fine-tuning process and is described below. One or more of the NIC pre-trained parameters in the pre-trained NIC framework (900) may be further trained (e.g., fine-tuned) based on one or more blocks to be encoded and / or transmitted, wherein one or more blocks may differ from the set of training blocks. One or more of the pre-trained parameters used in the NIC pre-trained parameters may be fine-tuned by optimizing the co-RD loss L based on one or more blocks. One or more pre-trained parameters fine-tuned by one or more blocks are referred to as one or more replacement parameters or one or more fine-tuned parameters. In an embodiment, after one or more of the NIC pre-trained parameters are fine-tuned (e.g., replaced) by one or more replacement parameters, neural network update information is encoded into a bitstream to indicate one or more replacement parameters or a subset of one or more replacement parameters. In one example, the NIC framework (900) is updated (or fine-tuned), where one or more pre-trained parameters are each replaced with one or more alternative parameters.

[0168] In the first scenario, one or more pre-trained parameters include a first subset of one or more pre-trained parameters and a second subset of one or more pre-trained parameters. One or more alternative parameters include a first subset of one or more alternative parameters and a second subset of one or more alternative parameters.

[0169] A first subset of one or more pre-trained parameters is used in a pre-trained video encoder and, for example, is replaced by a first subset of one or more replacement parameters during the training process. Thus, the pre-trained video encoder is updated with a video encoder updated by the training process. The neural network update information may indicate a second subset of one or more replacement parameters to replace a second subset of one or more replacement parameters. One or more blocks may be encoded using the updated video encoder and transmitted as a bitstream along with the neural network update information.

[0170] At the decoder side, a second subset of one or more pre-trained parameters is used in a pre-trained video decoder. In an embodiment, the pre-trained video decoder receives and decodes neural network update information to determine a second subset of one or more replacement parameters. The pre-trained video decoder is updated to an updated video decoder when the second subset of one or more pre-trained parameters in the pre-trained video decoder is replaced by a second subset of one or more replacement parameters. One or more encoded blocks can be decoded using the updated video decoder.

[0171] FIGS. 16a and 16b illustrate an example of a first scenario. For example, one or more pre-trained parameters include N1 pre-trained parameters in a pre-trained context model NN (916) and N2 pre-trained parameters in a pre-trained main decoder network (915). Thus, a first subset of one or more pre-trained parameters includes N1 pre-trained parameters, and a second subset of one or more pre-trained parameters is identical to one or more pre-trained parameters. Thus, the N1 pre-trained parameters in the pre-trained context model NN (916) can be replaced with N1 corresponding replacement parameters so that the pre-trained video encoder (1600A) can be updated to the updated video encoder (1600A). The pre-trained context model NN (916) is also updated to become the updated context model NN (916). At the decoder side, N1 pre-trained parameters can be replaced with N1 corresponding replacement parameters, and N2 pre-trained parameters can be replaced with N2 corresponding replacement parameters, so that the pre-trained context model NN (916) is updated to become the updated context model NN (916), and the pre-trained main decoder network (915) is updated to become the updated main decoder network (915). Thus, the pre-trained video decoder (1600B) can be updated to the updated video decoder (1600B).

[0172] In the second scenario, none of the one or more pre-trained parameters are used in the pre-trained video encoder on the encoder side. Rather, one or more pre-trained parameters are used in the pre-trained video decoder on the decoder side. Therefore, the pre-trained video encoder is not updated and remains a pre-trained video encoder after the training process. In the embodiment, the neural network update information indicates one or more alternative parameters. One or more blocks can be encoded using the pre-trained video encoder and transmitted as a bitstream along with the neural network update information.

[0173] At the decoder side, a pre-trained video decoder can receive and decode neural network update information to determine one or more surrogate parameters. The pre-trained video decoder is updated to an updated video decoder when one or more pre-trained parameters in the pre-trained video decoder are replaced by one or more surrogate parameters. One or more encoded blocks can be decoded using the updated video decoder.

[0174] FIGS. 16a and FIGS. 16b illustrate an example of a second scenario. For example, one or more pre-trained parameters include N2 pre-trained parameters in the pre-trained main decoder network (915). Thus, none of the one or more pre-trained parameters are used in the pre-trained video encoder on the encoder side (e.g., pre-trained video encoder (1600A)). Thus, the pre-trained video encoder (1600A) remains a pre-trained video encoder after the training process. On the decoder side, the N2 pre-trained parameters can be replaced with N2 corresponding replacement parameters that update the pre-trained main decoder network (915) to an updated main decoder network (915). Thus, the pre-trained video decoder (1600B) can be updated to an updated video decoder (1600B).

[0175] In the third scenario, one or more pre-trained parameters are used in a pre-trained video encoder and, for example, are replaced by one or more alternative parameters during the training process. Thus, the pre-trained video encoder is updated with a video encoder updated by the training process. One or more blocks can be encoded using the updated video encoder and transmitted as a bitstream. Neural network update information is not encoded into the bitstream. On the decoder side, the pre-trained video decoder is not updated and remains as a pre-trained video decoder. One or more encoded blocks can be decoded using the pre-trained video decoder.

[0176] FIGS. 16a and FIGS. 16b illustrate an example of a third scenario. For example, one or more pre-trained parameters are in a pre-trained main encoder network (911). Thus, one or more pre-trained parameters in the pre-trained main encoder network (911) can be replaced with one or more alternative parameters so that the pre-trained video encoder (1600A) can be updated to become the updated video encoder (1600A). The pre-trained main encoder network (911) is also updated to become the updated main encoder network (911). On the decoder side, the pre-trained video decoder (1600B) is not updated.

[0177] In various examples as described in the first, second, and third scenarios, video decoding can be performed by pre-trained decoders having different capabilities, including decoders having and not having the capability to update pre-trained parameters.

[0178] In the example, compression performance can be increased by coding one or more blocks with an updated video encoder and / or an updated video decoder compared to coding one or more blocks with a pre-trained video encoder and a pre-trained video decoder. Thus, a content-adaptive online training method can be used to adapt a pre-trained NIC framework (e.g., a pre-trained NIC framework (900)) to target block content (e.g., one or more blocks to be transmitted) and thus fine-tune the pre-trained NIC framework. Thus, the video encoder on the encoder side and / or the video decoder on the decoder side can be updated.

[0179] The content-adaptive online training method can be used as a preprocessing step (e.g., a pre-encoding step) to boost the compression performance of a pre-trained E2E NIC compression method.

[0180] In an embodiment, one or more blocks include a single input block, and the fine-tuning process is performed on the single input block. The NIC framework (900) is trained and updated (e.g., fine-tuned) based on the single input block. An updated video encoder on the encoder side and / or an updated video decoder on the decoder side may be used to code the single input block and optionally other input blocks. Neural network update information may be encoded into a bitstream along with the encoded single input block.

[0181] In an embodiment, one or more blocks include multiple input blocks, and the fine-tuning process is performed with multiple input blocks. The NIC framework (900) is trained and updated (e.g., fine-tuned) based on multiple input blocks. An updated video encoder on the encoder side and / or an updated decoder on the decoder side may be used to code multiple input blocks and optionally other input blocks. Neural network update information may be encoded into a bitstream along with the encoded multiple input blocks.

[0182] Rate loss R can increase depending on the signaling of neural network update information in the bitstream. When one or more blocks contain a single input block, the neural network update information is signaled for each encoded block, and a first increase in rate loss R is used to indicate the increase in rate loss R due to the signaling of neural network update information per block. When one or more blocks contain multiple input blocks, the neural network update information is signaled for multiple input blocks and shared by multiple input blocks, and a second increase in rate loss R is used to indicate the increase in rate loss R due to the signaling of neural network update information per block. Because the neural network update information is shared by multiple input blocks, the second increase in rate loss R may be smaller than the first increase in rate loss R. Therefore, in some examples, it may be advantageous to fine-tune the NIC framework using multiple input blocks.

[0183] In an embodiment, one or more pre-trained parameters to be updated are in one component of the pre-trained NIC framework (900). Accordingly, one component of the pre-trained NIC framework (900) is updated based on one or more alternative parameters, and other components of the pre-trained NIC framework (900) are not updated.

[0184] One component may be a pre-trained context model NN (916), a pre-trained entropy parameter NN (917), a pre-trained main encoder network (911), a pre-trained main decoder network (915), a pre-trained hyper-encoder (921), or a pre-trained hyper-decoder (925). The pre-trained video encoder and / or pre-trained video decoder may be updated depending on which of the components within the pre-trained NIC framework (900) is updated.

[0185] In the example, one or more pre-trained parameters to be updated are in the pre-trained context model NN (916), and thus the pre-trained context model NN (916) is updated and the remaining components (911), (915), (921), (917), and (925) are not updated. In the example, the pre-trained video encoder on the encoder side and the pre-trained video decoder on the decoder side include the pre-trained context model NN (916), and thus both the pre-trained video encoder and the pre-trained video decoder are updated.

[0186] In the example, one or more pre-trained parameters to be updated are in the pre-trained hyperdecoder (925), and thus the pre-trained hyperdecoder (925) is updated, and the remaining components (911), (915), (916), (917), and (921) are not updated. Thus, the pre-trained video encoder is not updated, and the pre-trained video decoder is updated.

[0187] In an embodiment, one or more pre-trained parameters to be updated are in a plurality of components of the pre-trained NIC framework (900). Thus, a plurality of components of the pre-trained NIC framework (900) are updated based on one or more alternative parameters. In an example, a plurality of components of the pre-trained NIC framework (900) include all components composed of neural networks (e.g., DNN, CNN). In an example, a plurality of components of the pre-trained NIC framework (900) include CNN-based components: a pre-trained main encoder network (911), a pre-trained main decoder network (915), a pre-trained context model NN (916), a pre-trained entropy parameter NN (917), a pre-trained hyper-encoder (921), and a pre-trained hyper-decoder (925).

[0188] As described above, in the example, one or more pre-trained parameters to be updated are in the pre-trained video encoder of the pre-trained NIC framework (900). In the example, one or more pre-trained parameters to be updated are in the pre-trained video decoder of the NIC framework (900). In the example, one or more pre-trained parameters to be updated are in the pre-trained video encoder and pre-trained video decoder of the pre-trained NIC framework (900).

[0189] The NIC framework (900) may be based on a neural network, for example, one or more components within the NIC framework (900) may include a neural network such as a CNN, a DNN, etc. As previously mentioned, neural networks may be specified by different types of parameters such as weights, biases, etc. Each neural network-based component within the NIC framework (900) (e.g., context model NN (916), entropy parameter NN (917), main encoder network (911), main decoder network (915), hyperencoder (921), or hyperdecoder (925)) may be composed of appropriate parameters such as weights, biases, or combinations of weights and biases. When CNN(s) are used, the weights may include elements within convolution kernels. One or more types of parameters may be used to specify a neural network. In an embodiment, one or more pre-trained parameters to be updated are bias term(s), and only the bias term(s) are replaced by one or more replacement parameters. In an embodiment, one or more pre-trained parameters to be updated are weights, and only the weights are replaced by one or more replacement parameters. In an embodiment, one or more pre-trained parameters to be updated include weights and bias term(s), and all pre-trained parameters including weights and bias term(s) are replaced by one or more replacement parameters. In an embodiment, other parameters may be used to specify the neural network, and other parameters may be fine-tuned.

[0190] The fine-tuning process may include multiple epochs (e.g., iterations) in which one or more pre-trained parameters are updated in an iterative fine-tuning process. The fine-tuning process may stop when the training loss is flattened or is about to be flattened. In the example, the fine-tuning process stops when the training loss (e.g., RD ​​loss L) is below a first threshold. In the example, the fine-tuning process stops when the difference between two consecutive training losses is below a second threshold.

[0191] Two hyperparameters (e.g., step size and maximum number of steps) can be used in the fine-tuning process along with a loss function (e.g., RD ​​loss L). The maximum number of iterations can be used as a threshold for the maximum number of iterations to terminate the fine-tuning process. In the example, the fine-tuning process stops when the number of iterations reaches the maximum number of iterations.

[0192] The step size can represent the learning rate of an online training process (e.g., an online fine-tuning process). The step size can be used for gradient descent algorithms or backpropagation calculations performed in the fine-tuning process. The step size can be determined using any suitable method.

[0193] The step size for each block within the image may be different. In the embodiment, different step sizes may be assigned to the image to achieve better compression results (e.g., better RD loss L).

[0194] In some examples, a video encoder and a video decoder based on a NIC framework (e.g., NIC framework (900)) can directly encode and decode images. Thus, a block-based content-adaptive online training method can be adapted to update specific parameters in the NIC framework and thus the video encoder and / or video decoder by directly using one or more images. Different images may have different step sizes to achieve optimized compression results.

[0195] In the embodiments, different step sizes are used for blocks having different types of content to achieve optimal results. Different types may refer to different variances. In the example, the step size is determined based on the variance of the block used to update the NIC framework. For example, the step size of a block with high variance is larger than the step size of a block with low variance, where high variance is greater than low variance.

[0196] In the embodiments, the step size is selected based on characteristics of the block or image, such as the RGB dispersion of the block. In the embodiments, the step size is selected based on the RD performance of the block (e.g., RD ​​loss L). Multiple sets of alternative parameter(s) may be generated based on different step sizes, and a set having better compression performance (e.g., smaller RD loss) may be selected.

[0197] In an embodiment, a first step size may be used to execute a specific number of iterations (e.g., 100). Then, a second step size (e.g., a first step size plus or minus increment) may be used to execute a specific number of iterations. The results from the first step size and the second step size may be compared to determine the step size to be used. More than two step sizes may be tested to determine the optimal step size.

[0198] The step size may change during the fine-tuning process. The step size may have an initial value at the start of the fine-tuning process, and the initial value may be reduced (e.g., halved) after a certain number of iterations in later stages of the fine-tuning process to achieve finer tuning. The step size or the learning rate may be changed by the scheduler during iterative online training. The scheduler may include a parameter tuning method used to adjust the step size. The scheduler may determine a value for the step size such that the step size can increase, decrease, or remain constant over multiple intervals. In the example, the learning rate is changed by the scheduler at each step. A single scheduler or multiple different schedulers may be used in different blocks. Thus, multiple sets of alternative parameter(s) may be generated based on multiple schedulers, and one of the multiple sets of alternative parameter(s) having better compression performance (e.g., smaller RD loss) may be selected.

[0199] In the embodiment, multiple learning rate schedules are assigned to different blocks to achieve better compression results. In the embodiment, all blocks within the image share the same learning rate schedule. In the embodiment, the selection of learning rate schedules is based on block characteristics such as the RGB variance of the blocks. In the embodiment, the selection of learning rate schedules is based on the RD performance of the blocks.

[0200] In an embodiment, different blocks may be used to update different parameters in different components within the NIC framework (e.g., context model NN (916) or hyperdecoder (925)). For example, a first block is used to update parameters in the context model NN (916), and a second block is used to update parameters in the hyperdecoder (925).

[0201] In an embodiment, different blocks may be used to update different types of parameters (e.g., biases or weights) in the NIC framework. For example, a first block is used to update at least one bias in one or more neural networks within the NIC framework, and a second block is used to update at least one weight in one or more neural networks within the NIC framework.

[0202] In an embodiment, multiple blocks (e.g., all blocks) within an image update one or more identical parameters.

[0203] In an embodiment, one or more parameters to be updated are selected based on characteristics of the block, such as the RGB variance of the block. In an embodiment, one or more parameters to be updated are selected based on the RD performance of the block.

[0204] At the end of the fine-tuning process, one or more updated parameters may be calculated for each of one or more surrogate parameters. In an embodiment, one or more updated parameters are calculated as the difference between one or more surrogate parameters and one or more corresponding pre-trained parameters. In an embodiment, one or more updated parameters are each of one or more surrogate parameters.

[0205] In the embodiments, one or more updated parameters may be generated from one or more alternative parameters using, for example, a specific linear or non-linear transformation, and one or more updated parameters are representative parameter(s) generated based on one or more alternative parameters. One or more alternative parameters are transformed into one or more updated parameters for better compression.

[0206] A first subset of one or more updated parameters corresponds to a first subset of one or more replacement parameters, and a second subset of one or more updated parameters corresponds to a second subset of one or more replacement parameters.

[0207] In the embodiments, different blocks have different relationships between one or more updated parameters and one or more replacement parameters. For example, for the first block, one or more updated parameters are calculated as the difference between one or more replacement parameters and one or more pre-trained parameters. For the second block, one or more updated parameters are each one or more replacement parameters.

[0208] In the embodiment, multiple blocks (e.g., all blocks) within the image have the same relationship between one or more updated parameters and one or more replacement parameters.

[0209] In the embodiments, the relationship between one or more updated parameters and one or more replacement parameters is selected based on the characteristics of the block, such as the RGB variance of the block. In the embodiments, the relationship between one or more updated parameters and one or more replacement parameters is selected based on the RD performance of the block.

[0210] In the example, one or more updated parameters may be compressed using, for example, LZMA2, a variant of the Lempel-Ziv-Markov chain algorithm (LZMA), the bzip2 algorithm, etc. In the example, compression is omitted for one or more updated parameters. In some embodiments, one or more updated parameters or a second subset of one or more updated parameters may be encoded as a bitstream as neural network update information, wherein the neural network update information represents one or more replacement parameters or a second subset of one or more replacement parameters.

[0211] In the embodiments, compression methods for one or more updated parameters differ for different blocks. For example, for the first block, LZMA2 is used to compress one or more updated parameters, and for the second block, bzip2 is used to compress one or more updated parameters. In the embodiments, the same compression method is used to compress one or more updated parameters for multiple blocks (e.g., all blocks) within the image. In the embodiments, the compression method is selected based on characteristics of the blocks, such as the RGB dispersion of the blocks. In the embodiments, the compression method is selected based on the RD performance of the blocks.

[0212] After the fine-tuning process, in some examples, a pre-trained video encoder on the encoder side may be updated or fine-tuned based on (i) a first subset of one or more alternate parameters or (ii) one or more alternate parameters. An input block (e.g., one of one or more blocks used in the fine-tuning process) may be encoded into a bitstream using the updated video encoder. Thus, the bitstream contains both the encoded block and the neural network update information.

[0213] Where applicable, in the example, neural network update information is decoded (e.g., decompressed) by a pre-trained video decoder to obtain one or more updated parameters or a second subset of one or more updated parameters. In the example, one or more alternative parameters or a second subset of one or more alternative parameters may be obtained based on the relationship between one or more updated parameters and the aforementioned one or more alternative parameters. As described above, the pre-trained video decoder may be fine-tuned, and the decoded updated video may be used to decode an encoded block.

[0214] The NIC framework may include any type of neural network and may use any neural network-based image compression method, such as a context-hyperprior encoder-decoder framework (e.g., the NIC framework shown in FIG. 9b), a scale-hyperprior encoder-decoder framework, a Gaussian mixture likelihood framework and a variation of the Gaussian mixture likelihood framework, an RNN-based recursive compression method and a variation of the RNN-based recursive compression method.

[0215] Compared to related E2E image compression methods, the content-adaptive online training methods and apparatus of the present disclosure may have the following advantages. An adaptive online training mechanism is used to improve NIC coding efficiency. The use of a flexible and general framework can accommodate various types of pre-trained frameworks and quality metrics. For example, specific pre-trained parameters in various types of pre-trained frameworks can be replaced by using online training with blocks to be encoded and transmitted.

[0216] FIG. 19 illustrates a flowchart outlining a process (1900) according to an embodiment of the present disclosure. The process (1900) may be used to encode blocks, such as blocks in a raw image or blocks in a residual image. In various embodiments, the process (1900) is executed by a processing circuit in terminal devices (310), (320), (330), and (340), a processing circuit that performs the functions of a video encoder (1600A), and a processing circuit that performs the functions of a video encoder (1700). In an example, the processing circuit performs (i) one of a video encoder (403), (603), and (703) and (ii) a combination of the functions of one of a video encoder (1600A) and a video encoder (1700). In some embodiments, the process (1900) is implemented with software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit performs the process (1900). The process starts at (S1901). In the example, the NIC framework is based on a neural network. In the example, the NIC framework is the NIC framework (900) described with reference to FIG. 9b. The NIC framework may be based on CNNs such as those described with reference to FIG. 10 through 15. The video encoder (e.g., (1600A) or (1700)) and the corresponding video decoder (e.g., (1600B) or (1800)) may include a number of components within the NIC framework as described above. The NIC framework based on neural networks is pre-trained, and thus the video encoder and video decoder are pre-trained. The process (1900) proceeds to (S1910).

[0217] In (S1910), a fine-tuning process for the NIC framework is performed based on one or more blocks (or input block(s)). The input block(s) may be any suitable block(s) having any suitable size(s). In some examples, the input block(s) include blocks within raw images, natural images, computer-generated images, etc., in the spatial domain.

[0218] In some examples, the input block(s) contain residual data in the spatial domain, calculated, for example, by a residual calculator (e.g., residual calculator (723)). For example, referring to FIGS. 7 and 9, components within various devices can be appropriately combined to achieve (S1910), and the residual block from the residual calculator can be supplied to a main encoder network (911) within the NIC framework.

[0219] One or more parameters (e.g., one or more pre-trained parameters) in one or more neural networks (e.g., one or more pre-trained neural networks) within a NIC framework (e.g., a pre-trained NIC framework) may each be updated with one or more alternative parameters as described above. In an embodiment, one or more parameters in one or more neural networks are updated, for example, at each step during the training process described in (S1910).

[0220] In an embodiment, at least one neural network in a video encoder (e.g., a pre-trained video encoder) is composed of a first subset of one or more pre-trained parameters, and thus at least one neural network in the video encoder can be updated based on a corresponding first subset of one or more alternative parameters. In an example, the first subset of one or more alternative parameters includes all of one or more alternative parameters. In an example, at least one neural network in the video encoder is updated when the first subset of one or more pre-trained parameters is replaced by the first subset of one or more alternative parameters, respectively. In an example, at least one neural network in the video encoder is updated iteratively in a fine-tuning process. In an example, none of one or more pre-trained parameters are included in the video encoder, and thus the video encoder is not updated and retains the pre-trained video encoder.

[0221] In (S1920), one of the one or more blocks may be encoded using a video encoder having at least one updated neural network, wherein the video encoder is composed of a first subset of one or more replacement parameters. In the example, one of the one or more blocks is encoded after at least one neural network in the video encoder has been updated.

[0222] Step (S1920) can be adapted appropriately. For example, the video encoder is not updated when none of the one or more alternative parameters are included in at least one neural network in the video encoder, and thus one or more blocks can be encoded using a pre-trained video encoder (e.g., a video encoder including at least one pre-trained neural network).

[0223] In (S1930), neural network update information representing a second subset of one or more alternative parameters may be encoded into a bitstream. In the example, the second subset of one or more alternative parameters is used to update at least one neural network in a video decoder on the decoder side. Step (S1930) may be omitted, and, for example, if the second subset of one or more alternative parameters does not contain any parameters and no neural network update information is signaled into the bitstream, none of the neural networks in the video decoder are updated.

[0224] In (S1940), a bitstream containing one or more encoded blocks and neural network update information may be transmitted. Step (S1940) can be appropriately adapted. For example, if step (S1930) is omitted, the bitstream does not contain neural network update information. The process (1900) proceeds to (S1999) and terminates.

[0225] The process (1900) can be adapted to suit various scenarios, and the steps in the process (1900) can be adjusted accordingly. One or more steps in the process (1900) may be adapted, omitted, repeated, and / or combined. Any appropriate order may be used to implement the process (1900). Additional step(s) may be added. For example, in addition to encoding one of one or more blocks, one or more blocks are encoded in (S1920) and transmitted in (S1940).

[0226] In some examples of the process (1900), one or more blocks are encoded by an updated video encoder and transmitted as a bitstream. Since the fine-tuning process is based on one or more blocks, the fine-tuning process is based on the context to be encoded and is therefore context-based.

[0227] In some examples, the neural network update information further indicates which parameter(s) are a second subset of one or more pre-trained parameters (or a corresponding second subset of one or more alternative parameters), thereby enabling the corresponding pre-trained parameter(s) in the video decoder to be updated. The neural network update information may indicate component information (e.g., (915)), layer information (e.g., layer 4 DeConv: 5x5 c3 s2), channel information (e.g., second channel) of the second subset of one or more pre-trained parameters. Thus, referring to FIG. 11, the second subset of one or more alternative parameters includes the convolution kernel of the second channel of DeConv: 5x5 c3 s2 in the main decoder network (915). Thus, the convolution kernel of the second channel of DeConv: 5x5 c3 s2 in the pre-trained main decoder network (915) is updated. In some examples, component information of a second subset of one or more pre-trained parameters (e.g., (915)), layer information (e.g., layer 4 DeConv: 5x5 c3 s2), channel information (e.g., second channel), etc. are predetermined and stored in a pre-trained video decoder and are therefore not signaled.

[0228] FIG. 20 illustrates a flowchart outlining a process (2000) according to an embodiment of the present disclosure. The process (2000) may be used for the reconstruction of an encoded block. In various embodiments, the process (2000) is executed by a processing circuit within terminal devices (310), (320), (330), and (340), such as a processing circuit that performs the functions of a video decoder (1600B) and a processing circuit that performs the functions of a video decoder (1800). In an example, the processing circuit performs a combination of functions of (i) one of a video decoder (410), a video decoder (510), and a video decoder (810) and (ii) one of a video decoder (1600B) or a video decoder (1800). In some embodiments, the process (2000) is implemented with software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit performs the process (2000). The process starts at (S2001). In the example, the NIC framework is based on a neural network. In the example, the NIC framework is the NIC framework (900) described with reference to FIG. 9b. The NIC framework may be based on CNNs such as those described with reference to FIG. 10 through 15. The video decoder (e.g., (1600B) or (1800)) may include a number of components within the NIC framework as described above. The NIC framework based on neural networks may be pre-trained. The video decoder may be pre-trained with pre-trained parameters. The process (2000) proceeds to (S2010).

[0229] In (S2010), first neural network update information in the coded bitstream can be decoded. The first neural network update information may be for the first neural network in the video decoder. The first neural network may be composed of first pre-trained parameters. The first neural network update information may correspond to a first block in the image to be reconstructed and may indicate a first replacement parameter corresponding to the first pre-trained parameter among the first pre-trained parameters.

[0230] In the example, the first pre-trained parameter is the pre-trained bias term.

[0231] In the example, the first pre-trained parameter is the pre-trained weight coefficient.

[0232] In an embodiment, the video decoder includes a plurality of neural networks. The first neural network update information may indicate update information for one or more of the remaining neural networks among the plurality of neural networks. For example, the first neural network update information further indicates one or more alternative parameters for one or more of the remaining neural networks among the plurality of neural networks. The one or more alternative parameters correspond to one or more pre-trained parameters for each of the one or more remaining neural networks. In the example, the first pre-trained parameter and each of the one or more pre-trained parameters are each pre-trained bias term. In the example, the first pre-trained parameter and each of the one or more pre-trained parameters are each pre-trained weight coefficients. In the example, the first pre-trained parameter and the one or more pre-trained parameters include one or more pre-trained bias terms and one or more pre-trained weight coefficients in the plurality of neural networks.

[0233] In the example, the first neural network update information displays update information for a subset of multiple neural networks, and the remaining subset of multiple neural networks is not updated.

[0234] In the example, the video decoder is the video decoder (1800) shown in FIG. 18. The first neural network is the main decoder network (915).

[0235] In the example, the video decoder is the video decoder (1600B) illustrated in FIG. 16b. In the video decoder, a plurality of neural networks include a main decoder network (915), a context model NN (916), an entropy parameter NN (917), and a hyperdecoder (925). The first neural network is one of the main decoder network (915), the context model NN (916), the entropy parameter NN (917), and the hyperdecoder (925), such as the context model NN (916). In the example, the first neural network update information further includes one or more alternative parameters for one or more remaining neural networks in the video decoder (e.g., the main decoder network (915), the entropy parameter NN (917), and / or the hyperdecoder (925)).

[0236] In (S2020), the first replacement parameter may be determined based on the first neural network update information. In the embodiment, the updated parameter is obtained from the first neural network update information. In the example, the updated parameter may be obtained from the first neural network update information by decompression (e.g., LZMA2 or bzip2 algorithm).

[0237] In the example, the first neural network update information indicates that the updated parameter is the difference between the first surrogate parameter and the first pre-trained parameter. The first surrogate parameter can be calculated based on the sum of the updated parameter and the first pre-trained parameter.

[0238] In the embodiment, the first replacement parameter is determined to be the updated parameter.

[0239] In an embodiment, the updated parameter is a representative parameter generated based on a first replacement parameter on the encoder side (e.g., using a linear or non-linear transformation), and the first replacement parameter is obtained based on the representative parameter.

[0240] In (S2030), the first neural network in the video decoder may be updated (or fine-tuned) based on the first replacement parameter, for example, by replacing the first pre-trained parameter in the first neural network with the first replacement parameter. If the video decoder includes a plurality of neural networks and the first neural network update information indicates update information for the plurality of neural networks (e.g., additional replacement parameter(s)), the plurality of neural networks may be updated. For example, the first neural network update information may additionally include one or more replacement parameters for one or more remaining neural networks in the video decoder, and one or more remaining neural networks may be updated based on the one or more replacement parameters.

[0241] In (S2040), the first encoded block in the bitstream can be decoded by a video decoder that is updated, for example, based on an updated first neural network. The output block generated in (S2040) can be any suitable block having any suitable size. In some examples, the output block is a reconstructed block within an image reconstructed in the spatial domain.

[0242] In some examples, the output block of the video decoder contains residual data in the spatial domain, and thus additional processing may be used to generate a reconstructed block based on the output block. For example, the reconstruction module (874) is configured to combine the residual data and the prediction results (output by the inter or intra prediction module) in the spatial domain to form reconstructed blocks that may be part of the reconstructed image. To improve visual quality, additional appropriate operations, such as deblocking operations, may be performed. For example, referring to FIGS. 8 and 9, components within various devices may be appropriately combined to achieve (S2040) in order to generate a reconstructed image by supplying the residual data and the corresponding prediction results from the main decoder network (915) in the video decoder to the reconstruction module (874).

[0243] In the example, the bitstream further includes one or more encoded bits used to determine a context model for decoding the encoded block. The video decoder may include a main decoder network (e.g., (911)), a context model network (e.g., (916)), an entropy parameter network (e.g., (917)), and a hyperdecoder network (e.g., (925)). The neural network is one of the main decoder network, the context model network, the entropy parameter NN, and the hyperdecoder network. One or more encoded bits may be decoded using the hyperdecoder network. The entropy model (e.g., context model) may be determined using the context model network and the entropy parameter network based on the decoded bits of the encoded block and the quantized potentials available to the context model network. The encoded block may be decoded using the main decoder network and the entropy model.

[0244] The process (2000) proceeds to (S2099) and terminates.

[0245] The process (2000) can be adapted to suit various scenarios, and the steps in the process (2000) can be adjusted accordingly. One or more of the steps in the process (2000) may be adapted, omitted, repeated, and / or combined. Any appropriate order may be used to implement the process (2000). Additional step(s) may be added.

[0246] In the example, at (S2040), another block in the coded bitstream is decoded based on the first neural network updated for the first block.

[0247] In the example, at (S2010), the second neural network update information in the coded bitstream for the second neural network is decoded in the video decoder. The second neural network consists of second pre-trained parameters. The second neural network update information corresponds to a second block in the image to be reconstructed and indicates a second alternative parameter corresponding to the second pre-trained parameter among the second pre-trained parameters. The second neural network (e.g., context model NN (916)) may be different from the first neural network (e.g., main decoder network (915)). At (S2030), the second neural network in the video decoder may be updated based on the second alternative parameter. At (S2040), the second block may be decoded based on the updated second neural network for the second block. In the example, the first pre-trained parameter is one of the pre-trained weight coefficients and the pre-trained bias term. In the example, the second pre-trained parameter is the other of the pre-trained weight coefficients and the pre-trained bias term.

[0248] The embodiments of the present disclosure may be used individually or combined in any order. Additionally, each of the methods (or embodiments), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transient computer-readable medium.

[0249] The present disclosure makes no limitations on methods used in encoders, such as neural network-based encoders, or decoders, such as neural network-based decoders. The neural network(s) used in encoders, decoders, etc., may be any suitable type of neural network(s), such as DNNs, CNNs, etc.

[0250] Accordingly, the content-adaptive online training methods of the present disclosure can accommodate different types of NIC frameworks, for example, different types of encoding DNNs, decoding DNNs, encoding CNNs, decoding CNNs, etc.

[0251] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 21 illustrates a computer system (2100) suitable for implementing specific embodiments of the disclosed subject matter.

[0252] Computer software may be coded using any suitable machine code or computer language that can undergo assembly, compilation, linking, or similar mechanisms to generate code containing instructions that can be executed directly or through interpretation, micro-code execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0253] The commands can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0254] The components of the computer system (2100) illustrated in FIG. 21 are essentially exemplary and are not intended to imply any limitation on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be interpreted as having any dependency or requirement on any one or combination of the components illustrated in the exemplary embodiments of the computer system (2100).

[0255] The computer system (2100) may include specific human interface input devices. These human interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture specific media that are not necessarily directly related to conscious input by humans, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0256] Input human interface devices may include one or more of a keyboard (2101), a mouse (2102), a trackpad (2103), a touch screen (2110), a data glove (not shown), a joystick (2105), a microphone (2106), a scanner (2107), and a camera (2108) (only one of each is shown).

[0257] The computer system (2100) may also include specific human interface output devices. Such human interface output devices may stimulate one or more human user's senses through, for example, tactile output, sound, light, and smell / taste. These human interface output devices may include haptic output devices (e.g., haptic feedback via a touch-screen (2110), data glove (not shown), or joystick (2105), but there may also be haptic feedback devices that do not serve as input devices), audio output devices (e.g., speaker (2109), headphones (not shown)), visual output devices (e.g., screen (2110) including a CRT screen, LCD screen, plasma screen, OLED screen - each having or not having touch screen input capability, each having or not having haptic feedback capability, and some of them may output output beyond three dimensions through means such as two-dimensional visual output or stereographic output -; virtual reality glasses (not shown), holographic display and smoke tank (not shown)), and printer (not shown).

[0258] The computer system (2100) may also include human-accessible storage devices and associated media, such as optical media including a CD / DVD ROM / RW (2120) having a CD / DVD media (2121), thumb drives (2122), removable hard drives or solid-state drives (2123), legacy magnetic media such as tapes and floppy disks (not shown), specialized ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0259] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carriers, or other transient signals.

[0260] The computer system (2100) may also include an interface (2154) to one or more communication networks (2155). The networks may be, for example, wireless, wireline, optical. The networks may additionally be local, wide-area, metropolitan, automotive and industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks, such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, automotive and industrial including CANBus, etc. Certain networks generally require external network interface adapters attached to certain general-purpose data ports or peripheral buses (2149) (e.g., USB ports of the computer system (2100)); Others are generally integrated into the core of the computer system (2100) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2100) can communicate with other entities. Such communication may be unidirectional receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., CANbus to specific CANbus devices), or bidirectional with other computer systems using, for example, local or wide-area digital networks. Specific protocols and protocol stacks may be used for each of the aforementioned networks and network interfaces.

[0261] The human interface devices, human-accessible storage devices, and network interfaces described above can be attached to the core (2140) of the computer system (2100).

[0262] The core (2140) may include one or more central processing units (CPUs) (2141), specialized programmable processing units in the form of a graphics processing unit (GPU) (2142), a field programmable gate area (FPGA) (2143), hardware accelerators (2144) for specific tasks, a graphics adapter (2150), etc. These devices may be connected via a system bus (2148), along with internal mass storage (2147), such as read-only memory (ROM) (2145), random access memory (2146), internal non-user accessible hard drives, SSDs, etc. In some computer systems, the system bus (2148) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (2148) or via a peripheral bus (2149). In the example, the screen (2110) can be connected to a graphics adapter (2150). Architectures for peripheral buses include PCI, USB, etc.

[0263] CPUs (2141), GPUs (2142), FPGAs (2143), and accelerators (2144) can be combined to execute specific instructions that can constitute the aforementioned computer code. The computer code may be stored in ROM (2145) or RAM (2146). Transient data may also be stored in RAM (2146), while persistent data may be stored, for example, in internal mass storage (2147). High-speed storage and retrieval of any of the memory devices may be made possible through the use of cache memory, which may be closely associated with one or more CPUs (2141), GPUs (2142), mass storage (2147), ROM (2145), RAM (2146), etc.

[0264] A computer-readable medium may have computer code for performing various computer-implemented operations. The medium and the computer code may be those specifically designed and configured for the purposes of this disclosure, or they may be of a kind well known and available to those skilled in the field of computer software technology.

[0265] As an example rather than a limitation, a computer system (2100) having an architecture, and specifically a core (2140), may provide functionality as a result of processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software implemented on one or more types of tangible computer-readable media. Such computer-readable media may be media associated with specific storage of the core (2140) that is of a non-transient nature, such as user-accessible mass storage as described above, as well as internal mass storage (2147) or ROM (2145). Software implementing various embodiments of the present disclosure may be stored on these devices and executed by the core (2140). The computer-readable media may include one or more memory devices or chips as needed. Software may enable the core (2140) and, specifically, the processors within it (including a CPU, GPU, FPGA, etc.) to execute specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (2146) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise implemented in a circuit (e.g., accelerator (2144)) that can operate instead of or with the software to execute specific processes or specific parts of specific processes described herein. Reference to software may include logic, where appropriate, and vice versa. Reference to a computer-readable medium may include, where appropriate, a circuit storing software for execution (e.g., an integrated circuit (IC)), a circuit implementing logic for execution, or both.The present disclosure includes any suitable combination of hardware and software.

[0266] Appendix A: Acronyms

[0267] JEM: joint exploration model

[0268] VVC: versatile video coding

[0269] BMS: benchmark set

[0270] MV: Motion Vector

[0271] HEVC: High Efficiency Video Coding

[0272] SEI: Supplementary Enhancement Information

[0273] VUI: Video Usability Information

[0274] GOPs: Groups of Pictures

[0275] TUs: Transform Units,

[0276] PUs: Prediction Units

[0277] CTUs: Coding Tree Units

[0278] CTBs: Coding Tree Blocks

[0279] PBs: Prediction Blocks

[0280] HRD: Hypothetical Reference Decoder

[0281] SNR: Signal Noise Ratio

[0282] CPUs: Central Processing Units

[0283] GPUs: Graphics Processing Units

[0284] CRT: Cathode Ray Tube

[0285] LCD: Liquid-Crystal Display

[0286] OLED: Organic Light-Emitting Diode

[0287] CD: Compact Disc

[0288] DVD: Digital Video Disc

[0289] ROM: Read-Only Memory

[0290] RAM: Random Access Memory

[0291] ASIC: Application-Specific Integrated Circuit

[0292] PLD: Programmable Logic Device

[0293] LAN: Local Area Network

[0294] GSM: Global System for Mobile communications

[0295] LTE: Long-Term Evolution

[0296] CANBus: Controller Area Network Bus

[0297] USB: Universal Serial Bus

[0298] PCI: Peripheral Component Interconnect

[0299] FPGA: Field Programmable Gate Areas

[0300] SSD: solid-state drive

[0301] IC: Integrated Circuit

[0302] CU: Coding Unit

[0303] NIC: Neural Image Compression

[0304] RD: Rate-Distortion

[0305] E2E: End to End

[0306] ANN: Artificial Neural Network

[0307] DNN: Deep Neural Network

[0308] CNN: Convolution Neural Network

[0309] Although the present disclosure describes several exemplary embodiments, there are modifications, substitutions, and various alternative equivalents that fall within the scope of the present disclosure.

Claims

Claim 1 A method for video decoding in a video decoder, comprising: a step of decoding first neural network update information in a coded bitstream for a first neural network in the video decoder, wherein the first neural network is composed of first pre-trained parameters, and the first neural network update information corresponds to a first block in an image to be reconstructed and indicates a first alternative parameter corresponding to a first pre-trained parameter among the first pre-trained parameters; a step of updating the first neural network in the video decoder based on the first alternative parameter; a step of decoding the first block based on the first neural network updated for the first block; and a step of decoding second neural network update information in a coded bitstream for a second neural network in the video decoder, wherein the second neural network is composed of second pre-trained parameters, and the second neural network update information corresponds to a second block in the image to be reconstructed and indicates a second alternative parameter corresponding to a second pre-trained parameter among the second pre-trained parameters, and the second neural network is the first A method comprising: a step of updating the second neural network in the video decoder based on the second alternative parameter; and a step of decoding the second block based on the updated second neural network for the second block. Claim 2 A method according to claim 1, wherein the first neural network update information further indicates one or more alternative parameters for one or more remaining neural networks in the video decoder, and the updating step further includes the step of updating one or more remaining neural networks based on the one or more alternative parameters. Claim 3 A method according to claim 1, wherein the first pre-trained parameter is one of a pre-trained weight coefficient and a pre-trained bias term. Claim 4 In paragraph 3, the method wherein the second pre-trained parameter is the other of the pre-trained weight coefficient and the pre-trained bias term. Claim 5 A method according to claim 1, further comprising the step of decoding a second block in the coded bitstream based on the first neural network updated for the first block. Claim 6 A method according to claim 1, wherein the first neural network update information indicates the difference between the first alternative parameter and the first pre-trained parameter, and the method further comprises the step of determining the first alternative parameter according to the sum of the difference and the first pre-trained parameter. Claim 7 A method according to claim 1, wherein the step of decoding the first neural network update information comprises decoding the first neural network update information based on one of a variation of the Lempel-Ziv-Markov chain algorithm (LZMA2) and the bzip2 algorithm. Claim 8 In claim 7, the step of decoding the second neural network update information comprises the step of decoding the second neural network update information based on the other of the LZMA2 and the bzip2 algorithm. Claim 9 A video decoding device comprises a processing circuit, wherein the processing circuit: decodes first neural network update information in a coded bitstream for a first neural network in a video decoder—the first neural network is composed of first pre-trained parameters, and the first neural network update information corresponds to a first block in an image to be reconstructed and indicates a first alternative parameter corresponding to a first pre-trained parameter among the first pre-trained parameters—; updates the first neural network in the video decoder based on the first alternative parameter; decodes the first block based on the first neural network updated for the first block; and decodes second neural network update information in a coded bitstream for a second neural network in the video decoder—the second neural network is composed of second pre-trained parameters, and the second neural network update information corresponds to a second block in the image to be reconstructed and indicates a second alternative parameter corresponding to a second pre-trained parameter among the second pre-trained parameters, and the second neural network is the first A video decoding device configured to update the second neural network in the video decoder based on the second alternative parameter and to decode the second block based on the updated second neural network for the second block. Claim 10 A video decoding device according to claim 9, wherein the first neural network update information further indicates one or more alternative parameters for one or more remaining neural networks in the video decoder, and the processing circuit is configured to update the one or more remaining neural networks based on the one or more alternative parameters. Claim 11 In claim 9, the processing circuit is: a video decoding device configured to decode a second block in the coded bitstream based on the first neural network updated for the first block. Claim 12 A video decoding device according to claim 9, wherein the first neural network update information indicates the difference between the first alternative parameter and the first pre-trained parameter, and the processing circuit is configured to determine the first alternative parameter according to the sum of the difference and the first pre-trained parameter. Claim 13 A non-transient computer-readable storage medium storing a program executable by at least one processor to perform the operations of the method of any one of claims 1 to 8. Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images