Content adaptive online training with scaling factors and / or offsets in neural image compression

Abstract

Aspects of the present application provide methods, apparatuses, and non-volatile computer-readable storage media for video encoding and video decoding. An apparatus for video decoding may include processing circuitry. The processing circuitry is configured to decode neural network update information in the encoded code stream for at least one neural network in the video decoder. The at least one neural network is configured with a group of pre-training parameters, and the neural network update information indicates a first modification parameter. The processing circuitry is configured to update the set of pre-training parameters in the at least one neural network in the video decoder based on the first modification parameters. The processing circuit is configured to decode the encoded image based on the updated at least one neural network.

Description

Incorporation by Reference

[0001] This application claims priority to U.S. Patent Application No. 17 / 730,000, filed on April 26, 2022, “CONTENT-ADAPTIVE ONLINE TRAINING WITH SCALING FACTORS AND / OR OFFSETS IN NEURAL IMAGE COMPRESSION,” which claims priority to U.S. Provisional Application No. 63 / 182,418, filed on April 30, 2021, “Content-Adaptive Online Training with Scaling Factor and Offset in Neural Image Compression.” The entire contents of the prior applications are incorporated herein by reference. Technical Field

[0002] Embodiments described herein generally relate to video encoding and decoding. Background Art

[0003] The background description provided herein is intended to present the background of the present application. The work of the named inventors, the work described in this background section, and the contents within the scope of the embodiments of this specification may not be considered as prior art at the time of filing, and are not explicitly or implicitly admitted as prior art against the present application.

[0004] Video encoding and decoding can be performed using inter-image prediction with motion compensation. An uncompressed digital image and / or video may include a series of images, each image having a certain spatial dimension, such as 1920x1080 luminance samples and associated chrominance samples. The image sequence may have a fixed or variable image rate (also commonly known as frame rate), for example, 60 images per second or 60 Hz. Uncompressed images and / or videos have specific bit rate requirements. For example, 1080p60 4:2:0 video (1920×1080 luminance sample resolution at 60Hz, frame rate) with 8 bits per sample requires close to 1.5Gbit / s bandwidth. One hour of such video requires more than 600GB of storage space.

[0005] One purpose of video encoding and decoding can be to reduce redundancy in input images and / or video signals by compression. Compression can help reduce the above-mentioned bandwidth and / or storage space requirements, in some cases reducing two orders of magnitude or more. Although the description herein uses video encoding / decoding as an illustrative example, the same technology can be applied to image encoding / decoding in a similar manner without departing from the spirit of the present application. Lossless compression and lossy compression, as well as their combinations, can be used. Lossless compression refers to a technology that can reconstruct an exact copy of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may be different from the original signal, but the distortion between the original signal and the reconstructed signal is small enough so that the reconstructed signal can be used for the desired purpose. Lossy compression is widely used in the video field. The amount of tolerable distortion depends on the application; for example, users of certain consumer streaming applications can tolerate greater distortion than users of television broadcast applications. The achievable compression ratio can reflect that a higher allowable / tolerable distortion can produce a higher compression ratio.

[0006] Video encoders and decoders may utilize several broad categories of techniques including, for example, motion compensation, Fourier transforms, quantization, and entropy coding.

[0007] Video codec techniques may include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data in a previously reconstructed reference picture. In some video codecs, an image is spatially subdivided into blocks of samples. An image may be an intra-image when all blocks of samples are encoded or decoded in intra-mode. Intra-images and their derivatives (e.g., independent decoder refresh images) may be used to reset decoder states and may thus be used as the first image in an encoded video stream and video session, or as a still image. Samples of an intra-predicted block may be transformed, and the transform coefficients may be quantized prior to entropy coding. Intra-prediction may be a technique for minimizing sample values ​​in a pre-transform domain. In some cases, the smaller the DC value after transformation, the smaller the AC coefficient, and the fewer bits are required to represent the block with a given quantization step size after entropy coding.

[0008] For example, conventional intra-frame coding known from, for example, MPEG-2 generation coding techniques does not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to start from, for example, surrounding sample data and / or metadata that are obtained during encoding and / or decoding of spatially adjacent, preceding data blocks in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. Note that, in at least some cases, intra-frame prediction uses only reference data from the current image being reconstructed, and does not use reference data from reference images.

[0009] There can be many different forms of intra-frame prediction. When more than one such technique can be used in a given video coding technique, the techniques used can be encoded in intra-frame prediction modes. In some cases, a mode can have sub-modes and / or parameters, and these sub-modes and / or parameters can be encoded separately or included in a mode codeword. Such codewords for a given mode, sub-mode and / or parameter combination can affect the coding efficiency gain through intra-frame prediction, and therefore can affect the entropy coding technique used to convert the codeword into a bitstream.

[0010] Some mode of intra prediction was introduced with H.264, improved in H.265, and further refined in newer coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). The predictor block can be formed using neighboring sample values ​​belonging to samples that are already available. Sample values ​​of neighboring samples are copied into the predictor block according to the direction. A reference to the direction of use can be encoded in the bitstream or can itself be predicted.

[0011] refer to Figure 1A , a subset of the nine known predictor directions of the 33 possible predictor directions known from H.265 (corresponding to the 33 angular modes of the 35 intra modes) is depicted at the bottom right. The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which the sample is being predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the upper right at an angle of 45 degrees to the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the lower left of sample (101) at an angle of 22.5 degrees to the horizontal.

[0012] Still reference Figure 1A , a square block (104) of 4x4 samples is shown in the upper left (indicated by bold dashed lines). The square block (104) includes 16 samples, each of which is labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (starting from the top) and the first sample in the X dimension (starting from the left). Similarly, sample S44 in block (104) is the fourth sample in both the Y dimension and the X dimension. Since the size of the block is 4×4 samples, S44 is located in the lower right corner. Reference samples are also shown according to a similar numbering scheme. The reference samples are labeled with R and their Y position (e.g., row index) and X position (column index) relative to the block (104). In H.264 and H.265, the prediction samples are adjacent to the block being reconstructed; therefore, there is no need to use negative values.

[0013] Intra-frame image prediction can work by copying reference sample values ​​from neighboring samples according to a signaled prediction direction. For example, assume that the encoded video bitstream includes signaling that indicates, for this block, a prediction direction consistent with arrow (102) - that is, the sample is predicted from one or more prediction samples to the upper right at a 45 degree angle to the horizontal. In this case, samples S41, S32, S23, and S14 are predicted using the same reference sample R05. Sample S44 is then predicted using reference sample R08.

[0014] In some cases, particularly when the direction is not divisible by 45 degrees, the values ​​of multiple reference samples may be combined (eg, by interpolation) in order to calculate the reference sample.

[0015] As video coding techniques have evolved, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This was increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions when published. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding are used to represent those possible directions with a small number of bits, accepting some penalty for less likely directions. Further, the directions themselves can sometimes be predicted based on the neighboring directions used by neighboring decoded blocks.

[0016] Figure 1B A schematic diagram (110) of 65 intra prediction directions of JEM is shown to illustrate the increasing number of prediction directions over time.

[0017] The mapping from intra prediction directions to the bits representing that direction in the coded video bitstream can vary in different video coding techniques; it can range, for example, from simple direct mappings from prediction directions to intra prediction modes or to codewords, to complex adaptive schemes involving most possible modes, and similar techniques. However, in all cases, there may be some directions that are statistically less likely to occur in the video content than other directions. Since the goal of video compression is to reduce redundancy, in video coding techniques that perform well, those less likely directions require more bits to represent than more likely directions.

[0018] Motion compensation may be a lossy compression technique, and may involve a technique where a block of sample data from a previously reconstructed image (reference image) or portion thereof is spatially shifted in a direction indicated by a motion vector (hereafter MV) to a position which may then be used to predict a newly reconstructed image or image portion. In some cases, the reference image may be the same as the image currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, the third dimension being an indication of the reference image currently being used (similar to the time dimension).

[0019] In some video compression techniques, the MV for a region of sample data can be predicted using those MVs associated with another region of sample data that is spatially adjacent to the region being reconstructed and precedes the MV in decoding order. Doing so can significantly reduce the amount of data required to encode the MV by relying on removing redundancy, thereby improving compression efficiency. MV prediction can work effectively, for example, because when encoding an input video signal derived from a camera (called natural video), there is a statistical possibility that a larger area than the area to which a single MV can be applied moves in a similar direction, and therefore, in some cases, similar motion vectors derived from MVs of adjacent regions can be used for prediction. This results in the MV found for a given region being similar or identical to the MV predicted from the surrounding MVs, so that after entropy coding, the MV can be represented with fewer bits than the number of bits used to encode the MV directly instead of predicting the MV from one or more adjacent MVs. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from an original signal (i.e., a sample stream). In other cases, MV prediction itself may be lossy, for example due to rounding errors introduced when calculating predicted values ​​from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T H.265 Recommendation, High Efficiency Video Coding, December 2016). Among the many MV prediction mechanisms provided by H.265, a technique referred to as spatial combining is described here.

[0021] Referring to FIG. 2 , the current block ( 201 ) includes samples that have been found by the encoder during a motion search process to be predicted from a previous block of the same size for which a spatial offset has been generated. Rather than encoding the MV directly, the MV is derived from metadata associated with one or more reference images, such as the most recent reference image (in decoding order), e.g., using the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use the predictor of the same reference image used by neighboring blocks. Summary of the invention

[0022] Various aspects of the present application provide methods and devices for video encoding and decoding. In some examples, a video decoding device includes a processing circuit. The processing circuit is used to decode neural network update information for at least one neural network in a video decoder in an encoded bitstream. The at least one neural network is configured with a set of pre-trained parameters. The neural network update information indicates a first modification parameter. The processing circuit is used to update the set of pre-trained parameters in the at least one neural network in the video decoder based on the first modification parameter. The processing circuit is used to decode an encoded image based on the updated at least one neural network.

[0023] In one embodiment, the first modified parameter is a scaling factor. A set of pre-trained parameters are replaced with a set of replacement parameters. The set of replacement parameters is a set of pre-trained parameters multiplied by the scaling factor. In one example, the neural network update information indicates an offset for modifying the set of pre-trained parameters. The set of replacement parameters in the at least one neural network is added with the offset.

[0024] In one embodiment, the first modification parameter is an offset, and the set of pre-trained parameters in the at least one neural network is added with the offset.

[0025] In one embodiment, the encoded bitstream indicates one or more encoded bits for determining a context model for decoding the encoded image. The at least one neural network includes one or more of a main decoder network, a context model neural network, an entropy parameter neural network, and a super decoder network in a video decoder. The one or more encoded bits are decoded using the super decoder network. The processing circuit uses the context model network and the entropy parameter network to determine the context model based on the decoded bits and a quantized potential representation of the encoded image available to the context model network. The processing circuit uses the main decoder network and the context model to decode the encoded image.

[0026] In one embodiment, the at least one neural network is one or more of a main decoder network, a context model neural network, an entropy parameter neural network, and a super decoder network in a video decoder.

[0027] In one embodiment, the set of pre-trained parameters includes a bias in a convolutional layer of a neural network in at least one of the neural networks.

[0028] In one embodiment, the set of pre-trained parameters includes weights in a convolution kernel of a convolution layer of a neural network in at least one neural network.

[0029] In one embodiment, the neural network update information indicates a second modified parameter of the at least one neural network configured with a second set of pre-trained parameters. The processing circuit updates the second set of pre-trained parameters in the at least one neural network with the second modified parameter.

[0030] In one example, one set of pre-trained parameters includes weights in a convolution kernel of a convolution layer of one of the at least one neural network. The second set of pre-trained parameters includes a bias of a convolution layer of one of the at least one neural network.

[0031] In one example, the processing circuit decodes another encoded image in the encoded code stream based on the updated at least one neural network. Aspects of the present application also provide a non-volatile computer-readable storage medium having a program stored therein, which can be executed by at least one processor to implement a method for video encoding and decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Further features, properties and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0033] Figure 1A is a schematic diagram of an exemplary subset of intra prediction modes.

[0034] Figure 1B is a schematic diagram of exemplary intra prediction directions.

[0035] FIG. 2 is a schematic diagram of a current block ( 201 ) and surrounding samples according to an embodiment.

[0036] Figure 3 is a schematic diagram of a simplified block diagram of a communication system (300) of one embodiment.

[0037] Figure 4 is a schematic diagram of a simplified block diagram of a communication system (400) of one embodiment.

[0038] Figure 5 is a schematic diagram of a simplified block diagram of a decoder of an embodiment.

[0039] Figure 6 is a schematic diagram of a simplified block diagram of an encoder of an embodiment.

[0040] Figure 7 A block diagram of an encoder of another embodiment is shown.

[0041] Figure 8 A block diagram of a decoder of another embodiment is shown.

[0042] Fig. 9An exemplary NIC framework according to one embodiment of the present application is shown.

[0043] Fig.10 An exemplary convolutional neural network (CNN) of the main encoder network of one embodiment of the present application is shown.

[0044] Fig.11 An exemplary CNN of the main decoder network of one embodiment of the present application is shown.

[0045] Fig.12 An exemplary CNN of a super encoder according to one embodiment of the present application is shown.

[0046] Fig.13 An exemplary CNN of a super decoder according to one embodiment of the present application is shown.

[0047] Fig.14 An exemplary CNN of a context model network according to an embodiment of the present application is shown.

[0048] Fig.15 An exemplary CNN of an entropy parameter network according to an embodiment of the present application is shown.

[0049] Fig.16A An exemplary video encoder according to an embodiment of the present application is shown.

[0050] Fig. 16B An exemplary video decoder according to an embodiment of the present application is shown.

[0051] Fig.17 An exemplary video encoder according to an embodiment of the present application is shown.

[0052] Fig.18 An exemplary video decoder according to an embodiment of the present application is shown.

[0053] Fig.19 The flowchart shows an overview of a method according to an embodiment of the present application.

[0054] Fig. 20 The flowchart shows an overview of a method according to an embodiment of the present application. Fig.21 is a schematic diagram of a computer system of an embodiment. DETAILED DESCRIPTION

[0055] I. Communication system, video encoder and video decoder

[0056] Figure 3A simplified block diagram of a communication system (300) according to an embodiment of the present application is illustrated. The communication system (300) includes a plurality of terminal devices that are capable of communicating with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). Figure 3 In the example, a first pair of terminal devices (310) and (320) can perform unidirectional data transmission. For example, the terminal device (310) can encode video data and transmit it to another terminal device (320) via the network (350). The video data can be, for example, video data of a video image stream collected by the terminal device (310). The encoded video data can be transmitted in the form of one or more encoded video code streams. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to restore the video image, and display the video image based on the restored video data. Unidirectional data transmission can be common in media service applications, etc.

[0057] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340). The second pair of terminal devices can, for example, perform bidirectional transmission of encoded video data during a video conference. For bidirectional transmission of data, in one embodiment, each of the terminal devices (330) and (340) can encode video data (e.g., a video image stream captured by the terminal device) and transmit the video data to another terminal device of the terminal devices (330) and (340) via a network (350). Each of the terminal devices (330) and (340) can also receive the encoded video data transmitted by another terminal device of the terminal devices (330) and (340), can decode the encoded video data to restore the video image, and can display the video image on an accessible display device based on the restored video data.

[0058] exist Figure 3In the example of, terminal devices (310), (320), (330) and (340) can be shown by servers, personal computers and smart phones, but the principles of the present application may not be limited to this. Embodiments of the present application can be applied to laptops, tablet computers, media players and / or dedicated video conferencing equipment. Network (350) represents any number of networks that transmit encoded video data between terminal devices (310), (320), (330) and (340), including, for example, cables (wired) and / or wireless communication networks. Communication network (350) can exchange data in circuit-switched, packet-switched channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purpose of discussion herein, unless otherwise specified below, the architecture and topology of network (350) may be irrelevant to the operations disclosed in the present application.

[0059] As examples of applications of the disclosed subject matter, Figure 4 The deployment of the video encoder and video decoder in a video streaming environment is shown. The disclosed subject matter can also be applied to other video applications, including, for example, video conferencing, digital TV, storing compressed video on digital media (including CD, DVD, memory stick, etc.), etc.

[0060] The streaming system may include a video capture subsystem (413). The video capture subsystem (413) may include a video source (401), such as a digital camera, for creating an uncompressed video image or image stream (402). In one embodiment, the video image stream (402) includes samples captured by a digital camera. The video image stream (402) (depicted by thick lines to emphasize that its data volume is larger than that of the encoded video data (404) (or encoded video code stream)) can be processed by an electronic device (420), which includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. The encoded video data (402) (or encoded video code stream (404)) (depicted by thin lines to emphasize that its data volume is lower than that of the video image stream) can be stored on a streaming server (405) for future use. One or more streaming client subsystems, such as Figure 4The client subsystem (406) and the client subsystem (408) in the video transmission system can access the streaming server (405) to retrieve the copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in the electronic device (430). The video decoder (410) decodes the input copy (407) of the encoded video data and creates an output stream (411) of video images that can be presented on a display (412) (e.g., a display screen) or other presentation device (not shown). In some streaming transmission systems, the encoded video data (404), (407) and (409) (e.g., video bitstreams) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T Recommendation H.265. In one embodiment, the video coding and decoding standard under development is informally referred to as the Versatile Video Coding (VVC) standard. The subject matter disclosed in the present application can be used in an environment where the VVC standard is applied.

[0061] It should be noted that the electronic device (420) and the electronic device (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0062] Figure 5 The block diagram of a video decoder (510) according to an embodiment of the present application is shown. The video decoder (510) may be arranged in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used to replace Figure 4 A video decoder (410) of an embodiment.

[0063] The receiver (531) can receive one or more encoded video sequences to be decoded by the video decoder (510). In the same embodiment or another embodiment, one encoded video sequence can be decoded at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequence can be received from a channel (501), which can be a hardware / software link to a storage device storing the encoded video data. The receiver (531) can receive the encoded video data and other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective consumption objects (not shown). The receiver (531) can separate the encoded video sequence from the other data. To combat network jitter, a buffer memory (515) can be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other applications, the buffer memory can be external to the video decoder (510) (not shown). In yet other applications, there may be a buffer memory (not shown) external to the video decoder (510) to, for example, combat network jitter, and there may be an additional buffer memory (515) internal to the video decoder (510), for example, to handle playout timing. When the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (515) may not be required, or the buffer memory (515) may be very small. For use over a best-effort packet network such as the Internet, the buffer memory (515) may be required and may be relatively large, and preferably may have an adaptive size, and may be at least partially implemented in an operating system or similar element (not shown) external to the video decoder (510).

[0064] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (510) and information that may be used to control a presentation device such as a presentation device (512) (e.g., a display screen), which is an integral part of the electronic device (530) but can be coupled to the electronic device (530), such as Figure 5. The control information for one or more rendering devices may be in the form of a Supplementary Enhancement Information (SEI message) or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the encoded video sequence. The encoding of the encoded video sequence may be performed according to a video coding technique or standard, and may be performed according to various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Based on at least one parameter corresponding to the group, the parser (520) may extract a set of subgroup parameters for at least one subgroup of the subgroups of pixels in the video decoder from the encoded video sequence. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), etc. The parser (520) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence information.

[0065] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to generate symbols (521).

[0066] Depending on the type of the coded video picture or part of the coded video picture (e.g., inter-frame and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different units. The units involved and the manner in which they are involved may be controlled by subgroup control information parsed from the coded video sequence by the parser (520). For the sake of clarity, the propagation of such subgroup control information between the parser (520) and the multiple units below is not described.

[0067] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a number of functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, the following subdivision based on the concepts of functional units is appropriate.

[0068] The first unit is a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) receives quantized transform coefficients in the form of one or more symbols (521) and control information from the parser (520), including the transform used, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (551) can output blocks including sample values, which can be input into an aggregator (555).

[0069] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed image, but may use prediction information from a previously reconstructed portion of the current image. Such prediction information may be provided by an intra-prediction unit (552). In some cases, the intra-prediction unit (552) may use reconstructed surrounding block information obtained from a current image buffer (558) to generate a block of the same size and shape as the block being reconstructed. For example, the current image buffer (558) buffers a partially reconstructed current image and / or a fully reconstructed current image. In some cases, the aggregator (555) may add the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0070] In other cases, the output samples of the scaler / inverse transform unit (551) may relate to inter-frame coded and potentially motion compensated blocks. In this case, the motion compensated prediction unit (553) may access the reference image memory (557) to obtain samples for prediction. After the extracted samples are motion compensated according to the block-related symbols (521), these samples can be added to the output of the scaler / inverse transform unit (551) (in this case called residual samples or residual signals) by the aggregator (555) to generate output sample information. The motion compensated prediction unit (553) obtains the predicted samples from addresses in the reference image memory (557), which addresses can be controlled by motion vectors, which can be used by the motion compensated prediction unit (553) in the form of symbols (521), which symbols can have, for example, X, Y and reference image components. Motion compensation can also include interpolation of sample values ​​extracted from the reference image memory (557) when using sub-sample accurate motion vectors, as well as motion vector prediction mechanisms, etc.

[0071] The output samples of the aggregator (555) are subjected to various loop filtering techniques in a loop filter unit (556). The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream) and available to the loop filter unit (556) as symbols (521) from the parser (520), but the video compression techniques may also be responsive to meta-information obtained during decoding of a previous portion (in decoding order) of the coded picture or coded video sequence, and to previously reconstructed and loop filtered sample values.

[0072] The output of the loop filter unit (556) may be a sample stream that may be output to a rendering device (512) and stored in a reference picture memory (557) for subsequent inter-picture prediction.

[0073] Certain coded pictures, once fully reconstructed, may be used as reference pictures for subsequent predictions. For example, once a coded picture corresponding to a current picture is fully reconstructed and that coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557) and a new current picture buffer may be reallocated before starting reconstruction of a subsequent coded picture.

[0074] The video decoder (510) may perform decoding operations according to a predetermined video compression technology in a standard such as ITU-T H.265. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the encoded video sequence conforms to the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as the only tools available under the profile. It is also necessary to conform to the standard that the complexity of the encoded video sequence may be within the range defined by the level of the video compression technology or standard. In some cases, the level limits the maximum image size, the maximum frame rate, the maximum reconstruction sampling rate (e.g., measured in megasamples per second), the maximum reference image size, etc. In some cases, the limits set by the level may be further limited by the Hypothetical Reference Decoder (HRD) specification and the HRD buffer management metadata sent in the encoded video sequence.

[0075] In one embodiment, a receiver (531) may receive encoded video and additional (redundant) data. The additional data may be considered as part of the encoded video sequence. The additional data may be used by a video decoder (510) to correctly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0076] Figure 6 A block diagram of a video encoder (603) according to an embodiment of the present application is shown. The video encoder (603) is set in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used instead of Figure 4 The video encoder (403) in the example.

[0077] The video encoder (603) can be used to obtain the video source (601) (not Figure 6 In another embodiment, the video source (601) is a part of the electronic device (620) in the example, and receives the video samples. The video source can collect one or more video images to be encoded by the video encoder (603). In another embodiment, the video source (601) is a part of the electronic device (620).

[0078] The video source (601) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (603), which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCb, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (601) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate images that exhibit motion when viewed sequentially. The image itself may be constructed as a spatial pixel array, where each pixel may include one or more samples, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by a person skilled in the art. The following description focuses on describing samples.

[0079] According to one embodiment, the video encoder (603) can encode and compress the images of the source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) can control other functional units described below and be functionally coupled to these functional units. For the sake of simplicity, the coupling method is not shown. The parameters set by the controller (650) may include rate control related parameters (image skipping, quantizer, lambda value of rate distortion optimization technology, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may have other appropriate functions related to the video encoder (603) optimized for a certain system design.

[0080] In some example embodiments, the video encoder (603) is configured to operate in a coding loop. As a simplified description, in one embodiment, the coding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input image to be encoded and one or more reference images), and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to that of the (remote) decoder to create sample data (because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (634). Since the decoding of the symbol stream produces bit-accurate results that are independent of the position of the (local or remote) decoder, the contents of the reference image memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the prediction portion of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using the prediction during decoding. The basic principle of reference image synchronicity (and drift if synchronicity cannot be maintained, for example due to channel errors) is also applied in some related techniques.

[0081] The operation of the "local" decoder (633) can be combined with the above Figure 5 The "remote" decoder described in detail for the video decoder (510) is identical. However, additional brief reference is made to Figure 5 , when symbols are available and the entropy encoder (645) and parser (520) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510) including the buffer memory (515) and the parser (520) may not be fully implemented in the local decoder (633).

[0082] In one embodiment, any decoder technology other than parsing / entropy decoding present in the decoder also exists in the corresponding encoder in substantially the same functional form. Thus, the disclosed subject matter focuses on the decoder operation. Because the encoder technology is the reverse operation of the fully described decoder technology, the description of the encoder technology can be simplified. A more detailed description is provided below in certain areas.

[0083] In some examples, during operation, the source encoder (630) may perform motion compensated predictive coding. Motion compensated predictive coding predictively encodes an input image with reference to one or more previously encoded images from a video sequence designated as "reference images." In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input image and pixel blocks of one or more reference images that may be selected as one or more prediction references for the input image.

[0084] The local video decoder (633) may decode the encoded video data of the image that may be designated as the reference image based on the symbols created by the source encoder (630). Advantageously, the operation of the encoding engine (632) may be a lossy process. When the encoded video data is available at the video decoder ( Figure 6 When the video encoder (403) is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) repeats the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (403) may locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.

[0085] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new image to be encoded, the predictor (635) may search the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that may serve as suitable prediction references for the new image. The predictor (635) may operate on a sample block-by-pixel block basis to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (635), it may be determined that the input image may have prediction references taken from a plurality of reference images stored in the reference image memory (634).

[0086] The controller (650) may manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0087] The outputs of all the above functional units may be entropy encoded in an entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0088] The transmitter (640) may buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission over a communication channel (660), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) may combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0089] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain coded picture type to each coded picture, which may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned one of the following picture types:

[0090] An Intra Picture (I Picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow for different types of I pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those of ordinary skill in the art are aware of those variations of I pictures and their corresponding applications and features.

[0091] A predictive picture (P picture) may be a picture that may be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0092] Bi-directionally predictive pictures (B pictures) can be pictures that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstructing a single block.

[0093] A source picture may typically be spatially subdivided into a plurality of blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and encoded on a block-by-block basis. Blocks may be predictively coded with reference to other (already coded) blocks, which are determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or the blocks may be predictively coded (spatial prediction or intra prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0094] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard used. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard used.

[0095] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, as well as other forms of redundant data, such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0096] The captured video may be taken as a plurality of source images (video images) in a temporal sequence. Intra-image prediction (often shortened to intra-prediction) exploits spatial correlations in a given image, while inter-frame prediction exploits temporal or other correlations between images. For example, a particular image being encoded / decoded, referred to as the current image, is divided into blocks. When a block in the current image is similar to a reference block in a reference image that was previously encoded in the video and is still cached, the block in the current image may be encoded by a vector called a motion vector. In the case of using multiple reference images, the motion vector points to a reference block in a reference image and may have a third dimension indicating the reference image.

[0097] In some embodiments, bidirectional prediction techniques may be used for inter-frame prediction. According to the bidirectional prediction technique, two reference images are used, such as a first reference image and a second reference image, both of which precede the current image in the video in decoding order (but may be in the past or future in display order). A block in the current image may be encoded by a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. The block may be predicted by a combination of the first reference block and the second reference block.

[0098] In addition, merge mode techniques can be used in inter-frame prediction to improve coding efficiency.

[0099] According to some embodiments of the present application, prediction, such as inter-frame prediction and intra-frame prediction, is performed in blocks. For example, according to the HEVC annotation, an image in a video image sequence is divided into coding tree units (CTUs) for compression, and the CTUs in the image may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU may include three coding tree blocks (CTBs), namely a luminance CTB and two chrominance CTBs. Each CTU may be recursively split into one or more coding units (coding units, CUs) using a quadtree. For example, a 64×64 pixel CTU may be split into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In one example, each CU is analyzed to determine the prediction type (e.g., inter-frame prediction type or intra-frame prediction type) of the CU. Depending on the predictability in time and / or space, the CU is split into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in the codec (encoding / decoding) is performed in units of prediction blocks. For example, the prediction block is a luma prediction block, and the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 samples, etc.

[0100] Figure 7 A diagram of a video encoder (703) of another embodiment of the present application is shown. The video encoder (703) is used to receive a processing block (e.g., a prediction block) of sample values ​​within a current video image in a video image sequence, and encode the processing block into an encoded image that is part of an encoded video sequence. The example of the video encoder (703) is used to replace Figure 4 The video encoder (403) in the example.

[0101] In the example of HEVC, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) can use, for example, rate-distortion optimization (RDO) to determine whether it is best to use intra mode, inter mode, or bidirectional prediction mode to encode the processing block. When the processing block is encoded in intra mode, the video encoder (703) can use intra prediction technology to encode the processing block into an encoded image; when it is determined that the processing block is encoded in inter mode or bidirectional prediction mode, the video encoder (703) can use inter prediction or bidirectional prediction technology to encode the processing block into the encoded image, respectively. In some video coding and decoding technologies, the merge mode can be a sub-mode of inter prediction, in which the motion vector is derived from the one or more motion vector predictors without the aid of the encoded motion vector components other than the one or more motion vector predictors. In some other video coding and decoding technologies, there may be motion vector components applicable to the subject block. For example, the video encoder (703) includes Figure 7 Other components (not shown), such as a mode decision module (not shown), are used to determine the mode of the processing block.

[0102] exist Figure 7 In the example of FIG. 7 , the video encoder ( 703 ) includes a Figure 7 An inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721) and an entropy encoder (725) are coupled together.

[0103] The inter-frame encoder (730) is used to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in a previous image and a subsequent image), generate inter-frame prediction information (e.g., redundant information description according to the inter-frame coding technology, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a predicted block) based on the inter-frame prediction information using any suitable technology. In some examples, the reference image is a decoded reference image that is decoded based on the encoded video information.

[0104] The intra-frame encoder (722) is used to receive samples of a current block (e.g., a processing block), compare the block with an encoded block in the same image in some cases, and generate quantization coefficients after transformation, and in some cases (e.g., intra-frame prediction direction information according to one or more intra-frame coding techniques) to generate intra-frame prediction information. In one example, the intra-frame encoder (722) also calculates an intra-frame prediction result (e.g., a predicted block) based on the intra-frame prediction information and a reference block in the same image.

[0105] The general controller (721) is used to determine the general control data and control other components of the video encoder (703) based on the general control data. In one embodiment, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra-frame mode, the general controller (721) controls the switch (726) to select the intra-frame mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select the intra-frame prediction information and add the intra-frame prediction information to the bitstream; when the mode is inter-frame mode, the general controller (721) controls the switch (726) to select the inter-frame prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select the inter-frame prediction information and add the inter-frame prediction information to the bitstream.

[0106] The residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra-frame encoder (722) or the inter-frame encoder (730). The residual encoder (724) can be used to operate based on the residual data to encode the residual data to generate a transform coefficient. In one example, the residual encoder (724) is used to convert the residual data from the spatial domain to the frequency domain and generate a transform coefficient. The transform coefficient is then quantized to obtain a quantized transform coefficient. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra-frame encoder (722) and the inter-frame encoder (730). For example, the inter-frame encoder (730) can generate a decoded block based on the decoded residual data and the inter-frame prediction information, and the intra-frame encoder (722) can generate a decoded block based on the decoded residual data and the intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded picture, and in some embodiments, the decoded picture may be buffered in a memory circuit (not shown) and used as a reference image in some examples.

[0107] The entropy encoder (725) is used to format the code stream to include the encoded blocks. The entropy encoder (725) is used to include various information according to appropriate standards. For example, the entropy encoder (725) is used to include general control data, selected prediction information (such as intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the code stream. Note that according to the present application, when the block is encoded and decoded in the merge sub-mode of the inter-frame mode or the bidirectional prediction mode, there is no residual information.

[0108] Figure 8 FIG. 8 is a diagram of a video decoder (810) according to another embodiment of the present invention. The video decoder (810) is configured to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed image. In one embodiment, the video decoder (810) is configured to replace Figure 4 A video decoder (410) of an embodiment.

[0109] exist Figure 8 In the example of FIG. 8 , the video decoder ( 810 ) includes a Figure 8 An entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874) and an intra-frame decoder (872) are coupled together.

[0110] The entropy decoder (871) can be used to reconstruct certain symbols from the encoded image, which represent syntax elements that constitute the encoded image. Such symbols can include, for example, the mode in which the block is encoded (e.g., intra mode, inter mode, bidirectional prediction mode, the latter two in a merge submode or another submode), prediction information (e.g., intra prediction information or inter prediction information) that can identify certain samples or metadata for prediction by the intra decoder (872) or the inter decoder (880), respectively, residual information in the form of, for example, quantized transform coefficients, etc. In one embodiment, when the prediction mode is an inter prediction mode or a bidirectional prediction mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information can be inverse quantized and provided to the residual decoder (873).

[0111] The inter-frame decoder (680) is used for receiving the inter-frame prediction information and generating an inter-frame prediction result based on the inter-frame prediction information.

[0112] The intra-frame decoder (672) is used to receive intra-frame prediction information and generate a prediction result based on the intra-frame prediction information.

[0113] The residual decoder (873) is used to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to include quantizer parameters (QP)), which may be provided by the entropy decoder (871) (the data path is not shown because this may only be a small amount of control information).

[0114] The reconstruction module (874) can be used to combine the residual output by the residual decoder (873) and the prediction result (output by the inter-frame prediction module or the intra-frame prediction module according to the situation) in the spatial domain to form a reconstructed block, which can be part of the reconstructed image, which can be part of the reconstructed video. It should be noted that other appropriate operations (such as deblocking operations, etc.) can also be performed to improve visual quality.

[0115] It should be noted that the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) can be implemented using any suitable technology. In one embodiment, the video encoders (403), (603) and (703) and the video decoders (410), (510) and (810) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603) and (603) and the video decoders (410), (510) and (810) can be implemented using one or more processors that execute software instructions.

[0116] II. Neural Network-Based Video Encoding and Decoding

[0117] The present application describes video coding techniques related to neural image compression techniques and / or neural video compression techniques, such as neural image compression (NIC) based on artificial intelligence (AI). Various aspects of the present application include content-adaptive online training in NIC, such as NIC methods for end-to-end (E2E) optimized image coding frameworks based on neural networks. Neural networks (NN) may include artificial neural networks (ANN), such as deep neural networks (DNN) or convolutional neural networks (CNN), etc.

[0118] In one embodiment, the related hybrid video codec is difficult to optimize as a whole. For example, improvements to a single module (e.g., an encoder) in a hybrid video codec may not result in coding gains in overall performance. In a NN-based video coding framework, different modules from input to output can be jointly optimized by performing a learning process or a training process (e.g., a machine learning process) to improve the final goal (e.g., rate-distortion performance, such as the rate-distortion loss L described in this application), and thus produce an E2E-optimized NIC.

[0119] An exemplary NIC framework or system can be described as follows. The NIC framework can use an input image x as an input to a neural network encoder (e.g., an encoder based on a neural network such as a DNN) to compute a compressed representation (e.g., a compact representation) The compressed representation can be compact, for example, for storage and transmission purposes. A neural network decoder (e.g., a decoder based on a neural network such as a DNN) can use the compressed representation As input to reconstruct the output image (also called the reconstructed image) In various embodiments, the input image x and the reconstructed image In the spatial domain, and the compressed representation In a domain different from the spatial domain. In some examples, the compressed representation quantized and entropy coded.

[0120] In some examples, the NIC framework can use a variational auto encoder (VAE) structure. In the VAE structure, the neural network encoder can directly use the entire input image x as the input of the neural network encoder. The entire input image x can be passed through a set of neural network layers working as a black box to calculate the compressed representation Compressed representation is the output of the neural network encoder. The neural network decoder can represent the entire compressed As input. The compressed representation The reconstructed image can be computed by another set of neural network layers working as another black box The rate-distortion (RD) loss L can be optimized To achieve image reconstruction Distortion loss With compact representation The trade-off between the bit consumption R and the trade-off hyperparameter is λ.

[0121] A neural network (e.g., ANN) can learn to perform tasks from examples without programming for specific tasks. ANN can be configured with interconnected nodes or artificial neurons. The connection between the nodes can transmit a signal from a first node to a second node (e.g., a receiving node), and the signal can be modified by a weight, which can be indicated by a weight coefficient of the connection. The receiving node can process one or more signals from one or more nodes that transmit one or more signals to the receiving node (i.e., one or more input signals of the receiving node), and then generate an output signal by applying a function to the input signal. The function can be a linear function. In one example, the output signal is a weighted sum of one or more input signals. In one example, the output signal is further modified by a bias, which can be indicated by a bias term, so that the output signal is the sum of the weighted sum of the bias and one or more input signals. The function can include, for example, a nonlinear operation of a weighted sum of one or more input signals or a bias and a weighted sum. The output signal can be sent to one or more nodes (one or more downstream nodes) connected to the receiving node. ANN can be represented or configured by parameters (e.g., weights and / or biases of the connection). The weights and / or biases can be obtained by training the ANN using examples to iteratively adjust the weights and / or biases therein. The trained ANN configured with the determined weights and / or the determined biases can be used to perform the task.

[0122] The nodes in the ANN can be organized in any suitable architecture. In various embodiments, the nodes in the ANN are organized into multiple layers, including an input layer that receives one or more input signals of the ANN and an output layer that outputs one or more output signals from the ANN. In one embodiment, the ANN further includes one or more layers, such as one or more hidden layers between the input layer and the output layer. Different layers can perform different types of transformations on the respective inputs of different layers. Signals can propagate from the input layer to the output layer.

[0123] An ANN with multiple layers between the input layer and the output layer may be referred to as a DNN. In one embodiment, the DNN is a feed-forward network, where data flows from the input layer to the output layer without looping back. In one example, the DNN is a fully connected network, where each node in one layer is connected to all nodes in the next layer. In one embodiment, the DNN is a recurrent neural network (RNN), where data can flow in any direction. In one embodiment, the DNN is a CNN.

[0124] A CNN may include an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. The one or more hidden layers may include one or more convolutional layers (e.g., used in an encoder) that perform convolution (e.g., two-dimensional (2D) convolution). In one embodiment, the 2D convolution performed in the convolutional layer is performed between a convolution kernel (also referred to as a filter or channel, such as a 5×5 matrix) and an input signal of the convolutional layer (e.g., a 2D matrix such as a 2D image, a 256×256 matrix). In various examples, the size of the convolution kernel (e.g., 5×5) is smaller than the size of the input signal (e.g., 256×256). Therefore, the portion of the input signal (e.g., a 256×256 matrix) covered by the convolution kernel (e.g., a 5×5 area) is smaller than the area of ​​the input signal (e.g., a 256×256 area), and can therefore be referred to as the receptive field of the corresponding node in the next layer.

[0125] During convolution, the dot product of the convolution kernel and the corresponding receptive field in the input signal is calculated. Therefore, each element of the convolution kernel is a weight applied to the corresponding sample in the receptive field, and thus the convolution kernel includes multiple weights. For example, a convolution kernel represented by a 5×5 matrix has 25 weights. In some examples, a bias is applied to the output signal of the convolution layer, and the output signal is based on the sum of the dot product and the bias.

[0126] The convolution kernel can be moved along the input signal (e.g., a 2D matrix) by a size called a stride, and thus the convolution operation generates a feature map or an activation map (e.g., another 2D matrix), which in turn is part of the input to the next layer in the CNN. For example, the input signal is a 2D image with 256×256 samples, and the stride is 2 samples (e.g., a stride of 2). For a stride of 2, the convolution kernel moves 2 samples along the X direction (e.g., horizontal direction) and / or the Y direction (e.g., vertical direction).

[0127] Multiple convolution kernels may be applied to the input signal in the same convolution layer to generate multiple feature maps, respectively, where each feature map may represent a specific feature of the input signal. Typically, a convolution layer with N channels (i.e., N convolution kernels), each convolution kernel having M×M samples, and a stride of S may be indicated as Conv:M×M cN sS. For example, a convolution layer with 192 channels, each convolution kernel having 5×5 samples, and a stride of 2 is indicated as Conv:5×5 c192 s2. One or more hidden layers may include one or more deconvolution layers (e.g., used in a decoder) for performing, for example, 2D deconvolution. Deconvolution is the inverse process of convolution. A deconvolution layer with 192 channels, each deconvolution kernel having 5×5 samples, and a stride of 2 is indicated as DeConv:5×5 c192 s2.

[0128] In various embodiments, CNN has the following advantages. The number of learnable parameters (i.e., parameters to be trained) in CNN can be significantly smaller than the number of learnable parameters in DNN, such as feedforward DNN. In CNN, relatively more nodes can share the same filter (e.g., the same weight) and the same bias (if bias is used), and thus memory usage can be reduced because all receptive fields sharing the same filter can use one bias and one weight vector. For example, for an input signal with 100×100 samples, a convolutional layer with a convolution kernel of 5×5 samples has 25 learnable parameters (e.g., weights). If bias is used, one channel uses 26 learnable parameters (e.g., 25 weights and one bias). If the convolutional layer has N channels, the total learnable parameters are 26×N. On the other hand, for a fully connected layer in a DNN, 100×100 (i.e., 10,000) weights are used for each node in the next layer. If the next layer has L nodes, the total learnable parameters are 10,000×L.

[0129] CNN may further include one or more other layers, such as one or more pooling layers, one or more fully connected layers that can connect each node in one layer to each node in another layer, and / or one or more normalization layers. The layers in CNN can be organized in any suitable order and in any suitable architecture (e.g., feedforward architecture, recursive architecture). In one example, the convolutional layer is followed by one or more other layers, such as one or more pooling layers, one or more fully connected layers, and / or one or more normalization layers.

[0130] A pooling layer can be used to reduce the dimensionality of the data by merging the outputs of multiple nodes in one layer into a single node in the next layer. The following describes the pooling operation of a pooling layer that takes a feature map as input. The description can be appropriately adapted to other input signals. The feature map can be divided into sub-regions (e.g., rectangular sub-regions), and the features in each sub-region can be independently downsampled (or pooled) into a single value, such as by taking the average value in average pooling or the maximum value in maximum pooling.

[0131] The pooling layer can perform pooling, such as local pooling, global pooling, maximum pooling, and / or average pooling. Pooling is a form of nonlinear downsampling. Local pooling merges a small number of nodes in a feature map (e.g., a local node cluster, such as 2×2 nodes). Global pooling can merge, for example, all nodes of a feature map.

[0132] Pooling layers can reduce the size of representations and thus reduce the number of parameters, memory usage, and computation in CNNs. In one example, pooling layers are inserted between consecutive convolutional layers in a CNN. In one example, pooling layers are followed by activation functions, such as rectified linear unit (ReLU) layers. In one example, pooling layers are omitted between consecutive convolutional layers in a CNN.

[0133] The normalization layer can be ReLU, leaky ReLU, generalized divisive normalization (GDN), or inverse GDN (IGDN), etc. Relu can apply a non-saturated activation function to remove negative values ​​from the input signal (e.g., feature map) by setting negative values ​​to zero. Leaky ReLU can have a small slope (e.g., 0.01) for negative values ​​instead of a flat slope (e.g., 0). Accordingly, if the value x is greater than 0, the output of the leaky ReLU is x. Otherwise, the output of the leaky ReLU is the value x multiplied by a small slope (e.g., 0.01). In one example, the slope is determined before training, so the slope is not learned during training.

[0134] Fig. 9 An exemplary NIC framework (900) (e.g., NIC system) of one embodiment of the present application is shown. The NIC framework (900) can be based on a neural network, such as a DNN and / or a CNN. The NIC framework (900) can be used to compress (e.g., encode) an image and decompress (e.g., decode or reconstruct) a compressed image (e.g., an encoded image). The NIC framework (900) can include two sub-neural networks implemented using a neural network, namely, a first sub-NN (951) and a second sub-NN (952).

[0135] The first sub-NN (951) can be similar to an autoencoder and can be trained to generate a compressed image of the input image x And for the compressed image Decompress to get the reconstructed image The first sub-NN (951) may include a plurality of components (or modules), such as a main encoder neural network (or main encoder network) (911), a quantizer (912), an entropy encoder (913), an entropy decoder (914), and a main decoder neural network (or main encoder network) (915). Fig. 9, the main encoder network (911) can generate a potential representation or a potential representation y from an input image x (e.g., an image to be compressed or encoded). In one example, the main encoder network (911) is implemented using a CNN. The relationship between the potential representation y and the input image x can be described using Equation 2. y = f1(x; θ1) Equation 2 Here, the parameter θ1 represents the parameters, such as the weights and biases used in the convolution kernel in the main encoder network (911) (if a bias is used in the main encoder network (911)).

[0136] The latent representation y may be quantized using a quantizer (912) to generate a quantized latent representation Quantitative potential representation may be compressed, for example, by an entropy encoder (913) using lossless compression to generate a compressed image (eg, an encoded image) (931), compressed image (931) is the compressed representation of the input image x The entropy encoder (913) may use entropy coding techniques, such as Huffman coding or arithmetic coding, etc. In one example, the entropy encoder (913) uses arithmetic coding and is an arithmetic encoder. In one example, the encoded image (931) is transmitted in an encoded bitstream.

[0137] The encoded image (931) can be decompressed (e.g., entropy decoded) by an entropy decoder (914) to generate an output. The entropy decoder (914) can use an entropy coding and decoding technique corresponding to the entropy coding and decoding technique used in the entropy encoder (913), such as Huffman coding or arithmetic coding. In one example, the entropy decoder (914) uses arithmetic decoding and is an arithmetic decoder. In one example, lossless compression is used in the entropy encoder (913), lossless decompression is used in the entropy decoder (914), and noise, such as noise caused by the transmission of the encoded image (931), is negligible, and the output of the entropy decoder (914) is a quantized potential representation.

[0138] The main decoder network (915) can quantize the latent representation Decode to generate the reconstructed image In one example, the main decoder network (915) is implemented using CNN. Reconstructing the Image (i.e., the output of the main decoder network (915)) and the quantized latent representation The relationship between (i.e., the input of the main decoder network (915)) can be described using Equation 3. Wherein, parameter θ2 represents parameters, such as weights and biases used in the convolution kernel of the main decoder network (915) (if biases are used in the main decoder network (915)). Therefore, the first sub-NN (951) can compress (e.g., encode) the input image x to obtain an encoded image (931), and decompress (e.g., decode) the encoded image (931) to obtain a reconstructed image Due to the quantization loss introduced by the quantizer (912), the reconstructed image can be different from the input image x.

[0139] The second sub-NN (952) can learn the quantized latent representation for entropy coding An entropy model (e.g., a prior probability model) of the context model NN (916) may be used. Therefore, the entropy model may be a conditional entropy model, such as a Gaussian mixture model (GMM), a Gaussian scale model (GSM) that depends on the input image x. The second sub-NN (952) may include a context model NN (916), an entropy parameter NN (917), a super encoder (921), a quantizer (922), an entropy encoder (923), an entropy decoder (924), and a super decoder (925). The entropy model used in the context model NN (916) may be a potential representation (e.g., a quantized potential representation ) is an autoregressive model of a context model. In one example, a super encoder (921), a quantizer (922), an entropy encoder (923), an entropy decoder (924), and a super decoder (925) form a super neural network (e.g., a hyperprior NN). The super neural network can represent information for correcting context-based predictions. Data from the context model NN (916) and the super neural network can be merged by an entropy parameter NN (917). The entropy parameter NN (917) can generate parameters of an entropy model, such as a conditional Gaussian entropy model (e.g., GMM), and parameters such as mean and scale parameters.

[0140] refer to Fig. 9 , at the encoder side, the quantized latent representation from the quantizer (912) is fed into the context model NN (916). On the decoder side, the quantized latent representation from the entropy decoder (914) is sent to the context model NN (916). The context model NN (916) can be implemented using a neural network such as CNN. The context model NN (916) can be based on the context Generate output o cm,i, which is a quantized latent representation available to the context model NN (916) Context The output of the context model NN (916) may include a previously quantized latent representation at the encoder side or a previously entropy decoded quantized latent representation at the decoder side. cm,i With input (for example, ) can be described using Equation 4. Here, parameter θ3 represents parameters, such as weights and biases used in the convolution kernel of the context model NN (916) (if biases are used in the context model NN (916)).

[0141] Output o of the context model NN (916) cm,i and the output o from the super decoder (925) hc is fed into the entropy parameter NN (917) to generate the output o ep The entropy parameter NN (917) may be implemented using a neural network such as a CNN. The output of the entropy parameter NN (917) is ep With input (for example, o cm,i and hc ) can be described using Equation 5. o ep =f4(o cm,i , o hc ;θ4) Equation 5 Wherein, parameter θ4 represents parameters, such as weights and biases used in the convolution kernel of entropy parameter NN (917) (if biases are used in entropy parameter NN (917)). The output o of entropy parameter NN (917) ep can be used to determine (e.g., adjust) an entropy model, and thus the adjusted entropy model can be, for example, outputted by the super decoder (925). hc , which is determined by the input image x. In an example, the output o ep Includes parameters for tuning entropy models (e.g., GMM), such as mean and scale parameters. Fig. 9 , the entropy encoder (913) and the entropy decoder (914) can adopt an entropy model (e.g., an adjusted entropy model) in entropy encoding and entropy decoding, respectively.

[0142] The second sub-NN (952) can be described as follows. The latent representation y can be fed into the super encoder (921) to generate a super latent representation z. In one example, the super encoder (921) is implemented using a neural network such as a CNN. The relationship between the super latent representation z and the latent representation y can be described using Equation 6. z = f5(y;θ5) Equation 6 Here, parameter θ5 represents parameters such as weights and biases used in the convolution kernel of the super encoder (921) (if a bias is used in the super encoder (921)).

[0143] The super latent representation z is quantized by a quantizer (922) to generate a quantized latent representation Quantified potential representation The side information, such as the coded bits (932) from the hyper-neural network, may be compressed by an entropy encoder (923), for example using lossless compression. The entropy encoder (923) may use entropy coding techniques, such as Huffman coding or arithmetic coding, etc. In one example, the entropy encoder 923 uses arithmetic coding and is an arithmetic encoder. In one example, the side information, such as the coded bits (932), may be transmitted in a coded bitstream, for example, together with the coded image (931).

[0144] The side information, such as the encoded bits (932), can be decompressed (e.g., entropy decoded) by an entropy decoder (924) to generate an output. The entropy decoder (924) can use an entropy coding technique, such as Huffman coding or arithmetic coding, etc. In one example, the entropy decoder (924) uses arithmetic decoding and is an arithmetic decoder. In one example, lossless compression is used in the entropy encoder (923), lossless decompression is used in the entropy decoder (924), and noise, such as noise caused by the transmission of the side information, is negligible, and the output of the entropy decoder (924) can be a quantized latent representation. The super decoder (925) can be used to represent the quantized latent Decode to generate output o hc Output hc and the quantified potential representation The relationship between can be described using Equation 7. Wherein, parameter θ6 represents parameters such as weights and biases used in the convolution kernel of the super decoder (925) (if biases are used in the super decoder (925)).

[0145] As described above, compressed or encoded bits (932) may be added to the encoded bitstream as side information, which enables the entropy decoder (914) to use a conditional entropy model. Thus, the entropy model may be image-dependent and spatially adaptive, and may therefore be more accurate than a fixed entropy model.

[0146] The NIC framework (900) may be adapted to, for example, omit Fig. 9 One or more of the components, modifications shown in Fig. 9 One or more parts, components and / or components shown in Fig. 9 In one example, the NIC framework using the fixed entropy model includes the first sub-NN (951) but does not include the second sub-NN (952). In one example, the NIC framework includes components other than the entropy encoder (923) and the entropy decoder (924) in the NIC framework (900).

[0147] In one embodiment, Fig. 9 One or more components in the NIC framework (900) shown in the figure are implemented using one or more neural networks, such as one or more CNNs. Each NN-based component (e.g., the main encoder network (911), the main decoder network (915), the context model NN (916), the entropy parameter NN (917), the super encoder (921), or the super decoder (925)) in the NIC framework (e.g., the NIC framework (900)) can include any suitable architecture (e.g., having any suitable combination of layers), including any suitable type of parameters (e.g., weights, biases, and / or a combination of weights and biases, etc.), and including any suitable number of parameters.

[0148] In one embodiment, corresponding CNNs are used to implement the main encoder network (911), the main decoder network (915), the context model NN (916), the entropy parameter NN (917), the super encoder (921) and the super decoder (925).

[0149] Fig.10 An exemplary CNN of the main encoder network (911) of an embodiment of the present application is shown. For example, the main encoder network (911) includes four groups of layers, each of which includes a convolutional layer 5×5 c192 s2 and a subsequent GDN layer. Fig.10 One or more of the layers shown in may be modified and / or omitted. One or more additional layers may be added to the main encoder network (911).

[0150] Fig.11 An exemplary CNN of the main decoder network (915) according to an embodiment of the present application is shown. For example, the main decoder network (915) includes three groups of layers, each of which includes a deconvolution layer 5×5 c192 s2 and a subsequent IGDN layer. In addition, the three groups of layers can be followed by a deconvolution layer 5×5 c3 s2 and then a IGDN layer. Fig.11 One or more of the layers shown in may be modified and / or omitted. One or more additional layers may be added to the main decoder network (915).

[0151] Fig.12An exemplary CNN of the super encoder (921) of an embodiment of the present application is shown. For example, the super encoder (921) includes a convolutional layer 3×3 c192 s1 and a leaky ReLU connected thereto, a convolutional layer 5×5 c192 s2 and a leaky ReLU connected thereto, and a convolutional layer 5×5 c192 s2. Fig.12 One or more of the layers shown in may be modified and / or omitted. One or more additional layers may be added to the super encoder (921).

[0152] Fig.13 An exemplary CNN of the super decoder (925) of an embodiment of the present application is shown. For example, the super decoder (925) includes a deconvolution layer 5×5 c192 s2 and a leaky ReLU connected thereto), a deconvolution layer 5×5 c288 s2 and a leaky ReLU connected thereto, and a deconvolution layer 3×3 c384 s1. Fig.13 One or more of the layers shown in may be modified and / or omitted. One or more additional layers may be added to the super encoder (925).

[0153] Fig.14 An exemplary CNN of the context model NN (916) of an embodiment of the present application is shown. For example, the context model NN (916) includes a mask convolution 5×5 c384 s1 for context prediction, and thus the context in Equation 4 Include limited context (e.g., 5×5 convolution kernel). Fig.14 The convolutional layers in may be modified. One or more additional layers may be added to the context model NN (916).

[0154] Fig.15 An exemplary CNN of the entropy parameter NN (917) of an embodiment of the present application is shown. For example, the entropy parameter NN (917) includes a convolution layer 1×1 c640 s1 and a leaky ReLU connected thereto, a convolution layer 1×1 c512 and a leaky ReLU connected thereto, and a convolution layer 1×1 c384 s1. Fig.15 One or more of the layers shown in may be modified and / or omitted. One or more additional layers may be added to the entropy parameter NN (917).

[0155] The NIC framework (900) can be implemented using CNN, such as Figures 10 to 15As described. The NIC framework (900) may be appropriately modified so that one or more components (e.g., (911), (915), (916), (917), (921), and / or (925)) in the NIC framework (900) are implemented using any suitable type of neural network (e.g., a CNN-based or non-CNN-based neural network). One or more other components of the NIC framework (900) may be implemented using one or more neural networks.

[0156] The NIC framework (900) including a neural network (e.g., CNN) can be trained to learn parameters used in the neural network. For example, when using CNN, the parameters represented by θ1-θ6, such as the weights and biases used in the convolution kernels of the main encoder network (911) (if biases are used in the main encoder network (911)), the weights and biases used in the convolution kernels of the main decoder network (915) (if biases are used in the main decoder network (915)), the weights and biases used in the convolution kernels of the super encoder (921) (if biases are used in the super encoder (921)), the weights and biases used in the convolution kernels in the super decoder (925) (if biases are used in the super decoder (925)), the weights and biases used in one or more convolution kernels in the context model NN (916) (if biases are used in the context model NN (916)), and the weights and biases used in the convolution kernels of the entropy parameter NN (917) (if biases are used in the entropy parameter NN (917)).

[0157] In an example, refer to Fig.10 , the main encoder network (911) includes four convolutional layers, each of which has a 5×5 convolution kernel and 192 channels. Therefore, the number of weights used in the convolution kernel of the main encoder network (911) is 19200 (i.e., 4×5×5×192). The parameters used by the main encoder network (911) include 19200 weights and, optionally, a bias. When a bias and / or one or more additional NNs are used in the main encoder network (911), one or more additional parameters may be included.

[0158] refer to Fig. 9, the NIC framework (900) includes at least one component or module built on one or more neural networks. The at least one component may include one or more of a main encoder network (911), a main decoder network (915), a super encoder (921), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). The at least one component can be trained separately. In one example, the parameters of each component are learned through each training process respectively. The at least one component can be jointly trained as a group. In one example, the training process is used to jointly learn the parameters of a subset of the at least one component. In one example, the training process is used to learn the parameters of all components in the at least one component, and is therefore called E2E optimization.

[0159] During the training process of one or more components in the NIC framework (900), weights (or weight coefficients) of the one or more components may be initialized. In one example, the weights are initialized based on one or more pre-trained corresponding neural network models (e.g., DNN models, CNN models). In one example, the weights are initialized by setting the weights to random numbers.

[0160] For example, after the weights are initialized, a training image set may be used to train one or more components. The training image set may include any suitable image having one or more of any suitable size. In some examples, the training image set includes original images, natural images, and / or computer-generated images in the spatial domain, etc. In some examples, the training image set includes residual images having residual data in the spatial domain. The residual data may be calculated by a residual calculator (e.g., residual calculator (723)). In some examples, the training images in the training image set (e.g., original images and / or residual images including residual data) may be divided into blocks of suitable size, and these blocks and / or images may be used to train a neural network in the NIC framework. Therefore, the original image, the residual image, the block from the original image, and / or the block from the residual image may be used to train a neural network in the NIC framework.

[0161] For the sake of brevity, the following describes the training process using a training image as an example. The description can be modified appropriately to fit the training block. The training image t of the training image set can be obtained by Fig. 9 The encoding process in the code generates a compressed representation (e.g., the encoded information (e.g., encoded into the bitstream)). The encoded information can be Fig. 9 The decoding process described in is used to reconstruct the image Calculate and reconstruct.

[0162] For the NIC framework (900), two conflicting objectives, such as reconstruction quality and bit consumption, are balanced. Quality loss function (e.g., distortion or distortion loss) Can be used to indicate reconstruction quality, such as the quality of the reconstructed image (e.g., ) and the original image (e.g., training image t). A ratio (or rate loss) R can be used to indicate the bit consumption of the compressed representation. In one example, the rate loss R also includes side information, such as side information used to determine a context model.

[0163] For neural image compression, differentiable approximations of quantization can be used for E2E optimization. In various examples, during the training process of neural network-based image compression, noise injection (e.g., adding noise) is used to simulate the quantization operation, and thus the quantization operation is simulated by the noise injection rather than being performed by a quantizer (e.g., quantizer (912)). In one example, a noise injection module can perform noise injection. Therefore, training with noise injection can perform a variational approximation of the quantization error. A bits per pixel (BPP) estimator can be used to simulate an entropy encoder, and thus entropy encoding is simulated by the BPP estimator rather than being performed by an entropy encoder (e.g., (913)) and an entropy decoder (e.g., (914)). Reference Fig. 9 , the NIC framework (900) can be adapted to employ noise injection and an estimator to simulate the training process of an entropy encoder. For example, during the training process, the quantizer (912), entropy encoder (913) and entropy decoder (914) in the NIC framework (900) can be replaced by, for example, a noise injection module and a BPP estimator, respectively.

[0164] Therefore, during the training process, the rate loss R in the loss function L shown in Equation 1 can be estimated, for example, based on noise injection and a BPP estimator. In general, a higher rate R can allow for lower distortion D, while a lower rate R can result in higher distortion D. Therefore, the tradeoff hyperparameter λ in Equation (1) can be used to optimize the joint RD loss L, where the sum L of λD and R can be optimized. The training process can be used to adjust the parameters of one or more components (e.g., (911), (915)) in the NIC framework (900) so that the joint RD loss L is minimized or optimized.

[0165] Various models can be used to determine the distortion loss D and the rate loss R, and thus determine the joint RD loss L in Equation 1. In one example, the distortion loss It is expressed as peak signal-to-noise ratio (PSNR), which is a metric based on mean square error, multiscale structural similarity (MS-SSIM) quality indicator, a weighted combination of PSNR and MS-SSIM, etc.

[0166] In one example, the goal of the training process is to train an encoding neural network (e.g., encoding DNN) (e.g., a video encoder to be used on the encoder side) and to train a decoding neural network (e.g., decoding DNN) (e.g., a video decoder to be used on the decoder side). Fig. 9 , the encoding neural network may include a main encoder network (911), a super encoder (921), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). The decoding neural network may include a main decoder network (915), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). The video encoder and / or the video decoder may include one or more other components based on one or more NNs and / or not based on one or more NNs.

[0167] The NIC framework (e.g., NIC framework (900)) can be trained in an E2E manner. In one example, during the training process, the encoding neural network and the decoding neural network are jointly updated in an E2E manner based on the gradient value of back propagation. In one example, the gradient value includes the gradient of the RD loss of the parameters to be trained (e.g., weights and / or biases) in the NIC framework.

[0168] After the parameters of the neural network in the NIC framework (900) are trained, one or more components in the NIC framework (900) can be used to encode and / or decode images. In one embodiment, on the encoder side, a video encoder is used to encode an input image x into an encoded image (931) to be transmitted in a bitstream. The video encoder may include multiple components in the NIC framework (900). In one embodiment, on the decoder side, a corresponding video decoder is used to decode the encoded image (931) in the bitstream into a reconstructed image The video decoder may include multiple components in the NIC framework (900).

[0169] In one example, such as when content-adaptive online training is employed, the video encoder includes all components in the NIC framework (900).

[0170] Fig.16A An exemplary video encoder (1600A) according to an embodiment of the present application is shown. The video encoder (1600A) includes a reference Fig. 9 The main encoder network (911), quantizer (912), entropy encoder (913) and second sub-NN (952) are described, and their detailed descriptions are omitted for the sake of brevity. Fig. 16B An exemplary video decoder (1600B) of an embodiment of the present application is shown. The video decoder (1600B) may correspond to the video encoder (1600A). The video decoder (1600B) may include a main decoder network (915), an entropy decoder (914), a context model NN (916), an entropy parameter NN (917), an entropy decoder (924) and a super decoder (925). FIG. 16A to FIG. 16B On the encoder side, the video encoder (1600A) can generate coded images (931) and coded bits (932) to be transmitted in a bitstream. On the decoder side, the video decoder (1600B) can receive the coded images (931) and coded bits (932) and decode them.

[0171] Figure 17 to Figure 18 An exemplary video encoder (1700) and a corresponding video decoder (1800) according to an embodiment of the present application are shown respectively. Fig.17 , the encoder (1700) includes a main encoder network (911), a quantizer (912) and an entropy encoder (913). Examples of the main encoder network (911), the quantizer (912) and the entropy encoder (913) can be referenced Fig. 9 Description of . Reference Fig.18 , the video decoder (1800) includes a main decoder network (915) and an entropy decoder (914). Examples of the main decoder network (915) and the entropy decoder (914) can be referenced. Fig. 9 Description of . Reference Fig.17 and Fig.18 The video encoder (1700) can generate a coded image (931) to be transmitted in a code stream. The video decoder (1800) can receive the coded image (931) and decode it.

[0172] III. Content-Adaptive Online Training in Video Coding Solutions

[0173] The NIC framework (900) may be based on a neural network, such as a CNN, a DNN, and / or other neural networks. For example, one or more components in the NIC framework (900) (e.g., the main encoder network (911)) may include one or more neural networks. These neural networks may be specified by parameters such as weights, biases, etc.

[0174] As described above, a NIC framework (900) including a video encoder and a video decoder can be trained based on images and / or blocks in a training image set. In some examples, one or more images to be compressed (e.g., encoded) and / or transmitted have properties that are significantly different from the training image set. Therefore, encoding and decoding one or more images using a video encoder and a video decoder trained separately based on the training image set may result in a relatively poor RD loss L (e.g., relatively large distortion and / or relatively large bit rate). Therefore, aspects of the present application describe a content-adaptive online training method for NIC.

[0175] In order to distinguish between a training process based on a training image set and a content-adaptive online training process based on one or more images to be compressed (e.g., encoded) and / or transmitted, the NIC framework (900), video encoder, and video decoder trained by the training image set are respectively referred to as a pre-trained NIC framework (900), a pre-trained video encoder, and a pre-trained video decoder. The parameters in the pre-trained NIC framework (900), the pre-trained video encoder, or the pre-trained video decoder are respectively referred to as NIC pre-trained parameters, encoder pre-trained parameters, and decoder pre-trained parameters. In one example, the NIC pre-trained parameters include encoder pre-trained parameters and decoder pre-trained parameters. In one example, the encoder pre-trained parameters and the decoder pre-trained parameters do not overlap, wherein the decoder pre-trained parameters do not include any of the encoder pre-trained parameters. For example, the encoder pre-trained parameters in (1700) (e.g., the pre-trained parameters in the main encoder network (911)) and the decoder pre-trained parameters in (1800) (e.g., the pre-trained parameters in the main decoder network (915)) do not overlap. In one example, the encoder pre-trained parameters and the decoder pre-trained parameters overlap, wherein the decoder pre-trained parameters include at least one parameter in the encoder pre-trained parameters. For example, the encoder pre-trained parameters in (1600A) (e.g., pre-trained parameters in the context model NN (916)) and the decoder pre-trained parameters in (1600B) (e.g., pre-trained parameters in the context model NN (916)) overlap. The NIC pre-trained parameters can be obtained based on blocks and / or images in the training image set.

[0176] The content adaptive online training process can be referred to as a fine-tuning process. In the content adaptive online training process of type I, the content adaptive online training process described in the present application can be implemented to determine (e.g., train) one or more modification parameters, such as one or more scaling factors and / or one or more offsets. Each of the one or more modification parameters can modify a set of pre-trained parameters (e.g., multiple pre-trained parameters) in the NIC pre-trained parameters, such as a set of pre-trained weights, a set of pre-trained biases, or a set of weights and biases used in the NIC framework (900). A set of pre-trained parameters can be updated using the same method based on the modification parameters. In one example, the modification parameter is a scaling factor that modifies a set of pre-trained parameters. Therefore, a set of replacement parameters can be generated by multiplying a set of pre-trained parameters based on the scaling factor. For example, a set of pre-trained parameters can be multiplied with the same scaling factor. A set of pre-trained parameters in the NIC framework can be replaced by a set of replacement parameters respectively to update the NIC framework.

[0177] In a content-adaptive online training process of type II, a content-adaptive online training process may be performed to train (e.g., update) one or more pre-trained parameters in the NIC pre-trained parameters. The one or more pre-trained parameters may include one or more pre-trained weights and / or pre-trained biases used in the NIC framework (900). When the one or more pre-trained parameters include a set of pre-trained parameters, the set of pre-trained parameters may be updated in different ways.

[0178] In both the Type I content adaptive online training process and the Type II content adaptive online training process, one or more pre-trained parameters in the NIC framework can be updated, and therefore one or more neural networks in the NIC framework configured with the one or more pre-trained parameters can be updated. In some examples, for example, in the Type I content adaptive online training process, one or more pre-trained parameters in the NIC framework are updated using one or more modified parameters, wherein the one or more modified parameters are trained using the content adaptive online training process. In some examples, for example, in the Type II content adaptive online training process, one or more pre-trained parameters in the NIC framework are updated using the content adaptive online training process, such as directly updating without using modified parameters.

[0179] Examples of a content-adaptive online training process of Type I and a content-adaptive online training process of Type II may be described as follows.

[0180] A. Type I content-adaptive online training

[0181] The modification parameters can be determined based on (i) one or more images to be encoded or (ii) one or more blocks to be encoded (in the image), thereby modifying a set of pre-trained parameters. The process of determining the modified parameters can also be regarded as a process of fine-tuning the modified parameters. For example, the process of determining the scaling factor can be regarded as a process of fine-tuning the scaling parameters from a "pre-trained" scaling parameter (e.g., a scaling parameter of 1 that has not been modified to a set of pre-trained parameters) to a scaling parameter (e.g., 1.1). The training process for the offset can be regarded as a process of fine-tuning the offset from a "pre-trained" offset (e.g., an offset of 0 that has not been modified to a set of pre-trained parameters) to an offset (e.g., 0.1).

[0182] One or more modification parameters can be used to modify pre-trained parameters (e.g., weights and / or biases) in a pre-trained NIC framework, thereby modifying the pre-trained NIC framework. The modification parameters can modify a set of pre-trained parameters (e.g., a set of pre-trained weights), thereby modifying the pre-trained NIC framework. The modification parameters can modify one or more elements of a convolution kernel, one or more rows of a convolution kernel, and / or one or more columns of a convolution kernel. The modification parameters can be scaling factors, offsets, and / or other parameters, etc. The scaling factor can modify the set of pre-trained parameters by multiplying the set of pre-trained parameters by the scaling factor to obtain a set of replacement parameters. The offset can modify the set of pre-trained parameters by adding the offset to a set of pre-trained parameters to obtain a set of replacement parameters. A set of replacement parameters can replace a set of pre-trained parameters in one or more corresponding neural networks. In some embodiments, the scaling factor can be applied to the offset.

[0183] In some examples, it may be advantageous to modify a relatively large number of parameters using a single modification parameter when similar changes to the relatively large number of parameters may improve coding efficiency.

[0184] In some embodiments, one or more modification parameters (e.g., scaling factors, offsets) may be applied to modify one or more feature maps, which are one or more outputs of one or more convolutional layers. The modification parameters may modify the entire feature map (e.g., a 2D matrix including the elements) or a subset of the elements in the feature map. The modification parameters may modify one or more elements in the feature map. The modification parameters may modify one or more rows of the feature map. The modification parameters may modify one or more columns of the feature map. The feature map may be modified by one or more modification parameters. The modification parameters may be used to modify multiple portions of one or more feature maps or multiple feature maps.

[0185] In one embodiment, which feature map (e.g., indication information of the NN of the main decoder network (915), NN layer, CNN channel) and / or which part of the feature map (e.g., indication information of which elements, which rows and / or which columns) are modified by one or more modification parameters is signaled in the bitstream. In one embodiment, which feature map and / or which part of the feature map is modified by one or more modification parameters is predetermined and known to the video decoder, and is therefore not signaled.

[0186] One or more modification parameters may be determined (e.g., trained or fine-tuned) based on one or more images to be encoded and / or transmitted, wherein the one or more images may be different from a set of training images. The one or more modification parameters may be trained by optimizing a joint RD loss L based on the one or more images. In one embodiment, after the one or more modification parameters have been determined through a training process, neural network update information is encoded into a bitstream to indicate the one or more modification parameters or a subset of the one or more modification parameters. The one or more modification parameters correspond to one or more sets of pre-trained parameters, wherein the one or more sets of pre-trained parameters are modified by the one or more modification parameters to generate one or more sets of replacement parameters. In one example, the NIC framework (900) is updated (or fine-tuned), wherein the one or more sets of pre-trained parameters are respectively replaced by one or more sets of replacement parameters.

[0187] In the first scenario, the one or more modification parameters include a first subset of the one or more modification parameters and a second subset of the one or more modification parameters. The first subset of the one or more modification parameters respectively corresponds to a first subset of one or more sets of pre-trained parameters. The second subset of the one or more modification parameters respectively corresponds to a second subset of one or more sets of pre-trained parameters. In the first scenario, the first subset of one or more sets of pre-trained parameters is used in a pre-trained video encoder, and the second subset of one or more sets of pre-trained parameters is used in a pre-trained video decoder.

[0188] On the encoder side, the first subset of one or more groups of pre-trained parameters is modified by the first subset of one or more modified parameters, for example, by multiplying with a scaling factor, adding with an offset, or by a combination of multiplication and addition. The first subset of one or more groups of pre-trained parameters can be replaced by the first subset of one or more groups of replacement parameters. Therefore, the pre-trained video encoder can be updated to an updated video encoder through a training process. The updated video encoder can be used to encode one or more images.

[0189] Neural network update information indicating a second subset of the one or more modification parameters may be encoded and signaled in the bitstream. The one or more encoded images and the neural network update information may be transmitted in the bitstream.

[0190] On the decoder side, the pre-trained video decoder receives the neural network update information and decodes it to determine the second subset of the one or more modified parameters. When the second subset of one or more sets of pre-trained parameters in the pre-trained video decoder is respectively modified by the second subset of the one or more modified parameters, the pre-trained video decoder is updated to an updated video decoder. In one example, the second subset of one or more sets of pre-trained parameters in the pre-trained video decoder is replaced by the second subset of one or more sets of replacement parameters. The updated video decoder can be used to decode one or more encoded images.

[0191] FIG. 16A to FIG. 16B An example of a first scenario is shown. For example, the one or more modification parameters include a first modification parameter (e.g., a scaling factor of 1.1) and a second modification parameter (e.g., an offset of 0.1). The scaling factor modifies a first set of pre-trained parameters in a pre-trained context model NN (916), the first set including N1 pre-trained parameters. The offset modifies a second set of pre-trained parameters in a pre-trained main decoder network (915), the second set including N2 pre-trained parameters.

[0192] The first subset of one or more modified parameters includes a scaling factor, and the second subset of one or more modified parameters includes a scaling factor and an offset. The scaling factor modifies a first subset of one or more groups of pre-trained parameters (e.g., the first set of pre-trained parameters). The scaling factor and the offset respectively modify a second subset of one or more groups of pre-trained parameters (e.g., the first set of pre-trained parameters and the second set of pre-trained parameters). Accordingly, on the encoder side, N1 pre-trained parameters in the pre-trained context model NN (916) can be replaced by N1 corresponding replacement parameters, so that the pre-trained video encoder (1600A) can be updated. The pre-trained context model NN (916) is also updated to an updated context model NN (916). On the decoder side, N1 pre-trained parameters can be replaced by N1 corresponding replacement parameters, and N2 pre-trained parameters can be replaced by N2 corresponding replacement parameters, the pre-trained context model NN (916) is updated to an updated context model NN (916), and the pre-trained main decoder network (915) is updated to an updated main decoder network (915). Therefore, the pre-trained video decoder (1600B) can be updated.

[0193] In a second scenario, none of the one or more modification parameters are used to modify one or more pre-trained parameters in a pre-trained video encoder on the encoder side. Instead, the one or more modification parameters are used to modify one or more sets of pre-trained parameters in a pre-trained video decoder on the decoder side. Therefore, the pre-trained video encoder is not updated and continues to function as a pre-trained video encoder after the training process. In one embodiment, the neural network update information indicates the one or more modification parameters. One or more images may be encoded using the pre-trained video encoder and transmitted in a bitstream together with the neural network update information.

[0194] On the decoder side, the pre-trained video decoder can receive the neural network update information and decode it to determine one or more modified parameters. When one or more sets of pre-trained parameters in the pre-trained video decoder are modified by one or more modified parameters or replaced by one or more sets of replacement parameters, the pre-trained video decoder is updated to an updated video decoder. The updated video decoder can be used to decode the one or more encoded images.

[0195] FIG. 16A to FIG. 16B An example of a second scenario is shown. For example, the one or more modification parameters include a second modification parameter that modifies N2 pre-trained parameters in the pre-trained main decoder network (915). Therefore, none of the one or more modification parameters is used in the pre-trained video encoder on the encoder side (e.g., the pre-trained video encoder (1600A)). Therefore, the pre-trained video encoder (1600A) continues to serve as a pre-trained video encoder after the training process. On the decoder side, the N2 pre-trained parameters can be modified by the second modification parameter or replaced by N2 corresponding replacement parameters, which process updates the pre-trained main decoder network (915). Therefore, the pre-trained video decoder (1600B) can be updated.

[0196] In a third scenario, one or more modified parameters modify one or more sets of pre-trained parameters in the pre-trained video encoder, and one or more pre-trained parameters in the pre-trained video decoder are not modified. Therefore, the pre-trained video encoder is updated to an updated video encoder through a training process. One or more images can be encoded using the updated video encoder and transmitted in a bitstream. No neural network update information is encoded into the bitstream. On the decoder side, the pre-trained video decoder is not updated and continues to act as a pre-trained video decoder. The pre-trained video decoder can be used to decode one or more encoded images.

[0197] FIG. 16A to FIG. 16BAn example of a third scenario is shown. For example, the one or more modification parameters include modification parameters that modify a set of pre-trained parameters in the pre-trained master encoder network (911). Accordingly, a set of pre-trained parameters in the pre-trained master encoder network (911) can be replaced by a set of replacement parameters so that the pre-trained video encoder (1600A) can be updated to an updated video encoder (1600A). The pre-trained master encoder network (911) is also updated to an updated master encoder network (911). On the decoder side, the pre-trained video decoder (1600B) is not updated.

[0198] In various examples, such as described in the first, second, and third scenarios, video decoding may be performed by pre-trained decoders having different capabilities, including decoders with and without the ability to update pre-trained parameters.

[0199] In one example, compression performance can be improved by encoding and decoding one or more images using an updated video encoder and / or an updated video decoder compared to encoding and decoding one or more images using a pre-trained video encoder and a pre-trained video decoder. Therefore, the content-adaptive online training method can be used to adapt a pre-trained NIC framework (e.g., a pre-trained NIC framework (900)) to target image content (e.g., one or more images to be transmitted) and thereby fine-tune the pre-trained NIC framework. Accordingly, the video encoder on the encoder side and / or the video decoder on the decoder side can be updated.

[0200] The content-adaptive online training method can be used as a pre-processing step (e.g., a pre-encoding step) to improve the compression performance of the pre-trained E2E NIC compression method.

[0201] In one embodiment, the one or more images include a single input image, and the fine-tuning process is performed using the single input image. One or more modification parameters, and thus the NIC framework (900), are trained and updated (e.g., fine-tuned) based on the single input image. The updated video encoder on the encoder side and / or the updated video decoder on the decoder side can be used to encode and decode the single input image and, optionally, other input images. The neural network update information can be encoded into the bitstream together with the encoded single input image.

[0202] In one embodiment, the one or more images include multiple input images, and the multiple input images are fine-tuned. Based on the multiple input images, one or more modification parameters, and thus the NIC framework (900), are trained and updated (e.g., fine-tuned). The updated video encoder on the encoder side and / or the updated decoder on the decoder side can be used to encode the multiple input images and, optionally, other input images. The neural network update information can be encoded into the bitstream together with the encoded multiple input images.

[0203] As the signaling of the neural network update information in the bitstream increases, the rate loss R also increases. When one or more images include a single input image, the neural network update information for each encoded image is represented by a signal, and a first increase in the rate loss R is used to indicate the increase in the rate loss R due to the signaling of the neural network update information for each image. When one or more images include multiple input images, the neural network update information for the multiple input images is represented by a signal and shared by the multiple input images, and a second increase in the rate loss R is used to indicate the increase in the rate loss R due to the signaling of the neural network update information for each image. Because the neural network update information is shared by multiple input images, the second increase in the rate loss R can be less than the first increase in the rate loss R. Therefore, in some examples, it may be more advantageous to use multiple input images to fine-tune one or more modification parameters and thereby fine-tune the NIC framework.

[0204] In one embodiment, the one or more modification parameters to be trained are used to modify one or more sets of pre-trained parameters in one component of the pre-trained NIC framework (900). Thus, the one component of the pre-trained NIC framework (900) is updated based on the one or more modification parameters, while other components of the pre-trained NIC framework (900) are not updated.

[0205] The one component may be a pre-trained context model NN (916), a pre-trained entropy parameter NN (917), a pre-trained main encoder network (911), a pre-trained main decoder network (915), a pre-trained super encoder (921), or a pre-trained super decoder (925). The pre-trained video encoder and / or pre-trained video decoder may be updated based on which components in the pre-trained NIC framework (900) are updated.

[0206] In one example, the one or more modified parameters to be trained are used to modify one or more groups of pre-trained parameters in the pre-trained context model NN (916), so that the pre-trained context model NN (916) is updated, while the remaining components (911), (915), (921), (917) and (925) are not updated. In one example, the pre-trained video encoder on the encoder side and the pre-trained video decoder on the decoder side include the pre-trained context model NN (916), and so both the pre-trained video encoder and the pre-trained video decoder are updated.

[0207] In one example, the one or more modified parameters to be trained are used to modify one or more sets of pre-trained parameters in the pre-trained super decoder (925), so that the pre-trained super decoder (925) is updated, while the remaining components (911), (915), (916), (917) and (921) are not updated. Therefore, the pre-trained video encoder is not updated, but the pre-trained video decoder is updated.

[0208] In one embodiment, the one or more modification parameters to be trained are used to modify one or more groups of pre-trained parameters in multiple components of the pre-trained NIC framework (900). Therefore, multiple components of the pre-trained NIC framework (900) are updated based on the one or more modification parameters. In one example, the multiple components of the pre-trained NIC framework (900) include all components configured with a neural network (e.g., DNN, CNN). In one example, the multiple components of the pre-trained NIC framework (900) include CNN-based components: a pre-trained main encoder network (911), a pre-trained main decoder network (915), a pre-trained context model NN (916), a pre-trained entropy parameter NN (917), a pre-trained super encoder (921) and a pre-trained super decoder (925).

[0209] As described above, in one example, the one or more modification parameters to be updated modify one or more sets of pre-trained parameters in a pre-trained video encoder of a pre-trained NIC framework (900). In one example, the one or more modification parameters to be updated are used to modify one or more sets of pre-trained parameters in a pre-trained video decoder of the NIC framework (900). In one example, the one or more modification parameters to be updated are used to modify one or more sets of pre-trained parameters in a pre-trained video encoder and a pre-trained video decoder of a pre-trained NIC framework (900).

[0210] Each neural network-based component in the NIC framework (900) (e.g., context model NN (916), entropy parameter NN (917), main encoder network (911), main decoder network (915), super encoder (921), or super decoder (925)) can be configured with appropriate parameters, such as respective weights, biases, or a combination of weights and biases. In some embodiments, modification parameters are used to modify weights and biases in one or more of the neural network-based components. When one or more CNNs are used, the weights may include elements in a convolution kernel. One or more types of parameters may be used to specify a neural network. In one embodiment, one or more modification parameters are used to modify one or more bias items, and only the one or more bias items are modified by the one or more modification parameters. In one embodiment, one or more modification parameters modify weights, and only weights are modified by the one or more modification parameters. In one embodiment, one or more modification parameters are used to modify weights and one or more bias items.

[0211] In one embodiment, the one or more modified parameters include a scaling factor, and the set of pre-trained parameters to be updated is modified by the scaling factor. The scaling factor can modify a set of pre-trained parameters, such as a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases, etc. After the training process, the scaling factor is determined, and the set of pre-trained parameters can be replaced with a set of replacement parameters, respectively, wherein the set of replacement parameters is a set of pre-trained parameters multiplied by the same scaling factor.

[0212] A scaling factor may be used to modify a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases of one or more of the neural network-based components in a NIC framework (e.g., NIC framework (900)).

[0213] In one embodiment, a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in one of the components based on the neural network in a NIC framework (e.g., NIC framework (900)) is modified using a scaling factor. For example, a set of pre-trained weights includes a 5×5 convolution kernel for a specified channel used in a context model NN (916). The scaling factor (e.g., 1.1) is obtained through a context-based online training process. Therefore, a set of pre-trained weights (e.g., a 5×5 convolution kernel for a specified channel used in the context model NN (916)) is updated to a set of replacement weights. The set of replacement weights can be a set of pre-trained weights multiplied by a scaling factor (e.g., 1.1).

[0214] In one example, a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in a CNN based on a component of a neural network in a NIC framework (e.g., NIC framework (900)) are modified using a scaling factor. Figures 10 to 15 , the CNN based on the neural network component in the NIC framework (900) can be Fig.10 The main encoder network (911) shown in CNN, Fig.11 The CNN of the main decoder network (911) shown in Fig.12 The CNN of the super encoder (921) shown in Fig.13 The CNN of the super decoder (925) shown in Fig.14 A CNN of the context model NN (916) shown in FIG. 1 , or Fig.15 The entropy parameters of the CNN NN (917) are shown in FIG.

[0215] In one example, a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in a convolutional layer or a deconvolutional layer in a CNN based on a component of a neural network in a NIC framework (e.g., NIC framework (900)) is modified using a scaling factor, wherein the convolutional layer or the deconvolutional layer is, for example, Fig.13 The third deconvolution layer (DeConv: 3×3 c384 s1) in the CNN of the main decoder network (915) shown in FIG.

[0216] In one example, a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases of a channel in a convolutional layer or a deconvolutional layer in a CNN of a neural network-based component in a NIC framework (e.g., NIC framework (900)) is modified using a scaling factor, the channel being, for example, Fig.13 The first channel (e.g., c1) of the third deconvolutional layer (DeConv: 3×3 c384 S1) in the CNN of the main decoder network (915) shown in FIG.

[0217] In one example, a set of pre-trained weights in a channel of a convolutional layer or a deconvolutional layer in a CNN of a neural network-based component in a NIC framework (e.g., NIC framework (900)) is modified using a scaling factor. The set of pre-trained weights in the channel (e.g., c1) may include multiple elements, one or more rows and / or one or more columns, etc. In one example, the set of pre-trained weights in the channel (e.g., c1) includes all elements (e.g., 9 elements) in the channel.

[0218] A set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases of a plurality of components in a neural network-based component in a NIC framework (e.g., the NIC framework (900)) may be modified using a scaling factor. For example, the scaling factor is obtained through a context-based online training process. Thus, the set of pre-trained weights is updated to a set of replacement weights based on the scaling factor. The set of replacement weights may be a set of pre-trained weights (e.g., a 5×5 convolution kernel for a specific channel used in the context model NN (916) and a 5×5 convolution kernel for a specific channel in the first deconvolution layer used in the main decoder network (915)) multiplied by a scaling factor (e.g., 1.1).

[0219] In one embodiment, a scaling factor may be utilized to modify a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in all neural network-based components in a NIC framework (e.g., NIC framework (900)).

[0220] The present description may be suitably adapted to accommodate another scaling factor for modifying another set of pre-trained parameters, such as another set of pre-trained weights, another set of pre-trained biases, another set of pre-trained weights and pre-trained biases of one or more of the neural network-based components in a NIC framework (e.g., NIC framework (900)).

[0221] In some embodiments, one or more scaling factors are determined by a context-based online training process and can be used to modify one or more feature maps. The scaling factor can modify the entire feature map or a subset of the feature map. The scaling factor can modify the one or more elements in the feature map by multiplying them respectively. Therefore, the one or more elements in the feature map are replaced by one or more elements multiplied by the scaling factor. The scaling factor can modify one or more rows of the feature map. The scaling factor can modify one or more columns of the feature map. The feature map can be modified by one or more scaling factors. The scaling factor can be used to modify one or more feature maps or multiple parts of multiple feature maps.

[0222] In one embodiment, the one or more modified parameters include an offset, and the set of pre-trained parameters to be updated is modified by the offset. The offset may modify the set of pre-trained parameters, such as a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases, etc. After the training process, the offset is determined, and the set of pre-trained parameters may be replaced with a set of replacement parameters, respectively, wherein the set of replacement parameters is a set of pre-trained parameters plus the same offset.

[0223] The offset may be utilized to modify a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases of one or more of the neural network-based components in a NIC framework (e.g., NIC framework (900)).

[0224] In one embodiment, an offset is applied to modify a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases of one of the neural network-based components in a NIC framework (e.g., the NIC framework (900)). For example, a set of pre-trained weights includes a 5×5 convolution kernel for a specified channel used in the context model NN (916). The offset (e.g., 0.1) is obtained through a context-based online training process. Therefore, a set of pre-trained weights (e.g., a 5×5 convolution kernel for a specified channel used in the context model NN (916)) is updated to a set of replacement weights. A set of replacement weights can be a set of pre-trained weights plus an offset (e.g., 0.1) respectively.

[0225] In one example, an offset is used to modify a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in a CNN of a neural network-based component in a NIC framework (e.g., NIC framework (900)). Figures 10 to 15 , the CNN of the neural network based component in the NIC framework (900) can be Fig.10 The main encoder network (911) shown in CNN, Fig.11 The CNN of the main decoder network (911) shown in Fig.12 The CNN of the super encoder (921) shown in Fig.13 The CNN of the super decoder (925) shown in Fig.14 The context model NN (916) shown in FIG. Fig.15 The entropy parameters of the CNN NN (917) are shown in FIG.

[0226] In one example, the offset is used to modify a convolutional layer or a deconvolutional layer (e.g., a convolutional layer or a deconvolutional layer) in a CNN of a neural network-based component in a NIC framework (e.g., NIC framework (900)). Fig.13 A set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in the third deconvolution layer (DeConv: 3×3 c384 s1)) in the CNN of the main decoder network (915) shown in .

[0227] In one example, an offset is utilized to modify a set of pre-trained biases in a convolutional layer or a deconvolutional layer in a CNN of a neural network-based component in a NIC framework (e.g., NIC framework (900)).

[0228] In one example, an offset is used to modify a channel (e.g., Fig.13 A set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in the first channel (e.g., c1)) of the third deconvolution layer (DeConv: 3×3 c384 S1) in the CNN of the main decoder network (915) shown in .

[0229] An offset can be applied to modify a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in multiple components of a neural network-based component in a NIC framework (e.g., the NIC framework (900)). For example, the offset is obtained through a context-based online training process. Therefore, a set of pre-trained weights is updated to a set of replacement weights based on the offset. The set of replacement weights can be a set of pre-trained weights (e.g., a 5×5 convolution kernel for a specified channel used in the context model NN (916) and a 5×5 convolution kernel for a specified channel in the first deconvolution layer used in the main decoder network (915)) plus an offset (e.g., 0.1).

[0230] In one embodiment, an offset may be applied to modify a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in all neural network-based components in a NIC framework (e.g., NIC framework (900)).

[0231] This description may be suitably adapted to apply to modifying another set of pre-trained parameters, such as another set of pre-trained weights, another set of pre-trained biases, or another set of pre-trained weights and pre-trained offsets in one or more of the neural network-based components in a NIC framework (e.g., NIC framework (900)).

[0232] In some embodiments, one or more offsets may be applied to modify one or more feature maps. An offset may modify an entire feature map or a subset of a feature map. An offset may modify one or more elements in a feature map by adding the offset to one or more elements, respectively. Thus, one or more elements in a feature map are replaced by one or more replacement elements, which are one or more elements plus a scaling factor, respectively. An offset may modify one or more rows of a feature map. An offset may modify one or more columns of a feature map. A feature map may be modified by one or more offsets. An offset may be used to modify one or more feature maps or portions of a feature map.

[0233] In one embodiment, the one or more modified parameters include a scaling factor and a corresponding offset, and the set of pre-trained parameters to be updated are modified by the scaling factor and the corresponding offset. The scaling factor and the corresponding offset can modify a set of pre-trained parameters, such as a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases, etc. After the training process, the scaling factor and the corresponding offset are determined, and the set of pre-trained parameters can be replaced with a set of replacement parameters, respectively.

[0234] A set of replacement parameters can be determined based on a set of pre-trained parameters, a scaling factor, and a corresponding offset. In one example, a set of replacement parameters is a set of intermediate parameters plus an offset, and a set of intermediate parameters is a set of pre-trained parameters multiplied by a scaling factor. In one example, a set of replacement parameters is the product of a set of intermediate parameters and a scaling factor, and a set of intermediate parameters is a set of pre-trained parameters and an offset.

[0235] A set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in one or more of the neural network-based components in a NIC framework (e.g., NIC framework (900)) may be modified using a scaling factor and a corresponding offset.

[0236] In one embodiment, a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases of one of the neural network-based components in a NIC framework (e.g., the NIC framework (900)) is modified using a scaling factor and a corresponding offset. For example, a set of pre-trained weights includes a 5×5 convolution kernel for a specified channel used in the context model NN (916). The scaling factor (e.g., 1.1) and the corresponding offset (e.g., 0.1) are obtained through a context-based online training process. As described above, a set of pre-trained weights (e.g., a 5×5 convolution kernel for a specific channel used in the context model NN (916)) is thus updated to a set of replacement weights.

[0237] In one example, a set of replacement parameters is the sum of a set of intermediate parameters and an offset (eg, 0.1), and a set of intermediate parameters is a set of pre-trained parameters multiplied by a scaling factor (eg, 1.1).

[0238] In one example, a set of replacement parameters is the product of a set of intermediate parameters and a scaling factor (eg, 1.1), and a set of intermediate parameters is the sum of a set of pre-trained parameters and an offset (eg, 0.1).

[0239] In one example, a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in a CNN of a neural network-based component in a NIC framework (e.g., NIC framework (900)) are modified using a scaling factor and a corresponding offset.

[0240] In one example, a convolutional layer or a deconvolutional layer (e.g., a convolutional layer) in a CNN of a neural network-based component in a NIC framework (e.g., NIC framework (900)) is modified using a scaling factor and a corresponding offset. Fig.13 A set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in the third deconvolution layer (DeConv: 3×3 c384 s1)) in the CNN of the main decoder network (915) shown in .

[0241] In one example, a channel (e.g., Fig.13 A set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in the first channel (e.g., c1)) of the third deconvolution layer (DeConv: 3×3 c384 S1) in the CNN of the main decoder network (915) shown in .

[0242] A set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and pre-trained biases in multiple components of a neural network-based component in a NIC framework (e.g., the NIC framework (900)) can be modified using a scaling factor and a corresponding offset. For example, the scaling factor and the corresponding offset are obtained through a context-based online training process. Therefore, based on the scaling factor and the corresponding offset, the set of pre-trained weights are updated to a set of replacement weights. As described above, the set of replacement weights can be a set of pre-trained weights modified by a scaling factor (e.g., 1.1) and an offset (e.g., 0.1) (e.g., a 5×5 convolution kernel of a specific channel used in the context model NN (916) and a 5×5 convolution kernel of a specific channel in the first deconvolution layer used in the main decoder network (915)).

[0243] In one embodiment, a set of pre-trained weights, a set of pre-trained biases, or a set of pre-trained weights and a set of pre-trained biases in all neural network-based components in a NIC framework (e.g., NIC framework (900)) may be modified using a scaling factor and a corresponding offset.

[0244] This description may be suitably adapted to apply to another scaling factor and corresponding offset for modifying another set of pre-trained parameters in one or more of the neural network-based components in a NIC framework (e.g., NIC framework (900)).

[0245] In some embodiments, one or more feature maps may be modified using one or more scaling factors and corresponding one or more offsets. A scaling factor and corresponding offset may modify an entire feature map or a subset of a feature map. A scaling factor and corresponding offset may modify one or more elements in a feature map. A scaling factor and corresponding offset may modify one or more rows of a feature map. A scaling factor and corresponding offset may modify one or more columns of a feature map. A feature map may be modified using one or more scaling factors and corresponding one or more offsets. A scaling factor and corresponding offset may be used to modify multiple portions of one or more feature maps or multiple feature maps.

[0246] The one or more modification parameters to be determined based on the training process may include one or more scaling factors and one or more offsets. In one example, each value in the one or more scaling factors and the one or more offsets modifies a different set of pre-trained parameters, as described above. In one example, a scaling factor in the one or more scaling factors and an offset in the one or more offsets modify the same set of pre-trained parameters, as described above. In one example, each amount in the one or more scaling factors and the one or more offsets modifies a different feature map or a different portion of a feature map, as described above. In one example, a scaling factor in the one or more scaling factors and an offset in the one or more offsets modify the same feature map or the same portion of a feature map, as described above.

[0247] The fine-tuning process may include multiple training epochs (e.g., iterations), wherein one or more modification parameters (e.g., scaling factors, offsets) are updated or determined during the repeated fine-tuning process. The fine-tuning process may stop when the training loss has flattened or is about to flatten. In one example, the fine-tuning process stops when the training loss (e.g., RD ​​loss L) is below a first threshold. In one example, the fine-tuning process stops when the difference between two consecutive training losses is below a second threshold.

[0248] Two hyperparameters (e.g., step size and maximum number of steps) can be used in the fine-tuning process together with the loss function (e.g., RD ​​loss L). As described above, in one example, the rate loss R in the loss function L during the fine-tuning process is estimated based on, for example, noise injection and a BPP estimator. The maximum number of iterations can be used as a threshold for the maximum number of iterations for terminating the fine-tuning process. In one example, when the number of iterations reaches the maximum number of iterations, the fine-tuning process stops.

[0249] The step size may indicate the learning rate of an online training process (e.g., an online fine-tuning process). The step size may be used in a gradient descent algorithm or a back propagation calculation performed during the fine-tuning process. For example, each adjustment to the modified parameter being trained is based on the step size used in the gradient descent algorithm and the corresponding gradient (e.g., the gradient of the RD loss with respect to the modified parameter being trained). The step size may be determined using any suitable method. In one embodiment, different step sizes are used for images with different types of content to obtain optimal results. Different types may refer to different variances. In one example, the step size is determined based on the variance of the image used to update the NIC framework. For example, the step size of an image with a high variance is greater than the step size of an image with a low variance, where the high variance is greater than the low variance.

[0250] In one embodiment, a first step size may be used to run a certain number of iterations (e.g., 100). Then, a second step size (e.g., the first step size plus or minus the size increment) may be used to run a certain number of iterations. The results obtained with the first step size and the second step size may be compared to determine the step size to use. More than two step sizes may be tested to determine the best step size.

[0251] The step size may vary during the fine-tuning process. The step size may have an initial value at the beginning of the fine-tuning process, and the initial value may be reduced (e.g., halved) at a later stage of the fine-tuning process (e.g., after a certain number of iterations) to achieve finer tuning. During iterative online training, the step size or learning rate may be changed by a scheduler. The scheduler may include a parameter adjustment method for adjusting the step size. The scheduler may determine the value of the step size so that the step size may increase, decrease, or remain constant over multiple time intervals. In one example, the learning rate is changed by the scheduler in each step. Different images may use a single scheduler or multiple different schedulers. Therefore, multiple sets of replacement parameters may be generated based on multiple schedulers, and a set of replacement parameters having better compression performance (e.g., smaller RD loss) may be selected from the multiple sets of replacement parameters.

[0252] At the end of the fine-tuning process, one or more updated parameters may be calculated for one or more modification parameters (e.g., scaling factors or offsets), respectively. In one embodiment, the one or more updated parameters are calculated as the difference between the one or more modification parameters and the corresponding one or more "pre-trained" modification parameters (e.g., 1 for scaling factors and 0 for offsets). In one embodiment, the one or more updated parameters are the one or more modification parameters, respectively.

[0253] One or more updated parameters can be generated from one or more modified parameters, respectively, for example using some linear or non-linear transformation, and the one or more updated parameters are one or more representative parameters generated based on the one or more modified parameters. The one or more modified parameters are transformed into one or more updated parameters for better compression.

[0254] In one example, the one or more updated parameters may be compressed, for example, using LZMA2, which is a variation of the Lempel–Ziv–Markov chain algorithm (LZMA) or the bzip2 algorithm, etc. In one example, the compression step is omitted for the one or more updated parameters.

[0255] In some embodiments, one or more updated parameters or a second subset of one or more updated parameters may be encoded into the bitstream as neural network update information, wherein the neural network update information indicates one or more replacement parameters or a second subset of one or more replacement parameters.

[0256] The updated parameters corresponding to the modified parameters can be encoded into the bitstream as neural network update information, where the neural network update information indicates the modified parameters. In one example, a set of pre-trained parameters modified by the modified parameters include one or more pre-trained decoder parameters in the pre-trained decoder parameters configured for the pre-trained video decoder. Therefore, the neural network update information indicating the modified parameters is encoded into the bitstream and represented by a signal. In one example, a set of pre-trained parameters modified by the modified parameters does not include any pre-trained decoder parameters in the pre-trained decoder parameters configured for the pre-trained video decoder. Therefore, the neural network update information indicating the modified parameters is not encoded and is not represented by a signal.

[0257] After the fine-tuning process, in some examples, the pre-trained video encoder on the encoder side can be updated or fine-tuned based on the modified parameters. For example, a set of pre-trained parameters modified by the modified parameters includes one or more pre-trained encoder parameters configured for the pre-trained video encoder, and thus one or more pre-trained encoder parameters can be updated based on the modified parameters, and the pre-trained video encoder is updated. The input image (e.g., one of the one or more images used in the fine-tuning process) can be encoded into a bitstream using the updated video encoder. Therefore, both the encoded image and the neural network update information are included in the bitstream.

[0258] If applicable, in one example, the neural network update information is decoded (e.g., decompressed) by the pre-trained video decoder to obtain updated parameters. In one example, the modified parameters can be obtained based on the relationship between the updated parameters and the modified parameters described above. As described above, the pre-trained video decoder can be modified, and the updated video decoder can be used to decode the encoded image.

[0259] The NIC framework may include any type of neural network and use any neural network-based image compression method, such as the context-hyper-prior encoder-decoder framework (e.g., Fig. 9 The NIC framework shown in the figure), the scale-hyper-prior encoder-decoder framework, the Gaussian mixture likelihood framework and the deformation of the Gaussian mixture likelihood framework, the RNN-based recursive compression method and the deformation of the RNN-based recursive compression method, etc.

[0260] Compared with the related E2E image compression method, the content adaptive online training method and device in the present application can have the following benefits. An adaptive online training mechanism can be used to improve NIC coding efficiency. The use of a flexible and universal framework can be compatible with various types of pre-training frameworks and quality metrics. For example, certain pre-training parameters in various types of pre-training frameworks can be replaced by one or more modification parameters, which are determined by online training using images to be encoded and transmitted.

[0261] B. Type II Content Adaptive Online Training

[0262] One or more of the NIC pre-trained parameters in the pre-trained NIC framework (900) can be further trained (e.g., fine-tuned) using one or more images to be encoded and / or transmitted, wherein the one or more images may be different from the training image set. One or more pre-trained parameters used in the NIC pre-trained parameters can be fine-tuned by optimizing the joint RD loss L based on the one or more images. One or more pre-trained parameters that have been fine-tuned using the one or more images are referred to as one or more replacement parameters or one or more fine-tuning parameters. In one embodiment, after one or more of the NIC pre-trained parameters have been fine-tuned (e.g., replaced) by one or more replacement parameters, neural network update information is encoded into the bitstream to indicate one or more replacement parameters or a subset of one or more replacement parameters. In one example, the NIC framework (900) is updated (or fine-tuned), wherein one or more pre-trained parameters are replaced by one or more replacement parameters, respectively.

[0263] In a first scenario, the one or more pre-trained parameters include a first subset of the one or more pre-trained parameters and a second subset of the one or more pre-trained parameters. The one or more replacement parameters include a first subset of the one or more replacement parameters and a second subset of the one or more replacement parameters.

[0264] A first subset of one or more pre-trained parameters is used in a pre-trained video encoder and, for example, replaced by a first subset of one or more replacement parameters during a training process. Thus, the pre-trained video encoder is updated to an updated video encoder through a training process. The neural network update information may indicate a second subset of one or more replacement parameters to replace a second subset of one or more replacement parameters. The one or more images may be encoded using the updated video encoder and transmitted in a bitstream together with the neural network update information.

[0265] On the decoder side, the second subset of one or more pre-trained parameters is used in the pre-trained video decoder. In one embodiment, the pre-trained video decoder receives the neural network update information and decodes it to determine the second subset of one or more replacement parameters. When the second subset of one or more pre-trained parameters in the pre-trained video decoder is replaced by the second subset of one or more replacement parameters, the pre-trained video decoder is updated to an updated video decoder. The updated video decoder can be used to decode the one or more encoded images.

[0266] FIG. 16A to FIG. 16B An example of the first scenario is shown. For example, the one or more pre-trained parameters include N1 pre-trained parameters in the pre-trained context model NN (916) and N2 pre-trained parameters in the pre-trained main decoder network (915). Therefore, a first subset of the one or more pre-trained parameters includes the N1 pre-trained parameters, and a second subset of the one or more pre-trained parameters is the same as the one or more pre-trained parameters. Accordingly, the N1 pre-trained parameters in the pre-trained context model NN (916) can be replaced by N1 corresponding replacement parameters, so that the pre-trained video encoder (1600A) can be updated to the updated video encoder (1600A). The pre-trained context model NN (916) is also updated to the updated context model NN (916). On the decoder side, N1 pre-trained parameters may be replaced by N1 corresponding replacement parameters, and N2 pre-trained parameters may be replaced by N2 corresponding replacement parameters, the pre-trained context model NN (916) is updated to the updated context model NN (916), and the pre-trained main decoder network (915) is updated to the updated main decoder network (915). Therefore, the pre-trained video decoder (1600B) may be updated to the updated video decoder (1600B).

[0267] In a second scenario, none of the one or more pre-trained parameters are used in a pre-trained video encoder on the encoder side. Instead, the one or more pre-trained parameters are used in a pre-trained video decoder on the decoder side. Therefore, the pre-trained video encoder is not updated and continues to function as a pre-trained video encoder after the training process. In one embodiment, the neural network update information indicates one or more replacement parameters. One or more images may be encoded using the pre-trained video encoder and transmitted in a bitstream along with the neural network update information.

[0268] On the decoder side, the pre-trained video decoder can receive the neural network update information and decode it to determine one or more replacement parameters. When one or more pre-trained parameters in the pre-trained video decoder are replaced by one or more replacement parameters, the pre-trained video decoder is updated to an updated video decoder. The updated video decoder can be used to decode one or more encoded images.

[0269] FIG. 16A to FIG. 16B An example of a second scenario is shown. For example, the one or more pre-trained parameters include N2 pre-trained parameters in the pre-trained main decoder network (915). Therefore, none of the one or more pre-trained parameters are used in the pre-trained video encoder (e.g., pre-trained video encoder (1600A)) on the encoder side. Therefore, the pre-trained video encoder (1600A) continues to serve as a pre-trained video encoder after the training process. On the decoder side, the N2 pre-trained parameters can be replaced by N2 corresponding replacement parameters, which updates the pre-trained main decoder network (915) to the updated main decoder network (915). Therefore, the pre-trained video decoder (1600B) can be updated to the updated video decoder (1600B).

[0270] In a third scenario, one or more pre-trained parameters are used in a pre-trained video encoder, and the one or more pre-trained parameters are replaced by one or more replacement parameters, for example, during a training process. Thus, the pre-trained video encoder is updated to an updated video encoder through a training process. One or more images may be encoded using the updated video encoder and transmitted in a bitstream. No neural network update information is encoded in the bitstream. On the decoder side, the pre-trained video decoder is not updated and continues to function as a pre-trained video decoder. The pre-trained video decoder may be used to decode one or more encoded images.

[0271] FIG. 16A to FIG. 16BAn example of a third scenario is shown. For example, one or more pre-trained parameters are in the pre-trained main encoder network (911). Accordingly, one or more pre-trained parameters in the pre-trained main encoder network (911) can be replaced by one or more replacement parameters, so that the pre-trained video encoder (1600A) can be updated to the updated video encoder (1600A). The pre-trained main encoder network (911) is also updated to the updated main encoder network (911). On the decoder side, the pre-trained video decoder (1600B) is not updated.

[0272] In various examples, such as described in the first, second, and third scenarios, video decoding may be performed by pre-trained decoders having different capabilities, including decoders with and without the ability to update pre-trained parameters.

[0273] In one example, compression performance can be improved by encoding and decoding one or more images using an updated video encoder and / or an updated video decoder compared to encoding one or more images using a pre-trained video encoder and a pre-trained video decoder. Therefore, the content-adaptive online training method can be used to adapt a pre-trained NIC framework (e.g., a pre-trained NIC framework (900)) to target image content (e.g., one or more images to be transmitted), thereby fine-tuning the pre-trained NIC framework. Accordingly, the video encoder on the encoder side and / or the video decoder on the decoder side can be updated.

[0274] The content-adaptive online training method can be used as a pre-processing step (e.g., a pre-encoding step) to improve the compression performance of the pre-trained E2E NIC compression method.

[0275] In one embodiment, the one or more images include a single input image, and the fine-tuning process is performed using the single input image. The NIC framework (900) is trained and updated (e.g., fine-tuned) based on the single input image. The updated video encoder on the encoder side and / or the updated video decoder on the decoder side can be used to encode and decode the single input image and, optionally, other input images. The neural network update information can be encoded into the bitstream together with the encoded single input image.

[0276] In one embodiment, the one or more images include a plurality of input images, and the fine-tuning process is performed using the plurality of input images. The NIC framework (900) is trained and updated (e.g., fine-tuned) based on the plurality of input images. The updated video encoder on the encoder side and / or the updated decoder on the decoder side can be used to encode the plurality of input images and, optionally, other input images. The neural network update information can be encoded into a bitstream along with the encoded plurality of input images.

[0277] As the signaling of the neural network update information in the bitstream increases, the rate loss R also increases. When one or more images include a single input image, the neural network update information of each encoded image is represented by a signal, and the first increase in the rate loss R is used to indicate the increase in the rate loss R due to the signaling of the neural network update information of each image. When one or more images include multiple input images, the neural network update information of the multiple input images is represented by a signal, and the neural network update information is shared by the multiple input images, and the second increase in the rate loss R is used to indicate the increase in the rate loss R due to the signaling of the neural network update information of each image. Because the neural network update information is shared by multiple input images, the second increase in the rate loss R can be less than the first increase in the rate loss R. Therefore, in some examples, it may be more advantageous to use multiple input images to fine-tune the NIC framework.

[0278] In one embodiment, the one or more pre-trained parameters to be updated are in one component of the pre-trained NIC framework (900). Thus, one component of the pre-trained NIC framework (900) is updated based on the one or more replacement parameters, and other components of the pre-trained NIC framework (900) are not updated.

[0279] The one component may be a pre-trained context model NN (916), a pre-trained entropy parameter NN (917), a pre-trained main encoder network (911), a pre-trained main decoder network (915), a pre-trained super encoder (921), or a pre-trained super decoder (925). The pre-trained video encoder and / or pre-trained video decoder may be updated based on which components in the pre-trained NIC framework (900) are updated.

[0280] In one example, one or more pre-trained parameters to be updated are in the pre-trained context model NN (916), thereby updating the pre-trained context model NN (916), and not updating the remaining components (911), (915), (921), (917) and (925). In one example, the pre-trained video encoder on the encoder side and the pre-trained video decoder on the decoder side include the pre-trained context model NN (916), thereby updating both the pre-trained video encoder and the pre-trained video decoder.

[0281] In one example, one or more pre-trained parameters to be updated are in the pre-trained super decoder (925), so the pre-trained super decoder (925) is updated, and the remaining components (911), (915), (916), (917) and (921) are not updated. Therefore, the pre-trained video encoder is not updated, and the pre-trained video decoder is updated.

[0282] In one embodiment, one or more pre-trained parameters to be updated are in multiple components of the pre-trained NIC framework (900). Therefore, multiple components of the pre-trained NIC framework (900) are updated based on one or more replacement parameters. In one example, the multiple components of the pre-trained NIC framework (900) include all components configured with a neural network (e.g., DNN, CNN). In one example, the multiple components of the pre-trained NIC framework (900) include CNN-based components: a pre-trained main encoder network (911), a pre-trained main decoder network (915), a pre-trained context model NN (916), a pre-trained entropy parameter NN (917), a pre-trained super encoder (921), and a pre-trained super decoder (925).

[0283] As described above, in one example, the one or more pre-trained parameters to be updated are in the pre-trained video encoder of the pre-trained NIC framework (900). In one example, the one or more pre-trained parameters to be updated are in the pre-trained video decoder of the NIC framework (900). In one example, the one or more pre-trained parameters to be updated are in the pre-trained video encoder and the pre-trained video decoder of the pre-trained NIC framework (900).

[0284] The NIC framework (900) may be based on a neural network, for example, one or more components in the NIC framework (900) may include a neural network, such as a CNN and / or a DNN, etc. As described above, a neural network may be specified by different types of parameters (e.g., weights, biases, etc.).

[0285] Each neural network-based component in the NIC framework (900) (e.g., context model NN (916), entropy parameter NN (917), main encoder network (911), main decoder network (915), super encoder (921), or super decoder (925)) can be configured with appropriate parameters, such as corresponding weights, biases, or a combination of weights and biases. When one or more CNNs are used, the weights can include elements in a convolution kernel. One or more types of parameters can be used to indicate the neural network. In one embodiment, the one or more pre-trained parameters to be updated are one or more bias items, and only the one or more bias items are replaced by one or more replacement parameters. In one embodiment, the one or more pre-trained parameters to be updated are multiple weights, and only these weights are replaced by one or more replacement parameters. In one embodiment, the one or more pre-trained parameters to be updated include multiple weights and one or more bias items, and all pre-trained parameters, including the multiple weights and the one or more bias items, are replaced by one or more replacement parameters. In one embodiment, other parameters can be used to indicate the neural network, and these other parameters can be fine-tuned.

[0286] The fine-tuning process may include multiple training rounds (e.g., iterations), wherein one or more pre-trained parameters (e.g., weights, biases) are updated or learned in the iterative fine-tuning process. The fine-tuning process may stop when the training loss has flattened or is about to flatten. In one example, the fine-tuning process stops when the training loss (e.g., RD ​​loss L) is below a first threshold. In one example, the fine-tuning process stops when the difference between two consecutive training losses is below a second threshold.

[0287] As described above in content-adaptive online training of type I, two hyperparameters (e.g., step size and maximum number of steps) can be used in the fine-tuning process together with the loss function (e.g., RD ​​loss L).

[0288] At the end of the fine-tuning process, one or more updated parameters may be calculated for the corresponding one or more replacement parameters. In one embodiment, the one or more updated parameters are calculated as the difference between the one or more replacement parameters and the corresponding one or more pre-trained parameters. In one embodiment, the one or more updated parameters are respectively one or more replacement parameters.

[0289] In one embodiment, one or more replacement parameters may be used to generate one or more updated parameters, for example using some linear or non-linear transformation, and the one or more updated parameters are one or more representative parameters generated based on the one or more replacement parameters. The one or more replacement parameters are transformed into one or more updated parameters for better compression.

[0290] The first subset of the one or more updated parameters corresponds to the first subset of the one or more replacement parameters, and the second subset of the one or more updated parameters corresponds to the second subset of the one or more replacement parameters.

[0291] In one example, the one or more updated parameters may be compressed, for example, using LZMA2, which is a variation of the Lempel–Ziv–Markov chain algorithm (LZMA) or the bzip2 algorithm, etc. In one example, the compression operation is omitted for the one or more updated parameters. In some embodiments, the one or more updated parameters or a second subset of the one or more updated parameters may be encoded into the bitstream as neural network update information, wherein the neural network update information indicates the one or more replacement parameters or a second subset of the one or more replacement parameters.

[0292] After the fine-tuning process, in some examples, the pre-trained video encoder on the encoder side can be updated or fine-tuned based on (i) the first subset of one or more replacement parameters or (ii) the one or more replacement parameters. The updated video encoder can be used to encode an input image (e.g., one of the one or more images used in the fine-tuning process) into a bitstream. Therefore, both the encoded image and the neural network update information are included in the bitstream.

[0293] If applicable, in one example, the neural network update information is decoded (e.g., decompressed) by the pre-trained video decoder to obtain one or more updated parameters or a second subset of one or more updated parameters. In one example, one or more replacement parameters or a second subset of one or more replacement parameters can be obtained based on the relationship between the one or more updated parameters and the one or more replacement parameters described above. As described above, the pre-trained video decoder can be fine-tuned, and the updated video decoder can be used to decode the encoded image.

[0294] The NIC framework may include any type of neural network and use any neural network-based image compression method, such as the context-hyper-prior encoder-decoder framework (e.g., Fig. 9 The NIC framework shown in the figure), the scale-hyper-prior encoder-decoder framework, the Gaussian mixture likelihood framework and the deformation of the Gaussian mixture likelihood framework, the RNN-based recursive compression method and the deformation of the RNN-based recursive compression method, etc.

[0295] Compared with the related E2E image compression method, the content adaptive online training method and device in the present application can have the following benefits. The encoding efficiency of NIC is improved by using an adaptive online training mechanism. The use of a flexible and universal framework can be compatible with various types of pre-training frameworks and quality metrics. For example, certain pre-training parameters in various types of pre-training frameworks can be replaced by online training using images to be encoded and transmitted.

[0296] In some embodiments, content adaptive online training including type I content adaptive online training and type II content adaptive online training can utilize one or more input blocks in an image to be encoded and / or transmitted (e.g., an original image or an image with residual data). Type I content adaptive online training based on one or more input blocks can be referred to as block-based type I content adaptive online training.

[0297] In one embodiment, content adaptive online training of type I and content adaptive online training of type II are combined such that one or more pre-trained parameters (e.g., one or more weights and / or one or more biases) used in a pre-trained NIC framework are updated using content adaptive online training and one or more modified parameters (e.g., one or more scaling factors and / or one or more offsets) of one or more sets of pre-trained parameters are modified.

[0298] IV. Overview Flowchart of Exemplary Encoding and Decoding Process Based on Content-Adaptive Online Training

[0299] Fig.19 An overview flow chart of the method (1900) of an embodiment of the present application is shown. The method (1900) can be used to encode an image, such as an original image or a residual image. In various embodiments, the method (1900) is performed by a processing circuit, such as a processing circuit in a terminal device (310), (320), (330) and (340), a processing circuit that performs the functions of a video encoder (1600A), a processing circuit that performs the functions of a video encoder (1700), or a processing circuit that performs the functions of a NIC framework (900). In one example, the processing circuit performs (i) one of the video encoders (403), (603) and (703), and (ii) a combination of the functions of one of the video encoder (1600A) and the video encoder (1700). In some embodiments, the method (1900) is implemented as software instructions, so that when the processing circuit executes the software instructions, the processing circuit executes the method (1900). The process starts at (S1901). In one example, the NIC framework is based on a neural network. In one example, the NIC framework is a reference Fig. 9 The NIC framework (900) described herein can be based on CNN, for example, Figures 10 to 15 As described above, the video encoder (e.g., (1600A) or (1700)) and the corresponding video decoder (e.g., (1600B) or (1800)) may include multiple components in the NIC framework. The neural network-based NIC framework is pre-trained, so the video encoder and video decoder are also pre-trained. Method (1900) proceeds to (S1910).

[0300] At (S1910), a fine-tuning process is performed on the NIC framework based on one or more images (or one or more input images). The one or more input images can be one or more any suitable images having one or more any suitable sizes. In some examples, the one or more input images include one or more original images in a spatial domain, one or more natural images, and / or one or more computer-generated images, etc.

[0301] In some examples, one or more input images include, for example, residual data in the spatial domain calculated by a residual calculator (e.g., residual calculator (723)). The components in various devices can be appropriately combined to implement (S1910), for example, referring to Figure 7 and Fig. 9 , the residual data from the residual calculator are combined into an image and fed into the main encoder network (911) in the NIC framework.

[0302] One or more modification parameters may be determined based on one or more images, for example, adjusted based on default values ​​(e.g., 1 for a scaling factor and 0 for an offset). In one embodiment, during the training process described in (S1910), for example, in each step, the one or more modification parameters are updated. The one or more modification parameters may be one or more scaling factors or one or more offsets. The one or more modification parameters may respectively modify one or more sets of pre-trained parameters in at least one neural network in the NIC framework to generate one or more sets of replacement parameters. Each of the one or more modification parameters may modify a corresponding set of pre-trained parameters.

[0303] In one embodiment, a first subset of one or more sets of pre-trained parameters modified by a first subset of one or more modification parameters is used in at least one neural network in a video encoder (e.g., a pre-trained video encoder), and thus the at least one neural network in the video encoder can be updated based on the first subset of the one or more modification parameters. In one example, when the first subset of one or more sets of pre-trained parameters are respectively replaced by the first subset of one or more sets of replacement parameters, the at least one neural network in the video encoder is updated. In one example, at least one neural network in the video encoder is iteratively updated during the fine-tuning process.

[0304] In one example, the video encoder does not include any of the one or more sets of pre-trained parameters, and thus the video encoder is not updated and remains a pre-trained video encoder.

[0305] At (S1920), one of the one or more images may be encoded using a video encoder having at least one updated neural network. In one example, after at least one neural network in the video encoder is updated, one of the one or more images is encoded.

[0306] Step (S1920) may be appropriately changed. For example, when the at least one neural network in the video encoder does not include any of the one or more sets of pre-trained parameters, the video encoder is not updated, and thus a pre-trained video encoder (e.g., a video encoder including at least one pre-trained neural network) may be used to encode one of the one or more images.

[0307] At (S1930), neural network update information indicating the second subset of the one or more modification parameters may be encoded into the bitstream. In one example, the second subset of the one or more modification parameters is used to update at least one neural network in the video decoder at the decoder side. For example, if the second subset of the one or more modification parameters does not include modification parameters and the neural network update information is not signaled in the bitstream, step (S1930) may be omitted and none of the neural networks in the video decoder is updated.

[0308] At (S1940), a codestream including an encoded image of the one or more images and the neural network update information may be transmitted. Step (S1940) may be appropriately changed. For example, if step (S1930) is omitted, the codestream does not include the neural network update information. The method (1900) proceeds to (S1999) and ends.

[0309] The method (1900) may be appropriately modified to accommodate various scenarios, and the steps in the method (1900) may be adjusted accordingly. One or more of the steps in the method (1900) may be modified, omitted, repeated, and / or combined. The method (1900) may be implemented in any suitable order. One or more additional steps may be added. For example, in addition to encoding one of the one or more images, one or more other images, such as the remaining one or more of the one or more images, are encoded in (S1920) and transmitted in (S1940).

[0310] In some examples of the method (1900), one of the one or more images is encoded by the updated video encoder and transmitted in the bitstream. Since the fine-tuning process is based on the one or more images, the fine-tuning process is based on the context to be encoded and is therefore context-based.

[0311] In some examples, the neural network update information further indicates which pre-trained parameters in the video decoder are modified by the second subset of the one or more modified parameters, so that one or more corresponding pre-trained parameters in the video decoder can be updated. The neural network update information may indicate component information (e.g., (915)), layer information (e.g., fourth layer DeConv: 5×5 c3 s2), and / or channel information (e.g., second channel) of the one or more pre-trained parameters in the video decoder modified by the second subset of the one or more modified parameters. Therefore, referring to Fig.11 , the one or more pre-trained parameters in the video decoder modified by the second subset of the one or more modification parameters include the convolution kernel of the second channel of DeConv: 5×5 c3 s2 in the main decoder network (915). Therefore, the convolution kernel of the second channel of DeConv: 5×5 c3 s2 in the main decoder network (915) is updated. In some examples, the component information (e.g., (915)), layer information (e.g., the fourth layer DeConv: 5×5 c3 s2) and / or channel information (e.g., the second channel) of the one or more pre-trained parameters in the video decoder modified by the second subset of the one or more modification parameters are predetermined and stored in the pre-trained video decoder, and therefore are not represented by a signal.

[0312] Fig. 20 An overview flow chart of a method (2000) of an embodiment of the present application is shown. The method (2000) can be used for reconstruction of an encoded image. In various embodiments, the method (2000) is performed by a processing circuit, such as a processing circuit in a terminal device (310), (320), (330) and (340), a processing circuit that performs the functions of a video decoder (1600B), a processing circuit that performs the functions of a video decoder (1800), or a processing circuit that performs the functions of a NIC framework (900). In one example, the processing circuit performs a combination of the following functions: (i) one of a video decoder (410), a video decoder (510) and a video decoder (810), and (ii) one of a video decoder (1600B) or a video decoder (1800). In some embodiments, the method (2000) is implemented as software instructions, so that when the processing circuit executes the software instructions, the processing circuit performs the method (2000). The process starts at (S2001). In one example, the NIC framework is based on a neural network. In one example, the NIC framework is a reference Fig. 9 The NIC framework (900) described herein can be based on CNN, for example, Figures 10 to 15As described above, the video decoder (e.g., (1600B) or (1800)) may include multiple components in the NIC framework. The neural network-based NIC framework may be pre-trained. The video decoder may be pre-trained using pre-trained parameters. The method (2000) proceeds to (S2010).

[0313] At (S2010), neural network update information in the encoded bitstream may be decoded. The neural network update information may be used for at least one neural network in the video decoder. The at least one neural network may be predefined or indicated in the encoded bitstream or the neural network update information. The at least one neural network may be configured with a set of pre-trained parameters. The neural network update information may correspond to the encoded image to be reconstructed and indicate a first modification parameter.

[0314] In one example, the first modified parameter is a scaling factor. The neural network update information may further indicate an offset for modifying the set of pre-trained parameters.

[0315] In one example, the first modification parameter is an offset.

[0316] In one example, the set of pre-trained parameters includes biases in a convolutional layer of a neural network in at least one neural network.

[0317] In one example, the set of pre-trained parameters includes weights in a convolution kernel of a convolution layer of a neural network in at least one neural network.

[0318] In one example, the video decoder is Fig.18 The video decoder (1800) shown in FIG. 1 is the main decoder network (915).

[0319] In one example, the video decoder is Fig. 16B The video decoder (1600B) shown in . In one example, the at least one neural network is one of the main decoder network (915), the context model NN (916), the entropy parameter NN (917), and the super decoder (925). For example, the at least one neural network is the context model NN (916). The at least one neural network in the video decoder may include multiple main decoder networks (915), context models NN (916), entropy parameters NN (917), and super decoders (925).

[0320] At (S2020), a first modification parameter may be determined based on the neural network update information. In one embodiment, the updated parameter is obtained from the neural network update information. In one example, the updated parameter may be obtained from the neural network update information by decompression. In one example, the neural network update information indicates that the updated parameter is the difference (e.g., 0.1) between the first modification parameter (e.g., scaling factor 1.1) and the pre-trained modification parameter (e.g., scaling factor 1), and the first modification parameter may be calculated based on the sum of the updated parameter and the pre-trained modification parameter. In one embodiment, the first modification parameter is determined as the updated parameter.

[0321] At (S2030), a set of pre-trained parameters in at least one neural network in the video decoder may be updated based on the first modification parameter. Fig.13 This corresponds to the nine weights in the first channel (e.g., c1) of the third deconvolution layer (DeConv: 3×3 c384 s1) in the CNN of the super decoder (925) shown in FIG.

[0322] In one example, when the first modification parameter is a scaling factor, the set of pre-trained parameters are respectively replaced by a set of replacement parameters, wherein the set of replacement parameters is a set of pre-trained parameters multiplied by the scaling factor.

[0323] In one example, the first modified parameter is an offset that is added to a set of pre-trained parameters in at least one neural network to update the set of pre-trained parameters.

[0324] At (S2040), the encoded images in the code stream can be decoded by the updated video decoder, for example, based on the updated at least one neural network. The output image generated at (S2040) can be one or more any suitable images having one or more any suitable sizes. In some examples, the output image includes a reconstructed original image in the spatial domain, a natural image, and / or a computer-generated image, etc.

[0325] In some examples, the output image of the video decoder includes residual data in the spatial domain, and thus further processing can be used to generate a reconstructed image based on the output image. For example, the reconstruction module (874) is used to combine the residual data and the prediction result (which is the output of the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which can be part of the reconstructed image. Additional appropriate operations, such as deblocking operations, etc., can be performed to improve visual quality. The components in various devices can be appropriately combined to implement (S2040), for example, referring to Figure 8 and Fig. 9, the residual data and the corresponding prediction results from the main decoder network (915) in the video decoder are fed into the reconstruction module (874) to generate a reconstructed image.

[0326] In one example, the code stream further includes one or more coded bits for determining a context model for decoding a coded image. The video decoder may include a main decoder network (e.g., (911)), a context model network (e.g., (916)), an entropy parameter network (e.g., (917)), and a super decoder network (e.g., (925)). At least one neural network includes one or more of a main decoder network, a context model network, an entropy parameter NN, and a super decoder network. The super decoder network may be used to decode the one or more coded bits. The entropy model (e.g., context model) may be determined based on the decoded bits and a quantized potential representation of the coded image available to the context model network using the context model network and the entropy parameter network. The coded image may be decoded using the main decoder network and the entropy model.

[0327] Method (2000) proceeds to (S2099) and ends.

[0328] The method (2000) may be appropriately adapted to suit various scenarios, and the steps in the method (2000) may be adjusted accordingly. One or more steps in the method (2000) may be modified, omitted, repeated, and / or combined. The method (2000) may be implemented in any suitable order. One or more additional steps may be added.

[0329] For example, at (S2040), one or more other encoded images in the encoded code stream are decoded based on the updated neural network. Therefore, the encoded image and the one or more other encoded images can share the same neural network update information.

[0330] In one example, the neural network update information indicates a second modified parameter of at least one neural network configured with a second set of pre-trained parameters. At (S2030), the second set of pre-trained parameters in the at least one neural network may be updated based on the second modified parameter. In one example, the set of pre-trained parameters includes weights in a convolution kernel of a convolution layer of one of the at least one neural network, and the second set of pre-trained parameters includes a bias of a convolution layer of one of the at least one neural network.

[0331] The embodiments in this application may be used alone or in combination in any order. Furthermore, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-volatile computer-readable medium.

[0332] The present application does not impose any restrictions on the methods used for encoders (e.g., encoders based on neural networks), decoders (e.g., decoders based on neural networks). The one or more neural networks used in the encoder and / or decoder, etc. can be one or more neural networks of any suitable type, such as DNN and CNN, etc.

[0333] Therefore, the content adaptive online training method of the present application can be compatible with different types of NIC frameworks, such as different types of encoding DNNs, decoding DNNs, encoding CNNs and / or decoding CNNs, etc.

[0334] V. Exemplary Computer System for Implementing Embodiments of the Application

[0335] The techniques described above may be implemented by computer software using computer-readable instructions physically stored in one or more computer-readable media. Fig.21 A computer system (2100) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0336] Computer software may be encoded using any suitable machine code or computer language, and may be generated through assembly, compilation, linking or similar mechanisms, and the code may include instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through operations such as code interpretation and microcode execution.

[0337] These instructions can be executed in various types of computers or computer components, including, for example, personal computers, tablets, servers, smart phones, gaming devices, IoT devices, etc.

[0338] Fig.21 The components shown for the computer system (2100) are exemplary in nature and are not intended to impose any limitations on the scope of use or functionality of computer software implementing embodiments of the present application. Nor should the configuration of the components be interpreted as having any dependency or requirement on any one or combination of components in the exemplary embodiment of the computer system (2100).

[0339] The computer system (2100) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, taps), visual input (e.g., gestures), olfactory input (not shown). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to a person's conscious input, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0340] Input human-machine interface devices may include one or more of the following (only one of each is depicted): keyboard (2101), mouse (2102), touchpad (2103), touch screen (2110), data gloves (not shown), joystick (2105), microphone (2106), scanner (2107), camera (2108).

[0341] The computer system (2100) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (2110), a data glove (not shown), or a joystick (2105), but there may also be tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (2109), headphones (not shown)), visual output devices (e.g., for screens (2110) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities, some of which are capable of outputting two-dimensional visual outputs or more than three-dimensional outputs by means such as stereo output; virtual reality glasses (not shown), holographic displays and smoke canisters (not shown), and printers (not shown).

[0342] The computer system (2100) may also include human-accessible storage devices and their associated media, such as optical media including media (2121) such as CD / DVD ROM / RW (2120) with CD / DVD, thumb drives (2122), removable hard drives or solid-state drives (2123), traditional magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices such as security software dogs (not shown), and the like.

[0343] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other volatile signals.

[0344] The computer system (2100) may also include an interface (2154) to one or more communication networks (2155). The network may be, for example, wireless, wired, optical. The network may further be local, wide-area, metropolitan, vehicle-mounted, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks (e.g., Ethernet), wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide-area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicle and industrial networks (including CAN buses), and the like. Some networks typically require an external network interface adapter attached to some common data port or peripheral bus (2149) (e.g., a USB port of the computer system (2100)); other networks are typically integrated into the kernel of the computer system (2100) by attaching to a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smartphone computer system). Using any of these networks, the computer system (2100) can communicate with other entities. Such communications may be one-way, receive-only (e.g., broadcast TV), one-way send-only (e.g., CANbus to certain CANbus devices), or two-way (e.g., to other computer systems using a local area digital network or a wide area digital network). Certain protocols and protocol stacks may be used on each of those networks and network interfaces as described above.

[0345] The above-mentioned human interface device, human-accessible storage device, and network interface may be attached to the kernel (2140) of the computer system (2100).

[0346] The kernel (2140) may include one or more central processing units (CPUs) (2141), graphics processing units (GPUs) (2142), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (2143), hardware accelerators for certain tasks (2144), graphics adapters (2150), etc. These devices, along with read-only memory (ROM) (2145), random access memory (2146), internal mass storage devices (2147) such as internal non-user accessible hard drives, SSDs, etc., may be connected via a system bus (2148). In some computer systems, the system bus (2148) may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the kernel's system bus (2148), or to the system bus (2148) via a peripheral bus (2149). In one embodiment, a screen (2110) may be connected to a graphics adapter (2150). The architecture of the peripheral bus includes PCI, USB, etc.

[0347] The CPU (2141), GPU (2142), FPGA (2143) and accelerator (2144) can execute certain instructions, and the combination of these instructions can constitute the aforementioned computer code. The computer code can be stored in ROM (2145) or RAM (2146). Transition data can also be stored in RAM (2146), while permanent data can be stored in, for example, an internal mass storage device (2147). Fast storage and retrieval of any memory device can be enabled by using a cache memory, which can be closely associated with one or more CPUs (2141), GPUs (2142), mass storage devices (2147), ROMs (2145), RAMs (2146), etc.

[0348] The computer readable medium may have computer codes for performing various computer-implemented operations. The media and computer codes may be those specially designed and constructed for the purposes of this application, or they may be of a type well known and available to those skilled in the art of computer software.

[0349] As an example and not limitation, a computer system having the architecture (2100) and in particular the kernel (2140) can provide functionality as a result of one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as described above, as well as certain storage devices of the kernel (2140) having non-volatile properties (e.g., kernel internal mass storage devices (2147) or ROM (2145)). Software that implements various embodiments of the present application can be stored in such devices and executed by the kernel (2140). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can cause the kernel (2140) and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (2146) and modifying such data structures according to software-defined processes. In addition or as an alternative, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (2144)) that may replace software or operate in conjunction with software to perform specific processes or specific portions of specific processes described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuits (e.g., integrated circuits (ICs)) storing software for execution, circuits containing logic for execution, or both. The present application includes any suitable combination of hardware and software. Appendix A: Acronyms JEM: joint exploration model VVC: versatile video coding BMS: benchmark set MV:motion vector HEVC: High Efficiency Video Coding SEI: Supplementary Enhancement Information VUI:Video Usability Information GOP:Groups of Pictures TU:Transform Unit PU: Prediction Unit CTU: Coding Tree Unit CTB: Coding Tree Block PB: Prediction Block HRD: Hypothetical Reference Decoder SNR:Signal Noise Ratio CPU:Central Processing Unit GPU: Graphics Processing Unit CRT: Cathode Ray Tube LCD: Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM:Read-Only Memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD:Programmable Logic Device LAN:Local Area Network GSM:Global System for Mobile communications Global Mobile Communications System LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Array SSD: solid-state drive IC:Integrated Circuit CU:Coding Unit NIC: Neural Image Compression RD:Rate-Distortion E2E: End to End ANN:Artificial Neural Network DNN: Deep Neural Network CNN: Convolution Neural Network

[0350] Although the present application has described several exemplary embodiments, there are changes, permutations and various equivalent substitutions that fall within the scope of the present application. Therefore, it should be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present application and are therefore within the spirit and scope of the present application.

Claims

1. A method for video decoding in a video decoder, characterized in that: include: Decoding neural network update information for at least one neural network in the encoded bitstream for a video decoder, the at least one neural network being configured with a set of pre-trained parameters, the neural network update information indicating a first modification parameter; updating the set of pre-trained parameters in the at least one neural network in the video decoder based on the first modification parameters; and The encoded image is decoded based on the updated at least one neural network.

2. The method according to claim 1, characterized in that The first modification parameter is a scaling factor, and The updating includes replacing the set of pre-trained parameters respectively with a set of replacement parameters, the set of replacement parameters being the set of pre-trained parameters multiplied by the scaling factor.

3. The method according to claim 1, characterized in that The first modification parameter is an offset, and The updating includes adding the offset to the set of pre-trained parameters in the at least one neural network.

4. The method according to claim 2, characterized in that: The neural network update information further indicates an offset for modifying the set of pre-trained parameters, and The updating includes adding the offset to the set of replacement parameters in the at least one neural network.

5. The method according to claim 1, characterized in that The encoded code stream further indicates one or more encoded bits for determining a context model for decoding the encoded image, The at least one neural network comprises one or more of a main decoder network, a context model neural network, an entropy parameter neural network, and a super decoder network in the video decoder, The method further comprises: decoding the one or more encoded bits using the superdecoder network, and determining a context model based on decoded bits and a quantized latent representation of the encoded image available to the context model network using the context model network and the entropy parameter network, and Decoding the encoded image includes decoding the encoded image using the primary decoder network and the context model.

6. The method according to claim 1, characterized in that The at least one neural network is one or more of a main decoder network, a context model neural network, an entropy parameter neural network, and a super decoder network in the video decoder.

7. The method according to claim 1, characterized in that The set of pre-trained parameters includes biases in a convolutional layer of a neural network in the at least one neural network.

8. The method according to claim 1, characterized in that: The set of pre-trained parameters includes weights in a convolution kernel of a convolutional layer of a neural network in the at least one neural network.

9. The method according to claim 1, characterized in that: The neural network update information indicates second modified parameters of the at least one neural network configured with a second set of pre-trained parameters, and The updating includes updating the second set of pre-trained parameters in the at least one neural network based on the second modification parameters.

10. The method according to claim 9, characterized in that The set of pre-trained parameters includes weights in convolution kernels of a convolution layer of a neural network in the at least one neural network, and The second set of pre-trained parameters includes biases of the convolutional layer of one of the at least one neural network.

11. The method according to claim 1, characterized in that Further including: Another encoded image in the encoded code stream is decoded based on the updated at least one neural network.

12. A video decoding device, characterized in that: comprising a processing circuit for: Decoding neural network update information for at least one neural network in the encoded bitstream for a video decoder, the at least one neural network being configured with a set of pre-trained parameters, the neural network update information indicating a first modification parameter; updating the set of pre-trained parameters in the at least one neural network in the video decoder based on the first modification parameters; and The encoded image is decoded based on the updated at least one neural network.

13. The device according to claim 12, characterized in that The first modification parameter is a scaling factor, and The processing circuit is used to replace the set of pre-trained parameters with a set of replacement parameters, respectively, and the set of replacement parameters is the set of pre-trained parameters multiplied by the scaling factor.

14. The device according to claim 12, characterized in that The first modification parameter is an offset, and The processing circuit is configured to add the offset to the set of pre-trained parameters in the at least one neural network.

15. The device according to claim 13, characterized in that The neural network update information further indicates an offset for modifying the set of pre-trained parameters, and The processing circuit is configured to add the offset to the set of replacement parameters in the at least one neural network.

16. The device according to claim 12, characterized in that The set of pre-trained parameters includes biases in a convolutional layer of a neural network of the at least one neural network.

17. The device according to claim 12, characterized in that The set of pre-trained parameters includes weights in a convolution kernel of a convolutional layer of a neural network in the at least one neural network.

18. The device according to claim 12, characterized in that The neural network update information indicates second modified parameters of the at least one neural network configured with a second set of pre-trained parameters, and The processing circuit is configured to update the second set of pre-trained parameters in the at least one neural network based on the second modification parameter.

19. The device according to claim 18, characterized in that The set of pre-trained parameters includes weights in convolution kernels of a convolution layer of a neural network in the at least one neural network, and The second set of pre-trained parameters includes biases of the convolutional layer of one of the at least one neural network.

20. A non-volatile computer-readable storage medium storing a program, characterized in that: The program is executable by at least one processor to: Decoding neural network update information for at least one neural network in the encoded bitstream for a video decoder, the at least one neural network being configured with a set of pre-trained parameters, the neural network update information indicating a first modification parameter; updating the set of pre-trained parameters in the at least one neural network in the video decoder based on the first modification parameters; and The encoded image is decoded based on the updated at least one neural network.

Images