Video decoding method, video decoding device and storage medium

Abstract

Aspects of the present disclosure provide a video decoding method, apparatus, and non-transitory computer-readable storage medium. The apparatus may include processing circuitry. The processing circuitry is configured to decode first neural network update information in a coded bitstream, the first neural network update information being used for a first neural network. The first neural network is configured with a first set of pre-trained parameters. The first neural network update information corresponds to a first block in an image to be reconstructed and indicates a first replacement parameter corresponding to a first pre-trained parameter in the first set of pre-trained parameters. The processing circuitry is configured to update the first neural network based on the first replacement parameter. The processing circuitry may decode the first block based on the updated first neural network, the updated first neural network being used for the first block.

Description

[0001] Incorporation by reference

[0002] This application claims priority to U.S. Patent Application No. 17 / 729,978, filed on April 26, 2022, entitled “BLOCK-WISE CONTENT-ADAPTIVE ONLINE TRAINING IN NEURAL IMAGE,” which claims priority to U.S. Provisional Application No. 63 / 182,366, filed on April 30, 2021, entitled “Block-wise Content-Adaptive Online Training in Neural Image Compression.” The disclosures of the prior applications are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure describes embodiments generally related to video encoding. Background Art

[0004] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. To the extent described in this background section, the work of the presently named inventors and aspects of the description that may not have been prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art for the present disclosure.

[0005] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. An uncompressed digital image and / or video can include a series of pictures, each picture having spatial dimensions of, for example, 1920×1080 luma samples and associated chroma samples. The series of pictures can have a fixed or variable picture rate (also informally referred to as a frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed images and / or videos have specific bit rate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (a resolution of 1920×1080 luma samples at a frame rate of 60 Hz) requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires over 600 GBytes of storage space.

[0006] One goal of video encoding and decoding is to reduce redundancy in the input image and / or video signal through compression. Compression helps reduce the bandwidth and / or storage space requirements mentioned above, in some cases by two orders of magnitude or more. Although the description herein uses video encoding / decoding as an illustrative example, the same techniques can be applied in a similar manner to image encoding / decoding without departing from the spirit of the present disclosure. Lossless compression and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to techniques that can reconstruct an exact replica of the original signal from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is sufficiently small that the reconstructed signal is useful for the intended application. In the case of video, lossy compression is widely used. The amount of distortion allowed depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect this: higher allowable / tolerable distortion can result in higher compression ratios.

[0007] Video encoders and decoders may utilize several broad categories of techniques, including, for example, motion compensation, transforms, quantization, and entropy coding.

[0008] Video codec techniques can include a technique known as intra coding. In intra coding, sample values ​​are represented without reference to samples or other data from a previously reconstructed reference picture. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are encoded in intra mode, the picture can be an intra picture. Intra pictures and their derivatives (e.g., independent decoder refresh pictures) can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and video session, or as a still image. The samples of an intra block can be transformed, and the transform coefficients can be quantized before entropy coding. Intra prediction can be a technique for minimizing sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficients, the fewer bits are required to represent the entropy coded block at a given quantization step size.

[0009] For example, conventional intra-frame coding known from the MPEG-2 generation of coding techniques does not use intra-frame prediction. However, some newer video compression techniques include techniques that attempt to use surrounding sample data and / or metadata obtained during encoding and / or decoding of, for example, spatially adjacent and preceding data blocks in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. Note that, at least in some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, and not reference data from reference pictures.

[0010] There are many different forms of intra-frame prediction. When more than one such technique can be used in a given video coding technique, the technique used can be encoded in the intra-frame prediction mode. In some cases, a mode can have sub-modes and / or parameters, which can be encoded separately or contained in the mode codeword. For a given mode, sub-mode and / or parameter combination, which codeword is used will have an impact on the coding efficiency gain achieved through intra-frame prediction, and therefore, the entropy coding technique used to convert the codeword into the bitstream.

[0011] H.264 introduced specific intra-frame prediction modes, which were improved in H.265 and further improved in newer coding technologies such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). A prediction block can be formed using the values ​​of neighboring samples belonging to already available samples. Sample values ​​from neighboring samples are copied into the prediction block based on the direction. A reference to the direction in use can be encoded in the bitstream or predicted itself.

[0012] refer to Figure 1A , depicted in the lower right corner is a subset of 9 known prediction directions from the 33 possible prediction directions of H.265 (corresponding to the 33 angular modes of the 35 intra modes). The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction of the predicted sample. For example, arrow (102) indicates that sample (101) is predicted based on one or more samples to the upper right and at a 45° angle to the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more samples to the lower left of sample (101) at a 22.5° angle to the horizontal.

[0013] Still refer to Figure 1A , a square block (104) of 4×4 samples is depicted in the upper left (indicated by the thick dashed line). The square block (104) includes 16 samples, each sample labeled "S", its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. Since the size of the block is 4×4 samples, S44 is located in the lower right corner. Reference samples are also shown that follow a similar numbering scheme. The reference samples are labeled (104) with R, their Y position (e.g., row index) and X position (column index) relative to the block (104). In H.264 and H.265, the prediction samples are adjacent to the block being reconstructed; therefore, there is no need to use negative values.

[0014] Intra-picture prediction works by copying reference sample values ​​from neighboring samples, as appropriate via the signaled prediction direction. For example, suppose the coded video bitstream contains a signal indicating a prediction direction consistent with arrow (102) for this block—that is, multiple samples are predicted based on one or more prediction samples to the upper right, at a 45° angle to the horizontal. In this case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Sample S44 is then predicted based on reference sample R08.

[0015] In some cases, the values ​​of multiple reference samples may be combined, for example, by interpolation, in order to calculate the reference sample; in particular, when the direction is not divisible by 45°.

[0016] As video coding technology has developed, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This was increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of disclosure. Experiments have been done to identify the most likely directions, and certain techniques in entropy coding are used to represent those possible directions with a small number of bits, with some penalty for less likely directions. In addition, the direction itself can sometimes be predicted from adjacent directions used in adjacent already decoded blocks.

[0017] Figure 1B A schematic diagram (110) is shown depicting 65 intra prediction directions according to JEM to illustrate that the number of prediction directions increases over time.

[0018] The mapping of the intra-prediction direction bits representing directions in the coded video bitstream can vary depending on the video coding technique; and can include, for example, simple, direct mappings such as mapping prediction directions to intra-prediction modes, mapping prediction directions to codewords, to complex adaptive schemes involving most probable modes, and similar techniques. However, in all cases, some directions are statistically less likely to occur in the video content than others. Since the goal of video compression is to reduce redundancy, in well-performing video coding techniques, those less likely directions will be represented by more bits than more likely directions.

[0019] Motion compensation can be a lossy compression technique and can involve a technique in which a block of sample data from a previously reconstructed picture or part thereof (reference picture) is used to predict a newly reconstructed picture or part of a picture after being spatially shifted in a direction indicated by a motion vector (hereafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, the third being an indication of the reference picture in use (the latter indirectly being a temporal dimension).

[0020] In some video compression techniques, the MV applicable to a region of sample data can be predicted based on other MVs, for example, based on the MV associated with another region of sample data that is spatially adjacent to the region being reconstructed and precedes the MV in decoding order. This can significantly reduce the amount of data required to encode the MV, thereby eliminating redundancy and improving compression. MV prediction works effectively, for example, because when encoding an input video signal derived from a camera (referred to as natural video), there is a statistical probability that multiple regions larger than the region to which a single MV applies will move in similar directions. Therefore, in some cases, similar MVs (motion vectors) derived from multiple MVs in adjacent regions can be used for prediction. This results in the MV discovered for a given region being similar or identical to the MV predicted based on surrounding MVs, and after entropy coding, this can be represented using fewer bits than would be used if the MV were encoded directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, such as when the predicted value is calculated from several surrounding MVs, MV prediction itself can be lossy due to rounding errors.

[0021] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, “High Efficiency Video Coding”, December 2016). In addition to the many MV prediction mechanisms provided by H.265, a technique referred to as “spatial merging” is described here.

[0022] refer to Figure 2, the current block (201) includes samples that the encoder found during motion search so that it can be predicted based on a previous block of the same size that has been spatially shifted. Instead of encoding the MV directly, the MV associated with any of the five surrounding samples (denoted A0, A1 and B0, B1, B2 (202 to 206 respectively)) can be used, and the MV can be derived based on metadata associated with one or more reference pictures, for example, based on the most recent (in decoding order) reference picture. In H.265, MV prediction can use multiple predictors that make predictions based on the same reference pictures being used by neighboring blocks. Summary of the Invention

[0023] Various aspects of the present disclosure provide video encoding and decoding methods and devices. In some examples, a video decoding device includes processing circuitry. The processing circuitry is configured to decode first neural network update information in an encoded bitstream, the first neural network update information being used for a first neural network. The first neural network is configured with a first set of pre-trained parameters. The first neural network update information corresponds to a first block in an image to be reconstructed and indicates a first replacement parameter corresponding to a first pre-trained parameter in the first set of pre-trained parameters. The processing circuitry may update the first neural network based on the first replacement parameter and decode the first block based on the updated first neural network, the updated first neural network being used for the first block.

[0024] In one embodiment, the first neural network update information further indicates one or more replacement parameters for one or more remaining neural networks in the plurality of neural networks other than the first neural network. The processing circuitry may update the one or more remaining neural networks based on the one or more replacement parameters.

[0025] In one embodiment, processing circuitry decodes second neural network update information from a coded bitstream, the second neural network update information being for a second neural network. The second neural network is configured with a second set of pre-trained parameters. The second neural network update information corresponds to a second block in an image to be reconstructed and indicates second replacement parameters corresponding to second pre-trained parameters in the second set of pre-trained parameters. In one example, the second neural network is different from the first neural network. The processing circuitry may update the second neural network based on the second replacement parameters and decode the second block based on the updated second neural network, the updated second neural network being for the second block.

[0026] In one embodiment, the first pre-training parameter is one of a pre-training weight coefficient and a pre-training bias term.

[0027] In one embodiment, the second pre-training parameter is the other of a pre-training weight coefficient and a pre-training bias term.

[0028] In one embodiment, the processing circuit decodes the second block in the encoded bitstream based on the updated first neural network that was used for the first block.

[0029] In one embodiment, the first neural network update information indicates a difference between the first replacement parameter and the first pre-trained parameter, and the processing circuit determines the first replacement parameter based on a sum of the difference and the first pre-trained parameter.

[0030] In one embodiment, the processing circuit decodes the first neural network update information based on one of a variation of a Lempel-Ziv-Markov chain algorithm (LZMA2) and a bzip2 algorithm.

[0031] In one example, the processing circuit decodes the second neural network update information based on the other of an LZMA2 and a bzip2 algorithm.

[0032] Aspects of the present disclosure also provide a non-transitory computer-readable storage medium storing a program executable by at least one processor to perform a video decoding method. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Further features, nature, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0034] Figure 1A is a schematic diagram of an exemplary subset of intra prediction modes;

[0035] Figure 1B is a diagram of exemplary intra prediction directions;

[0036] Figure 2 shows a current block and surrounding samples according to one embodiment;

[0037] Figure 3 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment;

[0038] Figure 4 is a schematic diagram of a simplified block diagram of a communication system according to one embodiment;

[0039] Figure 5 is a schematic diagram of a simplified block diagram of a decoder according to one embodiment;

[0040] Figure 6 is a schematic diagram of a simplified block diagram of an encoder according to one embodiment;

[0041] Figure 7 shows a block diagram of an encoder according to another embodiment;

[0042] Figure 8 shows a block diagram of a decoder according to another embodiment;

[0043] Figure 9A An example of block image encoding according to an embodiment of the present disclosure is shown;

[0044] Figure 9B An exemplary NIC framework according to an embodiment of the present disclosure is shown;

[0045] Figure 10 An exemplary convolutional neural network (CNN) of a main encoder network according to an embodiment of the present disclosure is shown;

[0046] Figure 11 An exemplary CNN of a main decoder network according to an embodiment of the present disclosure is shown;

[0047] Figure 12 An exemplary CNN of a hyper encoder according to an embodiment of the present disclosure is shown;

[0048] Figure 13 An exemplary CNN of a hyper decoder according to an embodiment of the present disclosure is shown;

[0049] Figure 14 An exemplary CNN of a context model network according to an embodiment of the present disclosure is shown;

[0050] Figure 15 An exemplary CNN of an entropy parameter network according to an embodiment of the present disclosure is shown;

[0051] Figure 16A An exemplary video encoder according to an embodiment of the present disclosure is shown;

[0052] Figure 16B An exemplary video decoder according to an embodiment of the present disclosure is shown;

[0053] Figure 17 An exemplary video encoder according to an embodiment of the present disclosure is shown;

[0054] Figure 18 An exemplary video decoder according to an embodiment of the present disclosure is shown;

[0055] Figure 19 shows a flowchart outlining a process according to an embodiment of the present disclosure;

[0056] Figure 20 shows a flowchart outlining a process according to an embodiment of the present disclosure;

[0057] Figure 21 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION

[0058] Figure 3 A simplified block diagram of a communication system (300) according to an embodiment of the present disclosure is shown. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). Figure 3 In the example, a first pair of terminal devices (310) and (320) performs unidirectional transmission of data. For example, the terminal device (310) can encode video data (e.g., a video picture stream captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to restore the video picture, and display the video picture based on the restored video data. One-way data transmission is common in media service applications, etc.

[0059] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, such as may occur during a video conference. For the bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) can encode video data (e.g., a video picture stream captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) can also receive the encoded video data transmitted by the other of the terminal devices (330) and (340), and can decode the encoded video data to restore the video picture, and can display the video picture on an accessible display device based on the restored video data.

[0060] exist Figure 3In the example of , terminal devices (310), (320), (330) and (340) may be shown as servers, personal computers and smart phones, but the principles of the present disclosure may not be limited to this. Embodiments of the present disclosure are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. Network (350) represents any number of networks that transmit encoded video data between terminal devices (310), (320), (330) and (340), including, for example, line (wired) and / or wireless communication networks. The communication network (350) can exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purposes of the present discussion, the architecture and topology of the network (350) may not be important for the operation of the present disclosure, unless explained below.

[0061] As examples of applications of the disclosed subject matter, Figure 4 The placement of a video encoder and a video decoder in a streaming environment is shown. The disclosed subject matter is equally applicable to other video-enabled applications including, for example, video conferencing, digital television, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0062] The streaming system may include: a capture subsystem (413), which may include a video source (401), such as a digital camera; creating, for example, an uncompressed video picture stream (402). In one example, the video picture stream (402) includes samples captured by the digital camera. The video picture stream (402) is depicted as a thick line to emphasize the high amount of data when compared to the encoded video data (404) (or encoded video bitstream), which may be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to implement or implement various aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is depicted as a thin line to emphasize the lower amount of data when compared to the video picture stream (402), which may be stored on a streaming server (405) for future use. One or more streaming client subsystems (e.g., Figure 4The client subsystems (406) and (408) in the video server (405) can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the input copy (407) of the encoded video data and creates an output video picture stream (411) that can be presented on a display (412) (e.g., a display screen) or other presentation device (not shown). In some streaming systems, the encoded video data (404), (407) and (409) (e.g., a video bitstream) can be encoded according to certain video coding / compression standards. Examples of such standards include ITU-T Recommendation H.265. In one example, the video coding standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed subject matter can be used in the context of VVC.

[0063] Note that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0064] Figure 5 A block diagram of a video decoder (510) according to an embodiment of the present disclosure is shown. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used to replace Figure 4 A video decoder (410) is shown in the example.

[0065] A receiver (531) may receive one or more encoded video sequences to be decoded by a video decoder (510); in the same or another embodiment, one encoded video sequence at a time, wherein decoding of each encoded video sequence is independent of the decoding of the other encoded video sequences. The encoded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data and other data, such as encoded audio data and / or ancillary data streams, which may be forwarded to their respective consuming entities (not shown). The receiver (531) may separate the encoded video sequence from the other data. To combat network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, it may be external to the video decoder (510) (not shown). In other cases, there may be a buffer memory (not shown) outside the video decoder (510), for example, to combat network jitter, and another buffer memory (515) may be inside the video decoder (510), for example, to handle playback timing. When the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (515) may not be needed or may be small. For use over a best-effort packet network such as the Internet, the buffer memory (515) may be required, which may be relatively large and may advantageously have an adaptive size and may be implemented at least partially in an operating system or similar element (not shown) external to the video decoder (510).

[0066] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The categories of these symbols include information for managing the operation of the video decoder (510) and potentially controlling a presentation device such as a presentation device (512) (e.g., a display screen) that is not part of the electronic device (530) but can be coupled to the electronic device (530), such as Figure 5As shown. The control information for the rendering device may be in the form of a supplemental enhancement information (SEI message) or a video usability information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be based on a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a set of subgroup parameters of at least one pixel subgroup in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (Cu), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (520) may also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0067] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0068] Depending on the type of coded video picture or portion thereof (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors, the reconstruction of the symbol (521) may involve a number of different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). For clarity, the flow of this subgroup control information between the parser (520) and the following units is not described.

[0069] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a number of functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purposes of describing the disclosed subject matter, the conceptual subdivision into the following functional units is appropriate.

[0070] The first unit is a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives quantized transform coefficients and control information, including which transform to use, block size, quantization factor, quantization scaling matrix, etc., as symbols (521) from the parser (520). The scaler / inverse transform unit (551) can output blocks of sample values, which can be input into the aggregator (555).

[0071] In some cases, the output samples of the scaler / inverse transform (551) may belong to an intra-coded block; that is, a block that does not use prediction information from a previously reconstructed image, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses surrounding already reconstructed information obtained from the current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. The current picture buffer (558) buffers, for example, partially reconstructed current pictures and / or fully reconstructed current pictures. In some cases, the aggregator (555) adds the prediction information already generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0072] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-coded and possibly motion compensated block. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to obtain samples for prediction. After the extracted samples are motion compensated according to the symbols (521) associated with the block, these samples may be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (in this case referred to as residual samples or residual signal) to generate output sample information. The addresses within the reference picture memory (557) from which the motion compensated prediction unit (553) obtains the predicted samples may be controlled by motion vectors, which the motion compensated prediction unit (553) may obtain in the form of symbols (521) having, for example, X, Y, and reference picture components. When using sub-sampled accurate motion vectors, motion compensation may also include interpolation of sample values ​​obtained from the reference picture memory (557), motion vector prediction mechanisms, etc.

[0073] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). The video compression techniques may include loop filtering techniques that are controlled by parameters contained in the coded video sequence (also referred to as the coded video bitstream) and made available to the loop filtering unit (556) as symbols (521) from the parser (520), but may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of the coded picture or coded video sequence and to previously reconstructed and loop filtered sample values.

[0074] The output of the loop filter unit (556) may be a sample stream that may be output to a rendering device (512) and stored in a reference picture memory (557) for use in future inter-picture prediction.

[0075] Once fully reconstructed, certain coded pictures can be used as reference pictures for future predictions. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557) and a new current picture buffer can be reallocated before starting reconstruction of the next coded picture.

[0076] The video decoder (510) may perform decoding operations according to a predetermined video compression technique in a standard such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the coded video sequence conforms to the syntax of the video compression technology or standard and to the profile documented in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as the only tools available under the profile. Conformance to the standard also requires that the complexity of the coded video sequence be within the range defined by the level of the video compression technology or standard. In some cases, the level limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the hypothetical reference decoder (HRD) specification and metadata of the HRD buffer management signaled in the coded video sequence.

[0077] In one embodiment, a receiver (531) can receive additional (redundant) data with the encoded video. The additional data can be included as part of the encoded video sequence. The video decoder (510) can use the additional data to correctly decode the data and / or more accurately reconstruct the original video data. The additional data can be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, and the like.

[0078] Figure 6 A block diagram of a video encoder (603) according to an embodiment of the present disclosure is shown. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used to replace Figure 4 The video encoder (403) in the example.

[0079] The video encoder (603) can be used to generate a video from a video source (601) (which is not Figure 6 In another example, the video source (601) is a part of the electronic device (620) that receives the video samples. The video source can capture video pictures to be encoded by the video encoder (603). In another example, the video source (601) is a part of the electronic device (620).

[0080] The video source (601) can provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) can be a storage device that stores previously prepared videos. In a video conferencing system, the video source (601) can be a camera that captures local image information as a video sequence. The video data can be provided as multiple individual pictures that impart motion when viewed sequentially. The pictures themselves can be organized as a spatial array of pixels, where each pixel can include one or more samples, depending on the sampling structure, color space, etc. in use. The relationship between pixels and samples can be readily understood by those skilled in the art. The following description focuses on samples.

[0081] According to one embodiment, the video encoder (603) can encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units described below and is functionally coupled to the other functional units. For clarity, the coupling is not described. The parameters set by the controller (650) can include rate control related parameters (picture skipping, quantizer, lambda value of rate-distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other suitable functions related to the video encoder (603) optimized for a specific system design.

[0082] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. As an oversimplified description, in one example, the encoding loop can include a source encoder (630) (e.g., responsible for creating symbols, e.g., a symbol stream, based on the input picture to be encoded and the reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols in a manner similar to what the (remote) decoder would also create to create sample data (because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream results in bit-accurate results independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, when prediction is used during decoding, the predicted part of the encoder "sees" as reference picture samples exactly the same sample values ​​as the sample values ​​"seen" by the decoder. This basic principle of reference picture synchronicity (and the resulting drift if synchronicity cannot be maintained, eg due to channel errors) is also used in some related techniques.

[0083] The operation of the "local" decoder (633) may be identical to the operation of a "remote" decoder such as the video decoder (510), described above in conjunction with Figure 5 However, a brief reference to Figure 5 , since the symbols are available and the symbol encoding / decoding of the encoded video sequence by the entropy encoder (645) and the parser (520) can be lossless, the entropy decoding part of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0084] In one embodiment, except for the parsing / entropy decoding present in the decoder, the decoder technology is present in the corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on the decoder operation. The description of the encoder technology can be simplified because these technologies are the inverse of the fully described decoder technology. In some areas, a more detailed description is provided below.

[0085] During operation, in some examples, the source encoder (630) may perform motion compensated predictive coding, which predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as "reference pictures." In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a prediction reference for the input picture.

[0086] The local video decoder (633) can decode the coded video data of the picture that can be designated as the reference picture based on the symbols created by the source encoder (630). The operation of the encoding engine (632) can advantageously be a lossy process. When the coded video data can be decoded at the video decoder ( Figure 6 When decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference pictures and may cause the reconstructed reference pictures to be stored in a reference picture cache (634). In this way, the video encoder (603) may locally store copies of the reconstructed reference pictures that have the same content (absent transmission errors) as the reconstructed reference pictures that will be obtained by the remote video decoder.

[0087] The predictor (635) may perform a prediction search on the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new picture. The predictor (635) may operate on a sample-block-by-pixel-block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0088] The controller (650) can manage the encoding operations of the source encoder (630), including, for example, the setting of parameters and sub-group parameters for encoding video data.

[0089] The outputs of all the aforementioned functional units may undergo entropy coding in an entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0090] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which can be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (source not shown).

[0091] The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a specific coded picture type to each coded picture, which can affect the encoding techniques that can be applied to the corresponding picture. For example, a picture can generally be assigned one of the following picture types:

[0092] An intra picture (I picture) can be a picture that is encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of those variations of I pictures and their corresponding applications and features.

[0093] A predicted picture (P picture) may be a picture that is encoded and decoded using intra prediction or inter prediction by predicting sample values ​​of each block using at most one motion vector and a reference index.

[0094] Bidirectionally predicted pictures (B pictures) can be pictures that use up to two motion vectors and reference indices to predict the sample values ​​of each block, and are encoded and decoded using intra-frame prediction or inter-frame prediction. Similarly, multi-predicted pictures can use more than two reference pictures and related metadata to reconstruct a single block.

[0095] A source picture may typically be spatially subdivided into a plurality of blocks of samples (e.g., 4×4, 8×8, 4×8, or 16×16 blocks of samples each) and coded on a block-by-block basis. A block may be predictively coded with reference to other (already coded) blocks as determined by the coding allocation applied to the block's corresponding picture. For example, a block of an I picture may be non-predictively coded or may be predictively coded (spatial prediction or intra prediction) with reference to already coded blocks of the same picture. Pixel blocks of a P picture may be predictively coded with reference to one previously coded reference picture, either via spatial prediction or via temporal prediction. Blocks of a B picture may be predictively coded with reference to one or two previously coded reference pictures, either via spatial prediction or via temporal prediction.

[0096] The video encoder (603) may perform encoding operations according to a predetermined video coding technique or standard (e.g., ITU-T Rec. H.265). In its operation, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video coding technique or standard being used.

[0097] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data (e.g., redundant pictures and slices), SEI messages, VUI parameter set fragments, etc.

[0098] Video can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often abbreviated to intra prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In one example, a particular picture in encoding / decoding, referred to as the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that was previously encoded in the video and is still buffered, the block in the current picture can be encoded by a vector called a motion vector. In the case of using multiple reference pictures, the motion vector points to a reference block in a reference picture and can have a third dimension that identifies the reference picture.

[0099] In some embodiments, bidirectional prediction techniques can be used in inter-frame picture prediction. According to bidirectional prediction techniques, two reference pictures are used, for example, a first reference picture and a second reference picture, both of which precede the current picture in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current picture can be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block can be predicted by a combination of the first reference block and the second reference block.

[0100] In addition, merge mode technology can be used in inter-frame picture prediction to improve coding efficiency.

[0101] According to some embodiments of the present disclosure, prediction is performed in units of blocks, for example, inter-picture prediction and intra-picture prediction. For example, according to the HEVC standard, a picture in a video picture sequence is partitioned into coding tree units (CTUs) for compression, and the CTUs in the picture have the same size, for example, 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU includes three coding tree blocks (CTBs), namely one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64×64 pixel CTU can be partitioned into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of the CU, for example, an inter-frame prediction type or an intra-frame prediction type. Based on temporal and / or spatial predictability, the CU is divided into one or more prediction units (PUs). Typically, each PU includes one luma prediction block (PB) and two chroma PBs. In an embodiment, a prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0102] Figure 7 A diagram of a video encoder (703) according to another embodiment of the present disclosure is shown. The video encoder (703) is configured to receive a processed block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures and encode the processed block into an encoded image as part of an encoded video sequence. In one example, the video encoder (703) is used instead of Figure 4 The video encoder (403) in the example.

[0103] In the HEVC example, a video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) determines whether to best encode the processing block using intra mode, inter mode, or a bi-prediction mode such as rate-distortion optimization. When the processing block is to be encoded in intra mode, the video encoder (703) may encode the processing block into a coded picture using intra prediction techniques; and when the processing block is to be encoded in inter mode or bi-prediction mode, the video encoder (703) may encode the processing block into a coded picture using inter prediction or bi-prediction techniques, respectively. In some video coding techniques, the merge mode may be an inter-picture prediction submode, in which motion vectors are derived from one or more motion vector predictors without benefiting from a coded motion vector component outside the predictor. In some other video coding techniques, there may be a motion vector component applicable to the object block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the mode of the processing block.

[0104] exist Figure 7 In the example of , the video encoder (703) includes Figure 7 Shown are an inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together.

[0105] The inter-frame encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-frame prediction information (e.g., a description of redundant information according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a prediction block) based on the inter-frame prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture decoded based on the encoded video information.

[0106] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block with blocks already encoded in the same picture, generate quantized coefficients after transformation, and, in some cases, generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). In one example, the intra encoder (722) further calculates an intra prediction result (e.g., a prediction block) based on the intra prediction information and a reference block in the same picture.

[0107] The general controller (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines a mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream; when the mode is inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0108] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) is configured to operate based on the residual data to encode the residual data to generate transform coefficients. In one example, the residual encoder (724) is configured to convert the residual data from the spatial domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (703) further includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and intra prediction information. The decoded blocks are processed appropriately to generate a decoded picture, and the decoded picture may be buffered in memory circuitry (not shown) and used as a reference picture in some examples.

[0109] The entropy encoder (725) is configured to format the bitstream to include the coded blocks. The entropy encoder (725) is configured to include various information according to a suitable standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. Note that according to the disclosed subject matter, when encoding a block in inter mode or the merge sub-mode of bi-prediction mode, there is no residual information.

[0110] Figure 8A diagram of a video decoder (810) according to another embodiment of the present disclosure is shown. The video decoder (810) is configured to receive an encoded picture as part of an encoded video sequence and decode the encoded picture to generate a reconstructed picture. In one example, the video decoder (810) is used instead of Figure 4 A video decoder (410) is shown in the example.

[0111] exist Figure 8 In the example of , the video decoder (810) includes Figure 8 Shown are an entropy decoder (871), an inter-frame decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-frame decoder (872) coupled together.

[0112] The entropy decoder (871) can be configured to reconstruct certain symbols representing syntax elements that make up the coded image from the coded image. Such symbols can include, for example, the mode in which the block is coded (e.g., intra mode, inter mode, bidirectional prediction mode, merge submode, or the latter two modes in another submode), prediction information (e.g., intra prediction information or inter prediction information) that can identify specific samples or metadata used for prediction by the intra decoder (872) or inter decoder (880), respectively, and residual information such as quantized transform coefficients. In one example, when the prediction mode is inter or bidirectional prediction mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information can be subjected to inverse quantization and provided to the residual decoder (873).

[0113] The inter-frame decoder (880) is configured to receive inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0114] The intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0115] The residual decoder (873) is configured to perform inverse quantization to extract dequantized transform coefficients and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to include quantizer parameters (QP)), and this information may be provided by the entropy decoder (871) (the data path is not shown because this may only be a small amount of control information).

[0116] The reconstruction module (874) is configured to combine the residual output by the residual decoder (873) and the prediction result (output by the inter-frame or intra-frame prediction module, as appropriate) in the spatial domain to form a reconstructed block, which can be part of a reconstructed picture, which in turn can be part of a reconstructed video. Note that other suitable operations such as deblocking can be performed to improve visual quality.

[0117] Note that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented using any suitable technology. In one embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) can be implemented using one or more processors executing software instructions.

[0118] This disclosure describes video coding techniques related to neural image compression techniques and / or neural video compression techniques, such as artificial intelligence (AI)-based neural image compression (NIC). Aspects of this disclosure include content-adaptive online training in NIC, such as a block-based content-adaptive online training NIC method for an end-to-end (E2E) optimized image coding framework based on a neural network. The neural network (NN) may include an artificial neural network (ANN), such as a deep neural network (DNN), a convolutional neural network (CNN), and the like.

[0119] In one embodiment, the related hybrid video codec is difficult to be optimized as a whole. For example, improvements to a single module (e.g., an encoder) in a hybrid video codec may not result in coding gains in overall performance. In an NN-based video coding framework, different modules can be jointly optimized from input to output by performing a learning process or training process (e.g., a machine learning process) to improve the final target (e.g., rate-distortion performance, such as the rate-distortion loss L described in the present disclosure), thereby generating an E2E optimized NIC.

[0120] An exemplary NIC framework or system can be described as follows. The NIC framework can use an input block x as input to a neural network encoder (e.g., an encoder based on a neural network (e.g., DNN)) to compute a compressed representation (e.g., a compact representation) The compressed representation can be compact, for example, for storage and transmission purposes. A neural network decoder (e.g., a decoder based on a neural network (e.g., DNN)) can use the compressed representation as input to reconstruct an output block (also referred to as a reconstructed block). In various embodiments, the input block x and the reconstruction block In the spatial domain, the compressed representation In a domain other than the spatial domain. In some examples, the compressed representation quantized and entropy coded.

[0121] In some examples, the NIC framework can use a variational autoencoder (VAE) structure. In the VAE structure, the neural network encoder can directly use the entire input block x as the input of the neural network encoder. The entire input block x can be passed through a set of neural network layers that work as a black box to calculate the compressed representation Compressed Representation is the output of the neural network encoder. The neural network decoder can represent the entire compressed As input. Compressed representation can be passed through another set of neural network layers that work as another black box to compute the reconstruction block Rate-distortion (RD) loss can be optimized to achieve the reconstruction block with a tradeoff hyperparameter λ Distortion loss and compact representation A trade-off between the bit consumption R.

[0122]

[0123] A neural network (e.g., an ANN) can learn to perform tasks from examples without requiring task-specific programming. An ANN can be configured with connected nodes or artificial neurons. The connection between the nodes can transmit a signal from a first node to a second node (e.g., a receiving node), and the signal can be modified by a weight, which can be indicated by a weight coefficient of the connection. The receiving node can process a signal from a node that sends a signal to the receiving node (i.e., an input signal of the receiving node) and then generate an output signal by applying a function to the input signal. The function can be a linear function. In one example, the output signal is a weighted sum of the input signals. In one example, the output signal is further modified by a bias, which can be indicated by a bias term, so that the output signal is the sum of the bias and the weighted sum of the input signals. The function can include, for example, a nonlinear operation on the weighted sum of the input signals or the sum of the bias and the weighted sum. The output signal can be sent to a node connected to the receiving node (downstream node). An ANN can be represented or configured by parameters (e.g., weights of the connection and / or bias). The weights and / or biases can be obtained by training the ANN with examples in which the weights and / or biases can be iteratively adjusted. The trained ANN configured with the determined weights and / or the determined biases may be used to perform a task.

[0124] The nodes in an ANN can be organized using any suitable architecture. In various embodiments, the nodes in the ANN are organized into layers, including an input layer that receives input signals to the ANN and an output layer that outputs output signals from the ANN. In one embodiment, the ANN also includes layers between the input layer and the output layer, such as hidden layers. Different layers can perform different types of transformations on their respective inputs. Signals can be transmitted from the input layer to the output layer.

[0125] An ANN with multiple layers between the input and output layers may be referred to as a DNN. In one embodiment, the DNN is a feed-forward network, where data flows from the input layer to the output layer without looping back. In one example, the DNN is a fully connected network, where every node in one layer is connected to all nodes in the next layer. In one embodiment, the DNN is a recurrent neural network (RNN), where data can flow in any direction. In one embodiment, the DNN is a CNN.

[0126] A CNN may include an input layer, an output layer, and a hidden layer between the input layer and the output layer. The hidden layer may include a convolution layer (e.g., used in an encoder) that performs convolution (e.g., two-dimensional (2D) convolution). In one embodiment, the 2D convolution performed in the convolution layer is between a convolution kernel (also referred to as a filter or channel, e.g., a 5×5 matrix) and an input signal to the convolution layer (e.g., a 2D matrix, e.g., a 2D block, a 256×256 matrix). In various examples, the dimension of the convolution kernel (e.g., 5×5) is smaller than the dimension of the input signal (e.g., 256×256). Therefore, the portion of the input signal (e.g., a 256×256 matrix) covered by the convolution kernel (e.g., a 5×5 area) is smaller than the area of ​​the input signal (e.g., a 256×256 area), and can therefore be referred to as a receptive field in a corresponding node in the next layer.

[0127] During convolution, the dot product of the convolution kernel and the corresponding receptive field in the input signal is calculated. Therefore, each element of the convolution kernel is a weight applied to the corresponding sample in the receptive field, and the convolution kernel therefore includes weights. For example, a convolution kernel represented by a 5×5 matrix has 25 weights. In some examples, a bias is applied to the output signal of the convolution layer, and the output signal is based on the sum of the dot product and the bias.

[0128] The convolution kernel can move along the input signal (e.g., a 2D matrix) by an amount called the stride, so that the convolution operation generates a feature map or activation map (e.g., another 2D matrix), which in turn contributes to the input of the next layer in the CNN. For example, if the input signal is a 2D block with 256×256 samples, the stride is 2 samples (e.g., a stride of 2). For a stride of 2, the convolution kernel moves 2 samples along the X direction (e.g., horizontally) and / or the Y direction (e.g., vertically).

[0129] Multiple convolution kernels can be applied to the input signal in the same convolution layer to generate multiple feature maps, respectively, where each feature map can represent a specific feature of the input signal. In general, a convolution layer with N channels (i.e., N convolution kernels) (each convolution kernel has M×M samples and a stride of S) can be specified as Conv:MxM cN sS. For example, a convolution layer with 192 channels (each convolution kernel has 5×5 samples and a stride of 2) is specified as Conv:5x5 c192s2. The hidden layer may include a deconvolution layer (e.g., used in a decoder) that performs deconvolution (e.g., 2D deconvolution). Deconvolution is the inverse operation of convolution. A deconvolution layer with 192 channels (each deconvolution kernel has 5×5 samples and a stride of 2) is specified as DeConv:5x5 c192 s2.

[0130] In various embodiments, CNN has the following benefits. The number of learnable parameters (i.e., parameters to be trained) in a CNN can be significantly smaller than the number of learnable parameters in a DNN (e.g., a feedforward DNN). In a CNN, a relatively large number of nodes can share the same filter (e.g., the same weight) and the same bias (if a bias is used), thereby reducing memory usage because a single bias and a single weight vector can be used on all receptive fields that share the same filter. For example, for an input signal with 100×100 samples, a convolutional layer with a convolution kernel of 5×5 samples has 25 learnable parameters (e.g., weights). If a bias is used, one channel uses 26 learnable parameters (e.g., 25 weights and one bias). If the convolutional layer has N channels, the total learnable parameters are 26xN. On the other hand, for a fully connected layer in a DNN, 100x100 (i.e., 10,000) weights are used for each node in the next layer. If the next layer has L nodes, the total learnable parameters are 10,000xL.

[0131] CNN may also include one or more other layers, such as a pooling layer, a fully connected layer that can connect each node in one layer to each node in another layer, a normalization layer, etc. The layers in a CNN can be arranged in any suitable order and in any suitable architecture (e.g., a feedforward architecture, a recursive architecture). In one example, the convolutional layer is followed by other layers, such as a pooling layer, a fully connected layer, a normalization layer, etc.

[0132] Pooling layers can be used to reduce the dimensionality of data by combining the outputs of multiple nodes from one layer into a single node in the next layer. The following describes the pooling operation of a pooling layer that takes a feature map as input. The description can be appropriately applied to other input signals. The feature map can be divided into sub-regions (e.g., rectangular sub-regions), and the features in the corresponding sub-regions can be independently downsampled (or pooled) into a single value, for example, by taking the average in average pooling or taking the maximum value in maximum pooling.

[0133] The pooling layer can perform pooling, such as local pooling, global pooling, maximum pooling, average pooling, etc. Pooling is a form of nonlinear downsampling. Local pooling combines a small number of nodes in the feature map (e.g., a local cluster of nodes, such as 2×2 nodes). For example, global pooling can combine all nodes of the feature map.

[0134] Pooling layers can reduce the size of representations, thereby reducing the number of parameters, memory usage, and computational effort in CNNs. In one example, a pooling layer is inserted between consecutive convolutional layers in a CNN. In one example, the pooling layer is followed by an activation function, such as a rectified linear unit (ReLU) layer. In one example, the pooling layer is omitted between consecutive convolutional layers in a CNN.

[0135] The normalization layer can be ReLU, leaky ReLU, generalized division normalization (GDN), inverse GDN (IGDN), etc. ReLU can apply a non-saturating activation function to remove negative values ​​from the input signal (e.g., feature map) by setting negative values ​​to zero. For negative values, the leaky ReLU can have a small slope (e.g., 0.01) instead of a flat slope (e.g., 0). Therefore, if the value x is greater than 0, the output from the leaky ReLU is x. Otherwise, the output from the leaky ReLU is the value x multiplied by the small slope (e.g., 0.01). In one example, the slope is determined before training and is therefore not learned during training.

[0136] In NN-based image compression methods, such as DNN-based or CNN-based image compression methods, instead of directly encoding the entire image, a block-based or block-based coding mechanism can effectively compress images in DNN-based video coding standards (e.g., FVC). The entire image can be divided into blocks of equal (or different) sizes, and these blocks can be compressed individually. In one embodiment, the image can be divided into blocks of equal or unequal sizes. The divided blocks can be compressed instead of the image. Figure 9A An example of block image coding according to an embodiment of the present disclosure is shown. The image (980) can be divided into blocks, for example, blocks (981)-(996). For example, blocks (981)-(996) can be compressed according to the scanning order. Figure 9AIn the example shown, blocks (981)-(989) have already been compressed, and blocks (990)-(996) will be compressed.

[0137] The image can be viewed as a block. In one embodiment, the image is compressed without being divided into blocks. The entire image can be input to the E2E NIC framework.

[0138] Figure 9B An exemplary NIC framework (900) (e.g., a NIC system) according to an embodiment of the present disclosure is shown. The NIC framework (900) can be based on a neural network, such as a DNN and / or a CNN. The NIC framework (900) can be used to compress (e.g., encode) a block and decompress (e.g., decode or reconstruct) a compressed block (e.g., an encoded block). The NIC framework (900) can include two sub-neural networks implemented using a neural network, namely a first sub-NN (951) and a second sub-NN (952).

[0139] The first sub-NN (951) can be similar to an autoencoder and can be trained to generate a compressed block of the input block x And decompress the compressed block To obtain the reconstructed block The first sub-NN (951) may include multiple components (or modules), such as a primary encoder neural network (or primary encoder network) (911), a quantizer (912), an entropy encoder (913), an entropy decoder (914), and a primary decoder neural network (or primary encoder network) (915). Figure 9B , the primary encoder network (911) can generate a latent or potential representation y from an input block x (e.g., a block to be compressed or encoded). In one example, the primary encoder network (911) is implemented using a CNN. The relationship between the latent representation y and the input block x can be described using Equation 2.

[0140] y = f1(x; θ1) Equation 2

[0141] Wherein, the parameter θ1 represents the parameters, e.g., the weights and biases used in the convolution kernels in the main encoder network (911) (if biases are used in the main encoder network (911)).

[0142] The latent representation y may be quantized using a quantizer (912) to generate a quantized latent The potential for quantification can be compressed For example, the entropy encoder (913) uses lossless compression to generate a compressed block (eg, an encoded block) (931), which is a compressed representation of the input block x The entropy encoder (913) can use an entropy coding technique, such as Huffman coding, arithmetic coding, etc. In one example, the entropy encoder (913) uses arithmetic coding and is an arithmetic encoder. In one example, the coded block (931) is transmitted in a coded bitstream.

[0143] The coded block (931) can be decompressed (e.g., entropy decoded) by an entropy decoder (914) to generate an output. The entropy decoder (914) can use an entropy coding technique corresponding to the entropy coding technique used in the entropy encoder (913), such as Huffman coding, arithmetic coding, etc. In one example, the entropy decoder (914) uses arithmetic decoding and is an arithmetic decoder. In one example, lossless compression is used in the entropy encoder (913), lossless decompression is used in the entropy decoder (914), and noise such as that generated by the transmission of the coded block (931) can be ignored. The output from the entropy decoder (914) is a quantized potential.

[0144] The main decoder network (915) can decode the quantized latent To generate reconstruction blocks In one example, the main decoder network (915) is implemented using a CNN. Reconstruction Block (i.e., the output of the main decoder network (915)) and the quantized potential The relationship between (i.e., the inputs to the main decoder network (915)) can be described using Equation 3.

[0145]

[0146] Wherein, the parameter θ2 represents the parameters, e.g., weights and biases used in the convolution kernels in the main decoder network (915) (if biases are used in the main decoder network (915)). Thus, the first sub-NN (951) can compress (e.g., encode) the input block x to obtain the encoded block (931) and decompress (e.g., decode) the encoded block (931) to obtain the reconstructed block Due to the quantization loss introduced by the quantizer (912), the reconstructed block x can be different from the input block x.

[0147] The second sub-NN (952) can be used for entropy coding in the quantization potential The entropy model (e.g., a prior probability model) is learned on the context model. Therefore, the entropy model can be a conditional entropy model, for example, a Gaussian mixture model (GMM), a Gaussian scale model (GSM) depending on the input block x. The second sub-NN (952) can include a context model NN (916), an entropy parameter NN (917), a super encoder (921), a quantizer (922), an entropy encoder (923), an entropy decoder (924), and a super decoder (925). The entropy model used in the context model NN (916) can be a latent image (e.g., a quantized latent image). ) autoregressive model. In one example, the super encoder (921), quantizer (922), entropy encoder (923), entropy decoder (924), and super decoder (925) form a super neural network (e.g., super NN). The super neural network can represent information useful for correcting context-based predictions. Data from the context model NN (916) and the super neural network can be combined by entropy parameters NN (917). The entropy parameters NN (917) can generate parameters, such as mean and scale parameters for an entropy model such as a conditional Gaussian entropy model (e.g., GMM).

[0148] refer to Figure 9B , on the encoder side, the quantized potential from the quantizer (912) is fed into the context model NN (916). On the decoder side, the quantized potential from the entropy decoder (914) is fed into the context model NN (916). The context model NN (916) can be implemented using a neural network such as CNN. The context model NN (916) can be based on the context Generate output o cm,i , which outputs a quantized potential available to the context model NN (916) Context The output of the context model NN (916) may include a previous quantized latent image at the encoder side or a previous entropy decoded quantized latent image at the decoder side. cm,i and input (e.g., ) can be described using Equation 4.

[0149]

[0150] Wherein, the parameter θ3 represents the parameters, e.g., the weights and biases used in the convolution kernel in the context model NN (916) (if biases are used in the context model NN (916)).

[0151] Output o from context model NN (916) cm,i and the output o from the super decoder (925)hc is fed into the entropy parameter NN (917) to generate the output o ep The entropy parameter NN (917) may be implemented using a neural network such as a CNN. The output of the entropy parameter NN (917) is ep and input (e.g., cm,i and o hc ) can be described using Equation 5.

[0152] o ep =f4(o cm,i ,o hc ;θ4) Equation 5

[0153] Wherein, the parameter θ4 represents the parameters, such as the weights and biases used in the convolution kernel in the entropy parameter NN (917) (if a bias is used in the entropy parameter NN (917)). The output o of the entropy parameter NN (917) ep can be used to determine (eg, adjust) the entropy model, and thus the adjusted entropy model can depend on the input block x, for example, via the output o from the super decoder (925). hc In one example, the output o ep Includes parameters used to tune entropy models (e.g., GMM), such as mean and scale parameters. Figure 9B , the entropy encoder (913) and the entropy decoder (914) can use an entropy model (e.g., a conditional entropy model) in entropy encoding and entropy decoding, respectively.

[0154] The second sub-NN (952) can be described as follows. The potential y can be fed into the super encoder (921) to generate a super potential z. In one example, the super encoder (921) is implemented using a neural network such as a CNN. The relationship between the super potential z and the potential y can be described using Equation 6.

[0155] z = f5(y; θ5) Equation 6

[0156] Wherein, the parameter θ5 represents the parameters, such as the weights and biases used in the convolution kernel in the super encoder (921) (if biases are used in the super encoder (921)).

[0157] The quantizer (922) quantizes the super potential z to generate a quantized potential The potential for quantification can be compressed For example, side information, such as coded bits (932) from a super neural network, is generated using lossless compression by an entropy encoder (923). The entropy encoder (923) can use entropy coding techniques, such as Huffman coding, arithmetic coding, and the like. In one example, the entropy encoder (923) uses arithmetic coding and is an arithmetic encoder. In one example, side information such as coded bits (932) can be transmitted in a coded bitstream, such as with the coded block (931).

[0158] Side information such as coded bits (932) can be decompressed (e.g., entropy decoded) by an entropy decoder (924) to generate an output. The entropy decoder (924) can use an entropy coding technique, such as Huffman coding, arithmetic coding, etc. In one example, the entropy decoder (924) uses arithmetic decoding and is an arithmetic decoder. In one example, lossless compression is used in the entropy encoder (923), lossless decompression is used in the entropy decoder (924), and noise such as that caused by the transmission of the side information can be ignored. The output from the entropy decoder (924) can be a quantized potential. The super decoder (925) can decode the quantized potential To generate the output to hc Output to hc and quantified potential The relationship between can be described using Equation 7.

[0159]

[0160] Wherein, the parameter θ6 represents the parameters, such as the weights and biases used in the convolution kernel in the super decoder (925) (if biases are used in the super decoder (925)).

[0161] As described above, compressed or coded bits (932) can be added to the coded bitstream as side information, which enables the entropy decoder (914) to use a conditional entropy model. Thus, the entropy model can be block-dependent and spatially adaptive, and thus can be more accurate than a fixed entropy model.

[0162] The NIC framework (900) may be modified appropriately, for example, by omitting Figure 9B Modify one or more components shown in Figure 9B One or more components shown in, and / or including Figure 9B In one example, the NIC framework using the fixed entropy model includes the first sub-NN (951) and does not include the second sub-NN (952). In one example, the NIC framework includes the components of the NIC framework (900) except the entropy encoder (923) and the entropy decoder (924).

[0163] In one embodiment, Figure 9B One or more components in the illustrated NIC framework (900) are implemented using a neural network (e.g., a CNN). Each NN-based component (e.g., a primary encoder network (911), a primary decoder network (915), a context model NN (916), an entropy parameter NN (917), a super encoder (921), or a super decoder (925)) in the NIC framework (e.g., the NIC framework (900)) can include any suitable architecture (e.g., having any suitable combination of layers), include any suitable type of parameters (e.g., weights, biases, a combination of weights and biases, and / or the like), and include any suitable number of parameters.

[0164] In one embodiment, the main encoder network (911), the main decoder network (915), the context model NN (916), the entropy parameter NN (917), the super encoder (921) and the super decoder (925) are implemented using corresponding CNNs.

[0165] Figure 10 An exemplary CNN of the main encoder network (911) according to an embodiment of the present disclosure is shown. For example, the main encoder network (911) includes four groups of layers, where each group of layers includes a convolutional layer 5x5 c192 s2 followed by a GDN layer. The following examples may be modified and / or omitted: Figure 10 Additional layers may be added to the main encoder network (911).

[0166] Figure 11 An exemplary CNN of the main decoder network (915) according to an embodiment of the present disclosure is shown. For example, the main decoder network (915) includes three groups of layers, where each group of layers includes a deconvolution layer 5x5 c192 s2 followed by an IGDN layer. In addition, the three groups of layers are followed by a deconvolution layer 5x5 c3 s2 followed by an IGDN layer. The above examples may be modified and / or omitted. Figure 11 Additional layers may be added to the main decoder network (915).

[0167] Figure 12 An exemplary CNN of a super encoder (921) according to an embodiment of the present disclosure is shown. For example, the super encoder (921) includes a convolutional layer 3x3 c192 s1, followed by a leaky ReLU, a convolutional layer 5x5 c192s2, followed by a leaky ReLU, and a convolutional layer 5x5 c192 s2. The following may be modified and / or omitted: Figure 12 Additional layers may be added to the super encoder (921).

[0168] Figure 13 An exemplary CNN of a super decoder (925) according to an embodiment of the present disclosure is shown. For example, the super decoder (925) includes a deconvolution layer 5x5 c192 s2, followed by a leaky ReLU, a convolution layer 5x5 c288s2, followed by a leaky ReLU, and a convolution layer 3x3 c384 s1. The above can be modified and / or omitted. Figure 13 Additional layers may be added to the super encoder (925).

[0169] Figure 14 An exemplary CNN of the context model NN (916) according to an embodiment of the present disclosure is shown. For example, the context model NN (916) includes a masked convolution 5x5 c384 s1 for context prediction, so the context in Equation 4 is Include limited context (e.g., 5×5 convolution kernel). Can be modified Figure 14 Additional layers may be added to the context model NN (916).

[0170] Figure 15 An exemplary CNN of an entropy parameter NN (917) according to an embodiment of the present disclosure is shown. For example, the entropy parameter NN (917) includes a convolutional layer 1x1 c640 s1, followed by a leaky ReLU, a convolutional layer 1x1 c512s1, followed by a leaky ReLU, and a convolutional layer 1x1 c384 s1. The following examples may be modified and / or omitted: Figure 15 Additional layers may be added to the entropy parameter NN (917).

[0171] The NIC framework (900) can be implemented using CNN, as shown in Figure 10-15 As described. The NIC framework (900) may be suitably adapted such that one or more components (e.g., (911), (915), (916), (917), (921), and / or (925)) in the NIC framework (900) are implemented using any suitable type of neural network (e.g., a CNN-based or non-CNN-based neural network). One or more other components of the NIC framework (900) may be implemented using a neural network.

[0172] The NIC framework (900) including a neural network (e.g., CNN) can be trained to learn parameters used in the neural network. For example, when using a CNN, parameters represented by θ1-θ6, such as weights and biases used in the convolution kernels in the main encoder network (911) (if biases are used in the main encoder network (911)), weights and biases used in the convolution kernels in the main decoder network (915) (if biases are used in the main decoder network (915)), weights and biases used in the convolution kernels in the super encoder (921) (if biases are used in the super encoder (921)), weights and biases used in the convolution kernels in the super decoder (925) (if biases are used in the super decoder (925)), weights and biases used in the convolution kernels in the context model NN (916) (if biases are used in the context model NN (916)), and weights and biases used in the convolution kernels in the entropy parameter NN (917) (if biases are used in the entropy parameter NN (917)), can be learned during training.

[0173] In one example, reference Figure 10 , the main encoder network (911) includes four convolutional layers, where each convolutional layer has a 5×5 convolution kernel and 192 channels. Therefore, the number of weights used in the convolution kernels in the main encoder network (911) is 19200 (i.e., 4x5x5x192). The parameters used in the main encoder network (911) include 19200 weights and an optional bias. Additional parameters may be included when bias and / or additional NNs are used in the main encoder network (911).

[0174] refer to Figure 9B , the NIC framework (900) includes at least one component or module built on a neural network. The at least one component may include one or more of a main encoder network (911), a main decoder network (915), a super encoder (921), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). The at least one component can be trained individually. In one example, the training process is used to learn the parameters of each component separately. The at least one component can be trained together as a group. In one example, the training process is used to learn the parameters of a subset of at least one component together. In one example, the training process is used to learn the parameters of all at least one component, which is therefore called end-to-end optimization.

[0175] During the training of one or more components in the NIC framework (900), weights (or weight coefficients) of the one or more components may be initialized. In one example, the weights are initialized based on a pre-trained corresponding neural network model (e.g., a DNN model, a CNN model). In one example, the weights are initialized by setting them to random numbers.

[0176] For example, after initializing the weights, a set of training blocks can be used to train one or more components. The set of training blocks can include any suitable blocks of any suitable size. In some examples, the set of training blocks includes blocks of original images, natural images, computer-generated images, etc. in the spatial domain. In some examples, the set of training blocks includes blocks of residual blocks or residual images with residual data in the spatial domain. The residual data can be calculated by a residual calculator (e.g., residual calculator (723)). In some examples, the original image and / or the residual image including the residual data can be used directly to train the neural network in the NIC framework. Therefore, the original image, the residual image, the block from the original image, and / or the block from the residual image can be used to train the neural network in the NIC framework.

[0177] For the sake of brevity, the following uses a training block as an example to describe the training process. The description can be applied to the training block as appropriate. A training block t in the set of training blocks can be represented by Figure 9B The encoding process in , to generate a compressed representation (e.g., coded information, e.g., a bitstream). The coded information can be obtained by Figure 9B The decoding process described in the above is used to calculate and reconstruct the reconstruction block

[0178] For the NIC framework (900), two competing objectives are balanced, e.g., reconstruction quality and bit consumption. The quality loss function (e.g., distortion or distortion loss) Can be used to indicate reconstruction quality, e.g., reconstruction (e.g., reconstructed blocks ) and the difference between the original block (e.g., training block t). The rate (or rate loss) R can be used to indicate the bit consumption of the compressed representation. In one example, the rate loss R also includes side information used, for example, in determining the context model.

[0179] For neural image compression, a differentiable approximation of quantization can be used in E2E optimization. In various examples, during training of neural network-based image compression, noise injection is used to simulate quantization, so that quantization is simulated by noise injection rather than being performed by a quantizer (e.g., quantizer (912)). Therefore, training with noise injection can variably approximate the quantization error. A bits per pixel (BPP) estimator can be used to simulate an entropy encoder, so that entropy encoding is simulated by the BPP estimator rather than being performed by an entropy encoder (e.g., (913)) and an entropy decoder (e.g., (914)). Therefore, for example, the rate loss R in the loss function L shown in Equation 1 during training can be estimated based on the noise injection and the BPP estimator. In general, a higher rate R can achieve lower distortion D, while a lower rate R will result in higher distortion D. The trade-off hyperparameter λ in Equation 1 can be used to optimize a common RD loss L, where L can be optimized as the sum of λD and R. The training process can be used to adjust the parameters of one or more components (e.g., (911), (915)) in the NIC framework (900) so that the joint RD loss L is minimized or optimized. In one example, the joint rate-distortion (RD) loss can be optimized using a tradeoff hyperparameter λ as follows:

[0180]

[0181] where E measures the distortion of the decoded block residual compared to the original block residual before encoding, which acts as a regularization loss for the residual encoding / decoding DNN and the encoding / decoding DNN. β is a hyperparameter used to balance the importance of the regularization loss.

[0182] Various models can be used to determine the distortion loss D and rate loss R, thereby determining the common RD loss L in Equation 1. In one example, the distortion loss It is expressed as Peak Signal-to-Noise Ratio (PSNR), which is a metric based on mean square error, Multi-Scale Structural Similarity (MS-SSIM) quality index, a weighted combination of PSNR and MS-SSIM, etc.

[0183] In one example, the goal of the training process is to train an encoding neural network (e.g., encoding DNN), e.g., a video encoder to be used on the encoder side, and to train a decoding neural network (e.g., decoding DNN), e.g., a video decoder to be used on the decoder side. Figure 9BThe encoding neural network may include a main encoder network (911), a super encoder (921), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). The decoding neural network may include a main decoder network (915), a super decoder (925), a context model NN (916), and an entropy parameter NN (917). The video encoder and / or video decoder may include other components based on NN and / or not based on NN.

[0184] The NIC framework (e.g., NIC framework (900)) can be trained in an E2E manner. In one example, the encoding neural network and the decoding neural network are jointly updated in an E2E manner based on back-propagation gradients during training.

[0185] After training the parameters of the neural network in the NIC framework (900), one or more components in the NIC framework (900) can be used to encode and / or decode the block. In one embodiment, on the encoder side, the video encoder is configured to encode the input block x into an encoded block (931) to be transmitted in the bitstream. The video encoder can include multiple components in the NIC framework (900). In one embodiment, on the decoder side, a corresponding video decoder is configured to decode the encoded block (931) in the bitstream into a reconstructed block x. The video decoder can include multiple components in the NIC framework (900).

[0186] In one example, for example, when content-adaptive online training is employed, the video encoder includes all components in the NIC framework (900).

[0187] Figure 16A An exemplary video encoder (1600A) according to an embodiment of the present disclosure is shown. The video encoder (1600A) includes a reference Figure 9B The main encoder network (911), quantizer (912), entropy encoder (913) and second sub-NN (952) are described, and detailed description is omitted for the sake of brevity. Figure 16B An exemplary video decoder (1600B) according to an embodiment of the present disclosure is shown. The video decoder (1600B) may correspond to the video encoder (1600A). The video decoder (1600B) may include a main decoder network (915), an entropy decoder (914), a context model NN (916), an entropy parameter NN (917), an entropy decoder (924), and a super decoder (925). Figures 16A-16B On the encoder side, the video encoder (1600A) can generate coded blocks (931) and coded bits (932) to be transmitted in a bitstream. On the decoder side, the video decoder (1600B) can receive and decode the coded blocks (931) and coded bits (932).

[0188] Figure 17-18 An exemplary video encoder (1700) and a corresponding video decoder (1800) according to an embodiment of the present disclosure are shown, respectively. Figure 17 , the encoder (1700) includes a main encoder network (911), a quantizer (912) and an entropy encoder (913). Figure 9B Describes an example of a main encoder network (911), a quantizer (912), and an entropy encoder (913). Figure 18 , the video decoder (1800) includes a main decoder network (915) and an entropy decoder (914). Figure 9B Describes an example of a main decoder network (915) and an entropy decoder (914). Figure 17 and 18 , the video encoder (1700) can generate a coded block (931) to be transmitted in a bitstream. The video decoder (1800) can receive and decode the coded block (931).

[0189] As described above, the NIC framework (900) including a video encoder and a video decoder can be trained based on images and / or blocks in the set of training images. In some examples, one or more blocks to be compressed (e.g., encoded) and / or transmitted have properties that are significantly different from those of the set of training blocks. Therefore, directly encoding and decoding one or more blocks using a video encoder and a video decoder trained based on the set of training blocks, respectively, can result in relatively poor RD loss L (e.g., relatively large distortion and / or relatively large bit rate). Therefore, aspects of the present disclosure describe content-adaptive online training methods for NICs, for example, block-wise content-adaptive online training methods for NICs.

[0190] In the block-based content adaptive online training method, the input image can be divided into blocks, and one or more blocks can be used to update one or more parameters in the pre-trained NIC framework to one or more replacement parameters by optimizing rate-distortion performance. Neural network update information indicating one or more replacement parameters or a subset of one or more replacement parameters can be encoded into the bitstream together with the encoded one or more blocks. On the decoder side, the video decoder can decode the encoded one or more blocks and can achieve better compression performance by using one or more replacement parameters or a subset of one or more replacement parameters. The block-based content adaptive online training method can be used as a preprocessing step (e.g., a pre-encoding step) to improve the compression performance of the pre-trained E2ENIC compression method.

[0191] To distinguish between a training process based on the set of training blocks and a content-adaptive online training process based on one or more blocks to be compressed (e.g., encoded) and / or transmitted, the NIC framework (900), video encoder, and video decoder trained by the set of training blocks are referred to as a pre-trained NIC framework (900), a pre-trained video encoder, and a pre-trained video decoder, respectively. Parameters in the pre-trained NIC framework (900), pre-trained video encoder, or pre-trained video decoder are referred to as NIC pre-trained parameters, encoder pre-trained parameters, and decoder pre-trained parameters, respectively. In one example, the NIC pre-trained parameters include encoder pre-trained parameters and decoder pre-trained parameters. In one example, the encoder pre-trained parameters and the decoder pre-trained parameters do not overlap, wherein none of the encoder pre-trained parameters are included in the decoder pre-trained parameters. For example, the encoder pre-trained parameters in (1700) (e.g., the pre-trained parameters in the main encoder network (911)) and the decoder pre-trained parameters in (1800) (e.g., the pre-trained parameters in the main decoder network (915)) do not overlap. In one example, the encoder pre-trained parameters and the decoder pre-trained parameters overlap, wherein at least one of the encoder pre-trained parameters is included in the decoder pre-trained parameters. For example, the encoder pre-trained parameters in (1600A) (e.g., pre-trained parameters in context model NN (916)) and the decoder pre-trained parameters in (1600B) (e.g., pre-trained parameters in context model NN (916)) overlap. The NIC pre-trained parameters can be obtained based on blocks and / or images in the set of training blocks.

[0192] The content-adaptive online training process may be referred to as a fine-tuning process, as described below. One or more pre-trained parameters in the NIC pre-trained parameters in the pre-trained NIC framework (900) may be further trained (e.g., fine-tuned) based on one or more blocks to be encoded and / or transmitted, wherein the one or more blocks may be different from the set of training blocks. The one or more pre-trained parameters used in the NIC pre-trained parameters may be fine-tuned by optimizing a common RD loss L based on the one or more blocks. The one or more pre-trained parameters that have been fine-tuned by the one or more blocks are referred to as one or more replacement parameters or one or more fine-tuning parameters. In one embodiment, after one or more pre-trained parameters in the NIC pre-trained parameters have been fine-tuned (e.g., replaced) by the one or more replacement parameters, neural network update information is encoded into the bitstream to indicate the one or more replacement parameters or a subset of the one or more replacement parameters. In one example, the NIC framework (900) is updated (or fine-tuned), wherein the one or more pre-trained parameters are respectively replaced by the one or more replacement parameters.

[0193] In a first embodiment, the one or more pre-trained parameters include a first subset of one or more pre-trained parameters and a second subset of one or more pre-trained parameters. The one or more replacement parameters include a first subset of one or more replacement parameters and a second subset of one or more replacement parameters.

[0194] A first subset of one or more pre-trained parameters is used in a pre-trained video encoder and, for example, replaced by a first subset of one or more replacement parameters during a training process. Thus, the pre-trained video encoder is updated to an updated video encoder through the training process. The neural network update information may indicate a second subset of one or more replacement parameters that will replace the second subset of one or more replacement parameters. One or more blocks may be encoded using the updated video encoder and transmitted in a bitstream with the neural network update information.

[0195] On the decoder side, the second subset of one or more pre-trained parameters is used in the pre-trained video decoder. In one embodiment, the pre-trained video decoder receives and decodes the neural network update information to determine the second subset of one or more replacement parameters. When the second subset of one or more pre-trained parameters in the pre-trained video decoder is replaced with the second subset of one or more replacement parameters, the pre-trained video decoder is updated to an updated video decoder. The updated video decoder can be used to decode the one or more coded blocks.

[0196] Figures 16A-16B An example of the first case is shown. For example, the one or more pre-trained parameters include N1 pre-trained parameters in the pre-trained context model NN (916) and N2 pre-trained parameters in the pre-trained main decoder network (915). Therefore, the first subset of the one or more pre-trained parameters includes the N1 pre-trained parameters, and the second subset of the one or more pre-trained parameters is the same as the one or more pre-trained parameters. Therefore, the N1 pre-trained parameters in the pre-trained context model NN (916) can be replaced by the replacement parameters corresponding to N1, so that the pre-trained video encoder (1600A) can be updated to the updated video encoder (1600A). The pre-trained context model NN (916) is also updated to the updated context model NN (916). On the decoder side, the N1 pre-trained parameters can be replaced by the replacement parameters corresponding to N1, the N2 pre-trained parameters can be replaced by the replacement parameters corresponding to N2, the pre-trained context model NN (916) is updated to the updated context model NN (916), and the pre-trained main decoder network (915) is updated to the updated main decoder network (915). Therefore, the pre-trained video decoder (1600B) can be updated to the updated video decoder (1600B).

[0197] In the second case, one or more pre-trained parameters are not used in the pre-trained video encoder on the encoder side. Instead, one or more pre-trained parameters are used in the pre-trained video decoder on the decoder side. Thus, the pre-trained video encoder is not updated, and the pre-trained video encoder continues after the training process. In one embodiment, the neural network update information indicates one or more replacement parameters. One or more blocks can be encoded using the pre-trained video encoder and transmitted in the bitstream with the neural network update information.

[0198] On the decoder side, the pre-trained video decoder can receive and decode the neural network update information to determine one or more replacement parameters. When one or more pre-trained parameters in the pre-trained video decoder are replaced with one or more replacement parameters, the pre-trained video decoder is updated to an updated video decoder. The updated video decoder can be used to decode one or more coded blocks.

[0199] Figures 16A-16B An example of the second case is shown. For example, one or more pre-trained parameters include N2 pre-trained parameters (915) in the pre-trained main decoder network. Therefore, one or more pre-trained parameters are not used in the pre-trained video encoder on the encoder side (e.g., pre-trained video encoder (1600A)). Therefore, after the training process, the pre-trained video encoder (1600A) continues to be a pre-trained video encoder. On the decoder side, the N2 pre-trained parameters can be replaced by the replacement parameters corresponding to N2, which updates the pre-trained main decoder network (915) to the updated main decoder network (915). Therefore, the pre-trained video decoder (1600B) can be updated to the updated video decoder (1600B).

[0200] In the third case, one or more pre-trained parameters are used in a pre-trained video encoder and, for example, replaced by one or more replacement parameters during the training process. Thus, the pre-trained video encoder is updated to an updated video encoder through the training process. One or more blocks can be encoded using the updated video encoder and transmitted in the bitstream. The neural network update information is not encoded in the bitstream. On the decoder side, the pre-trained video decoder is not updated and remains the pre-trained video decoder. The pre-trained video decoder can be used to decode the one or more encoded blocks.

[0201] Figures 16A-16BAn example of the third case is shown. For example, one or more pre-trained parameters are in the pre-trained master encoder network (911). Therefore, one or more pre-trained parameters in the pre-trained master encoder network (911) can be replaced by one or more replacement parameters, so that the pre-trained video encoder (1600A) can be updated to the updated video encoder (1600A). The pre-trained master encoder network (911) is also updated to the updated master encoder network (911). On the decoder side, the pre-trained video decoder (1600B) is not updated.

[0202] In various examples, such as described in the first, second, and third scenarios, video decoding may be performed by pre-trained decoders with different capabilities, including decoders with and without the ability to update pre-trained parameters.

[0203] In one example, compression performance can be improved by encoding one or more blocks with an updated video encoder and / or an updated video decoder compared to encoding the one or more blocks with a pre-trained video encoder and a pre-trained video decoder. Thus, a content-adaptive online training method can be used to adapt a pre-trained NIC framework (e.g., pre-trained NIC framework (900)) to target block content (e.g., one or more blocks to be transmitted), thereby fine-tuning the pre-trained NIC framework. Thus, the video encoder on the encoder side and / or the video decoder on the decoder side can be updated.

[0204] The content-adaptive online training method can be used as a pre-processing step (e.g., a pre-encoding step) to improve the compression performance of the pre-trained E2E NIC compression method.

[0205] In one embodiment, one or more blocks include a single input block, and the fine-tuning process is performed using the single input block. The NIC framework (900) is trained and updated (e.g., fine-tuned) based on the single input block. An updated video encoder on the encoder side and / or an updated video decoder on the decoder side can be used to encode the single input block and optionally other input blocks. Neural network update information can be encoded into the bitstream along with the encoded single input block.

[0206] In one embodiment, one or more blocks include multiple input blocks, and the fine-tuning process is performed using the multiple input blocks. The NIC framework (900) is trained and updated (e.g., fine-tuned) based on the multiple input blocks. An updated video encoder on the encoder side and / or an updated decoder on the decoder side can be used to encode the multiple input blocks and optionally other input blocks. Neural network update information can be encoded into the bitstream along with the encoded multiple input blocks.

[0207] The rate loss R may increase as the neural network update information is signaled in the bitstream. When the one or more blocks include a single input block, the neural network update information is signaled for each coding block, and a first increase in the rate loss R is used to indicate the increase in rate loss R due to the signaling of the neural network update information for each block. When the one or more blocks include multiple input blocks, the neural network update information is signaled for multiple input images and shared by multiple input blocks, and a second increase in the rate loss R is used to indicate the increase in rate loss R due to the signaling of the neural network update information for each block. Because the neural network update information is shared by multiple input blocks, the second increase in the rate loss R may be less than the first increase in the rate loss R. Therefore, in some examples, it may be advantageous to fine-tune the NIC framework using multiple input blocks.

[0208] In one embodiment, the one or more pre-trained parameters to be updated are in one component of the pre-trained NIC framework (900). Thus, the one component of the pre-trained NIC framework (900) is updated based on the one or more replacement parameters, and other components of the pre-trained NIC framework (900) are not updated.

[0209] A component can be a pre-trained context model NN (916), a pre-trained entropy parameter NN (917), a pre-trained primary encoder network (911), a pre-trained primary decoder network (915), a pre-trained super encoder (921), or a pre-trained super decoder (925). Depending on which component in the pre-trained NIC framework (900) is updated, the pre-trained video encoder and / or the pre-trained video decoder can be updated.

[0210] In one example, one or more pre-trained parameters to be updated are in the pre-trained context model NN (916), and thus the pre-trained context model NN (916) is updated without updating the remaining components (911), (915), (921), (917), and (925). In one example, the pre-trained video encoder on the encoder side and the pre-trained video decoder on the decoder side include the pre-trained context model NN (916), and thus the pre-trained video encoder and the pre-trained video decoder are updated.

[0211] In one example, one or more pre-trained parameters to be updated are in the pre-trained super decoder (925), so the pre-trained super decoder (925) is updated without updating the remaining components (911), (915), (916), (917), and (921). Therefore, the pre-trained video encoder is not updated, but the pre-trained video decoder is updated.

[0212] In one embodiment, the one or more pre-trained parameters to be updated are in multiple components of the pre-trained NIC framework (900). Therefore, the multiple components of the pre-trained NIC framework (900) are updated based on the one or more replacement parameters. In one example, the multiple components of the pre-trained NIC framework (900) include all components configured with a neural network (e.g., DNN, CNN). In one example, the multiple components of the pre-trained NIC framework (900) include CNN-based components: a pre-trained primary encoder network (911), a pre-trained primary decoder network (915), a pre-trained context model NN (916), a pre-trained entropy parameter NN (917), a pre-trained super encoder (921), and a pre-trained super decoder (925).

[0213] As described above, in one example, the one or more pre-trained parameters to be updated are in a pre-trained video encoder of a pre-trained NIC framework (900). In one example, the one or more pre-trained parameters to be updated are in a pre-trained video decoder of a pre-trained NIC framework (900). In one example, the one or more pre-trained parameters to be updated are in both a pre-trained video encoder and a pre-trained video decoder of a pre-trained NIC framework (900).

[0214] The NIC framework (900) can be based on a neural network. For example, one or more components in the NIC framework (900) can include a neural network, such as a CNN, a DNN, etc. As described above, a neural network can be specified by different types of parameters, such as weights, biases, etc. Each neural network-based component in the NIC framework (900) (e.g., context model NN (916), entropy parameter NN (917), main encoder network (911), main decoder network (915), super encoder (921), or super decoder (925)) can be configured with appropriate parameters, such as corresponding weights, biases, or a combination of weights and biases. When using a CNN, the weights can include elements in a convolution kernel. One or more types of parameters can be used to specify a neural network. In one embodiment, the one or more pre-trained parameters to be updated are bias terms, and only the bias terms are replaced by one or more replacement parameters. In one embodiment, the one or more pre-trained parameters to be updated are weights, and only the weights are replaced by one or more replacement parameters. In one embodiment, the one or more pre-trained parameters to be updated include weights and bias terms, and all pre-trained parameters including weights and bias terms are replaced by one or more replacement parameters. In one embodiment, other parameters may be used to specify the neural network and may be fine-tuned.

[0215] The fine-tuning process may include multiple stages (e.g., iterations), wherein one or more pre-trained parameters are updated during the iterative fine-tuning process. The fine-tuning process may stop when the training loss is flat or about to be flat. In one example, the fine-tuning process stops when the training loss (e.g., RD ​​loss L) is below a first threshold. In one example, the fine-tuning process stops when the difference between two consecutive training losses is below a second threshold.

[0216] Two hyperparameters (e.g., step size and maximum number of steps) can be used in the fine-tuning process together with the loss function (e.g., RD ​​loss L). The maximum number of iterations can be used as a threshold for terminating the fine-tuning process. In one example, the fine-tuning process stops when the number of iterations reaches the maximum number of iterations.

[0217] The step size can indicate the learning rate of an online training process (e.g., an online fine-tuning process). The step size can be used for the gradient descent algorithm or backpropagation calculation performed during the fine-tuning process. Any suitable method can be used to determine the step size.

[0218] The step size of each block in the image may be different. In one embodiment, different step sizes may be assigned to images in order to achieve better compression results (eg, better RD loss L).

[0219] In some examples, a video encoder and video decoder based on a NIC framework (e.g., NIC framework (900)) can directly encode and decode images. Thus, the block-wise content-adaptive online training method can be adapted to update certain parameters in the NIC framework by directly using one or more images, thereby updating certain parameters in the video encoder and / or video decoder. Different images can have different step sizes to achieve optimized compression results.

[0220] In one embodiment, different step sizes are used for blocks with different types of content to achieve optimal results. Different types can refer to different differences. In one example, the step size is determined based on the difference of the blocks used to update the NIC framework. For example, the step size of a block with a high difference is larger than the step size of a block with a low difference, where a high difference is larger than a low difference.

[0221] In one embodiment, the step size is selected based on block or image characteristics, such as the RGB difference of the block. In one embodiment, the step size is selected based on the RD performance of the block (e.g., RD ​​loss L). Multiple sets of replacement parameters can be generated based on different step sizes, and the set with better compression performance (e.g., smaller RD loss) can be selected.

[0222] In one embodiment, a first step size can be used to run a certain number of iterations (e.g., 100). Then, a second step size (e.g., the first step size plus or minus the size increment) can be used to run a certain number of iterations. The results of the first and second step sizes can be compared to determine the step size to use. More than two step sizes can be tested to determine the optimal step size.

[0223] During the fine-tuning process, the step size can vary. The step size may have an initial value at the beginning of the fine-tuning process, and the initial value may be reduced (e.g., halved) at a later stage of the fine-tuning process (e.g., after a certain number of iterations) to achieve finer tuning. During iterative online training, the step size or learning rate may be changed by a scheduler. The scheduler may include a parameter adjustment method for adjusting the step size. The scheduler may determine the value of the step size so that the step size may increase, decrease, or remain constant over multiple intervals. In one example, the learning rate is changed by the scheduler in each step. A single scheduler or multiple different schedulers may be used for different blocks. Therefore, multiple sets of replacement parameters may be generated based on multiple schedulers, and a set of replacement parameters having better compression performance (e.g., smaller RD loss) may be selected.

[0224] In one embodiment, multiple learning rate schedules are assigned to different blocks to achieve better compression results. In one embodiment, all blocks in the image share the same learning rate schedule. In one embodiment, the learning rate schedule is selected based on block characteristics, such as RGB differences of the block. In one embodiment, the learning rate schedule is selected based on the RD performance of the block.

[0225] In one embodiment, different blocks can be used to update different parameters in different components in the NIC framework, such as the context model NN (916) or the super decoder (925). For example, the first block is used to update the parameters in the context model NN (916), and the second block is used to update the parameters in the super decoder (925).

[0226] In one embodiment, different blocks can be used to update different types of parameters (e.g., biases or weights) in the NIC framework. For example, a first block can be used to update at least one bias in one or more neural networks in the NIC framework, and a second block can be used to update at least one weight in one or more neural networks in the NIC framework.

[0227] In one embodiment, multiple blocks (eg, all blocks) in an image update the same parameter or parameters.

[0228] In one embodiment, one or more parameters to be updated are selected based on characteristics of the block, eg, RGB differences of the block. In one embodiment, one or more parameters to be updated are selected based on RD performance of the block.

[0229] At the end of the fine-tuning process, one or more updated parameters may be calculated for the corresponding one or more replacement parameters. In one embodiment, the one or more updated parameters are calculated as the difference between the one or more replacement parameters and the corresponding one or more pre-trained parameters. In one embodiment, the one or more updated parameters are respectively the one or more replacement parameters.

[0230] In one embodiment, one or more updated parameters can be generated from one or more replacement parameters, for example, using a specific linear or nonlinear transformation, and the one or more updated parameters are representative parameters generated based on the one or more replacement parameters. The one or more replacement parameters are converted into one or more updated parameters for better compression.

[0231] The first subset of the one or more update parameters corresponds to the first subset of the one or more replacement parameters, and the second subset of the one or more update parameters corresponds to the second subset of the one or more replacement parameters.

[0232] In one embodiment, different blocks have different relationships between the one or more updated parameters and the one or more replacement parameters. For example, for a first block, the one or more updated parameters are calculated as the difference between the one or more replacement parameters and the corresponding one or more pre-trained parameters. For a second block, the one or more updated parameters are respectively the one or more replacement parameters.

[0233] In one embodiment, multiple blocks (eg, all blocks) in an image have the same relationship between the one or more update parameters and the one or more replacement parameters.

[0234] In one embodiment, the relationship between the one or more update parameters and the one or more replacement parameters is selected based on characteristics of the block, such as RGB differences of the block. In one embodiment, the relationship between the one or more update parameters and the one or more replacement parameters is selected based on RD performance of the block.

[0235] In one example, the one or more updated parameters can be compressed, for example, using LZMA2, a bzip2 algorithm, or the like, where LZMA2 is a variant of the Lempel–Ziv–Markov chain algorithm (LZMA). In one example, compression is omitted for the one or more updated parameters. In some embodiments, the one or more updated parameters or a second subset of the one or more updated parameters can be encoded into the bitstream as neural network update information, where the neural network update information indicates the one or more replacement parameters or the second subset of the one or more replacement parameters.

[0236] In one embodiment, the compression method for one or more updated parameters is different for different blocks. For example, for a first block, LZMA2 is used to compress the one or more updated parameters, and for a second block, bzip2 is used to compress the one or more updated parameters. In one embodiment, the same compression method is used to compress the one or more updated parameters for multiple blocks (e.g., all blocks) in the image. In one embodiment, the compression method is selected based on block characteristics, such as RGB differences between blocks. In one embodiment, the compression method is selected based on RD performance of the blocks.

[0237] After the fine-tuning process, in some examples, a pre-trained video encoder on the encoder side can be updated or fine-tuned based on (i) a first subset of one or more replacement parameters or (ii) one or more replacement parameters. The updated video encoder can be used to encode an input image (e.g., one of the one or more blocks used for the fine-tuning process) into a bitstream. Thus, the bitstream includes the encoded block and the neural network update information.

[0238] If applicable, in one example, the neural network update information is decoded (e.g., decompressed) by the pre-trained video decoder to obtain one or more updated parameters or a second subset of the one or more updated parameters. In one example, the one or more replacement parameters or the second subset of the one or more replacement parameters can be obtained based on a relationship between the one or more updated parameters and the one or more replacement parameters. As described above, the pre-trained video decoder can be fine-tuned, and the decoded update video can be used to decode the encoded block.

[0239] The NIC framework can include any type of neural network and use any neural network-based image compression method, such as a context-superior encoder-decoder framework (e.g., the NIC framework shown in FIG. (9B)), a scale-superior encoder-decoder framework, a Gaussian mixture likelihood framework and variants of the Gaussian mixture likelihood framework, an RNN-based recursive compression method and variants of the RNN-based recursive compression method, etc.

[0240] Compared to related E2E image compression methods, the content-adaptive online training method and device disclosed herein have the following advantages: An adaptive online training mechanism is utilized to improve NIC encoding efficiency. A flexible and versatile framework can accommodate various types of pre-trained frameworks and quality metrics. For example, certain pre-trained parameters in various pre-trained frameworks can be replaced by using online training of the blocks to be encoded and transmitted.

[0241] Figure 19A flowchart outlining a process (1900) according to an embodiment of the present disclosure is shown. The process (1900) may be used to encode a block, e.g., a block in an original image or a block in a residual image. In various embodiments, the process (1900) is performed by a processing circuit, e.g., a processing circuit in a terminal device (310), (320), (330), and (340), a processing circuit that performs the functionality of a video encoder (1600A), a processing circuit that performs the functionality of a video encoder (1700). In one example, the processing circuit performs a combination of (i) one of the video encoders (403), (603), and (703) and (ii) the functionality of one of the video encoder (1600A) and the video encoder (1700). In some embodiments, the process (1900) is implemented in software instructions so that when the processing circuit executes the software instructions, the processing circuit performs the process (1900). The process begins at (S1901). In one example, the NIC framework is based on a neural network. In one example, the NIC framework is a reference Figure 9B The NIC framework (900) described herein can be based on CNN, for example, Figure 10-15 As described above, the video encoder (e.g., (1600A) or (1700)) and the corresponding video decoder (e.g., (1600B) or (1800)) can include multiple components in the NIC framework. The neural network-based NIC framework is pre-trained to pre-train the video encoder and video decoder. The process (1900) proceeds to (S1910).

[0242] At (S1910), a fine-tuning process is performed on the NIC framework based on one or more blocks (or input blocks). The input blocks can be any suitable blocks of any suitable size. In some examples, the input blocks include blocks in the original image, natural image, computer-generated image, etc. in the spatial domain.

[0243] In some examples, the input block includes residual data in the spatial domain, for example, calculated by a residual calculator (e.g., residual calculator (723)). The components in the various devices can be appropriately combined, for example, to refer to Figure 7 and Figure 9B , implementing (S1910) the residual block from the residual calculator can be fed into the main encoder network (911) in the NIC framework.

[0244] As described above, one or more parameters (e.g., one or more pretrained parameters) in one or more neural networks (e.g., one or more pretrained neural networks) in a NIC framework (e.g., a pretrained NIC framework) can be updated to one or more replacement parameters, respectively. In one embodiment, during the training process described in (S1910), for example, in each step, one or more parameters in the one or more neural networks are updated.

[0245] In one embodiment, at least one neural network in a video encoder (e.g., a pre-trained video encoder) is configured with a first subset of one or more pre-trained parameters, such that the at least one neural network in the video encoder can be updated based on a corresponding first subset of one or more replacement parameters. In one example, the first subset of one or more replacement parameters includes all of the one or more replacement parameters. In one example, the at least one neural network in the video encoder is updated when the first subset of one or more pre-trained parameters is respectively replaced by the first subset of one or more replacement parameters. In one example, the at least one neural network in the video encoder is iteratively updated during a fine-tuning process. In one example, none of the one or more pre-trained parameters are included in the video encoder, such that the video encoder is not updated and the pre-trained video encoder remains.

[0246] At (S1920), one of the one or more blocks may be encoded using a video encoder having the at least one updated neural network, wherein the video encoder is configured with a first subset of the one or more replacement parameters. In one example, after updating the at least one neural network in the video encoder, one of the one or more blocks is encoded.

[0247] Step (S1920) may be modified appropriately. For example, when one or more replacement parameters are not included in at least one neural network in the video encoder, the video encoder is not updated, and thus a pre-trained video encoder (e.g., a video encoder including at least one pre-trained neural network) may be used to encode one of the one or more blocks.

[0248] At (S1930), neural network update information indicating a second subset of the one or more replacement parameters may be encoded into the bitstream. In one example, the second subset of the one or more replacement parameters is used to update at least one neural network in a video decoder on the decoder side. Step (S1930) may be omitted, and for example, if the second subset of the one or more replacement parameters does not include parameters and the neural network update information is not signaled in the bitstream, then the neural network in the video decoder is not updated.

[0249] At (S1940), a bitstream including the coded blocks in the one or more blocks and the neural network update information may be transmitted. Step (S1940) may be modified as appropriate. For example, if step (S1930) is omitted, the bitstream does not include the neural network update information. The process (1900) proceeds to (S1999) and terminates.

[0250] The process (1900) can be adapted to various circumstances, and the steps in the process (1900) can be adjusted accordingly. One or more steps in the process (1900) can be modified, omitted, repeated, and / or combined. The process (1900) can be implemented in any suitable order. Additional steps can be added. For example, in addition to encoding one of the one or more blocks, the one or more blocks can also be encoded in (S1920) and transmitted in (S1940).

[0251] In some examples of the process (1900), one of the one or more blocks is encoded by the updated video encoder and transmitted in a bitstream. Because the fine-tuning process is based on the one or more blocks, the fine-tuning process is based on the context to be encoded and is therefore context-based.

[0252] In some examples, the neural network update information further indicates what parameters the second subset of one or more pre-trained parameters (or the corresponding second subset of one or more replacement parameters) are, so that the corresponding pre-trained parameters in the video decoder can be updated. The neural network update information can indicate component information (e.g., (915)), layer information (e.g., the fourth layer DeConv: 5x5 c3 s2), channel information (e.g., the second channel), etc. of the second subset of one or more pre-trained parameters. Therefore, referring to Figure 11 , the second subset of one or more replacement parameters includes the convolution kernel (915) of the second channel DeConv: 5x5 c3 s2 in the main decoder network. Therefore, the convolution kernel of the second channel DeConv: 5x5 c3 s2 in the pre-trained main decoder network (915) is updated. In some examples, component information (e.g., (915)), layer information (e.g., fourth layer DeConv: 5x5 c3 s2), channel information (e.g., second channel), etc. of the second subset of one or more pre-trained parameters are predetermined and stored in the pre-trained video decoder and are therefore not signaled.

[0253] Figure 20A flowchart outlining a process (2000) according to an embodiment of the present disclosure is shown. The process (2000) can be used for reconstruction of a coding block. In various embodiments, the process (2000) is performed by a processing circuit, for example, a processing circuit in a terminal device (310), (320), (330) and (340), a processing circuit that performs the functions of a video decoder (1600B), a processing circuit that performs the functions of a video decoder (1800). In one example, the processing circuit performs a combination of the functions of (i) one of a video decoder (410), a video decoder (510) and a video decoder (810) and (ii) one of a video decoder (1600B) or a video decoder (1800). In some embodiments, the process (2000) is implemented in software instructions so that when the processing circuit executes the software instructions, the processing circuit performs the process (2000). The process starts at (S2001). In one example, the NIC framework is based on a neural network. In one example, the NIC framework is a reference Figure 9B The NIC framework (900) described herein can be based on CNN, for example, Figure 10-15 As described above, the video decoder (e.g., (1600B) or (1800)) can include multiple components in the NIC framework. The neural network-based NIC framework can be pre-trained. The video decoder can be pre-trained using pre-trained parameters. The process (2000) proceeds to (S2010).

[0254] At (S2010), first neural network update information in the encoded bitstream may be decoded. The first neural network update information may be used for a first neural network in a video decoder. The first neural network may be configured with a first set of pre-trained parameters. The first neural network update information may correspond to a first block in an image to be reconstructed and indicate a first replacement parameter corresponding to a first pre-trained parameter in the first set of pre-trained parameters.

[0255] In one example, the first pre-training parameter is a pre-training bias term.

[0256] In one example, the first pre-trained parameter is a pre-trained weight coefficient.

[0257] In one embodiment, the video decoder includes multiple neural networks. The first neural network update information may indicate update information for one or more remaining neural networks in the multiple neural networks. For example, the first neural network update information further indicates one or more replacement parameters for one or more remaining neural networks in the multiple neural networks. The one or more replacement parameters correspond to one or more corresponding pre-trained parameters of the one or more remaining neural networks. In one example, each of the first pre-trained parameter and the one or more pre-trained parameters is a corresponding pre-trained bias term. In one example, each of the first pre-trained parameter and the one or more pre-trained parameters is a corresponding pre-trained weight coefficient. In one example, the first pre-trained parameter and the one or more pre-trained parameters include one or more pre-trained bias terms and one or more pre-trained weight coefficients in the multiple neural networks.

[0258] In one example, the first neural network update information indicates update information for a subset of the plurality of neural networks, and the remaining subset of the plurality of neural networks is not updated.

[0259] In one example, the video decoder is Figure 18 The video decoder (1800) shown in FIG. 1 is a first neural network and is the main decoder network (915).

[0260] In one example, the video decoder is Figure 16B The video decoder (1600B) shown in FIG. A plurality of neural networks in the video decoder include a main decoder network (915), a context model NN (916), an entropy parameter NN (917), and a super decoder (925). The first neural network is one of the main decoder network (915), the context model NN (916), the entropy parameter NN (917), and the super decoder (925), for example, the context model NN (916). In one example, the first neural network update information also includes one or more replacement parameters of one or more remaining neural networks in the video decoder (e.g., the main decoder network (915), the entropy parameter NN (917), and / or the super decoder (925)).

[0261] At (S2020), first replacement parameters may be determined based on the first neural network update information. In one embodiment, the updated parameters are obtained from the first neural network update information. In one example, the updated parameters may be obtained from the first neural network update information by decompression (e.g., using an LZMA2 or bzip2 algorithm).

[0262] In one example, the first neural network update information indicates that the updated parameter is a difference between the first replacement parameter and the first pre-trained parameter. The first replacement parameter can be calculated based on the sum of the updated parameter and the first pre-trained parameter.

[0263] In one embodiment, the first replacement parameter is determined as the updated parameter.

[0264] In one embodiment, the updated parameters are representative parameters generated based on the first replacement parameters at the encoder side (eg, using a linear or non-linear transformation), and the first replacement parameters are obtained based on the representative parameters.

[0265] At (S2030), the first neural network in the video decoder may be updated (or fine-tuned) based on the first replacement parameters, for example, by replacing the first pre-trained parameters with the first replacement parameters in the first neural network. If the video decoder includes multiple neural networks, and the first neural network update information indicates update information (e.g., additional replacement parameters) for the multiple neural networks, the multiple neural networks may be updated. For example, the first neural network update information may also include one or more replacement parameters for one or more remaining neural networks in the video decoder, and the one or more remaining neural networks may be updated based on the one or more replacement parameters.

[0266] At (S2040), the encoded first block in the bitstream can be decoded by, for example, an updated video decoder based on the updated first neural network. The output block generated at (S2040) can be any suitable block of any suitable size. In some examples, the output block is a reconstructed block in a reconstructed image in the spatial domain.

[0267] In some examples, the output block of the video decoder includes residual data in the spatial domain, and further processing can therefore be used to generate a reconstructed block based on the output block. For example, the reconstruction module (874) is configured to combine the residual data and the prediction result (output by the inter-frame or intra-frame prediction module) in the spatial domain to form a reconstructed block that can be part of the reconstructed image. Additional appropriate operations, such as deblocking operations, can be performed to improve visual quality. The components in the various devices can be appropriately combined, for example, with reference to Figure 8 and 9, implementing (S2040) the residual data and corresponding prediction results from the main decoder network (915) in the video decoder are fed into the reconstruction module (874) to generate a reconstructed image.

[0268] In one example, the bitstream also includes one or more coded bits for determining a context model for decoding a coded block. The video decoder may include a main decoder network (e.g., (911)), a context model network (e.g., (916)), an entropy parameter network (e.g., (917)), and a super decoder network (e.g., (925)). The neural network is one of the main decoder network, the context model network, the entropy parameter NN, and the super decoder network. The super decoder network may be used to decode the one or more coded bits. The context model network and the entropy parameter network may be used to determine an entropy model (e.g., a context model) based on the decoded bits and quantized potential bits of the coded block available to the context model network. The coded block may be decoded using the main decoder network and the entropy model.

[0269] The process (2000) proceeds to (S2099) and terminates.

[0270] The process (2000) can be adapted to various circumstances, and the steps in the process (2000) can be adjusted accordingly. One or more steps in the process (2000) can be modified, omitted, repeated, and / or combined. The process (2000) can be implemented in any suitable order. Additional steps can be added.

[0271] In one example, at (S2040), another block in the encoded bitstream is decoded based on the updated first neural network of the first block.

[0272] In one example, at (S2010), second neural network update information in a coded bitstream of a second neural network in a video decoder is decoded. The second neural network is configured with a second set of pre-trained parameters. The second neural network update information corresponds to a second block in an image to be reconstructed and indicates a second replacement parameter corresponding to a second pre-trained parameter in the second set of pre-trained parameters. The second neural network (e.g., context model NN (916)) may be different from the first neural network (e.g., main decoder network (915)). At (S2030), the second neural network in the video decoder may be updated based on the second replacement parameter. At (S2040), the second block may be decoded based on the updated second neural network of the second block. In one example, the first pre-trained parameter is one of a pre-trained weight coefficient and a pre-trained bias term. In one example, the second pre-trained parameter is the other of the pre-trained weight coefficient and the pre-trained bias term.

[0273] The embodiments of the present disclosure may be used alone or in any combination. In addition, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer-readable medium.

[0274] The present disclosure does not impose any restrictions on the methods used for encoders (e.g., neural network-based encoders) and decoders (e.g., neural network-based decoders). The neural networks used in the encoders, decoders, etc. can be any suitable type of neural network, such as DNN, CNN, etc.

[0275] Therefore, the content-adaptive online training method disclosed in the present invention can adapt to different types of NIC frameworks, for example, different types of encoding DNNs, decoding DNNs, encoding CNNs, decoding CNNs, etc.

[0276] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 21 A computer system (2100) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0277] Computer software may be encoded using any suitable machine code or computer language, which may be subjected to assembly, compilation, linking, or similar mechanisms to create code comprising instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through interpretation, microcode execution, etc.

[0278] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, and the like.

[0279] Figure 21 The components of the computer system (2100) shown in the example are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as constituting any dependency or requirement on any one component or combination of components shown in the exemplary embodiment of the computer system (2100).

[0280] The computer system (2100) may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0281] Input human-machine interface devices may include one or more of the following (only one of each is depicted): keyboard (2101), mouse (2102), trackpad (2103), touch screen (2110), data gloves (not shown), joystick (2105), microphone (2106), scanner (2107), camera (2108).

[0282] The computer system (2100) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (2110), a data glove (not shown), or a joystick (2105), although there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speakers (2109), headphones (not shown)), visual output devices (e.g., screens (2110), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities—some of which are capable of outputting two-dimensional visual output or more than three-dimensional output through means such as stereo output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

[0283] The computer system (2100) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (2120) with CD / DVD or similar media (2121), thumb drives (2122), removable hard drives or solid-state drives (2123), traditional magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD devices such as security dongles (not shown), etc.

[0284] Those skilled in the art will also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0285] The computer system (2100) may also include an interface (2154) to one or more communication networks (2155). The network may be, for example, wireless, wired, or optical. The network may also be local, wide, metropolitan, vehicular, industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like, television wired or wireless wide area digital networks including cable, satellite, and terrestrial broadcast television, vehicle and industrial networks including CANBus, and the like. Some networks typically require an external network interface adapter connected to some common data port or peripheral bus (2149) (e.g., a USB port of the computer system (2100)); others are typically integrated into the core of the computer system (2100) by connecting to a system bus as described below (e.g., an Ethernet interface in a PC computer system or a cellular network interface in a smartphone computer system). Using any of these networks, the computer system (2100) can communicate with other entities. Such communication can be one-way, receive-only (e.g., broadcast television), one-way, send-only (e.g., CANbus to certain CANbus devices), or two-way, for example, to other computer systems using local or wide area digital networks. As described above, certain protocols and protocol stacks can be used on each of these networks and network interfaces.

[0286] The aforementioned human interface device, human-accessible storage device, and network interface may be attached to the core (2140) of the computer system (2100).

[0287] The core (2140) may include one or more central processing units (CPUs) (2141), graphics processing units (GPUs) (2142), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (2143), hardware accelerators (2144) for specific tasks, a graphics adapter (2150), and the like. These devices, along with read-only memory (ROM) (2145), random access memory (2146), and internal mass storage (2147) such as an internal non-user accessible hard drive or solid-state drive (SSD), may be connected via a system bus (2148). In some computer systems, the system bus (2148) may be accessible in the form of one or more physical plugs to allow expansion of additional CPUs, GPUs, and the like. Peripheral devices may be connected to the core's system bus (2148) directly or via a peripheral bus (2149). In one example, a screen (2110) may be connected to a graphics adapter (2150). Peripheral bus architectures include PCI, USB, and the like.

[0288] The CPU (2141), GPU (2142), FPGA (2143), and accelerator (2144) can execute certain instructions, which, when combined, can constitute the aforementioned computer code. The computer code can be stored in ROM (2145) or RAM (2146). Transient data can also be stored in RAM (2146), while permanent data can be stored, for example, in internal mass storage (2147). Fast storage and retrieval of any storage device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (2141), GPUs (2142), mass storage (2147), ROM (2145), RAM (2146), etc.

[0289] The computer readable medium may have computer code for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of this disclosure, or may be of a type well known and available to those skilled in the art of computer software.

[0290] As an example and not limitation, a computer system having architecture (2100), in particular core (2140), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with a user-accessible mass storage as described above and certain memories of the core (2140) that are non-transitory, for example, a core internal mass storage (2147) or ROM (2145). Software that implements various embodiments of the present disclosure can be stored in such a device and executed by the core (2140). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the core (2140) and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (2146) and modifying such data structures according to software-defined processes. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise contained in circuitry (e.g., accelerator (2144)), which may operate in place of or in conjunction with software to perform specific processes or specific portions of specific processes described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuitry (e.g., an integrated circuit (IC)) storing software for execution, circuitry containing logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0291] Appendix A: Abbreviations

[0292] JEM: joint exploration model

[0293] VVC: versatile video coding

[0294] BMS: benchmark set

[0295] MV: Motion Vector

[0296] HEVC: High Efficiency Video Coding

[0297] SEI: Supplementary Enhancement Information, Supplementary Enhancement Information

[0298] VUI: Video Usability Information, video availability information

[0299] GOP: Groups of Pictures, picture group

[0300] TU: Transform Units, Transformation Unit

[0301] PU: Prediction Units, prediction unit

[0302] CTU: Coding Tree Units, coding tree unit

[0303] CTB: Coding Tree Blocks, coding tree blocks

[0304] PB: Prediction Blocks

[0305] HRD: Hypothetical Reference Decoder, Hypothetical Reference Decoder

[0306] SNR: Signal Noise Ratio

[0307] CPU: Central Processing Units, central processing unit

[0308] GPU: Graphics Processing Units, Graphics Processing Unit

[0309] CRT: Cathode Ray Tube

[0310] LCD: Liquid-Crystal Display

[0311] OLED: Organic Light-Emitting Diode

[0312] CD: Compact Disc

[0313] DVD: Digital Video Disc

[0314] ROM: Read-Only Memory

[0315] RAM: Random Access Memory

[0316] ASIC: Application-Specific Integrated Circuit

[0317] PLD: Programmable Logic Device

[0318] LAN: Local Area Network

[0319] GSM: Global System for Mobile communications

[0320] LTE: Long-Term Evolution

[0321] CANBus: Controller Area Network Bus

[0322] USB: Universal Serial Bus

[0323] PCI: Peripheral Component Interconnect

[0324] FPGA: Field Programmable Gate Areas, field programmable gate area

[0325] SSD: solid-state drive

[0326] IC: Integrated Circuit

[0327] CU: Coding Unit

[0328] NIC: Neural Image Compression

[0329] RD: Rate-Distortion

[0330] E2E: End to End

[0331] ANN: Artificial Neural Network

[0332] DNN: Deep Neural Network

[0333] CNN: Convolution Neural Network

[0334] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various substitute equivalents that fall within the scope of the present disclosure. Therefore, it should be understood that those skilled in the art will be able to devise many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. A video decoding method, characterized in that: include: decoding first neural network update information in a coded bitstream, the first neural network update information being used for a first neural network, the first neural network being configured with a first set of pre-trained parameters, the first neural network update information corresponding to a first block in an image to be reconstructed and indicating a first replacement parameter corresponding to a first pre-trained parameter in the first set of pre-trained parameters, the first neural network update information further indicating one or more replacement parameters, the one or more replacement parameters being used for one or more remaining neural networks in a plurality of neural networks other than the first neural network; updating the first neural network based on the first replacement parameter, the updating further comprising updating the one or more remaining neural networks based on the one or more replacement parameters; as well as The first block is decoded based on the updated first neural network, the updated first neural network being used for the first block.

2. The method according to claim 1, characterized in that Also includes: decoding second neural network update information in the coded bitstream, the second neural network update information being for a second neural network, the second neural network being configured with a second set of pre-trained parameters, the second neural network update information corresponding to a second block in the image to be reconstructed and indicating a second replacement parameter corresponding to a second pre-trained parameter in the second set of pre-trained parameters, the second neural network being different from the first neural network; updating the second neural network based on the second replacement parameters; as well as A second block is decoded based on the updated second neural network, wherein the updated second neural network is used for the second block.

3. The method according to claim 2, wherein: The first pre-training parameter is one of a pre-training weight coefficient and a pre-training bias item.

4. The method according to claim 3, characterized in that The second pre-training parameter is the other of the pre-training weight coefficient and the pre-training bias term.

5. The method according to claim 1, wherein Also includes: Based on the updated first neural network, a second block in the encoded bit stream is decoded.

6. The method according to claim 1, characterized in that The first neural network update information indicates a difference between the first replacement parameters and the first pre-trained parameters, and The method further includes determining the first replacement parameter based on the difference and the sum of the first pre-trained parameter.

7. The method according to claim 2, characterized in that Decoding the first neural network update information includes decoding the first neural network update information based on one of a variant of the Lempel-Ziv-Markov chain algorithm, LZMA2, and a bzip2 algorithm.

8. The method according to claim 7, characterized in that The decoding of the second neural network update information includes: decoding the second neural network update information based on the other of the LZMA2 algorithm and the bzip2 algorithm.

9. A video decoding device, characterized in that: include a processing circuit, the processing circuit being configured to: decoding first neural network update information in a coded bitstream, the first neural network update information being used for a first neural network, the first neural network being configured with a first set of pre-trained parameters, the first neural network update information corresponding to a first block in an image to be reconstructed and indicating a first replacement parameter corresponding to a first pre-trained parameter in the first set of pre-trained parameters, the first neural network update information further indicating one or more replacement parameters, the one or more replacement parameters being used for one or more remaining neural networks in a plurality of neural networks other than the first neural network; updating the first neural network based on the first replacement parameter, the updating further comprising updating the one or more remaining neural networks based on the one or more replacement parameters; as well as The first block is decoded based on the updated first neural network, the updated first neural network being used for the first block.

10. The device according to claim 9, characterized in that The processing circuit is configured to: decoding second neural network update information in the coded bitstream, the second neural network update information being for a second neural network, the second neural network being configured with a second set of pre-trained parameters, the second neural network update information corresponding to a second block in the image to be reconstructed and indicating a second replacement parameter corresponding to a second pre-trained parameter in the second set of pre-trained parameters, the second neural network being different from the first neural network; updating the second neural network based on the second replacement parameters; as well as A second block is decoded based on the updated second neural network, wherein the updated second neural network is used for the second block.

11. The device according to claim 10, characterized in that The first pre-training parameter is one of a pre-training weight coefficient and a pre-training bias item.

12. The device according to claim 11, characterized in that The second pre-training parameter is the other of the pre-training weight coefficient and the pre-training bias term.

13. The device according to claim 9, characterized in that The processing circuit is configured to decode a second block in the encoded bitstream based on the updated first neural network.

14. The device according to claim 9, characterized in that The first neural network update information indicates a difference between the first replacement parameters and the first pre-trained parameters, and The processing circuit is configured to determine the first replacement parameter based on a sum of the difference and the first pre-trained parameter.

15. The device according to claim 10, characterized in that The processing circuit is configured to: Decoding the first neural network update information includes decoding the first neural network update information based on one of a variant of the Lempel–Ziv–Markov chain algorithm, LZMA2, and a bzip2 algorithm.

16. The device according to claim 15, characterized in that The processing circuit is configured to: The second neural network update information is decoded based on the other of the LZMA2 and the bzip2 algorithms.

17. A non-transitory computer-readable storage medium, characterized in that The medium stores a program executable by at least one processor to perform the method according to any one of claims 1 to 8.

Images