Method, system and computer program executed by at least one processor of a video encoder and method executed by at least one processor of a video decoder - Patents.com

Abstract

A system and method are provided for block-wise entropy coding in neural image compression, including receiving a bitstream containing an image, partitioning the image into a plurality of blocks, compressing each of the plurality of blocks with a neural network-based encoder, obtaining block-wise compressed features from among the plurality of blocks in the image to obtain the compressed features, processing the compressed features with an entropy encoder to generate a first compressed bitstream, concatenating the compressed features to obtain a plurality of reshaped compressed features, processing the plurality of reshaped compressed features with an entropy encoder to generate a second compressed bitstream, and encoding the bitstream containing the image based on the second compressed bitstream.

Description

[Technical field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Patent Application No. 63 / 212,469, filed June 18, 2012, and U.S. Patent Application No. 17 / 825,575, filed May 26, 2022. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] [Field] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure are directed to systems and methods for block-wise entropy coding in neural image compression. [Background technology]

[0003] In a deep neural network-based video or image coding framework, by utilizing a large amount of training data set and advanced machine learning algorithms, various modules can be jointly optimized from input to output to improve the final objective (e.g., minimizing rate-distortion loss). In contrast, traditional hybrid video codecs usually focus on optimizing terminal modules, which may not bring about coding improvement in overall performance. Therefore, an end-to-end (E2E) optimized neural image compression (NIC) is designed to be optimized as a whole to achieve better coding performance.

[0004] In recent years, ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) has been actively exploring the potential needs for future video codec technology standards. ISO / IEC JPEG has established the JPEG-AI group, which focuses on AI-based end-to-end neural image compression. China AVS Standards has also formed an AVS-AI special group to address neural image and video compression technology. At the same time, many technology companies have also launched special research projects for NIC. Due to the success of recent approaches, there has been an increasing industrial interest in advanced neural image and video compression methodologies.

[0005] The general framework of NIC is described as the following steps: Given an input image x, the goal of NIC is to compute a compressed representation that is compact for storage and transmission using the image as an input to a deep neural network (DNN) encoder. The compressed representation is then quantized for arithmetic coding to generate a bitstream. The bitstream is returned to arithmetic decoding, which is used as an input to a DNN decoder to reconstruct the image. Additionally, hyperpriors and context models can be used to further improve the compression performance.

[0006] During model training, the rate-distortion (RD) loss is calculated based on the number of iterations of the reconstructed image using a trade-off hyperparameter λ, as shown in Equation 1 below. (outside 1) JPEG0007666783000001.jpg12125, (outside 2) JPEG0007666783000002.jpg11125 is optimized to achieve a trade-off between bit consumption R:

number

[0007] The disclosed embodiments provide a block-wise entropy coding method that optimizes the rate-distortion performance of image compression in an end-to-end (E2E) framework. The E2E optimized framework can be a DNN-based image or video coding framework that is fully pre-trained for both the encoder and the decoder.

[0008] According to an embodiment of the present disclosure, a method is provided for execution by at least one processor of a video encoder, the method comprising: receiving a bitstream containing an image; Partitioning the image into a number of blocks; compressing each of the plurality of blocks by a neural network based encoder; obtaining compressed features by obtaining block-wise compressed features from among the plurality of blocks in the image; processing the compressed features by an entropy encoder to generate a first compressed bitstream; obtaining a plurality of reshaped compressed features by concatenating the compressed features; processing the reshaped compressed features by the entropy encoder to generate a second compressed bitstream; encoding the bitstream containing the image based on the second compressed bitstream; Includes.

[0009] According to an embodiment of the present disclosure, there is provided a system including at least one memory configured to store computer program code, and at least one processor configured to access the computer program code and operate as directed by the computer program code, the computer program code causing the at least one processor to: Partitioning an image from a bitstream into a number of blocks; compressing each of the plurality of blocks by a neural network based encoder; obtaining compressed features by obtaining block-wise compressed features from among the plurality of blocks in the image; processing the compressed features by an entropy encoder to generate a first compressed bitstream; obtaining a plurality of reshaped compressed features by concatenating the compressed features; processing the reshaped compressed features by the entropy encoder to generate a second compressed bitstream; encoding the bitstream containing the image based on the second compressed bitstream; The method includes encoding code configured to cause the

[0010] According to an embodiment of the present disclosure, when executed by at least one processor, the at least one processor is receiving a bitstream containing an image; Partitioning the image into a number of blocks; compressing each of the plurality of blocks with a neural network based encoder; obtaining compressed features by obtaining block-wise compressed features from among the plurality of blocks in the image; processing the compressed features by an entropy encoder to generate a first compressed bitstream; obtaining a plurality of reshaped compressed features by concatenating the compressed features; processing the reshaped compressed features by the entropy encoder to generate a second compressed bitstream; encoding the bitstream containing the image based on the second compressed bitstream; A non-transitory computer-readable medium storing computer code configured to implement a video encoder is provided.

[0011] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic block diagram of a communication system according to an embodiment. [Diagram 2] FIG. 1 is a schematic block diagram of a communication system according to an embodiment. [Diagram 3] FIG. 2 is a schematic block diagram of a decoder according to an embodiment; [Figure 4] FIG. 2 is a schematic block diagram of an encoder according to an embodiment. [Diagram 5] 1 illustrates a NIC encoding method according to an embodiment of the present disclosure that may be performed by an encoder. [Figure 6] 6 illustrates a NIC decoding method 600 according to an embodiment of the present disclosure that may be performed by a decoder. [Figure 7] FIG. 2 is a block diagram of computer code according to an embodiment. [Figure 8] FIG. 2 is a block diagram of computer code according to an embodiment. [Figure 9] FIG. 1 illustrates a computer system suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] 1 illustrates a simplified block diagram of a communication system 100 according to an embodiment of the present disclosure. The communication system 100 may include at least two terminals 110, 120 interconnected via a network 150. For a one-way transmission of data, the first terminal 110 may code video data at a local location for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive the coded video data of the other terminal from the network 150, decode the coded data, and display the recovered video data. One-way data transmission may be common in media serving applications, etc.

[0014] 1 depicts a second pair of terminals 130, 140 equipped to support bidirectional transmission of coded video, such as may occur during a video conference. For bidirectional transmission of data, each terminal device 130, 140 may code video data captured at a local location for transmission to the other terminal over network 150. Each terminal may also receive coded video data transmitted by the other terminal, decode the coded video data, and display the recovered video data on a local display device.

[0015] In FIG. 1, terminals 110-140 may be represented as servers, personal computers, and smartphones, and / or any other type of terminal. For example, terminals 110-140 may be laptop computers, tablet computers, media players, and / or dedicated video conferencing devices. Network 150 represents any number of networks that convey coded video data between terminals 110-140, including, for example, wireline and / or wireless communication networks. Communication network 150 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network 150 is not important to the operation of the present disclosure unless otherwise described hereinafter.

[0016] 2 depicts the arrangement of a video encoder and a video decoder in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter can be similarly applicable to other video-enabled applications including, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0017] As depicted in FIG. 2, the streaming system 200 may include a capture subsystem 213 that may include a video source 201 and an encoder 203. The video source 201 may be, for example, a digital camera and may be configured to generate an uncompressed video sample stream 202. The uncompressed video sample stream 202 may provide a high data volume when compared to an encoded video bitstream and may be processed by an encoder 203 coupled to the video source 201. The encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data 204 may include a lower data volume when compared to the sample stream and may be stored in a streaming server 205 for future use. One or more streaming clients 206 may access the streaming server 205 to retrieve a video bitstream 209, which may be a copy of the encoded video bitstream 204.

[0018] In an embodiment, the streaming server 205 may also function as a Media-Aware Network Element (MANE). For example, the streaming server 205 may be configured to prune the encoded video bitstreams 204 in order to tailor potentially different bitstreams to one or more streaming clients 206. In an embodiment, a MANE may be provided separately from the streaming server 205 in the streaming system 200.

[0019] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 may, for example, decode a video bitstream 209, which is an incoming copy of the encoded video bitstream 204, and generate an outgoing video sample stream 211 that may be rendered on a display 212 or other rendering device (not shown). In some streaming systems, the video bitstreams 204, 209 may be encoded according to a particular video coding / compression standard. An example of such a standard includes, but is not limited to, ITU-T Recommendation H.265. The developing video coding standard is colloquially known as Versatile Video Coding (VVC). Embodiments of the present disclosure may be used in conjunction with VVC.

[0020] FIG. 3 illustrates an example functional block diagram of a video decoder 210 associated with a display 212 according to an embodiment of the present disclosure.

[0021] The video decoder 210 may include a channel 312, a receiver 310, a buffer memory 315, an entropy decoder / parser 320, a scaler / inverse transform unit 351, an intra-picture prediction unit 352, a motion compensated prediction unit 353, an aggregator 355, a loop filter unit 356, a reference picture memory 357, and a current picture memory. In at least one embodiment, the video decoder 210 may include an integrated circuit, a series of integrated circuits, and / or other electronic circuitry. The video decoder 210 may also be embodied partially or wholly in software executing on one or more CPUs with associated memory.

[0022] In this and other embodiments, the receiver 310 may receive one or more coded video sequences to be decoded by the video decoder 210, one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from a channel 312, which may be a hardware / software link to a storage device storing the coded video data. The receiver 310 may receive the coded video data together with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective use entities (not shown). The receiver 310 may separate the coded video sequences from the other data. To combat network jitter, a buffer memory 315 may be coupled between the receiver 310 and the entropy decoder / parser 320 (hereafter "parser"). When the receiver 310 is receiving data from a store / forward device of sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory 315 may not be used or may be small. For use with best-effort packet networks such as the Internet, the buffer memory 315 may be required and can be relatively large and of adaptive size.

[0023] The video decoder 210 may include a parser 320 for reconstructing symbols 321 from the entropy coded video sequence. The categories of symbols include information used to manage the operation of the decoder 210 and potentially information for controlling a rendering device, such as a display 212, which may be coupled to the decoder as shown in FIG. 2. The control information for the rendering device may take the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser 320 may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, context-dependent or non-context-dependent arithmetic coding, etc. The parser 320 may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. Parser 320 may also extract information from the coded video sequence information, such as transform coefficients, quantization parameter values, motion vectors, etc.

[0024] Parser 320 may perform an entropy decoding / parsing operation on the video sequence received from buffer memory 315 to generate symbols 321.

[0025] The reconstruction of symbols 321 may have many different units depending on the type of coded video picture or portion thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. Which units are included and how they are included may be controlled by subgroup control information parsed by parser 320 from the coded video sequence. The flow of such subgroup control information between parser 320 and the following units is not shown for clarity.

[0026] Beyond the functional blocks already mentioned, the decoder 210 may be conceptually subdivided into a number of functional units, which are described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is adequate.

[0027] One unit may be a scalar / inverse transform unit 351, which may receive quantized transform coefficients as symbols 321 from the parser 320 along with control information including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The scalar / inverse transform unit 351 may output a block containing sample values ​​that may be input to an aggregator 355.

[0028] In some cases, the output samples of the scaler / inverse transformer 351 may relate to intra-coded blocks, i.e. blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed part of the current picture. Such prediction information may be provided by the intra-picture prediction unit 3552. In some cases, the intra-picture prediction unit 352 generates a block of the same size and shape as the block being reconstructed, using surrounding already reconstructed information fetched from the current (partially reconstructed) picture from the current picture memory 358. The aggregator 355 adds, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit 352 to the output sample information provided by the scaler / inverse transformer 351.

[0029] In other cases, the output samples of the scalar / inverse transform unit 351 may relate to an inter-coded and potentially motion-compensated block. In such cases, the motion compensated prediction unit 353 may access the reference picture memory 357 to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols 321 related to the block, the samples may be added by the aggregator 355 to the output of the scalar / inverse transform unit 351 (in this case, called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory 357 from which the motion compensated prediction unit 353 fetches the prediction samples may be controlled by a motion vector. The motion vector may be available to the motion compensated prediction unit 353 in the form of the symbols 321, which may have, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory 357 when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0030] The output samples of the aggregator 355 may be subjected to various loop filtering techniques in the loop filter unit 356. Video compression techniques may include in-loop filter techniques that are controlled by parameters contained in the coded video bitstream and made available to the loop filter unit 356 as symbols 321 from the parser 320, but may also be responsive to meta-information obtained during the decoding of previous parts (in decoding order) of the coded picture or coded video sequence, or even to previously constructed loop filtered sample values.

[0031] The output of the loop filter unit 356 may be a sample stream that may be output to a render device, such as the display 212, or further stored in a reference picture memory 357 for use in future inter-picture prediction.

[0032] Once a particular coded picture is fully reconstructed, it may be used as a reference picture for future prediction. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 320), the current reference picture may become part of reference picture memory 357, and any unused current picture memory may be reallocated before beginning reconstruction of a subsequent coded picture.

[0033] The video decoder 210 may perform decoding operations according to a given video compression technique, which may be documented in a standard specification, such as ITU-T Recommendation H.265. The coded video sequence may conform to a syntax prescribed by the video compression technique or standard in use, in the sense that it conforms to the video compression technique or standard as specified in the video compression technique document or standard, and in particular in a profile document therein. Also, for compliance with some video compression techniques or standards, the complexity of the coded video sequence may be within a bound defined by a level of the video compression technique or standard. In some cases, the level limits a maximum picture size, a maximum frame rate, a maximum reconstruction sample rate (e.g., measured in megasamples / second), a maximum reference picture size, etc. The limits set by the level can be further limited in some cases through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0034] In an embodiment, the receiver 310 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder 210 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0035] FIG. 4 illustrates an example functional block diagram of a video encoder 203 associated with a video source 201, according to an embodiment of the present disclosure.

[0036] The video encoder 203 may include, for example, an encoder, which is a source coder 430, a coding engine 432, a (local) decoder 433, a reference picture memory 434, a predictor 435, a transmitter 440, an entropy coder 445, a controller 450, and a channel 460.

[0037] The encoder 203 may receive video samples from a video source 201 (not part of the encoder) that may capture video images to be coded by the encoder 203 .

[0038] The video source 201 may provide a source video sequence to be coded by the encoder 203 in the form of a digital video sample stream that can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCB, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source 201 may be a storage device that stores previously prepared video. In a video conferencing system, the video source 201 may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that, when viewed in sequence, impart motion. The picture itself may be organized as a spatial array of pixels, each pixel may have one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. The remainder of this specification focuses on samples.

[0039] According to an embodiment, the encoder 203 may code and compress pictures of a source video sequence into a coded video sequence 443 in real time or under any other time constraint required by the application. Forcing an appropriate coding rate is one function of the controller 450. The controller 450 may also control and be operatively coupled to other functional units as described below. Couplings are not shown for clarity. Parameters set by the controller 450 may include parameters related to rate control (picture skip, quantizer, lambda value for rate distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. One skilled in the art can readily identify other functions of the controller 450 as they may relate to a video encoder 203 optimized for a particular system design.

[0040] Some video encoders operate in what one skilled in the art would easily recognize as a "coding loop". As an oversimplified description, in an example, the coding loop can consist of the encoding part of the source coder 430 (responsible for generating symbols based on the input picture to be coded and on the reference pictures) and a (local) decoder 433 embedded in the video encoder 203. The decoder 433 reconstructs the symbols to generate sample data that the (remote) decoder would also generate if the compression between the symbols and the coded video stream was lossless for the particular video compression technique. That reconstructed sample stream is input to the reference picture memory 434. Since the decoding of the symbol stream results in a bit-exact result independent of the location of the decoder (local or remote), the contents in the reference picture memory are also bit-perfect between the local and remote encoders. That is, the predictive part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder would "see" when using the prediction during decoding. This basic principle of reference picture synchronicity (and the resulting drift when synchronicity cannot be maintained, for example due to channel errors) is known to those skilled in the art.

[0041] The operation of the "local" decoder 433 may be the same as the "remote" decoder 210 already described above in detail in conjunction with Figure 3. However, the entropy decoding portion of the decoder 210, including the channel 312, the receiver 310, the buffer memory 315 and the parser 320, may not be fully implemented in the local decoder 433, since symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder 445 and the parser 320 may be lossless.

[0042] It can be observed at this point that any decoder technique, except for parsing / entropy decoding, present in the decoder may need to be present in approximately the same functional form in the corresponding encoder. For this reason, the disclosed subject matter focuses on the operation of the decoder. The description of the encoder techniques is omitted, in that they may be the inverse of the decoder techniques described generically. Only to certain extents are more detailed descriptions required and are given below.

[0043] As part of its operation, the source coder 430 may perform motion-compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from the video sequence, designated as “reference frames.” In this manner, the coding engine 432 codes differences between pixel blocks of the input frame and pixel blocks of a reference frame that may be selected as a predictive reference for the input frame.

[0044] The local decoder 433 may decode the coded video data of the frames that may be designated as reference frames based on the symbols generated by the source coder 430. The operation of the coding engine 432 may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence may usually be a copy of the source video sequence with some errors. The local decoder 433 may reproduce the decoding process that may be performed by the video decoder on the reference frames, causing the reconstructed reference frames to be stored in the reference picture memory 434. In this way, the encoder 403 may locally store copies of the reconstructed reference frames that have common content with the reconstructed reference frames that would be obtained by the far-end video decoder (without transmission errors).

[0045] The predictor 435 may perform a prediction search for the coding engine 432. That is, for a new frame to be coded, the predictor 435 may look for certain metadata or sample data (as candidate reference pixel blocks) from the reference picture memory 434, such as reference picture motion vectors, block shapes, etc., that can be suitable prediction references for the new frame. The predictor 435 may operate on a sample block-by-pixel block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor 435, the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory 434.

[0046] Controller 450 may manage the coding operations of source coder 430, including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0047] The output of all the above functional units may undergo entropy coding in the entropy coder 445. The entropy coder converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc.

[0048] The transmitter 440 may buffer the coded video sequence generated by the entropy coder 445 to prepare it for transmission over a communication channel 460, which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter 440 may merge the coded video data from the source coder 430 with other data to be transmitted, such as coded audio data and / or ancillary data streams (sources not shown).

[0049] The controller 450 may manage the operation of the encoder 203. During coding, the controller 450 may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as an intra picture (I picture), a predicted picture (P picture), or a bidirectionally predicted picture (B picture).

[0050] An Intra Picture (I-picture) may be a picture that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow various types of Intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of such variations of I-pictures and their respective applications and characteristics.

[0051] A Predictive Picture (P-picture) may be a picture that can be coded and decoded by intra- or inter-prediction using at most one motion vector and reference index to predict sample values ​​for each block.

[0052] A Bi-directionally Predictive Picture (B-picture) may be a picture that can be coded and decoded by intra- or inter-prediction using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple-predictive picture(s) can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0053] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to each picture of the block. For example, blocks of I pictures may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of P pictures may be coded non-predictively, with spatial prediction, or with temporal prediction with reference to one previously coded reference picture. Blocks of B pictures may be coded non-predictively, with spatial prediction, or with temporal prediction with reference to one or two previously coded reference pictures.

[0054] Video encoder 203 may perform coding operations according to a given video coding technique or standard, such as ITU-T Recommendation H.265. During its operation, video encoder 203 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax defined by the video coding technique or standard being used.

[0055] In an embodiment, the transmitter 440 may transmit additional data along with the encoded video. The source coder 430 may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, or Visual Usability Information (VUI) parameter set fragments, etc.

[0056] According to an embodiment of the present disclosure, a block-wise entropy coding method in a NIC is provided to improve coding efficiency.

[0057] For example, with reference to FIGS. 5 and 6, an exemplary method of the present disclosure is described below.

[0058] FIG. 5 illustrates a NIC encoding method according to an embodiment of the present disclosure that may be performed by an encoder of the present disclosure.

[0059] 5, for each input image 510 to be compressed, the input image 510 may be partitioned into smaller blocks 520_1, 520_2, ..., 520_n. For example, the input image 510 may be divided into eight blocks 520_1, 520_2, ..., 520_n as shown in Fig. 5. Since the hardware may have memory limitations, it makes sense to split the high-resolution image into smaller blocks for parallel processing.

[0060] Each of the blocks 520_1, 520_2, ..., 520_n may be compressed by the NN-based encoder 530 to obtain compressed features 540_1, 540_2, ..., 540_n. For example, the blocks 520_1, 520_2, ..., 520_n may be compressed to obtain compressed features 540_1, 540_2, ..., 540_n, respectively. As shown in Figure 5, the blocks 520_1, 520_2, ..., 520_n are processed by the NN-based encoder 530 such that the eight compressed features 540_1, 540_2, ..., 540_n correspond to the eight blocks 520_1, 520_2, ..., 520_n, respectively. According to an embodiment, the compressed features 540_1, 540_2, ..., 540_n may be compressed feature blocks.

[0061] Each compressed feature 540_1, 540_2, ..., 540_n may be compressed by an entropy encoder 560 to generate a compressed bitstream 570. In accordance with an embodiment of the present disclosure, prior to entropy encoding, all or a portion of the compressed features 540_1, 540_2, ..., 540_n may be transformed (e.g., concatenated) to obtain reshaped compressed features 550_1, 550_2, ..., 550_m, where "m" may be less than "n", i.e., the total number of reshaped compressed features 550_1, 550_2, ..., 550_m is less than the total number of compressed features 540_1, 540_2, ..., 540_n. 5, in order to achieve better compression performance of the entropy coder, eight compressed features 540_1, 540_2,..., 540_n are transformed into four reshaped compressed features 550_1, 550_2,..., 550_m. According to an embodiment, the transformation can be a concatenation, upsampling, or NN-based method. According to an embodiment, the reshaped compressed features 550_1, 550_2,..., 550_m can be reshaped compressed feature blocks. According to an embodiment, the reshaped compressed feature blocks can have a larger size than the compressed feature blocks from which they were obtained.

[0062] Following transformation, the reshaped compressed features 550_1, 550_2,..., 550_m may be processed by an entropy encoder 560 to generate a compressed bitstream 570. The compressed bitstream 570 may be transmitted to a decoder, for example.

[0063] FIG. 6 illustrates a NIC decoding method 600 according to an embodiment of the present disclosure that may be performed by a decoder of the present disclosure.

[0064] 6, the compressed bitstream 570 is processed by an entropy decoder to generate decompressed features 620_1, 620_2,..., 620_m. According to an embodiment, the decompressed features 620_1, 620_2,..., 620_m may be decompressed feature blocks.

[0065] The decompressed features 620_1, 620_2,..., 620_m may be transformed (e.g., split) into reshaped decompressed features 630_1, 630_2,..., 630_n, where "n" is greater than "m". That is, the total number of decompressed features 620_1, 620_2,..., 620_m is less than the total number of reshaped decompressed features 630_1, 630_2,..., 630_n. According to an embodiment, the reshaped decompressed features 630_1, 630_2,..., 630_n may be reshaped decompressed feature blocks. According to an embodiment, the reshaped decompressed feature blocks may have a smaller size than the decompressed feature blocks from which they were obtained.

[0066] According to an embodiment, the total number of decompressed features 620_1, 620_2, ..., 620_m may be the same as the total number of reshaped compressed features 550_1, 550_2, ..., 550_m, and the total number of reshaped decompressed features 630_1, 630_2, ..., 630_n may be the same as the total number of compressed features 540_1, 540_2, ..., 540_n, although embodiments are not so limited and the total numbers may differ, for example, as shown in Figures 5-6. For example, as a non-limiting example, there may be eight compressed features 540_1, 540_2, ..., 540_n, four reshaped compressed features 550_1, 550_2, ..., 550_m, eight decompressed features 620_1, 620_2, ..., 620_m, and sixteen reshaped decompressed features 630_1, 630_2, ..., 630_n.

[0067] After obtaining the reshaped decompressed features 630_1, 630_2, ..., 630_n, the reshaped decompressed features 630_1, 630_2, ..., 630_n may be decoded by the NN-based decoder 640 to obtain a reconstructed block 650. According to an embodiment, the reconstructed block 650 may be a reconstructed picture. The reconstructed picture as a whole may be further processed by a post-enhancement 660 to improve the visual quality of the reconstructed picture. For example, the post-enhancement 660 may include deblocking and / or noise reduction. According to an embodiment, the reconstructed picture may then be displayed on a display.

[0068] In the conventional NIC framework, the NN-based entropy coder only encodes a single block of compressed features at a time, and the scale and variance of the entropy model are only evaluated on a block-by-block basis. According to embodiments of the present disclosure, multiple blocks are processed by the NN-based encoder and then combined for entropy coding.

[0069] In one embodiment of the present disclosure, during training of the models utilized by the NN-based encoder and the NN-based decoder of the present disclosure, the encoder and the decoder may be trained together or separately. A transformation operation between the compressed features and the reshaped features may also be added to the bitstream, and this information may be treated as a signal to the decoder performing the decoding process. For example, according to an embodiment of the present disclosure, an encoder performing the encoding process illustrated in FIG. 5 may send an indicator (e.g., a syntax element) of the transformation operation used by the encoder in the encoded bitstream to a decoder performing the decoding process illustrated in FIG. 6. The decoder may obtain the indicator and transform the decompressed features into reshaped decompressed features based on the transformation signaled by the indicator. For example, the decoder may use an inverse transformation corresponding to the transformation used by the encoder.

[0070] The shape of the compressed feature blocks, the reshaped compressed feature blocks, the decompressed feature blocks, and / or the reshaped decompressed feature blocks may be square or rectangular. The blocks in one type of block (e.g., the compressed feature blocks or the decompressed feature blocks) may be the same shape and / or size as each other, or may have a different shape and / or size than at least one other block in the one type of block. According to an embodiment, the blocks in one type (e.g., the compressed feature blocks or the decompressed feature blocks) may be transformed (e.g., split or combined) into different sizes.

[0071] According to an embodiment, at least one processor and a memory storing computer program instructions may be provided. The computer program instructions, when executed by the at least one processor, may implement an encoder or a decoder and may perform any number of functions described in this disclosure. For example, with reference to FIG. 7, the at least one processor may implement an encoder 700. The encoder 700 may be a NIC encoder. The computer program instructions may include, for example, an encoding code 710 configured to cause the at least one processor to code at least one picture received, and a transmitting code 760 configured to cause the at least one processor to transmit a bitstream obtained by coding the at least one picture, for example to a decoder. The encoding code 710 may include, for example, a splitting code 720, a first acquisition code 730, a second acquisition code 740, and a third acquisition code 750.

[0072] The partitioning code 720 may be configured to cause at least one processor to partition an image into multiple blocks according to an embodiment of the present disclosure.

[0073] The first acquisition code 730 may be configured to cause the at least one processor to obtain compressed features by encoding the plurality of blocks according to an embodiment of the present disclosure. According to an embodiment, compressed feature blocks are obtained. According to an embodiment, the plurality of blocks are encoded with a neural network by using the NN-based encoder 530.

[0074] The second acquisition code 740 may be configured to cause the at least one processor to obtain reshaped compressed features by transforming the compressed features according to an embodiment of the present disclosure, where the total number of reshaped compressed features is less than the total number of compressed features. According to an embodiment, a reshaped compressed feature block is obtained, and transforming the compressed features includes transforming the compressed feature block into a reshaped compressed feature block that is larger than the compressed feature block. According to an embodiment, the transformation includes concatenation, upsampling, or a NN-based method.

[0075] The third retrieval code 750 may be configured to cause the at least one processor to retrieve a bitstream including the encoded image by entropy encoding the reshaped compressed features according to an embodiment of the present disclosure. According to an embodiment, the third retrieval code 750 may be further configured to cause the at least one processor to include in the bitstream an indicator (e.g., a syntax element) indicating the transformation operation used by the encoder 700.

[0076] According to an embodiment, the compressed feature block or the reshaped compressed feature block have a square or rectangular shape, respectively. According to an embodiment, the reshaped compressed feature blocks are the same shape as each other. According to an embodiment, at least two of the reshaped compressed feature blocks have different shapes from each other. According to an embodiment, the reshaped compressed feature blocks are the same size as each other. According to an embodiment, at least two of the reshaped compressed feature blocks are different sizes from each other.

[0077] According to an embodiment, a decoder-side process corresponding to the above process may be implemented by a decoding code for decoding a picture, as would be understood by one of ordinary skill in the art based on the above description.

[0078] 8, the at least one processor may implement a decoder 800. The decoder 800 may be a NIC decoder. The computer program instructions stored in the memory of the decoder 800 may include, for example, a receive code 810, a decode code 820, and a display code 870.

[0079] The receiving code 810 may be configured to cause at least one processor to receive a bitstream including at least one encoded image.

[0080] The decoding code may be configured to cause the at least one processor to decode the received at least one encoded image. According to an embodiment, the decoding code 820 may include, for example, a first acquisition code 830, a second acquisition code 840, a third acquisition code 850, and a post-enhancement code 860.

[0081] The first acquisition code 830 may be configured to cause the at least one processor to obtain decompressed features of the encoded image by entropy decoding the bitstream. According to an embodiment, decompressed feature blocks are obtained.

[0082] The second acquisition code 840 may be configured to cause the at least one processor to obtain reshaped decompressed features by transforming the decompressed features, where the total number of reshaped decompressed features is greater than the total number of decompressed features. According to an embodiment, a reshaped compressed feature block is obtained, and transforming the decompressed features includes transforming the decompressed feature block into a reshaped decompressed feature block that is smaller than the decompressed feature block. According to an embodiment, the transforming includes splitting the decompressed feature block into reshaped compressed feature blocks. According to an embodiment, the second acquisition code 840 may be further configured to cause the at least one processor to obtain an indicator included in the bitstream, and to transform the decompressed features into reshaped decompressed features based on a transformation signaled by the indicator.

[0083] The third acquisition code 850 may be configured to cause the at least one processor to acquire a reconstructed image by decoding the reshaped decompressed features. According to an embodiment, the reshaped decompressed features are decoded by a neural network by using a NN-based decoder.

[0084] The post-enhancement code 860 may be configured to cause the at least one processor to perform post-enhancement on the reconstructed image. Post-enhancement may include, for example, deblocking and / or noise reduction.

[0085] The display code 870 may be configured to cause the at least one processor to control a display to display the reconstructed image after post-enhancement has been performed.

[0086] According to an embodiment, the decompressed feature block or the reshaped decompressed feature block have a square or rectangular shape, respectively. According to an embodiment, the reshaped decompressed feature blocks are the same shape as each other. According to an embodiment, at least two of the reshaped decompressed feature blocks have different shapes from each other. According to an embodiment, the reshaped decompressed feature blocks are the same size as each other. According to an embodiment, at least two of the reshaped decompressed feature blocks are different sizes from each other.

[0087] The embodiments of the present disclosure disclose improvements over previous E2E image compression methods.

[0088] For example, embodiments of the present disclosure include aspects of splitting and transforming blocks at the encoder and decoder sides, which improves the processing speed at the encoder side with reasonable compression performance and reduces the memory burden at the decoder side.

[0089] Additionally, embodiments of the present disclosure provide a flexible and general framework that accommodates various types of quality metrics.

[0090] The techniques of the embodiments of the present disclosure described above can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 9 shows a computer system suitable for implementing embodiments of the disclosed subject matter.

[0091] The computer software can be coded in any suitable machine code or computer language that can be subject to mechanisms such as assembly, compilation, linking, etc. to generate code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., directly, or through interpretation, microcode execution, etc.

[0092] The instructions may be executable by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming consoles, Internet of Things devices, and the like.

[0093] 9 for computer system 900 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 900.

[0094] The computer system 900 may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more users through, for example, tactile input (e.g., keystrokes, swipes, dataglove actions), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still camera), video (e.g., two-dimensional video, three-dimensional video including stereoscopic video), and the like.

[0095] The input human interface devices may include one or more of a keyboard 901, a mouse 902, a trackpad 903, a touch screen 910, a data glove, a joystick 905, a microphone 906, a scanner 907, and a camera 908 (only one of each is shown).

[0096] The computer system 900 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 910, data gloves, or joystick 905, although there may also be haptic feedback devices that do not function as input devices). For example, such devices may include audio output devices (e.g., speakers 909, headphones (not shown)), visual output devices (e.g., screens 910 including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or output in more than three dimensions by means of stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0097] The computer system 900 may also include human accessible storage devices and their associated media, such as CD / DVD ROM / RW 920 with CD / DVD or similar media 921, thumb drives 922, removable hard disks or solid state drives 923, legacy magnetic media, such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices, such as security dongles (not shown), and the like.

[0098] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0099] The computer system 900 may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wireline, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial networks including CANBus, etc. Certain networks generally require an external network interface adapter attached to a particular general purpose digital port or peripheral bus 949 (e.g., a USB port of the computer system 900, etc.). Others are generally built into the core of the computer system 900 by attachment to a system bus as described below (e.g., an Ethernet network to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 900 may communicate with other entities. Such communications may be one-way receive-only (e.g., broadcast TV) or one-way transmit-only (e.g., CANBus to a specific CANBus device), or may be two-way to other computer systems, for example, using local or wide area digital networks. Such communications may include communications to cloud computing environment 955. Specific protocols or protocol stacks may be used with each of the networks and network interfaces as described above.

[0100] The above-mentioned human interface devices, human accessible storage devices, and network interface 954 may be attached to the core 940 of the computer system 900 .

[0101] The core 940 may include one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, dedicated programmable processing units in the form of field programmable gate areas (FPGAs) 943, hardware accelerators for specific tasks 944, etc. These devices may be connected through a system bus 948, along with read only memory (ROM) 945, random access memory 946, internal mass storage such as an internal non-user accessible hard drive, SSD, etc. 947. In some computer systems, the system bus 948 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals may be attached directly to the core's system bus 948 or through a peripheral bus 949. Architectures for peripheral buses include PCI, USB, etc. A graphics adapter 950 may be included in the core 940.

[0102] The CPU 941, GPU 942, FPGA 943, and accelerator 944 can execute certain instructions that may combine to constitute the above computer code. The computer code may be stored in ROM 945 or RAM 946. Temporary data may also be stored in RAM 946, while persistent data may be stored, for example, in an internal mass storage device 947. Rapid storage and retrieval from any of the memory devices may be enabled by the use of a cache memory. A cache memory may be closely associated with one or more of the CPU 941, GPU 942, mass storage device 947, ROM 945, RAM 946, etc.

[0103] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those of ordinary skill in the computer software arts.

[0104] By way of example, and not by way of limitation, the architecture, and in particular the computer system 900 having the core 940, may provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with the user-accessible mass storage devices introduced above, in addition to specific storage of the core 940 that is non-transitory in nature, such as the core internal mass storage device 947 or the ROM 945. Software implementing various embodiments of the present disclosure may be stored in such devices and executable by the core 940. The computer-readable media may include one or more memory devices or chips, depending on the particular needs. The software may cause the core 940, and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in the RAM 946 and modifying such data structures according to the software-defined processes. Additionally or alternatively, the computer system may provide functionality as a result of hardwired or otherwise embodied logic in circuitry (e.g., accelerator 944) that may operate in place of or in conjunction with software to perform certain processes or certain portions of certain processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (e.g., integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0105] While the present disclosure has described several non-limiting examples, there are alterations, permutations, and various substitute equivalents that are within the scope of the present disclosure. Thus, it will be apparent to those skilled in the art that numerous systems and methods will be able to embody the principles of the present disclosure and thus be within its spirit and scope, even if not explicitly shown or described herein.

Claims

1. 1. A method performed by at least one processor of a video encoder, comprising: receiving a bitstream containing an image; Partitioning the image into a number of blocks; compressing each of the plurality of blocks by a neural network based encoder; obtaining a plurality of compressed features by obtaining block-wise compressed features from among the plurality of blocks in the image; obtaining a plurality of reshaped compressed features by concatenating at least some of the plurality of compressed features, wherein at least some of the plurality of compressed features are concatenated such that the plurality of reshaped compressed features have respective sizes or shapes; processing the reshaped compressed features by an entropy encoder to generate a compressed bitstream; encoding the bitstream including the image based on the compressed bitstream; having and encoding the bitstream includes including in the encoded bitstream an indicator of a transformation operation used by the video encoder between at least a portion of the plurality of compressed features and the plurality of reshaped compressed features. method.

2. obtaining a plurality of compressed features includes obtaining a plurality of compressed feature blocks; and concatenating at least some of the compressed feature blocks includes concatenating at least some of the compressed feature blocks into a reshaped compressed feature block that is larger than each of the at least some compressed feature blocks. The method of claim 1.

3. the compressed feature block or the reshaped compressed feature block each have a square shape; The method of claim 2.

4. the compressed feature block or the reshaped compressed feature block each have a rectangular shape; The method of claim 2.

5. The reshaped compressed feature blocks are identical in shape to each other. The method of claim 2.

6. At least two of the reshaped compressed feature blocks have different shapes from each other. The method of claim 2.

7. the reshaped compressed feature blocks are equal in size to each other; The method of claim 2.

8. At least two of the reshaped compressed feature blocks are different in size from each other. The method of claim 2.

9. transmitting the encoded bitstream to a decoder. The method of claim 1.

10. at least one memory configured to store computer program code; at least one processor configured to access the computer program code and to act as directed by the computer program code; having The computer program code causes the at least one processor to: A system comprising encoded code configured to perform the method of any one of claims 1 to 9.

11. When executed by at least one processor, the at least one processor is A computer program for carrying out the method according to any one of claims 1 to 9.

12. 1. A method performed by at least one processor of a video decoder, comprising: receiving a compressed bitstream; processing the compressed bitstream with an entropy decoder to generate decompressed features; obtaining an indicator from the compressed bitstream and obtaining reshaped decompressed features by partitioning the decompressed features with a size or shape corresponding to a transformation indicated by the indicator; decoding the reshaped decompressed features by a neural network based decoder to obtain a reconstructed block; The method according to claim 1,

Images