Method, medium and device for block-by-block entropy coding in neural image compression

Abstract

A block-by-block entropy coding method, device and storage medium in neural image compression are provided. A method includes: receiving a code stream including an image; partitioning the image into a plurality of blocks; compressing each of the plurality of blocks by a neural network-based encoder; obtaining a compression feature of the image by obtaining a compression feature of each of the plurality of blocks in the image; processing the compression feature of the image by an entropy encoder to generate a first compressed code stream; obtaining a plurality of reshaped compression features by cascading the compression features of the image; processing the plurality of reshaped compression features by an entropy encoder to generate a second compressed code stream; and encoding the code stream including the image based on the second compressed code stream.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Provisional Application No. 63 / 212,469, filed on June 18, 2021, the disclosure of which is incorporated by reference in its entirety into this application. Technical Field

[0003] Embodiments of the present disclosure relate to block-wise entropy coding systems and methods in neural image compression. Background Art

[0004] In a video or image coding framework based on deep neural networks, different modules can be jointly optimized from input to output to improve the final goal (e.g., minimizing rate-distortion loss) by leveraging large training datasets and complex processing learning algorithms. In contrast, traditional hybrid video codecs usually focus on optimizing a single module, which may not lead to coding gains in overall performance. Therefore, end-to-end (E2E) optimized Neural Image Compression (NIC) is designed as a holistic optimization to achieve better coding performance.

[0005] Recently, ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) has been actively looking for potential needs for standardization of future video coding technologies. ISO / IEC JPEG has established the JPEG-AI group focusing on AI-based end-to-end neural image compression. At the same time, many technology companies have also funded dedicated research projects at NIC. The success of recent methods has led to increasing interest in advanced neural image and video compression methods in the industry.

[0006] The general framework of NIC is described as the following steps. Given an input image x, the goal of NIC is to use the image as the input to a deep neural network (DNN) encoder to compute a compact compressed representation for storage and transmission. This compressed representation will then be quantized for arithmetic coding to generate a bitstream. The bitstream will be returned to the arithmetic decoding, which will be used as the input to the DNN decoder to reconstruct the image. In addition, hyperprior and context models can be used to further improve the compression performance.

[0007] During model training, a trade-off between the rate-distortion (RD) loss to achieve the distortion loss of the reconstructed image and the bit consumption R of the compact representation with a trade-off hyperparameter λ is optimized, as shown in the following Equation 1:

[0008]

[0009] where x is the input image, is the reconstructed image, and is a compressed representation. Summary of the invention

[0010] Embodiments of the present disclosure provide a block-by-block entropy coding method for optimizing rate-distortion performance of image compression in an end-to-end (E2E) framework. The E2E optimized framework may be a DNN-based image or video encoding and decoding framework that has been well pre-trained for both encoders and decoders.

[0011] According to an embodiment of the present disclosure, a method performed by at least one processor in a video encoder is provided. The method includes: receiving a code stream including an image; partitioning the image into a plurality of blocks; compressing each of the plurality of blocks by a neural network-based encoder; obtaining a compression feature by obtaining a compression feature of each of the plurality of blocks in the image; processing the compression feature by an entropy encoder to generate a first compressed code stream; obtaining a plurality of reshaped compression features by cascading the compression features; processing the plurality of reshaped compression features by an entropy encoder to generate a second compressed code stream; and encoding the code stream including the image based on the second compressed code stream.

[0012] According to an embodiment of the present disclosure, a system is provided, the system comprising: at least one memory configured to store computer program code; and at least one processor configured to access the computer program code and operate as indicated by the computer program code. The computer program code comprises an encoding code, the encoding code being configured to cause at least one processor to: partition an image in a code stream into a plurality of blocks; compress each of the plurality of blocks by an encoder based on a neural network; obtain a compression feature by obtaining a compression feature of each of the plurality of blocks in the image; process the compression feature by an entropy encoder to generate a first compressed code stream; obtain a plurality of reshaped compressed features by cascading the compressed features; process the plurality of reshaped compressed features by an entropy encoder to generate a second compressed code stream; and encode a code stream including an image based on the second compressed code stream.

[0013] According to an embodiment of the present disclosure, a non-transitory computer-readable medium storing computer code is provided, and the computer code is configured to cause the at least one processor to implement a video encoder when executed by at least one processor: receiving a code stream including an image; partitioning the image into multiple blocks; compressing each of the multiple blocks by a neural network-based encoder; obtaining compression features by obtaining compression features of each of the multiple blocks in the image; processing the compression features by an entropy encoder to generate a first compressed code stream; obtaining multiple reshaped compression features by cascading the compression features; processing the multiple reshaped compression features by the entropy encoder to generate a second compressed code stream; and encoding the code stream including the image based on the second compressed code stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Other features, properties and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0015] Figure 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0016] Figure 2 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0017] Figure 3 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment.

[0018] Figure 4 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0019] Figure 5 The NIC encoding method that can be performed by an encoder according to an embodiment of the present disclosure is illustrated.

[0020] Figure 6 A NIC decoding method 600 that may be performed by a decoder according to an embodiment of the present disclosure is illustrated.

[0021] Figure 7 is a block diagram of computer code according to an embodiment.

[0022] Figure 8 is a block diagram of computer code according to an embodiment.

[0023] Fig. 9 is a diagram of a computer system suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] Figure 11 is a simplified block diagram of a communication system 100 according to an embodiment disclosed in the present application. The communication system 100 includes at least two terminals (110, 120), which can communicate with each other through a network 150. For unidirectional data transmission, a first terminal 110 can encode video data locally for transmission to another terminal 120 through the network 150. A second terminal 220 can receive encoded video data from other terminals from the network 150, decode the encoded video data, and display the recovered video data. Unidirectional data transmission is more common in applications such as media services.

[0025] Figure 1 A second pair of terminals 130 and 140 is shown to support bidirectional transmission of encoded video, which may occur, for example, during a video conference. For bidirectional data transmission, each of the terminals 130 and 140 may encode locally captured video data for transmission to the other of the terminals 130 and 140 via the network 150. Each of the terminals 130 and 140 may also receive encoded video data transmitted by the other of the terminals 130 and 140, and may decode the encoded video data and display the recovered video data on a local display device.

[0026] exist Figure 1 In an embodiment of the present invention, the terminals 110-140 may be servers, personal computers and smart phones, and / or any other type of terminals. For example, the terminals 110-140 may be laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. The network 150 represents any number of networks that transmit encoded video data between the terminals 110-140, including, for example, wired (wired) and / or wireless communication networks. The communication network 150 may exchange data in circuit switching and / or packet switching channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purposes of this discussion, unless explained below, the architecture and topology of the network 150 may be irrelevant to the operations disclosed in the present application.

[0027] As an example, Figure 2 The video encoder and video decoder are shown in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0028] like Figure 2As shown, the streaming system 200 may include a capture subsystem 213, which may include a video source 201 and an encoder 203. The video source may be, for example, a digital camera and is used to create an uncompressed video sample stream 202. The uncompressed video sample stream 202 may provide a high data volume compared to the encoded video stream, and the uncompressed video sample stream 202 may be processed by an encoder 203, which is coupled to the camera 201. The encoder 203 may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the video sample stream, the encoded video stream 204 may include a lower data volume, which may be stored on the streaming server 205 for future use. One or more streaming clients 206 may access the streaming server 205 to retrieve a video bitstream 209, which is a copy of the encoded video stream 204.

[0029] In an embodiment, the streaming server 205 may also function as a Media-Aware Network Element (MANE). For example, the streaming server 205 may be configured to prune the encoded video bitstream 204 so as to customize a possibly different bitstream for one or more streaming clients 206. In an embodiment, the MANE may be provided separately from the streaming server 205 in the streaming system 200.

[0030] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 may, for example, decode a video stream 209 that is an input copy of the encoded video stream 204, and the video decoder 210 creates an output video sample stream 211 that can be rendered on a display 212 or another rendering device (not depicted). In some streaming systems, the video streams 204 and 209 may be encoded according to certain video codec / compression standards. Examples of such standards include, but are not limited to, ITU-T Recommendation H.265. The video codec standard being developed is informally referred to as Versatile Video Coding (VVC). Embodiments of the present disclosure may be used in the context of VVC.

[0031] Figure 3 An exemplary functional block diagram of a video decoder 210 attached to a display 212 is shown according to an embodiment of the present disclosure.

[0032] The video decoder 210 may include a channel 312, a receiver 310, a buffer memory 315, an entropy decoder / parser 320, a scaler / inverse transform unit 351, an intra picture prediction unit 352, a motion compensation prediction unit 353, an aggregator 355, a loop filter unit 356, a reference picture memory 357, and a current picture memory 358. In at least one embodiment, the video decoder 210 may include an integrated circuit, a series of integrated circuits, and / or other electronic circuits. The video decoder 210 may also be partially or fully embodied as software that runs on one or more CPUs with associated memory.

[0033] In some embodiments and other embodiments, the receiver 310 may receive one or more encoded video sequences to be decoded by the decoder 210; in the same embodiment or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequence may be received from a channel 312, which may be a hardware / software link to a storage device storing the encoded video data. The receiver 310 may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective consuming entities (not shown). The receiver 310 may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory 315 may be coupled between the receiver 310 and the entropy decoder / parser 320 (hereinafter referred to as "parser"). When the receiver 310 receives data from a storage / forward device with sufficient bandwidth and controllability or from an isochronous synchronous network, it may not be necessary to configure the buffer memory 315, or the buffer memory may be made smaller. Of course, in order to be used on a traffic packet network such as the Internet, a buffer memory 315 may also be required, and the buffer memory may be relatively large and may have an adaptive size.

[0034] The video decoder 210 may include a parser 320 to reconstruct symbols 321 from an entropy-encoded coded video sequence. The types of symbols include, for example, information for managing the operation of the decoder 210 and potential information for controlling a display device (e.g., display 212), which may be coupled to a display device such as a display 212. Figure 4The decoder shown in . The control information for the display device may be a parameter set fragment (not indicated) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser 320 may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be performed according to a video coding technique or standard, and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser 320 may extract a subgroup parameter set of at least one subgroup of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), and the like. The parser 320 may also extract information such as transform coefficients, quantizer parameter values, motion vectors, and the like from the coded video sequence. .

[0035] Parser 320 may perform entropy decoding / parsing operations on the video sequence received from buffer memory 315 , thereby creating symbols 321 .

[0036] Depending on the type of the coded video picture or a portion of the coded video picture (e.g., inter- and intra-pictures, inter- and intra-blocks), and other factors, the reconstruction of the symbol 321 may involve a plurality of different units. Which units are involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by the parser 320. For the sake of brevity, such subgroup control information flow between the parser 320 and the plurality of units below is not described.

[0037] In addition to the functional blocks already mentioned, decoder 210 may be conceptually subdivided into several functional units as described below. In a practical embodiment operating under commercial constraints, many of these units closely interact with each other and may be integrated with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide into the functional units below.

[0038] One unit is a sealer / inverse transform unit 351. The sealer / inverse transform unit 351 may receive quantized transform coefficients as symbols 321 from the parser 320, as well as control information, including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit 351 may output a block including sample values, which may be input into an aggregator 355.

[0039] In some cases, the output samples of the sealer / inverse transform unit 351 may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by the intra-picture prediction unit 352. In some cases, the intra-picture prediction unit 352 uses surrounding already reconstructed information obtained from the current (partially reconstructed) picture of the current picture memory 358 to generate a block of the same size and shape as the block being reconstructed. In some cases, the aggregator 355 adds the prediction information generated by the intra-prediction unit 352 to the output sample information provided by the sealer / inverse transform unit 351 on a per-sample basis.

[0040] In other cases, the output samples of the scaler / inverse transform unit 351 may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit 353 may access the reference picture memory 357 to extract samples for prediction. After the extracted samples are motion compensated according to the symbol 321, these samples may be added to the output of the scaler / inverse transform unit 351 (in this case referred to as residual samples or residual signals) by the aggregator 355, thereby generating output sample information. The acquisition of the prediction samples by the motion compensated prediction unit 353 from the address in the reference picture memory 357 may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit 353 in the form of the symbol 321, for example, including X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory 357 when using sub-sample accurate motion vectors, motion vector prediction mechanisms, and the like.

[0041] The output samples of aggregator 355 may be employed by various loop filtering techniques in loop filter unit 356. The video compression techniques may include in-loop filter techniques controlled by parameters included in the encoded video bitstream and available to loop filter unit 356 as symbols 321 from parser 320. However, in other embodiments, the video compression techniques may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, as well as to previously reconstructed and loop filtered sample values.

[0042] The output of the loop filter unit 356 may be a sample stream that may be output to a display device such as the display 212 and stored in the reference picture memory 357 for subsequent inter-picture prediction.

[0043] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. Once a coded picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by parser 320), the current reference picture buffer may become part of reference picture memory 357, and a new current picture memory may be reallocated before starting reconstruction of a subsequent coded picture.

[0044] The video decoder 210 may perform decoding operations according to a predetermined video compression technique as recorded, for example, in the ITU-T H.265 standard. The coded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that the coded video sequence conforms to the syntax of the video compression technique or standard and the profiles recorded in the video compression technique or standard, in a sense that it conforms to the syntax of the video compression technique or standard, as specified in the video compression technique document or standard, particularly the profiles therein. In addition, in order to conform to certain video compression techniques or standards, the complexity of the coded video sequence may be within the range defined by the video compression technique or standard level. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata of the HRD buffer management signaled in the coded video sequence.

[0045] In an embodiment, the receiver 310 may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder 210 to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0046] Figure 4 An exemplary functional block diagram of a video encoder 203 associated with a video source 201 according to an embodiment of the present disclosure is illustrated.

[0047] The video encoder 203 may include, for example, an encoder as a source encoder 430 , a codec engine 432 , a (local) decoder 433 , a reference picture memory 434 , a predictor 435 , a transmitter 440 , an entropy encoder 445 , a controller 450 , and a channel 460 .

[0048] Encoder 203 may receive video samples from a video source 201 (not part of the encoder), which may capture video images to be encoded by encoder 203 .

[0049] The video source 201 may provide a source video sequence in the form of a digital video sample stream to be encoded by the encoder 203, wherein the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source 201 may be a storage device storing previously prepared videos. In a video conferencing system, the video source 201 may be a camera that collects local image information as a video sequence. The video data may be provided as a plurality of separate pictures that are given motion when viewed sequentially. The picture itself may be constructed as a spatial pixel array, wherein each pixel may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following focuses on describing samples.

[0050] According to an embodiment, the encoder 203 may encode and compress the pictures of the source video sequence into an encoded video sequence 443 in real time or under any other time constraints required by the application. Implementing the appropriate encoding speed is a function of the controller 450. The controller 450 controls other functional units as described below and is functionally coupled to these units. For the sake of simplicity, the coupling is not indicated in the figure. The parameters set by the controller 450 may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology, etc.), picture size, picture group (group of pictures, GOP) layout, maximum motion vector search range, etc. Those skilled in the art can easily identify other functions of the controller 450, which are related to the video encoder 203 optimized for a certain system design.

[0051] Some video encoders operate in what is easily recognizable to those skilled in the art as an encoding loop. As a simple description, the encoding loop consists of the encoding portion of the source encoder 430 (e.g., responsible for creating symbols, such as a symbol stream, based on the input picture to be encoded and the reference picture) and a (local) decoder 433 embedded in the encoder 203. The decoder 433 reconstructs the symbols to create sample data, which the (remote) decoder will also create when the compression between the symbols and the encoded video bitstream is lossless in certain video compression techniques. The reconstructed sample stream can be input to the reference picture memory 434. Since the decoding of the symbol stream produces a bit-accurate result that is independent of the decoder location (local or remote), the reference picture memory contents are also bit-accurately corresponding between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction portion of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, such as due to channel errors) is well known to those skilled in the art.

[0052] The operation of the "local" decoder 433 can be combined with the above Figure 3 The detailed description is the same as for the "remote" decoder 210. However, when symbols are available and the entropy encoder 445 and the parser 320 are capable of losslessly encoding / decoding the symbols into the encoded video sequence, the entropy decoding portion of the video decoder 310, including the channel 312, the receiver 310, the buffer memory 315 and the parser 320, may not be fully implemented in the local decoder 433.

[0053] At this point, it can be observed that any decoder technology other than the parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the present application focuses on decoder operation. The description of encoder technology can be simplified because encoder technology is mutually inverse to the decoder technology described comprehensively. A more detailed description is only needed in certain areas and is provided below.

[0054] As part of the operation, the source encoder 430 may perform motion compensated predictive coding. The motion compensated predictive coding predictively encodes the input frame with reference to one or more previously encoded frames from the video sequence designated as "reference frames." In this way, the encoding engine 432 encodes the differences between the pixel blocks of the input frame and the pixel blocks of the reference frame, which may be selected as a prediction reference for the input picture.

[0055] The local decoder 433 may decode the encoded video data of the frame that may be designated as the reference frame based on the symbol created by the source encoder 430. The operation of the encoding engine 432 may be a lossy process. When the encoded video data is available at the video decoder ( Figure 4 When the video sequence is decoded at a remote video decoder (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local decoder 433 replicates the decoding process that may be performed by the video decoder on the reference frame and may cause the reconstructed reference frame to be stored in the reference picture memory 434. In this way, the encoder 203 may locally store a copy of the reconstructed reference frame that has common content (absent transmission errors) with the reconstructed reference frame to be obtained by the remote video decoder.

[0056] The predictor 435 may perform a prediction search for the encoding engine 432. That is, for a new picture to be encoded, the predictor 435 may search the reference picture memory 434 for sample data (as a candidate reference pixel block) or certain metadata, such as a reference picture motion vector, block shape, etc., that may serve as an appropriate prediction reference for the new picture. The predictor 435 may operate pixel-by-pixel based on a sample block to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor 435, it may be determined that the input picture may have a prediction reference taken from a plurality of reference pictures stored in the reference picture memory 434.

[0057] The controller 450 may manage encoding operations of the video encoder 430 , including, for example, setting parameters and subgroup parameters for encoding video data.

[0058] The outputs of all the above functional units may be entropy encoded in the entropy encoder 445. The entropy encoder 445 performs lossless compression on the symbols generated by the various functional units according to techniques well known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0059] Transmitter 440 may buffer the encoded video sequence created by entropy encoder 445 in preparation for transmission over communication channel 460, which may be a hardware / software link to a storage device where the encoded video data will be stored. Transmitter 440 may combine the encoded video data from source encoder 430 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0060] The controller 450 may manage the operation of the encoder 203. During encoding, the controller 450 may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned an intra picture (I picture), a predictive picture (P picture), or a bidirectional predictive picture (B picture).

[0061] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0062] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0063] Bi-directional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0064] The source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or the blocks may be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be non-predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be non-predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0065] The video encoder 203 may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder 203 may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0066] In an embodiment, the transmitter 440 may transmit additional data when transmitting the encoded video. The source encoder 430 may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as time / space / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0067] According to an embodiment of the present disclosure, a method for block-by-block entropy coding in a NIC is provided for improving coding and decoding efficiency.

[0068] For example, refer to Figures 5 and 6 , an example method of the present disclosure is described below.

[0069] Figure 5 The NIC encoding method according to an embodiment of the present disclosure that can be performed by the encoder of the present disclosure is illustrated.

[0070] refer to Figure 5 For each input image 510 to be compressed, the input image 510 may be partitioned into smaller blocks 520_1, 520_2, ..., 520_n. Figure 5 As shown, the input image 510 can be divided into eight blocks 520_1, 520_2, ..., 520_n. Because the hardware may have memory limitations, it is reasonable to divide the high-resolution image into small blocks for parallel processing.

[0071] Each of the blocks 520_1, 520_2, ..., 520_n may be compressed by the NN-based encoder 530 to obtain compressed features 540_1, 540_2, ..., 540_n. For example, the blocks 520_1, 520_2, ..., 520_n may be compressed to obtain compressed features 540_1, 540_2, ..., 540_n, respectively. Figure 5 As shown, blocks 520_1, 520_2, ..., 520_n are processed by the NN-based encoder 530, so that eight compressed features 540_1, 540_2, ..., 540_n correspond to the eight blocks 520_1, 520_2, ..., 520_n, respectively. According to an embodiment, the compressed features 540_1, 540_2, ..., 540_n may be compressed feature blocks.

[0072] Each compressed feature 540_1, 540_2, ..., 540_n may be processed by an entropy encoder 560 to generate a compressed code stream 570. According to an embodiment of the present disclosure, before entropy encoding, all or some of the compressed features 540_1, 540_2, ..., 540_n may be transformed (e.g., concatenated together) to obtain reshaped compressed features 550_1, 550_2, ..., 550_m, where "m" may be less than "n". In other words, the total number of reshaped compressed features 550_1, 550_2, ..., 550_m may be less than the total number of compressed features 540_1, 540_2, ..., 540_n. Figure 5As shown, in order to achieve better compression performance of the entropy encoder, eight compressed features 540_1, 540_2, ..., 540_n are transformed into four reshaped compressed features 550_1, 550_2, ..., 550_m. According to an embodiment, the transformation can be a cascade, upsampling or NN-based method. According to an embodiment, the reshaped compressed features 550_1, 550_2, ..., 550_m can be reshaped compressed feature blocks. According to an embodiment, the reshaped compressed feature blocks can have a larger size than the one or more compressed feature blocks from which they are obtained.

[0073] After the transformation, the reshaped compressed features 550_1, 550_2, ..., 550_m may be processed by an entropy encoder 560 to generate a compressed code stream 570, which may be sent to, for example, a decoder.

[0074] Figure 6 A NIC decoding method 600 according to an embodiment of the present disclosure that may be performed by a decoder of the present disclosure is illustrated.

[0075] refer to Figure 6 The compressed code stream 570 is processed by the entropy decoder 610 to generate decompressed features 620_1, 620_2, ..., 620_m. According to an embodiment, the decompressed features 620_1, 620_2, ..., 620_m may be decompressed feature blocks.

[0076] The decompressed features 620_1, 620_2, ..., 620_m may be transformed (e.g., divided) into reshaped decompressed features 630_1, 630_2, ..., 630_n, where "n" is greater than "m". In other words, the total number of decompressed features 620_1, 620_2, ..., 620_m may be less than the total number of reshaped decompressed features 630_1, 630_2, ..., 630_n. According to an embodiment, the reshaped decompressed features 630_1, 630_2, ..., 630_n may be reshaped decompressed feature blocks. According to an embodiment, the reshaped decompressed feature blocks may have a smaller size than the one or more decompressed feature blocks from which they are obtained.

[0077] According to an embodiment, the total number of decompressed features 620_1, 620_2, ..., 620_m may be the same as the total number of reshaped compressed features 550_1, 550_2, ..., 550_m, and the total number of reshaped decompressed features 630_1, 630_2, ..., 630_n may be the same as the total number of compressed features 540_1, 540_2, ..., 540_n. However, the embodiment is not limited thereto, and the total number may be different, for example, Figures 5 and 6For example, as a non-limiting example embodiment, there may be eight compressed features 540_1, 540_2, ..., 540_n; four reshaped compressed features 550_1, 550_2, ..., 550_m; eight decompressed features 620_1, 620_2, ..., 620_m; and 16 reshaped decompressed features 630_1, 630_2, ..., 630_n.

[0078] After obtaining the reshaped decompressed features 630_1, 630_2, ..., 630_n, the reshaped decompressed features 630_1, 630_2, ..., 630_n can be decoded by the NN-based decoder 640 to obtain a reconstructed block 650. According to an embodiment, the reconstructed block 650 can be a reconstructed picture. The entire reconstructed picture can be further processed by post-enhancement 660 to improve the visual quality of the reconstructed picture. For example, post-enhancement 660 can include deblocking and / or noise reduction. According to an embodiment, the reconstructed picture can then be displayed on a display.

[0079] In the traditional NIC framework, the NN-based entropy encoder encodes only a single block of compressed features at a time, and only evaluates the scale and variance of the entropy model block by block. According to an embodiment of the present disclosure, multiple blocks are processed by the NN-based encoder and then combined together for entropy encoding.

[0080] In one embodiment of the present disclosure, during the training of the model utilized by the NN-based encoder and NN-based decoder of the present disclosure, the encoder and decoder may be trained jointly or separately. Transform operations between compressed features and reshaped features may also be added to the bitstream, and this information may be considered as a signal to the decoder performing the decoding process. For example, according to an embodiment of the present disclosure, performing Figure 5 The encoder of the encoding process shown in FIG. 1 may send an indicator (e.g., a syntax element) of the transform operation used by the encoder in the encoded bitstream to the processor performing the transform operation. Figure 6 The decoder may obtain the indicator and transform the decompressed features into reshaped decompressed features based on the transform signaled by the indicator. For example, the decoder may use an inverse transform corresponding to the transform used by the encoder.

[0081] According to an embodiment, the shape of the compressed feature block, the reshaped compressed feature block, the decompressed feature block and / or the reshaped decompressed feature block may be a square or a rectangle. Each block from a single type of block (e.g., a compressed feature block or a decompressed feature block) may be the same shape and / or size as each other, or may be different in shape and / or size from at least one other block from a single type of block. According to an embodiment, a single type of block (e.g., a compressed feature block or a decompressed feature block) may be transformed (e.g., divided or combined) into different sizes.

[0082] According to an embodiment, at least one processor and a memory storing computer program instructions may be provided. The computer program instructions, when executed by the at least one processor, implement an encoder or a decoder and may perform any number of functions described in the present disclosure. For example, referring to Figure 7 , at least one processor may implement the encoder 700. The encoder 700 may be a NIC encoder. The computer program instructions may include, for example, an encoding code 710 and a sending code 760, wherein the encoding code 710 is configured to cause the at least one processor to encode the received at least one picture, and the sending code 760 is configured to cause the at least one processor to send a code stream to, for example, a decoder, wherein the code stream is obtained from encoding the at least one picture. The encoding code 710 may include, for example, a partitioning code 720, a first acquisition code 730, a second acquisition code 740, and a third acquisition code 750.

[0083] According to an embodiment of the present disclosure, the partition code 720 may be configured to cause at least one processor to partition an image into a plurality of blocks.

[0084] According to an embodiment of the present disclosure, the first acquisition code 730 may be configured to enable at least one processor to obtain a compression feature by encoding a plurality of blocks. According to an embodiment, a compression feature block is obtained. According to an embodiment, a plurality of blocks are encoded using a neural network by using an NN-based encoder.

[0085] According to an embodiment of the present disclosure, the second acquisition code 740 may be configured to enable at least one processor to obtain reshaped compressed features by transforming compressed features, the total number of reshaped compressed features being less than the total number of compressed features. According to an embodiment, a reshaped compressed feature block is obtained, and transforming the compressed features includes transforming the compressed feature block into a reshaped compressed feature block, the reshaped compressed feature block being larger than the compressed feature block. According to an embodiment, the transformation includes a cascade, upsampling, or NN-based method.

[0086] According to an embodiment of the present disclosure, the third acquisition code 750 may be configured to cause at least one processor to obtain a code stream including an encoded image by entropy encoding the reshaped compression feature. According to an embodiment, the third acquisition code 750 may be further configured to cause at least one processor to include an indicator (e.g., a syntax element) in the code stream, the indicator indicating the transform operation used by the encoder 700.

[0087] According to an embodiment, the compressed feature blocks or the reshaped compressed feature blocks are each square or rectangular. According to an embodiment, the reshaped compressed feature blocks are of the same shape as each other. According to an embodiment, at least two reshaped compressed feature blocks are of different shapes from each other. According to an embodiment, the reshaped compressed feature blocks are of the same size as each other. According to an embodiment, at least two reshaped compressed feature blocks are of different sizes from each other.

[0088] According to an embodiment, as understood by those of ordinary skill in the art based on the above description, a decoder-side process corresponding to the above process may be implemented by a decoding code, which is used to decode a picture.

[0089] For example, refer to Figure 8 , at least one processor may implement decoder 800. Decoder 800 may be a NIC decoder. Computer program instructions stored in the memory of decoder 800 may include, for example, receiving code 810, decoding code 820, and display code 870.

[0090] The receiving code 810 may be configured to enable at least one processor to receive a code stream including at least one encoded image.

[0091] The decoding code 820 may be configured to cause at least one processor to decode the received at least one encoded image. According to an embodiment, the decoding code 820 may include, for example, a first acquisition code 830, a second acquisition code 840, a third acquisition code 850, and a post-enhancement code 860.

[0092] The first obtaining code 830 may be configured to enable at least one processor to obtain the decompressed feature of the encoded image by entropy decoding the code stream. According to an embodiment, a decompressed feature block is obtained.

[0093] The second acquisition code 840 may be configured to cause at least one processor to obtain reshaped decompressed features by transforming the decompressed features, the total number of reshaped decompressed features being greater than the total number of decompressed features. According to an embodiment, a reshaped decompressed feature block is obtained, and transforming the decompressed features includes transforming the decompressed feature block into a reshaped decompressed feature block, the reshaped decompressed feature block being smaller than the compressed feature block. According to an embodiment, the transformation includes dividing the decompressed feature block into reshaped decompressed feature blocks. According to an embodiment, the second acquisition code 840 may be further configured to cause at least one processor to obtain an indicator included in the code stream, and based on the transformation signaled by the indicator, transform the decompressed features into the reshaped decompressed features.

[0094] The third obtaining code 850 may be configured to cause at least one processor to obtain a reconstructed image by decoding the reshaped decompressed features. According to an embodiment, the reshaped decompressed features are decoded using a neural network by using an NN-based decoder.

[0095] The post-enhancement code 860 may be configured to cause at least one processor to perform post-enhancement on the reconstructed image. Post-enhancement may include, for example, deblocking and / or noise reduction.

[0096] The display code 870 may be configured to cause the at least one processor to control the display to display the reconstructed image after performing the post-enhancement.

[0097] According to an embodiment, the decompressed feature blocks or the reshaped decompressed feature blocks are each square or rectangular. According to an embodiment, the reshaped decompressed feature blocks are of the same shape as each other. According to an embodiment, at least two reshaped decompressed feature blocks are of different shapes from each other. According to an embodiment, the reshaped decompressed feature blocks are of the same size as each other. According to an embodiment, at least two reshaped decompressed feature blocks are of different sizes from each other.

[0098] The embodiments of the present disclosure improve upon previous E2E image compression methods.

[0099] For example, embodiments of the present disclosure include aspects of segmentation blocks and transform blocks at both the encoder and decoder sides, which improves the processing speed at the encoder side with reasonable compression performance and reduces the memory burden at the decoder side.

[0100] Furthermore, embodiments of the present disclosure provide a flexible and general framework that accommodates various types of quality metrics.

[0101] The techniques of the embodiments of the present disclosure described above may be implemented as computer software through computer-readable instructions and physically stored in one or more computer-readable media. Fig. 9 A computer system 900 is shown that is suitable for implementing embodiments of the disclosed subject matter.

[0102] The computer software may be encoded in any suitable machine code or computer language, and may be assembled, compiled, linked, or other mechanisms to create a code comprising instructions, which may be directly executed by a computer central processing unit (CPU), graphics processing unit (GPU), or the like, or executed by decoding, microcode, or the like.

[0103] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, etc.

[0104] Fig. 9The components shown for computer system 900 are exemplary in nature and are not intended to limit the scope of use or functionality of computer software implementing embodiments of the present application. The configuration of the components should not be interpreted as having any dependency or requirement on any component or combination thereof shown in the exemplary embodiment of computer system 900.

[0105] The computer system 900 may include certain human-computer interface input devices. Such human-computer interface input devices may respond to input from one or more human users through tactile input (e.g., keyboard input, sliding, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-computer interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0106] The human-machine interface input device may include one or more of the following (only one of which is depicted): keyboard 901 , mouse 902 , touch pad 903 , touch screen 910 , data gloves, joystick 905 , microphone 906 , scanner 907 , and camera 908 .

[0107] The computer system 900 may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen 910, a data glove, or a joystick 905, but there may also be tactile feedback devices that are not used as input devices). For example, these devices may be audio output devices (e.g., speakers 909, headphones (not shown)), visual output devices (e.g., screens 910 including cathode ray tube screens, liquid crystal screens, plasma screens, organic light emitting diode screens, each of which has or does not have a touch screen input function, each of which has or does not have a tactile feedback function - some of which may output two-dimensional visual outputs or outputs of more than three dimensions by means such as stereoscopic picture output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)) and printers (not shown).

[0108] The computer system 900 may also include human-accessible storage devices and their associated media, such as optical media including high-density read-only / rewritable optical disks (CD / DVD ROM / RW) 920 with CD / DVD or similar media 921, a thumb drive 922, a removable hard disk drive or solid state drive 923, traditional magnetic media such as tapes and floppy disks (not shown), special-purpose ROM / ASIC / PLD-based devices such as security software protectors (not shown), and the like.

[0109] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0110] The computer system 900 may also include an interface to one or more communication networks. For example, the network may be wireless, wired, or optical. The network may also be a local area network, a wide area network, a metropolitan area network, an in-vehicle network, an industrial network, a real-time network, a delay-tolerant network, and the like. The network also includes local area networks such as Ethernet, wireless local area networks, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), in-vehicle and industrial networks (including CANBus), and the like. Some networks typically require an external network interface adapter for connecting to some universal data ports or peripheral buses 949 (e.g., a USB port of the computer system 900); other systems are typically integrated into the core of the computer system 900 by connecting to the system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smart phone computer system). By using any of these networks, the computer system 900 can communicate with other entities. The communication may be one-way, for reception only (e.g., wireless television), one-way, for transmission only (e.g., CAN bus to certain CAN bus devices), or two-way, such as to other computer systems via a local or wide area digital network. Such communication may include communication with a cloud computing environment 955. Each of the networks and network interfaces described above may use certain protocols and protocol stacks.

[0111] The above-mentioned human-machine interface devices, human-accessible storage devices, and network interface 954 can be connected to the core 940 of the computer system 900.

[0112] The core 940 may include one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 943, hardware accelerators 944 for specific tasks, and the like. These devices, as well as read-only memory (ROM) 945, random access memory 946, internal mass storage (e.g., internal non-user accessible hard disk drives, solid state drives, etc.) 947, and the like may be connected via a system bus 948. In some computer systems, the system bus 948 may be accessed in the form of one or more physical plugs so that it may be expanded by additional central processing units, graphics processing units, and the like. Peripheral devices may be directly attached to the system bus 948 of the core, or connected via a peripheral bus 949. The architecture of the peripheral bus includes a peripheral controller interface PCI, a universal serial bus USB, and the like. A graphics adapter 950 may be included in the core 940.

[0113] The CPU 941, GPU 942, FPGA 943, and accelerator 944 may execute certain instructions, which in combination may constitute the above-mentioned computer code. The computer code may be stored in ROM 945 or RAM 946. Transitional data may also be stored in RAM 946, while permanent data may be stored in, for example, internal mass storage 947. Fast storage and retrieval of any memory device may be achieved by using a cache memory, which may be closely associated with one or more CPUs 941, GPUs 942, mass storage 947, ROM 945, RAM 946, and the like.

[0114] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purpose of this application, or may be medium and code well known and available to those skilled in the art of computer software.

[0115] As an embodiment and not limitation, a computer system having architecture 900, in particular core 940, can provide the function of executing software contained in one or more tangible computer-readable media as a processor (including CPU, GPU, FPGA, accelerator, etc.). Such computer-readable media can be a medium associated with the above-mentioned user-accessible mass storage, and a specific memory of the core 940 having non-volatile properties, such as core internal mass storage 947 or ROM 945. The software implementing various embodiments of the present application can be stored in such a device and executed by the core 940. According to specific needs, the computer-readable medium may include one or more storage devices or chips. The software can enable the core 940, in particular the processor therein (including CPU, GPU, FPGA, etc.) to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in RAM 946 and modifying such a data structure according to a software-defined process. In addition or as an alternative, the computer system can provide a function that is logically hardwired or otherwise contained in a circuit (e.g., accelerator 944), which can replace the software or run with the software to perform a specific process or a specific part of a specific process described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuits (such as integrated circuits (ICs)) storing the executing software, circuits containing the executing logic, or both. The present application includes any suitable combination of hardware and software.

[0116] Although the present application has described a number of non-limiting exemplary embodiments, various changes, arrangements and various equivalent substitutions of the embodiments are within the scope of the present application. It should therefore be understood that those skilled in the art are capable of designing a variety of systems and methods, which, although not explicitly shown or described herein, embody the principles of the present application and are therefore within the spirit and scope of the present application.

Claims

1. A block-by-block entropy coding method in neural image compression, performed by a video encoder, characterized in that The method comprises: receiving a bitstream, the bitstream comprising an image; partitioning the image into a plurality of blocks; compressing each of the plurality of blocks by a neural network-based encoder; Obtaining a compression feature of the image by obtaining a compression feature of each of the plurality of blocks in the image; Obtaining a plurality of reshaped compressed features by cascading the compressed features of some or each of the plurality of blocks in the image; processing the plurality of reshaped compression features by an entropy encoder to generate a second compressed code stream; and The second compressed code stream is encoded, wherein the encoding of the second compressed code stream comprises: including an indicator in the encoded code stream, wherein the indicator indicates that the encoder uses a transform operation, and the transform operation is cascaded.

2. The method according to claim 1, characterized in that The obtaining of compressed features comprises obtaining compressed feature blocks; and The cascading the compressed features includes: cascading the compressed feature blocks into a reshaped compressed feature block, the reshaped compressed feature block being larger than the compressed feature block.

3. The method according to claim 2, characterized in that The compressed feature blocks or the reshaped compressed feature blocks are each square.

4. The method according to claim 2, characterized in that: The compressed feature block or the reshaped compressed feature block are each rectangular.

5. The method according to claim 2, characterized in that: The reshaped compressed feature blocks have the same shape as each other.

6. The method according to claim 2, characterized in that At least two of the reshaped compressed feature blocks have different shapes from each other.

7. The method according to claim 2, characterized in that The reshaped compressed feature blocks have the same size as each other.

8. The method according to claim 2, characterized in that: At least two of the reshaped compressed feature blocks have different sizes from each other.

9. The method according to claim 1, characterized in that: The method further includes sending the encoded code stream to a decoder.

10. A non-volatile computer-readable storage medium storing computer code, characterized in that: The computer code is configured to implement the method of any one of claims 1-9 when executed by at least one processor.

11. A computer device, characterized in that: The method comprises a processor and a memory; the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method according to any one of claims 1 to 9.

12. A method for processing a video code stream, characterized in that: The video code stream is generated according to the method described in any one of claims 1-9.

Images