Efficient activation function in neural network image compression decoder

By adopting a converter-based attention module and simplified upsampling and activation function in image compression technology, the problem of high decoding complexity is solved, and efficient and low-complexity image compression is achieved, and image quality and computing efficiency are improved. .

CN119072922APending Publication Date: 2024-12-03TENCENT AMERICA LLC

Patent Information

Application Number
CN202480001466.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-04-02
Filing Date
2024-04-03
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing neural network-based image compression technology has high decoding complexity problems, resulting in high computing cost, practicality and low efficiency, and the existing solutions fail to completely reduce complexity without affecting performance.

Method used

Using a converter-based attention module, a simplified upsampling module and a streamlined activation function, the decoder network structure is simplified, and the calculation complexity is reduced and performance is maintained, including the use of a converter-based attention module to replace the residual non- The local attention block uses pixel rearrangement as the upsampling method to remove the hyperbolic tangent function and its element-by-element operation.

Benefits of technology

Significantly reduces the computational complexity of the decoder (kMAC/pixel reduction from 833 to 172), and improves efficiency without affecting performance, achieving higher image compression and lower bitrate, enhancing images quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119072922A_ABST
    Figure CN119072922A_ABST
Patent Text Reader

Abstract

A method and apparatus including computer code configured to cause one or more processors to receive a video bitstream including a current block in a current picture, and to reconstruct the current block by transforming the current block by a neural network including a plurality of up-sampling modules and an activation module, and at least one of the activation modules comprises a Leaky ReLu function and a convolution function.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 456,782, filed on April 3, 2023, and U.S. Application No. 18 / 624,728, filed on April 2, 2024, the disclosures of which are incorporated herein by reference in their entireties. Technical Field

[0003] The present disclosure provides an effective neural network module for image compression. The structure can provide reasonable compression performance while significantly reducing the complexity of the image compression model. Background Art

[0004] Currently, ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) has been actively looking for potential needs for standardization of future video coding technologies. ISO / IEC JPEG has established the JPEG-AI group, focusing on AI-based end-to-end (E2E) neural image compression using deep neural networks (DNN). The Chinese AVS standard has also established an AVS-AI special group dedicated to neural image and video compression technology. At the same time, the team has also funded special research projects for NIC. The recent success of these methods has brought more and more industrial interest in advanced neural image and video compression technologies.

[0005] Traditional hybrid video codecs are difficult to optimize. Improvements in a single module may not bring coding gains in overall performance. In contrast, in an artificial neural network-based video / image coding framework, by performing a machine learning process, different modules can be jointly optimized from input to output to improve the final goal (e.g., rate-distortion performance), thereby achieving end-to-end optimized Neural Image Compression (NIC).

[0006] Therefore, due to these reasons, people are eager to obtain technical solutions to such problems arising in computer technology. Summary of the invention

[0007] According to one aspect of the present disclosure, there is an apparatus, and similarly, there is a method and a computer-readable medium, the apparatus comprising: at least one memory configured to store computer program code; and at least one processor configured to access the computer program code and operate according to instructions of the computer program code, the computer program code comprising: a receiving code configured to cause the at least one processor to receive a video code stream including a current block in a current picture; and a reconstruction code configured to cause the at least one processor to reconstruct the current block by transforming the current block by a neural network including multiple upsampling modules and activation modules, and at least one activation module of the activation module includes a LeakyReLu function and a convolution function.

[0008] Furthermore, in at least one of the activation modules, the output of the LeakyReLu function may be the input of the convolution function.

[0009] Furthermore, in at least one of the activation modules, the output of the convolution function may be an input of the multiplication function of at least one of the activation modules.

[0010] Furthermore, in at least one of the activation modules, the output of the multiplication function may be an input of the addition function of at least one of the activation modules.

[0011] Furthermore, in at least one of the activation modules, the output of the convolution function may be an input of the addition function of at least one of the activation modules.

[0012] Furthermore, in at least one of the activation modules, the output of the convolution function may be an input of a second LeakyReLu function of at least one of the activation modules.

[0013] Furthermore, in at least one of the activation modules, the output of the second LeakyReLu function may be an input of a multiplication function of at least one of the activation modules.

[0014] Furthermore, in at least one of the activation modules, the output of the multiplication function may be an input of the addition function of at least one of the activation modules.

[0015] Additionally, the convolution function may include a 1x1 convolution kernel.

[0016] Furthermore, at least one of the upsampling modules may include a pixel shuffle layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above-mentioned features and aspects and other features and aspects of embodiments of the present disclosure will become apparent from the following description considered in conjunction with the accompanying drawings, in which:

[0018] Figure 1 is a schematic diagram of an illustration according to an embodiment;

[0019] Figure 2 is a simplified flow chart according to an embodiment;

[0020] Figure 3 is a simplified diagram according to an embodiment;

[0021] Figure 4 is a simplified flow chart according to an embodiment;

[0022] Figure 5 is a simplified diagram according to an embodiment;

[0023] Figure 6 is a simplified diagram according to an embodiment;

[0024] Figure 7 is a simplified diagram according to an embodiment;

[0025] Figure 8 is a simplified diagram according to an embodiment;

[0026] Fig. 9 is a simplified diagram according to an embodiment;

[0027] Fig.10 is a simplified diagram according to an embodiment;

[0028] Fig.11 is a simplified diagram according to an embodiment;

[0029] Fig.12 is a simplified diagram according to an embodiment;

[0030] Fig.13 is a simplified diagram according to an embodiment;

[0031] Fig.14 is a simplified diagram according to an embodiment;

[0032] Fig.15 is a simplified diagram according to an embodiment;

[0033] Fig.16 is a simplified flow chart according to an embodiment;

[0034] Fig.17 is a simplified diagram according to an embodiment;

[0035] Fig.18 is a simplified diagram according to an embodiment;

[0036] Fig.19is a schematic diagram according to an embodiment;

[0037] Fig. 20 is a simplified diagram according to an embodiment;

[0038] Fig.21 is a simplified diagram according to an embodiment;

[0039] Fig. 22 is a simplified diagram according to an embodiment;

[0040] Fig.23 is a simplified diagram according to an embodiment; and

[0041] Fig.24 is a simplified diagram according to an embodiment. DETAILED DESCRIPTION

[0042] The following detailed description of example embodiments refers to the accompanying drawings.The same reference numbers in different drawings may identify the same or similar elements.

[0043] Figure 1 A simplified block diagram of a communication system 100 according to an embodiment of the present disclosure is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional transmission of data, the first terminal 103 may encode video data at a local location for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the encoded video data of the other terminal from the network 105, decode the encoded video data and display the restored video data. Unidirectional data transmission may be common in applications such as media services.

[0044] Figure 1 A second pair of terminals 101 and 104 are shown, which are provided to support bidirectional transmission of encoded video, such as may occur during a video conference. For bidirectional transmission of data, each terminal 101 and 104 can encode video data captured at a local location for transmission to the other terminal via a network 105. Each terminal 101 and 104 can also receive encoded video data sent by another terminal, decode the encoded data, and can display the recovered video data on a local display device.

[0045] exist Figure 1In the embodiment of the present invention, terminals 101, 102, 103 and 104 can be shown as servers, personal computers and smart phones, but the principles of the present invention are not limited thereto. Embodiments of the present invention are applicable to laptop computers, tablet computers, media players and / or dedicated video conferencing devices. Network 105 represents any number of networks that transmit encoded video data between terminals 101, 102, 103 and 104, including, for example, wired and / or wireless communication networks. Communication network 105 can exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purpose of this discussion, unless explained below, the architecture and topology of network 105 may be irrelevant to the operation of the present invention.

[0046] Figure 2 The placement of a video encoder and a video decoder in a streaming environment is shown as an example of application of the disclosed subject matter. The disclosed subject matter can be equally applicable to other video-enabled applications, including, for example, video conferencing, digital television, storing compressed video on digital media including CDs, DVDs, memory sticks, etc., and the like.

[0047] The streaming system may include a capture subsystem 203, which may include a video source 201, such as a digital camera, for creating, for example, an uncompressed video sample stream 213. The sample stream 213 may be emphasized as a high data volume when compared to an encoded video stream, and may be processed by an encoder 202 coupled to the camera 201. The encoder 202 may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. The encoded video stream 204 may be emphasized as a low data volume when compared to the sample stream, and may be stored on a streaming server 205 for future use. One or more streaming clients 212 and 207 may access the streaming server 205 to retrieve copies 208 and 206 of the encoded video stream 204. Client 212 may include a video decoder 211 that can decode an input copy of the encoded video stream 208, and the video decoder can create an output video sample stream 210 that can be rendered on a display 209 or other rendering device (not depicted). In some streaming systems, video streams 204, 206, and 208 may be encoded according to certain video encoding / compression standards. Examples of these standards are mentioned above and further described herein.

[0048] Figure 3 It may be a functional block diagram of a video decoder 300 according to an embodiment of the present invention.

[0049] Receiver 302 may receive one or more encoded video sequences to be decoded by decoder 300; in the same or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequence may be received from channel 301, which may be a hardware / software link to a storage device storing the encoded video data. Receiver 302 may receive the encoded video data as well as other data, for example, receiver 302 may receive encoded audio data and / or auxiliary data streams that may be forwarded to their respective use entities (not depicted). Receiver 302 may separate the encoded video sequence from the other data. To combat network jitter, a buffer memory 303 may be coupled between receiver 302 and entropy decoder / parser 304 (hereinafter referred to as "parser"). Buffer 303 may also not be needed, or may be made smaller, when receiver 302 receives data from a storage / forwarding device with sufficient bandwidth and controllability or from an isochronous network. For use over a best-effort packet network such as the Internet, a buffer 303 may also be desired, which may be relatively large and may advantageously be of adaptive size.

[0050] The video decoder 300 may include a parser 304 to reconstruct symbols 313 from an entropy-coded video sequence. The categories of these symbols include information for managing the operation of the decoder 300, and potential information for controlling a display device (e.g., a display 312) that is not part of the decoder but may be coupled to the decoder. The control information for the (one or more) display devices may be in the form of a Supplemental Enhancement Information (SEI message) or a Video Usability Information (VUI) parameter set fragment (not depicted). The parser 304 may parse / entropy decode the received coded video sequence. The encoding of the coded video sequence may be based on a video coding technique or standard, and may follow principles known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser 304 may extract a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. Subgroups may include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, Coding Units (CU), blocks, Transform Units (TU), Prediction Units (PU), etc. The entropy decoder / parser may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0051] The parser 304 may perform an entropy decoding / parsing operation on the video sequence received from the buffer 303, thereby creating the symbol 313. The parser 304 may receive the encoded data and selectively decode a specific symbol 313. In addition, the parser 304 may determine whether to provide the specific symbol 313 to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra prediction unit 307, or the loop filter 311.

[0052] Depending on the type of the coded video picture or a portion of the coded video picture (e.g., inter- and intra-pictures, inter- and intra-blocks) and other factors, the reconstruction of the symbol 313 may involve multiple different units. Which units are involved and how they are involved can be controlled by the parser 304 through subgroup control information parsed from the coded video sequence. For the sake of brevity, such subgroup control information flow between the parser 304 and the multiple units below is not described.

[0053] In addition to the functional blocks already mentioned, decoder 300 can be conceptually subdivided into several functional units as described below. In a practical implementation running under commercial constraints, many of these units interact closely with each other and can be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide into the functional units below.

[0054] The first unit is a sealer / inverse transform unit 305. The sealer / inverse transform unit 305 receives quantized transform coefficients from the parser 304 as well as control information including which transform to use, block size, quantization factor, quantization scaling matrix, etc. as symbols 313. The sealer / inverse transform unit 305 may output a block including sample values, which may be input into an aggregator 310.

[0055] In some cases, the output samples of the scaler / inverse transform unit 305 may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by the intra-picture prediction unit 307. In some cases, the intra-picture prediction unit 307 generates a block of the same size and shape as the block being reconstructed using surrounding reconstructed information extracted from the current (partially reconstructed) picture 309. In some cases, the aggregator 310 adds the prediction information generated by the intra-prediction unit 307 to the output sample information provided by the scaler / inverse transform unit 305 on a per-sample basis.

[0056] In other cases, the output samples of the sealer / inverse transform unit 305 may belong to an inter-coded and possibly motion compensated block. In this case, the motion compensated prediction unit 306 may access the reference picture memory 308 to extract samples for prediction. After the extracted samples are motion compensated according to the symbols 313 associated with the block, these samples may be added to the output of the sealer / inverse transform unit (in this case referred to as residual samples or residual signals) by the aggregator 310, thereby generating output sample information. The address within the reference picture memory from which the motion compensation unit extracts the predicted samples may be controlled by a motion vector, which is available to the motion compensation prediction unit in the form of a symbol 313, which may have, for example, an X component, a Y component, and a reference picture component. Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory when using sub-sample accurate motion vectors, motion vector prediction mechanisms, and the like.

[0057] The output samples of aggregator 310 may be subjected to various loop filtering techniques in loop filter unit 311. The video compression techniques may include in-loop filter techniques controlled by parameters included in the encoded video bitstream and available to loop filter unit 311 as symbols 313 from parser 304, and may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, and to previously reconstructed and loop filtered sample values.

[0058] The output of the loop filter unit 311 may be a sample stream that may be output to the display device 312 as well as stored in the reference picture memory 557 for future inter-picture prediction.

[0059] Certain coded pictures, once fully reconstructed, can be used as reference images for future prediction. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 304), current reference picture 309 can become part of reference picture buffer 308, and a new current picture memory can be reallocated before starting to reconstruct a subsequent coded picture.

[0060] The video decoder 300 can perform decoding operations according to a predetermined video compression technology recorded in a standard such as ITU-T Rec. H.265. The encoded video sequence can conform to the syntax specified by the video compression technology or standard used. In the sense that the encoded video sequence follows the syntax of the video compression technology or standard, the encoded video sequence can conform to the syntax specified in the video compression technology document or standard and in particular in the profile in the video compression technology document or standard. Compliance must also be that the complexity of the encoded video sequence is within the range defined by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, megasamples per second), the maximum reference picture size, etc. In some cases, the limits set by the level can be further defined by the Hypothetical Reference Decoder (HRD) specification and the metadata of the HRD buffer management written in the encoded video sequence.

[0061] In an embodiment, the receiver 302 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder 300 to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0062] Figure 4 It may be a functional block diagram of a video encoder 400 according to an embodiment of the present disclosure.

[0063] The encoder 400 may receive video samples from a video source 401 (which is not part of the encoder) that may capture video images to be encoded by the encoder 400 .

[0064] The video source 401 may provide a source video sequence in the form of a digital video sample stream to be encoded by the encoder (303), which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source 401 may be a storage device storing previously prepared videos. In a video conferencing system, the video source 401 may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that impart motion when viewed sequentially. The pictures themselves may be organized as a spatial pixel array, where each pixel may include one or more samples, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.

[0065] According to an embodiment, the encoder 400 can encode and compress the pictures of the source video sequence into an encoded video sequence 410 in real time or under any other time constraints required by the application. It is a function of the controller 402 to implement the appropriate encoding speed. The controller controls other functional units as described below and is functionally coupled to these units. For the sake of brevity, the coupling is not depicted. The parameters set by the controller may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology, etc.), picture size, picture group layout, maximum motion vector search range, etc. Those skilled in the art can easily identify other functions of the controller 402 because they may be related to the video encoder 400 optimized for a specific system design.

[0066] Some video encoders operate in what is readily recognizable to those skilled in the art as a "coding loop". As an overly simplified description, the coding loop may include an encoding portion of an encoder 402 (hereinafter referred to as a "source encoder") (responsible for creating symbols based on an input picture to be encoded and (one or more) reference pictures) and a (local) decoder 406 embedded in the encoder 400, which reconstructs the symbols to create sample data that will also be created by a (remote) decoder (because in the video compression techniques contemplated by the disclosed subject matter, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream is input into a reference picture memory 405. Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference picture buffer are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction portion of the encoder are exactly the same sample values ​​that the decoder will "see" when using prediction during decoding. This basic principle of reference picture synchronization (and the drift that results when synchronization cannot be maintained, for example due to channel errors) is well known to those skilled in the art.

[0067] The operation of the "local" decoder 406 can be combined with the above Figure 3 The operation of the "remote" decoder 300 described in detail is identical. However, reference is also briefly made to Figure 4 , when symbols are available and the entropy encoder 408 and the parser 304 are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the decoder 300, including the channel 301, the receiver 302, the buffer 303 and the parser 304, may not be fully implemented in the local decoder 406.

[0068] At this point it can be observed that any decoder techniques other than parsing / entropy decoding present in the decoder must also be present in a substantially identical functional form in the corresponding encoder. The description of encoder techniques can be simplified because encoder techniques are reciprocal to the fully described decoder techniques. A more detailed description is only required in certain areas and is provided below.

[0069] As part of the operation of source encoder 403, source encoder 403 may perform motion compensated predictive coding that predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as “reference frames.” In this manner, encoding engine 407 encodes the differences between pixel blocks of an input frame and pixel blocks of reference frame(s) that may be selected as prediction reference(s) for the input picture.

[0070] The local video decoder 406 can decode the encoded video data of the frame that can be designated as the reference frame based on the symbol created by the source encoder 403. The operation of the encoding engine 407 can advantageously be a lossy process. When the encoded video data can be decoded at the video decoder ( Figure 4 When decoded at a remote video decoder (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder 406 replicates the decoding process that may be performed by the video decoder on the reference frame and that may cause the reconstructed reference frame to be stored in the reference picture cache 405. In this way, the encoder 400 may locally store a copy of the reconstructed reference frame that has common content (absent transmission errors) with the reconstructed reference frame to be obtained by the remote video decoder.

[0071] The predictor 404 may perform a prediction search for the encoding engine 407. That is, for a new frame to be encoded, the predictor 404 may search the reference image memory 405 for sample data (as a candidate reference pixel block) or certain metadata such as reference picture motion vectors, block shapes, etc. that may serve as a suitable prediction reference for the new picture. The predictor 404 may operate pixel-by-pixel based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor 404, it may be determined that the input picture may have a prediction reference taken from a plurality of reference pictures stored in the reference picture memory 405.

[0072] The controller 402 may manage encoding operations of the video encoder 403 , including, for example, setting parameters and sub-group parameters for encoding video data.

[0073] The outputs of all the above functional units may be entropy encoded in the entropy encoder 408. The entropy encoder converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc.

[0074] Transmitter 409 may buffer the encoded video sequence created by entropy encoder 408 in preparation for transmission over communication channel 411, which may be a hardware / software link to a storage device where the encoded video data will be stored. Transmitter 409 may combine the encoded video data from video encoder 403 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0075] The controller 402 may manage the operation of the encoder 400. During encoding, the controller 405 may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following frame types.

[0076] Intra-pictures (I-pictures) can be pictures that can be encoded and decoded without using any other pictures in the sequence as prediction sources. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh Pictures. Those skilled in the art are aware of these variations of I-pictures and their corresponding applications and features.

[0077] A predictive picture (P picture), which may be a picture that can be encoded and decoded using intra prediction or inter prediction, uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0078] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, use up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-directional predictive pictures can use more than two reference pictures and associated metadata for reconstructing a single block.

[0079] The source picture is typically spatially subdivided into blocks of samples (e.g., blocks of 4×4 samples, blocks of 8×8 samples, blocks of 4×8 samples, or blocks of 16×16 samples) and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or may be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be non-predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be non-predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.

[0080] The video encoder 400 may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T Rec. H.265. In the operation of the video encoder 400, the video encoder 400 may perform various compression operations, including predictive encoding operations that exploit temporal redundancy and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0081] In an embodiment, the transmitter 409 may transmit additional data along with the encoded video. The source encoder 403 may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set segments, etc.

[0082] According to an embodiment, the general framework of NIC is described as follows: Given an input image x, the goal of NIC is to use the image x as the input of the DNN encoder to compute a compressed representation Should It is compact for storage and transmission. Then, Used as input to the DNN decoder to reconstruct the image Other NIC methods adopt a variational autoencoder (VAE) structure, where the DNN encoder directly uses the entire image x as its input, and it computes the output representation through a set of network layers that work like a black box. Accordingly, the DNN decoder converts the entire representation As its input, this input passes through another set of network layers, which works like another black box to compute the reconstructed The rate-distortion (RD) loss is optimized to achieve the distortion loss of the reconstructed image With compact representation The trade-off between the bit consumption R and the trade-off hyperparameter is λ:

[0083]

[0084] Figure 5 A general architecture 500 of a NIC according to an exemplary embodiment is shown. The encoder includes a set of visual modules (such as convolutional networks or visual transformers) with downsampling functions; while the decoder includes a set of visual modules (such as convolutional networks or visual transformers) with upsampling functions.

[0085] Figure 6 An example 600 of a detailed structure of a decoder with activation functions 601, 602, and 603 according to an exemplary embodiment is shown. Typically, a set of upsampling methods is followed by an activation function. The numbers on top of the upsampling modules indicate the order of the activation functions. In some embodiments, cropping is ignored or removed. The function of the upsampling module is to increase the width and height of the input tensor. Typically, in NIC, upsampling is 2 times.

[0086] In this embodiment, the position of the attention model 612 is after the third upsampling module 611. In another embodiment, the position of the attention model can be before the first upsampling. In other embodiments, the position of the attention model can be after the first / second upsampling. The attention model can also be removed from the general architecture. The activation function increases the nonlinearity of the network.

[0087] Key aspects of embodiments include efficient activation in the decoder. Figure 7 An example 700 of a detailed activation function module, such as any one of activation functions 601 , 602 , and 603 , is shown.

[0088] All variances of the activation function include a convolution layer with a 1x1 convolution kernel and a leakyReLu function. It is worth noting that in another embodiment, Relu can be used instead of leakyRelu. The ReLU formula is: f(x) = max(0, x). The LeakyReLU formula is: f(x) = max(0.01*x, x).

[0089] The example activation function 701 is any one or more of the activation functions 601, 602, and 603 according to an embodiment, and the example activation function 701 shows an embodiment in which the input is first subjected to a leakyRelu operation, then a convolution 1x1 operation, and another leakyRelu operation. The output of this set of operations is then multiplied by the input and then added back to the input.

[0090] The example activation function 702 is any one or more of the activation functions 601, 602, and 603 according to an embodiment, and the example activation function 702 shows an embodiment in which the input is first subjected to a leakyRelu operation and then to a convolution 1x1 operation. The output of this set of operations is then multiplied by the input and then added back to the input.

[0091] The example activation function 703 is any one or more of the activation functions 601, 602 and 603 according to an embodiment, and the example activation function 702 shows an embodiment in which the input is first subjected to a leakyRelu operation and then to a convolution 1x1 operation. The output of this set of operations is added back to the input.

[0092] The upsampling module 611 or any upsampling module in example 600 can be a transposed convolution layer or a pixel rearrangement, or any trainable upsampling method. If it includes a convolution layer or a transposed convolution layer, its embodiments may not constrain the details of these layers. For example, in one embodiment, the filter size of the convolution network (or transposed convolution) is 3×3; in another preferred embodiment, the filter size of the convolution network (or transposed convolution) is 5×5; in another preferred embodiment, the filter size of the convolution network (or transposed convolution) is 7×7.

[0093] Therefore, the embodiments of this invention provide a network structure of a decoder in network compression. Figure 6 Under the structure shown, (from Figure 7 The upsampling method and activation function can be any combination. Compared with the traditional module in neural image compression, the embodiment of this paper achieves a reasonable compression rate while greatly reducing the complexity of the image compression model.

[0094] JPEG AI VM has a synthetic transform network with an attention module (RNAB). Although JPEG AI VM can provide better RD performance, it also brings high complexity to the synthetic network in the decoder. This paper describes the simplification of the attention module in the synthetic transform network in VM2.0. A transformer-based attention module is proposed to replace the current Residual Non-local Attention Block (RNAB) module. The transformer-based attention module can reduce the computational complexity (i.e., Kilo Multiply-Accumulates (kmac)) and maintain comparable coding performance. It is reported that the computational complexity on the decoder side can be reduced by more than 80%, while the performance loss is about 2%.

[0095] JPEG AI VM has a synthetic transform network with an attention module (RNAB). Although JPEG AI VM can provide better RD performance, it also brings high complexity to the synthetic network in the encoder and decoder. The main purpose of this CE is to simplify the attention module in the synthetic transform network in VM2.0 through the proposed scheme.

[0096] The synthetic transform network in the decoder of the JPEG AI VM, such as Example 700 or Figure 8 Example 800 in has four consecutive convolutional layers 801, 802, 803 and 804, as shown in Figure 8 As shown in , these convolutional layers have a downsampling stride of 2 and include an attention module. The attention module in the synthetic transformer network is the residual non-local attention block 811 (RNAB, as shown in Fig. 9 9).

[0097] For RNAB 811, it is located between the first downsampling convolution layer and the second downsampling convolution layer, and plays the role of an attention mechanism. RNAB includes a downsampling convolution layer and an upsampling convolution layer, a series of residual blocks (ResidualBlock (RB), such as Fig.10 As shown in Example 1000 in FIG. 1 , and a padding layer. Since RNAB introduces a high computational complexity, the embodiment of this invention simplifies the attention module to reduce the complexity of the synthetic transformation network.

[0098] For example, see Fig.11 Example 1100, which shows that the embodiment has replaced the RNAB module with Fig.11 The following transformer-based blocks are shown. Specifically, example 1100 includes two modules. Module 502 first normalizes the input and uses a 1x1 convolution layer together with a 3x3 convolution layer to generate Q / K / V to calculate the attention score. After that, module 502 multiplies the attention score by V and performs a reshaping operation. Finally, the output passes through a 1x1 convolution layer and is added to the input to obtain the output of module #1. Module 501 further processes the output of module 502 through two parallel branches including a 1×1 convolution layer and a 3×3 convolution layer. Then, module 501 multiplies the results and adds them to the input to obtain the output of module 501. The proposed attention block is optionally used in conjunction with a downsampling block and an upsampling block. Compared with RNAB 811, by replacing RNAB 811 with example 1100, the computational complexity is reduced because there are fewer convolution operations in the attention block.

[0099] All tests are performed according to CTC and a configuration file is provided for tool shutdown testing. The basic description of the method according to the embodiment is as follows Fig.12 As shown in Example 1200.

[0100] According to an example embodiment, after each convolution operation 801 , 802 , and 803 , the RNAB block 811 is removed from the decoder and a transformer based block 1201 , 1202 , and 1203 is placed.

[0101] According to other example embodiments, after each convolution operation 801 , 802 and 803 and the integration of downsampling and upsampling, the RNAB block 811 is removed from the decoder and a transformer based block 1201 , 1202 and 1203 is placed.

[0102] According to other example embodiments, after each convolution operation 801, 802 and 803 and integration of channel-wise splitting (i.e., splitting the input into two parts and passing through two transformer-based blocks and then connecting the outputs), the RNAB block 811 is removed from the decoder and two transformer-based blocks 1201, 1202 and 1203 are placed.

[0103] According to other example embodiments, after each convolution operation 801 , 802 and 803 , the RNAB block 811 is removed from the decoder and two transformer based blocks 1201 , 1202 and 1203 are placed.

[0104] exist Fig.13 The results for the "Tools Off" configuration on the A100 are shown in Table 1300. The results show that kMAC / pxl can be reduced from 833 to 215, 210, 179, and 143, respectively, compared to the VM 2.0 baseline, with a loss of 0.8%, 0.1%, 2.1%, and 2.2%, respectively.

[0105] Through these embodiments, the attention module (RNAB) in the synthetic network can be replaced with one of the simplified schemes with a transformer-based attention module, which can reduce the computational complexity on the decoder side by more than 80% with a performance loss of about 2%.

[0106] Furthermore, an objective of an embodiment relates to identifying the potential for further simplifying the lightweight model in CE 3.8 by using 1) a simplified activation module, 2) a simplified synthesis transformation network, and 3) a simplified super decoder network.

[0107] Fig.14 Example 1400 shows the activation function used in the synthetic transformation network of the decoder of the lightweight platform. And Fig.15 Example 1500 shows an activation function architecture simplified by removing the hyperbolic tangent function (tanh) according to an exemplary embodiment.

[0108] Fig.16 Example 1600 shows a super decoder network of a decoder 1601 of a lightweight platform according to an embodiment. As observed, there is a 1×1 convolution at the beginning of the super decoder network. However, the efficiency of 1×1 convolution is very low in mobile devices and will introduce additional complexity, so the embodiments of this article further simplify the super decoder network by removing the 1×1 convolution at the beginning of the super decoder. For example, see example 1602.

[0109] In addition, as observed, there are 3×3 convolutions after the second deconvolution of the synthesis transformation network for the Y component and the UV component. And for the YUV component, the embodiments herein simplify the synthesis transformation network by applying an asymmetric synthesis transformation network, that is, removing the 3×3 convolution after the second deconvolution of the UV component, while keeping the Y component as a default value to reduce complexity, as shown in the structure 1603 of the synthesis transformation network of the lightweight platform according to the exemplary embodiment.

[0110] Fig.17 Table 1700 shows a diagram such as Figure 14 to Figure 15 Comparison results of removing the activation function tanh, where the inference time of decoder Y and decoder UV on mobile devices can be reduced by 1.3% and 1.0% on NPU respectively. The total inference time on mobile devices can be reduced by 1.2% on NPU.

[0111] Fig.18 Table 1801 of Example 1800 shows comparison results of a simplified super-decoder network such as from Example 1602, where KMAC can be reduced by 0.03 and the inference time of super-decoder Y and super-decoder UV on the mobile device can be reduced by 0.78% and 6.56% on the NPU, respectively.

[0112] Fig.18 Table 1802 shows comparison results of a simplified synthetic transformation network such as from Example 1603, where KMAC can be reduced by 0.5, the inference time of the decoder UV on the mobile device can be reduced by 0.8% on the NPU, and the total inference time on the mobile device can be reduced by about 0.1% on the NPU.

[0113] Therefore, in view of these advantages, embodiments may 1) replace the activation function in the synthesis network with one of the above-mentioned activation function architectures, 2) replace the synthesis transformation network with a simplified synthesis transformation network, and 3) replace the super decoder network with a simplified super decoder network.

[0114] Learning-based image coding schemes, such as JPEG AI, have shown potential by greatly outperforming traditional image compression standards in terms of rate-distortion (RD) performance. However, their widespread application is hindered by high decoding complexity, especially the complexity of the upsampling module and the attention module. Existing work has attempted to reduce this complexity, but their solutions are not completely effective, leaving considerable complexity unresolved. Therefore, the embodiments of this article propose a simplified transform network architecture that uses an optimized attention module, a simplified upsampling module, and a streamlined activation function to solve this problem. Simulation results show that the complexity of the simplified decoder (measured in kMAC / pixel) is reduced by 80% (from 833 to 172), while the gain relative to Versatile Video Coding (VVC) is slightly increased (from 27.3% to 27.5%). Some of the methods described in this article can be integrated into the JPEG AI Verification Model (VM) software.

[0115] Machine learning has brought breakthroughs in various fields, especially in neural network-based image compression JPEG AI, a standard developed by JPEG that represents a major advancement in the field. JPEG AI is designed for both human visualization and computer vision tasks.

[0116] Despite the above progress, one of the main challenges of the verification model of JPEG AI (called JPEG AI VM) is its high complexity, especially the transform network in the decoder. This complexity leads to higher computational cost, reducing the practicality and efficiency of the standard. Although there have been some efforts to reduce the complexity of the decoder, such as simplifying the context model entropy decoding process and exploring sparse network structure techniques, little attention has been paid to the design of the transform network architecture in the decoder, and most existing solutions lead to performance degradation.

[0117] Therefore, the embodiments of this article address this challenge by providing three simplifications to the transformer network: 1) an improved attention module, 2) a new upsampling module, and 3) a simplified activation function. These improvements reduce the computational load and improve efficiency without affecting system performance. Specifically, a transformer-based attention module is introduced, which has a layered architecture to reduce complexity. The embodiment adopts a recovery transformer block, which applies a self-attention mechanism between channels instead of applying a self-attention mechanism in the spatial dimension, thereby effectively capturing global information and reducing complexity. A pixel rearrangement operation with a pre-1×1 convolution is also adopted as an upsampling scheme. Such a design reduces the computational complexity while maintaining network performance. In addition, a simpler activation function is provided, which does not have a hyperbolic tangent function and its subsequent element-by-element operation to further reduce the computational complexity.

[0118] Fig.19 A comparative analysis of details in the reconstructed image is provided. The top panel 1901 depicts the original, uncompressed image, the middle panel 1902 shows the reconstructed image after compression via JPEG AI VM, and the lower panel 1903 shows the reconstructed image achieved via a JPEG AI-based scheme according to an embodiment of the present invention. The visual comparison demonstrates the ability of the embodiments of the present invention to reduce KMAC while reducing bits per pixel (bpp) and enhancing image quality.

[0119] exist Fig. 20 The structure of the JPEG AI transform network 2001 is described in Example 200 of . The embodiment focuses on the decoder transformer network because the decoder transformer network is critical to the decoding complexity. The decoder transformer network includes two residual blocks (RBs) at the beginning of the network and a sequence of 4 convolutions with an upsampling step of 2 to increase the resolution of the output. In addition, the decoder transformer network includes a residual attention unit (Residual Attention Unit, ResAU) as an activation function and a residual non-local attention module (Residual Non-local Attention Block, RNAB) as an attention module to improve non-linear representation capabilities. Fig. 20The architecture of RNAB 2002, ResAU 2003 and RB 2004 is shown in FIG. The embodiment focuses on the decoder transformer network because the decoder transformer network is crucial to the decoding complexity. As can be seen, the JPEG AI transform network 2001 contains two RBs at the beginning of the network and a sequence of 4 convolutions with an upsampling step of 2 to increase the resolution of the output. In addition, the PEGAI transform network 2001 contains ResAU as an activation function and RNAB as an attention module to improve the nonlinear representation capability.

[0120] While achieving promising performance, JPEG AI also brings high computational complexity to the transform network decoder side. Specifically, the current transform network on the decoder side has a complexity of 833kMAC / pixel, which is computationally expensive. Specifically, among all components, RNAB and upsampling modules have the highest and second highest complexity, accounting for 95% of the complexity in total. To this end, an embodiment includes two low-complexity transformer-based attention modules and a pixel-reordering-based upsampling module to replace the current RNAB module and upsampling module, thereby reducing the complexity while maintaining comparable performance.

[0121] like Fig.21 As shown in example 2100 of , an embodiment provides a transformer-based attention module to collect global information to maintain performance and a hierarchical attention architecture to achieve low complexity. Fig.21 Architecture 2100 is shown in FIG.

[0122] The transformer architecture 2100 according to the embodiment has demonstrated its advantages over convolution in various visual tasks due to its ability to capture global information. However, the memory usage of the original visual transformer is very large, especially for high-resolution images. Secondly, the complexity increases quadratically with the increase of input width / height), resulting in higher kMAC / pixel. Some use Swin Transformer, which introduces a moving window-based attention mechanism, to reduce complexity. However, this cannot collect information of high-resolution images at a long distance and may cause performance degradation.

[0123] To address the above two challenges, the embodiment adopts a recovery transformer block (e.g., recovery transformer block 2101, recovery transformer block 2102, recovery transformer block 2013) as a basic module in such embodiments of the present invention. The recovery transformer block applies a self-attention mechanism between channels instead of applying a self-attention mechanism in the spatial dimension, that is, the time and memory complexity of the key-value query dot product grows linearly with the spatial resolution of the input. Fig. 22 The detailed structure of the recovery transformer block is depicted, which includes the following two sub-blocks.

[0124] There is a Multi-Dconv Head Transposed Attention (MDTA) block 2201. The MDTA block is used to generate a global attention map. Assume that the input is X. The embodiment first passes X to the layer normalization operator and obtains the normalized value. Then, the query (Q), key (K), and value (V) are obtained by using the normalized values ​​obtained by three sets of 1×1 convolutions and 3×3 depth-separable convolutions, respectively. This process can encode channel space context. Finally, we can obtain the transposed attention map by calculating the dot product of the reshaped query and key.

[0125] There is a Gated-Dconv Feed-Forward Network (GDFN) block 2202. GDFN expands the feature channel to make it the same as the input dimension. In general, the recovery transformer block first passes the input to the MDTA to collect features with contextual information, and then feeds the intermediate results back to the GDFN to control the information flow. The embodiment also designs the following two methods to optimize peak memory usage.

[0126] There is also channel splitting. That is, after the third convolution, it is shown that the output is split into two parts between channels and the two parts are passed to two recovery transformer blocks respectively.

[0127] There is also spatial down-sampling. Assume that the output after the third convolution is X∈R C ×H×W An embodiment may first down-convert X before inputting it into the transformer block, i.e., the shape of the intermediate output is C×(H / 2)×(W / 2). An embodiment then up-converts the output to keep the resolution unchanged.

[0128] There are also low-complexity upsampling modules. Fig. 20As shown in the JPEG AI Transform Network 2001 in Figure 2, the decoder structure includes four upsampling modules. The upsampling process of JPEG AI VM2.0 uses transposed convolutions with a stride of 2, where the kernel size is set to 3. This method is known for its high coding complexity. A closer look at the specifics of the upsampling modules in VM2.0 shows that their complexity increases from left to right. The computational cost (measured in KMAC / pixel) associated with the four upsampling modules is as follows: 5, 15, 60, and 2, respectively. This increasing trend in complexity corresponds to the growth in size of the feature maps as they pass through these modules. This relationship peaks in the third upsampling module, which exhibits the highest complexity due to the increase in feature map size. Interestingly, the last upsampling module exhibits a relatively low computational cost despite having the largest input size. This reduction is due to the fact that the last upsampling module only outputs three layers, limiting its overall complexity.

[0129] Essentially, the pixel shuffling operation rearranges the elements in the tensor so that the spatial resolution (height and width) of the tensor increases, while the depth (or number of channels) decreases. Given an input feature map F, its number of channels C = n*r 2 , the spatial resolution is H×W, the pixels are rearranged and the elements of F are reorganized to form a new feature map F'', the depth (or number of channels) of the new feature map is C=n, but the spatial resolution is rH×rW. Where r is an integer upsampling factor.

[0130] In practice, a pixel shuffling operation is often preceded by a standard convolution operation to increase the number of channels in the input tensor. This allows the pixel shuffling operation to properly redistribute the pixels to achieve the desired upsampling effect. When the filter size is 1×1, the complexity of this upsampling module (convolutional layer with filter 1×1 and pixel shuffling) is reduced compared to the transposed convolution with 3×3 filter. By replacing the third upsampling module from the right (which is the most computationally expensive in terms of KMAC / pixel), we can significantly reduce the complexity of the decoder.

[0131] There is also a low-complexity activation function module according to the embodiments of this article. There is an activation function module that includes a collection of several sub-modules, including LeakyReLU, 1×1 convolutional layer, and hyperbolic tangent. In addition, the activation function module also involves some complex operations, such as element-by-element multiplication. The complete module structure is shown in Fig. 20RESAU 2003. While many of these operations are considered basic components in a neural network, the embodiments herein are based on the identification of certain redundancies that, although not explicitly quantifiable in terms of KMAC / pixel, do add unnecessary complexity to the system. Therefore, the embodiments include a simplified model that simplifies the structure of the activation function module. Specifically, there is herein a removal of the hyperbolic tangent activation function and subsequent element-by-element operations according to the embodiments.

[0132] The embodiments herein have been compared with the JPEG AI common training and testing conditions defined on an Nvidia A100 GPU device. 0.06, 0.12, 0.25, 0.50 and 0.75 bitrates are mandatory for BD rate calculations. Objective quality metrics include MS-SSIM, FSIM, NLPD, IWSSIM, VMAF, VIF and PSNR-HVS-M between compressed and original images at these target bitrates. The average Bjorntgaard Difference Rate (MS-SSIM, FSIM, NLPD, IW-SSIM, VMAF, VIFP and PSNR-HVS-M) of the above seven metrics (MS-SSIM, FSIM, NLPD, IW-SSIM, VMAF, VIFP and PSNR-HVS-M) as our main distortion metrics Delta-rate, BD-rate) rate is used to measure image compression quality. Together, these seven metrics provide a comprehensive and accurate assessment of image quality, taking into account aspects of perceptual similarity, naturalness, and fidelity. To measure the complexity of the model, the widely used ptflops library is used to calculate kMAC / pixel. This allows us to measure the computational efficiency of the algorithm.

[0133] The embodiments mainly demonstrate and evaluate simplified alternatives to specific JPEG AI VM 2.0 components, which are used as our baseline. In Section IV-B, the embodiments evaluate the multi-level attention architecture with two variations, channel splitting and spatial downsampling strategies. The following description provides the results of the study of the effects of simplified upsampling modules, simplified activation functions (further reducing complexity), and combined testing of all proposed simplifications into a single model for evaluation.

[0134] Embodiments are evaluated by comparing simplified attention modules with channel splitting (Attention-CS) and simplified attention modules with spatial down sampling (Attention-DS) after the third convolutional layer to optimize peak memory usage. For simplified attention modules-CS, γ is set to 2.66 for the restoration transformer blocks after all convolutional layers. For simplified attention modules-DS, γ is set to 4, 4, and 2.66 for the restoration transformer blocks after the first convolutional layer, the second convolutional layer, and the third convolutional layer, respectively.

[0135] Fig.23 The results of these experiments are shown in Example Table 2300 in , and contain the following observations. First, Attention-CS and simplified attention module-DS can reduce decoding complexity (up to 75%) with little performance degradation (up to 0.8%). This demonstrates the effectiveness of the proposed multi-stage transformer-based attention architecture. Second, Attention-CS outperforms Attention-DS in both BD rate and decoding complexity. The reason may be the high resolution after the third convolution, and the spatial information is more important than the channel spatial information for the final performance. This suggests that the channel segmentation method is more suitable for the JPEG AI codec.

[0136] In Table 2300, negative values ​​mean encoding performance (lower is better), and for two trained models with the same initialization, the performance deviation is as high as 0.5%, which is caused by the randomness introduced in training and the calculations within the activation function cannot be explicitly quantified in kMAC / pixel.

[0137] The embodiment can also replace the third upsampling module from the right with our proposed simplified upsampling method. Specifically, compared with the original upsampling module (the baseline in VVC), the simplified upsampling method above reduces 39kMAC per pixel. Assuming that the embodiment ignores the RNAB module, in this case, this reduction is equivalent to about 43% of the computational cost.

[0138] When examining the overall effect of the simplified upsampling module, we observe an average BD rate reduction of 0.2%. Given the nature of deep learning models, especially when focusing on complex tasks such as JPEG AI, there is an inherent uncertainty in each training iteration. This uncertainty, or training error, typically varies around 0.5% for each JPEG AI training cycle. Therefore, when we drop the BD rate by 0.2% in this case, it will be well within the typical training error range. This leads to an important conclusion: the small loss incurred by implementing the simplified upsampling module can be considered negligible. Overall, our proposed simplified upsampling method provides substantial computational savings without compromising the rate-distortion efficiency of the model.

[0139] As depicted in example table 2300, by introducing a simplified activation function in the transform network, a coding gain loss of approximately 0.1% was observed. This small loss is essentially negligible given the inherent variability in the training process. This loss will be well within the bounds of the expected training error and will not affect the overall performance of the model according to the example embodiments herein.

[0140] Through such experiments, it is noted that the complexity of complex activation functions cannot be explicitly quantified in terms of KMAC / pixel by ptflops. However, it brings unnecessary complexity to the system, so this redundancy can be removed by the simplified scheme of the embodiments of this article.

[0141] In addition, combined tests were performed to evaluate the general applicability of each simplification method (simplified attention, upsampling, and activation) according to the embodiments in this article. Fig.23 The results shown in the example table 2300 show that the simplified module herein has remarkable robustness and effectiveness. The computational complexity with respect to kMAC / pixel is reduced from 833 to 172.1. Furthermore, the embodiments herein manage to achieve a coding gain of 0.2%.

[0142] The observations highlight the value of these simplifications. Each simplification contributes to a more computationally efficient model without degrading overall performance. Furthermore, combined testing again demonstrates the robustness of these simplifications with other modules, reinforcing their broad applicability and effectiveness in a variety of scenarios.

[0143] Therefore, the embodiments of this article have solved the high decoding complexity inherent in learning-based image coding schemes. Although learning-based image coding schemes have excellent rate-distortion performance, their practical applications are limited due to their high complexity. Specifically, a series of modifications to the decoder architecture of the JPEG AI standard are proposed, namely, simplification of the attention module, upsampling module and activation function. The modified attention module utilizes a low-complexity hierarchical attention structure, a transformer-based module to collect better global information and a novel memory optimization strategy to ensure feasible memory usage. The upsampling module combines pixel reordering techniques to reduce complexity without compromising network stability. The activation function has been simplified by removing irrelevant operations and retaining only necessary components, further reducing the complexity.

[0144] Modifications according to the embodiments of this paper reduce computational complexity, with the kMAC / pixel metric reduced from 852 to 180. Importantly, this improvement in computational efficiency also leads to a slight increase in VVC gain, from 27.3% to 27.5%. These findings balance the complexity and performance of learning-based image coding.

[0145] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media, or implemented by one or more specially configured hardware processors. Fig.24 A computer system 2400 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0146] Computer software may be encoded using any suitable machine code or computer language, which may be assembled, compiled, linked or similarly constructed to create code comprising instructions that may be directly executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., or executed through interpretation, microcode, etc.

[0147] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, etc.

[0148] Fig.24 The components of the computer system 2400 shown in the example are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement related to any one component or combination of components shown in the exemplary embodiment of the computer system 2400.

[0149] Computer system 2400 may include certain human interface input devices. Such human interface input devices may be responsive to input from one or more human users, for example, through: tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not depicted). Human interface devices may also be used to capture certain media that are not necessarily directly related to a person's conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), video (e.g., two-dimensional video, three-dimensional video including stereoscopic video), etc.

[0150] The input human-machine interface devices may include one or more of the following (only one of each is shown): keyboard 2401 , mouse 2402 , touch pad 2403 , touch screen 2410 , joystick 2405 , microphone 2406 , scanner 2408 , camera 2407 .

[0151] The computer system 2400 may also include certain human-machine interface output devices. Such human-machine interface output devices may, for example, stimulate one or more human user senses through tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen 2410 or a joystick 2405, but may also be a tactile feedback device that is not an input device), audio output devices (e.g., speakers 2409, headphones (not shown)), visual output devices (e.g., screens 2410 including cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, organic light-emitting diode (OLED) screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities, some of which are capable of outputting two-dimensional visual outputs or outputs of more than three dimensions through devices such as stereoscopic picture output, virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

[0152] The computer system 2400 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 2420 with CD / DVD 2411 or similar media, thumb drive 2422, removable hard drive or solid state drive 2423, traditional magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices such as security software dogs (not shown), and the like.

[0153] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0154] The computer system 2400 may also include an interface 2499 to one or more communication networks 2498. The network 2498 may be, for example, a wireless network, a wired network, an optical network. The network 2498 may also be a local network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a delay-tolerant network, and the like. Examples of the network 2498 include a local area network such as Ethernet, a wireless LAN, a cellular network including Global System for Mobile communications (GSM), 3G, 4G, 5G, Long-Term Evolution (LTE), etc., a television wired or wireless wide area digital network including cable television, satellite television, and terrestrial broadcast television, a vehicle and industrial network including a CAN bus, and the like. Some networks 2498 typically require external network interface adapters (e.g., USB ports of computer system 2400) connected to some general data ports or peripheral buses (2450 and 2451); other network interfaces are typically integrated into the kernel of computer system 2400 by connecting to the system bus described below (e.g., connecting to an Ethernet interface in a PC computer system or connecting to a cellular network interface in a smartphone computer system). Computer system 2400 can use any of these networks 2498 to communicate with other entities. Such communications can be one-way receive only (e.g., broadcast television), one-way send only (e.g., CANBus to certain CANBus devices), or two-way, such as connecting to other computer systems using a local area network or wide area network digital network. As described above, certain protocols and protocol stacks can be used on each of those networks and network interfaces.

[0155] The above-mentioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the kernel 2440 of the computer system 2400 .

[0156] The core 2440 may include one or more central processing units (CPUs) 2441, graphics processing units (GPUs) 2442, graphics adapters 2417, dedicated programmable processing units (FPGAs) in the form of field programmable gate areas (FPGAs) 2443, hardware accelerators 2444 for certain tasks, and the like. These devices, together with read-only memory (ROM) 2445, random access memory 2446, and internal mass storage 2447 such as internal non-user accessible hard disk drives, SSDs, and the like, may be connected via a system bus 2448. In some computer systems, the system bus 2448 may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, and the like. Peripheral devices may be connected directly to the core's system bus 2448, or may be connected to the core's system bus 2448 via a peripheral device bus 2449. The architecture of the peripheral bus includes PCI, USB, and the like.

[0157] The CPU 2441, GPU 2442, FPGA 2443, and accelerator 2444 may execute certain instructions, which may be combined to form the above-mentioned computer code. The computer code may be stored in ROM 2445 or RAM 2446. Transitional data may also be stored in RAM 2446, while permanent data may be stored, for example, in internal mass storage 2447. Fast storage and retrieval to any storage device may be achieved by using a cache, which may be closely associated with one or more CPUs 2441, GPU 2442, mass storage 2447, ROM 2445, RAM 2446, etc.

[0158] The computer readable medium may have thereon computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or the medium and computer code may be of a type well known and available to those skilled in the art of computer software.

[0159] As an example and not limitation, a computer system having architecture 2400, particularly kernel 2440, can provide functionality due to the execution of software contained in one or more tangible computer-readable media by (one or more) processors (including CPU, GPU, FPGA, accelerator, etc.). Such computer-readable media can be media associated with a user-accessible mass storage as described above, as well as some non-transitory memory of kernel 2440, such as kernel internal mass storage 2447 or ROM 2445. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by kernel 2440. Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can enable kernel 2440, particularly the processors therein (including CPU, GPU, FPGA, etc.) to perform specific processes or specific parts of specific processes described in the present disclosure, including defining data structures stored in RAM 2446 and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system provides functionality due to logic hardwired or otherwise embodied in a circuit (e.g., accelerator 2444), which can replace software or run in conjunction with software to perform a specific process or a specific portion of a specific process described herein. Where appropriate, a portion referring to software may include logic, and vice versa. Where appropriate, a portion referring to a computer-readable medium may include a circuit (e.g., an integrated circuit (IC)) storing software for execution, a circuit embodying logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0160] Although the present disclosure has described a number of exemplary embodiments, there are modifications, permutations, and various replacement equivalents that fall within the scope of the present disclosure. It should therefore be understood that those skilled in the art will be able to design a variety of systems and methods that, although not explicitly shown or described in the present disclosure, embody the principles of the present disclosure and therefore fall within the spirit and scope of the present disclosure.

Claims

1. A method for video decoding, performed by at least one processor, the method comprising: Receiving a video stream including a current block in a current picture; as well as reconstructing the current block by transforming the current block through a neural network including a plurality of upsampling modules and an activation module, Among them, at least one activation module of the activation modules includes a LeakyReLu function and a convolution function.

2. The method according to claim 1, characterized in that In at least one of the activation modules, the output of the LeakyReLu function is the input of the convolution function.

3. The method according to claim 2, characterized in that In the at least one of the activation modules, the output of the convolution function is an input to the multiplication function of the at least one of the activation modules.

4. The method according to claim 3, characterized in that In the at least one activation module of the activation modules, the output of the multiplication function is the input of the addition function of the at least one activation module of the activation modules.

5. The method according to claim 2, characterized in that: In the at least one of the activation modules, the output of the convolution function is an input to the addition function of the at least one of the activation modules.

6. The method according to claim 2, characterized in that In the at least one activation module of the activation modules, the output of the convolution function is the input of a second LeakyReLu function of the at least one activation module of the activation modules.

7. The method according to claim 6, characterized in that In at least one of the activation modules, the output of the second LeakyReLu function is an input to a multiplication function of the at least one of the activation modules.

8. The method according to claim 7, characterized in that In the at least one activation module of the activation modules, the output of the multiplication function is the input of the addition function of the at least one activation module of the activation modules.

9. The method according to claim 2, characterized in that: The convolution function includes a 1x1 convolution kernel.

10. The method according to claim 1, characterized in that At least one of the upsampling modules includes a pixel rearrangement layer.

11. An apparatus comprising: at least one memory configured to store computer program code; as well as At least one processor is configured to access the computer program code and operate according to the instructions of the computer program code, wherein the computer program code comprises: A receiving code configured to cause the at least one processor to receive a video code stream including a current block in a current picture; and reconstruction code configured to cause the at least one processor to reconstruct the current block by transforming the current block through a neural network including a plurality of upsampling modules and activation modules, Among them, at least one activation module of the activation modules includes a LeakyReLu function and a convolution function.

12. The device according to claim 11, characterized in that In at least one of the activation modules, the output of the LeakyReLu function is the input of the convolution function.

13. The device according to claim 12, characterized in that In the at least one of the activation modules, the output of the convolution function is an input to the multiplication function of the at least one of the activation modules.

14. The device according to claim 13, characterized in that In the at least one activation module of the activation modules, the output of the multiplication function is the input of the addition function of the at least one activation module of the activation modules.

15. The device according to claim 12, characterized in that In the at least one of the activation modules, the output of the convolution function is an input to the addition function of the at least one of the activation modules.

16. The device according to claim 12, characterized in that In the at least one activation module of the activation modules, the output of the convolution function is the input of a second LeakyReLu function of the at least one activation module of the activation modules.

17. The device according to claim 16, characterized in that In at least one of the activation modules, the output of the second LeakyReLu function is an input to a multiplication function of the at least one of the activation modules.

18. The device according to claim 17, characterized in that In the at least one activation module of the activation modules, the output of the multiplication function is the input of the addition function of the at least one activation module of the activation modules.

19. The device according to claim 12, characterized in that The convolution function includes a 1x1 convolution kernel.

20. A non-transitory computer-readable medium storing a program, wherein the program causes a computer to perform the following operations: Receiving a video stream including a current block in a current picture; and reconstructing the current block by transforming the current block through a neural network including a plurality of upsampling modules and an activation module, in, At least one activation module of the activation modules includes a LeakyReLu function and a convolution function.

Citation Information

Patent Citations

  • Efficient activation function in neural network image compression decoder

    US20240333942A1

Cited By

  • Method, apparatus, and medium for feature coding

    WO2026184650A1