Efficient up-sampling method in neural network image compression decoder
By simplifying the transformation network architecture, including the optimization attention module and the upsampling module, the high decoding complexity problem of image compression models based on neural networks is solved, efficient image compression is achieved and computing efficiency is improved.
Patent Information
- Application Number
- CN202480001463.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-03-26
- Filing Date
- 2024-03-27
- Publication Date
- 2024-11-29
AI Technical Summary
The existing neural network-based image compression model has high decoding complexity problems, resulting in high computational cost, practicality and efficiency, and the performance of existing solutions is degraded when reducing complexity.
Using a simplified transformation network architecture, including an optimized attention module, a streamlined upsampling module and a simplified activation function, reduces computational complexity and removes irrelevant operations by introducing a transformer-based attention module and pixel recombination operation. Simplify the activation function.
Significantly reduces the complexity of the decoder (kMAC/pixel decreases from 833 to 172), and improves computational efficiency while maintaining performance, increasing encoding gain (from 27.3% to 27.5%).
Smart Images

Figure BDA0004965041940000131 
Figure HDA0004965041960000011 
Figure HDA0004965041960000021
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. Provisional Application No. 63 / 454,821, filed on March 27, 2023, and U.S. Application No. 18 / 616,793, filed on March 26, 2024, the entire contents of which are incorporated herein by reference. Technical Field
[0003] The present invention provides an efficient neural network module for image compression. The architecture can provide reasonable compression performance while significantly reducing the complexity of the image compression model. Background Art
[0004] In recent years, ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) has been actively looking for potential needs for standardization of future video coding technology. ISO / IEC JPEG has established the JPEG-AI group, focusing on AI-based end-to-end neural image compression using deep neural networks (DNN). The Chinese AVS standard has also formed an AVS-AI special group to study neural image technology and video compression technology. At the same time, some companies have also provided funding for special research projects on neural image compression (NIC). The success of some methods in recent years has aroused increasing interest in the industry for advanced neural image and video compression methods.
[0005] Traditional hybrid video codecs are difficult to optimize. Improvements in a single module may not produce coding gains in overall performance. In contrast, in an artificial neural network-based video / image coding framework, different modules can be jointly optimized from input to output by performing a machine learning process to improve the final goal (e.g., rate-distortion performance), thereby achieving end-to-end (E2E) optimized neural image compression (NIC).
[0006] Therefore, for these reasons, it is desirable to find technical solutions to such problems arising in computer technology. Summary of the invention
[0007] According to one aspect of the present invention, a device is provided, and similarly a method and a computer-readable medium are also provided, the device comprising at least one memory configured to store computer program code; and at least one processor configured to access the computer program code and operate according to the instructions of the computer program code, the computer program code comprising: a receiving code and a reconstruction code; the receiving code is configured to: enable at least one processor to receive a video bitstream, the video bitstream comprising a current block in a current image; the reconstruction code is configured to: enable at least one processor to reconstruct the current block by transforming the current block by a neural network, the neural network comprising multiple upsampling modules and multiple activation modules; at least one of the multiple upsampling modules comprises a convolution layer and a pixel reorganization layer; and at least one of the multiple activation modules comprises a leaky rectified linear unit (LeakyReLu) function and a convolution function.
[0008] At least one upsampling module among the plurality of upsampling modules may include only a convolution layer and a pixel re-organization layer.
[0009] At least one activation module among the multiple activation modules may include multiple LeakyReLu functions, so that the output of the LeakyReLu function can be output to the convolution function, and the output of the convolution function can be output to a second LeakyReLu function among the multiple LeakyReLu functions.
[0010] The output of the LeakyReLu function may be output to a convolution function, and the output of the convolution function may be output to a multiplication function.
[0011] The output of the multiplication function can be output to the addition function.
[0012] The output of the LeakyReLu function may be output to the convolution function, and the output of the convolution function may be output to the addition function.
[0013] The convolution function may include a 1×1 filter.
[0014] Each of the multiple upsampling modules may include a different number of kilo multiply and accumulate (KMAC) operations.
[0015] The number of KMAC operations included in one of the multiple upsampling modules may be more than twice the number of KMAC operations included in any other upsampling module in the multiple upsampling modules.
[0016] The plurality of upsampling modules may be only four upsampling modules.
[0017] Other embodiments will be set forth in the detailed description which follows and, in part, will be obvious from the detailed description, and / or may be learned by practice of the presented embodiments of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The above features and aspects, as well as other features and aspects of the embodiments of the present invention will become more apparent from the following detailed description in conjunction with the accompanying drawings, in which:
[0019] Figure 1 is a schematic diagram according to an embodiment.
[0020] Figure 2 is a simplified flow chart according to an embodiment.
[0021] Figure 3 is a simplified diagram according to an embodiment.
[0022] Figure 4 is a simplified flow chart according to an embodiment.
[0023] Figure 5 is a simplified diagram according to an embodiment.
[0024] Figure 6 is a simplified diagram according to an embodiment.
[0025] Figure 7 is a simplified diagram according to an embodiment.
[0026] Figure 8 is a simplified diagram according to an embodiment.
[0027] Fig. 9 is a simplified diagram according to an embodiment.
[0028] Fig.10 is a simplified diagram according to an embodiment.
[0029] Fig.11 is a simplified diagram according to an embodiment.
[0030] Fig.12 is a simplified diagram according to an embodiment.
[0031] Fig.13 is a simplified diagram according to an embodiment.
[0032] Fig.14 is a simplified diagram according to an embodiment.
[0033] Fig.15 is a simplified diagram according to an embodiment.
[0034] Fig.16 is a simplified flow chart according to an embodiment.
[0035] Fig.17is a simplified diagram according to an embodiment.
[0036] Fig.18 is a simplified diagram according to an embodiment.
[0037] Fig.19 is a schematic diagram according to an embodiment.
[0038] Fig. 20 is a simplified diagram according to an embodiment.
[0039] Fig.21 is a simplified diagram according to an embodiment.
[0040] Fig. 22 is a simplified diagram according to an embodiment.
[0041] Fig.23 is a simplified diagram according to an embodiment.
[0042] Fig.24 is a simplified diagram according to an embodiment. DETAILED DESCRIPTION
[0043] The following detailed description of example embodiments refers to the accompanying drawings.The same reference numbers in different drawings may identify the same or similar elements.
[0044] Figure 1 A simplified block diagram of a communication system 100 according to an embodiment of the present invention is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected by a network 105. For unidirectional data transmission, a first terminal 103 may encode video data locally for transmission to another terminal 102 via the network 105. The second terminal 102 may receive the encoded video data of the other terminal from the network 105, decode the encoded data and display the recovered video data. Unidirectional data transmission is more common in applications such as media services.
[0045] Figure 1 A second pair of terminals 101 and 104 are shown configured to support bidirectional transmission of encoded video, which may occur, for example, during a video conference. For bidirectional data transmission, each terminal 101 and 104 may encode video data collected locally for transmission to the other terminal via a network 105. Each terminal 101 and 104 may also receive encoded video data sent by the other terminal, may decode the encoded data, and may display the recovered video data at a local display device.
[0046] exist Figure 1In the embodiment of the present invention, terminals 101, 102, 103 and 104 may be shown as servers, personal computers and smart phones, but the principles of the present invention are not limited thereto. Embodiments of the present invention may be applied to laptop computers, tablet computers, media players and / or dedicated video conferencing equipment. Network 105 represents any number of networks that transmit encoded video data between terminals 101, 102, 103 and 104, including, for example, wired and / or wireless communication networks. Communication network 105 may exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purpose of the present invention, unless explained below, the architecture and topology of network 105 may be irrelevant to the operation of the present invention.
[0047] Figure 2 The placement of a video encoder and a video decoder in a streaming environment is shown as an example of the application of the subject matter disclosed in this application. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital television (TV), storing compressed video on digital media including compact discs (CDs), digital video disks (DVDs), memory sticks, etc., and the like.
[0048] The streaming system may include an acquisition subsystem 203, which may include a video source 201, such as a digital camera, which creates an uncompressed video sample stream 213. The sample stream 213 may be emphasized as a high data volume compared to an encoded video bitstream and may be processed by an encoder 202 coupled to the camera 201. The encoder 202 may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. The encoded video bitstream 204 may be emphasized as a low data volume compared to the sample stream and is stored on a streaming server 205 for future use. One or more streaming clients 212 and 207 may access the streaming server 205 to obtain copies 208 and 206 of the encoded video bitstream 204. The client 212 may include a video decoder 211, which decodes the incoming copy 208 of the encoded video bitstream and produces an output video sample stream 210 that can be presented on a display 209 or another display device (not depicted). In some streaming systems, the video bitstreams 204, 206, and 208 may be encoded according to certain video encoding / compression standards. Examples of these standards are described above and further herein.
[0049] Figure 3 may be a functional block diagram of a video decoder 300 according to an embodiment of the present invention.
[0050] Receiver 302 may receive one or more encoded video sequences to be decoded by decoder 300; in the same or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the decoding of other encoded video sequences. The encoded video sequence may be received from channel 301, which may be a hardware / software link to a storage device storing the encoded video data. Receiver 302 may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective use entities (not depicted). Receiver 302 may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory 303 may be coupled between receiver 302 and entropy decoder / parser 304 (hereinafter referred to as "parser"). When receiver 302 receives data from a storage / forward device with sufficient bandwidth and controllability or from an isochronous synchronous network, it may not be necessary to configure buffer 303, or the buffer may be made smaller. For use over a traffic packet network such as the Internet, a buffer 303 may also be required, which may be relatively large and may advantageously be of an adaptive size.
[0051] The video decoder 300 may include a parser 304 to reconstruct symbols 313 from an entropy-encoded video sequence. The categories of these symbols include information for managing the operation of the decoder 300 and may include information for controlling a display device (e.g., a display 312) that is not part of the decoder but may be coupled to the decoder. The control information for the rendering device may be in the form of a parameter set fragment (not depicted) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser 304 may parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence may be performed according to a video encoding technique or standard and may follow various principles known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 304 may extract a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to the group. Subgroups may include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, Coding Units (CU), blocks, Transform Units (TU), Prediction Units (PU), etc. The entropy decoder / parser may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.
[0052] The parser 304 may perform an entropy decoding / parsing operation on the video sequence received from the buffer 303, thereby creating the symbol 313. The parser 304 may receive the encoded data and selectively decode a specific symbol 313. In addition, the parser 304 may determine whether to provide the specific symbol 313 to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra prediction unit 307, or the loop filter 311.
[0053] Depending on the type of the coded video picture or a portion of the coded video picture (e.g., inter-frame and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of the symbol 313 may involve multiple different units. Which units are involved and how they are involved can be controlled by the parser 304 through subgroup control information parsed from the coded video sequence. For the sake of brevity, such subgroup control information flow between the parser 304 and the multiple units below is not described.
[0054] In addition to the functional blocks already mentioned, decoder 300 may be conceptually subdivided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide into the functional units below.
[0055] The first unit is a sealer / inverse transform unit 305. The sealer / inverse transform unit 305 receives quantized transform coefficients as symbols 313 from the parser 304, as well as control information including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit may output a block including sample values, which may be input into the aggregator 310.
[0056] In some cases, the output samples of the sealer / inverse transform unit 305 may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by the intra-picture prediction unit 307. In some cases, the intra-picture prediction unit 307 generates a block of the same size and shape as the block being reconstructed using reconstructed information extracted from the current (partially reconstructed) picture 309. In some cases, the aggregator 310 adds the prediction information generated by the intra-prediction unit 307 to the output sample information provided by the sealer / inverse transform unit 305 on a per-sample basis.
[0057] In other cases, the output samples of the sealer / inverse transform unit 305 may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit 306 may access the reference picture memory 308 to extract samples for prediction. After the extracted samples are motion compensated according to the symbol 313 belonging to the block, these samples may be added to the output of the sealer / inverse transform unit (in this case referred to as residual samples or residual signal) by the aggregator 310, thereby generating output sample information. The address in the reference picture memory from which the prediction samples extracted by the motion compensation unit come from may be controlled by a motion vector, and the motion vector is provided to the motion compensation unit in the form of the symbol 313, which includes, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values extracted from the reference picture memory when using sub-sample accurate motion vectors, motion vector prediction mechanisms, and the like.
[0058] The output samples of aggregator 310 may be applied to various loop filtering techniques in loop filter unit 311. The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video bitstream and available to loop filter unit 311 as symbols 313 from parser 304. The video compression techniques may also be responsive to meta-information obtained during decoding of a coded picture or a previous (in decoding order) portion of a coded video sequence, and to previously reconstructed and loop filtered sample values.
[0059] The output of the loop filter unit 311 may be a sample stream, which may be output to the rendering device 312 and stored in the reference picture memory 557 for subsequent inter-picture prediction.
[0060] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. Once a coded picture is fully reconstructed, and the coded picture is identified (e.g., by parser 304) as a reference picture, current reference picture 309 may become part of reference picture buffer 308, and a new current picture memory may be reallocated before starting reconstruction of a subsequent coded picture.
[0061] The video decoder 300 may perform decoding operations according to a predetermined video compression technique in a standard such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax specified by the video compression technology or standard being used in the sense that the encoded video sequence follows the syntax of the video compression technology or standard (as specified in the video compression technology document or standard, in particular as specified in its profile document). For compliance, the complexity of the encoded video sequence may also be required to be within the range defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, mega samples per second), maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata of the HRD buffer management signaled in the encoded video sequence.
[0062] In an embodiment, the receiver 302 may receive the encoded video and additional (redundant) data. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder 300 to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.
[0063] Figure 4 may be a functional block diagram of a video encoder 400 according to an embodiment of the present invention.
[0064] The encoder 400 may receive video samples from a video source 401 (not a part of the encoder), which may capture video images to be encoded by the encoder 400 .
[0065] The video source 401 may provide a source video sequence, which will be encoded by the encoder (303) into a digital video sample stream, which may have any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits, ...), any color space (e.g., BT.601Y CrCB, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source 401 may be a storage device that stores previously prepared videos. In a video conferencing system, the video source 401 may be a camera that collects local image information as a video sequence. The video data may be provided as a plurality of separate pictures, which are given motion when viewed sequentially. The picture itself may be constructed as a spatial pixel array, wherein each pixel may include one or more samples according to the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on the samples.
[0066] According to an embodiment, the encoder 400 may encode and compress the pictures of the source video sequence into an encoded video sequence 410 in real time or under any other time constraints required by the application. It is a function of the controller 402 to implement the appropriate encoding speed. The controller controls other functional units as described below and is functionally coupled to these units. For the sake of brevity, the coupling is not depicted in the figure. The parameters set by the controller may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology, etc.), picture size, picture group (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can easily identify other functions of the controller 402, which may be related to the video encoder 400 optimized for a certain system design.
[0067] Some video encoders operate in what is commonly referred to by those skilled in the art as a "coding loop". As a simplified description, the coding loop may consist of an encoding portion including an encoder 402 (hereinafter referred to as a "source encoder") (e.g., responsible for creating symbols based on the input picture to be encoded and the reference picture) and a (local) decoder 406 embedded in the encoder 400, which reconstructs the symbols to create sample data that the (remote) decoder can also create (because in the video compression technology considered by the disclosed subject matter, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream is input to a reference picture memory 405. Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference picture buffer also correspond bit-accurately between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction portion of the encoder are exactly the same sample values that the decoder will "see" when using the prediction during decoding. The basic principle of this reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, for example due to channel errors) is well known to those skilled in the art.
[0068] The operation of the "local" decoder 406 may be combined with, for example, Figure 3 The operation of the "remote" decoder 300 described in detail is the same. However, additional brief reference is made to Figure 4 , when symbols are available and the entropy encoder 408 and the parser 304 are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the decoder 300 (including the channel 301, the receiver 302, the buffer 303 and the parser 304) may not be fully implemented in the local decoder 406.
[0069] At this point it can be observed that any decoder techniques other than parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. The description of encoder techniques can be simplified because encoder techniques are reciprocal to the decoder techniques described comprehensively. A more detailed description is only required in certain areas and is provided below.
[0070] As part of its operation, source encoder 403 may perform motion compensated predictive coding, which predictively encodes an input frame with reference to one or more previously encoded frames from a video sequence designated as “reference frames.” In this manner, encoding engine 407 encodes the differences between pixel blocks of an input frame and pixel blocks of a reference frame that may be selected as a prediction reference for the input frame.
[0071] The local video decoder 406 may decode the encoded video data of the frame that may be designated as the reference frame based on the symbol created by the source encoder 403. The operation of the encoding engine 407 may advantageously be a lossy process. When the encoded video data is available at the video decoder ( Figure 4 When decoded at a remote video decoder (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder 406 replicates the decoding process that may be performed by the video decoder on the reference frame and may cause the reconstructed reference frame to be stored in the reference picture cache 405. In this way, the encoder 400 may locally store a copy of the reconstructed reference frame that has common content (absent transmission errors) with the reconstructed reference frame to be obtained by the remote video decoder.
[0072] The predictor 404 may perform a prediction search for the encoding engine 407. That is, for a new frame to be encoded, the predictor 404 may search the reference picture memory 405 for sample data (as a candidate reference pixel block) or certain metadata, such as a reference picture motion vector, block shape, etc., that may serve as an appropriate prediction reference for the new picture. The predictor 404 may operate pixel-by-pixel based on a sample block to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor 404, it may be determined that the input picture may have a prediction reference taken from a plurality of reference pictures stored in the reference picture memory 405.
[0073] The controller 402 may manage encoding operations of the video encoder 403 , including, for example, setting parameters and subgroup parameters for encoding video data.
[0074] The outputs of all the above functional units may be entropy encoded in the entropy encoder 408. The entropy encoder may transform the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc.
[0075] The transmitter 409 may buffer the encoded video sequence created by the entropy encoder 408 in preparation for transmission over a communication channel 411, which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter 409 may combine the encoded video data from the video encoder 403 with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).
[0076] The controller 402 may manage the operation of the encoder 400. During encoding, the controller 405 may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following picture types:
[0077] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh pictures. Those skilled in the art are aware of variations of I pictures and their corresponding applications and features.
[0078] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values for each block.
[0079] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstructing a single block.
[0080] The source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples), and encoded block by block. The blocks may be predictively coded with reference to other (already coded) blocks, which are determined according to the coding allocation applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively coded, or may be predictively coded (spatial prediction or intra-prediction) with reference to already coded blocks of the same picture. Blocks of pixels of a P picture may be non-predictively coded by spatial prediction with reference to one previously coded reference picture or by temporal prediction. Blocks of a B picture may be non-predictively coded by spatial prediction with reference to one or two previously coded reference pictures or by temporal prediction.
[0081] The video encoder 400 may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder 400 may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.
[0082] In an embodiment, the transmitter 409 may send additional data when sending the encoded video. The source encoder 403 may include such data as part of the encoded video sequence. The additional data may include time / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, etc.
[0083] According to an embodiment, the overall framework of NIC is described as follows: Given an input image x, the goal of NIC is to use image x as the input of a DNN encoder to compute a compressed representation x, which is a compact representation of x for storage and transmission purposes. Then, use x as the input of a DNN decoder to reconstruct the image Other NIC methods adopt a variational autoencoder (VAE) structure, where the DNN encoder directly uses the entire image x as its input, which is passed through a set of network layers that work like a black box to compute the output representation x. Correspondingly, the DNN decoder takes the entire representation x as its input, which is passed to another set of network layers that work like another black box to compute the reconstructed The rate-distortion (RD) loss is optimized to achieve the distortion loss of the reconstructed image. The tradeoff between the bit consumption R of the compact representation x is λ, and has a tradeoff hyperparameter λ:
[0084]
[0085] Figure 5 The overall architecture 500 of a NIC according to an exemplary embodiment is shown. The encoder includes a set of visual modules (e.g., convolutional networks or visual transformers) with downsampling functions; however, the decoder consists of a set of visual modules (e.g., convolutional networks or visual transformers) with upsampling functions.
[0086] Figure 6An example 600 of a detailed structure of a decoder according to an exemplary embodiment is shown. In some embodiments, the residual block before the upsampling block 601 can be removed. The residual attention unit (Residual attention unit, ResAU) is an activation function. The residual non-local attention block (Residual Non-local Attention Block, RNAB) is an attention module. In one embodiment, RNAB consists of a group of residual blocks. In another embodiment, RNAB is a transformer-based attention module. According to an exemplary embodiment, the efficient neural network structure includes improved upsampling blocks 601, 602, 603 and 604 in the decoder.
[0087] A key aspect of an embodiment includes efficient tensor transformation with an upsampling module in the decoder. Figure 7 An example of a detailed upsampling module 700 is shown, such as any one of the upsampling modules 601 , 602 , 603 , and 604 .
[0088] The upsampling module 700 is composed of a convolution layer 701 and a pixel shuffle layer 702. The convolution layer 701 is used to increase the number of channels of the input tensor, and the pixel shuffle layer 702 is used to upsample the tensor.
[0089] Embodiments do not need to constrain the details of convolutional layer 701. In one embodiment, the convolutional network has a filter size of 3×3; in another preferred embodiment, the convolutional network has a filter size of 5×5; in another preferred embodiment, the convolutional network has a filter size of 7×7. In another embodiment, the upsampling module is a traditional transposed convolutional network.
[0090] According to an embodiment, for example Figure 6 As shown, there may be four upsampling modules in the decoder, such as upsampling blocks 601, 602, 603, and 604. In one embodiment, the four upsampling modules are identical (ie, they all use 1×1 convolution for pixel rebinning).
[0091] In one embodiment, the upsampling modules are different. In one embodiment, the first upsampling module (upsampling block 601) is a transposed convolution with a filter size of 3×3. The second upsampling module (upsampling block 602) is a transposed convolution with a filter size of 3×3. The third upsampling module (upsampling block 603) uses convolution 1×1 for pixel reorganization. The fourth upsampling module (upsampling block 603) uses convolution 1×1 for pixel reorganization. These upsampling modules can be combined in any way.
[0092] Compared to conventional modules in neural image compression (comparative examples described below), such exemplary embodiments significantly reduce the complexity of the image compression model while achieving favorable and reasonable compression rates.
[0093] Figure 8 The core network structure 800 of the proposed solution in the JPEG AI solicitation is shown as a simplified analysis and synthesis transformation that can be used in the verification model VMuC under consideration to achieve a better performance / complexity trade-off.
[0094] Fig. 9 Modules used in analysis and synthesis transformations according to an exemplary embodiment are shown (module 901 is a Res Block with Upsample (RBU), module 902 is a Res Block (RB), module 903 is a Res Unit (RU), module 904 is an Attention Block (AB), module 905 is a Res Block with Stride (RBS), and module 906 is a scaling net (SN)). The Res Block with Stride (RBS) is a downsampling module in the analysis transformation; and the Res Block with Upsample (RBU) is used as an upsampling module in the synthesis transformation according to an embodiment.
[0095] Fig.10 The synthetic transformation of VMuC according to an exemplary embodiment is shown, and Fig.10 It includes upsampling modules (convolutional network and ResAU) and Res blocks.
[0096] In the comparative tests performed, "Test 01" used the modified RBU (replacing convolution and ResAU) in the Verification Model (VM) and also removed the RNAB module; "Test 02" used the simplified RBU (replacing convolution and ResAU) in the latest VM and also removed the RNAB module; and "Test 03" used the modified RBU (replacing convolution and ResAU) in the VM and retained the RNAB module. According to an exemplary embodiment, the details of the modified RBU module 1101 are shown in FIG. Fig.11 The example of FIG. 110 adds a clipping module 1110. According to an exemplary embodiment, the details of the simplified RBU module 1102 are shown in FIG. Fig.11 Add any one of the simplified RBU module 1102 and the modified RBU module 1101 in the example to Fig.12In any one of the blocks 1201, 1202, 1203, 1204, 1205, 1206 of the modified / simplified RBU module shown in the example 1200.
[0097] Fig.13 A comparative example 1300 for Test 01 and Test 02 is shown.
[0098] Fig.14 Example 1400 in FIG. 1 shows a VM upsampling structure according to an exemplary embodiment, where the transform consists of four transposed convolutional layers. Among them, the upsampling 03 module has the largest number of KMACs. According to an exemplary embodiment, the simplified KMACs for each upsampling module are: upsampling 01: ~5, upsampling 02: ~15, upsampling 03: ~60, and upsampling 04: ~2.
[0099] Compared with other embodiments, Fig.14 The goal of the illustrated embodiment is to replace the transposed convolution with pixel reorganization using a conventional convolutional network. Therefore, a convolutional network with a 1×1 filter is used before pixel reorganization to change the number of channels. And, through testing, the convolutional network with a 3×3 filter does not provide a significant gain compared to the convolutional network with a 1×1 filter, and also introduces many KMACs. Therefore, as a technical solution, the embodiment of the present invention can replace the transposed convolution with pixel reorganization and a conventional convolutional network with a 1×1 filter.
[0100] Fig.15 The exemplary table shows the simplified upsampling implemented according to the exemplary embodiment of the present invention. Through testing, it is found that it is difficult to obtain loss convergence in the training in Scheme 1 and Scheme 3. And it is found that Scheme 2 has some color shifting problems during the training process. Therefore, according to the exemplary embodiment, Scheme 4 can be implemented as a simplified upsampling scheme. Compared with Anchor, the simplified upsampling reduces 39kMac / pixel (about 43% if the RNAB module is not considered) and introduces only 0.2% coding loss.
[0101] exist Fig.15 On the basis of Fig.16 The example 1600 in FIG. 1 shows an activation function implemented according to an exemplary embodiment of the present invention, in which three simplified activation function architectures are shown as follows: Fig.17As shown (in Scheme 1 of Embodiment 1701, a leaky rectified linear unit (LeakyReLU) function can be used to replace the tangent function, in Scheme 2 of Embodiment 1702, the tangent function can be removed, and in Scheme 3 of Embodiment 1703, the tangent function and the multiplication function can be removed), and through testing, Scheme 3 (Embodiment 1703) has shown the best trade-off between performance and complexity, as shown in FIG. Fig.18 The test results 1800 are shown in FIG.
[0102] Learning-based image coding schemes, such as JPEG AI, have greatly surpassed traditional image compression standards in terms of rate-distortion (RD) performance, showing great potential. However, the widespread application of learning-based image coding schemes is hindered by high decoding complexity, especially from the complexity of upsampling modules and attention modules. Existing work has attempted to reduce this complexity, but these solutions are not completely effective and leave considerable complexity problems unsolved. Therefore, an embodiment of the present invention proposes a simplified transform network architecture that uses an optimized attention module, a streamlined upsampling module, and a streamlined activation function to solve this problem. Simulation results show that the complexity of the simplified decoder (measured in kMAC / pixel) is reduced by 80% (from 833 to 172), and the gain is slightly increased compared to Versatile Video Coding (VVC) (from 27.3% to 27.5%). Some of the methods described in the present invention can be integrated into the JPEG AI Verification Model (VM) software.
[0103] Machine learning has led to breakthroughs in various fields, notably the neural network-based image compression JPEG AI, a standard developed by JPEG that represents a major advance in the field. Machine learning is designed for both human visualization and computer vision tasks.
[0104] Despite these advances, one of the main challenges of the JPEG AI verification model (called JPEG AI VM) is its high complexity, especially the complexity of the transform network in the decoder. This complexity leads to higher computational costs, reducing the practicality and efficiency of the standard. There have been some studies on reducing the complexity of the decoder, such as simplifying the context model entropy decoding process and exploring sparse network structure techniques, but most existing solutions have resulted in performance degradation due to insufficient attention to the design of the transform network architecture in the decoder.
[0105] Therefore, embodiments of the present invention address this challenge by providing three simplifications of the transformer network: 1) a modified attention module, 2) a new upsampling module, and 3) a simplified activation function. These modifications reduce the computational load and improve efficiency without affecting system performance. Specifically, a transformer-based attention module is introduced that uses a hierarchical structure to reduce complexity. The embodiment uses a Restormer block to apply self-attention across channels rather than across spatial dimensions, effectively acquiring global information and reducing complexity. A pixel reorganization operation using the previous 1×1 convolution is also used as an upsampling scheme. This design reduces computational complexity while maintaining network performance. In addition, a more streamlined activation function is provided that does not require hyperbolic tangent and subsequent element-by-element operations to further reduce computational complexity.
[0106] Fig.19 A comparative analysis of details in the reconstructed image is provided. The top panel 1901 depicts the uncompressed original image, the middle panel 1902 depicts the reconstruction after compression via JPEG AI VM, and the lower panel 1903 shows the reconstruction achieved via a JPEG AI-based scheme, in accordance with an embodiment of the present invention. The visual comparison demonstrates the ability of embodiments of the present invention to reduce kilomultiply-accumulate (KMAC) operations while reducing bits per pixel (bpp) and enhancing image quality.
[0107] The structure of JPEG AI Transformation Network 2001 is as follows Fig. 20 As shown in Example 200. The embodiment focuses on the decoder transformer network because the decoder transformer network is crucial to the decoding complexity. The transformer network has two residual blocks (RBs) placed at the beginning of the network and a sequence of four convolutions with an upsampling stride of 2 to increase the resolution of the output. In addition, the transformer network has a residual attention unit (ResAU) as an activation function and a residual non-local attention module (RNAB) as an attention module to improve nonlinear representation capabilities. Fig. 20 The architecture of RNAB 2002, ResAU 2003, and RB 2004 is shown. The embodiment focuses on the decoder transformer network because the decoder transformer network is crucial to the decoding complexity. As shown, the JPEG AI transform network 2001 has two RBs placed at the beginning of the network and a sequence of four convolutions with an upsampling stride of 2 to increase the resolution of the output. In addition, the transform network 2001 has ResAU as an activation function and RNAB as an attention module to improve nonlinear representation capabilities.
[0108] While achieving good performance, JPEG AI also brings huge computational complexity to the transform network decoder side. Specifically, the complexity of the current transform network on the decoder side is 833kMAC / pixel, which is a very large computational amount. Specifically, among all components, the RNAB module and the upsampling module have the highest complexity and the second highest complexity, respectively, accounting for 95% of the complexity in total. To this end, an embodiment includes two low-complexity transformer-based attention modules and a pixel-reorganization-based upsampling module to replace the current RNAB module and upsampling module, thereby reducing complexity while maintaining comparable performance.
[0109] like Fig.21 As shown in example 2100, embodiments provide a transformer-based attention module to obtain global information to maintain performance and a hierarchical attention architecture to achieve low complexity. Fig.21 shown.
[0110] The transformer architecture 2100 according to the embodiment has the ability to obtain global information, so it shows its advantage over convolution in various visual tasks. However, the memory usage of the original visual transformer is quite large, especially for high-resolution images. Secondly, the complexity grows quadratically with the input width / height, resulting in a higher kMAC / pixel. Some use Swin Transformer to introduce shifted window-based attention to reduce complexity. However, Swin Transformer cannot obtain information of high-resolution images at a long distance and may cause performance degradation.
[0111] In order to solve the above two challenges, the embodiment adopts a Restormer block (e.g., Restormer block 2101, Restormer block 2102, Restormer block 2013) as a basic module in such embodiments of the present invention. The Restormer block applies self-attention across channels rather than across spatial dimensions, that is, the time and memory complexity of the key query dot product grows linearly with the spatial resolution of the input. The detailed structure of the Restormer block is as follows: Fig. 22 As shown, the restoration block consists of the following two sub-blocks.
[0112] One is a multi-Dconv head transposed attention (Multi-Dconv Head Transposed Attention, MDTA) block 2201. This block is used to generate a global attention map. Assume that the input is X. The embodiment first passes X to the layer normalization operator and obtains the normalized value. After that, according to the normalized value, three groups of 1×1 convolutions and 3×3 depth-wise convolutions are used respectively to obtain the query (query, Q), key (key, K) and value (value, V). This process can encode the spatial context channel by channel. Finally, the transposed attention map is obtained by calculating the dot product of the reshaped query and the key.
[0113] The other is a gated-Dconv feed-forward network (GDFN) block 2202. GDFN expands the feature channel so that the feature channel has the same dimension as the input. In general, the restoration block first passes the input to MDTA to obtain features with context information, and then feeds the intermediate results to GDFN to control the information flow. The embodiment also designs the following two methods to optimize the use of peak memory.
[0114] One method is channel segmentation. That is, after the third convolution, the output is split into two parts across channels, and the two parts are passed to two restoration blocks respectively.
[0115] Another method is spatial downsampling. Assume that the output after the third convolution is An embodiment may first reduce the size of X before passing it to the transformer block, i.e., the shape of the intermediate output is C×(H / 2)×(W / 2). Afterwards, an embodiment increases the size of the output to keep the resolution unchanged.
[0116] like Fig. 20As shown in the JPEG AI Transform Network 2001 in Figure 2, there is also a low-complexity upsampling module, and the decoder structure consists of four upsampling modules. JPEG AI VM2.0 uses transposed convolutions with a stride of 2 for the upsampling process, where the convolution kernel size is set to 3. This method is known for its high coding complexity. Looking closely at the details of the upsampling modules in VM2.0, it can be found that the complexity increases gradually from right to left. The computational cost (measured in KMAC / pixel) associated with the four upsampling modules is as follows: 5, 15, 60, and 2. This increasing trend in complexity corresponds to the increasing feature map size when passing through these modules. This relationship peaks in the third upsampling module, which exhibits the highest complexity due to the increase in feature map size. Interestingly, despite having the largest input size, the last upsampling module has a relatively low computational cost. This reduction is attributed to the module only outputting three layers, limiting the overall complexity of the module.
[0117] The pixel reordering operation essentially rearranges the elements in the tensor so that the spatial resolution (height and width) of the tensor increases, while the depth (or number of channels) decreases. Assume that the channels of the input feature map F are C = n*r 2 With a spatial resolution of H×W, pixel reorganization reorganizes the elements of F to form a new feature map F'', the new feature map F'' has a depth (or channel) C=n, but a spatial resolution of rH×rW. Here, r is an integer upsampling factor.
[0118] In practice, the pixel reshuffling operation is often preceded by a standard convolution operation to increase the number of channels in the input tensor. This allows the pixel reshuffling operation to properly redistribute the pixels to achieve the desired upsampling effect. When the filter size is 1×1, the complexity of this upsampling module (convolution layer with filter 1×1 and pixel reshuffling) will be reduced compared to the transposed convolution with 3×3 filters. By replacing the third upsampling module from the right, which has the highest computational cost in terms of KMAC / pixel, we can significantly reduce the complexity of the decoder.
[0119] According to an embodiment of the present invention, there is also an activation function module with low complexity. The activation function module includes a set of several submodules, including: LeakyReLU, 1×1 convolution layer, and hyperbolic tangent. In addition, the activation function module also involves some complex operations, such as element-by-element multiplication operations. Fig. 20The complete module structure is shown in ResAU 2003 of . While many of these operations are considered important components in a neural network, embodiments of the present invention are based on the recognition of certain redundancies that, although not explicitly quantified in the form of KMAC / pixel, do add unnecessary complexity to the system. As such, embodiments have included a simplified model that streamlines the structure of the activation function module. Specifically, according to an embodiment, the present invention removes the hyperbolic tangent activation function and subsequent element-by-element operations. Embodiments of the present invention have been compared with the JPEGAI general training and test conditions defined in an Nvidia A100 GPU device. Bit rates of 0.06, 0.12, 0.25, 0.50, and 0.75 are mandatory and are all used for BD rate calculations. Objective quality metrics include MS-SSIM, FSIM, NLPD, IWSSIM, VMAF, VIF, and PSNR-HVS-M between compressed and original images at these target bit rates. The average of the above 7 indicators (MS-SSIM, FSIM, NLPD, IW-SSIM, VMAF, VIFP and PSNR-HVS-M) Delta-ratio( Delta-rate, BD-rate) is used as our main distortion metric to measure the quality of image compression. Taking into account various aspects of perceptual similarity, naturalness, and fidelity, these seven metrics together provide a comprehensive and accurate assessment of image quality. To measure the complexity of the model, the widely used ptflops library is used to calculate kMAC / pixel. This allows us to measure the computational efficiency of the algorithm.
[0120] The embodiments mainly present and evaluate simplified alternatives of specific JPEG AI VM 2.0 components as our baseline. In Section IV-B, the embodiments use two transformations of varying channel segmentation and spatial downsampling strategies to evaluate the multi-level attention architecture. The following description provides investigation results on the effects of combined testing of simplified upsampling modules, simplified activation functions, further complexity reduction, and merging all proposed simplifications into a single model for evaluation.
[0121] The embodiments are evaluated by comparing a simplified attention module (Attention-CS) using channel splitting after the third convolutional layer with a simplified attention module (Attention-DS) using spatial downsampling to optimize the use of peak memory. For the simplified attention module-CS, γ is set to 2.66 for the restoration blocks after all convolutional layers, respectively. For the simplified attention module-DS, multiple γ are set to 4, 4, and 2.66 for the restoration blocks after the first convolutional layer, the second convolutional layer, and the third convolutional layer, respectively.
[0122] The results of this experiment are Fig.23 As shown in the example table 2300 in , and the following findings are made. First, attention-CS and simplified attention module-DS can reduce the decoding complexity (up to 75%) with a small drop in performance (up to 0.8%). This demonstrates the effectiveness of the proposed multi-level transformer-based attention architecture. Second, attention-CS outperforms attention-DS in both BD rate and decoding complexity. The reason may be that the resolution after the third convolution is high, and spatial information is more important than channel information for the final performance. This shows that the channel segmentation method is more suitable for the JPEGAI codec.
[0123] In Table 2300, negative values mean encoding performance (lower is better), and for two trained models with the same initialization, the performance deviation reaches 0.5% due to the randomness introduced in training, and the calculations within the activation function cannot be explicitly quantified in terms of kMAC / pixel.
[0124] The embodiment can also use the simplified upsampling method we proposed to replace the third upsampling module from the right. Specifically, compared with the original upsampling module (Anchor in VVC), this simplified upsampling method reduces 39kMAC per pixel. Assuming that the embodiment ignores the RNAB module, in this case, this reduction is equivalent to about 43% of the computational cost.
[0125] When examining the overall effect of the simplified upsampling module, we observed an average BD rate reduction of 0.2%. Given the nature of deep learning models, especially when focusing on complex tasks such as JPEGAI, each training iteration is associated with an inherent uncertainty. This uncertainty, or training error, typically varies around 0.5% during each JPEG AI training cycle. Therefore, when we reduce the BD rate to 0.2% in this case, it falls well within the usual training error range. This leads to an important conclusion: the small loss incurred by implementing the simplified upsampling module can be considered negligible. Overall, our proposed simplified upsampling method saves a lot of computation without affecting the rate-distortion efficiency of the model.
[0126] By introducing a simplified activation function in the transform network, as shown in example table 2300, it can be observed that the coding gain loss is about 0.1%. Considering the inherent variability in the training process, this seemingly small loss is essentially negligible. According to an example embodiment of the present invention, this loss falls completely within the range of expected training errors and does not affect the overall performance of the model.
[0127] Through such experiments, it will be noted that the complexity of complex activation functions cannot be explicitly quantified in the form of KMAC / pixel through ptflops. However, complex activation functions will generate unnecessary complexity for the system, and therefore such redundancy can be removed through the simplified scheme of the embodiments of the present invention.
[0128] In addition, according to an embodiment of the present invention, a combined test was also conducted to evaluate the overall applicability of each simplified method (simplified attention, simplified upsampling, and simplified activation). The evaluation results are also reported in Fig.23 The results are given in an example table 2300, which demonstrates the remarkable robustness and effectiveness of the simplified module in the present invention. The computational complexity in terms of kMAC / pixel is reduced from 833 to 172.1. In addition, the present embodiment manages to achieve a coding gain of 0.2%.
[0129] The observations highlight the value of these simplifications. Each simplification contributes to a more computationally efficient model without degrading overall performance. Furthermore, combined testing reaffirms the robustness of these simplifications with other modules, reinforcing their broad applicability and effectiveness in a variety of scenarios.
[0130] Therefore, the embodiments of the present invention solve the problem of high decoding complexity inherent in learning-based image coding schemes. Although the learning-based image coding scheme has excellent rate-distortion performance, its practical application is very limited due to its high complexity. Specifically, the present invention proposes a series of modifications to the decoder architecture of the JPEG AI standard, namely, simplifications of the attention module, upsampling module and activation function. The modified attention module adopts a low-complexity hierarchical attention structure, uses a transformer-based module to obtain better global information, and adopts a new memory optimization strategy to ensure feasible memory usage. The upsampling module adopts pixel reconstruction technology to reduce the complexity without affecting the stability of the network. The activation function is streamlined by removing irrelevant operations and retaining only necessary components, further reducing the complexity.
[0131] Modifications according to embodiments of the present invention reduce computational complexity, with the kMAC / pixel metric reduced from 852 to 180. Importantly, the improvement in computational efficiency also resulted in a small increase in VVC gain, from 27.3% to 27.5%. These findings balance the complexity and performance of learning-based image coding.
[0132] The techniques described above may be implemented as computer software using computer-readable instructions and may be physically stored in one or more computer-readable media or implemented by one or more hardware processors of a specific configuration. Fig.24A computer system 2400 suitable for implementing certain embodiments of the disclosed subject matter is shown.
[0133] Computer software may be encoded using any suitable machine code or computer language, which may be subjected to assembly, compilation, linking or similar mechanisms to create code comprising instructions that may be executed directly by a computer central processing unit (CPU), graphics processing unit (GPU), etc., or through interpretation, microcode execution, etc.
[0134] The instructions may be executed on various types of computers or components thereof, such as computers or components thereof including personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.
[0135] Fig.24 The components of the computer system 2400 shown are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the invention. Neither should the configuration of the components be interpreted as having any dependency or requirement relating to any one or combination of components shown in the exemplary embodiment of the computer system 2400.
[0136] Computer system 2400 may include certain human-machine interface input devices. Such human-machine interface input devices may be responsive to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not depicted). Human-machine interface devices may also be used to acquire certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images acquired from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).
[0137] The input human-machine interface device may include one or more of the following (only one of each is shown): keyboard 2401 , mouse 2402 , touch pad 2403 , touch screen 2410 , joystick 2405 , microphone 2406 , scanner 2408 , camera 2407 .
[0138] The computer system 2400 may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate one or more human user senses, for example, through tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen 2410 or joystick 2405, but may also be a tactile feedback device that is not an input device), audio output devices (e.g., speakers 2409, headphones (not depicted)), visual output devices (e.g., screens 2410 including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which are capable of outputting two-dimensional visual outputs or outputs in more than three dimensions through devices such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).
[0139] The computer system 2400 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD / RW 2420 with CD / DVD etc. media 2411, a thumb drive 2422, a removable hard drive or solid state drive 2423, traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD based devices such as security dongles (not depicted), and the like.
[0140] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.
[0141] The computer system 2400 may also include an interface 2499 to one or more communication networks 2498. The network 2498 may be, for example, a wireless network, a wired network, an optical network. The network 2498 may further be a local network, a wide area network, a metropolitan area network, a vehicle and industrial network, a real-time network, a delay-tolerant network, etc. Examples of the network 2498 include a local area network such as Ethernet, a wireless LAN, a cellular network including GSM, 3G, 4G, 5G, LTE, etc., a television wired or wireless wide area digital network including cable television, satellite television, and terrestrial broadcast television, a vehicle and industrial network including CANBus, etc. Some networks 1098 typically require an external network interface adapter (e.g., a USB port of the computer system 2400) connected to some general data port or peripheral bus (2450 and 2451); as described below, other network interfaces are typically integrated into the kernel of the computer system 2400 by connecting to the system bus (e.g., connected to an Ethernet interface in a PC computer system or connected to a cellular network interface in a smartphone computer system). The computer system 2400 can communicate with other entities using any of these networks 2498. Such communications can be one-way receive only (e.g., broadcast television), one-way send only (e.g., a CANbus connected to certain CANbus devices), or bidirectional, for example, connecting to other computer systems using a local area network or wide area network digital network. As described above, certain protocols and protocol stacks can be used on each of those networks and network interfaces.
[0142] The aforementioned human-machine interface devices, human-machine accessible storage devices, and network interfaces may be attached to the kernel 2440 of the computer system 2400 .
[0143] The core 2440 may include one or more central processing units (CPUs) 2441, graphics processing units (GPUs) 1042, graphics adapters 2417, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 2443, hardware accelerators 2444 for certain tasks, and the like. These devices, as well as read-only memory (ROM) 2445, random access memory 2446, internal mass storage 2447 such as internal non-user accessible hard disk drives, SSDs, and the like, may be connected via a system bus 2448. In some computer systems, the system bus 2448 may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, and the like. Peripheral devices may be connected directly to the core's system bus 2448, or connected to the core's system bus 2448 via a peripheral bus 2451 connection. The architecture of the peripheral bus includes PCI, USB, and the like.
[0144] The CPU 2441, GPU 2442, FPGA 2443, and accelerator 2444 may execute certain instructions, which may be combined to form the above-mentioned computer code. The computer code may be stored in a read-only memory (ROM) 2445 or a random access memory (RAM) 2446. Transitional data may also be stored in a random access memory (RAM) 2446, while permanent data may be stored, for example, in an internal mass storage 2447. Fast storage and retrieval to any storage device may be performed by using a cache, which may be closely associated with one or more CPUs 2441, GPUs 2442, mass storage 2447, ROM 2445, RAM 2446, etc.
[0145] The computer readable medium may have thereon computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present invention, or the medium and computer code may be of a type well known and available to those skilled in the art of computer software.
[0146] As a non-limiting example, a computer system having architecture 2400, especially a computer system of kernel 2440, can provide functionality due to one or more processors (including CPU, GPU, FPGA, accelerator, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as some non-temporary memory of kernel 2440, such as kernel internal mass storage 2447 or ROM 2445. Software implementing various embodiments of the present invention can be stored in such devices and executed by kernel 2440. According to specific needs, the computer-readable medium may include one or more storage devices or chips. The software can enable the core 2440, especially the processors therein (including CPU, GPU, FPGA, etc.) to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in random access memory RAM 2446 and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system 2400 may provide functionality by logic hardwired or otherwise embodied in circuitry (e.g., accelerator 2444) that may replace software or operate in conjunction with software to perform specific processes or specific portions of specific processes described herein. Where appropriate, references to portions of software may include logic and vice versa. Where appropriate, references to portions of computer-readable media may include circuits (e.g., integrated circuits (ICs)) storing software for execution, circuits embodying logic for execution, or both. The present invention includes any suitable combination of hardware and software.
[0147] Although the present invention has described a number of exemplary embodiments, there are changes, permutations and various replacement equivalents that fall within the scope of the present invention. Therefore, it should be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present invention and therefore fall within the spirit and scope of the present invention.
Claims
1. A video decoding method, performed by at least one processor, the method comprising: receiving a video bitstream, the video bitstream comprising a current block in a current picture; as well as Transforming the current block through a neural network to reconstruct the current block, wherein the neural network includes multiple upsampling modules and multiple activation modules; At least one of the multiple upsampling modules comprises a convolution layer and a pixel reassembly layer, and Among them, at least one activation module among the multiple activation modules includes a leaky rectified linear unit LeakyReLu function and a convolution function.
2. The method according to claim 1, wherein: At least one up-sampling module among the multiple up-sampling modules consists of the convolution layer and the pixel reorganization layer.
3. The method according to claim 1, wherein: At least one activation module among the multiple activation modules includes multiple LeakyReLu functions, so that the output of the LeakyReLu function is output to the convolution function, and the output of the convolution function is output to a second LeakyReLu function among the multiple LeakyReLu functions.
4. The method according to claim 1, wherein: The output of the LeakyReLu function is output to the convolution function, and the output of the convolution function is output to the multiplication function.
5. The method according to claim 4, wherein: The output of the multiplication function is output to the addition function.
6. The method according to claim 1, wherein: The output of the LeakyReLu function is output to the convolution function, and the output of the convolution function is output to the addition function.
7. The method according to claim 1, wherein: The convolution function includes a 1×1 filter.
8. The method according to claim 1, wherein: Each of the plurality of up-sampling modules includes a different number of thousands of multiplication and accumulation KMAC operations.
9. The method according to claim 8, wherein: An upsampling module among the plurality of upsampling modules includes a number of KMAC operations that is greater than twice the number of KMAC operations included in any other upsampling module among the plurality of upsampling modules.
10. The method according to claim 8, wherein: The multiple up-sampling modules consist of four up-sampling modules.
11. An apparatus comprising: at least one memory configured to store computer program code; At least one processor is configured to access the computer program code and operate according to the instructions of the computer program code, wherein the computer program code comprises: receiving code configured to cause the at least one processor to receive a video bitstream, the video bitstream comprising a current block in a current picture; and A reconstruction code is configured to cause the at least one processor to transform the current block through a neural network to reconstruct the current block, wherein the neural network includes a plurality of upsampling modules and a plurality of activation modules; At least one of the multiple upsampling modules comprises a convolution layer and a pixel reassembly layer, and Among them, at least one activation module among the multiple activation modules includes a leaky rectified linear unit LeakyReLu function and a convolution function.
12. The device according to claim 11, wherein At least one up-sampling module among the multiple up-sampling modules consists of the convolution layer and the pixel reorganization layer.
13. The device according to claim 11, wherein: At least one activation module among the multiple activation modules includes multiple LeakyReLu functions, so that the output of the LeakyReLu function is output to the convolution function, and the output of the convolution function is output to a second LeakyReLu function among the multiple LeakyReLu functions.
14. The device according to claim 11, wherein: The output of the LeakyReLu function is output to the convolution function, and the output of the convolution function is output to the multiplication function.
15. The device according to claim 14, wherein: The output of the multiplication function is output to the addition function.
16. The device according to claim 11, wherein The output of the LeakyReLu function is output to the convolution function, and the output of the convolution function is output to the addition function.
17. The device according to claim 11, wherein: The convolution function includes a 1×1 filter.
18. The device according to claim 11, wherein Each of the plurality of up-sampling modules includes a different number of thousands of multiplication and accumulation KMAC operations.
19. The device according to claim 18, wherein: An upsampling module among the plurality of upsampling modules includes a number of KMAC operations that is greater than twice the number of KMAC operations included in any other upsampling module among the plurality of upsampling modules.
20. A non-transitory computer readable medium storing a program that causes a computer to: receiving a video bitstream, the video bitstream comprising a current block in a current picture; and Transforming the current block through a neural network to reconstruct the current block, wherein the neural network includes multiple upsampling modules and multiple activation modules; in, At least one of the plurality of upsampling modules comprises a convolution layer and a pixel reassembly layer, and At least one activation module among the multiple activation modules includes a leaky rectified linear unit (LeakyReLu) function and a convolution function.
