Decoder, encoder, image encoding method, image decoding method, and storage medium

Abstract

The invention discloses a decoder, an encoder, an image encoding method, an image decoding method and a storage medium, and belongs to the technical field of encoding and decoding. The decoder comprises a first decoding network, a super decoding network and a context network, and the first decoding network comprises a first sub-network, a second sub-network, a third sub-network and a fourth sub-network which are sequentially connected from input to output. The number of channels of the input tensor of the fourth sub-network is greater than the number of channels of the input tensor of the third sub-network, and the number of channels of the input tensor of the fourth sub-network is greater than the number of channels of the input tensor of the second sub-network. By combining the structure of the first decoding network, the channel number of the input tensor of the fourth sub-network is set to be greater than the channel number of the input tensors of the third sub-network and the second sub-network, so that the calculation complexity of the first decoding network can be reduced, and the quality of the image reconstructed by the first decoding network can be ensured not to be obviously reduced.

Description

Technical Field

[0001] The present application relates to the field of coding and decoding technology, and in particular to a decoder, an encoder, an image encoding method, an image decoding method and a storage medium. Background Art

[0002] With the widespread application of deep learning technology in image recognition, target detection and other fields, deep learning technology has also been applied to image compression tasks, that is, introducing a codec network built based on deep learning technology to achieve image compression. However, although using a codec network for image compression can greatly improve the codec performance and image compression effect, it also introduces a huge amount of computation. Therefore, how to build a codec network to reduce the amount of computation has become one of the current concerns. Summary of the invention

[0003] The present application provides a decoder, an encoder, an image encoding, an image decoding method and a storage medium, which can reduce the amount of decoding calculations. The technical solution is as follows:

[0004] In a first aspect, a decoder is provided, the decoder comprising a first decoding network, a super decoding network and a context network, the first decoding network comprising a first sub-network, a second sub-network, a third sub-network and a fourth sub-network sequentially connected from input to output, the number of channels of the input tensor of the fourth sub-network being greater than the number of channels of the input tensor of the third sub-network, and the number of channels of the input tensor of the fourth sub-network being greater than the number of channels of the input tensor of the second sub-network.

[0005] Since the fourth sub-network is close to the output of the first decoding network, the computational amount of the fourth sub-network is usually represented by the product of the number of channels of the input tensor of the fourth sub-network and the number of channels of the output tensor of the first decoding network, and the number of channels of the output tensor of the first decoding network is usually small, so setting the number of channels of the input tensor of the fourth sub-network larger will not increase the computational amount by much. Since the first sub-network is close to the input of the first decoding network, the computational amount of the first sub-network is usually represented by the product of the number of channels of the input tensor of the first decoding network and the number of channels of the output tensor of the first sub-network, and the number of channels of the input tensor of the first decoding network is usually large, so setting the number of channels of the output tensor of the first sub-network smaller can reduce the computational amount of the first sub-network. Similarly, the computational amount of the second sub-network located in the middle is usually represented by the product of the number of channels of the input tensor of the second sub-network and the number of channels of the output tensor of the second sub-network. Usually, the computational amount of the intermediate process is large, so setting the number of channels of the output tensor of the second sub-network smaller, that is, the number of channels of the input tensor of the third sub-network is smaller, thereby reducing the computational amount of the intermediate process.

[0006] That is to say, by combining the structure of the first decoding network, the number of channels of the input tensor of the fourth subnetwork is set to be greater than the number of channels of the input tensors of the third subnetwork and the second subnetwork, that is, the number of channels of the tensor close to the input side of the first decoding network is smaller, and the number of channels of the tensor close to the output side of the first decoding network is larger, thereby reducing the computational complexity of the first decoding network, and further ensuring that the quality of the image reconstructed by the first decoding network is not significantly reduced, avoiding the problem of high computational complexity of the first decoding network when the number of channels of the tensor close to the input layer of the first decoding network is large and the number of channels of the tensor close to the output side of the first decoding network is small.

[0007] In the JPEG-AI compression model, it is necessary to separate the color components of the image, that is, to separate the image to be encoded into an image of the Y component and an image of the UV component, so that the first decoding network can decode the data of the Y component and the data of the UV component. When the first decoding network is used to decode the data of the Y component, the number of channels of the input tensor of the fourth subnetwork is 96, and the number of channels of the input tensors of the third subnetwork and the second subnetwork are both 64. When the first decoding network is used to decode the data of the UV component, the number of channels of the input tensor of the fourth subnetwork is 128, and the number of channels of the input tensors of the third subnetwork and the second subnetwork are both 64.

[0008] The above is just an example. In some other embodiments, other values ​​can be set as long as the number of channels of the tensor close to the input side of the first decoding network is smaller and the number of channels of the tensor close to the output side of the first decoding network is larger. For example, when the first decoding network is used to decode the data of the Y component, the number of channels of the input tensor of the fourth subnetwork is 96, and the number of channels of the input tensors of the third subnetwork and the second subnetwork are both 64. When the first decoding network is used to decode the data of the UV component, the number of channels of the input tensor of the fourth subnetwork is 96, and the number of channels of the input tensors of the third subnetwork and the second subnetwork are both 64. In other words, for the UV component, the same number of channels as the Y component can be used, thereby simplifying the parameter configuration process of the first decoding network.

[0009] Optionally, at least two of the first subnetwork, the second subnetwork, and the third subnetwork include residual activation units, the residual activation units include activation layers, and the activation layers in the at least two subnetworks are different. In this way, activation layers with different flexibility can be selected according to different requirements, avoiding problems in the data processing process of the first decoding network due to the same activation layers.

[0010] Optionally, the first decoding network is used to decode data of the Y component, the first subnetwork, the second subnetwork and the third subnetwork all include the residual activation unit, the activation layer in the first subnetwork is a leaky rectified linear unit LeakyReLU, and the activation layers in the second subnetwork and the third subnetwork are both parameterized rectified linear units PReLU.

[0011] Optionally, the first decoding network is used to decode data of the UV component, the second subnetwork and the third subnetwork both include the residual activation unit, the activation layer in the second subnetwork is a rectified linear unit ReLU, and the activation layer in the third subnetwork is a leaky rectified linear unit LeakyReLU.

[0012] Since the flexibility of PReLU is greater than that of LeakyReLU, and the flexibility of LeakyReLU is greater than that of ReLU. Usually, the Y component is related to vision, that is, vision is more sensitive to the Y component, and the data processing of the Y component is more flexible, so the activation layer of the Y component can be set with higher flexibility, that is, PReLU and LeakyReLU can be used to make the decoding performance of the Y component better. However, vision is not very sensitive to the UV component, and the use of a more flexible activation layer may cause data overfitting, affecting the decoding performance of the UV component, and then affecting the overall decoding performance. Therefore, the activation layer of the UV component is set with lower flexibility, that is, LeakyReLU and ReLU can be used to avoid the problem of poor decoding performance caused by overfitting of the UV component data.

[0013] Optionally, if the first decoding network is used to decode data of the Y component, the first sub-network, the second sub-network and the third sub-network all include a residual activation unit; if the first decoding network is used to decode data of the UV component, the second sub-network and the third sub-network all include the residual activation unit;

[0014] The residual activation unit includes an activation layer, and the activation layer in the second sub-network and the third sub-network is the same, but different from the activation layer in the first sub-network.

[0015] Optionally, the activation layer in the first sub-network is a leaky rectified linear unit (LeakyReLU), and the activation layers in the second sub-network and the third sub-network are both parameterized rectified linear units (PReLU).

[0016] For the UV component, if the number of channels of the input tensor of the fourth sub-network is relatively large, the flexibility of setting the activation layer of the UV component is lower, that is, LeakyReLU and ReLU are selected, which can avoid the problem of poor decoding performance caused by overfitting of the UV component data. However, if the number of channels of the input tensor of the fourth sub-network is relatively small, such as the same as that of the Y component, set to 96, then the flexibility of the activation layer of the UV component will have less impact on the decoding performance. Therefore, the flexibility of the activation layer of the UV component can be set higher, that is, PReLU is selected. In this way, not only will the decoding performance not be greatly affected, but the parameter configuration process of the first decoding network is also simplified by using the same activation layer as the Y component.

[0017] Optionally, the residual activation unit further includes a grouped convolutional layer, and the number of groups of the grouped convolutional layer changes with the number of channels of an input tensor of the grouped convolutional layer.

[0018] Typically, the number of a group of data that can be processed by the hardware for implementing the grouped convolution layer is fixed. Therefore, in an embodiment of the present application, the number of groups is set to vary with the number of channels of the input tensor of the grouped convolution layer, so that when the number of channels of the input tensor is different, the number of groups of the grouped convolution layer is also different, thereby ensuring that the number of a group of data is consistent with the number of a group of data that the hardware can process, and having good compatibility with the hardware, reducing data processing delays, and avoiding the problem of high data processing delays caused by hardware incompatibility due to the number of groups being inconsistent with the number of a group of data that the hardware can process when the number of groups is fixed.

[0019] Optionally, the number of groups is obtained by dividing the number of channels of the input tensor of the grouped convolution layer by a target value, and the target value is an integer multiple of 16. For example, the target value is 16 or 32.

[0020] Since the number of a group of data that can be processed by the hardware implementing the grouped convolution layer is usually a multiple of 16, such as 16, 32, etc., in an embodiment of the present application, the number of groups of the grouped convolution layer is set equal to the number of channels of the input tensor divided by a multiple of 16. In this way, it can be ensured that the number of a group of data is a multiple of 16, thereby having better compatibility with the hardware, reducing data processing delay, and avoiding the problem of high data processing delay caused by incompatibility with hardware due to the number of groups being fixed and the number of a group of data obtained after grouping the input tensor is not a multiple of 16.

[0021] The first decoding network mentioned above can be a standard decoding network, that is, a decoding network constructed by parameters published by the image coding standard. In some embodiments, the decoder also includes a second decoding network, which is a user-defined decoding network, that is, a decoding network constructed by user-defined decoding rules. The second decoding network is used to decode the feature map to obtain a reconstructed image, and the feature map is decoded from the bitstream based on the super decoding network and / or context network included in the decoder.

[0022] In a second aspect, an encoder is provided, the encoder comprising a first encoding network, a second encoding network, a first super encoding network, a super decoding network and a context network, the first encoding network is used to determine a feature map of an image to be encoded, the first super encoding network is used to determine a super prior feature based on the feature map, the super decoding network and the context network are used to encode the super prior feature and the feature map into a bitstream;

[0023] The code stream is used for decoding by a decoder, and the decoder includes the super decoding network, the context network and a first decoding network, and the first decoding network is the first decoding network included in the decoder of any one of claims 1-13.

[0024] Optionally, the second encoding network is a pre-trained encoding network.

[0025] Optionally, the encoder further comprises a second super encoding network, wherein the second super encoding network is a pre-trained super encoding network.

[0026] In a third aspect, an image encoding method is provided, characterized in that it is applied to an encoder, the encoder includes a first encoding network, a second encoding network, a first super encoding network, a super decoding network and a context network, and the method includes:

[0027] Inputting the image to be encoded into the first encoding network to obtain a feature map;

[0028] Inputting the feature map into the first super encoding network to obtain a super prior feature;

[0029] By means of the super decoding network and the context network, the super prior features and the feature map are encoded into a bitstream;

[0030] The code stream is used for decoding by a decoder, and the decoder includes the super decoding network, the context network and a first decoding network. The first decoding network is the first decoding network included in the decoder described in the first aspect above.

[0031] Optionally, the second encoding network is a pre-trained encoding network.

[0032] Optionally, the encoder further comprises a second super encoding network, wherein the second super encoding network is a pre-trained super encoding network.

[0033] Since the first encoding network is a custom encoding network, the first decoding network is a standard decoding network, and the decoder also includes a second decoding network, and the second decoding network is a custom decoding network. After the image is encoded by the first encoding network, the code stream can be decoded by the first decoding network or by the second decoding network. When the code stream is decoded by the first decoding network, the standard decoding network can successfully decode the code stream encoded by the custom encoding network, ensuring the compatibility of the image encoding and decoding standard. When the code stream is decoded by the second decoding network, the custom decoding network can successfully decode the code stream encoded by the custom encoding network, achieving the optimal encoding and decoding performance.

[0034] In a fourth aspect, an image decoding method is provided, which is applied to the decoder described in the first aspect, wherein the decoder includes the first decoding network, the second decoding network, the super decoding network, and the context network, and the method includes:

[0035] Decode the super-prior features from the bitstream;

[0036] Based on the super prior features, decoding a feature map from the bitstream through the super decoding network and the context network;

[0037] The feature map is decoded by the second decoding network to obtain a reconstructed image.

[0038] Optionally, the first decoding network is a pre-trained decoding network.

[0039] Since the second decoding network is a custom decoding network, the code stream can be a code stream encoded by the custom encoding network or a code stream encoded by the standard encoding network. If the code stream is a code stream encoded by the standard encoding network, after being decoded by the second decoding network, the custom decoding network can successfully decode the code stream obtained by the standard encoding network, thereby ensuring the compatibility of the image encoding and decoding standards. If the code stream is a code stream encoded by the custom encoding network, after being decoded by the second decoding network, the custom decoding network can successfully decode the code stream encoded by the custom encoding network, thereby achieving the optimal encoding and decoding performance.

[0040] In a fifth aspect, an image coding device is provided, wherein the image coding device has the function of implementing the image coding method in the second aspect. The image coding device includes at least one module, and the at least one module is used to implement the image coding method provided in the second aspect.

[0041] In a sixth aspect, an image decoding device is provided, wherein the image decoding device has the function of implementing the image decoding method in the third aspect. The image decoding device includes at least one module, and the at least one module is used to implement the image decoding method provided in the third aspect.

[0042] In a seventh aspect, a coding end device is provided, the coding end device comprising a processor and a memory, the memory being used to store a computer program for executing the image coding method provided in the second aspect. The processor is configured to execute the computer program stored in the memory to implement the image coding method described in the second aspect.

[0043] Optionally, the encoding end device may further include a communication bus, and the communication bus is used to establish a connection between the processor and the memory.

[0044] In an eighth aspect, a decoding end device is provided, the decoding end device comprising a processor and a memory, the memory being used to store a computer program for executing the image decoding method provided in the third aspect. The processor is configured to execute the computer program stored in the memory to implement the image decoding method described in the third aspect.

[0045] Optionally, the decoding end device may further include a communication bus, and the communication bus is used to establish a connection between the processor and the memory.

[0046] In a ninth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the storage medium, and when the instructions are executed on a computer, the computer executes the steps of the image encoding method described in the second aspect or the steps of the image decoding method described in the third aspect. Alternatively, the computer-readable storage medium stores a code stream obtained by the method described in the third aspect executed by one or more processors.

[0047] In a tenth aspect, a computer program product comprising instructions is provided, and when the instructions are executed on a computer, the computer executes the steps of the image encoding method described in the second aspect or the steps of the image decoding method described in the third aspect. In other words, a computer program is provided, and when the computer program is executed on a computer, the computer executes the steps of the image encoding method described in the second aspect or the steps of the image decoding method described in the third aspect.

[0048] The technical effects obtained in the above-mentioned fourth to tenth aspects are similar to the technical effects obtained by the corresponding technical means in the first to third aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0050] Figure 2 It is a schematic diagram of an AI coding model;

[0051] Figure 3 It is a schematic diagram of an AI decoding model;

[0052] Figure 4 It is a schematic diagram of a JPEG-AI encoding model;

[0053] Figure 5 It is a schematic diagram of a JPEG-AI decoding model;

[0054] Figure 6 is a schematic diagram of the structure of a first decoding network provided in an embodiment of the present application;

[0055] Figure 7 It is a structural diagram of a residual activation unit;

[0056] Figure 8 It is a schematic diagram of a multi-branch encoding and decoding network structure provided in an embodiment of the present application;

[0057] Fig. 9 is a flowchart of an image encoding method provided by an embodiment of the present application;

[0058] Fig.10 This is a flowchart of an image decoding method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0060] For ease of understanding, before explaining in detail the decoder, encoder, and image encoding and decoding method provided in the embodiments of the present application, the terms, implementation environment, and application scenarios involved in the embodiments of the present application are first introduced.

[0061] First, the terms involved in the embodiments of the present application are explained.

[0062] Artificial intelligence (AI) is a technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0063] AI compression algorithm (AI Codec): refers to a data compression method based on deep learning technology.

[0064] JPEG AI: An image coding standard based on AI compression algorithm, which aims to create a deep learning-based image coding standard and provide a single-stream, compact compressed domain representation. Compared with the commonly used image coding standards for human visualization at the same subjective quality, it significantly improves the image compression efficiency and the performance of image processing and computer vision tasks.

[0065] Tensor: is a high-dimensional matrix. A three-dimensional tensor is a three-dimensional matrix. The three dimensions are width, height, and channel. Tensors are usually divided into input tensors and output tensors. Input tensors refer to the data input to the network layer based on deep learning, and output tensors refer to the data output by the network layer based on deep learning.

[0066] Generally, the tensor output by the encoding network is also called a latent variable, which can be restored to the original data or data similar to the original data through the decoding network. The tensor output by the super encoding network is also called a super latent variable, which is obtained by further encoding the latent variable. The super latent variable can be restored to the latent variable or data similar to the latent variable through the super decoding network.

[0067] Rectified linear unit (ReLU): It is a commonly used activation function in deep learning, usually referring to nonlinear functions represented by ramp functions and their variants. Commonly used activation functions include leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU), etc.

[0068] ReLU sets all negative values ​​to zero, while LeakyReLU gives all negative values ​​a non-zero slope. PReLU is a variant of LeakyReLU, where the slope of the negative part is determined based on the data rather than being predefined. In general, both LeakyReLU and PReLU perform better than ReLU, and PReLU is more flexible than LeakyReLU, which is more flexible than ReLU.

[0069] BD-Rate: full name It is used to evaluate the rate-distortion (RD) performance of different video encoders. In the performance measurement between compression methods, the BD-Rate of method A compared to method B indicates the difference in bitrate between method A and method B under the same objective indicators. BD-Rate is usually expressed as a percentage. If it is -x%, it means that method A can save x% of space compared to method B; if it is x%, it means that x% of space is increased.

[0070] Peak signal to noise ratio (PSNR): is an objective standard for evaluating image quality. The PSNR value range is greater than zero. The larger the PSNR value, the better the image quality is, and the more similar the reconstructed image is to the original image.

[0071] Multi-scale structural similarity (MS-SSIM): It is a structural similarity (SSIM) index based on multiple scales (i.e., the images are scaled from large to small according to certain rules). The value range of MS-SSIM is 0-1. The larger the MS-SSIM value, the better the image quality is, and the more similar the reconstructed image is to the original image.

[0072] In some embodiments, PSNR and MS-SSIM are two indicators used to measure image reconstruction quality.

[0073] Conditional color separation (CCS): A coding method that encodes the Y component and the UV component separately, and can be conditioned on other components when encoding and decoding different components.

[0074] Secondly, the implementation environment involved in the embodiments of the present application is introduced.

[0075] Please refer to Figure 1 , Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. The implementation environment includes a source device 10, a destination device 20, a link 30, and a storage device 40. Among them, the source device 10 can generate an encoded image. Therefore, the source device 10 can also be referred to as an image encoding device or an encoding end. The destination device 20 can decode the encoded image generated by the source device 10. Therefore, the destination device 20 can also be referred to as an image decoding device or a decoding end. The link 30 can receive the encoded image generated by the source device 10, and can transmit the encoded image to the destination device 20. The storage device 40 can receive the encoded image generated by the source device 10, and can store the encoded image. Under such conditions, the destination device 20 can directly obtain the encoded image from the storage device 40. Alternatively, the storage device 40 can correspond to a file server or another intermediate storage device that can store the encoded image generated by the source device 10. Under such conditions, the destination device 20 can stream or download the encoded image stored in the storage device 40.

[0076] The source device 10 and the destination device 20 may each include one or more processors and a memory coupled to the one or more processors, the memory may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, any other medium that can be used to store desired program code in the form of instructions or data structures that can be accessed by a computer, etc. For example, the source device 10 and the destination device 20 may each include a mobile phone, a smart phone, a personal digital assistant (PDA), a wearable device, a pocket PC (PPC), a tablet computer, a smart car machine, a smart TV, a smart speaker, a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a telephone handset such as a so-called "smart" phone, a television, a camera, a display device, a digital media player, a video game console, a car computer, or the like.

[0077] The link 30 may include one or more media or devices capable of transmitting the encoded image from the source device 10 to the destination device 20. In one possible implementation, the link 30 may include one or more communication media that enable the source device 10 to directly send the encoded image to the destination device 20 in real time. In an embodiment of the present application, the source device 10 may modulate the encoded image based on a communication standard, which may be a wireless communication protocol, etc., and may send the modulated image to the destination device 20. The one or more communication media may include wireless and / or wired communication media, for example, the one or more communication media may include a radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, which may be a local area network, a wide area network, or a global network (e.g., the Internet), etc. The one or more communication media may include routers, switches, base stations, or other devices that facilitate communication from the source device 10 to the destination device 20, etc., and the embodiment of the present application does not specifically limit this.

[0078] In a possible implementation, the storage device 40 may store the received encoded image sent by the source device 10, and the destination device 20 may directly obtain the encoded image from the storage device 40. Under such conditions, the storage device 40 may include any of a variety of distributed or locally accessible data storage media, for example, any of the multiple distributed or locally accessible data storage media may be a hard disk drive, a Blu-ray disc, a digital versatile disc (DVD), a compact disc read-only memory (CD-ROM), a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded images.

[0079] In one possible implementation, the storage device 40 may correspond to a file server or another intermediate storage device that can store the encoded image generated by the source device 10, and the destination device 20 may stream or download the image stored in the storage device 40. The file server may be any type of server that can store the encoded image and send the encoded image to the destination device 20. In one possible implementation, the file server may include a network server, a file transfer protocol (FTP) server, a network attached storage (NAS) device, or a local disk drive, etc. The destination device 20 may obtain the encoded image through any standard data connection (including an Internet connection). Any standard data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of the two suitable for obtaining the encoded image stored on the file server. The transmission of the encoded image from the storage device 40 may be streaming transmission, download transmission, or a combination of the two.

[0080] Figure 1 The implementation environment shown is only one possible implementation method, and the technology of the embodiment of the present application can be applied not only to Figure 1 The source device 10 that can encode an image and the destination device 20 that can decode the encoded image shown can also be applicable to other devices that can encode an image and decode the encoded image, and the embodiments of the present application do not specifically limit this.

[0081] exist Figure 1In the illustrated implementation, the source device 10 includes a data source 120, an encoder 100, and an output interface 140. In some embodiments, the output interface 140 may include a modulator / demodulator (modem) and / or a transmitter, where the transmitter may also be referred to as a transmitter. The data source 120 may include an image capture device (e.g., a camera, etc.), an archive containing previously captured images, a feed interface for receiving images from an image content provider, and / or a computer graphics system for generating images, or a combination of these sources of images.

[0082] The data source 120 may send an image to the encoder 100, and the encoder 100 may encode the image received from the data source 120 to obtain an encoded image. The encoder may send the encoded image to the output interface. In some embodiments, the source device 10 directly sends the encoded image to the destination device 20 via the output interface 140. In other embodiments, the encoded image may also be stored in the storage device 40 for the destination device 20 to obtain and use for decoding and / or display later.

[0083] exist Figure 1 In the illustrated implementation environment, the destination device 20 includes an input interface 240, a decoder 200, and a display device 220. In some embodiments, the input interface 240 includes a receiver and / or a modem. The input interface 240 may receive an encoded image via the link 30 and / or from the storage device 40, and then send it to the decoder 200. The decoder 200 may decode the received encoded image to obtain a decoded image. The decoder may send the decoded image to the display device 220. The display device 220 may be integrated with the destination device 20 or may be external to the destination device 20. In general, the display device 220 displays the decoded image. The display device 220 may be any of a variety of types of display devices, for example, the display device 220 may be a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0084] although Figure 11 and 2. Although not shown in the drawings, in some aspects, encoder 100 and decoder 200 may be integrated with an encoder and decoder, respectively, and may include appropriate multiplexer-demultiplexer (MUX-DEMUX) units or other hardware and software for encoding both audio and video in a common data stream or in separate data streams. In some embodiments, if applicable, the MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol, or other protocols such as the user datagram protocol (UDP).

[0085] The encoder 100 and the decoder 200 may each be any of the following circuits: one or more microprocessors, digital signal processors (digital signal processing, DSP), application specific integrated circuits (application specific integrated circuits, ASIC), field programmable gate arrays (field-programmable gate array, FPGA), discrete logic, hardware or any combination thereof. If the technology of the embodiment of the present application is partially implemented in software, the device may store instructions for the software in a suitable non-volatile computer-readable storage medium, and may use one or more processors to execute the instructions in hardware to implement the technology of the embodiment of the present application. Any of the foregoing contents (including hardware, software, a combination of hardware and software, etc.) may be regarded as one or more processors. Each of the encoder 100 and the decoder 200 may be included in one or more encoders or decoders, and any of the encoders or decoders may be integrated as part of a combined encoder / decoder (encoder-decoder) in a corresponding device.

[0086] The embodiments of the present application may generally refer to the encoder 100 as "signaling" or "sending" certain information to another device, such as the decoder 200. The term "signaling" or "sending" may generally refer to the transmission of syntax elements and / or other data used to decode the compressed image. This transmission may occur in real time or near real time. Alternatively, this communication may occur over a period of time, such as when the syntax elements are stored in the encoded bitstream to a computer-readable storage medium at the time of encoding, and the decoding device may then retrieve the syntax elements at any time after the syntax elements are stored to this medium.

[0087] It should be noted that the application scenarios and implementation environments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of application scenarios and implementation environments, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0088] Finally, the application scenarios involved in the embodiments of the present application are introduced.

[0089] Coding and decoding technology refers to a technology that uses data characteristics such as spatial redundancy, visual redundancy and statistical redundancy to represent the original signal data with fewer bits in a lossy or lossless manner. It can achieve effective transmission and storage of information such as images / videos / audio, and plays an important role in the current media era where the types and amounts of information transmitted or stored are increasing.

[0090] Taking image compression as an example, image compression is divided into lossy compression and lossless compression. Lossy compression achieves a larger compression ratio at the cost of a certain degree of image quality degradation, while lossless compression does not cause loss of image details. Traditional image / video compression algorithms have undergone decades of development and have formed mature compression standards such as high efficiency video coding (HEVC) and versatile video coding (VVC). Traditional image / video compression algorithms have the advantages of comprehensive support functions, good versatility, and good hardware support, and have been widely used, but they have also encountered bottlenecks in improving coding efficiency.

[0091] As deep learning outperforms traditional algorithms in many computer vision tasks such as image recognition and target detection, more and more researchers are beginning to explore image / video compression methods based on deep learning. Different from traditional image algorithms that encode, quantize, entropy encode, etc. images through artificial design and optimize them separately, the modules in the AI ​​image compression algorithm (encoding network, entropy estimation network, decoding network, etc.) are optimized together as a whole, so the compression effect of the AI ​​image compression scheme is better. However, deep learning is a resource-intensive algorithm that not only requires a lot of computing costs, but also consumes a lot of memory. Although computing resources are increasing, optimizing the training and reasoning of deep learning networks is still very important for the implementation of models. In particular, more and more models will be moved from the server side to resource-limited devices such as edge devices, such as smartphones and embedded devices. How to deploy complex models on resource-limited devices is a problem that must be solved by current deep learning technology.

[0092] In the image compression method based on deep learning, AI compression model and JPEG-AI compression model are usually involved. For the sake of easy understanding, the encoding and decoding process of these two compression models is briefly introduced here.

[0093] (1) AI compression model

[0094] Please refer to Figure 2 , Figure 2 It is a schematic diagram of an AI coding model. The AI ​​coding model is also called an encoder, which is applied to the encoding end. When encoding, the image x to be encoded is input into the encoding network to obtain the feature map y, and the feature map y is input into the super encoding network to obtain the super prior feature z. The super prior feature z is entropy encoded according to the specified probability distribution to encode the super prior feature z into the bitstream. In addition, the super prior feature z is input into the super decoding network to obtain the prior feature p and the variance σ. Based on the prior feature p and the feature map y, the mean μ is obtained through the context network, and then based on the probability distribution N(μ, σ), the feature map y is entropy encoded to encode the feature map y into the bitstream.

[0095] Among them, the bit sequence obtained by entropy encoding the super prior feature z is a partial bit sequence included in the code stream, which can be called a super prior bit stream, denoted by bs z The bit sequence obtained by entropy encoding the feature map y is a partial bit sequence included in the code stream. This partial bit sequence can be called an image bit stream, denoted by bs y .

[0096] In some embodiments, the context network includes a context model and a probability distribution estimation model. In this case, the feature map y can be input into the context model to obtain the context feature φ. In combination with the prior feature p and the context feature φ, the mean μ is estimated by the probability distribution estimation model. Of course, this is just an example. In practical applications, it can also be implemented in other ways, and the embodiments of the present application do not limit this. Moreover, the above is introduced by taking entropy coding as an example. In practical applications, it can also be encoded by other encoding methods, and the embodiments of the present application do not limit this either.

[0097] Please refer to Figure 3 , Figure 3 It is a schematic diagram of an AI decoding model. This AI decoding model is also called a decoder. It is applied to the decoding end. When decoding, it calculates the super-prior bit stream bs included in the bit stream according to the specified probability distribution. z Perform entropy decoding to obtain super-prior features The super prior feature Input to the super decoding network to obtain prior features And variance σ. Determine the side information from the decoded partial image features and combine the prior features And side information, through the context network to get the mean μ, based on the probability distribution N(μ,σ), from the image bit stream bs included in the code stream y The feature map is decoded by entropy The feature map Input to the decoding network to get the reconstructed image

[0098] In some embodiments, the context network includes a context model and a probability distribution estimation model. In this case, based on the prior features The probability distribution estimation model can be used to estimate the mean μ corresponding to some image features. Based on the probability distribution N(μ,σ) of these image features, the image bit stream bs included in the code stream is obtained. y For other image features, the side information is determined from the decoded image features and input into the context model to obtain the context features. Combining prior features and contextual features The mean μ corresponding to other image features is estimated by the probability distribution estimation model. Of course, this is just an example, and it can be implemented in other ways in actual applications, which is not limited in the embodiments of the present application. Moreover, the above is introduced by taking entropy decoding as an example, and in actual applications, it can also be decoded by other decoding methods corresponding to the encoding method, which is also not limited in the embodiments of the present application.

[0099] It should be noted that if the above-mentioned probability distribution estimation network uses a Gaussian model (such as a single Gaussian model or a mixed Gaussian model) to model, the estimated probability distribution includes a mean and a variance. If the above-mentioned probability distribution estimation network uses a Laplace distribution model to model, the estimated probability distribution includes a location parameter and a scale parameter. If the above-mentioned probability distribution estimation network uses a logistic distribution model to model, the estimated probability distribution includes a mean and a scale parameter. The above is explained using a Gaussian model as an example.

[0100] In addition, the super decoding network includes a super decoding prediction network and a super scale decoding network, wherein the super decoding prediction network is used to determine the prior features, and the super scale decoding network is used to determine the variance. That is, the super prior features are input into the super decoding prediction network to obtain the prior features, and the super prior features are input into the super scale decoding network to obtain the variance.

[0101] (2) JPEG-AI compression model

[0102] The JPEG-AI compression model is based on the autoencoder structure of color component separation. It realizes the encoding and decoding of the image by encoding and decoding each separated component of the image to be encoded. Moreover, the basic structure of the JPEG-AI compression model is the same as the structure of the above-mentioned AI compression model, but the image input component of the JPEG-AI compression model is the YUV component, and the Y component and UV component are encoded and decoded separately. Among them, when encoding, the Y component is used alone as mentioned above. Figure 2 The architecture shown in the figure is used for encoding, and the UV component is encoded by combining the data of the Y component. When decoding, the Y component is encoded separately using the above Figure 3 The architecture shown is used for decoding, and the UV component is decoded by combining the data of the Y component.

[0103] Please refer to Figure 4 , Figure 4 This is a schematic diagram of a JPEG-AI encoding model. The JPEG-AI encoding model is also called an encoder. It is applied to the encoding end. When encoding, the image x in RGB (a color mode, where R represents red, G represents green, and B represents blue) format is converted into an image in YUV (a color encoding method, where Y represents brightness, i.e. grayscale value, and U and V represent chrominance) format through the color space, and then the color components of the YUV format image are separated to obtain the Y component image x Y and the UV component of the image x UV , the Y component of the image x Y Input to the encoding network to obtain the feature map y of the Y component Y , the feature map y of the Y component Y Input to the super encoding network to obtain the super prior feature z of the Y component Y , according to the specified probability distribution of the Y component of the hyper-prior feature z Y Entropy coding is performed to convert the super-prior feature z of the Y component into Y The super prior feature z of the Y component is encoded into the bitstream. Y Input to the super decoding network to obtain the prior feature p of the Y component Y and variance σ Y , based on the prior feature p Y and variance σ Y , the mean μ is obtained through the context network Y , and then based on the probability distribution N(μ Y , σ Y ), the feature map y of the Y component Y Entropy coding is performed to convert the feature map y of the Y component Y Encode into the code stream.

[0104] In addition, the Y component of the image x YAuxiliary image x' converted to UV components after downsampling UV , the UV component image x UV And the auxiliary image x' of the UV component UV Input to the encoding network to obtain the feature map y of the UV component UV , the characteristic map y of the UV component UV Input to the super encoding network to obtain the super prior feature z of the UV component UV , according to the specified probability distribution of the ultra-prior feature z of the UV component UV Entropy coding is performed to convert the ultra-prior feature z of the UV component UV The ultra-prior feature z of the UV component is encoded into the bitstream. UV Input to the super decoding network to obtain the prior feature p of the UV component UV and variance σ UV , based on the prior feature p of the UV component UV and feature map y UV , the mean value μ of the UV component is obtained through the context network UV , and then based on the probability distribution N(μ UV , σ UV ), the characteristic map y of the UV component UV Entropy encoding is performed to convert the feature map y of the UV component UV Encode into the code stream.

[0105] Among them, the super prior feature z of the Y component Y The bit sequence obtained by entropy coding is a partial bit sequence included in the code stream, which can be called the super-prior bit stream bs of the Y component. zY . The feature map y of the Y component Y The bit sequence obtained by entropy coding is a partial bit sequence included in the code stream, which can be called the image bit stream bs of the Y component. yY The hyper-prior feature z of the UV component UV The bit sequence obtained by entropy coding is a partial bit sequence included in the code stream, which can be called the super prior bit stream bs of the UV component. zUV . Characteristic map y for UV component UV The bit sequence obtained by entropy coding is a partial bit sequence included in the code stream, which can be called the image bit stream bs of the UV component. yUV .

[0106] Please refer to Figure 5 , Figure 5The JPEG-AI decoding model is a schematic diagram of a JPEG-AI decoding model. The JPEG-AI decoding model is also called a decoder. It is applied to the decoding end. When decoding, the super-prior bit stream bs of the Y component included in the bit stream is calculated according to the specified probability distribution. zY Perform entropy decoding to obtain the super-prior features of the Y component The hyper-prior feature of the Y component Input to the super decoding network to obtain the prior features of the Y component and variance σ Y Determine the side information from the decoded partial image features, combined with the prior features of the Y component And side information, the mean μ of the Y component is obtained through the context network Y , based on the probability distribution of the Y component N(μ Y , σ Y ), entropy decode the feature map of the Y component from the image bit stream of the Y component included in the code stream The feature map of the Y component Input to the decoding network to obtain the reconstructed Y component image

[0107] In addition, the feature map y of the Y component Y Auxiliary feature y' converted to UV component after downsampling UV Then, according to the specified probability distribution, the super-prior bit stream bs of the UV component included in the code stream is zUV Perform entropy decoding to obtain the ultra-prior features of the UV component The ultra-prior features of the UV component Input to the super decoding network to obtain the prior features of the UV component and variance σ UV Determine the side information from the decoded partial image features, combined with the prior features of the UV component and side information, the mean value μUV of the UV component is obtained through the context network, and the probability distribution N(μ UV , σ UV ), entropy decode the feature map of the UV component from the image bit stream of the UV component included in the code stream The characteristic map of the UV component and the auxiliary feature y' of the UV component UV Input to the decoding network to obtain the reconstructed UV component image

[0108] Afterwards, the reconstructed Y component image is filtered through the Inter Channel Correlation Information (ICCI) network. and UV component of the image Processing is performed to obtain a reconstructed image in YUV format, and finally the reconstructed image in YUV format is converted into RGB format through inverse color space conversion to obtain a reconstructed image

[0109] The above introduces the general process of encoding and decoding of the JPEG-AI compression model. For some details, please refer to the above explanations about the AI ​​compression model, which will not be repeated here.

[0110] Whether it is the above-mentioned AI compression model or the JPEG-AI compression model, a decoding network is involved. The computational complexity of the decoding network is relatively high. In order to reduce the computational complexity, an embodiment of the present application provides a decoder. The decoder includes a first decoding network, a super decoding network, and a context network. Please refer to Figure 6 , Figure 6 It is a structural diagram of a first decoding network provided in an embodiment of the present application. The first decoding network includes a first subnetwork, a second subnetwork, a third subnetwork and a fourth subnetwork connected in sequence from input to output, the number of channels of the input tensor of the fourth subnetwork is greater than the number of channels of the input tensor of the third subnetwork, and the number of channels of the input tensor of the fourth subnetwork is greater than the number of channels of the input tensor of the second subnetwork.

[0111] Since the first decoding network includes the first sub-network, the second sub-network, the third sub-network and the fourth sub-network connected in sequence from input to output, the input tensor of the first sub-network is the input tensor of the first decoding network, the output tensor of the first sub-network is the input tensor of the second sub-network, the output tensor of the second sub-network is the input tensor of the third sub-network, the input tensor of the third sub-network is the output tensor of the fourth sub-network, and the output tensor of the fourth sub-network is the output tensor of the first decoding network.

[0112] It should be noted that the "connected in sequence" involved in the embodiments of the present application refers to a logical connection, not a physical connection. In other words, "connected in sequence" means that the output tensor of the previous network is the input tensor of the next network. The subsequent connection of network layers is the same.

[0113] For ease of description, Figure 6 As shown, the number of channels of the input tensor of the first decoding network is recorded as C, that is, the number of channels of the input tensor of the first sub-network is C. The number of channels of the output tensor of the first sub-network is recorded as C3, that is, the number of channels of the input tensor of the second sub-network is C3. The number of channels of the output tensor of the second sub-network is recorded as C2, that is, the number of channels of the input tensor of the third sub-network is C2. The number of channels of the output tensor of the third sub-network is recorded as C1, that is, the number of channels of the input tensor of the fourth sub-network is C1. The number of channels of the output tensor of the fourth sub-network is recorded as C in, that is, the number of channels of the output tensor of the first decoding network is C in .

[0114] Since the fourth sub-network is close to the output of the first decoding network, the computational complexity of the fourth sub-network is usually represented by the product of the number of channels of the input tensor of the fourth sub-network and the number of channels of the output tensor of the first decoding network, that is, C1*C in , and the number of channels of the output tensor of the first decoding network is usually small, so setting the number of channels of the input tensor of the fourth sub-network to be larger will not increase the amount of calculation much. Since the first sub-network is close to the input of the first decoding network, the amount of calculation of the first sub-network is usually represented by the product of the number of channels of the input tensor of the first decoding network and the number of channels of the output tensor of the first sub-network, that is, C*C3, and the number of channels of the input tensor of the first decoding network is usually large, so setting the number of channels of the output tensor of the first sub-network to be smaller can reduce the amount of calculation of the first sub-network. Similarly, the amount of calculation of the second sub-network in the middle is usually represented by the product of the number of channels of the input tensor of the second sub-network and the number of channels of the output tensor of the second sub-network, that is, C3*C2. Usually, the amount of calculation in the intermediate process is large, so setting the number of channels of the output tensor of the second sub-network to be smaller, that is, setting the number of channels of the input tensor of the third sub-network to be smaller, thereby reducing the amount of calculation in the intermediate process.

[0115] That is to say, by combining the structure of the first decoding network, the number of channels of the input tensor of the fourth subnetwork is set to be greater than the number of channels of the input tensors of the third subnetwork and the second subnetwork, that is, the number of channels of the tensor close to the input side of the first decoding network is smaller, and the number of channels of the tensor close to the output side of the first decoding network is larger, thereby reducing the computational complexity of the first decoding network, and further ensuring that the quality of the image reconstructed by the first decoding network is not significantly reduced, avoiding the problem of high computational complexity of the first decoding network when the number of channels of the tensor close to the input layer of the first decoding network is large and the number of channels of the tensor close to the output side of the first decoding network is small.

[0116] Based on the above description, in the JPEG-AI compression model, it is necessary to separate the color components of the image, that is, to separate the image to be encoded into an image of the Y component and an image of the UV component, so that the first decoding network can decode the data of the Y component and the data of the UV component. When the first decoding network is used to decode the data of the Y component, the number of channels of the input tensor of the fourth subnetwork is 96, and the number of channels of the input tensors of the third subnetwork and the second subnetwork are both 64. When the first decoding network is used to decode the data of the UV component, the number of channels of the input tensor of the fourth subnetwork is 128, and the number of channels of the input tensors of the third subnetwork and the second subnetwork are both 64.

[0117] For example, the number of channels of the input tensors of the fourth sub-network, the third sub-network, and the second sub-network included in the first decoding network can be illustrated by the following Table 1.

[0118] Table 1

[0119] Y component UV Component C 160 32 <![CDATA[C3]]> 64 64 <![CDATA[C2]]> 64 64 <![CDATA[C1]]> 96 128

[0120] It should be noted that C is the number of channels of the input tensor of the first decoding network, and the input tensor of the first decoding network is usually the output tensor of other networks. Therefore, the embodiment of the present application does not adjust the number of channels of the input tensor of the first decoding network.

[0121] The channel numbers shown in Table 1 above are only examples. In some other embodiments, other values ​​can be set as long as the channel number of the tensor close to the input side of the first decoding network is smaller and the channel number of the tensor close to the output side of the first decoding network is larger. For example, when the first decoding network is used to decode the data of the Y component, the number of channels of the input tensor of the fourth subnetwork is 96, and the number of channels of the input tensors of the third subnetwork and the second subnetwork are both 64. When the first decoding network is used to decode the data of the UV component, the number of channels of the input tensor of the fourth subnetwork is 96, and the number of channels of the input tensors of the third subnetwork and the second subnetwork are both 64. In other words, for the UV component, the same number of channels as the Y component can be used, thereby simplifying the parameter configuration process of the first decoding network.

[0122] In the above Figure 6 In the case where the index number (the syntax element is compIdx) is equal to 0, it indicates that the first decoding network is used to decode the data of the Y component, and when the index number is equal to 1, it indicates that the first decoding network is used to decode the data of the UV component. When the first decoding network is used to decode the data of the Y component, the first subnetwork of the first decoding network includes a residual module, a first deconvolution layer, a first cropping layer, and a first residual activation unit. When the first decoding network is used to decode the data of the UV component, the first subnetwork of the first decoding network includes a synthesis module. Regardless of whether the first decoding network is used to decode the data of the Y component or the data of the UV component, the second subnetwork of the first decoding network includes a second deconvolution layer, a second cropping layer, and a second residual activation unit, the third subnetwork includes a first convolution layer and a third residual activation unit, and the fourth subnetwork includes a second convolution layer, a pixel shifting layer, and a third cropping layer.

[0123] That is, when the first decoding network is used to decode the data of the Y component, the first sub-network, the second sub-network and the third sub-network all include a residual activation unit, and when the first decoding network is used to decode the data of the UV component, the second sub-network and the third sub-network all include a residual activation unit. That is, at least two of the first sub-network, the second sub-network and the third sub-network include a residual activation unit.

[0124] Please refer to Figure 7 , Figure 7 The present invention is a schematic diagram of the structure of a residual activation unit. The residual activation unit includes an activation layer, a grouped convolution layer and a convolution layer. In the case where at least two sub-networks among the first sub-network, the second sub-network and the third sub-network include residual activation units, the activation layers in the at least two sub-networks are different. In this way, activation layers with different flexibility can be selected according to different requirements, avoiding problems in the data processing process of the first decoding network due to the same activation layers.

[0125] In some embodiments, when the first decoding network is used to decode data of the Y component, the first subnetwork, the second subnetwork, and the third subnetwork all include residual activation units, the activation layer in the first subnetwork is LeakyReLU, and the activation layers in the second subnetwork and the third subnetwork are both PReLU.

[0126] In some other embodiments, the first decoding network is used to decode data of the UV component, the second sub-network and the third sub-network both include residual activation units, the activation layer in the second sub-network is ReLU, and the activation layer in the third sub-network is LeakyReLU.

[0127] Based on the above description, the flexibility of PReLU is greater than that of LeakyReLU, and the flexibility of LeakyReLU is greater than that of ReLU. Usually, the Y component is related to vision, that is, vision is more sensitive to the Y component, and the data processing of the Y component is more flexible, so the activation layer of the Y component can be set with higher flexibility, that is, PReLU and LeakyReLU can be used to make the decoding performance of the Y component better. However, vision is not very sensitive to the UV component, and the use of a more flexible activation layer may cause data overfitting, affecting the decoding performance of the UV component, and then affecting the overall decoding performance. Therefore, the activation layer of the UV component is set with lower flexibility, that is, LeakyReLU and ReLU can be used to avoid the problem of poor decoding performance caused by overfitting of the UV component data.

[0128] For example, the activation layers in the first sub-network, the second sub-network, and the third sub-network may be illustrated by the following Table 2.

[0129] Table 2

[0130] Y component UV Component First subnetwork LeakyReLU - Second sub-network PReLU ReLU The third sub-network PReLU LeakyReLU

[0131] It should be noted that when the first decoding network is used to decode the data of the UV component, the first network does not include a residual activation unit, so the activation layer of the first network corresponding to the UV component is not shown in the above Table 2.

[0132] The type of activation layer included in each sub-network shown in Table 2 above is only an example. In some other embodiments, other types of activation layers may be set, as long as the activation layer of the Y component is different and has higher flexibility, and the activation layer of the UV component is different and has lower flexibility.

[0133] In other embodiments, the activation layer in the second subnetwork and the third subnetwork is the same, but different from the activation layer in the first subnetwork. For example, the activation layer in the first subnetwork is LeakyReLU, and the activation layers in the second subnetwork and the third subnetwork are both PReLU. That is, when the first decoding network is used to decode the data of the Y component, the first subnetwork, the second subnetwork, and the third subnetwork all include residual activation units, and the activation layer in the first subnetwork is LeakyReLU, and the activation layers in the second subnetwork and the third subnetwork are both PReLU. When the first decoding network is used to decode the data of the UV component, the second subnetwork and the third subnetwork all include residual activation units, and the activation layers in the second subnetwork and the third subnetwork are both PReLU.

[0134] For the UV component, if the number of channels of the input tensor of the fourth sub-network is relatively large, the flexibility of setting the activation layer of the UV component is lower, that is, LeakyReLU and ReLU are selected, which can avoid the problem of poor decoding performance caused by overfitting of the UV component data. However, if the number of channels of the input tensor of the fourth sub-network is relatively small, such as the same as that of the Y component, set to 96, then the flexibility of the activation layer of the UV component will have less impact on the decoding performance. Therefore, the flexibility of the activation layer of the UV component can be set higher, that is, PReLU is selected. In this way, not only will the decoding performance not be greatly affected, but the parameter configuration process of the first decoding network is also simplified by using the same activation layer as the Y component.

[0135] Based on the above description, the residual activation unit also includes a grouped convolution layer, and the number of groups of the grouped convolution layer changes with the number of channels of the input tensor of the grouped convolution layer.

[0136] Typically, the number of a group of data that can be processed by the hardware for implementing the grouped convolution layer is fixed. Therefore, in an embodiment of the present application, the number of groups is set to vary with the number of channels of the input tensor of the grouped convolution layer, so that when the number of channels of the input tensor is different, the number of groups of the grouped convolution layer is also different, thereby ensuring that the number of a group of data is consistent with the number of a group of data that the hardware can process, and having good compatibility with the hardware, reducing data processing delays, and avoiding the problem of high data processing delays caused by hardware incompatibility due to the number of groups being inconsistent with the number of a group of data that the hardware can process when the number of groups is fixed.

[0137] In some embodiments, the number of groups is obtained by dividing the number of channels of the input tensor of the grouped convolution layer by a target value, and the target value is an integer multiple of 16. For example, the target value is 16 or 32.

[0138] Since the number of a group of data that can be processed by the hardware implementing the grouped convolution layer is usually a multiple of 16, such as 16, 32, etc., in an embodiment of the present application, the number of groups of the grouped convolution layer is set equal to the number of channels of the input tensor divided by a multiple of 16. In this way, it can be ensured that the number of a group of data is a multiple of 16, thereby having better compatibility with the hardware, reducing data processing delay, and avoiding the problem of high data processing delay caused by incompatibility with hardware due to the number of groups being fixed and the number of a group of data obtained after grouping the input tensor is not a multiple of 16.

[0139] Based on the above description, the first decoding network includes a first deconvolution layer and a second deconvolution layer, and the structures of the first deconvolution layer and the second deconvolution layer are both CONV-1 (4×4) structures, that is, the first deconvolution layer and the second deconvolution layer are both structures that upsample the input tensor by 2 times. In an embodiment of the present application, other structures that can upsample the input tensor by 2 times can also be used to replace the first deconvolution layer and the second deconvolution layer. Similarly, the first decoding network includes a first convolution layer and a second convolution layer, and the structures of the first convolution layer and the second convolution layer are both CONV (3×3) structures, that is, the first convolution layer and the second convolution layer are both structures that keep the input tensor unchanged. In an embodiment of the present application, other structures that can keep the input tensor unchanged can also be used to replace the first convolution layer and the second convolution layer.

[0140] When the input tensors of the first deconvolution layer and the second deconvolution layer are three-dimensional tensors, the first deconvolution layer and the second deconvolution layer are structures that upsample the length and width of the input tensor by a factor of 2. Similarly, when the input tensors of the first convolution layer and the second convolution layer are three-dimensional tensors, the first convolution layer and the second convolution layer are structures that keep the length and width of the input tensor unchanged.

[0141] Next, the performance of the first decoding network provided in the embodiment of the present application will be illustrated through the following example.

[0142] Assuming that the number of channels of the input tensors of the fourth sub-network, the third sub-network and the second sub-network included in the first decoding network provided in the embodiment of the present application is as shown in Table 1 above, at 4 bit rate points, compared with the VVC encoding standard, the BD-Rate of the existing decoding network and the first decoding network of the embodiment of the present application is compared, as shown in Table 3 below.

[0143] Table 3

[0144]

[0145] The first row of data in Table 3 above is the test situation of the existing decoding network, and the second row of data is the test situation of the first decoding network provided in the embodiment of the present application. The evaluation indicators considered in Table 3 above include multiple-scale structural similarity (MSSSIM), visual quality fidelity (VIF), feature similarity (FSIM), normalization laplacian pyramid distance (NLPD), information content weighted structural similarity (IW-SSIM), video multi-method assessment fusion (VMAF), PSNR-human visual system (HSV), monotonicity, maximum multiply accumulate (MAC) / pixel, and average MAC / pxl. Among them, the average distortion rate is determined based on the values ​​of multiple indicators for evaluating image reconstruction quality, such as MSSSIM, VIF, FSIM, IW-SSIM, VMAF and PSNR-HSV, which can reflect the effect of reconstructing the image by the decoding network compared with the VVC method.

[0146] It can be seen from Table 3 that the computational complexity of the first decoding network provided in the embodiment of the present application is reduced compared to the existing decoding network, that is, both Max kMAC / Pxl and AVG kMAC / Pxl are reduced, and the BD-Rate on the test image has basically no significant change. In other words, the first decoding network provided in the embodiment of the present application can reduce the computational complexity and has no significant impact on the quality of the reconstructed image.

[0147] The first decoding network mentioned above can be a standard decoding network, that is, a decoding network constructed by parameters released by the image coding standard. In some embodiments, the decoder also includes a second decoding network, which is a user-defined decoding network, that is, a decoding network constructed by user-defined decoding rules. The second decoding network is used to decode the feature map to obtain a reconstructed image, and the feature map is obtained by decoding the bitstream based on the super decoding network and / or context network included in the decoder. The super decoding network and the context network can be standard networks, that is, networks constructed by parameters sent by the image coding standard.

[0148] That is to say, the decoder includes two decoding networks, namely a standard decoding network and a custom decoding network, and the two decoding networks share a standard super decoding network and a context network. Of course, for the second decoding network, one of the standard super decoding network and the context network can also be shared, and the other can be constructed using custom rules.

[0149] When the image coding and decoding standards are released, the decoding side rules are generally released, and the encoding side rules can be implemented by users themselves. That is, the decoding side decoding rules and the parameters of the corresponding network models (such as super decoding network, decoding network, context network) will be strictly released, and the encoding side rules will not be strictly released. Users can use customized encoding side rules or standard published encoding side rules. In this way, after users use customized encoding side rules to encode images, they need to ensure that the standard decoding side rules can be successfully decoded.

[0150] Of course, in some embodiments, the user can also customize the decoding end rules, so that after the user uses the standard encoding end rules to encode the image, it is necessary to ensure that the user-defined decoding end rules can be successfully decoded. In other embodiments, in order to ensure the optimal encoding and decoding performance, the user can also customize the encoding end rules and the decoding end rules. In this way, after the user uses the customized encoding end rules to encode the image, it is necessary to ensure that the customized decoding end rules can be successfully decoded, and the optimal encoding and decoding performance can also be achieved. However, in this case, it is also necessary to ensure that other users can successfully decode using the standard decoding end rules.

[0151] Based on the above requirements, the present application embodiment provides a multi-branch encoding and decoding network structure, please refer to Figure 8 The multi-branch encoding and decoding network structure includes a custom encoding network, a custom super encoding network, a standard encoding network, a standard super encoding network, a standard super decoding network, a standard context network, a custom decoding network and a standard decoding network.

[0152] Among them, the custom encoding network, custom super encoding network and custom decoding network are networks constructed by users according to custom rules, and the standard encoding network, standard super encoding network, standard super decoding network, standard context network and standard decoding network are networks constructed according to image encoding and decoding standards.

[0153] Through the above multi-branch codec network structure, a variety of training methods compatible with the standard codec network can be constructed to train custom encoding networks, custom super encoding networks and custom decoding networks. In the various training methods described below, the network constructed by the standard is a trained network, so the standard network can also be called a pre-trained network.

[0154] The first training method: training a custom decoding network by combining a standard encoding network, a standard super encoding network, a standard super decoding network, a standard context network and a custom decoding network. That is, the above decoding network is a custom decoding network, which is obtained by combining a standard encoding network, a standard super encoding network, a standard super decoding network and a standard context network.

[0155] Above Figure 8 There are four switches, namely switch ec, switch ep, switch dc and switch dp. For the first training method, switch ep and switch dc can be closed, and switch ec and switch dp can be opened, that is, switch ep = switch dc = 1, switch ec = switch dp = 0. In this way, the custom decoding network can be trained by combining the standard encoding network, the standard super encoding network, the standard super decoding network, the standard context network and the custom decoding network, so that the user-defined decoding network can successfully decode the code stream encoded by the standard encoding end rule, ensuring the compatibility of the standard.

[0156] The second training method is to train the custom encoding network and the custom super encoding network by combining the standard super decoding network, the standard context network, the standard decoding network, the custom encoding network and the custom super encoding network. That is, the above decoding network is a standard decoding network, and the standard decoding network is used to train the custom encoding network and the custom super encoding network by combining the standard super decoding network, the standard context network, the custom encoding network and the custom super encoding network.

[0157] In the above Figure 8In the case of switch ec, switch ep, switch dc and switch dp, for the second training method, switch ec and switch dp can be closed, and switch ep and switch dc can be opened, that is, switch ep = switch dc = 0, switch ec = switch dp = 1. In this way, the custom encoding network and the custom super encoding network can be trained by combining the standard super decoding network, the standard context network, the standard decoding network, the custom encoding network and the custom super encoding network, so that the code stream encoded by the custom encoding end rule can be successfully decoded by the standard decoding network, ensuring the compatibility of the standard.

[0158] The third training method is to train the custom decoding network, the custom encoding network and the custom super encoding network by combining the standard encoding network, the standard super encoding network, the standard super decoding network, the standard context network, the custom encoding network, the custom super encoding network and the custom decoding network. That is, the above decoding network is a custom decoding network, and the custom decoding network is used to train the custom decoding network, the custom encoding network and the custom super encoding network by combining the standard encoding network, the standard super encoding network, the standard super decoding network, the standard context network, the custom encoding network and the custom super encoding network.

[0159] In the above Figure 8 In the case where switch ec, switch ep, switch dc and switch dp exist, for the third training method, switch ep, switch dc and switch ec can be closed, and switch dp can be opened, that is, switch ep=switch dc=switch ec=1, switch dp=0. In this way, the custom decoding network, the custom encoding network and the custom super encoding network can be trained by combining the standard encoding network, the standard super encoding network, the standard super decoding network, the standard context network, the custom encoding network, the custom super encoding network and the custom decoding network, so that after the user uses the custom encoding end rule to encode the image, the custom decoding network can be used to successfully achieve decoding, achieve optimal encoding and decoding performance, ensure standard compatibility, and enable different user-defined encoding ends and decoding ends to encode and decode each other.

[0160] The fourth training method is to train a custom decoding network, a custom encoding network, and a custom super encoding network by combining a standard encoding network, a standard super encoding network, a standard super decoding network, a standard context network, a standard decoding network, a custom encoding network, a custom super encoding network, and a custom decoding network. That is, the above decoding network is a custom decoding network, and the custom decoding network is used to train a custom decoding network, a custom encoding network, and a custom super encoding network by combining a standard encoding network, a standard super encoding network, a standard super decoding network, a standard context network, a standard decoding network, a custom encoding network, and a custom super encoding network.

[0161] In the above Figure 8 In the case where switch ec, switch ep, switch dc and switch dp exist, for the fourth training method, switch ep, switch dc, switch ec and switch dp can be closed, that is, switch ep=switch dc=switch ec=switch dp=1. In this way, the custom decoding network, custom encoding network and custom super encoding network can be trained by combining the standard encoding network, standard super encoding network, standard super decoding network, standard context network, custom encoding network, custom super encoding network, custom decoding network and standard decoding network, so that after the user uses the custom encoding end rule to encode the image, the custom decoding network can be used to successfully achieve decoding and achieve optimal encoding and decoding performance. It can also ensure that other users can successfully achieve decoding using the standard decoding network, ensure the compatibility of the standard, and can also achieve mutual encoding and decoding between the encoding ends and decoding ends customized by different users.

[0162] The fifth training method is to train a custom decoding network, a custom encoding network and a custom super encoding network by combining a standard super decoding network, a standard context network, a standard decoding network, a custom encoding network, a custom super encoding network and a custom decoding network. That is, the above-mentioned decoding network is a custom decoding network, and the custom decoding network is used to train a custom decoding network, a custom encoding network and a custom super encoding network by combining a standard super decoding network, a standard context network, a standard decoding network, a custom encoding network and a custom super encoding network.

[0163] In the above Figure 8 In the case where switch ec, switch ep, switch dc and switch dp exist, for the fifth training method, switch ec, switch dc and switch dp can be closed, and switch ep can be opened, that is, switch ec = switch dc = switch dp = 1, switch ep = 0. In this way, the custom decoding network, the custom encoding network and the custom super encoding network can be trained by combining the standard super decoding network, the standard context network, the standard decoding network, the custom encoding network, the custom super encoding network and the custom decoding network, so that after the user uses the custom encoding end rule to encode the image, the custom decoding network can be used to successfully achieve decoding and achieve optimal encoding and decoding performance. It can also ensure that other users can successfully achieve decoding using the standard decoding network, ensure the compatibility of the standard, and enable the encoding and decoding ends customized by different users to encode and decode each other.

[0164] Next, the performance of the codec network after training according to the multi-branch codec network structure provided in the embodiment of the present application is illustrated through the following example.

[0165] At 4 bitrate points, compared with the VVC coding standard, the BD-Rates of various codecs trained according to the multi-branch codec network structure provided in the embodiment of the present application are compared, as shown in Table 4 below.

[0166] Table 4

[0167]

[0168] The first row of data in Table 4 above is the test situation of the existing decoding network, the second row, the third row and the fourth row of data are the test situations obtained after the custom decoding network is trained according to the first training method, and the model complexity of the custom decoding network corresponding to these three rows of data decreases successively, the fifth row and the sixth row of data are the test situations obtained after the custom encoding network is trained according to the second training method, and the model complexity of the custom encoding network corresponding to these two rows of data decreases successively, and the seventh row, the eighth row and the ninth row of data are the test situations obtained after the custom decoding network obtained according to the first training method and the custom encoding network obtained according to the second training method are combined. Moreover, the seventh row is the same as the custom decoding network corresponding to the second row, the seventh row is the same as the custom encoding network corresponding to the fifth row, the eighth row is the same as the custom decoding network corresponding to the fourth row, the eighth row is the same as the custom encoding network corresponding to the fifth row, the ninth row is the same as the custom decoding network corresponding to the fourth row, and the ninth row is the same as the custom encoding network corresponding to the sixth row.

[0169] Taking the first row of data in Table 4 as the baseline data, it can be seen from the data in the second, third and fourth rows of Table 4 that, compared with the baseline data, after the custom decoding network is trained according to the first training method, the computational complexity of the custom decoding network is basically reduced, that is, Max kMAC / Pxl and AVG kMAC / Pxl are basically reduced, and the BD-Rate on the test image is reduced to a limited extent. In other words, after the custom decoding network is trained according to the method provided in the embodiment of the present application, the computational complexity can be basically reduced, and the quality of the reconstructed image is limited. Among them, the structure of the custom decoding network corresponding to the fourth row is very simple, so it is normal that the average distortion rate decreases more.

[0170] Similarly, it can be seen from the data in the fifth and sixth rows of Table 4 that, compared with the baseline data, after the custom encoding network is trained according to the second training method, the computational complexity of decoding through the decoding network does not change much, that is, Max kMAC / Pxl and AVG kMAC / Pxl basically do not change much, and the BD-Rate on the test image decreases only slightly. In other words, after the custom encoding network is trained according to the method provided in the embodiment of the present application, the computational complexity of decoding the data encoded by the custom encoding network does not change basically, and the quality of the reconstructed image decreases only slightly.

[0171] Similarly, it can be seen from the data in the seventh, eighth and ninth rows of Table 4 that, compared with the baseline data, after combining the custom decoding network obtained by the first training method and the custom encoding network obtained by the second training method, the encoding calculation complexity is basically reduced, that is, Max kMAC / Pxl and AVG kMAC / Pxl are basically reduced, and the BD-Rate on the test image is limited. In other words, after training the custom encoding network and the custom decoding network according to the method provided in the embodiment of the present application, the calculation complexity can be basically reduced, and the quality of the reconstructed image is limited. Among them, the network structure corresponding to the last two rows is very simple, so it is normal that the average distortion rate decreases more.

[0172] After training through the above-mentioned joint training method, both the encoder and the decoder can be set to a multi-branch structure, that is, the encoder includes two encoding networks, namely the standard encoding network and the custom encoding network, and the decoder includes two decoding networks, namely the standard decoding network and the custom decoding network. When the encoder encodes an image, one of the encoding networks can be selected for encoding, and when the decoder decodes an image, one of the decoding networks can be selected for decoding. Next, the method of using this codec for image encoding and decoding is described in detail.

[0173] Please refer to Fig. 9 , Fig. 9 This is a flowchart of an image encoding method provided by an embodiment of the present application. The method is applied to an encoder, the encoder includes a first encoding network, a second encoding network, a first super encoding network, a super decoding network and a context network, and the method includes the following steps.

[0174] Step 901: Input the image to be encoded into the first encoding network to obtain a feature map.

[0175] The first coding network is a custom coding network, and the second coding network is a pre-trained coding network, that is, the second coding network is a standard coding network. Of course, in other embodiments, the two coding networks included in the multi-branch encoder can also be custom coding networks with different structures, which is not limited in the embodiments of the present application.

[0176] Since the encoder includes two encoding networks, when the encoder performs image encoding, different encoding networks can be selected for encoding according to different requirements, which is more flexible and can achieve better encoding performance.

[0177] Step 902: Input the feature map into the first super encoding network to obtain super prior features.

[0178] The first supercoding network is a custom supercoding network. In some embodiments, the encoder further includes a second supercoding network, which is a pre-trained supercoding network, that is, the second supercoding network is a standard supercoding network.

[0179] Of course, in other embodiments, the two super coding networks included in the multi-branch encoder may also be customized super coding networks with different structures, which is not limited in the embodiments of the present application.

[0180] Since the encoder includes two super coding networks, when the encoder performs image coding, different super coding networks can be selected for coding according to different requirements, which has higher flexibility and can achieve better coding performance.

[0181] Step 903: Encode the super-prior features and feature maps into the bitstream through the super-decoding network and the context network.

[0182] The code stream is used for decoding by a decoder, and the decoder includes the above-mentioned super decoding network, context network and first decoding network. The first decoding network is the first decoding network included in the decoder introduced above.

[0183] For the relevant technical details of the above encoding method, please refer to the AI ​​encoding model or JPEG-AI encoding model mentioned above, which will not be repeated here.

[0184] Since the first encoding network is a custom encoding network, the first decoding network is a standard decoding network, and the decoder also includes a second decoding network, and the second decoding network is a custom decoding network. After the image is encoded by the first encoding network, the code stream can be decoded by the first decoding network or by the second decoding network. When the code stream is decoded by the first decoding network, the standard decoding network can successfully decode the code stream encoded by the custom encoding network, ensuring the compatibility of the image encoding and decoding standard. When the code stream is decoded by the second decoding network, the custom decoding network can successfully decode the code stream encoded by the custom encoding network, achieving the optimal encoding and decoding performance.

[0185] Please refer to Fig.10 , Fig.10 This is a flowchart of an image decoding method provided by an embodiment of the present application. The method is applied to the above-mentioned decoder, the encoder includes a first decoding network, a second decoding network, a super decoding network and a context network, and the method includes the following steps.

[0186] Step 1001: Decode the super-prior features from the bitstream.

[0187] The code stream may be a code stream obtained by encoding with a custom coding network, or a code stream obtained by encoding with a standard coding network, which is not limited in the embodiments of the present application.

[0188] Step 1002: Based on the super prior features, a feature map is decoded from the bitstream through a super decoding network and a context network.

[0189] Step 1003: Decode the feature map through a second decoding network to obtain a reconstructed image.

[0190] The first decoding network is a pre-trained decoding network, that is, the first decoding network is a standard decoding network, and the second decoding network is a custom decoding network. Of course, in other embodiments, the two decoding networks included in the multi-branch encoder can also be custom decoding networks with different structures, which is not limited in the embodiments of the present application.

[0191] Since the decoder includes two decoding networks, when the decoder decodes an image, different decoding networks can be selected for decoding according to different requirements, which is more flexible and can achieve better decoding performance.

[0192] When the code stream is obtained by encoding through a standard encoding network and the second decoding network is a custom decoding network, the compatibility of the custom decoding network with the standard decoding network can be achieved. When the code stream is obtained by encoding through a custom encoding network and the second decoding network is a custom decoding network, the encoding and decoding process can meet the needs of users and achieve the best encoding and decoding performance.

[0193] For the relevant technical details of the above decoding method, please refer to the AI ​​decoding model or JPEG-AI decoding model mentioned above, which will not be repeated here.

[0194] Since the second decoding network is a custom decoding network, the code stream can be a code stream encoded by the custom encoding network or a code stream encoded by the standard encoding network. If the code stream is a code stream encoded by the standard encoding network, after being decoded by the second decoding network, the custom decoding network can successfully decode the code stream obtained by the standard encoding network, thereby ensuring the compatibility of the image encoding and decoding standards. If the code stream is a code stream encoded by the custom encoding network, after being decoded by the second decoding network, the custom decoding network can successfully decode the code stream encoded by the custom encoding network, thereby achieving the optimal encoding and decoding performance.

[0195] As described above, both the encoder and the decoder include a super decoding network and a context network. When the encoder and the decoder are located on the same device, the device may include only one super decoding network and one context network. In this case, the encoder and the decoder share the super decoding network and the context network. Of course, multiple super decoding networks and multiple context networks may also be included. In this case, the encoder and the decoder use their own super decoding networks and context networks. Regardless of how the super decoding network and the context network are deployed, the super decoding network and the context network used by the encoder and the decoder are pre-trained networks, that is, networks constructed using parameters specified by the standard.

[0196] The embodiment of the present application further provides an encoder, which includes a first encoding network, a second encoding network, a first super encoding network, a super decoding network and a context network, wherein the first encoding network is used to determine a feature map of an image to be encoded, the first super encoding network is used to determine a super prior feature based on the feature map, and the super decoding network and the context network are used to encode the super prior feature and the feature map into a bitstream;

[0197] The code stream is used for decoding by a decoder, and the decoder includes a super decoding network, a context network and a first decoding network. The first decoding network is the first decoding network included in the decoder of any one of claims 1-13.

[0198] Optionally, the second encoding network is a pre-trained encoding network.

[0199] Optionally, the encoder further includes a second super encoding network, the second super encoding network being a pre-trained super encoding network.

[0200] The present application also provides an image encoding device, which has the function of implementing the above-mentioned Fig. 9The image coding method shown in the figure has the following functions: The image coding device comprises at least one module, and the at least one module is used to implement the above image coding method.

[0201] The present application also provides an image decoding device, which has the function of implementing the above-mentioned Fig.10 The image decoding method described above has the following functions. The image decoding device includes at least one module, and the at least one module is used to implement the above image decoding method.

[0202] The embodiment of the present application also provides a computer-readable storage medium, wherein the storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the above Fig. 9 The steps of the image encoding method or the execution of the above Fig.10 Alternatively, the computer-readable storage medium stores the above-mentioned steps executed by one or more processors. Fig. 9 The code stream obtained by the method.

[0203] The present application also provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the above Fig. 9 The steps of the image encoding method or the execution of the above Fig.10 In other words, a computer program is provided, which, when executed on a computer, enables the computer to perform the above-mentioned image decoding method. Fig. 9 The steps of the image encoding method or the execution of the above Fig.10 The steps of the image decoding method.

[0204] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiment of the present application may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0205] It should be understood that the "multiple" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

[0206] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0207] The above-mentioned embodiments are provided for the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A decoder, characterized in that: The decoder includes a first decoding network, a super decoding network and a context network, the first decoding network includes a first sub-network, a second sub-network, a third sub-network and a fourth sub-network connected in sequence from input to output, the number of channels of the input tensor of the fourth sub-network is greater than the number of channels of the input tensor of the third sub-network, and the number of channels of the input tensor of the fourth sub-network is greater than the number of channels of the input tensor of the second sub-network.

2. The decoder according to claim 1, characterized in that The first decoding network is used to decode the data of the Y component, the number of channels of the input tensor of the fourth sub-network is 96, and the number of channels of the input tensors of the third sub-network and the second sub-network are both 64.

3. The decoder according to claim 1, characterized in that The first decoding network is used to decode data of the UV component, the number of channels of the input tensor of the fourth subnetwork is 128, and the number of channels of the input tensors of the third subnetwork and the second subnetwork are both 64.

4. The decoder according to claim 1, characterized in that The first decoding network is used to decode data of the UV component, the number of channels of the input tensor of the fourth subnetwork is 96, and the number of channels of the input tensors of the third subnetwork and the second subnetwork are both 64.

5. The decoder according to any one of claims 1 to 4, characterized in that: At least two of the first sub-network, the second sub-network, and the third sub-network include residual activation units, the residual activation units include activation layers, and the activation layers in the at least two sub-networks are different.

6. The decoder according to claim 5, characterized in that The first decoding network is used to decode the data of the Y component, the first sub-network, the second sub-network and the third sub-network all include the residual activation unit, the activation layer in the first sub-network is a leaky rectified linear unit LeakyReLU, and the activation layers in the second sub-network and the third sub-network are both parameterized rectified linear units PReLU.

7. The decoder according to claim 5, characterized in that The first decoding network is used to decode data of the UV component, the second subnetwork and the third subnetwork both include the residual activation unit, the activation layer in the second subnetwork is a rectified linear unit ReLU, and the activation layer in the third subnetwork is a leaky rectified linear unit LeakyReLU.

8. The decoder according to any one of claims 1 to 4, characterized in that: If the first decoding network is used to decode the data of the Y component, the first sub-network, the second sub-network and the third sub-network all include a residual activation unit; if the first decoding network is used to decode the data of the UV component, the second sub-network and the third sub-network all include the residual activation unit; The residual activation unit includes an activation layer, and the activation layer in the second sub-network and the third sub-network is the same, but different from the activation layer in the first sub-network.

9. The decoder according to claim 8, characterized in that The activation layer in the first sub-network is a leaky rectified linear unit (LeakyReLU), and the activation layers in the second sub-network and the third sub-network are both parameterized rectified linear units (PReLU).

10. The decoder according to any one of claims 4 to 9, characterized in that: The residual activation unit also includes a grouped convolution layer, and the number of groups of the grouped convolution layer changes with the number of channels of the input tensor of the grouped convolution layer.

11. The decoder according to claim 10, characterized in that The number of groups is obtained by dividing the number of channels of the input tensor of the grouped convolutional layer by a target value, and the target value is an integer multiple of 16.

12. The decoder according to claim 11, characterized in that The target value is 16 or 32.

13. The decoder according to any one of claims 1 to 12, characterized in that: The decoder also includes a second decoding network, which is used to decode the feature map to obtain a reconstructed image, wherein the feature map is decoded from the bitstream based on the super decoding network and / or the context network.

14. An encoder, characterized in that: The encoder comprises a first encoding network, a second encoding network, a first super encoding network, a super decoding network and a context network, wherein the first encoding network is used to determine a feature map of an image to be encoded, the first super encoding network is used to determine a super prior feature based on the feature map, and the super decoding network and the context network are used to encode the super prior feature and the feature map into a bitstream; The code stream is used for decoding by a decoder, and the decoder includes the super decoding network, the context network and a first decoding network, and the first decoding network is the first decoding network included in the decoder of any one of claims 1-13.

15. The encoder according to claim 14, characterized in that The second encoding network is a pre-trained encoding network.

16. The encoder according to claim 14, characterized in that The encoder also includes a second super encoding network, which is a pre-trained super encoding network.

17. An image encoding method, characterized in that: Applied to an encoder, the encoder includes a first encoding network, a second encoding network, a first super encoding network, a super decoding network and a context network, the method includes: Inputting the image to be encoded into the first encoding network to obtain a feature map; Inputting the feature map into the first super encoding network to obtain a super prior feature; By means of the super decoding network and the context network, the super prior features and the feature map are encoded into a bitstream; The code stream is used for decoding by a decoder, and the decoder includes the super decoding network, the context network and a first decoding network, and the first decoding network is the first decoding network included in the decoder of any one of claims 1-13.

18. The method according to claim 17, characterized in that The second encoding network is a pre-trained encoding network.

19. The method according to claim 17, characterized in that The encoder also includes a second super encoding network, which is a pre-trained super encoding network.

20. An image decoding method, characterized in that: Applied to the decoder according to any one of claims 1 to 13, the decoder comprises the first decoding network, the second decoding network, the super decoding network and the context network, and the method comprises: Decode the super-prior features from the bitstream; Based on the super prior features, decoding a feature map from the bitstream through the super decoding network and the context network; The feature map is decoded by the second decoding network to obtain a reconstructed image.

21. The method of claim 20, wherein: The first decoding network is a pre-trained decoding network.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a code stream obtained by the method according to any one of claims 17 to 19 executed by one or more processors.

Images