Image decoding method, image encoding method, decoder, and encoder

Abstract

The embodiment of the invention provides an image decoding method, an image encoding method, a decoder and an encoder, the methods can be applied to the fields of image processing, video encoding and decoding, video live broadcast and the like, and the image decoding method comprises the following steps: decoding a code stream of a current image to obtain a residual value of the current image, determining a transformation value of the current image based on the residual value; processing the transformation value of the current image through a synthesis transformation network to obtain a reconstructed image of the current image; wherein one or more layers in the synthetic transformation network are first up-sampling layers, and each first up-sampling layer comprises a first convolution layer and a pixel recombination layer connected behind the first convolution layer. The decoding efficiency can be improved, and the decoding complexity can be effectively controlled.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image coding and decoding technology, and in particular, to an image decoding method, an image encoding method, a decoder, and an encoder. Background Art

[0002] With the rapid development of deep learning technology, deep learning technology has been applied in the field of image coding and decoding. In the current image coding and decoding technology based on deep learning, the encoder first maps the original image to latent variables through a synthetic transformation network, and encodes the latent variables to obtain a bitstream. Correspondingly, the decoder decodes the bitstream to obtain latent variables, and processes the latent variables through a synthetic transformation network to obtain a reconstructed image.

[0003] However, the current synthetic transformation network has the problem of not being able to balance computational efficiency and processing effect, resulting in low image encoding and decoding performance. Summary of the invention

[0004] The embodiments of the present application provide an image decoding method, an image encoding method, a decoder and an encoder, which can improve decoding efficiency and effectively control decoding complexity.

[0005] In a first aspect, an embodiment of the present application provides a decoding method, including:

[0006] Decoding a code stream of a current image to obtain a residual value of the current image, and determining a transformation value of the current image based on the residual value;

[0007] Processing the transformation value of the current image through a synthetic transformation network to obtain a reconstructed image of the current image;

[0008] Among them, one or more layers in the synthetic transformation network are a first upsampling layer, and the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer.

[0009] In a second aspect, an embodiment of the present application provides an encoding method, including:

[0010] Processing the current image through the analysis transformation network to obtain the transformation value of the current image;

[0011] The one or more layers in the analysis transformation network are first upsampling layers, and the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer;

[0012] Based on the transformation value of the current image, the current image is encoded to obtain a code stream.

[0013] In a third aspect, an embodiment of the present application provides a decoder, including:

[0014] A decoding unit, configured to decode a code stream of a current image to obtain a residual value of the current image, and determine a transformation value of the current image based on the residual value;

[0015] A reconstruction unit, used for processing the transformation value of the current image through a synthetic transformation network to obtain a reconstructed image of the current image;

[0016] Among them, one or more layers in the synthetic transformation network are a first upsampling layer, and the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer.

[0017] In a fourth aspect, an embodiment of the present application provides an encoder, including:

[0018] A transformation unit, used for processing the current image by analyzing the transformation network to obtain a transformation value of the current image;

[0019] The one or more layers in the analysis transformation network are first upsampling layers, and the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer;

[0020] The encoding unit is used to encode the current image based on the transformation value of the current image to obtain a code stream.

[0021] In a fifth aspect, an embodiment of the present application provides a decoder, including:

[0022] a processor adapted to implement computer instructions; and,

[0023] A computer-readable storage medium stores computer instructions, wherein the computer instructions are suitable for being loaded by a processor and executing the decoding method in the first aspect or its various implementation modes involved above.

[0024] In one implementation, the number of the processor is one or more, and the number of the memory is one or more.

[0025] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be disposed separately from the processor.

[0026] In a sixth aspect, an embodiment of the present application provides an encoder, including:

[0027] a processor adapted to implement computer instructions; and,

[0028] A computer-readable storage medium stores computer instructions, wherein the computer instructions are suitable for being loaded by a processor and executing the encoding method in the second aspect or its various implementation modes involved above.

[0029] In one implementation, the number of the processor is one or more, and the number of the memory is one or more.

[0030] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be disposed separately from the processor.

[0031] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are read and executed by a processor of a computer device, the computer device executes the decoding method involved in the first aspect mentioned above or the encoding method involved in the second aspect mentioned above.

[0032] In an eighth aspect, an embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program including a computer instruction, the computer instruction being stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the decoding method involved in the first aspect mentioned above or the encoding method involved in the second aspect mentioned above.

[0033] In a ninth aspect, an embodiment of the present application provides a code stream, which is a code stream as described in the method described in the first aspect above or a code stream generated by the method described in the second aspect above.

[0034] When decoding the current image, the decoding end of the present application first decodes the code stream of the current image to obtain the residual value of the current image, and determines the transformation value of the current image based on the residual value; the transformation value of the current image is processed by a synthetic transformation network to obtain a reconstructed image of the current image; wherein one or more layers in the synthetic transformation network are the first upsampling layer, and the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer. In the embodiment of the present application, by improving the structure of the upsampling layer in the synthetic transformation network, compared with the deconvolution upsampling layer, the number of deconvolution operations is reduced, the computational complexity of the upsampling layer can be reduced, and the decoding efficiency can be improved and the decoding complexity can be effectively controlled. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic structural diagram of the video encoding and decoding system provided in an embodiment of the present application.

[0036] Figure 2 It is a schematic structural diagram of the end-to-end image encoding and decoding model based on deep learning provided in an embodiment of the present application.

[0037] Figure 3It is a schematic structural diagram of the synthetic transformation network provided in an embodiment of the present application.

[0038] Figure 4 It is another schematic structural diagram of the synthetic transformation network provided in an embodiment of the present application.

[0039] Figure 5 It is a schematic flowchart of the image decoding method provided in an embodiment of the present application.

[0040] Figure 6 It is a schematic structural diagram of the first upsampling layer provided in an embodiment of the present application.

[0041] Figure 7 It is a schematic structural diagram of a reference residual block provided in an embodiment of the present application.

[0042] Figure 8 It is a schematic structural diagram of the first residual block provided in an embodiment of the present application.

[0043] Fig. 9 It is a schematic structural diagram of the first non-local residual block provided in an embodiment of the present application.

[0044] Fig.10 It is a schematic structural diagram of a reference non-local residual block provided in an embodiment of the present application.

[0045] Fig.11 It is another schematic structural diagram of the first non-local residual block provided in an embodiment of the present application.

[0046] Fig.12 It is a schematic structural diagram of the first multi-head transposition attention module provided in an embodiment of the present application.

[0047] Fig.13 It is a schematic structural diagram of the reference multi-head transposition attention module provided in an embodiment of the present application.

[0048] Fig.14 It is another schematic structural diagram of the first multi-head transposition attention module provided in an embodiment of the present application.

[0049] Fig.15 It is a schematic structural diagram of the first gated feedforward network provided in an embodiment of the present application.

[0050] Fig.16 is a schematic structural diagram of a reference gated feedforward network provided in an embodiment of the present application.

[0051] Fig.17 It is another schematic structural diagram of the first gated feedforward network provided in an embodiment of the present application.

[0052] Fig.18It is a schematic structural diagram of the first attention module provided in an embodiment of the present application.

[0053] Fig.19 It is a schematic structural diagram of the second attention module provided in an embodiment of the present application.

[0054] Fig. 20 It is a schematic structural diagram of the third attention module provided in an embodiment of the present application.

[0055] Figure 21 to Figure 24 This is an example of the distribution of the attention module provided in an embodiment of the present application.

[0056] Fig.25 It is a schematic flowchart of the image encoding method provided in an embodiment of the present application.

[0057] Fig.26 It is a schematic block diagram of a decoder provided in an embodiment of the present application.

[0058] Fig. 27 It is a schematic block diagram of the encoder provided in an embodiment of the present application.

[0059] Fig.28 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The solution provided by the present application can be applied to the field of digital compression technology.

[0061] For example, the solution provided by the present application can be applied in the field of video coding technology.

[0062] Among them, the field of video coding technology includes but is not limited to at least one of the following: image coding field, video coding field, hardware video coding field, dedicated circuit video coding field and real-time video coding field. In addition, the solution provided in the present application can be combined with Joint Photographic Experts Group Artificial Intelligence (JPEG AI). Alternatively, the solution of the present application can be combined with other exclusive or industry standards for operation, including ITU-TH.261, ISO / IECMPEG-1Visual, ITU-TH.262 or ISO / IECMPEG-2Visual, ITU-TH.263, ISO / IECMPEG-4Visual, ITU-TH.264 (also known as ISO / IECMPEG-4AVC), scalable video coding (SVC) and multi-view video coding (MVC) extensions. It should be understood that the technology of the present application is not limited to any specific coding standard or technology.

[0063] It should be noted that the terms used in the implementation method section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0064] For example, the term "and / or" in this article is only a way to describe the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" is only a way to describe the combination relationship of listed objects, indicating that one or more items may exist. For example, at least one of the following: A, B, C can mean the following combinations: A exists alone, B exists alone, C exists alone, A and B exist at the same time, A and C exist at the same time, B and C exist at the same time, and A, B, and C exist at the same time. The term "multiple" means two or more. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0065] For another example, the term "corresponding" may indicate that there is a direct or indirect correspondence between the two, or that there is an association relationship between the two, or that there is an indication and being indicated, configuration and being configured, etc. The term "indication" may be a direct indication, an indirect indication, or an indication of an association relationship. For example, A indicates B, which may indicate that A directly indicates B, such as B can be obtained through A; it may also indicate that A indirectly indicates B, such as A indicates C, B can be obtained through C; it may also indicate that there is an association relationship between A and B. The term "predefined" or "preconfigured" may refer to the pre-storage of corresponding codes, tables or other relevant information that can be used for indication in a device (for example, including an encoder or decoder), or it may refer to an agreement by protocol. "Protocol" may refer to any standard protocol in the field of encoding and decoding, and this application does not limit this. The term "when..." may be interpreted as "if" or "if" or "when..." or "in response to" and other similar descriptions. Similarly, depending on the context, the phrase "if determined" or "if (stated condition or event) is detected" may be interpreted as "when determined" or "in response to determining" or "when (stated condition or event) is detected" or "in response to detecting (stated condition or event)" and similar descriptions. The terms "first", "second", "third", "fourth", "Ath", "Bth", etc. are used to distinguish different objects rather than to describe a specific order. The terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. Among them, digital video compression technology mainly compresses huge digital image video data for easy transmission and storage.

[0066] To facilitate understanding, first combine Figure 1 The video encoding and decoding system involved in the embodiments of the present application is introduced.

[0067] Figure 1 This is a schematic block diagram of a coding and decoding system involved in an embodiment of the present application.

[0068] like Figure 1 As shown, the encoding and decoding system 100 includes an encoding device 110 and a decoding device 120 .

[0069] The encoding device 110 is used to encode (which can be understood as compressing) the video or image data to generate a code stream, and transmit the code stream to the decoding device 120. The decoding device 120 decodes the code stream generated by the encoding device 110 to obtain decoded video or image data.

[0070] The encoding device 110 can be understood as a device having a function of encoding a video or an image, and the decoding device 120 can be understood as a device having a function of decoding a video or an image. The encoding device 110 can modulate the encoded data according to the communication standard and transmit the modulated data to the decoding device 120. The encoding device 110 or the decoding device 120 includes a wider range of devices, such as smartphones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, car computers, etc.

[0071] The encoding device 110 may transmit the encoded data (eg, a code stream) to the decoding device 120 via the channel 130 .

[0072] The channel 130 may include one or more media and / or devices capable of transmitting the encoded data from the encoding device 110 to the decoding device 120. The channel 130 may include one or more communication media that enable the encoding device 110 to transmit the encoded data directly to the decoding device 120 in real time. The communication media includes wireless communication media, such as radio frequency spectrum. The communication media may also include wired communication media, such as one or more physical transmission lines. The channel 130 may include a storage medium that can store the encoded data of the encoding device 110. The storage medium includes a variety of local access data storage media, such as optical disks, DVDs, flash memories, etc. The decoding device 120 may obtain the encoded data from the storage medium. The channel 130 may include a storage server that can store the encoded data of the encoding device 110. The decoding device 120 may download the stored encoded data from the storage server. Optionally, the storage server can store the encoded data and can transmit the encoded data to the decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.

[0073] The encoding device 110 includes an encoder 112 and an output interface 113 .

[0074] The output interface 113 may include a modulator / demodulator (modem) and / or a transmitter. The encoder 112 transmits the encoded data directly to the decoding device 120 via the output interface 113. The encoded data may also be stored in a storage medium or a storage server for subsequent reading by the decoding device 120.

[0075] The encoding device 110 may include a video source 111 or an image source in addition to the encoder 112 and the input interface 113 .

[0076] The video source 111 may include at least one of a video acquisition device (e.g., a video camera), a video archive, a video input interface, and a computer graphics system, wherein the video input interface is used to receive video data from a video content provider, and the computer graphics system is used to generate video data. The encoder 112 encodes the video data from the video source 111 to generate a bitstream. The video data may include one or more pictures or a sequence of pictures. The bitstream contains the encoding information of the picture or the sequence of pictures in the form of a bitstream. The encoding information may include the encoded picture data and associated data. The associated data may include a sequence parameter set (SPS), a picture parameter set (PPS), and other syntax structures. The SPS may contain parameters applied to one or more sequences. The PPS may contain parameters applied to one or more pictures. The syntax structure refers to a set of zero or more syntax elements arranged in a specified order in the bitstream.

[0077] The decoding device 120 includes an input interface 121 and a decoder 122. The input interface 121 may include a receiver and / or a modem.

[0078] The decoding device 120 may include a display device 123 in addition to the input interface 121 and the decoder 122 .

[0079] The input interface 121 may receive the encoded data through the channel 130. The decoder 122 is used to decode the encoded data to obtain decoded data, and transmit the decoded data to the display device 123. The display device 123 displays the decoded data. The display device 123 may be integrated with the decoding device 120 or outside the decoding device 120. The display device 123 may include a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

[0080] It should be understood that Figure 1This is only an example of the present application and should not be understood as a display of the present application. In other words, the technical solution of the embodiment of the present application is not limited to Figure 1 The system framework shown, for example, the technology of the present application can also be applied to single-sided video encoding or single-sided video decoding.

[0081] In end-to-end image coding standards based on deep learning, such as JPEG AI, a transform-coding structure is usually adopted. When encoding, the encoder first inputs the input image into a neural network for encoding for a nonlinear transformation; then, the encoder can obtain the residual based on the changed latent variable; then the residual value is quantized and entropy encoded to generate a bitstream. When decoding, the decoder first recovers the value of the latent variable from the bitstream, and then inputs the recovered latent variable into another synthetic transform neural network for decoding for a nonlinear transformation to obtain a reconstructed image.

[0082] The following is an introduction to the end-to-end image encoding and decoding framework based on deep learning involved in the embodiments of the present application.

[0083] Figure 2 It is a schematic diagram of an end-to-end image encoding and decoding model based on deep learning involved in an embodiment of the present application.

[0084] like Figure 2 As shown in Figure 1, the current image x is processed by the analysis transformation network to obtain the transformation value y of the current image x. The transformation value y is processed by the super coding network to obtain the data to be coded Next, treat the encoded data After rounding, lossless encoding is performed to obtain bitstream 1. After lossless decoding of bitstream 1, the reconstruction is obtained. Reconstructed by super decoding network After decoding, it passes through the context model network and the prediction fusion network to obtain the predicted value μ of the current image. Subtract the predicted value μ of the current image from the transformed value y of the current image to obtain the residual value r of the current image, and quantize and round the residual value r to obtain the quantized residual value The quantized residual value Perform entropy coding to obtain code stream 2.

[0085] At the decoding end, the bitstream 1 is subjected to probability distribution The decoding of the bit stream 2 is based on the Gaussian distribution N(0, ) to obtain the quantized residual value The quantized residual value After dequantization, add it to the predicted value μ to get the transformed reconstruction value of the current image in, represents the standard deviation of the Gaussian distribution. Then, the decoder reconstructs the transform value through the synthetic transform network Process and obtain the reconstructed image of the current image

[0086] By the above Figure 2 As shown in the end-to-end image compression scheme based on deep learning, the brightness component (x Y ) and the chrominance component (x UV ) After nonlinear transformation of the analysis transformation network, the brightness components (y Y ) and the chrominance component (y UV ) is the result of the transformation. Then the brightness component (y Y ) and the chrominance component (y UV ) minus the output μ of the prediction fusion network Y and μ UV , and obtain the residual r of the luminance component and the chrominance component respectively Y and r UV , are all floating point types. Y and r UV Quantize it and convert it into an integer and The decoder decodes the bitstream to obtain and After dequantization, it is compared with the output μ of the prediction fusion network. Y and μ UV Add together to get the transformed values ​​of the brightness component and the chrominance component and Then the transformation values ​​of the brightness component and the chrominance component are and As input, through the synthesis transformation network, the reconstructed images of the brightness component and the chrominance component are obtained respectively. and

[0087] Figure 3 This is an example of the structure of the synthetic transformation network involved in the embodiments of the present application.

[0088] like Figure 3As shown, the synthesis transformation network may include a synthesis transformation network adopted by a luminance (Y) component and a chrominance (UV) component, and the network structure thereof is the same and includes, from top to bottom: two residual blocks (Res Block, RB), a deconvolution layer (DConv), a cropping module, a residual activation unit (ResAU), a deconvolution layer, a cropping module, a residual activation unit, a deconvolution layer, a non-local residual block (residual non-local attentionblock, RNAB), a cropping module, a residual activation unit, a deconvolution layer and a cropping module.

[0089] Among them, the residual block (ResBlock) is a module formed by stacking layers of convolution modules with jump connections, which can solve the problem of vanishing network gradients. The deconvolution layer is an upsampling layer, which is used to upsample the input feature map. The cropping unit is used to crop the input feature map. The residual activation unit is a unit formed by the residual block and the activation function. The residual activation unit is used to express the output as a linear superposition of the input and a nonlinear transformation of the input. A nonlinear change function can be used to describe the input and output of the residual activation unit, where the input is X and the output is F(x), and F usually includes operations such as convolution and activation. The non-local residual block is used as an attention module for attention-based feature extraction.

[0090] It is worth noting that the parameters (including at least one of the number of channels, convolution kernel, and upsampling multiple) of the deconvolution layer (Deconvolution, DConv) in the synthesis transformation network used by the luminance component and the deconvolution layer (Deconvolution, DConv) in the synthesis transformation network used by the chrominance component are different. For example, the number of channels of the first deconvolution layer from top to bottom in the synthesis transformation network used by the luminance component is 128, the convolution kernel is 3×3, and the upsampling multiple is 2, while the number of channels of the first deconvolution layer from top to bottom in the synthesis transformation network used by the chrominance component is 64, the convolution kernel is 3×3, and the upsampling multiple is 2.

[0091] Figure 4 It is another example of the structure of the synthetic transformation network involved in the embodiments of the present application.

[0092] like Figure 4 As shown, is the transformation value (or transformation reconstruction value) of the current image, which can be expressed in terms of the number of channels, width, and height as [C in , h, w]. is the reconstructed image of the current image, which can be expressed in terms of channel number, width and height as [C out, h0, w0]. The end-to-end image encoding and decoding scheme uses the four deconvolution layers as upsampling layers in the synthetic transformation network for upsampling. For example, the parameters of the four upsampling layers are: [C, h3, w3], [C, h2, w2], [C, h1, w1], [C out , h0, w0]. However, due to the large amount of computation of the deconvolution layer, the computational complexity of the synthetic transformation network is very large, which makes the synthetic transformation network bring higher complexity while improving the performance.

[0093] As can be seen from the above, in an end-to-end image encoding and decoding scheme based on deep learning, the encoder maps the original image into latent variables and writes them into the bitstream through the entropy encoder; after the decoder obtains the latent variables from the bitstream through the entropy decoder, it is necessary to input the latent variables into the synthetic transformation network to obtain the final reconstructed image. However, since the end-to-end image encoding and decoding scheme uses a series of deconvolution layers in the synthetic transformation network for upsampling, the computational complexity of the decoding end is very large. Therefore, how to design an efficient synthetic transformation network structure for end-to-end image compression and find a balance between performance and decoding computational complexity is a very important issue.

[0094] In order to further improve the performance of end-to-end image / video coding schemes based on deep learning and effectively control the complexity, this application proposes an efficient synthetic transformation network structure for end-to-end image compression. Specifically, in the new synthetic transformation network structure, the convolution layer close to the input uses a deconvolution operation, while the convolution layer in the middle uses a normal convolution operation, and finally restores the output to its original size through pixel reorganization (pixel shuffle), thereby reducing the number of deconvolution operations, thereby improving the coding efficiency and effectively controlling the complexity of decoding. In addition, the method is widely applicable and can be applied to any end-to-end image coding scheme based on deep learning. This application uses JPEG AI as a platform to discuss some implementation details of the proposed method, but it is worth noting that the method involved in this application is not limited to the JPEG AI coding standard.

[0095] The technical solutions of the embodiments of the present application are described in detail below through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes are not repeated in some embodiments. First, taking the decoding end as an example, the image decoding method provided by the embodiment of the present application is introduced.

[0096] Figure 5 is a schematic flow chart of the decoding method 200 provided in the present application. It should be understood that the decoding method 200 can be executed by a decoder. For example, the decoding method 200 can be executed by Figure 1The decoding device 120 or the video decoder 122 shown in the figure performs. For the convenience of description, the decoder is taken as an example for explanation below.

[0097] like Figure 5 As shown, the decoding method 200 may include:

[0098] S210, the decoder decodes the code stream of the current image to obtain a residual value of the current image, and determines a transformation value of the current image based on the residual value.

[0099] Exemplarily, the decoder decodes the bitstream to obtain the quantized residual value of the current image. The quantized residual value is dequantized to obtain the residual value r of the current image. Next, the predicted value μ of the current image is determined, for example, by using a prediction fusion network to determine the predicted value μ of the current image, and then the predicted value μ of the current image and the residual value r are added to obtain the transformation value of the current image.

[0100] S220, the decoder processes the transformation value of the current image through a synthetic transformation network to obtain a reconstructed image of the current image; wherein one or more layers in the synthetic transformation network are the first upsampling layers.

[0101] Figure 6 It is a schematic structural diagram of the first upsampling layer provided in an embodiment of the present application.

[0102] like Figure 6 As shown, the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer. Of course, in other alternative embodiments, the first upsampling layer may also include layers or modules other than the first convolutional layer and the pixel reorganization layer.

[0103] For example, the pixel reorganization layer can also be called a sub-pixel convolution layer, which fills the channel dimension data to achieve upsampling. Specifically, after the convolution of the previous network, s 2 channels, and s 2 The C×H×W feature map is converted to a C×sH×sW image.

[0104] Exemplarily, the synthetic transformation network may include K upsampling layers, the K upsampling layers include one or more first upsampling layers, and K is a positive integer. The sum of the upsampling multiples of each upsampling layer in the K upsampling layers is a fixed value. The upsampling multiples of different upsampling layers in the K upsampling layers may be the same or different. For example, each upsampling layer in the K upsampling layers is the first upsampling layer. For another example, some of the upsampling layers in the K upsampling layers are the first upsampling layers, and another part of the upsampling layers in the K upsampling layers are second upsampling layers of a type different from the first upsampling layer. Among them, the second upsampling layer may be a deconvolution layer, or may be other types of modules or layers that can be used for upsampling, and the present application does not specifically limit this. For example, the second upsampling layer may be a module or layer that can implement interpolation or up-pooling.

[0105] Exemplarily, when the synthetic transformation network includes a first upsampling layer, the last layer in the synthetic transformation network is the first upsampling layer, or the first layer in the synthetic transformation network is the first upsampling layer, or the layer located in the middle position in the synthetic transformation network is the first upsampling layer.

[0106] Exemplarily, when the synthetic transformation network includes multiple first upsampling layers, the multiple first upsampling layers may be continuous or discontinuous. Wherein, when two adjacent first upsampling layers are directly connected, the two first upsampling layers may be considered to be continuous, and when two adjacent first upsampling layers are connected through other layers in the synthetic transformation network, the two first upsampling layers may be considered to be discontinuous.

[0107] In this embodiment, the decoder first decodes the code stream of the current image to obtain the residual value of the current image, and determines the transformation value of the current image based on the residual value; the transformation value of the current image is processed by a synthetic transformation network to obtain a reconstructed image of the current image; wherein one or more layers in the synthetic transformation network are the first upsampling layer, and the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer. In the embodiment of the present application, by improving the structure of the upsampling layer in the synthetic transformation network, compared with the deconvolution upsampling layer, the number of deconvolution operations is reduced, the computational complexity of the upsampling layer can be reduced, and the decoding efficiency can be improved and the decoding complexity can be effectively controlled.

[0108] In some embodiments, the i-th layer in the synthetic transformation network is the first upsampling layer; wherein S220 may include:

[0109] The decoder uses the first i-1 layers of the synthetic transformation network to determine the i-1th feature information of the current image based on the transformation value of the current image; the format of the i-1th feature information is C×H×W, C represents the number of channels of the i-1th feature information, H represents the height of the i-1th feature information, and W represents the width of the i-1th feature information; C, H and W are all positive integers; using the first convolutional layer, the i-1th feature information is transformed into a channel, and the format is s 2 C×H×W feature information; s is a positive integer; using the pixel reorganization layer, the format is s 2 The feature information of C×H×W is reorganized to obtain the i-th feature information of the current image, and the format of the i-th feature information is C×sH×sW; the reconstructed image is determined based on the i+1-th feature information using the layers other than the first i layers in the synthetic transformation network.

[0110] For example, s 2 C=s 2 ×C,s 2 C is the number of channels of the first convolutional layer.

[0111] Exemplarily, sH=s×H, where sH is the height of the i-th feature information.

[0112] Exemplarily, sW=s×W, where sW is the width of the i-th feature information.

[0113] In some embodiments, the number of the first upsampling layers in the synthesis transformation network is negatively correlated with at least one of the following: a bit rate adopted by the current image, and an upper limit value of the computational complexity of the synthesis transformation network.

[0114] Exemplarily, the number of first upsampling layers in the synthesis transformation network can be any positive integer.

[0115] Exemplarily, the number of the first upsampling layers may be adjusted based on actual needs.

[0116] Exemplarily, when the number of upsampling layers in the synthesis transformation network is fixed, the greater the bit rate used by the current image and / or the greater the upper limit of the computational complexity of the synthesis transformation network, the smaller the number of the first upsampling layers in the synthesis transformation network.

[0117] Exemplarily, the upper limit of the computational complexity of the synthesis transformation network is a predefined value. For example, the upper limit of the computational complexity of the synthesis transformation network can be realized by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in the decoder, or the upper limit of the computational complexity of the synthesis transformation network can be agreed or defined by a standard protocol.

[0118] Exemplarily, the computational complexity of the synthesis transformation network can be the overall computational complexity of the current image decoded by the decoding end. The computational complexity of the synthesis transformation network can be characterized by the sum of the computational complexity of some layers or each layer in the synthesis transformation network. Among them, the embodiment of the present application does not specifically limit the measurement method of the computational complexity relative to the layer. For example, the number of floating point operations (Floating Point Operations, FLOPs) is used to measure the computational complexity of the layer; the smaller the value of FLOPs, the smaller the computational complexity, and the larger the value of FLOPs, the greater the computational complexity. For another example, kmac / pixel=FLOPs / (1000*w*h) can be used to represent the computational complexity, where h and w represent the height and width of the decoded image.

[0119] In some embodiments, one or more layers of the synthesis transformation network are second upsampling layers, and the structure of the second upsampling layer is different from the structure of the first upsampling layer.

[0120] Exemplarily, the computational complexity of the second upsampling layer is greater than the computational complexity of the first upsampling layer. In this case, it is equivalent to that the synthesis transform network, based on the first upsampling layer, can cooperate with the second upsampling layer with a greater computational complexity than the first upsampling layer to improve the decoding performance of the synthesis transform network, thereby improving the ability of the synthesis transform network to balance computational complexity and decoding performance.

[0121] In some embodiments, the second upsampling layer is a deconvolution layer.

[0122] Of course, in other alternative embodiments, the second upsampling layer may also be other types of layers or modules, which are not specifically limited in the present application. For example, the second upsampling layer may be a module or layer that can implement interpolation or up-pooling.

[0123] In some embodiments, the number of the second upsampling layers in the synthesis transformation network is positively correlated with at least one of the following: a bit rate adopted by the current image, and an upper limit value of the computational complexity of the synthesis transformation network.

[0124] Exemplarily, the number of second upsampling layers in the synthesis transformation network can be any positive integer.

[0125] Exemplarily, the number of the second upsampling layers may be adjusted based on actual needs.

[0126] Exemplarily, when the number of upsampling layers in the synthesis transformation network is fixed, the greater the bit rate used by the current image and / or the greater the upper limit of the computational complexity of the synthesis transformation network, the greater the number of second upsampling layers in the synthesis transformation network.

[0127] Exemplarily, the upper limit of the computational complexity of the synthesis transformation network is a predefined value. For example, the upper limit of the computational complexity of the synthesis transformation network can be realized by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in the decoder, or the upper limit of the computational complexity of the synthesis transformation network can be agreed or defined by a standard protocol.

[0128] Exemplarily, the computational complexity of the synthesis transformation network can be the overall computational complexity of the current image decoded by the decoding end. The computational complexity of the synthesis transformation network can be characterized by the sum of the computational complexity of some layers or each layer in the synthesis transformation network. Among them, the embodiment of the present application does not specifically limit the measurement method of the computational complexity relative to the layer. For example, the number of floating point operations (Floating Point Operations, FLOPs) is used to measure the computational complexity of the layer; the smaller the value of FLOPs, the smaller the computational complexity, and the larger the value of FLOPs, the greater the computational complexity. For another example, kmac / pixel=FLOPs / (1000*w*h) can be used to represent the computational complexity, where h and w represent the height and width of the decoded image.

[0129] In some embodiments, when the synthesis transformation network is a network used by the first component, the sum of upsampling multiples of each upsampling layer in the synthesis transformation network is different from the sum of upsampling multiples of each upsampling layer in the synthesis transformation network when the synthesis transformation network is a network used by the second component.

[0130] Exemplarily, the first component is a luminance component and the second component is a chrominance component.

[0131] Exemplarily, the first component and the second component are both components among the R component, the G component and the B component.

[0132] Exemplarily, the upsampling multiple may refer to the ratio of the width of the upsampled feature information to the width of the feature information that has not been upsampled, or the upsampling multiple may refer to the ratio of the height of the upsampled feature information to the height of the feature information that has not been upsampled.

[0133] In some embodiments, when the first component is a luminance component and the second component is a chrominance component, the sum of upsampling multiples of each upsampling layer in the synthesis transformation network when the synthesis transformation network is a network used for the first component is greater than the sum of upsampling multiples of each upsampling layer in the synthesis transformation network when the synthesis transformation network is a network used for the second component.

[0134] Of course, in other alternative embodiments, when the first component is a luminance component and the second component is a chrominance component, the sum of the upsampling multiples of each upsampling layer in the synthesis transformation network when the synthesis transformation network is the network used for the first component may also be equal to the sum of the upsampling multiples of each upsampling layer in the synthesis transformation network when the synthesis transformation network is the network used for the second component. This application does not make specific limitations on this.

[0135] In some embodiments, one or more layers in the synthetic transformation network are convolutional layers.

[0136] Exemplarily, the convolution kernel size of the convolution layer in the synthetic transformation network can be any size such as 1×1, 3×3, 5×5 or 7×7.

[0137] Exemplarily, it is assumed that the synthetic transformation network includes 2 upsampling layers and 2 convolutional layers, wherein, in the direction from input to output, the 2 convolutional layers are located between the two upsampling layers, and the 2 upsampling layers include a first upsampling layer and a deconvolutional layer, wherein the first upsampling layer is an upsampling layer close to the output of the synthetic transformation network, and the deconvolutional layer is an upsampling layer close to the input of the synthetic transformation network. The convolution kernel size of the two convolutional layers can be any size such as 1×1, 3×3, 5×5 or 7×7, and the convolution kernel sizes of the convolution kernels of the two convolutional layers can be different; the convolution kernel size of the first convolutional layer in the first upsampling layer can be any size such as 1×1, 3×3, 5×5 or 7×7.

[0138] In some embodiments, the synthesis transform network includes a first residual block.

[0139] Exemplarily, the first residual block may be a residual block obtained by simplifying the structure of the reference residual block. The simplification of the structure of the reference residual block includes but is not limited to: reducing the number of modules in the reference residual block or simplifying the structure of the modules in the reference residual block.

[0140] Figure 7 It is a schematic structural diagram of a reference residual block provided in an embodiment of the present application.

[0141] like Figure 7 As shown, the reference residual block may include two convolutional layers, the two convolutional layers are connected by an activation function, and the input feature information of the two convolutional layers is jump-connected to the output of the two convolutional layers. The first residual block may be a residual block obtained by simplifying the structure of the reference residual block.

[0142] In this embodiment, the first residual block may be Figure 7 The residual block is obtained by simplifying the structure of the reference residual block shown in FIG. Figure 7The residual block obtained after the last convolutional layer in the residual reference block shown.

[0143] Figure 8 It is a schematic structural diagram of the first residual block provided in an embodiment of the present application.

[0144] like Figure 8 As shown, the first residual block includes: a second convolutional layer, a first activation function and a first connection layer, the second convolutional layer is connected to the first connection layer through the first activation function, and the input feature of the second convolutional layer is jump-connected to the first connection layer. In the specific information processing process, the input feature information is processed by the second convolutional layer and then processed by the first activation function to obtain feature information, which is added to the input feature information to obtain feature information output by the first residual block.

[0145] In some embodiments, one or more layers in the synthesis transformation network are the first residual block.

[0146] Exemplarily, the first residual block may be located at any position in the synthesis transformation network. In other words, any one or more layers in the synthesis transformation network may be the first residual block. For example, the layers located at the first position and the second position in the order of input to output in the synthesis transformation network are the first residual blocks.

[0147] In some embodiments, one or more layers in the synthetic transformation network is a first attention module, and the first attention module is located at least one of the following positions in the synthetic transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit; wherein the first attention module includes the first residual block.

[0148] Exemplarily, when one or more layers in the synthetic transformation network is a first attention module connected after the upsampling layer in the synthetic transformation network, each first upsampling layer in the synthetic transformation network or part of the first upsampling layers is connected to the first attention module, and / or each deconvolution layer in the synthetic transformation network or part of the deconvolution layers is connected to the first attention module.

[0149] Exemplarily, when one or more layers in the synthetic transformation network is a first attention module connected after the convolutional layer in the synthetic transformation network, each convolutional layer or each convolutional layer in part of the synthetic transformation network is connected to the first attention module.

[0150] It should be noted that the first attention module can be any module that includes the first residual block and can implement attention processing.

[0151] In some embodiments, the first attention module is a first non-local residual block.

[0152] Exemplarily, the first non-local residual block may be a non-local residual block obtained by simplifying the structure of the reference non-local residual block. The simplification method of the structure of the reference non-local residual block includes but is not limited to: reducing the number of modules in the reference non-local residual block or simplifying the structure of the modules in the reference non-local residual block.

[0153] Fig. 9 It is a schematic structural diagram of the first non-local residual block provided in an embodiment of the present application.

[0154] like Fig. 9 As shown, the first non-local residual block includes: a first branch, a second branch, a second connection layer and a third connection layer, the first branch and the second branch are both connected to the third connection layer through the second connection layer, the input feature information of the first non-local residual block is jump-connected to the third connection layer, and at least one of the first branch and the second branch includes the first residual block.

[0155] Exemplarily, the first branch and the second branch are both used for feature extraction. The structures of the first branch and the second branch are different; or, when the structures of the first branch and the second branch are the same, the parameters used by the first branch are different from the parameters used by the second branch. For example, the size of the convolution kernel used by the convolution layer in the first branch is different from the size of the convolution kernel used by the convolution layer in the second branch.

[0156] Exemplarily, the first branch may include a plurality of residual blocks connected in series, and the plurality of residual blocks include the first residual block.

[0157] Exemplarily, the second branch may include at least one residual convolution unit connected in series and an activation function, wherein the residual convolution unit includes at least one residual block and at least one convolution layer connected in series, and the at least one residual block includes the first residual block.

[0158] Fig.10 It is another schematic structural diagram of a reference non-local residual block provided in an embodiment of the present application.

[0159] like Fig.10 As shown, the reference non-local residual block includes: a first branch, a second branch, a second connection layer and a third connection layer, the first branch and the second branch are connected to the third connection layer through the second connection layer, and the input feature information of the first non-local residual block is jump-connected to the third connection layer. Among them, the first branch includes 3 residual blocks (RB) connected in series, and the second branch includes 2 residual blocks, 1 convolution layer, 2 residual blocks, 1 convolution layer, 2 residual blocks, 1 convolution layer and 1 activation function in sequence.

[0160] In this embodiment, the first residual block may be Fig.10 The residual block obtained by simplifying the structure of the reference non-local residual block shown in FIG. Fig.10 The non-local residual block is obtained by replacing part or all of the residual blocks in the reference non-local residual block with the first residual block mentioned above. For example, the first non-local residual block may be: Fig.10 The partial residual blocks or non-local residual blocks obtained after the partial convolutional layers are shown.

[0161] Fig.11 It is another schematic structural diagram of the first non-local residual block provided in an embodiment of the present application.

[0162] like Fig.11 As shown, the first non-local residual block includes: a first branch, a second branch, a second connection layer and a third connection layer, the first branch and the second branch are connected to the third connection layer through the second connection layer, and the input feature information of the first non-local residual block is jump-connected to the third connection layer. Among them, the first branch includes 3 residual blocks (RBs) connected in series, the second residual block of the 3 residual blocks connected in series is the first residual block mentioned above, and the second branch includes 2 residual blocks, 1 convolution layer, 2 residual blocks, 1 convolution layer, 2 residual blocks, 1 convolution layer and 1 activation function in sequence.

[0163] In some embodiments, the first attention module includes a first multi-head transposition attention module.

[0164] Fig.12 It is a schematic structural diagram of the first multi-head transposition attention module provided in an embodiment of the present application.

[0165] like Fig.12 As shown, the first multi-head transposition attention module includes: a third branch, a fourth branch, a fifth branch, a fourth connection layer, a fifth connection layer and a sixth connection layer, the third branch and the fourth branch are both connected to the fifth connection layer through the fourth connection layer, the fifth branch is connected to the fifth connection layer through the fourth connection layer, the input feature information of the first multi-head transposition attention module is jump-connected to the fifth connection layer, and at least one of the third branch, the fourth branch and the fifth branch includes the first residual block.

[0166] Exemplarily, the third branch, the fourth branch, and the fifth branch are all used for feature extraction. The structures of the third branch, the fourth branch, and the fifth branch are different from each other; or, when the structures of some branches of the third branch, the fourth branch, and the fifth branch are the same, different parameters are used in different branches of the partial branches. For example, different convolution kernel sizes are used in the convolution layers of different branches of the partial branches.

[0167] Exemplarily, the first multi-head transposed attention module uses the third branch to determine the query information Q based on the input feature information, uses the fourth branch to determine the key information K based on the input feature information, and uses the fifth branch to determine the value item information V based on the input feature information. The query information Q is also called a query matrix, the key information K is also called a key matrix, and the value item information V is also called a value item matrix. Then, when the fourth connection layer is a matrix multiplication connection layer, the first multi-head transposed attention module multiplies the input information obtained based on the query information Q and the input information obtained based on the key information K, and then outputs the transposed attention map A based on the result obtained after the multiplication. When the fifth connection layer is a matrix multiplication connection layer, the first multi-head transposed attention module multiplies the transposed attention map A with the input information obtained based on the value item information V to obtain the transposed attention information; then, when the sixth connection layer is a matrix addition connection layer, the first multi-head transposed attention module can add the information obtained based on the transposed attention information to the feature information input to the first multi-head transposed attention module to obtain the feature information output by the first multi-head transposed attention module.

[0168] Exemplarily, the first multi-head transposition attention module can be a multi-head transposition attention module obtained by simplifying the structure of a reference multi-head transposition attention module. The simplification method of the structure of the reference multi-head transposition attention module includes but is not limited to: reducing the number of modules in the reference multi-head transposition attention module or simplifying the structure of the modules in the reference multi-head transposition attention module.

[0169] Fig.13 It is a schematic structural diagram of the reference multi-head transposition attention module provided in an embodiment of the present application.

[0170] like Fig.13 As shown, the reference multi-head transposition attention module includes: a third branch, a fourth branch, a fifth branch, a fourth connection layer, a fifth connection layer and a sixth connection layer. The third branch and the fourth branch are connected to the fifth connection layer through the fourth connection layer, the fifth branch is connected to the fifth connection layer through the fourth connection layer, and the input feature information of the first multi-head transposition attention module is jump-connected to the fifth connection layer.

[0171] Among them, the third branch, the fourth branch and the fifth branch each include two convolutional layers and a connection layer (for example, the circle with R shown in the figure) in sequence; in addition, the fourth connection layer is connected to the fifth connection layer through a normalized exponential function, and the fifth connection layer is connected to the sixth connection layer through a connection layer (for example, the circle with R shown in the figure) and a convolutional layer.

[0172] During data processing, the reference multi-head transposed attention module uses 1x1 convolution to aggregate pixel-level cross-channel context and 3x3 deep convolution to encode channel-level spatial context to obtain Q, K, and V respectively. After reshaping Q and K, self-attention is calculated in the channel dimension, and the transposed attention map A is generated by dot product. The dot product of Q and K represents the similarity between the elements of Q and K (each element is a vector), but this similarity is not normalized, so a normalized exponential function (softmax) is needed to normalize the results of Q and K. Then the result after softmax is a mask matrix with all values ​​​​of 0-1 (which can be understood as an attention score matrix), and V represents the features after the linear transformation of the input. Then multiplying the mask matrix by V can get the weighted features. Q and K are introduced to obtain a mask matrix with all values ​​​​of 0-1, and V is introduced to retain the features of the input.

[0173] In this embodiment, the first multi-head transposition attention module can be Fig.13 The residual block is obtained by simplifying the structure of the reference multi-head transposition attention module shown in FIG. For example, the first multi-head transposition attention module can be: Fig.13 The multi-head transposed attention module shown is obtained by replacing the 1×1 convolutional layer and the 3×3 convolutional layer on some or all branches of the multi-head transposed attention module with the first residual block mentioned above. For another example, the first multi-head transposed attention module can be: Fig.13 The multi-head transposed attention module obtained after the partial convolutional layer shown (e.g., the 1×1 convolutional layer connected to the sixth connection layer).

[0174] Fig.14 It is a schematic structural diagram of the reference multi-head transposition attention module provided in an embodiment of the present application.

[0175] like Fig.14As shown, the first multi-head transposition attention module includes: a third branch, a fourth branch, a fifth branch, a fourth connection layer, a fifth connection layer and a sixth connection layer, the third branch and the fourth branch are connected to the fifth connection layer through the fourth connection layer, the fifth branch is connected to the fifth connection layer through the fourth connection layer, and the input feature information of the first multi-head transposition attention module is jumped to the fifth connection layer. Wherein, the third branch, the fourth branch and the fifth branch each include a first residual block and a connection layer (such as the circle with R shown in the figure) in sequence; in addition, the fourth connection layer is connected to the fifth connection layer through a normalized exponential function, and the fifth connection layer is connected to the sixth connection layer through a connection layer (such as the circle with R shown in the figure).

[0176] In some embodiments, the first attention module comprises a first gated feed-forward network.

[0177] Fig.15 It is a schematic structural diagram of the first gated feedforward network provided in an embodiment of the present application.

[0178] like Fig.15 As shown, the first gated feedforward network includes: a sixth branch, a seventh branch, a seventh connection layer and an eighth connection layer, the sixth branch and the seventh branch are both connected to the eighth connection layer through the seventh connection layer, and at least one of the sixth branch and the seventh branch includes the first residual block.

[0179] Exemplarily, the sixth branch and the seventh branch are both used for feature extraction. The sixth branch and the seventh branch have different structures; or, when the sixth branch and the seventh branch have the same structure, the parameters used by the sixth branch are different from the parameters used by the seventh branch. For example, the convolution kernel size used in the convolution layer in the sixth branch is different from the convolution kernel size used in the convolution layer in the seventh branch.

[0180] Exemplarily, the first gated feedforward network uses a gating mechanism for feature extraction, wherein the gating mechanism is a fusion of elements of two parallel paths of linear transformation layers, one of which is activated by a seventh connection layer.

[0181] Exemplarily, the first gated feedforward network may be a gated feedforward network obtained by simplifying the structure of a reference gated feedforward network. The simplification of the structure of the reference gated feedforward network includes, but is not limited to, reducing the number of modules in the reference gated feedforward network or simplifying the structure of the modules in the reference gated feedforward network.

[0182] Fig.16 is a schematic structural diagram of a reference gated feedforward network provided in an embodiment of the present application.

[0183] like Fig.16 As shown, the reference gated feedforward network includes: a sixth branch, a seventh branch, a seventh connection layer and an eighth connection layer, and the sixth branch and the seventh branch are connected to the eighth connection layer through the seventh connection layer. Among them, the sixth branch includes two convolutional layers in sequence, and the seventh branch includes two convolutional layers and an activation function (such as the circle with S line shown in the figure) in sequence. In addition, the seventh connection layer is connected to the eighth connection layer through a convolutional layer.

[0184] In this embodiment, the first gated feedforward network may be Fig.16 The residual block is obtained by simplifying the structure of the reference gated feedforward network shown in FIG. For example, the first gated feedforward network may be: Fig.16 The gated feedforward network shown in the figure is obtained by replacing the 1×1 convolutional layer and the 3×3 convolutional layer on some or all branches with the first residual block mentioned above. For another example, the first gated feedforward network can be: Fig.16 The gated feed-forward network obtained after the partial convolutional layer shown (e.g., the 1×1 convolutional layer connected to the eighth connection layer).

[0185] Fig.17 It is another schematic structural diagram of the first gated feedforward network provided in an embodiment of the present application.

[0186] like Fig.17 As shown, the reference gated feedforward network includes: a sixth branch, a seventh branch, a seventh connection layer and an eighth connection layer, and the sixth branch and the seventh branch are connected to the eighth connection layer through the seventh connection layer. Among them, the sixth branch only includes a first residual block, and the seventh branch sequentially includes a first residual block and an activation function (such as a circle with an S line shown in the figure).

[0187] Of course, in other alternative embodiments, the first attention module may also include a multi-head transposition attention module and a gated feedforward network at the same time, and the embodiments of the present application do not specifically limit this.

[0188] Fig.18 It is a schematic structural diagram of the first attention module provided in an embodiment of the present application.

[0189] like Fig.18As shown, the first attention module may include a first multi-head transposition attention module and a first gated feedforward network connected after the first multi-head transposition attention module. Of course, in other alternative embodiments, the first attention module may include a first multi-head transposition attention module and a first gated feedforward network connected after the reference multi-head transposition attention module. Alternatively, the first attention module may include a reference multi-head transposition attention module and a first gated feedforward network connected after the first multi-head transposition attention module. The embodiments of the present application do not specifically limit this.

[0190] In some embodiments, one or more layers in the synthetic transformation network is a second attention module, and the second attention module is located at least one of the following positions in the synthetic transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit.

[0191] Exemplarily, the structure of the second attention module is different from the structure of the first attention module.

[0192] Fig.19 It is a schematic structural diagram of the second attention module provided in an embodiment of the present application.

[0193] like Fig.19 As shown, the second attention module includes a window attention unit and a grid attention unit connected to the window attention unit; wherein the attention window unit is used to divide the feature information input to the second attention module into multiple sub-blocks and perform attention processing on the multiple sub-blocks to obtain local feature information, and the grid attention unit is used to divide the local feature information into multiple grids and perform attention processing on the multiple grids to obtain output feature information.

[0194] For example, for the input feature Figure X ∈R H×W×C , where H is the height of the input feature map, W is the width of the input feature map, and C is the number of channels of the input feature map. Instead of focusing on the flat spatial dimension HW, the window attention unit divides the first feature information into non-overlapping sub-blocks, each of size P×P, and focuses self-attention on the local spatial dimension (i.e., P×P), which will be used for local interaction. Meanwhile, the grid attention unit grids the input into shape (G×G, HG×WG, C) using a fixed G×G uniform grid, resulting in a grid with adaptive size H / G×W / G. Global spatial blending is performed using self-attention on the decomposed grid axis (i.e., G×G).

[0195] Exemplarily, the window attention unit includes a block self-attention (Block Self Attention, Block-SA) and a feed forward neural network (Feed Forward Neural Network, FNN) and two connection layers; the block self-attention is connected to the feed forward neural network through a connection layer, and the input feature information of the block self-attention is jumped to the one connection layer; the feed forward neural network is connected to the other connection layer of the two connection layers, and the input feature information of the feed forward neural network is jumped to the other connection layer. The block self-attention is a module that can be used to implement block partitioning and block-based attention calculation.

[0196] Exemplarily, the grid window attention unit includes a grid self-attention (Grid-SA) and a feed forward neural network (FNN) and two connection layers; the grid self-attention is connected to the feed forward neural network through a connection layer, and the input feature information of the grid self-attention is jumped to the one connection layer; the feed forward neural network is connected to the other connection layer of the two connection layers, and the input feature information of the feed forward neural network is jumped to the other connection layer. The grid self-attention is a module that can be used to implement grid division and grid-based attention calculation.

[0197] Of course, in other alternative embodiments, the second attention module may also include other modules, which is not specifically limited in this application.

[0198] For example, the second attention module may also include an inverted linear bottleneck unit with depthwise separable convolution. As an example, the inverted linear bottleneck unit with depthwise separable convolution includes a convolution layer, a depthwise separable (Depth-wise) convolution layer, a squeeze excitation layer (SE), a convolution layer, and a connection layer in series, wherein the input feature information of the inverted linear bottleneck unit with depthwise separable convolution jumps to the connection layer. In the data processing process, the input feature information is subjected to point-by-point convolution channel dimension enhancement through the first convolution layer, and the depth-wise convolution is performed in the projection space after the dimension enhancement. The SE immediately following is used to enhance the representation of the important channel, and finally the second convolution layer is used again to perform point-by-point convolution to restore the dimension. Since the depth-wise convolution layer is connected to the normalization process, and the normalization process is not very friendly to the image compression task. Therefore, as another example, the inverted linear bottleneck unit with depthwise separable convolution includes a first convolution layer, a squeeze excitation layer (ie, SE layer) and a second convolution layer in series in order to improve the image compression efficiency.

[0199] In some embodiments, one or more layers in the synthetic transformation network is a third attention module, and the third attention module is located at least one of the following positions in the synthetic transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit.

[0200] Exemplarily, the structure of the third attention module is different from the structure of the first attention module.

[0201] Fig. 20 It is a schematic structural diagram of the third attention module provided in an embodiment of the present application.

[0202] like Fig. 20 As shown, the third attention module includes a depth-separable convolutional layer, at least one convolutional layer and a ninth connection layer, the depth-separable convolutional layer is connected to the ninth connection layer through the at least one convolutional layer, and the input feature information of the depth-separable convolutional layer is jump-connected to the ninth connection layer.

[0203] It should be noted that the present application does not specifically limit the distribution of the attention modules in the synthetic transformation network.

[0204] For example, the synthetic transformation network may include one or more of the first attention module, the second attention module and the third attention module mentioned above. When the synthetic transformation network includes multiple attention modules, the combination of the first attention module, the second attention module and the third attention module can be any combination. The multiple attention modules may be adjacent to each other, or some of the multiple attention modules may be adjacent to each other, or any two of the multiple attention modules may be separated by other layers in the synthetic transformation network, that is, some of the multiple attention modules may be connected through other layers in the synthetic transformation network. For the attention modules that are adjacent to each other, they may be connected in series or in parallel, and a connection layer may be introduced for connection, which is not specifically limited in the present application.

[0205] Combine the following Figure 21 to Figure 24 An exemplary description of the layout of the attention module in the synthetic transformer network.

[0206] Figure 21 to Figure 24 This is an example of the distribution of the attention module provided in an embodiment of the present application.

[0207] like Fig.21 As shown, is the transformation value (or transformation reconstruction value) of the current image, is the reconstructed image of the current image. Assume that the synthetic transformation network includes 4 upsampling layers, the second upsampling layer of the 4 upsampling layers is connected to the attention module A, and the third upsampling layer is connected to the attention module B. Among them, the attention module A can be the first attention module, the second attention module or the third attention module mentioned above. Similarly, the attention module B can be the first attention module, the second attention module or the third attention module mentioned above.

[0208] like Fig. 22 As shown, is the transformation value (or transformation reconstruction value) of the current image, is the reconstructed image of the current image. Assume that the synthetic transformation network includes 4 upsampling layers, and the third upsampling layer of the 4 upsampling layers is connected to the series-connected attention module B and attention module A. Among them, the attention module A can be the first attention module, the second attention module or the third attention module mentioned above. Similarly, the attention module B can be the first attention module, the second attention module or the third attention module mentioned above.

[0209] like Fig.23 As shown, is the transformation value (or transformation reconstruction value) of the current image, is the reconstructed image of the current image. Assume that the analysis transformation network includes 4 upsampling layers, and the third upsampling layer of the 4 upsampling layers is connected to the series-connected attention module B and attention module A. In addition, the feature information output by the attention module A is connected to a connection layer, and the feature information input to the attention module B is jumped to the connection layer connected to the attention module B. Among them, the attention module A can be the first attention module, the second attention module or the third attention module mentioned above. Similarly, the attention module B can be the first attention module, the second attention module or the third attention module mentioned above.

[0210] like Fig.24 As shown, is the transformation value (or transformation reconstruction value) of the current image, is the reconstructed image of the current image. Assume that the synthetic transformation network includes 4 upsampling layers, and the third upsampling layer of the 4 upsampling layers is connected to the attention module B and the attention module A in parallel. Among them, the attention module A can be the first attention module, the second attention module or the third attention module mentioned above. Similarly, the attention module B can be the first attention module, the second attention module or the third attention module mentioned above.

[0211] It should be noted that the present application does not limit the specific type of activation function involved. For example, it may include but is not limited to: Sigmoid activation function, Rule activation function, LeakyReLu activation function, Exponential Linear Units (ELU) activation function, GELU activation function, etc.

[0212] In addition, the present application does not limit the specific type of the connection layer involved, for example, it includes but is not limited to: a connection layer for realizing element wise addition (for example Fig.14 The circles with plus signs shown in ), the connection layers for implementing element wise multiplication (e.g. Fig.16 The circles with black dots shown in ), the connection layers for implementing matrix multiplication (e.g. Fig.14 The circles with multiplication signs shown in ), the connection layers used to implement reshape (e.g. Fig.14 The circles with R shown in FIG) and the connection layers (eg, Fig.16 The circle with the S line shown in ).

[0213] In addition, the convolution layer involved in the present application may be a traditional convolution layer, or a grouped convolution or other types of convolution layers, and the present application does not make any specific limitations on this.

[0214] In addition, the term "connection" involved in this application may include direct connection or indirect connection. For example, module 1 is connected to module 2, which may include: module 1 is directly connected to module 2, or module 1 is indirectly connected to module 2. Among them, module 1 is directly connected to module 2 means: the output of module 1 is the input of module 2, and module 1 is indirectly connected to module 2 means: the output of module 1 is processed by other modules as the input of module 2. Module 1 is connected to module 2 can be equivalently replaced by: module 1 is connected to module 2.

[0215] The following will be combined Fig.25 The encoding method according to the embodiment of the present application is described from the perspective of the encoder.

[0216] Fig.25 It is a schematic flowchart of the encoding method 300 provided in this application.

[0217] It should be understood that the encoding method 300 may be performed by an encoder. Figure 1 The encoding device 110 or video encoder 112 is shown to perform.

[0218] like Fig.25 As shown, the encoding method 300 may include:

[0219] S310, processing the current image by analyzing the transformation network to obtain a transformation value of the current image;

[0220] The one or more layers in the analysis transformation network are first upsampling layers, and the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer;

[0221] S320: Encode the current image based on the transformation value of the current image to obtain a code stream.

[0222] In some embodiments, the i-th layer in the analysis transformation network is the first upsampling layer; wherein S220 may include:

[0223] Using the first i-1 layers of the analysis transformation network, the i-1th feature information of the current image is determined based on the transformation value of the current image; the format of the i-1th feature information is C×H×W, C represents the number of channels of the i-1th feature information, H represents the height of the i-1th feature information, and W represents the width of the i-1th feature information; C, H and W are all positive integers; using the first convolutional layer, the i-1th feature information is transformed into a channel, and the format is s 2 C×H×W feature information; s is a positive integer; using the pixel reorganization layer, the format is s 2 The feature information of C×H×W is reorganized to obtain the i-th feature information of the current image, and the format of the i-th feature information is C×sH×sW; the layers except the first i layers in the analysis transformation network are used to determine the reconstructed image based on the i+1-th feature information.

[0224] In some embodiments, the number of the first upsampling layers in the analysis transformation network is negatively correlated with at least one of the following: a bit rate adopted by the current image, and an upper limit value of the computational complexity of the analysis transformation network.

[0225] In some embodiments, one or more layers of the analysis transformation network are second upsampling layers, and the structure of the second upsampling layer is different from the structure of the first upsampling layer.

[0226] In some embodiments, the second upsampling layer is a deconvolution layer.

[0227] In some embodiments, the number of the second upsampling layers in the analysis transformation network is positively correlated with at least one of the following: a bit rate adopted by the current image, and an upper limit value of the computational complexity of the analysis transformation network.

[0228] In some embodiments, when the analysis transform network is a network used by the first component, the sum of upsampling multiples of each upsampling layer in the analysis transform network is different from the sum of upsampling multiples of each upsampling layer in the analysis transform network when the analysis transform network is a network used by the second component.

[0229] In some embodiments, when the first component is a luminance component and the second component is a chrominance component, the sum of upsampling multiples of each upsampling layer in the analysis transform network when the analysis transform network is a network used for the first component is greater than the sum of upsampling multiples of each upsampling layer in the analysis transform network when the analysis transform network is a network used for the second component.

[0230] In some embodiments, one or more layers in the analysis transformation network are convolutional layers.

[0231] In some embodiments, the analysis transformation network includes a first residual block; the first residual block includes: a second convolutional layer, a first activation function and a first connection layer, the second convolutional layer is connected to the first connection layer through the first activation function, and the input features of the second convolutional layer are jump-connected to the first connection layer.

[0232] In some embodiments, one or more layers in the analysis transform network are the first residual block.

[0233] In some embodiments, one or more layers in the analysis transformation network is a first attention module, and the first attention module is located at least one of the following positions in the synthesis transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit; wherein the first attention module includes the first residual block.

[0234] In some embodiments, the first attention module is a first non-local residual block, which includes: a first branch, a second branch, a second connection layer and a third connection layer, the first branch and the second branch are both connected to the third connection layer through the second connection layer, the input feature information of the first non-local residual block is jump connected to the third connection layer, and at least one of the first branch and the second branch includes the first residual block.

[0235] In some embodiments, the first attention module includes a first multi-head transposition attention module; the first multi-head transposition attention module includes: a third branch, a fourth branch, a fifth branch, a fourth connection layer, a fifth connection layer and a sixth connection layer, the third branch and the fourth branch are both connected to the fifth connection layer through the fourth connection layer, the fifth branch is connected to the fifth connection layer through the fourth connection layer, the input feature information of the first multi-head transposition attention module is jump connected to the fifth connection layer, and at least one of the third branch, the fourth branch and the fifth branch includes the first residual block.

[0236] In some embodiments, the first attention module includes a first gated feedforward network; the first gated feedforward network includes: a sixth branch, a seventh branch, a seventh connection layer and an eighth connection layer, the sixth branch and the seventh branch are both connected to the eighth connection layer through the seventh connection layer, and at least one of the sixth branch and the seventh branch includes the first residual block.

[0237] In some embodiments, one or more layers in the analysis transformation network are a second attention module, and the second attention module is located at at least one of the following positions in the synthesis transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit; wherein the second attention module includes a window attention unit and a grid attention unit connected after the window attention unit; the attention window unit is used to divide the feature information input to the second attention module into multiple sub-blocks and perform attention processing on the multiple sub-blocks to obtain local feature information, and the grid attention unit is used to divide the local feature information into multiple grids and perform attention processing on the multiple grids to obtain output feature information.

[0238] In some embodiments, one or more layers in the analysis transformation network is a third attention module, and the third attention module is located at least one of the following positions in the synthesis transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit; wherein the third attention module includes a depthwise separable convolution layer, at least one convolution layer and a ninth connection layer, the depthwise separable convolution layer is connected to the ninth connection layer through the at least one convolution layer, and the input feature information of the depthwise separable convolution layer is jump connected to the ninth connection layer.

[0239] It should be understood that the encoding method can be understood as the inverse process of the decoding method. Therefore, the specific scheme of the encoding method 300 can refer to the relevant content of the decoding method 200. For the convenience of description, this application will not go into details.

[0240] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the embodiments mentioned above. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the specific embodiments mentioned above can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0241] It should also be understood that in the various method embodiments of the present application, the size of the serial numbers of the processes involved above does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0242] Combined with the following Figure 26 to Figure 28 The device embodiments of the present application are described in detail.

[0243] Fig.26 is a schematic block diagram of a decoder 400 provided in the present application.

[0244] like Fig.26 As shown, the decoder 400 may include:

[0245] The decoding unit 410 is used to decode the code stream of the current image to obtain a residual value of the current image, and determine a transformation value of the current image based on the residual value;

[0246] A reconstruction unit 420, configured to process the transformation value of the current image through a synthetic transformation network to obtain a reconstructed image of the current image;

[0247] Among them, one or more layers in the synthetic transformation network are a first upsampling layer, and the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer.

[0248] In some embodiments, the i-th layer in the synthetic transformation network is the first upsampling layer; wherein the reconstruction unit 420 is specifically used for:

[0249] Using the first i-1 layers of the synthetic transformation network, determine the i-1th feature information of the current image based on the transformation value of the current image; the format of the i-1th feature information is C×H×W, C represents the number of channels of the i-1th feature information, H represents the height of the i-1th feature information, and W represents the width of the i-1th feature information; C, H and W are all positive integers; using the first convolutional layer, perform channel transformation on the i-1th feature information to obtain a format of s 2 C×H×W feature information; s is a positive integer; using the pixel reorganization layer, the format is s 2 The feature information of C×H×W is reorganized to obtain the i-th feature information of the current image, and the format of the i-th feature information is C×sH×sW; the reconstructed image is determined based on the i+1-th feature information using the layers other than the first i layers in the synthetic transformation network.

[0250] In some embodiments, the number of the first upsampling layers in the synthesis transformation network is negatively correlated with at least one of the following: a bit rate adopted by the current image, and an upper limit value of the computational complexity of the synthesis transformation network.

[0251] In some embodiments, one or more layers of the synthesis transformation network are second upsampling layers, and the structure of the second upsampling layer is different from the structure of the first upsampling layer.

[0252] In some embodiments, the second upsampling layer is a deconvolution layer.

[0253] In some embodiments, the number of the second upsampling layers in the synthesis transformation network is positively correlated with at least one of the following: a bit rate adopted by the current image, and an upper limit value of the computational complexity of the synthesis transformation network.

[0254] In some embodiments, when the synthesis transformation network is a network used by the first component, the sum of upsampling multiples of each upsampling layer in the synthesis transformation network is different from the sum of upsampling multiples of each upsampling layer in the synthesis transformation network when the synthesis transformation network is a network used by the second component.

[0255] In some embodiments, when the first component is a luminance component and the second component is a chrominance component, the sum of upsampling multiples of each upsampling layer in the synthesis transformation network when the synthesis transformation network is a network used for the first component is greater than the sum of upsampling multiples of each upsampling layer in the synthesis transformation network when the synthesis transformation network is a network used for the second component.

[0256] In some embodiments, one or more layers in the synthetic transformation network are convolutional layers.

[0257] In some embodiments, the synthetic transformation network includes a first residual block; the first residual block includes: a second convolutional layer, a first activation function and a first connection layer, the second convolutional layer is connected to the first connection layer through the first activation function, and the input features of the second convolutional layer are jump-connected to the first connection layer.

[0258] In some embodiments, one or more layers in the synthesis transformation network are the first residual block.

[0259] In some embodiments, one or more layers in the synthetic transformation network is a first attention module, and the first attention module is located at least one of the following positions in the synthetic transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit; wherein the first attention module includes the first residual block.

[0260] In some embodiments, the first attention module is a first non-local residual block, which includes: a first branch, a second branch, a second connection layer and a third connection layer, the first branch and the second branch are both connected to the third connection layer through the second connection layer, the input feature information of the first non-local residual block is jump connected to the third connection layer, and at least one of the first branch and the second branch includes the first residual block.

[0261] In some embodiments, the first attention module includes a first multi-head transposition attention module; the first multi-head transposition attention module includes: a third branch, a fourth branch, a fifth branch, a fourth connection layer, a fifth connection layer and a sixth connection layer, the third branch and the fourth branch are both connected to the fifth connection layer through the fourth connection layer, the fifth branch is connected to the fifth connection layer through the fourth connection layer, the input feature information of the first multi-head transposition attention module is jump connected to the fifth connection layer, and at least one of the third branch, the fourth branch and the fifth branch includes the first residual block.

[0262] In some embodiments, the first attention module includes a first gated feedforward network; the first gated feedforward network includes: a sixth branch, a seventh branch, a seventh connection layer and an eighth connection layer, the sixth branch and the seventh branch are both connected to the eighth connection layer through the seventh connection layer, and at least one of the sixth branch and the seventh branch includes the first residual block.

[0263] In some embodiments, one or more layers in the synthetic transformation network are a second attention module, and the second attention module is located at least one of the following positions in the synthetic transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit; wherein the second attention module includes a window attention unit and a grid attention unit connected after the window attention unit; the attention window unit is used to divide the feature information input to the second attention module into multiple sub-blocks and perform attention processing on the multiple sub-blocks to obtain local feature information, and the grid attention unit is used to divide the local feature information into multiple grids and perform attention processing on the multiple grids to obtain output feature information.

[0264] In some embodiments, one or more layers in the synthetic transformation network is a third attention module, and the third attention module is located at least one of the following positions in the synthetic transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit; wherein the third attention module includes a depthwise separable convolution layer, at least one convolution layer and a ninth connection layer, the depthwise separable convolution layer is connected to the ninth connection layer through the at least one convolution layer, and the input feature information of the depthwise separable convolution layer is jump-connected to the ninth connection layer.

[0265] It should be understood that the device embodiment of the decoder and the method embodiment of the decoding method may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they are not described here. Specifically, Fig.26 The decoder 400 shown may correspond to the corresponding subject in the decoding method 200 of the embodiment of the present application, and the aforementioned and other operations and / or functions of each unit in the decoder 400 are respectively for implementing the corresponding processes in the decoding method 200.

[0266] Fig. 27 is a schematic block diagram of the encoder 500 provided in the present application.

[0267] like Fig. 27 As shown, the encoder 500 may include:

[0268] The transformation unit 510 is used to process the current image by analyzing the transformation network to obtain the transformation value of the current image;

[0269] The one or more layers in the analysis transformation network are first upsampling layers, and the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer;

[0270] The encoding unit 520 is used to encode the current image based on the transformation value of the current image to obtain a code stream.

[0271] In some embodiments, the i-th layer in the analysis transformation network is the first upsampling layer; wherein the transformation unit 510 is specifically used to:

[0272] Using the first i-1 layers of the analysis transformation network, the i-1th feature information of the current image is determined based on the transformation value of the current image; the format of the i-1th feature information is C×H×W, C represents the number of channels of the i-1th feature information, H represents the height of the i-1th feature information, and W represents the width of the i-1th feature information; C, H and W are all positive integers; using the first convolutional layer, the i-1th feature information is transformed into a channel, and the format is s 2 C×H×W feature information; s is a positive integer; using the pixel reorganization layer, the format is s 2 The feature information of C×H×W is reorganized to obtain the i-th feature information of the current image, and the format of the i-th feature information is C×sH×sW; the layers except the first i layers in the analysis transformation network are used to determine the reconstructed image based on the i+1-th feature information.

[0273] In some embodiments, the number of the first upsampling layers in the analysis transformation network is negatively correlated with at least one of the following: a bit rate adopted by the current image, and an upper limit value of the computational complexity of the analysis transformation network.

[0274] In some embodiments, one or more layers of the analysis transformation network are second upsampling layers, and the structure of the second upsampling layer is different from the structure of the first upsampling layer.

[0275] In some embodiments, the second upsampling layer is a deconvolution layer.

[0276] In some embodiments, the number of the second upsampling layers in the analysis transformation network is positively correlated with at least one of the following: a bit rate adopted by the current image, and an upper limit value of the computational complexity of the analysis transformation network.

[0277] In some embodiments, when the analysis transform network is a network used by the first component, the sum of upsampling multiples of each upsampling layer in the analysis transform network is different from the sum of upsampling multiples of each upsampling layer in the analysis transform network when the analysis transform network is a network used by the second component.

[0278] In some embodiments, when the first component is a luminance component and the second component is a chrominance component, the sum of upsampling multiples of each upsampling layer in the analysis transform network when the analysis transform network is a network used for the first component is greater than the sum of upsampling multiples of each upsampling layer in the analysis transform network when the analysis transform network is a network used for the second component.

[0279] In some embodiments, one or more layers in the analysis transformation network are convolutional layers.

[0280] In some embodiments, the analysis transformation network includes a first residual block; the first residual block includes: a second convolutional layer, a first activation function and a first connection layer, the second convolutional layer is connected to the first connection layer through the first activation function, and the input features of the second convolutional layer are jump-connected to the first connection layer.

[0281] In some embodiments, one or more layers in the analysis transform network are the first residual block.

[0282] In some embodiments, one or more layers in the analysis transformation network is a first attention module, and the first attention module is located at least one of the following positions in the synthesis transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit; wherein the first attention module includes the first residual block.

[0283] In some embodiments, the first attention module is a first non-local residual block, which includes: a first branch, a second branch, a second connection layer and a third connection layer, the first branch and the second branch are both connected to the third connection layer through the second connection layer, the input feature information of the first non-local residual block is jump connected to the third connection layer, and at least one of the first branch and the second branch includes the first residual block.

[0284] In some embodiments, the first attention module includes a first multi-head transposition attention module; the first multi-head transposition attention module includes: a third branch, a fourth branch, a fifth branch, a fourth connection layer, a fifth connection layer and a sixth connection layer, the third branch and the fourth branch are both connected to the fifth connection layer through the fourth connection layer, the fifth branch is connected to the fifth connection layer through the fourth connection layer, the input feature information of the first multi-head transposition attention module is jump connected to the fifth connection layer, and at least one of the third branch, the fourth branch and the fifth branch includes the first residual block.

[0285] In some embodiments, the first attention module includes a first gated feedforward network; the first gated feedforward network includes: a sixth branch, a seventh branch, a seventh connection layer and an eighth connection layer, the sixth branch and the seventh branch are both connected to the eighth connection layer through the seventh connection layer, and at least one of the sixth branch and the seventh branch includes the first residual block.

[0286] In some embodiments, one or more layers in the analysis transformation network are a second attention module, and the second attention module is located at at least one of the following positions in the synthesis transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit; wherein the second attention module includes a window attention unit and a grid attention unit connected after the window attention unit; the attention window unit is used to divide the feature information input to the second attention module into multiple sub-blocks and perform attention processing on the multiple sub-blocks to obtain local feature information, and the grid attention unit is used to divide the local feature information into multiple grids and perform attention processing on the multiple grids to obtain output feature information.

[0287] In some embodiments, one or more layers in the analysis transformation network is a third attention module, and the third attention module is located at least one of the following positions in the synthesis transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit; wherein the third attention module includes a depthwise separable convolution layer, at least one convolution layer and a ninth connection layer, the depthwise separable convolution layer is connected to the ninth connection layer through the at least one convolution layer, and the input feature information of the depthwise separable convolution layer is jump connected to the ninth connection layer.

[0288] It should be understood that the device embodiment of the encoder and the method embodiment of the encoding method may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they are not described here. Specifically, Fig. 27 The encoder 500 shown may correspond to the corresponding subject in the encoding method 300 of the embodiment of the present application, and the aforementioned and other operations and / or functions of each unit in the encoder 500 are respectively for implementing the corresponding processes in each method such as the encoding method 300.

[0289] It should also be understood that each unit in the decoder 400 or encoder 500 involved in the embodiment of the present application is divided based on logical functions. In practical applications, the function of a unit can also be realized by multiple units, or the function of multiple units is realized by one unit, and even, these functions can also be assisted by one or more other units. For example, part or all of the decoder 400 or encoder 500 is merged into one or several other units. For another example, a certain (some) unit in the decoder 400 or encoder 500 can also be split into multiple units smaller in function to constitute, which can achieve the same operation without affecting the realization of the technical effect of the embodiment of the present application. For another example, the decoder 400 or encoder 500 can also include other units, and in practical applications, these functions can also be assisted by other units, and can be realized by the collaboration of multiple units.

[0290] According to another embodiment of the present application, a computer program (including program code) capable of executing each step involved in the corresponding method can be run on a general computing device of a general-purpose computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM) to construct the decoder 400 or encoder 500 involved in the embodiment of the present application, and to implement the encoding method or decoding method of the embodiment of the present application. The computer program can be recorded on, for example, a computer-readable storage medium, and loaded into an electronic device through a computer-readable storage medium, and run therein to implement the corresponding method of the embodiment of the present application. In other words, the units involved above can be implemented in hardware form, can be implemented in software form, and can also be implemented in the form of a combination of hardware and software. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the integrated logic circuit of the hardware in the processor and / or the instruction in software form, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software in the decoding processor to perform. Optionally, the software may be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the method embodiment mentioned above in combination with its hardware.

[0291] Fig.28 It is a schematic structural diagram of an electronic device 600 provided in this application.

[0292] like Fig.28 As shown, the electronic device 600 at least includes a processor 610 and a computer-readable storage medium 620. The processor 610 and the computer-readable storage medium 620 may be connected via a bus or other means. The computer-readable storage medium 620 is used to store a computer program 621, which includes computer instructions, and the processor 610 is used to execute the computer instructions stored in the computer-readable storage medium 620. The processor 610 is the computing core and control core of the electronic device 600, which is suitable for implementing one or more computer instructions, and is specifically suitable for loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.

[0293] Exemplarily, the processor 610 may also be referred to as a central processing unit (CPU). The processor 610 may include, but is not limited to, a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, discrete hardware components, and the like.

[0294] Exemplarily, the computer-readable storage medium 620 may be a high-speed RAM memory, or a non-volatile memory (Non-VolatileMemory), such as at least one disk memory; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor 610. Specifically, the computer-readable storage medium 620 includes, but is not limited to: a volatile memory and / or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0295] Exemplarily, the electronic device 600 may be a decoder or decoding framework involved in an embodiment of the present application; a first computer instruction is stored in the computer-readable storage medium 620; the processor 610 loads and executes the first computer instruction stored in the computer-readable storage medium 620 to implement the corresponding steps in the decoding method provided in the present application; in other words, the first computer instruction in the computer-readable storage medium 620 is loaded by the processor 610 and the corresponding steps are executed, which will not be repeated here to avoid repetition.

[0296] Exemplarily, the electronic device 600 may be an encoder or encoding framework involved in an embodiment of the present application; a second computer instruction is stored in the computer-readable storage medium 620; the second computer instruction stored in the computer-readable storage medium 620 is loaded and executed by the processor 610 to implement the corresponding steps in the encoding method provided in the present application; in other words, the second computer instruction in the computer-readable storage medium 620 is loaded by the processor 610 and the corresponding steps are executed, which will not be repeated here to avoid repetition.

[0297] According to another aspect of the present application, the present application also provides a coding and decoding system, including the encoder and decoder mentioned above.

[0298] According to another aspect of the present application, the present application also provides a computer-readable storage medium (Memory), which is a memory device in a decoder or encoder for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in electronic devices and, of course, extended storage media supported by electronic devices. The computer-readable storage medium provides a storage space that stores the operating system of the electronic device. In addition, one or more computer instructions suitable for being loaded and executed by a processor are also stored in the storage space. These computer instructions can be one or more computer programs (including program codes).

[0299] According to another aspect of the present application, the present application further provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer executes the encoding method or the decoding method provided in the various optional modes mentioned above.

[0300] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process of the embodiment of the present application is run in whole or in part or the function of the embodiment of the present application is realized. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or can be transmitted between a computer-readable storage medium and another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center.

[0301] According to another aspect of the present application, the present application further provides a code stream, which may be a code stream decoded using the decoding method provided by the present application or a code stream generated using the encoding method provided by the present application.

[0302] Those of ordinary skill in the art will appreciate that the units and process steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0303] Finally, it should be noted that the above content is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image decoding method, characterized in that: include: Decoding a code stream of a current image to obtain a residual value of the current image, and determining a transformation value of the current image based on the residual value; Processing the transformation value of the current image through a synthetic transformation network to obtain a reconstructed image of the current image; Among them, one or more layers in the synthetic transformation network are a first upsampling layer, and the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer.

2. The method according to claim 1, characterized in that The i-th layer in the synthetic transformation network is the first upsampling layer; The step of processing the transformation value of the current image through a synthetic transformation network to obtain a reconstructed image of the current image includes: Determine the i-1th feature information of the current image based on the transformation value of the current image by using the first i-1 layers of the synthetic transformation network; the format of the i-1th feature information is C×H×W, C represents the number of channels of the i-1th feature information, H represents the height of the i-1th feature information, and W represents the width of the i-1th feature information; C, H and W are all positive integers; Using the first convolutional layer, the channel transformation is performed on the i-1th feature information to obtain a format of s 2 C×H×W feature information; s is a positive integer; Using the pixel reorganization layer, the format is s 2 Reorganize the pixels of the feature information of C×H×W to obtain the i-th feature information of the current image, where the format of the i-th feature information is C×sH×sW; The reconstructed image is determined based on the (i+1)th feature information using layers other than the first i layers in the synthetic transformation network.

3. The method according to claim 1, characterized in that The number of the first upsampling layers in the synthesis transformation network is negatively correlated with at least one of the following: a bit rate adopted by the current image and an upper limit value of the computational complexity of the synthesis transformation network.

4. The method according to claim 1, characterized in that: One or more layers of the synthetic transform network are second upsampling layers, and the structure of the second upsampling layer is different from the structure of the first upsampling layer.

5. The method according to claim 4, characterized in that The number of the second upsampling layers in the synthesis transformation network is positively correlated with at least one of the following: a bit rate adopted by the current image and an upper limit value of the computational complexity of the synthesis transformation network.

6. The method according to claim 1, characterized in that When the synthesis transformation network is a network used by the first component, the sum of upsampling multiples of each upsampling layer in the synthesis transformation network is different from the sum of upsampling multiples of each upsampling layer in the synthesis transformation network when the synthesis transformation network is a network used by the second component.

7. The method according to claim 6, characterized in that When the first component is a luminance component and the second component is a chrominance component, the sum of upsampling multiples of each upsampling layer in the synthesis transformation network when the synthesis transformation network is a network used for the first component is greater than the sum of upsampling multiples of each upsampling layer in the synthesis transformation network when the synthesis transformation network is a network used for the second component.

8. The method according to claim 1, characterized in that: The synthetic transformation network includes a first residual block; the first residual block includes: a second convolutional layer, a first activation function and a first connection layer, the second convolutional layer is connected to the first connection layer through the first activation function, and the input features of the second convolutional layer are jump-connected to the first connection layer.

9. The method according to claim 8, characterized in that One or more layers in the synthetic transformation network are first attention modules, and the first attention module is located at least one of the following positions in the synthetic transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit; Wherein, the first attention module includes the first residual block.

10. The method according to claim 9, characterized in that The first attention module is a first non-local residual block, which includes: a first branch, a second branch, a second connection layer and a third connection layer, the first branch and the second branch are both connected to the third connection layer through the second connection layer, the input feature information of the first non-local residual block is jump-connected to the third connection layer, and at least one of the first branch and the second branch includes the first residual block.

11. The method according to claim 9, characterized in that The first attention module includes a first multi-head transposition attention module; The first multi-head transposition attention module includes: a third branch, a fourth branch, a fifth branch, a fourth connection layer, a fifth connection layer and a sixth connection layer, the third branch and the fourth branch are both connected to the fifth connection layer through the fourth connection layer, the fifth branch is connected to the fifth connection layer through the fourth connection layer, the input feature information of the first multi-head transposition attention module is jump connected to the fifth connection layer, and at least one of the third branch, the fourth branch and the fifth branch includes the first residual block.

12. The method according to claim 9, characterized in that The first attention module includes a first gated feedforward network; the first gated feedforward network includes: a sixth branch, a seventh branch, a seventh connection layer and an eighth connection layer, the sixth branch and the seventh branch are both connected to the eighth connection layer through the seventh connection layer, and at least one of the sixth branch and the seventh branch includes the first residual block.

13. The method according to claim 1, characterized in that One or more layers in the synthetic transformation network are second attention modules, and the second attention module is located at least one of the following positions in the synthetic transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit; Among them, the second attention module includes a window attention unit and a grid attention unit connected to the window attention unit; the attention window unit is used to divide the feature information input to the second attention module into multiple sub-blocks and perform attention processing on the multiple sub-blocks to obtain local feature information, and the grid attention unit is used to divide the local feature information into multiple grids and perform attention processing on the multiple grids to obtain output feature information.

14. The method according to claim 1, characterized in that One or more layers in the synthetic transformation network are third attention modules, and the third attention module is located at least one of the following positions in the synthetic transformation network: before the upsampling layer, after the upsampling layer, before the convolution layer, after the convolution layer, before the residual activation unit, and after the residual activation unit; Among them, the third attention module includes a depth-separable convolutional layer, at least one convolutional layer and a ninth connection layer, the depth-separable convolutional layer is connected to the ninth connection layer through the at least one convolutional layer, and the input feature information of the depth-separable convolutional layer is jump-connected to the ninth connection layer.

15. An image encoding method, characterized in that: include: Processing the current image through an analysis transformation network to obtain a transformation value of the current image; The one or more layers in the analysis transformation network are first upsampling layers, and the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer; Based on the transformation value of the current image, the current image is encoded to obtain a code stream.

16. A decoder, characterized in that: include: A decoding unit, configured to decode a code stream of a current image to obtain a residual value of the current image, and determine a transformation value of the current image based on the residual value; A reconstruction unit, configured to process the transformation value of the current image through a synthetic transformation network to obtain a reconstructed image of the current image; Among them, one or more layers in the synthetic transformation network are a first upsampling layer, and the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer.

17. An encoder, characterized in that: include: A transformation unit, used for processing the current image by analyzing the transformation network to obtain a transformation value of the current image; The one or more layers in the analysis transformation network are first upsampling layers, and the first upsampling layer includes a first convolutional layer and a pixel reorganization layer connected after the first convolutional layer; The encoding unit is used to encode the current image based on the transformation value of the current image to obtain a code stream.

18. An electronic device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the method according to any one of claims 1 to 14 or the method according to claim 15 is implemented.

19. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 14 or the method according to claim 15.

20. A code stream, characterized in that The code stream is a code stream decoded by the method according to any one of claims 1 to 14, or the code stream is a code stream generated by the method according to claim 15.

Images