Learning image compression and decompression using long and short attention modules

Abstract

A method of feature extraction using a neural network, the method comprising: extracting a first set of features from an input feature set by processing the input feature set via at least two residual blocks of the neural network sequentially connected to each other; and extracting a second group of features from the input feature set, the extraction of the second group of features comprising the following steps: extracting a third group of features and a fourth group of features from the input feature set through non-local attention processing, and performing group convolution on the input feature set, the third group of features and the fourth group of features through a multi-head mechanism to obtain the second group of features.

Description

Technical Field

[0001] The present invention relates to the technical field of visual information compression and decompression, and more specifically, to a method and device for extracting features in learning image compression and decompression using a long-short attention module based on a neural network. Background Art

[0002] Visual information such as pictures or images (e.g., still pictures (still images) and moving pictures (moving images) (e.g., picture streams and videos)) is one of the main media for acquiring information. For example, nowadays, in order to exchange various information between users, the following actions are increasingly performed: transmitting still images through wired and / or wireless networks, transmitting videos and / or video streams through wired or wireless mobile networks, broadcasting digital television signals, conducting real-time video conversations (e.g., video chats or video conferences) through wired or wireless mobile networks, and storing images and videos on portable storage media (e.g., DVD disks or Blu-ray disks).

[0003] However, uncompressed images usually consume a lot of resources when saved and transmitted. Therefore, in order to efficiently store and transmit images, image compression and decompression algorithms are becoming increasingly important.

[0004] Image compression and decompression involve encoding and decoding respectively. Encoding is the process of compressing and possibly changing the format of the picture (image) content. Encoding is important because it reduces the bandwidth required to transmit pictures (images) over wired or wireless mobile networks. Decoding, on the other hand, is the process of decoding or decompressing a coded or compressed picture (image). Since encoding and decoding are applicable to different devices, encoding standards and decoding standards called codecs have been developed. A codec is generally an algorithm used to encode and decode images.

[0005] The Joint Photographic Experts Group (JPEG) was established in 1994. As an international standard for image compression, JPEG is still one of the most widely used image compression algorithms (see non-patent reference [1]). In addition, in the past few decades, a variety of image compression standards have been released, such as JPEG2000 (see non-patent reference [2]) and BPG (see non-patent reference [3]). The latest H.266 / Versatile Video Coding (VVC) standard was officially released in 2018, which represents the most advanced available coding technology (see non-patent reference [4]). Compared with the previous generation standard (H.265 / HEVC), H.266 further improves compression performance, reducing the data size for users by 50% while maintaining video quality. It is also constantly updated and iterated to achieve higher compression efficiency.

[0006] Like many other technical fields, deep neural networks have been widely used in the field of image compression and decompression. The two characteristics of local connection and parameter sharing in the convolutional neural network (CNN) operation reflect its advantages in image compression. Unlike traditional methods, end-to-end optimization based on CNN requires the function to be globally differentiable in gradient descent. To solve this problem, Ballé et al. (see non-patent reference [5]) proposed a CNN-based image coding framework based on generalized divergence normalization in 2016. Its network structure is mainly divided into two parts: one part is responsible for analyzing the potential representation of the image, and the other part is responsible for reconstruction and inverse process using the generalized divergence normalization function as the activation function. This method achieves coding performance comparable to JPEG2000.

[0007] Later, Ballé et al. (see non-patent reference [6]) combined the super-prior codec with the previous method to further reduce the spatial redundancy between feature maps and improve the compression efficiency.

[0008] These two works provide the feasibility of CNN-based image compression. The latest research proposed by Cheng et al. (see non-patent reference [7]) in 2020 is an image compression method based on Gaussian mixture model and attention mechanism, which achieves compression efficiency comparable to VVC on the Kodak dataset.

[0009] There are two modules in deep learning-based image compression that have a significant impact on bit savings. One module is used to build a flexible and accurate entropy model to help the network encode and decode simultaneously to save bits, while the other module is used to extract a more accurate latent representation of the image in the autoencoder stage, thereby reducing spatial redundancy to save bits.

[0010] Cited prior art

[0011] [Non-patent literature]

[0012] Non-patent literature 1Gregory K. Wallace, "The JPEG still picture compression standard", IEEE Transactions on Consumer Electronics, Vol. 38, No. 1, pp. xviii-xxxiv, 1992.

[0013] Non-patent literature 2Majid Rabbani and Rajan Joshi, "An overview of the jpeg 2000 still image compression standard", Signal Processing: Image Communications, Vol. 17, No. 1, pp. 3-48, 2002.

[0014] Non-patent literature 3 Gary J Sullivan, Jens-Rainer Ohm, Woo-Jin Han and Thomas Wiegand, "Overview of the high efficiency video coding (HEVC) standard", IEEE Transactions on Circuits and Systems for Video Technology, Vol. 22, No. 12, pp. 1649-1668, 2012.

[0015] Non-patent document 4Jens-Rainer Ohm and Gary J. Sullivan, "Versatile video coding toward the next generation of video compression," in Proceedings of the Symposium on Picture Coding, 2018, Vol. 2018.

[0016] Non-Patent Literature 5Johannes Balle, Valero Laparra, and Eero P. Simoncelli, “End-to-end optimized image compression,” arXiv preprint arXiv:1611.01704, 2016.

[0017] Non-Patent Literature 6Johannes Balle, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston, “Variational image compression with a scale hyperprior,” arXiv preprint arXiv:1802.01436, 2018.

[0018] Non-patent document 7 Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 7939-7948, 2020.

[0019] Non-patent literature 8 Jean Begaint, Fabien Racape, Simon Feltman, and Akshay Pushparaja, “Compressai: a pytorch library and evaluation platform for end-to-end compression research,” arXiv preprint arXiv:2011.03029, 2020.

[0020] Non-Patent Literature 9 David Minnen, Johannes Balle, and George D. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” Advances in Neural Information Processing Systems, vol. 31, 2018.

[0021] Technical issues

[0022] However, traditional image compression is very complex in the encoding stage. In the intra-frame prediction module of H.266 / VVC, 67 prediction modes need to be tried to find the prediction mode that best suits the current coding unit (CU). For example, traditional encoding methods require different encoding methods for each coding unit (CU) of the image, which increases the complexity of image compression while reducing the bit rate. Most deep learning methods based on end-to-end network structures use the same coding structure as the work of Ballé et al. (see non-patent reference [5]). Current deep learning-based image compression methods mainly focus on building accurate and flexible entropy models, while ignoring the accurate extraction of the potential representation of the image at the encoding end.

[0023] Therefore, there is a need to improve the accurate extraction of the latent representation of an image at the encoding end. Summary of the invention

[0024] The mentioned problems and disadvantages are solved by the subject matter of the independent claims. Further preferred embodiments are defined in the dependent claims. In particular, embodiments of the invention provide significant benefits regarding increased efficiency by saving bit rate in image compression.

[0025] According to a first aspect of the present invention, there is provided a method for feature extraction using a neural network, the method comprising:

[0026] extracting a first set of features from the input feature set by processing the input feature set through at least two residual blocks of a neural network connected sequentially to each other;

[0027] Extracting a second set of features from the input feature set, wherein extracting the second set of features comprises the following steps:

[0028] extracting a third set of features and a fourth set of features from the input feature set by non-local attention processing, and

[0029] Group convolution is performed on the input feature set, the third group of features, and the fourth group of features through a multi-head mechanism to obtain the second group of features.

[0030] According to a second aspect of the present invention, there is provided a method for learning image compression using a neural network, the method comprising performing the following steps:

[0031] extracting a set of features (x) from the input image data to be compressed; and

[0032] A set of features (y) is extracted from the set of features (x) extracted from the input image data to be compressed, the set of features (y) indicating a potential representation of the input image data to be compressed, by performing the following steps:

[0033] performing at least four steps of downsampling convolution on the set of features (x) extracted from the input image data by at least four convolutional layers of the neural network arranged in a streaming manner, and

[0034] After performing a two-step downsampling convolution process through two of the at least four convolutional layers, performing the step of feature extraction according to the first aspect at least once by using a set of features based on the output of the second convolutional layer of the two convolutional layers as the input feature set, thereby extracting the set of features (y) indicating the potential representation of the input image data; and

[0035] The method further comprises the following steps:

[0036] The extracted set of features (y) indicating a latent representation of the input image data is output.

[0037] According to a third aspect of the present invention, there is provided a method for learning image decompression using a neural network, the method comprising:

[0038] extracting a set of features from the input feature set to be decompressed; and

[0039] A set of features indicative of a reconstructed image of the input image data is extracted from the extracted set of features by performing the following steps:

[0040] The steps of the feature extraction method according to claim 1 are performed by using the set of features extracted from the input feature set to be decompressed as the input feature set,

[0041] performing an upsampling convolution process on the set of features extracted from the input feature set to be decompressed by at least four convolutional layers of the neural network arranged in a streaming manner, and

[0042] After two convolutional layers of the at least four convolutional layers perform two-step upsampling convolution processing, feature extraction according to the first aspect is performed at least once by using a set of features based on the output of a second convolutional layer of the two convolutional layers as an input feature set, thereby extracting a set of features indicating a reconstructed image of the input image data; and

[0043] The method further includes:

[0044] The extracted set of features indicative of a reconstructed image of the input image data is output.

[0045] According to a fourth aspect of the present invention, there is provided a method for learning image processing using a neural network, the method comprising:

[0046] Execute the steps of the second aspect above;

[0047] The output extracted set of features (y) indicative of a potential representation of the input image data is provided in the first processing path and the second processing path, wherein:

[0048] In the first processing path the following steps are performed:

[0049] The super encoder acquires (learns) modeling information from a set of features provided that indicate the latent representation,

[0050] The modeling information is quantized by a quantizer, and

[0051] The super decoder obtains decoding information from the quantized modeling information,

[0052] The following steps are performed in the second processing path:

[0053] Auxiliary information is obtained from the quantized latent representation through a context model, which indicates the latent representation quantized by the quantizer,

[0054] The entropy model combines the decoded information and the auxiliary information to calculate the modeling parameters; and

[0055] The output of the entropy decoder is calculated based on the modeling parameters; and the method further comprises the following steps:

[0056] The output of the entropy decoder is used as the input feature set to be decompressed, and a reconstructed image is generated by the image decompression method of the third aspect.

[0057] According to a fifth aspect of the present invention, there is provided an encoder for image data compression, the encoder comprising:

[0058] A first down-sampling processing device, the first down-sampling processing device is used to perform down-sampling processing on an input feature set obtained from the input image data;

[0059] a first filtering device, the first filtering device being used to extract a set of mixed features from the output of the first downsampling processing device, the mixed features comprising local features and global features;

[0060] A second down-sampling processing device, the second down-sampling processing device is used to perform down-sampling processing on the extracted set of mixed features; and

[0061] Second filter means for extracting a latent representation of the input image data from an output of second downsampling processing means for downsampling.

[0062] According to a sixth aspect of the present invention, there is provided a decoder for decompressing image data, the decoder comprising:

[0063] A first filtering device, the first filtering device being used to extract features from an input feature set to be decompressed;

[0064] A first upsampling processing device, the first upsampling processing device is used to perform upsampling processing on a set of features output from the pre-filter of the first filtering device;

[0065] a second filtering device, the second filtering device being used to extract features from a set of features output from the first upsampling processing device; and

[0066] The second up-sampling processing device is used to perform up-sampling processing on a set of features output from the second filtering device to obtain a set of features representing a reconstructed image of the input feature set. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Embodiments of the present invention will now be described with reference to the accompanying drawings, which are set forth for a better understanding of the concepts of the present invention but which should not be considered as limiting the present invention, in which:

[0068] Figure 1A A schematic diagram showing a general use case as an environment for adopting an embodiment of the present invention;

[0069] Figure 1B A schematic diagram showing the extraction of local features and global features by a long-short attention module according to an embodiment of the present invention;

[0070] Figure 1C A flow chart showing an embodiment of a method for feature extraction of the present invention is shown;

[0071] Figure 2 A schematic diagram showing the group convolution performed in global feature extraction in the long-short attention module;

[0072] Figure 3 A schematic diagram of a residual block in an embodiment of the present invention is shown;

[0073] Figure 4A A schematic diagram of a framework for image compression and decompression based on deep learning according to an embodiment of the present invention is shown;

[0074] Figure 4B Shows Figure 4A Schematic diagram of the neural network structure for deep learning-based image compression and decompression shown in;

[0075] Figure 5A Schematically illustrates a neural network structure of an encoder for learning image compression according to an embodiment of the present invention;

[0076] Figure 5B Schematically illustrates a neural network structure of a decoder for learning image decompression according to an embodiment of the present invention;

[0077] Figure 5C A flow chart showing an embodiment of a device for learning an encoder for image compression;

[0078] Figure 5D A flowchart showing an embodiment of a device for learning a decoder for image decompression is shown;

[0079] Figure 6 A schematic diagram showing a file format of a bit stream obtained by an image compression method according to an embodiment of the present invention;

[0080] Figure 7 A general apparatus embodiment of the present invention is shown;

[0081] Figure 8 Schematically illustrates a pipeline of a testing process for obtaining a quality score for a proposed learning image compression and decompression according to an embodiment of the present invention;

[0082] Fig. 9 A visual comparison between different methods in 00001-TE-960x720 of the JPEG AI dataset is shown;

[0083] Fig. 10A and Fig. 10B shows rate-distortion curves for comparing codec performance between different approaches; and

[0084] Fig.11 A table showing the average bitrate savings for five lambdas compared to the VVC (VTM 15.0) anchor point. DETAILED DESCRIPTION

[0085] In order to solve the above shortcomings of the current deep learning-based image processing methods, the present invention proposes to improve the feature extraction capability of the encoder and the reconstruction capability of the decoder, thereby improving the compression performance of the network. To this end, the present invention proposes a method for feature extraction using a neural network. The steps of the method are performed using the proposed neural network architecture described below, which is hereinafter referred to as a long-short attention module for feature extraction.

[0086] Figure 1A A schematic diagram of a general use case as an environment for adopting an embodiment of the present invention is shown. Devices 100-1 and 100-2 (e.g., a data center, a server, a processing device, and a data storage device, etc.) are arranged on the encoding side 1, which is used to store and process image data and generate one or more bit streams by encoding the image data.

[0087] In general, the term "image data" in the context of the present disclosure shall include all data that contains, indicates and / or can be processed to obtain images, pictures, picture / image streams, videos, and movies, etc., wherein in particular, a stream, video or movie may contain one or more images. Such data may also be referred to as visual data.

[0088] The image data may be monochrome data including grayscale information, or may include color information. Typically, each image is captured by at least one image capturing unit, thereby obtaining the image data.

[0089] At the encoding side 1, one or more bit streams generated are transmitted 500 to the decoding side 2 via any suitable network and data communication infrastructure, wherein, for example, a terminal such as a mobile device 200-1 is arranged, which receives the one or more bit streams, decodes the one or more bit streams, and processes the one or more bit streams to generate reconstructed image data for display on a display 200-2 of a (target) mobile device 200-1 or otherwise processes the same.

[0090] Figure 1C A flowchart of an embodiment of a general method for feature extraction using a neural network of the present invention is shown. More specifically, the method includes: step (110), extracting a first set of features from an input feature set by processing the input feature set via at least two residual blocks sequentially connected to each other of a neural network; and step (120), extracting a second set of features from the input feature set. The step (120) of extracting the second set of features includes: step (130), extracting a third set of features and a fourth set of features from the input feature set by non-local attention processing; and step (140), performing group convolution on the input feature set, the third set of features, and the fourth set of features by a multi-head mechanism to obtain the second set of features.

[0091] Figure 1B A schematic diagram of a long-short attention module for performing steps of a method for feature extraction using a neural network according to an embodiment of the present invention is shown. Figure 1A Having described a general use case, embodiments of the invention are further described below in the context of image data encoding and decoding. However, this is by no means limiting and the invention may also be applied to, for example, language modeling and machine translation or speech recognition.

[0092] Figure 1B A schematic diagram of a long-short attention module is shown for a neural network architecture and corresponding processing for performing the steps of the feature extraction method according to an embodiment of the present invention. The neural network may be, for example, a deep neural network whose structure consists of at least two layers. Therefore, it should be understood that the term "module" is used to represent the neural network architecture for performing the steps of the feature extraction method according to an embodiment of the present invention. Figure 1B In the other figures to be described, "N" is the number of channels of the neural network, and "h" is the number of heads of the multi-head attention mechanism to be described later. In the context of image data encoding and decoding, the extracted features may be small patches in the image data. In this context, each feature typically includes a feature keypoint and a feature descriptor. The feature keypoint can represent the two-dimensional (2D) position of the patch. The feature descriptor can represent the visual description of the patch. The feature descriptor is generally represented as a vector, also called a feature vector. As will be explained in detail above, Figure 1B At least some of the steps shown to be processed by the long-short attention module are optional and not limiting to the present invention.

[0093] The long-short attention module extracts the first set of features 11 from the input feature set 10 ( Figure 1C Step 110 in ) and the second set of features 12 ( Figure 1C 120 in step 120). The first set of features 11 is extracted from the input feature set by processing the input feature set via at least two residual blocks sequentially connected to each other of the neural network. It should be understood that in other embodiments of the present invention, more than two residual blocks sequentially connected to each other may also be used.

[0094] The input feature set 10 may be any type of feature set whose size and data type are suitable for the neural network structure of the long-short attention module of the embodiment of the present invention. The input feature set 10 is not limited to a specific data type, so the feature extraction performed by the long-short attention module can be used on general data types. For example, the data type may be image data.

[0095] In one embodiment of the present invention, the input feature set 10 is a multidimensional vector representing image data. For example, the feature extraction method for learning image processing according to the present invention can be applied to image encoding and / or image decoding, as further described in detail below. Therefore, the input feature set 10 is a multidimensional vector representing image data, and it should be understood that the input feature set directly or indirectly represents the image data, for example, it can directly represent the image data, or it can indirectly represent the image data by representing, for example, a potential representation of an encoded image.

[0096] In general, attention in deep learning can be described as a function that mimics cognitive attention, which focuses more on more important parts of the data. Attention overcomes the limitations of general encoder-decoder architectures with fixed-length internal representations.

[0097] The extraction of the first set of features 11 and the second set of features 12 are respectively shown in Figure 1B In the upper branch 20 and the lower branch 21 of the long-short attention module.

[0098] In one embodiment of the present invention, the extracted first set of features 11 may be a set of local features of the image data, and the extracted second set of features may be a set of global features of the image data.

[0099] Therefore, in one embodiment of the present invention, Figure 1B The upper branch 20 can extract a set of local features, and Figure 1B The lower branch 21 of can extract a set of global features. More specifically, Figure 1B The lower branch 21 of can extract a set of multi-scale global features. Then, Figure 1B The long-short attention module according to the present invention represented in FIG. 1 fuses the extracted first set of features 11 with the extracted second set of features 12.

[0100] In the present invention Figure 1B The upper branch 20 extracts a set of local features and Figure 1B In one embodiment of extracting a set of global features from the lower branch 21, the long-short attention module according to an embodiment of the present invention may fuse the extracted local features with the global features to obtain accurate potential features of the image data.

[0101] As described above in detail for the general method embodiment of the present invention, step 120 of extracting the second set of features includes: step 130, extracting the third set of features and the fourth set of features from the input feature set through non-local attention processing; and step 140, performing group convolution on the input feature set, the third set of features and the fourth set of features through a multi-head mechanism to obtain the second set of features.

[0102] to this end, Figure 1BThe first half of the lower branch 21 of the long-short attention module shown represents a simplified non-local attention module. The non-local attention module includes two paths based on a stacked network architecture and a residual block architecture. From the input feature set 10, the simplified non-local attention module performs processing by at least three residual blocks connected to each other in sequence to extract a third set of features 13 in the upper path 23. Figure 1B It is shown that the long-short attention module includes three residual blocks in the upper path 23, however, this is not limited to the present invention, and more than three residual blocks connected to each other in sequence may be used. However, using a larger number of residual blocks may affect the computation time and processing resources.

[0103] From the input feature set 10, the simplified non-local attention module extracts a fourth set of features 14 in the lower path 22. In order to extract the fourth set of features 14, the long-short attention module according to an embodiment of the present invention in the lower path 22 uses a neural network architecture based on at least three residual blocks connected to each other in sequence, at least one convolutional layer after the last residual block of the at least three residual blocks, and at least one activation layer after the at least one convolutional layer.

[0104] exist Figure 1B , it is shown that the long-short attention module also extracts the fourth set of features 14 through three residual blocks as used when extracting the third set of features 13 in the upper path 23, and an additional convolution layer and activation layer are used. However, the number of residual blocks may be more than three, and the number of convolution layers and activation layers in the lower path 22 may be more than one. In addition, the number of residual blocks in the upper path 23 does not have to be the same as the number of residual blocks in the lower path 22.

[0105] By having the activation layer on the lower path 22 apply an activation function to extract the fourth set of features 14, more complex calculations can be achieved by adding nonlinearity. By applying the activation function, the fourth set of features 14 may have values ​​between 0 and 1. For example, the fourth set of features 14 can be considered to represent the weights of the third set of features 13. However, the values ​​of the fourth set of features 14 are not limited to values ​​between 0 and 1, and may vary depending on the specific activation function applied.

[0106] Therefore, features in each of the third and fourth groups of features 13 and 14 that are less complex features (and thus may be less globally interpreted) may be distinguished from features that are more complex features (and thus may need to be interpreted as global features).

[0107] The activation layer can be, for example, one of a rectified linear unit (ReLU), leakyReLU, Sigmoid, Tanh, Softmax, or any other activation function. Activation functions can add nonlinearity to a neural network and can enable the neural network to solve computationally more complex tasks.

[0108] A convolutional layer with a 1×1 filter size can be used as a convolutional layer of a neural network.

[0109] Once the third set of features 13 and the fourth set of features 14 are extracted, the extracted third set of features 13, the fourth set of features 14 and the input feature set 10 are group convolved through a multi-head mechanism to obtain the second set of features 12 ( Figure 1B In step 140 in the group convolution, the input feature set 10, the third feature set 13 and the fourth feature set 14 are combined to obtain the second feature set 12, and the input feature set 10 uses a skip connection without processing; compared with the input feature set 10, the ability of the third feature set 13 to extract fields can be improved by processing through a residual block; compared with the third feature set 13, the fourth feature set 14 may be able to extract more complex features through an additional activation layer. That is, the three feature sets on which the group convolution is performed may contain features of different scales, so that the three feature sets can extract multi-scale global features. Therefore, the second feature set 12 may contain global features of the input of the long-short attention module.

[0110] Then, the upper branch 20 for extracting the first set of features 11 and the lower branch 21 for extracting the second set of features 12 may be fused into a hybrid feature map.

[0111] As described in detail above, in the long-short attention module according to an embodiment of the present invention, the first set of features 11 is extracted by at least two residual blocks of a neural network that are sequentially connected to each other to directly extract a set of local features. At the same time, in one embodiment of the present invention, the second set of features 12 that can represent global features can be extracted by processing through the above-mentioned non-local attention module and the multi-head attention mechanism through group convolution. The group convolution combines the input feature set 10, the third set of features 13, and the fourth set of features 14 to generate a second feature based on the input feature set 10.

[0112] The extracted second set of features 12 may be further processed by at least two convolutional layers of a neural network to obtain a fifth set of features 15. The processing may be able to reduce the dimensionality of the second set of features 12 to obtain the fifth set of features 15, such as Figure 1B As shown in the lower branch 21. Through this process, the size of the fifth group of features 15 can match the size of the first group of features 11.

[0113] Further, once the first set of features 11 and the fifth set of features 15 are obtained, and the dimensions of the first set of features 11 and the fifth set of features 15 match each other, the long-short attention module can perform cascade processing. The first set of features 11 and the fifth set of features 15 are then fused into a multidimensional vector, thereby representing a set of mixed features 17 suitable for input to the next processing step. For example, the next processing step may be a step in the encoding process, as described in further detail below.

[0114] In the above description, the process of obtaining the second set of features 12 and the corresponding fifth set of features 15 from the input feature set 10 is called long attention, because the second set of features 12 and the corresponding fifth set of features 15 contain global features. The process of obtaining the first set of features 11 from the input feature set 10 is called short attention, and the first set of features 11 only contains local features.

[0115] Further, the feature extraction method according to an embodiment of the present invention may include performing convolution processing on a set of mixed features 17 through at least one convolution layer of a neural network to obtain Figure 1B The sixth set of features 16 shown may represent the output of the long-short attention module, but this step is entirely optional.

[0116] Figure 2 A schematic diagram of performing group convolution by a multi-head attention mechanism in global feature extraction of a long-short attention module in an embodiment of the present invention is shown. The multi-head mechanism can be implemented by the following equation:

[0117]

[0118] Among them, f represents the input feature set 10, f ai and f bi are two sets of features extracted inside the long-short attention module. In the above embodiment, the two sets of features are the third set of features 13 and the fourth set of features 14. h represents the number of heads of the multi-head attention mechanism. For illustration purposes only, the number of heads of the multi-head attention mechanism is determined to be 3, but the number of heads of the multi-head attention mechanism is not limited to any specific number.

[0119] Specifically, for group convolution, the third group of features (f ai )13 and the fourth group of features (f bi )14. Afterwards, the input feature set 10 is added to the third set of features (f ai )13 and the fourth group of features (f bi )14 to obtain a set of result features. Then, the set of result features is connected in series according to the number of heads h of the multi-head attention mechanism to obtain a second set of features 12.

[0120] When the activation function applied by the activation layer is applied to the fourth set of features (f bi )14 is normalized so that the fourth set of features (f bi )14 has a value between 0 and 1 as described above, in the group convolution, due to the fourth group feature (f bi )14 has a value between 0 and 1, so the third set of features (f ai )13 and the fourth group of features (f bi )14 to extract the third set of features (f ai )13 important features.

[0121] As described above, the extraction of the third set of features 13 and the extraction of the fourth set of features include processing each of the extraction of the third set of features 13 and the extraction of the fourth set of features 14 through at least three residual blocks. In addition, as detailed above, the extraction of the first set of features 11 includes processing through at least two residual blocks of a neural network.

[0122] In an embodiment of the present invention, each residual block in the plurality of residual blocks or at least one residual block in the plurality of residual blocks may be defined as Figure 3 The residual block depicted. For example, the residual block may consist of at least two convolutional layers of a neural network and at least two activation layers applied after each of the at least two convolutional layers. In addition, there may be a skip connection that connects the input and output of the residual block, thereby passing the gradient directly to the deeper layers of the neural network without going through the activation function (activation layer).

[0123] The use of the residual block in the present invention can expand the corresponding domain of the neural network and improve its ability to extract features.

[0124] exist Figure 3 In the current embodiment of the present invention shown, the skip connection can be implemented by a convolutional layer of a neural network with a filter size of 1×1. By setting the filter size to 1×1, the skip connection can provide an alternative path to the deeper layers of the neural network.

[0125] like Figure 3 As depicted, on the lower branch of the residual block, the filter size of the convolutional layer may be larger than 1 × 1. For example, the filter size may be 3 × 3, and is able to learn the relationship between adjacent pixels of the target image.

[0126] According to the present invention, a method for learning image compression using a neural network is also provided. Figure 5A The neural network structure of the encoder 42 for executing the steps of the proposed learning image compression method according to an embodiment of the present invention is schematically shown.

[0127] According to the method of learning image compression using a neural network according to an embodiment of the present invention, a set of features x is extracted from input image data to be compressed, and a set of features y is extracted from the set of features x extracted from the input image data to be compressed, and the set of features y indicates a potential representation of the input image data to be compressed.

[0128] The extraction of the set of features y indicating the potential representation of the input image data is performed based on the following steps. The set of features x extracted from the input image data is downsampled through at least four convolutional layers of the neural network arranged in a streaming manner. For example, each time the downsampling process is performed at a corresponding convolutional layer in the at least four convolutional layers of the neural network, the size of the extracted set of features x can be reduced by half.

[0129] Downsampling in neural networks can be achieved by downsampling convolutional layers (3×3Conv,N, / 2), such as Figure 5A Depicted. Downsampling layers can be used to reduce the height and width of a feature map of a neural network. For example, the height and width of a feature map can be reduced by half through the downsampling process. Downsampling can be achieved by selecting a larger stride in the convolutional layer, configuring a pooling layer in the neural network, or by other means.

[0130] Then, after performing two-step downsampling convolution processing from at least four convolutional layers, the long-short attention module as described above is applied at least once by using a set of features based on the output of the second convolutional layer in the two convolutional layers as a feature input set in the long-short attention module, thereby extracting a set of features (y) indicating the potential representation of the input image data.

[0131] Figure 5A The encoder 42 depicted in the figure can process the input image data to be compressed through a downsampled convolution layer or a downsampled convolution block at each step of the downsampled convolution process. The downsampled convolution block can be composed of a downsampled convolution layer (3×3Conv,N, / 2) and at least one additional convolution layer (3×3Conv,N), which receives the output of the downsampled convolution layer as its input. These convolution layers may have a filter size of 3×3, but the present invention is not limited to a certain filter size. In addition, the downsampled convolution block may include a skip connection that directly connects the input and output of the downsampled convolution block through at least one convolution layer (i.e., Conv). The downsampled convolution block can then be followed by at least one residual block and another downsampled convolution block or downsampled convolution layer. The long-short attention module described above can then be applied to the output of the second downsampled convolution block or downsampled convolution layer to extract mixed features.

[0132] After applying the long-short attention module, a residual block, a downsampling convolution block, another residual block, and at least one convolution layer may be applied, followed by another long-short attention module, to extract a latent representation of the input image. The convolution layer applied before the second long-short attention module may be a downsampling convolution layer with a filter size of 3×3.

[0133] In other words, the present embodiment further includes inserting at least one step of processing by at least one residual block of a neural network between two steps of downsampling processing by two of the at least four downsampling convolutional layers arranged in a streaming manner. The term "streaming" should be understood in the sense that the output of the processing of one downsampling convolutional layer forms the basis of the input fed to the next downsampling convolutional layer. Figure 5A In the context of the present invention, the term "streaming" may also be understood as "downstream" from the input side of the encoder to the output side of the encoder. The term "forming a basis for" should be understood in the sense that additional processing steps may be inserted between the processing of multiple downsampling convolutional layers. For example, such processing may be the use of residual blocks. As detailed above, the use of residual blocks increases the receptive field and improves rate-distortion performance.

[0134] When a set of features indicative of a latent representation of the input image data is extracted, the method of learning image compression outputs the extracted set of features y.

[0135] According to the present invention, a method for learning image decompression using a neural network is also provided. Figure 5B The neural network structure of the decoder 47 for executing the steps of the proposed method for learning image decompression according to an embodiment of the present invention is schematically shown.

[0136] In this embodiment, the input feature set to be decompressed is A set of features is extracted, and a set of features indicating a reconstructed image of the input image data is extracted from the extracted set of features.

[0137] In order to extract a set of features of a reconstructed image of an input image indicating input image data, the feature extraction method detailed above by the long-short attention module is performed on a set of features extracted from the input feature set to be decompressed. Once the features are extracted by the long-short attention module, upsampling convolution is performed on the set of features extracted from the input feature set to be decompressed by at least four convolutional layers arranged in a streaming manner of a neural network. Then, after two steps of upsampling convolution processing by two convolutional layers of at least four convolutional layers, feature extraction of the long-short attention module is performed at least once using a set of features based on the output of the second convolutional layer of the two convolutional layers as a feature input set, thereby extracting a set of features of the reconstructed image indicating the input image data.

[0138] The term "streaming" should be understood in the sense that the output of one upsampling convolutional layer's processing forms the basis for the input fed to the next upsampling convolutional layer. Figure 5B In , this may be upstream from the input side of the decoder (right side of the figure) to the output side of the decoder (left side of the figure). The term "forming a basis for" is to be understood in the sense that further processing steps may be inserted between the processing of multiple upsampling convolutional layers. Such further processing may involve, for example, processing using the residual block detailed above.

[0139] When a set of features indicative of a reconstructed image of the input image data is extracted, the method of learning image decompression of the present embodiment then outputs the extracted set of features.

[0140] Upsampling in neural networks can be understood in a similar way to downsampling. Upsampling can be implemented by an upsampling convolution layer (3×3Conv,N,*2) or an upsampling convolution block, which includes an upsampling convolution layer (3×3Conv,N,*2) and at least one additional convolution layer (3×3Conv,N), such as Figure 5B Depicted. In addition, the upsampling convolution block may include a skip connection that directly connects the input and output of the upsampling convolution block through at least one convolutional layer (i.e., Conv). The upsampling convolution layer can be used to increase the height and width of the feature map of the neural network.

[0141] In one embodiment of the present invention, the method for learning image decompression also includes: inserting at least one step of processing through at least one residual block of a neural network between two steps of upsampling processing through two convolutional layers of at least four upsampling convolutional layers arranged in a streaming manner.

[0142] The residual blocks of the described encoder and decoder may have the same Figure 3 The residual block has the same structure as described.

[0143] Figure 5C A flow chart of an encoder for image data compression according to an embodiment of the present invention is shown. The encoder 42 can extract a potential representation of the input image data. The encoder 42 includes a first downsampling processing device 511 for downsampling processing. The first downsampling processing device 511 for downsampling processing receives an input feature set x obtained from the input image data and downsamples the feature set x (performs downsampling processing). By downsampling, the size of the input feature set x can be reduced. For example, the downsampling process can involve processing performed by a downsampling convolution block in the above-mentioned neural network model, a residual block connected in sequence, and a second downsampling convolution block connected to the output of the residual block.

[0144] The encoder 42 further includes a first filtering device 512, which is used to extract a set of mixed features from the output of the first downsampling processing device 511 for downsampling processing. The set of mixed features includes local features and global features of the image data. The first filtering device 512 can be referred to above. Figure 1B The encoder 42 further includes a second downsampling processing device 513 for downsampling processing, which receives the extracted set of mixed features as its input and performs downsampling (performs downsampling processing). The second downsampling processing device 513 for downsampling processing can be implemented by a first residual block, a downsampling convolution block, a second residual block and a downsampling convolution layer connected to each other in sequence.

[0145] The encoder 42 further comprises a second filtering means 514 for extracting a latent representation of the input image data from the output of the second downsampling processing means 513. The second filtering means 514 may be Figure 1B The long-short attention module of the embodiment of the present invention described in the embodiment of the present invention is implemented. The second filtering device 514 extracts the potential representation of the input image data because the encoder 42 of this embodiment of the present invention is designed to learn the important features of the input image data while reducing the dimensionality. Since the encoder 42 can be a trained model, the weights in each layer of the encoder 42 can be optimized to extract the potential representation of the input image data at the end of the encoder 42. In this regard, the second filtering device 514 can extract the potential representation of the input image data, while the first filtering device 512 can only extract mixed features.

[0146] Figure 5D FIG. 4 shows a flowchart of a decoder for decompressing image data according to an embodiment of the present invention. The decoder 47 includes a first filtering device 521. The first filtering device 521 extracts features from the input feature set to be decompressed. The first filtering device 521 can be obtained by referring to FIG. Figure 1B The long-short attention module described in the embodiment of the present invention is implemented. In the decoder, it may be advantageous to have as many features as possible. Therefore, the long-short attention module of the embodiment of the present invention can be used to extract as many features as possible in the decoder, thereby helping the decoder to reconstruct the image more accurately.

[0147] The decoder 47 further includes a first upsampling processing device 522. The first upsampling processing device 522 for upsampling processing receives the output of the first filtering device 521 and upsamples (performs upsampling processing) the output of the first filtering device 521. The first upsampling processing device 522 for upsampling processing can be composed of the first residual block, the first upsampled convolution residual block, the second residual block, and the second convolution residual block in the above-mentioned neural network model. Upsampling may increase the size of the input feature set to be decompressed.

[0148] The decoder 47 further comprises a second filtering means 523. The second filtering means 523 extracts features from the output of the first upsampling processing means 522 for upsampling processing. The second filtering means 521 may be Figure 1B The long-short attention module according to the embodiment of the present invention is described. The second filtering device 523 implemented by the long-short attention module according to the embodiment of the present invention can be used to help the decoder 47 reconstruct the image more accurately by extracting as many features as possible. In addition, the decoder 47 also includes a second upsampling processing device 524, which is used to perform upsampling processing on the output of the first filtering device 523 to obtain a set of features of the reconstructed image representing the input feature set. The reconstructed image corresponds to the decompressed image of the input feature set to be decompressed. The second upsampling processing device 524 for upsampling processing can be composed of a first residual block, an upsampling convolution block, a second residual block and an upsampling convolution layer connected to each other in sequence.

[0149] Figure 4A A schematic diagram of a framework for deep learning-based image compression and decompression according to an embodiment of the present invention is shown. Figure 4B The corresponding architectures of neural networks with exemplary types, numbers and arrangements of convolutional layers can be seen in .

[0150] According to the present embodiment, a method for learning image processing using a neural network is provided, the method comprising: performing image compression by an encoder 42; providing a set of extracted features y output from the encoder 42, the set of features y indicating a potential representation of input image data in a first processing path and a second processing path; and generating a reconstructed image by a decoder 47 using the outputs of the first processing path and the second processing path as input feature sets to be decompressed.

[0151] In the first processing path, the super encoder 43 obtains modeling information from a set of features y provided to indicate a potential representation. The modeling information is then quantized by a quantizer Q. The output is fed into an arithmetic encoder (AE) and a bitstream is obtained which is fed into an arithmetic decoder (AE). The quantized modeling information is represented The output of the arithmetic decoder (AE) is fed to the super decoder 44, and the super decoder 44 is fed from the quantized modeling information Get the decoding information Ψ.

[0152] In the second processing path, the context model 45 is derived from the quantized latent representation The auxiliary information Φ is obtained from the quantized potential representation Indicates the potential representation y quantized by the quantizer Q. Then, the modeling parameters N(μ,θ) (entropy parameters) are calculated by the entropy model 46 by combining the decoded information Ψ and the auxiliary information Φ. The output of the quantizer Q is also fed into the arithmetic encoder (AE), which generates a bit stream fed into the arithmetic decoder (AD). The modeling parameters N(μ,θ) are fed into the arithmetic decoder (AD), and the output of the arithmetic decoder (AD) is calculated based on the modeling parameters N(μ,θ) The modeling parameters N(μ,θ) are also fed to the arithmetic encoder (AE).

[0153] A super neural network consisting of a super encoder and a super decoder (e.g. Figure 4A and Figure 4B The right side of ) can improve the overall performance of the larger neural network by taking the weights of the larger main neural network or by acting as a regularizer on the larger neural network.

[0154] In image compression, a trade-off may need to be made between the bit rate size and the quality of the reconstructed image. The proposed network can be optimized using a rate-distortion loss function and a mean squared error. An example of a loss function can be defined as follows.

[0155] L=λ·255 2 ·D MSE +R

[0156]

[0157] The parameter λ can be selected from five options {0.0035, 0.0067, 0.015, 0.025, 0.0483}, and Figure 4A The corresponding N in is {128,128,128,196,196}.

[0158] Figure 6A schematic diagram of the file format of a bitstream obtained by an image compression method according to an embodiment of the present invention is shown. The first byte is assigned to a method index. For example, based on CompressAI, the method index may be 6 (see non-patent reference [8]). According to the method index, the number and quality number of the optimization method may be assigned in the bitstream file format. The quality numbers {1, 2, 3, 4, 5} may correspond to the five options {0.0035, 0.0067, 0.015, 0.025, 0.0483}, respectively. Then, the following items may be assigned in the bitstream file save format: the height H of the original image, the width W of the original image, the height z (h) of the Gaussian model, the width z (w) of the Gaussian model, the total length Len of the bitstream, the length Len-y of the image potential representation y encoded in the bitstream, the bitstream of the image potential representation y (\x0b…\x91), the length (Len-z) of the bitstream encoded with the Gaussian modeling information z, and the bitstream of the Gaussian modeling information z (\x06…\xf8).

[0159] Figure 7 A schematic diagram of a general device embodiment for applying any of the methods of the present invention described above is shown. Device 70 may include processing resources 71, memory access 72, and communication interface 73. The memory access 72 mentioned may store code, or may access code indicating that processing resources 71 perform one or more steps of any method embodiment of the present invention described and explained as described above and in conjunction with the present disclosure. Communication interface 73 may be suitable for receiving communication data via a network. The network may be a wired network or a wireless network. Device 70 may typically be a computer, a personal computer, a tablet computer, a laptop computer, a smart phone, a mobile phone, a video player, a TV set-top box, a receiver, etc., as they are known in the art.

[0160] The processing resources 71 may be implemented by one or more processing units (eg, a central processing unit (CPU)), or may be provided by distributed and / or shared processing capabilities (eg, present in a data center or in the form of so-called cloud computing).

[0161] The memory access 72 that can be implemented by the local memory may include, but is not limited to, a hard disk drive (HDD), a solid state drive (SSD), a random access memory (RAM), and a flash memory. Similarly, distributed and / or shared memory storage (memory storage) may also be applied, such as data center and / or cloud memory storage.

[0162] The communication interface 73 may be adapted to receive data conveying the multi-view image data 10 and to transmit communication data conveying the encoded at least one panoramic feature map and the plurality of encoded view blocks over a communication network. The communication network may be a wired or wireless mobile network.

[0163] The above-mentioned embodiments of the present invention achieve the beneficial effects of the technical problems to be solved. The technical effects of the embodiments can be reflected in several different tests. For example, different QP tests are performed on traditional methods such as JPEG2000 (non-patent reference [2]) and VVC (VTM15.0) (non-patent reference [4]). In addition, the technical effects can also be reflected in the testing of several deep learning models provided by CompressAI (https: / / github.com / InterDigitalInc / CompressAI, non-patent reference [8]), which includes bmshj (non-patent reference [6]), mbt2018 (non-patent reference [9]) and Cheng2020 (without GMM) (non-patent reference [7]). Therefore, in the following description, the abbreviations "JPEG2000", "VVC", "CompressAI", "bmshj" (or "bmshj2018"), "mbt2018" and "Cheng2020" or "Cheng" or Cheng(woGMM) are used to refer to the non-patent references [2], [4], [8], [6], [9] and [7] and the methods or codecs and models discussed therein, and the subsequent JPEG2000 codec, VVC codec, bmshj2018 model, mbt2018 model and chang2020 model, respectively.

[0164] The performance of the model can be optimized by the mean square error under different λ. In addition, the performance can be compared using visual quality, Peak to Signal Noise Ratio (PSNR), and Multi-Scale Structural Similarity (MS-SSIM) to verify the effectiveness of the long and short attention modules in saving bit rate.

[0165] To train the neural network, the hardware used is TITAN X 12G, the software used is Ubuntu16.04.6, PyTorch 1.7.0, and the data sets include the training data set JPEG-AIDataset (5283) and the test data set JPEG-AIDataset (40). The evaluation metrics used are PSNR: the higher the metric value, the better the image quality; MS-SSIM: the higher the metric value, the better the image quality; Video Multi-Method Assessment Fusion (VMAF): the higher the metric score, the better the image quality; Visual Information Fidelity (VIF): the higher the score, the better the image quality; NLP: the lower the score, the better the image quality; Feature Similarity Index (FSIM): the higher the metric value, the better the image quality. The parameter settings are as follows: Batch size: 8; Learning rate: 1e-4; Epochs: 300.

[0166] Figure 8 More details of the test process are shown. For example, the quality score can be obtained using the original RGB image and the decoded RGB image, which is encoded by the learning-based image encoder 42 and then decoded by the learning-based image decoder 47. Specifically, the RGB image can be converted to YUV 4:4:4, which indicates full color depth. The coding efficiency of the YUV color space may be higher than that of the RGB color model. Then, by adopting multiple indicators that can indicate similarity, such as MS-SSIM, IS-SSIM, VIF, PSNR-HVS-M, Normalized Laplacian Pyramid Distance (NLPD), FSIM, FSIMc and VMAF, the reconstruction quality can be scored and compared accordingly in the quality score.

[0167] Fig. 9 A visual comparison between different methods in 00001-TE-960x720 of the JPEG AI dataset is shown. The figure shows a lot of details to reflect the visual compression effect of these different methods. Bits per pixel (bpp), PSNR metric and MS-SSIM metric are also provided for the decoded results.

[0168] In the test, JPEG2000 has the lowest PSNR and MS-SSIM values, indicating that its compression effect is the worst. The proposed method is better than bmshj and mbt2018, but slightly worse than Cheng's results. In terms of evaluation metrics, VVC has close compression performance to Cheng.

[0169] VVC (VTM15.0), JPEG2000 and bmshj lead to large detail loss, such as blurred edges of wrinkles in the red frame (see details of white dots). mbt2018 and Cheng improved the reconstruction of the edge of the wedding dress (white dots were also reconstructed). And visually, Cheng's reconstruction result is better than mbt2018. The method of learning image processing of the present invention restores image details significantly better than other methods. The effectiveness of the LSA module in image compression is verified by visual comparison.

[0170] Fig.11 A table showing the average bitrate savings for five λs compared to the VVC (VTM 15.0) anchor point is shown. For further comparison, the average bitrate savings for bpp in [0.06, 0.50] when using VVC (VTM15.0) can be calculated. Among them, JPEG2000, as an earlier traditional method, has an average PSNR bitrate improvement of 26.91% and an average MS-SSIM bitrate improvement of 162.82%. Its compression effect is better than bmshj in PSNR, while it performs worst on MS-SSIM. The average bitrate savings of bmshj exceeds 20%, and its performance is worse than VVC. mbt2018 saves about 10% of the bitrate compared to VVC compression, and Cheng saves about 12% of the bitrate compared to VVC compression. The method proposed in an embodiment of the present invention (in Fig.11 The proposed method (labeled as “ours” in Figure 2) achieves the largest bitrate savings of 16.25% for PSNR and 20.59% for MS-SSIM, while achieving the best compression efficiency on both metrics. Thus, the results demonstrate the technical effectiveness of the long-short attention module in image compression.

[0171] As another comparison method, rate-distortion curves can be used. Rate-distortion curves can reflect codec performance. Fig. 10A and Fig. 10B The figure shows a rate-distortion curve for comparing the codec performance between different methods. The legend term 'ours' refers to the method proposed in the embodiment of the present invention. The horizontal axis represents the number of bits per pixel, and the vertical axis represents the evaluation metric.

[0172] Fig. 10AThe rate-distortion curves in clearly compare the PSNR performance of different methods at different bpp. At lower bit rates, the proposed method achieves the highest PSNR value, indicating the best compression efficiency. At higher bit rates, the proposed method achieves comparable performance to the methods of mbt2018 and Cheng (without GMM).

[0173] Fig. 10B The compression performance of various methods is compared in terms of MS-SSIM metric. The MS-SSIM is converted into decibels to show the performance. It can be observed that the proposed method achieves the best performance in all bpp. Fig. 10A and Fig. 10B The two rate-distortion curves shown verify the effectiveness of the method proposed in the present invention.

[0174] In summary, the present invention proposes a long-short attention module for feature extraction. Feature extraction can be used for learning image processing. As described in detail above, the long-short attention module has two branches. For learning image processing, one branch extracts local features, while the other branch extracts global features. The global features and local features are fused to obtain accurate potential features in the image. In the global feature branch, the proposed neural network architecture extracts multi-scale global features through a multi-head attention mechanism. The present invention uses group convolution between multiple features to achieve multi-scale global feature extraction to enhance the ability of the long-short attention module to extract global features.

[0175] In addition, the present invention proposes a learning image compression neural network architecture that combines long and short modules into the autoencoder so that the neural network can more accurately obtain the potential features of the image. Experiments show that at low bit rates in PSNR, the proposed network outperforms the state-of-the-art methods. At high bit rates, the proposed network achieves performance comparable to other networks. In the MS-SSIM metric, the proposed method achieves the best performance (nested texture recovery), which shows that the LSA module is good at reconstructing the structural information of the image. In other words, the experiments verify that accurate potential features of the image can improve coding efficiency.

[0176] Although detailed embodiments have been described, these embodiments are only intended to provide a better understanding of the invention as defined by the independent claims and are not to be considered as limiting the invention.

Claims

1. A method for feature extraction using a neural network, the method comprising: extracting a first set of features from the input feature set by processing the input feature set through at least two residual blocks of the neural network connected sequentially to each other; Extracting a second set of features from the input feature set, wherein extracting the second set of features comprises the following steps: Extracting a third set of features and a fourth set of features from the input feature set by non-local attention processing, Performing group convolution on the input feature set, the third group of features, and the fourth group of features through a multi-head mechanism to obtain the second group of features.

2. The method according to claim 1, wherein: The input feature set is a multi-dimensional vector representing image data.

3. The method according to claim 2, wherein: The first set of features is a set of local features of the image data, and the second set of features is a set of global features of the image data.

4. The method according to any one of claims 1 to 3, further comprising the following steps: A fifth set of features is obtained by processing the second set of features through at least two convolutional layers of the neural network.

5. The method according to any one of claims 1 to 4, further comprising the following steps: performing a cascade process on the first set of features and the fifth set of features, and The first group of features and the fifth group of features after cascading processing are fused to obtain a group of mixed features.

6. The method according to claim 5, further comprising: performing convolution processing on the set of mixed features by at least one convolutional layer of the neural network to obtain a sixth set of features, and The sixth set of features is output.

7. The method according to any one of claims 1 to 6, wherein: Implementing the group convolution includes the following steps: multiplying the third set of features by the fourth set of features, adding the input feature set to the product of the third set of features and the fourth set of features to obtain a set of result features, and The set of resulting features are concatenated by a plurality of heads in the multi-head attention mechanism to obtain the second set of features.

8. The method according to any one of claims 1 to 7, wherein: Extracting the third set of features includes: The input feature set is processed via at least three residual blocks of the neural network that are sequentially connected to each other.

9. The method according to any one of claims 1 to 8, wherein: Extracting the fourth set of features includes: Processing the input feature set by at least three residual blocks of the neural network connected sequentially to each other; processing an output of a last residual block of the at least three residual blocks by a convolutional layer of the neural network; and The activation layer of the neural network performs activation processing on the output of at least one convolutional layer; 10. A method for learning image compression using a neural network, the method comprising performing the following steps: extracting a set of features x from the input image data to be compressed; and A set of features y is extracted from the set of features x extracted from the input image data to be compressed, the set of features y indicating a potential representation of the input image data to be compressed, by performing the following steps: performing at least four steps of downsampling convolution on the set of features x extracted from the input image data by at least four convolutional layers of the neural network arranged in a streaming manner, and After performing a two-step downsampling convolution process through two of the at least four convolutional layers, performing the steps of the feature extraction method according to claim 1 at least once by using a set of features based on the output of a second convolutional layer of the two convolutional layers as the input feature set, thereby extracting the set of features y indicating the potential representation of the input image data; and The method further comprises the following steps: The extracted set of features y indicating a latent representation of the input image data is output.

11. The method according to claim 10, further comprising: At least a step of processing by at least one residual block of the neural network is inserted between two downsampling processes performed by two convolutional layers of the at least four downsampling convolutional layers arranged in a streaming manner.

12. A method for learning image decompression using a neural network, the method comprising: Extracting a set of features from the input feature set to be decompressed; as well as A set of features indicative of a reconstructed image of the input image data is extracted from the extracted set of features by performing the following steps: Using the set of features extracted from the input feature set to be decompressed as the input feature set to perform the steps of the feature extraction method according to claim 1, performing upsampling convolution processing on the set of features extracted from the input feature set to be decompressed by at least four convolutional layers of the neural network arranged in a streaming manner, and After two convolution layers of the at least four convolution layers perform a two-step upsampling convolution process, performing at least one step of the method according to claim 1 by using a set of features based on the output of a second convolution layer of the two convolution layers as the input feature set. The feature extraction method, thereby extracting a set of features indicative of a reconstructed image of the input image data; and The method further comprises: The extracted set of features indicative of a reconstructed image of the input image data is output.

13. The method according to claim 12, wherein: The method further comprises: Between two upsampling processes performed by two convolutional layers of the at least four upsampling convolutional layers arranged in a streaming manner, at least one step of processing by at least one residual block of the neural network is inserted.

14. The method according to any one of claims 1 to 13, wherein: The steps processed by the residual block include: performing convolution by at least two convolutional layers of the neural network; performing activation by at least one activation layer of the neural network after each of the at least two convolutional layers of the neural network; implementing a skip connection that provides an alternative path from an input of the residual block to an output of the residual block; and The output of the residual block is obtained by fusing the skip connection and the last activation layer of the residual block.

15. A method for learning image processing using a neural network, the method comprising: Execute the steps of the method according to claim 9; The extracted set of features y indicative of a latent representation of the input image data is provided as an output in a first processing path and a second processing path, wherein: In the first processing path the following steps are performed: The hyperencoder obtains modeling information from a set of features provided to indicate the latent representation. quantizing the modeling information by a quantizer, and Obtaining decoding information from the quantized modeling information by a super decoder, The following steps are performed in the second processing path: Obtaining auxiliary information from a quantized latent representation by a context model, the quantized latent representation indicating a latent representation quantized by a quantizer, The entropy model calculates modeling parameters by combining the decoded information and the auxiliary information; and The output of the entropy decoder is calculated according to the modeling parameters; and the method further comprises the following steps: The output of the entropy decoder is used as an input feature set to be decompressed, and a reconstructed image is generated by the image decompression method defined in claim 11.

16. A data processing apparatus for feature extraction, comprising a processing resource and access to a memory resource, the access to the memory resource being used to obtain code instructing the processing resource to perform the method according to claim 1 during operation.

17. A computer program comprising instructions, which, when a computer executes the program, cause the computer to execute the method according to claim 1.

18. A computer-readable storage medium comprising instructions, which, when executed by a computer, cause the computer to perform the method according to claim 1.

19. An encoding apparatus for learning image compression, comprising a processing resource and access to a memory resource, the access to the memory resource being used to obtain code instructing the processing resource to perform the method according to claim 9 during operation.

20. A computer program comprising instructions, which, when a computer executes the program, cause the computer to perform the method according to claim 10.

21. A computer-readable storage medium comprising instructions, which, when executed by a computer, cause the computer to perform the method according to claim 10.

22. A decoding apparatus for learning image decompression, comprising a processing resource and access to a memory resource, the access to the memory resource being used to obtain code instructing the processing resource to perform the method of claim 12 during operation.

23. A computer program comprising instructions, which, when a computer executes the program, cause the computer to perform the method according to claim 12.

24. A computer-readable storage medium comprising instructions, which, when executed by a computer, cause the computer to perform the method according to claim 12.

25. A system comprising one or more processing resources and access to one or more memory resources, the access to the one or more memory resources being used to obtain one or more codes that instruct the one or more processing resources to perform the method of claim 15 during operation.

26. An encoder for image data compression, the encoder comprising: a first down-sampling processing device, the first down-sampling processing device being used to perform down-sampling processing on an input feature set obtained from input image data; a first filtering device, the first filtering device being used to extract a set of mixed features from the output of the first downsampling processing device, the mixed features comprising local features and global features; A second downsampling processing device, the second downsampling processing device is used to perform downsampling processing on the extracted set of mixed features; as well as Second filter means for extracting a latent representation of said input image data from an output of said second downsampling processing means for downsampling.

27. A decoder for decompressing image data, the decoder comprising: a first filtering device, the first filtering device being used to extract features from an input feature set to be decompressed; a first upsampling processing device, the first upsampling processing device being used to perform upsampling processing on a set of features outputted from the pre-filter of the first filtering device; a second filtering device, the second filtering device being used to extract features from a set of features outputted from the first upsampling processing device; as well as The second upsampling processing device is used to perform upsampling processing on a set of features output from the second filtering device to obtain a set of features representing a reconstructed image of the input feature set.

Images