Image decoding and encoding method, apparatus, device, and storage medium

Abstract

The application belongs to the technical field of image processing, and discloses an image decoding and encoding method, device, equipment and storage medium. The application extracts image residual data or extended residual data packets from an image code stream, obtains a plurality of extended residual groups based on the extracted image residual data or extended residual data, respectively performs residual recovery on the plurality of extended residual groups to obtain image reconstruction features corresponding to each extended residual group, performs magnification spatial resolution processing on the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data, and performs image reconstruction according to the reconstruction feature data to obtain a reconstruction image block. Since the obtained extended residual data is residual data subjected to reduction spatial resolution processing, residual recovery processing can be performed on the whole group in a low resolution, thereby improving the overall residual recovery calculation efficiency and reducing the time complexity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image decoding and encoding method, device, equipment and storage medium. BACKGROUND

[0002] In a deep learning-based image compression scheme, using already decoded feature points as prior information to perform mean prediction on the feature points being decoded to reduce the spatial redundancy of the image is the current mainstream approach, and the mainstream scheme generally uses serial or wave-front coding and decoding, and the serial degree increases with the increase of the resolution of the features, and the overall time complexity of execution is high.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide an image decoding and encoding method, device, equipment and storage medium, aiming at solving the technical problem of high time complexity in the mean prediction process of the image coding and decoding of the prior art.

[0005] To achieve the above-mentioned purpose, the present application provides an image decoding method, which comprises the following steps:

[0006] extracting image residual data or extended residual data from an image code stream, and obtaining a plurality of extended residual groups based on the extracted image residual data or extended residual data;

[0007] performing residual recovery on the plurality of extended residual groups respectively to obtain image reconstruction features corresponding to each extended residual group;

[0008] performing upscaling spatial resolution processing on the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data;

[0009] performing image reconstruction according to the reconstruction feature data to obtain a reconstruction image block.

[0010] In a possible implementation of the present application, the step of extracting image residual data or extended residual data from an image code stream and obtaining a plurality of extended residual groups based on the extracted image residual data or extended residual data comprises:

[0011] extracting the image residual data from the image code stream;

[0012] performing downscaling spatial resolution processing on the image residual data to obtain the extended residual data, and the upscaling spatial resolution processing is the inverse process of the downscaling spatial resolution processing;

[0013] grouping the extended residual data to obtain a plurality of extended residual groups.

[0014] In a possible implementation of the present application, the method further comprises:

[0015] reducing the spatial size corresponding to the image residual data, and / or increasing the number of feature channels corresponding to the image residual data, to obtain the extended residual data.

[0016] In a possible implementation of the present application, the method further comprises:

[0017] reducing the spatial size corresponding to the image residual data according to the spatial information corresponding to the image residual data, and / or increasing the number of feature channels corresponding to the image residual data, to obtain the extended residual data.

[0018] In a possible implementation of the present application, the method further comprises:

[0019] constructing a residual recovery sequence according to the plurality of extended residual groups;

[0020] recovering the plurality of extended residual groups according to the residual recovery sequence to obtain image reconstruction features corresponding to each extended residual group.

[0021] In a possible implementation of the present application, the method further comprises:

[0022] traversing the residual recovery sequence to obtain a current extended residual group;

[0023] obtaining auxiliary information output by an auxiliary encoding network;

[0024] constructing prior information according to the auxiliary information;

[0025] recovering the current extended residual group according to the prior information to obtain image reconstruction features corresponding to the current extended residual group;

[0026] obtaining image reconstruction features corresponding to each extended residual group at the end of the traversal.

[0027] In a possible implementation of the present application, the constructing the prior information according to the auxiliary information comprises:

[0028] obtaining extended auxiliary information;

[0029] detecting whether the current extended residual group is the first element in the residual recovery sequence;

[0030] if the current extended residual group is the first element, constructing the prior information according to the extended auxiliary information;

[0031] if the current extended residual group is not the first element, splicing the extended auxiliary information and a convolution processing result corresponding to the image reconstruction feature of the recovered extended residual group to obtain spliced auxiliary information, and constructing the prior information according to the spliced auxiliary information.

[0032] In addition, to achieve the above object, the present application further provides an image coding method, which comprises:

[0033] performing down-spatial resolution processing on image features corresponding to a to-be-coded image to obtain extended image features;

[0034] grouping the extended image features to obtain a plurality of extended feature groups;

[0035] performing residual calculation on the plurality of extended feature groups respectively to obtain image residual data corresponding to each extended feature group;

[0036] generating an image code stream according to the image residual data, and sending the image code stream to an image decoding end.

[0037] In a possible implementation of the present application, the performing down-spatial resolution processing on image features corresponding to a to-be-coded image to obtain extended image features comprises:

[0038] obtaining image features corresponding to a to-be-coded image;

[0039] performing down-spatial resolution processing on data in the image features to obtain extended image features.

[0040] In a possible implementation of the present application, the performing down-spatial resolution processing on image features corresponding to a to-be-coded image to obtain extended image features comprises:

[0041] reducing spatial dimensions of image features of a to-be-coded image, and / or increasing a number of feature channels corresponding to the image features to obtain extended image features.

[0042] In a possible implementation of the present application, the method further includes: reducing a spatial size corresponding to an image feature of the image to be encoded, and / or increasing a feature channel number corresponding to the image feature, to obtain an extended image feature.

[0043] The spatial size corresponding to the image feature is reduced and / or the feature channel number corresponding to the image feature is increased according to space domain information corresponding to the image feature of the image to be encoded, to obtain an extended feature group.

[0044] In a possible implementation of the present application, the method further includes: performing residual calculation on the plurality of extended feature groups respectively based on the residual calculation sequence, to obtain image residual data corresponding to each extended feature group.

[0045] The residual calculation sequence is constructed according to the plurality of extended feature groups.

[0046] The residual calculation on the plurality of extended feature groups is performed respectively based on the residual calculation sequence, to obtain image residual data corresponding to each extended feature group.

[0047] In a possible implementation of the present application, the method further includes: performing residual calculation on the plurality of extended feature groups respectively based on the residual calculation sequence, to obtain image residual data corresponding to each extended feature group.

[0048] The residual calculation sequence is traversed to obtain a current extended feature group.

[0049] Auxiliary information output by an auxiliary encoding network is obtained.

[0050] Prior information is constructed according to the auxiliary information.

[0051] Residual calculation is performed on the current extended feature group based on the prior information, to obtain image residual data corresponding to the current extended feature group.

[0052] At the end of the traversal, image residual data corresponding to each extended feature group is obtained.

[0053] In a possible implementation of the present application, the method further includes: grouping the extended image feature to obtain a plurality of extended feature groups.

[0054] The extended image feature is grouped based on a feature channel corresponding to the extended feature data, to obtain a plurality of extended feature groups.

[0055] In a possible implementation of the present application, the method further includes: generating an image code stream according to the image residual data, and sending the image code stream to an image decoding end.

[0056] The image residual data corresponding to the to-be-encoded image is generated according to the image residual data corresponding to the to-be-encoded image, and the image code stream is sent to an image decoding end.

[0057] The image code stream is generated according to the image residual data, and the image code stream is sent to an image decoding end.

[0058] In addition, to achieve the above object, the present application further provides an image decoding device, which comprises:

[0059] A code stream decoding module is configured to extract image residual data or extended residual data from an image code stream, and obtain a plurality of extended residual groups based on the extracted image residual data or extended residual data.

[0060] A residual recovery module is configured to perform residual recovery on the plurality of extended residual groups respectively, and obtain image reconstruction features corresponding to each extended residual group.

[0061] A data combination module is configured to perform enlarged spatial resolution processing on the image reconstruction features corresponding to each extended residual group, and obtain reconstruction feature data.

[0062] An image reconstruction module is configured to perform image reconstruction according to the reconstruction feature data, and obtain a reconstructed image block.

[0063] In addition, to achieve the above object, the present application further provides an image encoding device, which comprises:

[0064] A feature extraction module is configured to perform reduced spatial resolution processing on image features corresponding to a to-be-encoded image, and obtain extended image features.

[0065] A data grouping module is configured to group the extended image features, and obtain a plurality of extended feature groups.

[0066] A residual calculation module is configured to perform residual calculation on the plurality of extended feature groups respectively, and obtain image residual data corresponding to each extended feature group.

[0067] A code stream generation module is configured to generate an image code stream according to the image residual data, and send the image code stream to an image decoding end.

[0068] In addition, to achieve the above object, the present application further provides a decoding device, which comprises a processor, a memory, and an image decoding program stored on the memory and executable on the processor, and the image decoding program is executed by the processor to implement the steps of the image decoding method.

[0069] Further, in order to achieve the above object, the present application also provides an encoding device, comprising a processor, a memory, an image decoding program and / or an image encoding program stored in the memory and executable on the processor, wherein the image decoding program is executed by the processor to implement the steps of the image decoding method as described above, and the image encoding program is executed by the processor to implement the steps of the image encoding method as described above.

[0070] Further, in order to achieve the above object, the present application also provides a storage medium, wherein the storage medium stores an image decoding program and / or an image encoding program, wherein the image decoding program is executed to implement the steps of the image decoding method as described above, and the image encoding program is executed to implement the steps of the image encoding method as described above.

[0071] Further, in order to achieve the above object, the present application also provides a computer program, which is configured to implement the steps of the image decoding method as described above or the steps of the image encoding method as described above when executed by a processor having a memory.

[0072] Further, in order to achieve the above object, the present application also provides a computer program product comprising computer program instructions, which are configured to implement the steps of the image decoding method as described above or the steps of the image encoding method as described above when executed by a processor having a memory.

[0073] The present application extracts image residual data or extended residual data from an image code stream, obtains a plurality of extended residual groups based on the extracted image residual data or extended residual data, respectively performs residual recovery on the plurality of extended residual groups to obtain image reconstruction features corresponding to each extended residual group, performs upsampling spatial resolution processing on the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data, and performs image reconstruction according to the reconstruction feature data to obtain a reconstructed image block. Since the obtained extended residual data is residual data subjected to downsampling spatial resolution processing, the residual recovery processing can be performed on each group in a grouped manner at a low resolution, thereby improving the overall residual recovery calculation efficiency and reducing the time complexity. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 is a structural schematic diagram of an electronic device of a hardware running environment related to an embodiment scheme of the present application;

[0075] Figure 2 is a flowchart of a first embodiment of the image decoding method of the present application;

[0076] Figure 3 is an overall framework diagram of an image compression of an embodiment of the present application;

[0077] Figure 4 Flowchart of the second embodiment of the image decoding method of the present application;

[0078] Figure 5 Flowchart of the spatial resolution processing of an embodiment of the present application

[0079] Figure 6 Flowchart of the third embodiment of the image decoding method of the present application;

[0080] Figure 7 Flowchart of the image decoding packet execution of an embodiment of the present application;

[0081] Figure 8 Flowchart of the secondary packet execution of an embodiment of the present application;

[0082] Figure 9 Flowchart of the feature enhancement packet execution of an embodiment of the present application;

[0083] Figure 10 Flowchart of the first embodiment of the image encoding method of the present application;

[0084] Figure 11 Flowchart of the second embodiment of the image encoding method of the present application;

[0085] Figure 12 Block diagram of the first embodiment of the image decoding device of the present application;

[0086] Figure 13 Block diagram of the first embodiment of the image encoding device of the present application.

[0087] The implementation, functional features and advantages of the present application will be further explained with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0088] It should be understood that the specific embodiments described herein merely exemplify the application and are not intended to limit the application.

[0089] Reference Figure 1 , Figure 1 Block diagram of the decoding device or encoding device structure of the hardware running environment involved in the embodiment scheme of the present application.

[0090] As Figure 1As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0091] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0092] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and an image decoding program and / or an image encoding program.

[0093] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the decoding device or the encoding device. The electronic device calls the image decoding program or image encoding program stored in the memory 1005 through the processor 1001 and executes the image decoding method or image encoding method provided in the embodiments of the present invention.

[0094] This invention provides an image decoding method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of an image decoding method according to the present invention.

[0095] In this embodiment, the image decoding method includes the following steps:

[0096] Step S10: extracting image residual data or extended residual data from the image code stream, and obtaining a plurality of extended residual groups based on the extracted image residual data or extended residual data.

[0097] It should be noted that the execution subject of the present embodiment can be a decoding device for decoding image data, which can be a personal computer, a server or other electronic device, and of course, can also be other devices that can achieve the same or similar functions. The present embodiment does not limit this, and the image decoding method of the present application is described below with the decoding device as an example.

[0098] In the image encoding process, the encoding device generally decodes the encoded image code stream after encoding is completed, and determines whether the parameters used in the encoding need to be adjusted according to the image quality of the decoded image. Therefore, the execution subject of the present embodiment can also be the encoding device.

[0099] It should be noted that the image code stream can be a code stream generated by the encoding device after encoding the image data to be compressed and encoded. When generating the image code stream, the encoding device reduces the spatial resolution of the image features to reduce the time complexity of the prediction mean in the encoding process. Finally, the encoding device directly encodes the generated image residual data or extended residual data into the image code stream. At this time, the decoding device can directly extract the image residual data or extended residual data from the image code stream. At this time, the decoding device can process the extracted image residual data or extended residual data to obtain a plurality of extended residual groups, and then perform residual recovery one group at a time, thereby reducing the time complexity of the mean prediction in the image decoding process.

[0100] The technical terms involved in the encoding or decoding of images include: JPEG (Joint Photographic Experts Group), JPEG-AI (Joint Photographic Experts Group Artificial Intelligence), entropy encoding (Entropy Encoding), neural network (Neural Network, NN), convolutional neural network (Convolutional Neural Network, CNN), feature (feature), rate-distortion principle (Rate-Distortion Optimized), etc., which are described below.

[0101] Wherein, JPEG (Joint Photographic Experts Group) is a standard for continuous tone static image compression, and the file suffix is.jpg or.jpeg. It is the most commonly used image file format. Its main joint coding mode is to use prediction coding (for example, Differential Pulse Code Modulation, DPCM), Discrete Cosine Transform (DCT) and entropy coding to remove redundant image and color data. It belongs to lossy compression format, which can compress the image into a small storage space, and will cause damage to the image data to a certain extent. Especially using too high compression ratio will reduce the quality of the image recovered after final decompression. If high-quality images are pursued, it is not appropriate to use too high compression ratio.

[0102] The scope of JPEG-AI is to create a learning-based image coding standard that provides a single-stream, compact, compressed-domain representation, outperforms the commonly used image coding standards in terms of compression efficiency and effective performance for image processing and computer vision tasks for human visualization. JPEG-AI is aimed at a wide range of applications, such as cloud storage, visual monitoring, self-driving cars and devices, image acquisition, storage and management, real-time monitoring of visual data and media distribution. The goal is to design an encoding solution that significantly improves the compression efficiency of commonly used coding standards with the same subjective quality, and provides effective compressed-domain processing for machine learning-based image processing and computer vision tasks. Other key requirements include hardware / software implementation-friendly encoding and decoding, support for 8-bit and 10-bit depth, efficient encoding of images using text and graphics, and progressive decoding.

[0103] Entropy coding is a coding process that does not lose any information according to the entropy principle. Information entropy is the average amount of information of the source (a measure of uncertainty). Common entropy coding includes Shannon coding, Huffman coding and arithmetic coding.

[0104] The neural network in the present application refers to an artificial neural network, rather than a biological neural network. The neural network is an operation model composed of a large number of nodes (or neurons) connected with each other. In the artificial neural network, the neuron processing unit can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in the network are divided into three categories: input units, output units and hidden units. The input units accept signals and data from the external world; the output units output the processing results of the system; the hidden units are the units between the input and output units, which cannot be observed from the outside of the system. The connection weights between neurons reflect the connection strength between units, and the information representation and processing are embodied in the connection relationship of the network processing units. The artificial neural network is a non-programmed, brain-like information processing method, which essentially obtains a parallel distributed information processing function through the transformation and dynamics of the network, and simulates the information processing function of the human brain neural system at different levels. At present, in the field of video processing, commonly used neural networks include convolutional neural network (CNN), recurrent neural network (RNN), fully connected network, etc.

[0105] The convolutional neural network is a kind of feedforward neural network, which is one of the most representative network structures in deep learning technology. The artificial neurons of the convolutional neural network can respond to a part of the surrounding units in the coverage range, and have excellent performance for large image processing. Generally, the basic structure of CNN includes two layers, one of which is the feature extraction layer (also called the convolution layer), and the input of each neuron is connected with the local receptive field of the previous layer, and the local features are extracted. Once the local features are extracted, the positional relationship between them and other features is also determined; the other is the feature mapping layer (also called the activation layer), each calculation layer of the network is composed of multiple feature mappings, and each feature mapping is a plane, and all the neurons on the plane have equal weights. The feature mapping structure can use Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, GDN function, etc. as the activation function of the convolutional network. In addition, since the neurons on a mapping plane share weights, the number of free parameters of the network is reduced. One of the advantages of CNN compared with traditional image processing algorithms is that it avoids the complex pre-processing process (extracting artificial features, etc.) of the image, and can directly input the original image for end-to-end learning. One of the advantages of CNN compared with traditional neural networks is that traditional neural networks all use full connection, that is, all the neurons from the input layer to the hidden layer are connected, which will result in a large number of parameters, making the network training time-consuming and even difficult to train, while CNN avoids this difficulty through local connection and weight sharing.

[0106] The feature involved in the present application is a three-dimensional feature matrix of CxWxH (such asFigure 3 As shown, Figure 3 As shown in the schematic diagram of the matrix structure of the embodiment). C represents the number of channels, H represents the feature height, and W represents the feature width. The feature matrix can be the input of the neural network or the output of the neural network.

[0107] The indicators for evaluating the coding efficiency include code rate, PSNR, MS-SSIM, VMAF FSIM, PSNRHVS, and various indicators, and of course, more indicators can be included, which are not limited herein. The smaller the bit stream is, the greater the compression rate is; the greater the PSNR is, the better the image coding efficiency is. In the mode selection, the discrimination formula is essentially a comprehensive evaluation of the two. The cost corresponding to the mode is J(mode) = D + λ * R. Wherein, D represents Distortion, which is usually measured by using the SSE indicator, and the SSE is the sum of squares of the difference between the reconstructed block and the source image; λ is the Lagrange multiplier; R is the actual number of bits required for image block coding under the mode, including the total number of bits required for encoding mode information, residual error, and the like.

[0108] In a possible implementation manner of the embodiment, the obtained extended residual error data is grouped according to the feature channels corresponding to the extended residual error data. At this time, the step S10 of the embodiment can include:

[0109] extracting the extended residual error data from the image code stream;

[0110] grouping the extended residual error data based on the feature channels corresponding to the extended residual error data to obtain a plurality of extended residual error groups.

[0111] It should be noted that grouping the extended residual error data based on the feature channels corresponding to the extended residual error data to obtain a plurality of extended residual error groups can be to divide the extended residual error data into a plurality of groups uniformly according to the corresponding feature channels. For example, assuming that the total number of feature channels corresponding to the extended residual error data is 20, the extended residual error data corresponding to the feature channels 1-10 can be divided into a group, and the extended residual error data corresponding to the feature channels 11-20 can be divided into a group. The number of groups of uniform division can be set in advance by the management personnel of the decoding device, and the embodiment does not limit this.

[0112] Of course, the grouping can also be uneven, in which case the extension residual data is grouped based on the corresponding feature channels, and the extension residual groups can also be obtained by grouping the extension residual data into multiple groups based on a preset grouping rule, where the preset grouping rule can be set by the management personnel of the decoding device according to actual needs, for example: the preset grouping rule is set to group the first m / n (n is the total number of feature channels, m is a preset value, and the value range is [1, n)) extension residual data into a group, and the remaining extension residual data is grouped into another group.

[0113] In actual use, when the extension residual data is grouped based on the corresponding feature channels, the extension residual data corresponding to one feature channel can also be divided into a group, for example: assuming that the total number of feature channels corresponding to the extension residual data is 20, then the extension residual data can be divided into 20 groups according to the different feature channels.

[0114] Step S20: Residual recovery is performed on the multiple extension residual groups respectively to obtain image reconstruction features corresponding to each extension residual group.

[0115] It should be noted that the residual recovery of the extension residual group to obtain the image reconstruction feature corresponding to the extension residual group can be mean prediction of the extension residual group, and then adding the mean value predicted by the residual data in the extension residual group, thereby obtaining the image reconstruction feature corresponding to the extension residual group.

[0116] Step S30: The image reconstruction features corresponding to each extension residual group are subjected to upsampling spatial resolution processing to obtain reconstruction feature data.

[0117] It should be noted that since the extension residual data has been subjected to downsampling spatial resolution processing, the spatial size and the number of channels corresponding to each data are different from the spatial size and the number of channels of the image features obtained by the encoding device when performing feature extraction on the original image. At this time, in order to ensure the smooth execution of image reconstruction, the image reconstruction features corresponding to each extension residual group can be subjected to upsampling spatial resolution processing to restore the spatial size and the number of channels corresponding to the image reconstruction features to be consistent with the image features obtained by performing feature extraction on the original image.

[0118] The upsampling spatial resolution processing can be the inverse process of the downsampling spatial resolution processing in the encoding device.

[0119] Step S40: Image reconstruction is performed according to the reconstruction feature data to obtain a reconstructed image block.

[0120] It should be noted that after obtaining reconstructed feature data that has the same spatial size and number of channels as the image features corresponding to the original image, image reconstruction can be performed based on the reconstructed feature data to obtain reconstructed image blocks.

[0121] In this process, image reconstruction based on reconstruction feature data to obtain reconstructed image patches can be achieved by using a pre-built synthetic transformation network to perform synthetic transformation processing on the reconstruction feature data, thereby obtaining reconstructed image patches. The synthetic transformation network can be a network built based on deep learning or a neural network.

[0122] To facilitate understanding, we will now combine... Figure 3 This explanation is provided, but it does not limit the scope of this solution. Figure 3 For image compression, the overall framework diagram is as follows: Figure 3 As shown, at the encoding device: x is the input image, which is processed by the main encoder (Analysis TransformNet) to generate a latent representation y. Directly encoding y requires a large bit rate, so a context model net (Context ModelNet) and a hyperparameter encoding network (Hyper Encoder net) and a hyperparameter decoding network (Hyper Decoder net) are introduced for prediction, obtaining the prediction result μ, and the residual resi = y - μ. The distribution parameter σ of the residual is obtained through the hyperparameter encoding network (Hyper Encoder net) and the probabilistic hyperparameter decoding network (Hyper Scale Decoder net). The distribution parameter helps entropy coding encode data with a lower bit rate. Therefore, at the decoding end, the predicted mean μ and distribution parameter σ for each y are needed to correctly perform entropy decoding. The G-unit component is used to scale resi and the distribution parameter σ to control the quantization loss. The quantized residual is obtained by rounding resi to the nearest integer. Finally, lossless entropy encoding is performed.

[0123] At the decoding device: entropy decoding is performed on the image bitstream to obtain... invG-unit component Scaling is performed, where the scaling factors for the G-unit and invG-unit modules are opposite. The scaled value is then combined with the predicted value μ to obtain... The synthesization transform net, which is input to the decoder, produces the reconstructed image patch.

[0124] The embodiment extracts image residual data or extended residual data from an image code stream, obtains a plurality of extended residual groups based on the extracted image residual data or extended residual data, respectively performs residual recovery on the plurality of extended residual groups to obtain image reconstruction features corresponding to each extended residual group, performs upsampling spatial resolution processing on the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data, and performs image reconstruction based on the reconstruction feature data to obtain a reconstructed image block. Since the obtained extended residual data is residual data subjected to downsampling spatial resolution processing, the residual recovery processing can be performed on the groups in a grouped manner at a low resolution, thereby improving the overall residual recovery calculation efficiency and reducing the time complexity.

[0125] Reference Figure 4 , Figure 4 FIG. 1 is a flowchart of an image decoding method according to a second embodiment of the present application.

[0126] Based on the first embodiment, the step S10 of the image decoding method of the present embodiment comprises the following steps.

[0127] Step S101: Extracting image residual data from an image code stream.

[0128] It should be noted that the encoding device can perform upsampling spatial resolution processing on the generated extended residual data to restore the image residual data having the same spatial size and channel number as the original image, and then encode the image residual data into the image code stream. In this case, the decoding device can only extract the image residual data from the image code stream when decoding the image code stream.

[0129] Step S102: Performing downsampling spatial resolution processing on the image residual data to obtain extended residual data.

[0130] It should be understood that, in order to facilitate subsequent grouping processing and reduce the time complexity, the image residual data can be subjected to downsampling spatial resolution processing to obtain extended residual data after the image residual data is obtained. In order to ensure the correctness of decoding, the downsampling spatial resolution processing performed by the encoding device and the decoding device needs to be consistent, and the upsampling spatial resolution processing is the inverse process of the downsampling spatial resolution processing.

[0131] After the downsampling spatial resolution processing on the image residual data, the spatial size of the image residual data is reduced, and a smaller convolution kernel can be used for processing the image residual data. For example, if a 5x5 convolution kernel is used for the original image residual data, a 3x3 convolution kernel is equivalent to a convolution kernel used after the downsampling spatial resolution processing.

[0132] In actual use, the step of performing the spatial resolution reduction processing on the image residual data to obtain extended residual data can include:

[0133] reducing the spatial size corresponding to the image residual data, and / or increasing the number of feature channels corresponding to the image residual data, to obtain extended residual data.

[0134] It should be noted that the reduction amplitude of the spatial size corresponding to the image residual data and the increase amplitude of the number of feature channels corresponding to the image residual data can be set in advance by the management personnel of the decoding device, and the present embodiment does not limit this.

[0135] In actual execution, only the reduction amplitude of the spatial size can be set, or only the increase amplitude of the number of feature channels can be set, and the decoding device can adaptively adjust the spatial size or the number of feature channels, of course, the decoding device can also be set with the reduction amplitude of the spatial size and the increase amplitude of the number of feature channels, and then the decoding device can be executed.

[0136] For example, assuming that the image feature corresponding to the image residual data is y∈R{H,W,C}, where H is the height of the image feature, W is the width of the image feature, and C is the number of feature channels corresponding to the image feature, at this time, H and W can be reduced to half of the original, then in order to ensure that the data volume does not change, the number of feature channels will be 4 times the original, and at this time, the image feature corresponding to the obtained extended residual data can be represented as y∈R{H / 2,W / 2,4C}.

[0137] In specific implementation, the spatial resolution reduction processing can be performed based on the spatial domain information or the frequency information corresponding to the image residual data, and the spatial resolution reduction processing can also be performed through a preset convolution layer, and at this time, the step of reducing the spatial size corresponding to the image residual data, and / or increasing the number of feature channels corresponding to the image residual data, to obtain extended residual data can include:

[0138] reducing the spatial size corresponding to the image residual data according to the spatial domain information corresponding to the image residual data, and / or increasing the number of feature channels corresponding to the image residual data, to obtain extended residual data;

[0139] or,

[0140] reducing the spatial size corresponding to the image residual data according to the frequency domain information corresponding to the image residual data, and / or increasing the number of feature channels corresponding to the image residual data, to obtain extended residual data;

[0141] or,

[0142] According to a preset convolutional layer, the spatial size corresponding to the image residual data is reduced, and / or the number of feature channels corresponding to the image residual data is increased, to obtain extended residual data.

[0143] For the convenience of understanding, the present application will be described below in conjunction with Figure 5 , but the present application is not limited thereto. Figure 5 The spatial domain resolution processing flowchart of the present embodiment is shown in Figure 1. Figure 5 SpaceShuffle is one of the ways to reduce the spatial domain resolution, assuming that the input of the process is a1∈R{H,W,C}, and the output is a2∈R{H / 2,W / 2,4*C}. The mathematical expression is as follows:

[0144] a2[h,w,c]=a1[2*h,2*w,c]

[0145] a2[h,w,C+c]=a1[2*h+1,2*w+1,c]

[0146] a2[h,w,2*C+c]=a1[2*h,2*w+1,c]

[0147] a2[h,w,3*C+c]=a1[2*h+1,2*w,c]

[0148] Where h represents the index value in the height dimension, which ranges from 0 to H-1, w represents the index value in the width dimension, which ranges from 0 to W-1, and c represents the index value in the channel dimension, which ranges from 0 to C-1. And a1[h,w,c] represents the size at the corresponding h,w,c index position in a1. Figure 5 unSpaceShuffle is the inverse process of SpaceShuffle, and unSpaceShuffle is one of the ways to increase the spatial domain resolution. After the input is processed by SpaceShuffle and unSpaceShuffle, the output is the same as the original input, so the process is lossless. The input and output of the above process can also be other data, and the present application does not limit this.

[0149] As shown in method (b), method (b) is another way to process the spatial domain resolution according to the spatial domain information corresponding to the image residual data. Pixshuffle has a similar sampling method as method (a), but the difference is that the data is interleaved in the channel dimension. The data of the previous channel is divided into four channels according to the spatial domain, and arranged in order.

[0150] Figure 5The manner (c) in the image residual data is processed according to the frequency domain information corresponding to the image residual data. As shown in the manner (c), the Wavelet transform is a two-dimensional wavelet transform, which can divide the data in the frequency domain and output four frequency domain subbands with the dimension of [H / 2, W / 2, C]. Each of the four frequency domain subbands represents different frequency domain characteristics. The Inv Wavelet transform is the inverse process of the Wavelet transform, which combines the frequency domain subbands into the original data. The process is lossless.

[0151] Figure 5 The manner (d) in the image residual data is processed according to the preset convolution layer. As shown in the manner (d), the Convolution is a convolution transform, which is directly input into the convolution layer for processing, and the output is [H / 2, W / 2, 4*C], so as to reduce the spatial resolution. The corresponding inverse process is also processed by the convolution layer, and the original image size is restored. However, the manner is lossy, and the data after the processing of the two manners is different from the source data.

[0152] In a possible implementation manner of the embodiment, before the image residual data is processed by the manner of reducing the spatial resolution, the image residual data can be grouped first, and then the image residual data in each group is processed by the manner of reducing the spatial resolution. In this case, the step S102 includes the following steps.

[0153] grouping the data according to the feature channels corresponding to the image residual data to obtain at least one image residual group;

[0154] processing the data in the image residual group by the manner of reducing the spatial resolution to obtain the extended residual data.

[0155] It should be noted that the data can be grouped according to the feature channels corresponding to the image residual data in the same or similar manner as the grouping of the extended residual data. The data in the image residual group can be processed by the manner of reducing the spatial resolution, and then the data after the processing of reducing the spatial resolution is aggregated to obtain the extended residual data.

[0156] For example, assuming that the image feature corresponding to the image residual data is y∈R{H,W,C}, the image feature corresponding to the data in each image residual group can be expressed as y∈R{H,W,C / 2} when the image feature is divided into two groups according to the feature channels.

[0157] Step S103: Grouping the extended residual data to obtain a plurality of extended residual groups.

[0158] It should be noted that grouping the extended residual data to obtain a plurality of extended residual groups can be uniformly grouping the extended residual data into a plurality of groups according to the corresponding feature channels. For example, assuming that the total number of feature channels corresponding to the extended residual data is 20, then the extended residual data corresponding to the feature channels 1-10 can be grouped into a group, and the extended residual data corresponding to the feature channels 11-20 can be grouped into a group. The number of groups of uniform division can be set by the manager of the decoding device in advance, and the present embodiment does not limit this.

[0159] Of course, the grouping can also be non-uniform. The extended residual data can be grouped based on the feature channels corresponding to the extended residual data to obtain a plurality of extended residual groups. The preset grouping rule can be set by the manager of the decoding device according to actual needs, for example, the preset grouping rule can be set to group the first m / n (n is the total number of feature channels, m is a preset value, and the value range is [1, n)) extended residual data into a group, and the remaining extended residual data into another group.

[0160] In actual use, when grouping the extended residual data based on the feature channels corresponding to the extended residual data to obtain a plurality of extended residual groups, the extended residual data corresponding to one feature channel can be divided into a group. For example, assuming that the total number of feature channels corresponding to the extended residual data is 20, then the extended residual data can be divided into 20 groups according to the different feature channels.

[0161] Before grouping, the present embodiment will also detect whether the extracted is image residual data or extended residual data. If it is image residual data, it will also be reduced in spatial resolution first to ensure that even if the image code stream transmitted by the encoding end contains image residual data, it can still be normally grouped and processed after processing, thereby improving the universality of the image decoding method of the present embodiment.

[0162] Reference Figure 6 , Figure 6 is a flowchart of a third embodiment of the image decoding method of the present application.

[0163] Based on the above first embodiment, the step S20 of the image decoding method of the present embodiment comprises:

[0164] Step S201: Constructing a residual recovery sequence according to the plurality of extended residual groups.

[0165] It should be noted that after the plurality of extended residual groups are divided, the residual recovery can be performed on the extended residual groups one by one, so as to reduce the time complexity of the mean prediction in the image decoding process. At this time, in order to determine the residual recovery order of each extended residual group, the residual recovery sequence can be constructed according to the plurality of extended residual groups.

[0166] Step S202: performing residual recovery on the plurality of extended residual groups respectively based on the residual recovery sequence, and obtaining image reconstruction features corresponding to each extended residual group.

[0167] It should be noted that performing residual recovery on the plurality of extended residual groups based on the residual recovery sequence can be sequentially performing residual recovery on the plurality of extended residual groups based on the sequence order in the residual recovery sequence.

[0168] In actual use, the residual recovery can be sequentially performed in a sequence traversal manner, and at this time, the step S202 in the embodiment can include:

[0169] traversing the residual recovery sequence to obtain a current extended residual group;

[0170] obtaining auxiliary information output by the auxiliary encoding network;

[0171] constructing prior information according to the auxiliary information;

[0172] performing residual recovery on the current extended residual group based on the prior information, and obtaining image reconstruction features corresponding to the current extended residual group;

[0173] At the end of the traversal, image reconstruction features corresponding to each extended residual group are obtained.

[0174] It should be noted that traversing the residual recovery sequence to obtain a current extended residual group can be traversing the residual recovery sequence and taking the traversed extended residual group as the current extended residual group. The auxiliary encoding network can be the auxiliary network (Hyper Encoder Net or Hyper Decoder Net) shown in the above Figure 3 .

[0175] In actual use, performing residual recovery on the current extended residual group based on the prior information to obtain image reconstruction features corresponding to the current extended residual group can be processing the prior information through a prediction parameter fusion network (Prediction Fusion Net) to obtain a predicted mean, and then adding the predicted mean and the residual in the current extended residual group to realize residual recovery and obtain the image reconstruction features corresponding to the current extended residual group.

[0176] In actual use, the step of constructing the prior information according to the auxiliary information in the embodiment can include:

[0177] obtaining extended auxiliary information;

[0178] detecting whether the current extended residual group is the first element in the residual recovery sequence;

[0179] if the current extended residual group is the first element in the residual recovery sequence, constructing the prior information according to the extended auxiliary information;

[0180] if the current extended residual group is not the first element in the residual recovery sequence, splicing the convolution processing result corresponding to the image reconstruction feature of the recovered extended residual group and the extended auxiliary information to obtain spliced auxiliary information, and constructing the prior information according to the spliced auxiliary information.

[0181] It should be noted that the spatial size and the number of feature channels of the auxiliary information output by the auxiliary coding network are actually consistent with those of the original image, and at this time, the extended residual data has actually been processed to reduce the spatial resolution. Therefore, in order to ensure smooth channel splicing, the auxiliary information needs to be processed first to obtain extended auxiliary information, and then the prior information is constructed according to the extended auxiliary information.

[0182] When constructing the prior information according to the extended auxiliary information, in order to increase the accuracy of the mean prediction, the image features corresponding to the reconstructed extended residual groups can also be used to construct the prior information. If the current extended residual group is the first element in the residual recovery sequence, it means that the current extended residual group is the first extended residual group to be recovered. At this time, there is no reconstructed extended residual group, so the prior information can be directly constructed according to the extended auxiliary information.

[0183] If the current extended residual group is not the first element in the residual recovery sequence, there is a reconstructed extended residual group at this time. Therefore, the image reconstruction features of the recovered extended residual group can be convolved by a convolution layer, then the channel splicing of the convolution processing result corresponding to the image reconstruction features of the recovered extended residual group and the extended auxiliary information is performed, and the prior information is constructed according to the spliced auxiliary information. When constructing the prior information, all recovered extended residual groups can be selected, or only a part of the recovered extended residual groups can be selected.

[0184] In a possible implementation manner of the embodiment, when the result is spliced, the image reconstruction features of the recovered extended residual group can be first enhanced to further improve the prediction effect. The step of splicing the convolution processing result corresponding to the image reconstruction features of the recovered extended residual group and the extended auxiliary information to obtain spliced auxiliary information can include:

[0185] obtain image reconstruction features corresponding to the recovered extended residual groups;

[0186] perform feature enhancement on the image reconstruction features to obtain enhanced reconstruction features;

[0187] splice the auxiliary information and the convolution processing result corresponding to the enhanced reconstruction features to obtain spliced auxiliary information.

[0188] In actual use, performing feature enhancement on the image reconstruction features to obtain enhanced reconstruction features can be missing value processing, outlier processing, and the like on the image reconstruction features.

[0189] It can be understood that, before splicing the auxiliary information and the image reconstruction features, performing feature enhancement on the image reconstruction features to obtain enhanced reconstruction features can increase the reliability of the enhanced reconstruction features, thereby improving the reliability of the prior information, and making the mean value prediction more accurate.

[0190] In actual use, when performing feature enhancement on the image reconstruction features corresponding to the recovered extended residual groups, the predicted mean value, the auxiliary information, the image residual data, and / or the residual data variance corresponding to the recovered extended residual groups can be used, and the step of performing feature enhancement on the image reconstruction features to obtain enhanced reconstruction features in this embodiment can include:

[0191] obtain the predicted mean value, the auxiliary information, the image residual data, and / or the residual data variance corresponding to the recovered extended residual groups;

[0192] perform feature enhancement on the image reconstruction features according to the predicted mean value, the auxiliary information, the image residual data, and / or the residual data variance corresponding to the recovered extended residual groups to obtain enhanced reconstruction features.

[0193] It should be noted that the predicted mean value corresponding to the recovered extended residual groups can be a value obtained when performing mean value prediction on the recovered extended residual groups during residual recovery. The residual data variance can be a variance value of the image residual data corresponding to the recovered extended residual groups.

[0194] In a possible implementation manner of this embodiment, in order to improve the reconstruction effect of image reconstruction, the step S30 in this embodiment can include:

[0195] perform feature enhancement on the image reconstruction features corresponding to each extended residual group to obtain enhanced reconstruction features corresponding to each extended residual group;

[0196] perform spatial resolution amplification processing on the enhanced reconstruction features corresponding to each extended residual group to obtain reconstruction feature data.

[0197] It should be noted that feature enhancement of the image reconstruction features corresponding to the extended residual group can be achieved by using the prediction mean, auxiliary information, image residual data and / or residual data variance of the extended residual group to enhance the image reconstruction features corresponding to the extended residual group.

[0198] It is understandable that before performing spatial resolution amplification processing on the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data, feature enhancement is first performed on the image reconstruction features corresponding to each extended residual group, and then spatial resolution amplification processing is performed on the enhanced reconstruction features corresponding to each extended residual group to obtain reconstruction feature data. This can ensure that the reliability of the final constructed reconstruction feature data is higher, and the quality of the reconstructed image blocks obtained by subsequent image reconstruction will be better.

[0199] To facilitate understanding, we will now combine... Figure 7 , 8 The following points are provided for explanation, but do not limit the scope of this scheme. Figure 7 This is a schematic diagram of the image decoding grouping execution process in this embodiment. Figure 8 This is a schematic diagram of the secondary grouping execution process in this embodiment. Figure 9 This is a schematic diagram of the feature enhancement grouping execution process in this embodiment.

[0200] like Figure 7 As shown, the image features corresponding to the image residual data are y∈R{H,W,C}. First, the spatial resolution is reduced, resulting in y∈R{H / 2,W / 2,4C}. This can then be divided into two groups (group1 and group2). A residual recovery sequence (group1-group2) is then constructed. The mean value mu of Group1 is obtained through a network using auxiliary (Psi) information, yielding the image reconstruction features of Group1. These features are then extracted and concatenated with Psi via channel concatenation. The mean value mu of Group2 is obtained through the network and then decoded to obtain Group2. Combining Group1 and Group2 via channel concatenation, a special spatial resolution amplification process (i.e., the reverse process of spatial resolution reduction) is performed to obtain the reconstructed feature data. Finally, image reconstruction is performed based on this reconstructed feature data.

[0201] However, if grouping is performed before reducing the spatial resolution, the execution flow is as follows: Figure 8As shown, the image feature corresponding to the image residual data is y e R{H,W,C}, at this time, according to the corresponding feature channel, it is divided into y1 and y2, at this time, y1 e R{H,W,C / 2}, at this time, it is respectively reduced in spatial resolution, y1 is divided into part1, part2, part3, part4, at this time, there are:

[0202] Part1 is represented in y1, the spatial position is in the even row, the odd column of y1 spatial index, and contains the data information of all channels. Part2 is represented in y1, the spatial position is in the odd row, the even column of y1 spatial index, and contains the data information of all channels. Part3 is represented in y1, the spatial position is in the even row, the odd column of y1 spatial index, and contains the data information of all channels. Part4 is represented in y1, the spatial position is in the odd row, the even column of y1 spatial index, and contains the data information of all channels.

[0203] Similarly, y2 can also be divided into similar four parts, part5, part6, part7, part8, at this time, the residual recovery sequence constructed is "part1-part2-part3-part4-part5-part6-part7-part8", at this time, the prior information can be generated according to the auxiliary information when part1 is recovered, the prior information can be generated according to the image reconstruction feature of part1 and auxiliary information when part2 is recovered, the prior information can be generated according to the image reconstruction feature of part1 and part2 and auxiliary information when part3 is recovered, and so on, until the image reconstruction feature corresponding to all parts is obtained;

[0204] Of course, two residual recovery sequences can also be constructed according to y1 and y2, at this time, the two residual recovery sequences are "part1-part2-part3-part4" and "part5-part6-part7-part8", at this time, y1 can be reconstructed according to the residual recovery sequence composed of part1-4 by using the similar process as above, to obtain the image reconstruction feature corresponding to each part in y1, then the prior information is generated according to the image reconstruction feature corresponding to y1 and auxiliary information, and y2 is reconstructed according to the residual recovery sequence composed of part5-8 and the generated prior information, to obtain the image reconstruction feature corresponding to each part in y2.

[0205] Wherein, since the y is first split into y1 and y2 and then respectively reduced in spatial resolution, after obtaining the image reconstruction features corresponding to each part of y1 and y2, the image reconstruction features corresponding to each part of y1 and y2 are respectively enlarged in spatial resolution and then aggregated, so as to obtain the complete reconstruction feature data.

[0206] If the feature enhancement is performed after grouping, the specific execution process is as shown in (a) of Figure 9 , the image feature corresponding to the image residual data is y∈R{H,W,C}, which is first reduced in spatial resolution to obtain y∈R{H / 2,W / 2,4C}, which can be divided into two groups (group1 and group2), and then the mean mu of the group1 points is obtained through the auxiliary (Psi) information through the network, the image reconstruction feature of group1 is obtained, the image reconstruction feature of group1 is enhanced using the prediction mean corresponding to group1, and the enhanced image feature group1_E is obtained, and then the features of the enhanced image feature are extracted and spliced with Psi, the mean mu of group2 is obtained through the network, and the image reconstruction feature of group2 is obtained through decoding, and then the image reconstruction feature of group2 is enhanced according to the mean mu of group2, and the enhanced image feature group2_E of group2 is obtained. Channel splicing is performed on group1_E and group2_E, and special spatial resolution enlargement processing (which is the inverse process of spatial resolution reduction processing) is performed, so as to obtain the reconstruction feature data, and then the image reconstruction is performed according to the reconstruction feature data. Wherein, in the feature enhancement in (a), the specific structure of the network (Enhance_Net) used for feature enhancement is as shown in (b) of Figure 9 . Figure 9

[0207] The embodiment constructs a residual recovery sequence according to the plurality of extended residual groups, and performs residual recovery on the plurality of extended residual groups based on the residual recovery sequence, to obtain image reconstruction features corresponding to each extended residual group. Since the residual recovery sequence is constructed according to the plurality of extended residual groups, the order of residual recovery can be determined through the residual recovery sequence, so that it can be quickly determined whether there is an already recovered extended residual group, and when there is an already recovered extended residual group, more accurate prior information can be constructed according to the image feature data corresponding to the already recovered extended residual group.

[0208] The embodiment of the present application provides an image coding method, referring to Figure 10 , Figure 10 is a flowchart of a first embodiment of an image coding method of the present application.

[0209] ​In this embodiment, the image encoding method comprises the following steps:

[0210] Step S100: Perform down-spatial resolution processing on the image features corresponding to the to-be-encoded image to obtain extended image features.

[0211] It should be noted that, in order to facilitate subsequent grouping processing to reduce the time complexity, after obtaining the image features corresponding to the to-be-encoded image, the image residual data can be subjected to down-spatial resolution processing to obtain extended image features. The to-be-encoded image is the original image mentioned in the image decoding method embodiment.

[0212] After the down-spatial resolution processing on the image features, the spatial size thereof will be smaller, and accordingly, a smaller convolution kernel can be used when processing the image features. For example, if a 5x5 convolution kernel is used on the original image residual data, a 3x3 convolution kernel is equivalent to being used on the image features after the down-spatial resolution processing.

[0213] In a possible implementation manner of this embodiment, the step S100 of this embodiment can comprise:

[0214] The spatial size of the image features of the to-be-encoded image is reduced, and / or the number of feature channels corresponding to the image features is increased, to obtain extended image features.

[0215] It should be noted that the reduction amplitude of the spatial size of the image features and the increase amplitude of the number of feature channels corresponding to the image features can be set in advance by the management personnel of the decoding device, and this embodiment does not limit this.

[0216] In actual execution, only the reduction amplitude of the spatial size can be set, or only the increase amplitude of the number of feature channels can be set, and the decoding device can adaptively adjust the spatial size or the number of feature channels. Of course, the device can also be set to adaptively adjust the reduction amplitude of the spatial size and the increase amplitude of the number of feature channels, and then the device is executed.

[0217] In a specific implementation, the down-spatial resolution processing can be performed based on the spatial information or frequency information of the image features, or the down-spatial resolution processing can be performed through a preset convolution layer. At this time, the step of reducing the spatial size of the image features of the to-be-encoded image and / or increasing the number of feature channels corresponding to the image features to obtain extended image features can comprise:

[0218] reducing a spatial size corresponding to the image feature of the to-be-encoded image according to spatial domain information corresponding to the image feature, and / or increasing a feature channel number corresponding to the image feature, to obtain an expanded image feature;

[0219] or,

[0220] reducing a spatial size corresponding to the image feature of the to-be-encoded image according to frequency domain information corresponding to the image feature, and / or increasing a feature channel number corresponding to the image feature, to obtain an expanded image feature;

[0221] or,

[0222] reducing a spatial size corresponding to the image feature according to a preset convolution layer, and / or increasing a feature channel number corresponding to the image feature, to obtain an expanded image feature.

[0223] In specific implementation, the explanation and description of the above-mentioned image decoding method embodiments can be referred to for the specific implementation, which will not be repeated here. Figure 5

[0224] Step S200: Grouping the expanded image features to obtain a plurality of expanded feature groups.

[0225] In specific implementation, when grouping the expanded image features, the expanded image features can be grouped according to the feature channels corresponding to the expanded image features. In this case, the step S200 of the embodiment can include:

[0226] grouping the expanded image features according to the feature channels corresponding to the expanded feature data to obtain a plurality of expanded feature groups.

[0227] It should be noted that grouping the expanded image features according to the feature channels corresponding to the expanded feature data to obtain a plurality of expanded feature groups can be to divide the expanded image features into a plurality of groups according to the corresponding feature channels. For example, assuming that the total number of feature channels corresponding to the expanded image features is 20, the expanded image features corresponding to the feature channels 1-10 can be divided into a group, and the expanded image features corresponding to the feature channels 11-20 can be divided into a group. The number of groups of uniform division can be set in advance by the management personnel of the decoding device, and the embodiment does not limit this.

[0228] ​Of course, the grouping can also be uneven. The extended image features are grouped based on the feature channels corresponding to the extended image features, and the plurality of extended feature groups can also be obtained by grouping the extended image features into a plurality of groups based on a preset grouping rule. The preset grouping rule can be set by the management personnel of the encoding device according to actual needs, for example, the preset grouping rule is set to group the first m / n (n is the total number of feature channels, and m is a preset value in the range of [1, n)) extended image features into a group, and the remaining extended image features are grouped into another group.

[0229] In actual use, when the extended image features are grouped based on the feature channels corresponding to the extended image features to obtain a plurality of extended feature groups, the extended image features corresponding to one feature channel can also be divided into a group, for example, assuming that the total number of feature channels corresponding to the extended image features is 20, then the extended image features can be divided into 20 groups according to the different feature channels.

[0230] Step S300: performing residual calculation on the plurality of extended feature groups respectively to obtain image residual data corresponding to each extended feature group.

[0231] It should be noted that the residual calculation on the extended feature groups respectively to obtain the image residual data corresponding to the extended feature groups can be mean prediction on the extended feature groups, and then subtracting the image features in the extended feature groups from the predicted mean to obtain the image residual data corresponding to the extended feature groups.

[0232] Step S400: generating an image code stream according to the image residual data, and sending the image code stream to an image decoding end.

[0233] It should be noted that the image code stream can be generated by writing the image residual data into the image code stream through entropy encoding.

[0234] In a possible implementation manner of the embodiment, in the process of reducing the spatial resolution of the image features corresponding to the image to be encoded, the image features can be first grouped, and then the spatial resolution of each group is reduced. At this time, the step S100 of the embodiment can include:

[0235] Obtaining image features corresponding to an image to be encoded;

[0236] Grouping the image features according to the feature channels corresponding to the image features to obtain at least one image feature group;

[0237] Performing spatial resolution reduction processing on the data in the image feature group to obtain extended image features.

[0238] In a possible implementation of the embodiment, the image feature corresponding to the to-be-encoded image is subjected to down-spatial resolution processing to obtain an extended image feature. Alternatively, after the image feature corresponding to the to-be-encoded image is obtained, the data in the image feature is subjected to down-spatial resolution processing to obtain the extended image feature. It should be noted that the image feature corresponding to the to-be-encoded image can be extracted by using a preset feature extraction network. The same or similar manner as that for grouping the extended image feature can be used when the data in the feature channel corresponding to the feature channel of the image feature is grouped.

[0239] In a possible implementation of the embodiment, the step S400 can include the following steps.

[0240] The image residual feature corresponding to each extended feature group is subjected to up-spatial resolution processing to obtain image residual data corresponding to the to-be-encoded image.

[0241] The image code stream is generated according to the image residual data corresponding to the to-be-encoded image, and the image code stream is sent to the image decoding end.

[0242] It should be noted that the up-spatial resolution processing can be the inverse process of the down-spatial resolution processing. After the image residual feature corresponding to each extended feature group is obtained, the image residual feature corresponding to each extended feature group is subjected to up-spatial resolution processing, so that the spatial size and the number of channels thereof are restored to be consistent with the image feature corresponding to the to-be-encoded image, thereby obtaining the image residual data corresponding to the to-be-encoded image. Then, the image residual data corresponding to the to-be-encoded image is subjected to entropy encoding to generate the image code stream, and the generated image code stream is sent to the image decoding end.

[0243] In the embodiment, the image feature corresponding to the to-be-encoded image is subjected to down-spatial resolution processing to obtain an extended image feature. The extended image feature is grouped to obtain a plurality of extended feature groups. The plurality of extended feature groups are subjected to residual calculation respectively to obtain image residual data corresponding to each extended feature group. The image code stream is generated according to the image residual data, and the image code stream is sent to the image decoding end. Since the image feature of the to-be-encoded image is subjected to down-spatial resolution processing after being obtained, and then grouped into a plurality of extended feature groups, the residual calculation processing can be implemented in a low resolution. Therefore, the overall residual calculation efficiency is improved, and the time complexity is reduced.

[0244] Reference Figure 11 , Figure 11 FIG. 2 is a flowchart of a second embodiment of an image encoding method according to the present application.

[0245] Based on the first embodiment, the step S300 of the image encoding method of the present embodiment comprises:

[0246] Step S3001: Constructing a residual calculation sequence according to the plurality of extended feature groups.

[0247] It should be noted that after the plurality of extended feature groups are divided, residual calculation can be performed on the groups. At this time, in order to determine the residual calculation order of each extended feature group, a residual calculation sequence can be constructed according to the plurality of extended feature groups.

[0248] Step S3002: Based on the residual calculation sequence, residual calculation is performed on the plurality of extended feature groups respectively to obtain image residual data corresponding to each extended feature group.

[0249] It should be noted that residual calculation can be performed on the plurality of extended feature groups respectively based on the sequence order in the residual calculation sequence.

[0250] In actual use, residual calculation can be sequentially performed through sequence traversal. At this time, the step S3002 of the present embodiment can comprise:

[0251] Traversing the residual calculation sequence to obtain a current extended feature group;

[0252] Obtaining auxiliary information output by the auxiliary encoding network;

[0253] Constructing prior information according to the auxiliary information;

[0254] Based on the prior information, residual calculation is performed on the current extended feature group to obtain image residual data corresponding to the current extended feature group;

[0255] At the end of traversal, image residual data corresponding to each extended feature group is obtained.

[0256] It should be noted that traversing the residual calculation sequence to obtain a current extended feature group can be traversing the residual calculation sequence and taking the traversed extended feature group as the current extended feature group. The auxiliary encoding network can be the auxiliary network (Hyper Encoder Net or Hyper Decoder Net) shown in the above Figure 3 .

[0257] In actual use, the residual calculation on the current extended feature group based on the prior information can be performed by processing the prior information through the Prediction Fusion Net to obtain a predicted mean value, and then subtracting the features in the current extended feature group from the predicted mean value to realize the residual calculation and obtain the image residual data corresponding to the current extended feature group.

[0258] In the process of constructing the prior information according to the auxiliary information, in order to increase the accuracy of the mean value prediction, the image features corresponding to the extended feature groups that have been subjected to the residual calculation can also be used to construct the prior information, and the specific implementation manner is the same as that applied in the image decoding process. The specific implementation steps can refer to the manner of constructing the prior information according to the auxiliary information provided in any of the above image decoding method embodiments.

[0259] The embodiment constructs a residual calculation sequence according to a plurality of extended feature groups, and performs residual calculation on the plurality of extended feature groups based on the residual calculation sequence to obtain image residual data corresponding to each extended feature group. Since the residual calculation sequence is constructed according to the plurality of extended feature groups, the order of the residual calculation can be determined through the residual calculation sequence, so that it can be quickly determined whether there is an already calculated extended feature group, and when there is an already calculated extended feature group, more accurate prior information can be constructed according to the convolution processing result corresponding to the image reconstruction features of the already recovered extended residual group.

[0260] In addition, the embodiment of the present application also provides a storage medium, and the storage medium stores an image decoding program and / or an image encoding program. The image decoding program is executed by a processor to implement the steps of the image decoding method described above, and the image encoding program is executed by the processor to implement the steps of the image encoding method described above.

[0261] Reference Figure 12 , Figure 12 is a structural block diagram of a first embodiment of an image decoding apparatus of the present application.

[0262] As Figure 12 shown, the image decoding apparatus provided by the embodiment of the present application comprises:

[0263] A bitstream decoding module 10 is configured to extract image residual data or extended residual data from an image bitstream, and obtain a plurality of extended residual groups based on the extracted image residual data or extended residual data.

[0264] A residual recovery module 20 is configured to perform residual recovery on the plurality of extended residual groups respectively to obtain image reconstruction features corresponding to each extended residual group.

[0265] The data combination module 30 is configured to perform up-sampling spatial resolution processing on the image reconstruction features corresponding to each extended residual group, to obtain reconstruction feature data.

[0266] The image reconstruction module 40 is configured to perform image reconstruction according to the reconstruction feature data, to obtain a reconstructed image block.

[0267] In the embodiment, the image residual data or the extended residual data extracted from the image code stream is obtained, and based on the image residual data or the extended residual data, a plurality of extended residual groups are obtained. Residual recovery is performed on each of the extended residual groups, to obtain image reconstruction features corresponding to each of the extended residual groups. Up-sampling spatial resolution processing is performed on the image reconstruction features corresponding to each of the extended residual groups, to obtain reconstruction feature data. Image reconstruction is performed according to the reconstruction feature data, to obtain a reconstructed image block. Since the extended residual data is residual data that has been subjected to down-sampling spatial resolution processing, residual recovery processing can be performed on each group in a low resolution, thereby improving the overall residual recovery calculation efficiency and reducing the time complexity.

[0268] In a possible implementation of the embodiment, the code stream decoding module 10 is further configured to extract extended residual data from the image code stream, and group the extended residual data according to the feature channels corresponding to the extended residual data, to obtain a plurality of extended residual groups.

[0269] In a possible implementation of the embodiment, the code stream decoding module 10 is further configured to extract image residual data from the image code stream, perform down-sampling spatial resolution processing on the image residual data, to obtain extended residual data, and the up-sampling spatial resolution processing is the inverse process of the down-sampling spatial resolution processing. The extended residual data is grouped, to obtain a plurality of extended residual groups.

[0270] In a possible implementation of the embodiment, the code stream decoding module 10 is further configured to group data according to the feature channels corresponding to the image residual data, to obtain at least one image residual group, and perform down-sampling spatial resolution processing on the data in the image residual group, to obtain extended residual data.

[0271] In a possible implementation of the embodiment, the code stream decoding module 10 is further configured to reduce the spatial size corresponding to the image residual data, and / or increase the number of feature channels corresponding to the image residual data, to obtain extended residual data.

[0272] In a possible implementation of the present embodiment, the code stream decoding module 10 is further configured to reduce the spatial size corresponding to the image residual data according to the spatial domain information corresponding to the image residual data, and / or increase the number of feature channels corresponding to the image residual data, to obtain extended residual data; or, reduce the spatial size corresponding to the image residual data according to the frequency domain information corresponding to the image residual data, and / or increase the number of feature channels corresponding to the image residual data, to obtain extended residual data; or, reduce the spatial size corresponding to the image residual data according to a preset convolution layer, and / or increase the number of feature channels corresponding to the image residual data, to obtain extended residual data.

[0273] In a possible implementation of the present embodiment, the residual recovery module 20 is further configured to construct a residual recovery sequence according to the plurality of extended residual groups; perform residual recovery on each of the plurality of extended residual groups based on the residual recovery sequence, to obtain image reconstruction features corresponding to each of the plurality of extended residual groups.

[0274] In a possible implementation of the present embodiment, the residual recovery module 20 is further configured to traverse the residual recovery sequence to obtain a current extended residual group; obtain auxiliary information output by the auxiliary encoding network; construct prior information according to the auxiliary information; perform residual recovery on the current extended residual group based on the prior information, to obtain image reconstruction features corresponding to the current extended residual group; and obtain image reconstruction features corresponding to each of the plurality of extended residual groups at the end of the traversal.

[0275] In a possible implementation of the present embodiment, the residual recovery module 20 is further configured to obtain extended auxiliary information; detect whether the current extended residual group is the first element in the residual recovery sequence; if the current extended residual group is the first element, construct prior information according to the extended auxiliary information; if the current extended residual group is not the first element, splice the extended auxiliary information and a convolution processing result corresponding to the image reconstruction features of the recovered extended residual group, to obtain spliced auxiliary information, and construct prior information according to the spliced auxiliary information.

[0276] In a possible implementation of the present embodiment, the residual recovery module 20 is further configured to obtain image reconstruction features corresponding to the recovered extended residual group; perform feature enhancement on the image reconstruction features, to obtain enhanced reconstruction features; and splice the auxiliary information and a convolution processing result corresponding to the enhanced reconstruction features, to obtain spliced auxiliary information.

[0277] In a possible implementation of the embodiment, the residual recovery module 20 is further configured to obtain a prediction mean, auxiliary information, image residual data and / or residual data variance corresponding to the recovered extended residual group; and perform feature enhancement on the image reconstruction feature according to the prediction mean, auxiliary information, image residual data and / or residual data variance corresponding to the recovered extended residual group to obtain an enhanced reconstruction feature.

[0278] In a possible implementation of the embodiment, the data combination module 30 is further configured to perform feature enhancement on the image reconstruction feature corresponding to each extended residual group to obtain an enhanced reconstruction feature corresponding to each extended residual group; and perform upscaling spatial resolution processing on the enhanced reconstruction feature corresponding to each extended residual group to obtain reconstruction feature data.

[0279] In a possible implementation of the embodiment, the image reconstruction module 40 is further configured to perform synthetic transformation processing on the reconstruction feature data by using a pre-constructed synthetic transformation network to implement the image reconstruction, to obtain the reconstructed image block, wherein the synthetic transformation network is a network constructed based on deep learning or a neural network.

[0280] In a possible implementation of the embodiment, the code stream decoding module 10 is further configured to group the extended residual data uniformly according to the feature channels corresponding to the extended residual data.

[0281] Reference Figure 13 , Figure 13 is a structural block diagram of a first embodiment of an image coding device of the present application.

[0282] As Figure 13 shown, the image coding device provided by the embodiment of the present application comprises:

[0283] a feature extraction module 100 configured to perform downscaling spatial resolution processing on image features corresponding to a to-be-coded image to obtain extended image features;

[0284] a data grouping module 200 configured to group the extended image features to obtain a plurality of extended feature groups;

[0285] a residual calculation module 300 configured to perform residual calculation on the plurality of extended feature groups respectively to obtain image residual data corresponding to each extended feature group;

[0286] a code stream generation module 400 configured to generate an image code stream according to the image residual data, and send the image code stream to an image decoding end.

[0287] The embodiment obtains extended image features by performing downsizing spatial resolution processing on image features corresponding to the image to be encoded, groups the extended image features to obtain a plurality of extended feature groups, respectively performs residual calculation on the plurality of extended feature groups to obtain image residual data corresponding to each extended feature group, generates an image code stream according to the image residual data, and sends the image code stream to an image decoding end. Since the image features of the image to be encoded are subjected to downsizing spatial resolution processing after being obtained, and then grouped into a plurality of extended feature groups, the residual calculation processing can be performed on the entire group at a low resolution, thereby improving the overall residual calculation efficiency and reducing the time complexity.

[0288] In a possible implementation manner of the embodiment, the feature extraction module 100 is further configured to obtain image features corresponding to the image to be encoded, and perform downsizing spatial resolution processing on data in the image features to obtain extended image features.

[0289] In a possible implementation manner of the embodiment, the feature extraction module 100 is further configured to

[0290] The spatial size corresponding to the image features of the image to be encoded is reduced, and / or the number of feature channels corresponding to the image features is increased, to obtain extended image features.

[0291] In a possible implementation manner of the embodiment, the feature extraction module 100 is further configured to reduce the spatial size corresponding to the image features of the image to be encoded according to spatial information corresponding to the image features, and / or increase the number of feature channels corresponding to the image features, to obtain extended image features; or, reduce the spatial size corresponding to the image features according to frequency information corresponding to the image features, and / or increase the number of feature channels corresponding to the image features, to obtain extended image features; or, reduce the spatial size corresponding to the image features according to a preset convolution layer, and / or increase the number of feature channels corresponding to the image features, to obtain extended image features.

[0292] In a possible implementation manner of the embodiment, the residual calculation module 300 is further configured to construct a residual calculation sequence according to the plurality of extended feature groups, and respectively perform residual calculation on the plurality of extended feature groups based on the residual calculation sequence to obtain image residual data corresponding to each extended feature group.

[0293] In a possible implementation of the embodiment, the residual calculation module 300 is further configured to traverse the residual calculation sequence to obtain a current extended feature group; obtain auxiliary information output by an auxiliary coding network; construct prior information according to the auxiliary information; perform residual calculation on the current extended feature group based on the prior information to obtain image residual data corresponding to the current extended feature group; and obtain image residual data corresponding to each extended feature group at the end of the traversal.

[0294] In a possible implementation of the embodiment, the data grouping module 200 is further configured to group the extended image features based on feature channels corresponding to the extended feature data to obtain a plurality of extended feature groups.

[0295] In a possible implementation of the embodiment, the code stream generation module 400 is further configured to perform upscaling spatial resolution processing on image residual features corresponding to each extended feature group to obtain image residual data corresponding to the to-be-encoded image, the upscaling spatial resolution processing being an inverse process of downscaling spatial resolution processing; generate an image code stream according to the image residual data corresponding to the to-be-encoded image, and send the image code stream to an image decoding end.

[0296] It should be understood that the above is only for illustration, and does not constitute any limitation on the technical solutions of the present application. In specific applications, those skilled in the art can set it up according to the needs, and the present application does not limit it.

[0297] It should be noted that the above-described workflow is only illustrative and does not limit the scope of protection of the present application. In actual applications, those skilled in the art can select part or all of them to achieve the purpose of the embodiment, and this place does not limit it.

[0298] In addition, technical details not described in detail in the embodiment can be referred to the image decoding method or image encoding method provided by any embodiment of the present application, which will not be described here.

[0299] In addition, it should be noted that in this document, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or system. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of another identical element in the process, method, article or system that includes the element.

[0300] The above embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0301] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, also can be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of contribution to the prior art can be embodied in the form of software product, the computer software product is stored in a storage medium (such as read only memory (Read Only Memory, ROM) / RAM, disk, optical disk), including a number of instructions to make a terminal device (may be a mobile phone, computer, server, or network equipment, etc.) executes the method described in various embodiments of the present application.

[0302] The above is only the preferred embodiment of the present application, not therefore limit the patent scope of the present application, any equivalent structure or equivalent flow transformation made by using the content of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image decoding method characterized by, The image decoding method comprises: grouping the extended residual data to obtain a plurality of extended residual groups; constructing a residual recovery sequence according to the plurality of extended residual groups; performing residual recovery on the plurality of extended residual groups respectively based on the residual recovery sequence to obtain image reconstruction features corresponding to each extended residual group; performing upscaling spatial resolution processing on the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data; performing image reconstruction according to the reconstruction feature data to obtain a reconstructed image block.

2. The image decoding method of claim 1, wherein, The residual recovery on the plurality of extended residual groups respectively based on the residual recovery sequence to obtain image reconstruction features corresponding to each extended residual group comprises: traversing the residual recovery sequence to obtain a current extended residual group; obtaining prior information; performing residual recovery on the current extended residual group based on the prior information to obtain image reconstruction features corresponding to the current extended residual group; obtaining image reconstruction features corresponding to each extended residual group at the end of the traversal.

3. The image decoding method of claim 2, wherein, The obtaining of the prior information comprises: obtaining extended auxiliary information based on information output by an auxiliary encoding network; detecting whether the current extended residual group is the first element in the residual recovery sequence; if the current extended residual group is the first element, constructing prior information according to the extended auxiliary information; if the current extended residual group is not the first element, splicing the extended auxiliary information and convolution processing results corresponding to image reconstruction features of a recovered extended residual group to obtain spliced auxiliary information, and constructing prior information according to the spliced auxiliary information.

4. The image decoding method of claim 2, wherein, The residual recovery on the current extended residual group based on the prior information to obtain image reconstruction features corresponding to the current extended residual group comprises: processing the prior information through a convolutional neural network to obtain a predicted mean value; adding the predicted mean value to residual in the current extended residual group to obtain image reconstruction features corresponding to the current extended residual group.

5. The image decoding method according to any one of claims 1-4, wherein when performing upscaling spatial resolution processing on the image reconstruction features corresponding to each extended residual group, the upscaling spatial resolution processing is used to restore a spatial size corresponding to the image reconstruction features to be consistent with a spatial size corresponding to an image obtained by performing feature extraction on an original image, and the upscaling spatial resolution processing is used to restore a number of channels corresponding to the high image reconstruction features to be consistent with a number of channels corresponding to the image obtained by performing feature extraction on the original image.

6. An image coding method characterized by, The image encoding method comprises: performing downscaling spatial resolution processing on image features corresponding to a to-be-encoded image to obtain extended image features; grouping the extended image features to obtain a plurality of extended feature groups; The residual calculation sequence is constructed according to the plurality of extended feature groups; residual calculation is performed on the plurality of extended feature groups respectively based on the residual calculation sequence, and image residual data corresponding to each extended feature group is obtained; wherein after image reconstruction features corresponding to each extended feature group are obtained, the image reconstruction features corresponding to each extended feature group are subjected to enlarged spatial resolution processing to obtain the image residual data corresponding to each extended feature group, and the enlarged spatial resolution processing is the inverse process of the reduced spatial resolution processing; An image code stream is generated according to the image residual data, and the image code stream is sent to an image decoding end.

7. An image decoding apparatus characterized by comprising: The image decoding device comprises: A code stream decoding module is configured to group the extended residual data to obtain a plurality of extended residual groups; A residual recovery module is configured to construct a residual recovery sequence according to the plurality of extended residual groups; residual recovery is performed on the plurality of extended residual groups respectively based on the residual recovery sequence, and image reconstruction features corresponding to each extended residual group are obtained; A data combination module is configured to perform enlarged spatial resolution processing on the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data; An image reconstruction module is configured to perform image reconstruction according to the reconstruction feature data to obtain a reconstructed image block.

8. An image coding apparatus characterized by comprising: The image encoding device comprises: A feature extraction module is configured to perform reduced spatial resolution processing on image features corresponding to a to-be-encoded image to obtain extended image features; A data grouping module is configured to group the extended image features to obtain a plurality of extended feature groups; A residual calculation module is configured to construct a residual calculation sequence according to the plurality of extended feature groups; residual calculation is performed on the plurality of extended feature groups respectively based on the residual calculation sequence, and image residual data corresponding to each extended feature group is obtained; wherein after image reconstruction features corresponding to each extended feature group are obtained, the image reconstruction features corresponding to each extended feature group are subjected to enlarged spatial resolution processing to obtain the image residual data corresponding to each extended feature group, and the enlarged spatial resolution processing is the inverse process of the reduced spatial resolution processing; A code stream generation module is configured to generate an image code stream according to the image residual data, and send the image code stream to an image decoding end.

9. A decoding device, comprising: The decoding device comprises a processor, a memory, and an image decoding program stored on the memory and executable on the processor, and the image decoding program, when executed by the processor, implements the image decoding method of any one of claims 1-5.

10. An encoding device, comprising: The encoding device comprises a processor, a memory, and an image encoding program stored on the memory and executable on the processor, and the image encoding program, when executed by the processor, implements the image encoding method of claim 6.

11. A storage medium, characterized by The storage medium stores an image decoding program and / or an image encoding program, the image decoding program, when executed, implements the image decoding method of any one of claims 1-5, and the image encoding program, when executed, implements the image encoding method of claim 6.

12. A computer program product, characterised in that, comprising computer program instructions configured to, when executed by a processor having memory, implement the image decoding method of any of claims 1-5, or implement the image encoding method of claim 6.

Images