Decoding and coding methods, devices and equipment based on multi-step context prediction

Abstract

The invention belongs to the technical field of image processing, and discloses a decoding and coding method, device and equipment based on multi-step context prediction. The method comprises the following steps: non-uniformly grouping image residual data extracted from an image code stream to obtain at least one extended residual group; performing residual recovery on the at least one extended residual group to obtain image reconstruction features corresponding to each extended residual group; and performing image reconstruction based on the image reconstruction features corresponding to the extended residual groups to obtain reconstructed image blocks. Since the image residual data extracted from the image code stream is firstly grouped, it is ensured that the whole group can be grouped for residual recovery processing, so that the overall residual recovery calculation efficiency is improved, the time complexity is reduced, non-uniform grouping can be performed according to detail information contained in the feature data during grouping, and the accuracy of residual recovery is improved. The method adapts to non-uniformity of feature data distribution, so that the coding performance is improved and the complexity is reduced.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a decoding and encoding method, device and equipment based on multi-step context prediction. Background Art

[0002] In image compression schemes based on deep learning, the current mainstream approach is to use the decoded feature points as prior information to perform mean prediction on the feature points currently being decoded to reduce the spatial redundancy of the image. Mainstream schemes generally use serial or wavefront methods for encoding and decoding, and the degree of serialization will increase with the number of features, resulting in a high overall execution time complexity.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present invention is to provide a decoding, encoding method, device and equipment based on multi-step context prediction, aiming to solve the technical problem of high time complexity of mean prediction in the image encoding and decoding process of the prior art.

[0005] To achieve the above object, the present invention provides a decoding method based on multi-step context prediction, the method comprising the following steps:

[0006] Non-uniformly grouping image residual data extracted from the image code stream to obtain at least one extended residual group;

[0007] Performing residual recovery on the at least one extended residual group respectively to obtain image reconstruction features corresponding to each extended residual group;

[0008] Image reconstruction is performed based on the image reconstruction features corresponding to each extended residual group to obtain a reconstructed image block.

[0009] In a possible implementation manner of the present application, the non-uniform grouping of the image residual data extracted from the image code stream to obtain at least one extended residual group includes:

[0010] Extract image residual data from the image code stream;

[0011] According to the feature channels corresponding to the image residual data, the image residual data are divided into at least one channel division group;

[0012] Perform spatial domain division on the image residual data in the channel division group to obtain at least one extended residual group.

[0013] In a possible implementation manner of the present application, dividing the image residual data into at least one channel division group according to the feature channels corresponding to each image residual data includes:

[0014] Classify according to the corresponding feature channels to obtain the image residual data corresponding to each feature channel;

[0015] Determine at least one data group and a characteristic channel label corresponding to each data group according to the channel division rule;

[0016] Based on the feature channel labels, the image residual data corresponding to each feature channel is added to the at least one data group to obtain at least one channel division group.

[0017] In a possible implementation manner of the present application, the channel division rule is a non-uniform division rule.

[0018] In a possible implementation manner of the present application, performing spatial domain division on the image residual data in the channel division group to obtain at least one extended residual group includes:

[0019] Performing phase division on the image residual data in the channel division group according to the phase division specification to obtain a plurality of characteristic phases corresponding to the channel division group;

[0020] Determine at least one airspace division group and an airspace label corresponding to each airspace division group based on the airspace division rule;

[0021] Spatially labeling the characteristic phases according to the number of the at least one spatial division group;

[0022] The image residual data in the channel division group is added to the at least one spatial domain division group based on the labeled characteristic phase to obtain at least one extended residual group.

[0023] In a possible implementation manner of the present application, the spatial domain division rule is a non-uniform division rule.

[0024] In a possible implementation of the present application, different channel division groups correspond to different airspace division rules.

[0025] In a possible implementation manner of the present application, different channel division groups correspond to different phase division specifications.

[0026] In a possible implementation manner of the present application, performing residual recovery on the at least one extended residual group to obtain image reconstruction features corresponding to each extended residual group includes:

[0027] constructing a residual recovery sequence according to the at least one extended residual group;

[0028] Based on the residual recovery sequence, residual recovery is performed on the at least one extended residual group respectively to obtain image reconstruction features corresponding to each extended residual group.

[0029] In a possible implementation manner of the present application, the step of performing residual recovery on the at least one extended residual group based on the residual recovery sequence to obtain image reconstruction features corresponding to each extended residual group includes:

[0030] Traversing the residual recovery sequence to obtain a current extended residual group;

[0031] Obtain auxiliary information output by the auxiliary encoding network;

[0032] constructing prior information based on the auxiliary information;

[0033] Performing residual recovery on the current extended residual group based on the prior information to obtain image reconstruction features corresponding to the current extended residual group;

[0034] At the end of the traversal, the image reconstruction features corresponding to each extended residual group are obtained.

[0035] In a possible implementation manner of the present application, constructing the prior information according to the auxiliary information includes:

[0036] Performing spatial domain resolution reduction processing on the auxiliary information to obtain extended auxiliary information;

[0037] Detecting whether the current extended residual group is the first element in the residual recovery sequence;

[0038] If it is the first element, constructing the prior information according to the extended auxiliary information;

[0039] If it is not the first element, the extended auxiliary information is spliced ​​with the convolution processing result corresponding to the image reconstruction feature of the restored extended residual group to obtain spliced ​​auxiliary information, and prior information is constructed according to the spliced ​​auxiliary information.

[0040] In a possible implementation manner of the present application, the step of splicing the extended auxiliary information with a convolution processing result corresponding to the image reconstruction feature of the restored extended residual group to obtain the spliced ​​auxiliary information includes:

[0041] Obtaining image reconstruction features corresponding to the restored extended residual group;

[0042] Performing feature enhancement on the image reconstruction feature to obtain an enhanced reconstruction feature;

[0043] The auxiliary information is spliced ​​with the convolution processing result corresponding to the enhanced reconstruction feature to obtain spliced ​​auxiliary information.

[0044] In a possible implementation manner of the present application, the step of performing feature enhancement on the image reconstruction feature to obtain the enhanced reconstruction feature includes:

[0045] Obtaining a predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the restored extended residual group;

[0046] The image reconstruction feature is enhanced according to the predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the restored extended residual group to obtain an enhanced reconstruction feature.

[0047] In a possible implementation manner of the present application, performing image reconstruction based on image reconstruction features corresponding to each extended residual group to obtain a reconstructed image block includes:

[0048] Amplifying the spatial resolution of the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data;

[0049] Image reconstruction is performed according to the reconstruction feature data to obtain a reconstructed image block.

[0050] In addition, to achieve the above object, the present invention also proposes a coding method based on multi-step context prediction, and the coding method based on multi-step context prediction includes:

[0051] Non-uniformly grouping image features corresponding to the image to be encoded to obtain at least one extended feature group;

[0052] Performing residual calculation on the at least one extended feature group respectively to obtain image residual data corresponding to each extended feature group;

[0053] An image code stream is generated according to the image residual data, and the image code stream is sent to an image decoding end.

[0054] In addition, to achieve the above object, the present invention further provides a decoding device, the decoding device comprising:

[0055] A grouping module, used for non-uniformly grouping the image residual data extracted from the image code stream to obtain at least one extended residual group;

[0056] A restoration module, used to perform residual restoration on the at least one extended residual group respectively to obtain image reconstruction features corresponding to each extended residual group;

[0057] The reconstruction module is used to perform image reconstruction based on the image reconstruction features corresponding to each extended residual group to obtain a reconstructed image block.

[0058] In addition, to achieve the above object, the present invention further provides an encoding device, the encoding device comprising:

[0059] A grouping module, used for non-uniformly grouping image features corresponding to the image to be encoded to obtain at least one extended feature group;

[0060] A calculation module, used to perform residual calculation on the at least one extended feature group respectively to obtain image residual data corresponding to each extended feature group;

[0061] The generating module is used to generate an image code stream according to the image residual data, and send the image code stream to the image decoding end.

[0062] In addition, to achieve the above-mentioned purpose, the present invention also proposes a decoding device, which includes: a processor, a memory, and a decoding program stored in the memory and executable on the processor, and when the decoding program is executed by the processor, the steps of the decoding method based on multi-step context prediction as described above are implemented.

[0063] In addition, to achieve the above-mentioned purpose, the present invention also proposes a coding device, which includes: a processor, a memory, and a decoding program and / or an encoding program stored in the memory and run on the processor, wherein the decoding program implements the steps of the decoding method based on multi-step context prediction as described above when executed by the processor, and the encoding program implements the steps of the encoding method based on multi-step context prediction as described above when executed by the processor.

[0064] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a decoding program and / or an encoding program is stored. When the decoding program is executed, the steps of the decoding method based on multi-step context prediction as described above are implemented, and when the encoding program is executed, the steps of the encoding method based on multi-step context prediction as described above are implemented.

[0065] The present invention obtains at least one extended residual group by performing non-uniform grouping of image residual data extracted from an image code stream; performs residual recovery on at least one extended residual group to obtain image reconstruction features corresponding to each extended residual group; and performs image reconstruction based on the image reconstruction features corresponding to each extended residual group to obtain a reconstructed image block. Since the image residual data extracted from the image code stream is first grouped, it is ensured that the residual recovery process can be performed in the group, thereby improving the overall residual recovery calculation efficiency and reducing the time complexity. When grouping, non-uniform grouping can be performed according to the detailed information contained in each feature data, adapting to the uneven distribution of the feature data, thereby improving the encoding performance and reducing the complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a schematic diagram of the structure of an electronic device in a hardware operating environment involved in an embodiment of the present invention;

[0067] Figure 2 It is a flowchart of a first embodiment of a decoding method based on multi-step context prediction of the present invention;

[0068] Figure 3 This is an overall framework diagram of image compression according to an embodiment of the present invention;

[0069] Figure 4 It is a flowchart diagram of a second embodiment of a decoding method based on multi-step context prediction of the present invention;

[0070] Figure 5 It is a flowchart diagram of a third embodiment of a decoding method based on multi-step context prediction according to the present invention;

[0071] Figure 6 It is a flowchart diagram of a fourth embodiment of a decoding method based on multi-step context prediction according to the present invention;

[0072] Figure 7 It is a flowchart diagram of a fifth embodiment of a decoding method based on multi-step context prediction according to the present invention;

[0073] Figure 8 This is a schematic diagram of channel non-uniform & spatial domain uniform grouping according to an embodiment of the present invention;

[0074] Fig. 9 This is a schematic diagram of channel uniform & spatial domain non-uniform grouping according to an embodiment of the present invention;

[0075] Fig.10 A schematic diagram of spatial non-uniform grouping of a 3×3 phase division specification according to an embodiment of the present invention;

[0076] Fig.11 A schematic diagram of a multi-network residual reconstruction structure according to an embodiment of the present invention;

[0077] Fig.12 It is a schematic flow chart of a first embodiment of a coding method based on multi-step context prediction according to the present invention;

[0078] Fig.13 is a structural block diagram of a first embodiment of a decoding device of the present invention;

[0079] Fig.14 FIG. 4 is a structural block diagram of a first embodiment of an encoding device according to the present invention.

[0080] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0081] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0082] Reference Figure 1 , Figure 1 The present invention is a schematic diagram of the structure of a decoding device or encoding device in a hardware operating environment involved in an embodiment of the present invention.

[0083] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0084] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0085] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a decoding program and / or an encoding program.

[0086] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in a decoding device or an encoding device, and the electronic device calls the decoding program or the encoding program stored in the memory 1005 through the processor 1001, and executes the decoding method based on multi-step context prediction or the encoding method based on multi-step context prediction provided in an embodiment of the present invention.

[0087] The embodiment of the present invention provides a decoding method based on multi-step context prediction, referring to Figure 2 , Figure 2 The figure is a flowchart of a first embodiment of a decoding method based on multi-step context prediction according to the present invention.

[0088] In this embodiment, the decoding method based on multi-step context prediction includes the following steps:

[0089] Step S10: non-uniformly grouping the image residual data extracted from the image code stream to obtain at least one extended residual group;

[0090] It should be noted that the executor of this embodiment can be a decoding device when encoding image data. The decoding device can be an electronic device such as a personal computer or a server. Of course, it can also be other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment and the following embodiments, the decoding method based on multi-step context prediction of the present invention is explained by taking a decoding device as an example.

[0091] Among them, since during the image encoding process, the encoding device generally decodes the encoded image code stream after the encoding is completed, and determines whether the parameters used in the encoding need to be adjusted according to the decoded image quality, the executor of this embodiment can also be the encoding device.

[0092] It should be noted that the image code stream can be a code stream generated by the encoding device after encoding the image data to be compressed and encoded. When generating the image code stream, the encoding device groups the image features, and then performs mean prediction on each group, and calculates the feature residual based on the predicted mean. Finally, the encoding device directly encodes the generated image residual data into the image code stream. At this time, the decoding device can directly extract the image residual data from the image code stream. At this time, the decoding device can divide the image residual data into at least one extended residual group in the same grouping method as the encoding device, and then perform residual recovery on each group, thereby reducing the time complexity of mean prediction in the image decoding process.

[0093] It should be noted that in the process of mean prediction, the feature data used may have a small number of channels used to represent low-frequency information, a part of the channels used to represent high-frequency detail information, and the remaining channels are all 0 and do not contain any information. This non-uniformity leads to the fact that the optimal coding performance and complexity cannot be achieved if uniform feature channels and spatial domain division are used. Therefore, the encoding device can use non-uniform grouping when grouping image features. Similarly, the encoding device can use non-uniform grouping when non-uniformly grouping image residual data. Among them, non-uniform grouping means that when grouping based on feature channels and spatial domains, non-uniform grouping is used for at least one of the feature channels and spatial domains, for example: uniform grouping is used for feature channels, but non-uniform grouping is used for spatial domains; uniform grouping is used for feature channels, but non-uniform grouping is used for spatial domains; non-uniform grouping is used for both feature channels and spatial domains.

[0094] The technical terms involved in encoding or decoding images include: JPEG (Joint Photographic Experts Group), JPEG-AI (Joint Photographic Experts Group Artificial Intelligence), entropy coding (Entropy Encoding), neural network (Neural Network, NN), convolutional neural network (Convolutional Neural Network, CNN), feature (feature), rate-distortion principle (Rate-Distortion Optimized), etc., which are explained here.

[0095] Among them, JPEG (Joint Photographic Experts Group) is a standard for continuous tone static image compression. The file suffix is ​​.jpg or .jpeg, and it is the most commonly used image file format. It mainly adopts a joint coding method of predictive coding (DPCM), discrete cosine transform (DCT) and entropy coding to remove redundant image and color data. It is a lossy compression format. It can compress images in a very small storage space, which will cause damage to image data to a certain extent. In particular, using too high a compression ratio will reduce the quality of the image restored after decompression. If you pursue high-quality images, it is not advisable to use too high a compression ratio.

[0096] The scope of JPEG AI is to create a learning-based image coding standard that provides a single-stream, compact compressed domain representation that targets both human visualization, significantly improves compression efficiency over commonly used image coding standards at equivalent subjective quality, and effective performance for image processing and computer vision tasks. JPEG AI targets a wide range of applications such as cloud storage, visual surveillance, autonomous vehicles and devices, image acquisition storage and management, real-time monitoring of visual data, and media distribution. The goal is to design a coding solution that needs to significantly improve the compression efficiency of commonly used coding standards at the same subjective quality, and provide efficient compressed domain processing for machine learning-based image processing and computer vision tasks. Other key requirements include hardware / software implementation-friendly encoding and decoding, support for 8-bit and 10-bit depths, efficient encoding of images with text and graphics, and progressive decoding.

[0097] Entropy coding is a coding process that does not lose any information according to the entropy principle. Information entropy is the average amount of information in the source (a measure of uncertainty). Common entropy coding methods include Shannon coding, Huffman coding, and arithmetic coding.

[0098] The neural network in this application refers to an artificial neural network, not a biological neural network. A neural network is a computational model consisting of a large number of nodes (or neurons) connected to each other. In an artificial neural network, a neuron processing unit can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in the network are divided into three categories: input units, output units, and hidden units. The input unit receives signals and data from the outside world; the output unit realizes the output of the system processing results; the hidden unit is a unit between the input and output units that cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units, and the representation and processing of information are reflected in the connection relationship of the network processing units. Artificial neural network is a non-programmed, brain-like information processing method. Its essence is to obtain a parallel distributed information processing function through the transformation and dynamic behavior of the network, and imitate the information processing function of the human brain nervous system to varying degrees and levels. At present, in the field of video processing, commonly used neural networks include convolutional neural networks (CNN), recurrent neural networks (RNN), fully connected networks, etc.

[0099] Convolutional neural network is a feedforward neural network and one of the most representative network structures in deep learning technology. Its artificial neurons can respond to surrounding units within a certain coverage area and have excellent performance in large-scale image processing. Generally, the basic structure of CNN includes two layers. One is the feature extraction layer (also called convolution layer). The input of each neuron is connected to the local receptive field of the previous layer and extracts the local features. Once the local feature is extracted, its positional relationship with other features is also determined; the second is the feature mapping layer (also called activation layer). Each computing layer of the network consists of multiple feature maps. Each feature map is a plane, and the weights of all neurons on the plane are equal. The feature mapping structure can use Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, GDN function, etc. as the activation function of the convolutional network. In addition, since neurons on a mapping surface share weights, the number of free parameters of the network is reduced. One of the advantages of CNN compared to traditional image processing algorithms is that it avoids the complex pre-processing process of the image (extracting artificial features, etc.), and can directly input the original image for end-to-end learning. One of the advantages of CNN over traditional neural networks is that traditional neural networks are fully connected, that is, all neurons from the input layer to the hidden layer are connected. This will result in a huge number of parameters, making network training time-consuming or even difficult to train. CNN avoids this difficulty through local connections, weight sharing and other methods.

[0100] The feature involved in this application is a three-dimensional feature matrix of CxWxH. C represents the number of channels, H represents the feature height, and W represents the feature width. The feature matrix can be the input or output of the neural network.

[0101] The indicators for evaluating coding efficiency include: bit rate, PSNR, MS-SSIM, VMAF FSIM, PSNRHVS and other indicators. Of course, there are more indicators, which are not limited here. The smaller the bit stream, the greater the compression rate; the larger the PSNR, the better the image coding efficiency. When selecting a mode, the discriminant formula is essentially a comprehensive evaluation of the two. The cost corresponding to the mode: J(mode) = D + λ*R. Among them, D represents Distortion, which is usually measured by the MSE indicator. SSE refers to the mean square sum of the differences between the reconstructed block and the source image; λ is the Lagrange multiplier; R is the actual number of bits required for encoding the image block in this mode, including the sum of bits required for encoding mode information, residuals, etc.

[0102] Step S20: performing residual recovery on the at least one extended residual group respectively to obtain image reconstruction features corresponding to each extended residual group.

[0103] It should be noted that performing residual recovery on the extended residual group to obtain the image reconstruction features corresponding to the extended residual group can be to perform mean prediction on the extended residual group, and then add the means obtained by predicting the residual data domain in the extended residual group to obtain the image reconstruction features corresponding to the extended residual group.

[0104] Step S30: performing image reconstruction based on the image reconstruction features corresponding to each extended residual group to obtain a reconstructed image block.

[0105] It can be understood that, based on the image reconstruction features corresponding to each extended residual group, image reconstruction is performed to obtain the reconstructed image block, which can be achieved by performing a synthetic transformation process on the image reconstruction features corresponding to each extended residual group through a pre-constructed synthetic transformation network, thereby obtaining the reconstructed image block. The synthetic transformation network can be a network constructed based on deep learning or a neural network.

[0106] In a possible implementation of this embodiment, in order to ensure that image reconstruction can be performed normally, step S30 in this embodiment may include:

[0107] Amplifying the spatial resolution of the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data;

[0108] Image reconstruction is performed according to the reconstruction feature data to obtain a reconstructed image block.

[0109] It should be noted that since the extended residual group is obtained by non-uniform grouping of the image residual data extracted from the image code stream, and the non-uniform grouping is based on feature channels and spatial domains, in this process, the spatial size and number of channels of the data may be different from the spatial size and number of channels of the image features obtained by feature extraction of the original image. Therefore, when reconstructing the image, the image reconstruction features corresponding to each extended residual group can be first enlarged in spatial domain resolution, and the spatial size and number of channels corresponding to the image reconstruction features can be restored to be consistent with the image features obtained by feature extraction of the original image, thereby obtaining reconstructed feature data.

[0110] Before enlarging the spatial resolution of the image reconstruction features corresponding to each extended residual group to obtain the reconstructed feature data, the image reconstruction features corresponding to each extended residual group may be enhanced first, and then the enhanced reconstruction features corresponding to each extended residual group are enlarged in spatial resolution to obtain the reconstructed feature data. This can ensure that the reliability of the reconstructed feature data finally constructed is higher, and the quality of the reconstructed image blocks obtained by subsequent image reconstruction will be better.

[0111] In actual use, the image reconstruction features corresponding to the extended residual group are feature enhanced. To obtain the enhanced reconstruction features corresponding to the extended residual group, the image reconstruction features corresponding to the extended residual group can be feature enhanced using the predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the extended residual group.

[0112] For ease of understanding, now combined Figure 3 This is for illustration only, but not for limitation. Figure 3 This is the overall framework diagram of image compression.

[0113] like Figure 3 As shown, in the framework, x and x_hat represent the input image (block) and the output reconstructed image (block), respectively. The image can be divided into one image block or multiple image blocks. The following encoding and decoding process for the image block can also be directly applied to the image. y represents the potential feature (feature of the latent domain) obtained after the input picture block is processed by the analysis transformation network. y then passes through the hyperparameter encoding network to obtain the hyperparameter latent feature information z, and z is quantized and encoded and written into the first bitstream (Bitstream#1). The encoding / decoding process of the first bitstream generally uses an algorithm encoding method with a fixed probability density model. Based on the decoding of the first bitstream, the z_hat feature is obtained, and z_hat passes through a probability superparameter decoding network to obtain the probability p required for decoding the second bitstream. In addition, based on the first bitstream, in addition to obtaining the decoding probability of the second bitstream, the first feature can also be jointly reconstructed to generate the predicted value mu of the first feature, and then the feature residual r is obtained. After quantization, the quantized feature residual r_q is obtained, and then the probability p is used for entropy encoding and encoded into the second bitstream. Based on the decoding of the second bitstream, the quantized feature residual r_q can be obtained, and then the reconstructed residual r_hat can be obtained through the residual recovery process. Based on mu and r_hat, the reconstructed first eigenvalue y_hat can be obtained. After the synthesis transformation network, y_hat can generate a reconstructed image block x_hat.

[0114] This embodiment obtains at least one extended residual group by non-uniformly grouping the image residual data extracted from the image code stream; performs residual recovery on the at least one extended residual group to obtain the image reconstruction features corresponding to each extended residual group; and performs image reconstruction based on the image reconstruction features corresponding to each extended residual group to obtain a reconstructed image block. Since the image residual data extracted from the image code stream will be grouped first, it is ensured that the residual recovery process can be performed in the group, thereby improving the overall residual recovery calculation efficiency and reducing the time complexity. When grouping, non-uniform grouping can be performed according to the detailed information contained in each feature data, adapting to the uneven distribution of feature data, thereby improving encoding performance and reducing complexity.

[0115] refer to Figure 4 , Figure 4 The figure is a flow chart of a second embodiment of a decoding method based on multi-step context prediction according to the present invention.

[0116] Based on the above first embodiment, step S10 of the decoding method based on multi-step context prediction in this embodiment includes:

[0117] Step S101: extracting image residual data from an image code stream.

[0118] It should be noted that extracting the image residual data from the image code stream may be performed by parsing the image code stream to obtain the image residual data.

[0119] Step S102: dividing the image residual data into at least one channel division group according to the feature channels corresponding to each image residual data.

[0120] It should be noted that, according to the feature channels corresponding to each image residual data, dividing the image residual data into at least one channel division group may be to group the image residual data corresponding to the feature channels marked as belonging to the same group into the same group, thereby obtaining at least one channel division group.

[0121] Among them, when the image residual data is divided into at least one channel division group based on the feature channel, when the data is divided into multiple channel division groups, the division can be performed in a non-uniform manner, that is, there is at least one channel division group in which the number of corresponding feature channels is inconsistent with that of other channel division groups.

[0122] Step S103: performing spatial domain division on the image residual data in the channel division group to obtain at least one extended residual group.

[0123] It should be noted that performing spatial domain division on the image residual data in the channel division group to obtain at least one extended residual group can be to group the image residual data marked as belonging to the same group in the channel division group into the same group, thereby obtaining the extended residual groups corresponding to each channel division group.

[0124] Among them, when performing spatial domain division on the image residual data in the channel division group to obtain at least one extended residual group, the spatial domain division can be performed in a non-uniform manner, that is, at least one extended residual group is obtained in which there is at least one extended residual group whose number of corresponding spatial domain positions is inconsistent with that of other extended residual groups.

[0125] This embodiment extracts image residual data from an image code stream; divides the image residual data into at least one channel division group according to the characteristic channels corresponding to each image residual data; and performs spatial domain division on the image residual data in the channel division group to obtain at least one extended residual group. Since the image residual data is grouped based on the characteristic channels first and then based on the spatial domain, it is ensured that the non-uniform grouping method can be adjusted according to actual needs, and it is ensured that the non-uniformity of the characteristic data distribution can be adapted.

[0126] refer to Figure 5 , Figure 5 The figure is a flowchart of a third embodiment of a decoding method based on multi-step context prediction according to the present invention.

[0127] Based on the above second embodiment, step S102 of the decoding method based on multi-step context prediction in this embodiment includes:

[0128] Step S1021: Classify according to the corresponding feature channels to obtain image residual data corresponding to each feature channel.

[0129] It should be noted that, in order to facilitate grouping based on feature channels, the image residual data can be classified according to the corresponding feature channels, and the image residual data corresponding to the same feature channel can be classified into the same category, so as to obtain the image residual data corresponding to each feature channel. Among them, a corresponding label can be set for each feature channel, so that different feature channels have different corresponding labels.

[0130] Step S1022: Determine at least one data group and a characteristic channel label corresponding to each data group according to the channel division rule.

[0131] It should be noted that the channel division rule may be set in advance by the administrator of the decoding device or the encoding device.

[0132] In actual use, determining at least one data group and the characteristic channel label corresponding to each data group according to the channel division rule may be to parse the channel division rule to obtain the number of groups and the set of characteristic channel numbers, and determining at least one data group according to the channel division rule may be to create data groups whose number is consistent with the number of groups, thereby obtaining at least one data group. Determining the characteristic channel label corresponding to each data group may be to set the corresponding characteristic channel label for each created data group according to the set of characteristic channel numbers.

[0133] Among them, the channel division rule can be a non-uniform division rule, that is, the number of feature channels corresponding to each channel division group is not exactly the same (that is, there is at least one channel division group, and the number of feature channels corresponding to it is inconsistent with the number of feature channels corresponding to other channel division groups). For example: assuming that the number of feature channels is 4, and the labels of each feature channel are 1, 2, 3, and 4 respectively, the channel division rule is set to the corresponding number of groups is 2, and the corresponding number of feature channels is 1 and 3 respectively. Then, two data groups A and B can be created at this time. The feature channel label corresponding to A is 1, and the feature channel label corresponding to B is 2, 3, and 4.

[0134] Step S1023: adding the image residual data corresponding to each feature channel to the at least one data group based on the feature channel label to obtain at least one channel division group.

[0135] It should be noted that, based on the feature channel label, the image residual data corresponding to each feature channel is added to the at least one data group, and obtaining at least one channel division group can be performed by matching the label corresponding to each feature channel with the feature channel label corresponding to each data group, and adding the image residual data corresponding to the successfully matched feature channel to the corresponding data group, thereby obtaining at least one channel division group.

[0136] In this embodiment, image residual data corresponding to each feature channel is obtained by classifying according to the corresponding feature channel; at least one data group and the feature channel label corresponding to each data group are determined according to the channel division rule; and the image residual data corresponding to each feature channel is added to the at least one data group based on the feature channel label to obtain at least one channel division group. Since the image residual data is first classified according to the corresponding feature channel before grouping based on the feature channel, it is ensured that batch addition can be performed directly during grouping, thereby improving the grouping efficiency.

[0137] refer to Figure 6 , Figure 6 The figure is a flowchart diagram of a fourth embodiment of a decoding method based on multi-step context prediction according to the present invention.

[0138] Based on the above second embodiment, step S103 of the decoding method based on multi-step context prediction in this embodiment includes:

[0139] Step S1031: performing phase division on the image residual data in the channel division group according to the phase division specification to obtain a plurality of characteristic phases corresponding to the channel division group.

[0140] It should be noted that the phase division specification may be the size of the divided phase, for example: 2×2 or 3×3 or more. Each channel division group may adopt the same phase division specification.

[0141] In actual use, the image residual data in the channel division group is phase-divided according to the phase division specification. To obtain multiple characteristic phases corresponding to the channel division group, the image residual data in the channel division group can be split into sub-blocks of 1X1 specification, and then the divided sub-blocks are divided into multiple characteristic phases whose sizes are consistent with the phase division specification.

[0142] In a possible implementation of this embodiment, different channel division groups may correspond to different phase division specifications. For example, assuming that there are two channel division groups, namely Y1 and Y2, the phase division specification corresponding to Y1 may be 2X2, and the phase division specification corresponding to Y2 may be 3X3.

[0143] Step S1032: Determine at least one airspace division group and the airspace label corresponding to each airspace division group based on the airspace division rule.

[0144] It should be noted that the spatial division rule may include the number of airspace pre-rentals of the total divided spatial division groups and the phase position corresponding to each spatial division group. The spatial division rule may be pre-set by the manager of the image decoding device or the image encoding device according to the actual position of the feature information. The spatial division rule may be pre-stored locally in the decoding device or obtained by parsing the image code stream. This embodiment does not limit this. Among them, the spatial division rule may be a non-uniform division rule, that is, the number of phase positions corresponding to each spatial division group is not exactly the same (that is, there is at least one spatial division group, and the number of phase positions corresponding to it is inconsistent with other spatial division groups). For example: assuming that the phase division specification is 2X2, each single phase will include 4 1X1 sub-blocks, and the phase positions are 4 in total. At this time, if the divided spatial division groups are 3, the number of phase positions corresponding to them can be set to 1, 1, and 2 respectively.

[0145] In actual use, determining at least one airspace division group and the airspace label corresponding to each airspace division group based on the airspace division rule can be to extract the number of airspace groups from the airspace division rule, generate airspace division groups whose number is consistent with the number of airspace groups, obtain at least one airspace division group, and set a corresponding airspace label for each airspace division group starting from the initial label. For example: assuming that the number of airspace groups extracted from the airspace division rule is 3, then 3 airspace division groups can be created at this time, and airspace labels (such as 1, 2, 3) can be set for them respectively.

[0146] Step S1033: Spatially label the characteristic phases according to the number of groups of the at least one spatial division group.

[0147] It should be noted that spatially labeling the characteristic phase according to the number of groups of at least one spatial division group may be setting a corresponding label for each sub-block in the characteristic phase according to the number of groups of at least one spatial division group.

[0148] In actual use, when setting the labels, the labels can be made according to the group position mapping relationship in the spatial division rule, wherein the group position mapping relationship may include phase positions in the phase that are divided uniformly for dividing the groups.

[0149] Step S1034: adding the image residual data in the channel division group to the at least one spatial domain division group based on the labeled characteristic phase to obtain at least one extended residual group.

[0150] It should be noted that, based on the labeled characteristic phase, the image residual data in the channel division group is added to at least one spatial domain division group, and at least one extended residual group is obtained by matching the labels corresponding to each sub-block in the characteristic phase with the spatial domain labels corresponding to each extended residual group, thereby determining the corresponding spatial domain division group to add each image residual data in the channel division group, and adding it to the corresponding spatial domain division group, thereby obtaining at least one extended residual group.

[0151] This embodiment performs phase division on the image residual data in the channel division group according to the phase division specification to obtain multiple characteristic phases corresponding to the channel division group; determines at least one spatial division group and the spatial labels corresponding to each spatial division group based on the spatial division rule; spatially labels the characteristic phases according to the number of groups of the at least one spatial division group; and adds the image residual data in the channel division group to the at least one spatial division group based on the labeled characteristic phases to obtain at least one extended residual group. Since the spatial domain labels are performed on the split characteristic phases according to the number of groups of at least one spatial division group when performing spatial domain grouping, it can support more different types of grouping numbers and phase division specifications.

[0152] refer to Figure 7 , Figure 7 The figure is a flowchart of a fifth embodiment of a decoding method based on multi-step context prediction according to the present invention.

[0153] Based on the above first embodiment, step S20 of the decoding method based on multi-step context prediction in this embodiment includes:

[0154] Step S201: construct a residual recovery sequence according to the at least one extended residual group.

[0155] It should be noted that after at least one extended residual group is divided, residual recovery can be performed in groups, thereby reducing the time complexity of mean prediction in the image decoding process. At this time, in order to determine the residual recovery order of each extended residual group, a residual recovery sequence can be constructed according to at least one extended residual group.

[0156] Step S202: performing residual recovery on the at least one extended residual group based on the residual recovery sequence to obtain image reconstruction features corresponding to each extended residual group.

[0157] It should be noted that performing residual recovery on at least one extended residual group based on the residual recovery sequence may be performing residual recovery on at least one extended residual group in sequence based on a sequence order in the residual recovery sequence.

[0158] In actual use, residual recovery may be performed sequentially by sequence traversal. In this case, step S202 of this embodiment may include:

[0159] Traversing the residual recovery sequence to obtain a current extended residual group;

[0160] Obtain auxiliary information output by the auxiliary encoding network;

[0161] constructing prior information based on the auxiliary information;

[0162] Performing residual recovery on the current extended residual group based on the prior information to obtain image reconstruction features corresponding to the current extended residual group;

[0163] At the end of the traversal, the image reconstruction features corresponding to each extended residual group are obtained.

[0164] It should be noted that traversing the residual recovery sequence to obtain the current extended residual group may be traversing the residual recovery sequence and taking the traversed extended residual group as the current extended residual group. Figure 3 The auxiliary network (AD&IQ) shown in .

[0165] In a possible implementation of the present embodiment, residual recovery is performed on the current extended residual group based on prior information to obtain image reconstruction features corresponding to the current extended residual group. The prior information can be processed by a convolution kernel to obtain features, and then the auxiliary information output by the auxiliary network is channel-spliced ​​with the obtained features. After splicing, the auxiliary information is input into the Prediction Fusion Net to obtain a predicted mean, and then the predicted mean is added to the residual in the current extended residual group to achieve residual recovery and obtain image reconstruction features corresponding to the current extended residual group.

[0166] For example, for the current extended residual group with sequence number i in the residual recovery sequence, the mean mu_i can be obtained through the mean prediction network based on the auxiliary information z_hat and the features y_hat_1 to y_hat_i-1 reconstructed in the previous stage. And the residual of the corresponding stage in r_hat is obtained and recorded as r_hat_i, and feature recovery is performed to obtain the image reconstruction feature y_hat_i corresponding to the current extended residual group.

[0167] In a possible implementation of this embodiment, after the extended residual group is grouped, residual recovery is not actually performed directly based on the data in the current extended residual group, but the residual corresponding to the current extended residual group is extracted from the bitstream according to the predicted parameters. At this time, residual recovery is performed on the current extended residual group based on the prior information to obtain the image reconstruction features corresponding to the current extended residual group. The prior information can also be processed by a convolution kernel to obtain the features, and then the auxiliary information output by the auxiliary network is channel-joined with the obtained features, and after the splicing, it is input into the Prediction Fusion Net to obtain the predicted mean and modeling parameters, and then the quantized residual data corresponding to the current extended residual group is extracted from the image bitstream according to the modeling parameters, and the residual data is dequantized to obtain the residual data, and the residual data is added to the predicted mean to obtain the quantized residual data corresponding to the current extended residual group.

[0168] For example, for the current extended residual group with sequence number i in the residual recovery sequence, the predicted mean mu_i and the modeling value sigma_i can be obtained through the mean prediction network based on the auxiliary information z_hat and the features y_hat_1 to y_hat_i-1 reconstructed in the previous stage. The quantized residual of the current extended residual group is obtained by decoding from the image bitstream according to sigma_i, and the inverse quantization is recorded as r_hat_i, and the mean mu_i is added for feature recovery to obtain the image reconstruction feature y_hat_i corresponding to the current extended residual group.

[0169] In actual use, the step of constructing the prior information according to the auxiliary information in this embodiment may include:

[0170] Performing spatial domain resolution reduction processing on the auxiliary information to obtain extended auxiliary information;

[0171] Detecting whether the current extended residual group is the first element in the residual recovery sequence;

[0172] If it is the first element, constructing the prior information according to the extended auxiliary information;

[0173] If it is not the first element, the extended auxiliary information is spliced ​​with the convolution processing result corresponding to the image reconstruction feature of the restored extended residual group to obtain spliced ​​auxiliary information, and prior information is constructed according to the spliced ​​auxiliary information.

[0174] It should be noted that the spatial size and number of feature channels of the auxiliary information output by the auxiliary coding network are actually consistent with those of the original image. However, due to the non-uniform grouping of the image residual data, the image residual data may have been processed by reducing the spatial resolution. Therefore, in order to ensure smooth channel stitching, the auxiliary information needs to be processed by reducing the spatial resolution first to obtain the extended auxiliary information, and then the prior information is constructed based on the extended auxiliary information. The spatial resolution reduction processing of the auxiliary information can be carried out in the same way as the spatial resolution reduction processing caused by the non-uniform grouping of the image residual data.

[0175] When constructing prior information based on the extended auxiliary information, in order to increase the accuracy of the mean prediction, when constructing the prior information, the image features corresponding to the previously reconstructed extended residual group can also be combined to construct the prior information. If the current extended residual group is the first element in the residual recovery sequence, it means that the extended residual group is the first extended residual group to perform residual recovery. At this time, there is no reconstructed extended residual group. Therefore, the prior information can be directly constructed based on the extended auxiliary information.

[0176] If the current extended residual group is not the first element in the residual recovery sequence, then there is an extended residual group that has been reconstructed. Therefore, the image reconstruction features of the restored extended residual group can be convolved through the convolution layer, and then the extended auxiliary information and the convolution processing results corresponding to the image reconstruction features of the restored extended residual group are channel-joined, and then the prior information is constructed based on the stitched auxiliary information. When constructing the prior information and selecting the restored extended residual group, all the restored extended residual groups can be selected, or only a part of the restored extended residual groups can be selected.

[0177] In a possible implementation of this embodiment, when performing result splicing, the image reconstruction features of the restored extended residual group may be enhanced to further improve the prediction effect. The step of splicing the extended auxiliary information with the convolution processing results corresponding to the image reconstruction features of the restored extended residual group to obtain the splicing auxiliary information may include:

[0178] Obtaining image reconstruction features corresponding to the restored extended residual group;

[0179] Performing feature enhancement on the image reconstruction feature to obtain an enhanced reconstruction feature;

[0180] The auxiliary information is spliced ​​with the convolution processing result corresponding to the enhanced reconstruction feature to obtain spliced ​​auxiliary information.

[0181] In actual use, the image reconstruction features are enhanced to obtain the enhanced reconstruction features by performing operations such as missing value processing and outlier processing on the image reconstruction features.

[0182] It can be understood that before splicing the auxiliary information with the image reconstruction features, the image reconstruction features are first enhanced to obtain enhanced reconstruction features, which can increase the reliability of the enhanced reconstruction features, thereby improving the reliability of the prior information, making the mean prediction based on the prior information more accurate.

[0183] In actual use, when the image reconstruction features corresponding to the restored extended residual group are enhanced, the predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the restored extended residual group may be used. At this time, the step of enhancing the image reconstruction features to obtain enhanced reconstruction features described in this embodiment may include:

[0184] Obtaining a predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the restored extended residual group;

[0185] The image reconstruction feature is enhanced according to the predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the restored extended residual group to obtain an enhanced reconstruction feature.

[0186] It should be noted that the predicted mean corresponding to the restored extended residual group may be a value obtained by performing mean prediction when performing residual recovery on the restored extended residual group. The residual data variance may be a variance value of the image residual data corresponding to the restored extended residual group.

[0187] For ease of understanding, now combined Figure 8 , 9 , 10 and 11 for explanation. Figure 8 This is a schematic diagram of channel non-uniform & spatial uniform grouping, such as Figure 8As shown, feature y contains a total of 4 feature channels. When grouping feature y, it is first grouped based on the feature channel and divided into two channel division groups y1 and y2 in a non-uniform manner, wherein y1 contains image residual data corresponding to one feature channel and y2 contains image residual data corresponding to three feature channels. Thereafter, spatial domain uniform division is performed based on the 2X2 phase division specification (the spatial domain label of each phase is calibrated by different colors), and y1 corresponds to 4 spatial domain division groups and y2 corresponds to 4 spatial domain division groups. Finally, a total of 8 extended residual groups are obtained. Afterwards, a residual recovery sequence is constructed based on the divided extended residual groups, and residual recovery is performed according to the residual recovery sequence to obtain image reconstruction features corresponding to each extended residual group. The spatial domain resolution is enlarged and spatial domain reset is performed to obtain image reconstruction features y1_hat corresponding to y1 and image reconstruction features y2_hat corresponding to y2. Finally, y1_hat and y2_hat can be aggregated according to the feature channels, and image reconstruction can be performed according to the aggregated features.

[0188] Fig. 9 This is a schematic diagram of channel uniform & spatial domain non-uniform grouping. At this time, feature y contains a total of 4 feature channels. When grouping feature y, it is now uniformly grouped based on the feature channels and divided into two channel groups y1 and y2. y1 and y2 contain image residual data corresponding to two feature channels, respectively. For example, Fig. 9 As shown, at this time, for y1 or y2, the spatial domain is non-uniformly divided with a phase division specification of 2X2. At this time, it can be divided into three spatial domain division groups: Y_1, Y_2, and Y_3. Among them, the spatial domain position corresponding to Y_3 is more than the spatial domain positions of Y_1 and Y_2. After that, a residual recovery sequence will be constructed according to Y_1, Y_2, and Y_3, and residual recovery will be performed based on the residual recovery sequence to obtain the image reconstruction features Y_hat_1, Y_hat_2, and Y_hat_3 corresponding to Y_1, Y_2, and Y_3, respectively.

[0189] Fig.10 FIG. 1 is a schematic diagram of spatial non-uniform grouping based on the 3X3 phase division specification. Fig.10 As shown in the figure, 3X3 phase is introduced for phase division, and the 6*6 specification features are divided into 1*1 specification sub-blocks. Then, phase division is performed with 3X3 phase to obtain 4 3X3 phases. Each sub-block in the phase is spatially labeled (labeled 1, 2, and 3 respectively). Then, the image data is added to three spatial division groups (Y_1, Y_2, and Y_3 respectively) according to the spatial labels. Then, a residual recovery sequence is constructed according to Y_1, Y_2, and Y_3. Residual recovery is performed according to the residual recovery sequence to obtain image reconstruction features Y_hat_1, Y_hat_2, and Y_hat_3 corresponding to Y_1, Y_2, and Y_3 respectively.

[0190] In the implementation process, the spatial domain division rules corresponding to each channel division group may be different (that is, the number of spatial domain division groups divided when performing spatial domain division for different channel division groups and the phase position corresponding to each group may be different). In this case, the network structure required for residual recovery for the extended residual groups corresponding to different channel division groups and the rules for performing spatial domain division within the network may also be different. In order to ensure the normal reconstruction, different networks can be used for reconstruction, such as Fig.11 As shown, Fig.11 It is a schematic diagram of the multi-network residual reconstruction structure. There are two networks used for residual reconstruction, namely Spatial4Net-1 network and Spatial2Net-2 network. Among them, Spatial4Net-1 network can be used for residual recovery of the extended residual group corresponding to y1, and Spatial2Net-2 network can be used for residual recovery of the extended residual group corresponding to y2.

[0191] In this embodiment, a residual recovery sequence is constructed according to the at least one extended residual group; and residual recovery is performed on the at least one extended residual group based on the residual recovery sequence to obtain image reconstruction features corresponding to each extended residual group. Since the residual recovery sequence is constructed according to the at least one extended residual group, the order of residual recovery can be determined by the residual recovery sequence, so that it is possible to quickly determine whether there is a restored extended residual group, and when there is a restored extended residual group, more accurate prior information is constructed according to the image feature data corresponding to the restored extended residual group.

[0192] The embodiment of the present invention provides a coding method based on multi-step context prediction, referring to Fig.12 , Fig.12 The figure is a flowchart of a first embodiment of a coding method based on multi-step context prediction according to the present invention.

[0193] In this embodiment, the encoding method based on multi-step context prediction includes the following steps:

[0194] Step S100: non-uniformly grouping image features corresponding to the image to be encoded to obtain at least one extended feature group.

[0195] When generating an image code stream, the encoding device groups the image features, then performs mean prediction on each group, and calculates the feature residual based on the predicted mean to reduce the time complexity of predicting the mean during the encoding process. Finally, the encoding device directly encodes the generated image residual data into the image code stream. Where necessary, the encoding device can also write the grouping rules used in the non-uniform grouping process into the image code stream to facilitate the decoding device to obtain it. The method adopted by the encoding device when performing non-uniform grouping is consistent with the method adopted by the above-mentioned decoding method. For specific implementation, please refer to the various embodiments of the above-mentioned decoding method based on multi-step context prediction, which will not be repeated here.

[0196] Step S200: performing residual calculation on the at least one extended feature group respectively to obtain image residual data corresponding to each extended feature group.

[0197] It should be noted that residual calculation is performed on the extended feature groups respectively to obtain image residual data corresponding to the extended feature groups. The extended feature groups can be predicted by mean, and then the image features in the extended feature groups are subtracted from the predicted mean to obtain image residual data corresponding to the extended feature groups.

[0198] Among them, when the encoding device performs residual calculation, the process used is the inverse process of the above-mentioned decoding device when performing residual recovery. The specific implementation method can refer to the decoding method based on multi-step context prediction provided in any of the above embodiments, which will not be repeated here.

[0199] Step S300: Generate an image code stream according to the image residual data, and send the image code stream to an image decoding end.

[0200] It should be noted that generating the image code stream according to the image residual data may be writing the image residual data into the image code stream through entropy coding.

[0201] In actual use, image coding can also be written into the image code stream in segments. For example, the original image x is generated through the corresponding analysis transformation network, and y is subjected to feature separation to obtain N stages of features to be encoded y_i (i=1...N) (i.e., at least one extended feature group).

[0202] Then the coding order of the decision components is determined, taking the Y component as an example (generally Y is encoded first):

[0203] 1. For the first stage feature y_1, y is passed through the hyperparameter coding network to obtain z, quantize z to z_hat, and encode it into bitstream 1. Decode based on bitstream 1 to obtain z_hat, and based on z_hat (the first stage cannot use features from other stages), obtain the mean mu_1 through the mean prediction network. Based on mu_1, obtain the residual r_1 of feature y_1, quantize r_1 to obtain r_q_1 and encode it into bitstream 2. Decode based on bitstream 2 to obtain r_q_1, and dequantize to obtain r_hat_1, and combine with the mean mu_1 to perform feature recovery to obtain the reconstructed feature y_hat_1.

[0204] 2. For the i-th stage feature y_i (i>1), z_hat is obtained based on bitstream 1 decoding, and the features y_hat_1 to y_hat_i-1 reconstructed in the previous stage are used to obtain the mean mu_i through the mean prediction network. Based on mu_i, the residual r_i of feature y_i is obtained, and r_q_i is encoded into bitstream 2. r_hat_i is obtained based on bitstream 2 decoding, and the feature is restored in combination with the mean mu_i to obtain the reconstructed feature y_hat_i.

[0205] 3. Gather y_hat_i (i=1...N) to obtain feature y_hat. The feature y_hat is passed through a synthetic transformation network to obtain the Y component to reconstruct the image.

[0206] This embodiment obtains at least one extended feature group by non-uniformly grouping the image features corresponding to the image to be encoded; performing residual calculation on the at least one extended feature group respectively to obtain image residual data corresponding to each extended feature group; generating an image code stream according to the image residual data, and sending the image code stream to the image decoding end. After the image features of the image to be encoded are obtained, they are non-uniformly grouped into at least one extended residual group, so that the residual calculation processing can be performed in groups, thereby improving the overall residual calculation efficiency and reducing the time complexity.

[0207] In addition, an embodiment of the present invention also proposes a storage medium, on which a decoding program and / or an encoding program is stored. When the decoding program is executed by a processor, the steps of the decoding method based on multi-step context prediction as described above are implemented; when the encoding program is executed by a processor, the steps of the encoding method based on multi-step context prediction as described above are implemented.

[0208] Reference Fig.13 , Fig.13 FIG. 4 is a structural block diagram of a first embodiment of a decoding device according to the present invention.

[0209] like Fig.13 As shown, the decoding device proposed in the embodiment of the present invention includes:

[0210] The grouping module 10 is used to perform non-uniform grouping on the image residual data extracted from the image code stream to obtain at least one extended residual group.

[0211] The restoration module 20 is used to perform residual restoration on the at least one extended residual group respectively to obtain image reconstruction features corresponding to each extended residual group.

[0212] The reconstruction module 30 is used to perform image reconstruction based on the image reconstruction features corresponding to each extended residual group to obtain a reconstructed image block.

[0213] This embodiment obtains at least one extended residual group by non-uniformly grouping the image residual data extracted from the image code stream; performs residual recovery on the at least one extended residual group to obtain the image reconstruction features corresponding to each extended residual group; and performs image reconstruction based on the image reconstruction features corresponding to each extended residual group to obtain a reconstructed image block. Since the image residual data extracted from the image code stream will be grouped first, it is ensured that the residual recovery process can be performed in the group, thereby improving the overall residual recovery calculation efficiency and reducing the time complexity. When grouping, non-uniform grouping can be performed according to the detailed information contained in each feature data, adapting to the uneven distribution of feature data, thereby improving encoding performance and reducing complexity.

[0214] In a possible implementation of this embodiment, the grouping module 10 is also used to extract image residual data from the image code stream; divide the image residual data into at least one channel division group according to the feature channels corresponding to each image residual data; and perform spatial domain division on the image residual data in the channel division group to obtain at least one extended residual group.

[0215] In a possible implementation of the present embodiment, the grouping module 10 is further used to perform classification according to corresponding feature channels to obtain image residual data corresponding to each feature channel; determine at least one data group and a feature channel label corresponding to each data group according to a channel division rule; and add the image residual data corresponding to each feature channel to the at least one data group based on the feature channel label to obtain at least one channel division group.

[0216] In a possible implementation manner of this embodiment, the channel division rule is a non-uniform division rule.

[0217] In a possible implementation of the present embodiment, the grouping module 10 is further used to perform phase division on the image residual data in the channel division group according to a phase division specification to obtain a plurality of characteristic phases corresponding to the channel division group; determine at least one spatial division group and a spatial label corresponding to each spatial division group based on a spatial division rule; perform spatial labeling on the characteristic phases according to the number of groups in the at least one spatial division group; and add the image residual data in the channel division group to the at least one spatial division group based on the labeled characteristic phases to obtain at least one extended residual group.

[0218] In a possible implementation manner of this embodiment, the spatial domain division rule is a non-uniform division rule.

[0219] In a possible implementation of this embodiment, different channel division groups correspond to different airspace division rules.

[0220] In a possible implementation of this embodiment, different channel division groups correspond to different phase division specifications.

[0221] In a possible implementation of this embodiment, the recovery module 20 is further used to construct a residual recovery sequence according to the at least one extended residual group; based on the residual recovery sequence, perform residual recovery on the at least one extended residual group respectively to obtain image reconstruction features corresponding to each extended residual group.

[0222] In a possible implementation of the present embodiment, the recovery module 20 is further used to traverse the residual recovery sequence to obtain a current extended residual group; obtain auxiliary information output by the auxiliary coding network; construct prior information based on the auxiliary information; perform residual recovery on the current extended residual group based on the prior information to obtain image reconstruction features corresponding to the current extended residual group; and at the end of the traversal, obtain image reconstruction features corresponding to each extended residual group.

[0223] In a possible implementation of the present embodiment, the recovery module 20 is also used to reduce the spatial resolution of the auxiliary information to obtain extended auxiliary information; detect whether the current extended residual group is the first element in the residual recovery sequence; if it is the first element, construct prior information based on the extended auxiliary information; if it is not the first element, splicing the extended auxiliary information with the convolution processing result corresponding to the image reconstruction feature of the restored extended residual group to obtain spliced ​​auxiliary information, and construct prior information based on the spliced ​​auxiliary information.

[0224] In a possible implementation of this embodiment, the recovery module 20 is also used to obtain image reconstruction features corresponding to the restored extended residual group; perform feature enhancement on the image reconstruction features to obtain enhanced reconstruction features; and splice the auxiliary information with the convolution processing results corresponding to the enhanced reconstruction features to obtain spliced ​​auxiliary information.

[0225] In a possible implementation of the present embodiment, the recovery module 20 is also used to obtain the predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the restored extended residual group; and perform feature enhancement on the image reconstruction features according to the predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the restored extended residual group to obtain enhanced reconstruction features.

[0226] In a possible implementation of this embodiment, the reconstruction module 30 is further used to enlarge the spatial resolution of the image reconstruction features corresponding to each extended residual group to obtain reconstructed feature data; and perform image reconstruction according to the reconstructed feature data to obtain a reconstructed image block.

[0227] Reference Fig.14 , Fig.14 It is a structural block diagram of the second embodiment of the encoding device of the present invention.

[0228] like Fig.14 As shown, the encoding device proposed in the embodiment of the present invention includes:

[0229] A grouping module 10, configured to non-uniformly group image features corresponding to the image to be encoded to obtain at least one extended feature group;

[0230] A calculation module 20, configured to perform residual calculation on the at least one extended feature group to obtain image residual data corresponding to each extended feature group;

[0231] The generating module 30 is used to generate an image code stream according to the image residual data, and send the image code stream to an image decoding end.

[0232] This embodiment obtains at least one extended feature group by non-uniformly grouping the image features corresponding to the image to be encoded; performing residual calculation on the at least one extended feature group respectively to obtain image residual data corresponding to each extended feature group; generating an image code stream according to the image residual data, and sending the image code stream to the image decoding end. After the image features of the image to be encoded are obtained, they are non-uniformly grouped into at least one extended residual group, so that the residual calculation processing can be performed in groups, thereby improving the overall residual calculation efficiency and reducing the time complexity.

[0233] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.

[0234] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0235] In addition, for technical details not described in detail in this embodiment, reference may be made to the decoding method based on multi-step context prediction or the encoding method based on multi-step context prediction provided in any embodiment of the present invention, which will not be repeated here.

[0236] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0237] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0238] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0239] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A decoding method based on multi-step context prediction, characterized in that: The decoding method based on multi-step context prediction includes: Non-uniformly grouping image residual data extracted from the image code stream to obtain at least one extended residual group; Performing residual recovery on the at least one extended residual group respectively to obtain image reconstruction features corresponding to each extended residual group; Image reconstruction is performed based on the image reconstruction features corresponding to each extended residual group to obtain a reconstructed image block.

2. The decoding method based on multi-step context prediction according to claim 1, characterized in that: The step of non-uniformly grouping the image residual data extracted from the image code stream to obtain at least one extended residual group includes: Extract image residual data from the image code stream; According to the feature channels corresponding to the image residual data, the image residual data are divided into at least one channel division group; Perform spatial domain division on the image residual data in the channel division group to obtain at least one extended residual group.

3. The decoding method based on multi-step context prediction as claimed in claim 2, characterized in that The step of dividing the image residual data into at least one channel division group according to the feature channels corresponding to each image residual data comprises: Classify according to the corresponding feature channels to obtain the image residual data corresponding to each feature channel; Determine at least one data group and a characteristic channel label corresponding to each data group according to the channel division rule; Based on the feature channel labels, the image residual data corresponding to each feature channel is added to the at least one data group to obtain at least one channel division group.

4. The decoding method based on multi-step context prediction as claimed in claim 3, characterized in that The channel division rule is a non-uniform division rule.

5. The decoding method based on multi-step context prediction as claimed in claim 2, characterized in that: The performing spatial domain division on the image residual data in the channel division group to obtain at least an extended residual group includes: Performing phase division on the image residual data in the channel division group according to the phase division specification to obtain a plurality of characteristic phases corresponding to the channel division group; Determine at least one airspace division group and an airspace label corresponding to each airspace division group based on the airspace division rule; Spatially labeling the characteristic phases according to the number of the at least one spatial division group; The image residual data in the channel division group is added to the at least one spatial domain division group based on the labeled characteristic phase to obtain at least one extended residual group.

6. The decoding method based on multi-step context prediction as claimed in claim 5, characterized in that: The airspace division rule is a non-uniform division rule.

7. The decoding method based on multi-step context prediction according to claim 5, characterized in that: Different channel division groups correspond to different airspace division rules.

8. The decoding method based on multi-step context prediction as claimed in claim 5, characterized in that: Different channel division groups correspond to different phase division specifications.

9. The decoding method based on multi-step context prediction according to claim 1, characterized in that: The performing residual recovery on the at least one extended residual group respectively to obtain image reconstruction features corresponding to each extended residual group includes: constructing a residual recovery sequence according to the at least one extended residual group; Based on the residual recovery sequence, residual recovery is performed on the at least one extended residual group respectively to obtain image reconstruction features corresponding to each extended residual group.

10. The decoding method based on multi-step context prediction according to claim 9, characterized in that: The step of performing residual recovery on the at least one extended residual group based on the residual recovery sequence to obtain image reconstruction features corresponding to each extended residual group includes: Traversing the residual recovery sequence to obtain a current extended residual group; Obtain auxiliary information output by the auxiliary encoding network; constructing prior information based on the auxiliary information; Performing residual recovery on the current extended residual group based on the prior information to obtain image reconstruction features corresponding to the current extended residual group; At the end of the traversal, the image reconstruction features corresponding to each extended residual group are obtained.

11. The decoding method based on multi-step context prediction according to claim 10, characterized in that: The constructing the prior information according to the auxiliary information comprises: Performing spatial domain resolution reduction processing on the auxiliary information to obtain extended auxiliary information; Detecting whether the current extended residual group is the first element in the residual recovery sequence; If it is the first element, constructing the prior information according to the extended auxiliary information; If it is not the first element, the extended auxiliary information is spliced ​​with the convolution processing result corresponding to the image reconstruction feature of the restored extended residual group to obtain spliced ​​auxiliary information, and prior information is constructed according to the spliced ​​auxiliary information.

12. The decoding method based on multi-step context prediction according to claim 11, characterized in that: The step of splicing the extended auxiliary information with the convolution processing result corresponding to the image reconstruction feature of the restored extended residual group to obtain the spliced ​​auxiliary information includes: Obtaining image reconstruction features corresponding to the restored extended residual group; Performing feature enhancement on the image reconstruction feature to obtain an enhanced reconstruction feature; The auxiliary information is spliced ​​with the convolution processing result corresponding to the enhanced reconstruction feature to obtain spliced ​​auxiliary information.

13. The decoding method based on multi-step context prediction according to claim 12, characterized in that: The step of performing feature enhancement on the image reconstruction feature to obtain an enhanced reconstruction feature includes: Obtaining a predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the restored extended residual group; The image reconstruction feature is enhanced according to the predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the restored extended residual group to obtain an enhanced reconstruction feature.

14. The decoding method based on multi-step context prediction according to any one of claims 1 to 13, characterized in that: The image reconstruction is performed based on the image reconstruction features corresponding to each extended residual group to obtain a reconstructed image block, including: Amplifying the spatial resolution of the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data; Image reconstruction is performed according to the reconstruction feature data to obtain a reconstructed image block.

15. A coding method based on multi-step context prediction, characterized in that: The encoding method based on multi-step context prediction includes: Non-uniformly grouping image features corresponding to the image to be encoded to obtain at least one extended feature group; Performing residual calculation on the at least one extended feature group respectively to obtain image residual data corresponding to each extended feature group; An image code stream is generated according to the image residual data, and the image code stream is sent to an image decoding end.

16. A decoding device, characterized in that: The decoding device comprises: A grouping module, used for non-uniformly grouping the image residual data extracted from the image code stream to obtain at least one extended residual group; A restoration module, used to perform residual restoration on the at least one extended residual group respectively to obtain image reconstruction features corresponding to each extended residual group; The reconstruction module is used to perform image reconstruction based on the image reconstruction features corresponding to each extended residual group to obtain a reconstructed image block.

17. A coding device, characterized in that: The encoding device comprises: A grouping module, used for non-uniformly grouping image features corresponding to the image to be encoded to obtain at least one extended feature group; A calculation module, used to perform residual calculation on the at least one extended feature group respectively to obtain image residual data corresponding to each extended feature group; The generating module is used to generate an image code stream according to the image residual data, and send the image code stream to the image decoding end.

18. A decoding device, characterized in that: The decoding device comprises: a processor, a memory, and a decoding program stored in the memory and executable on the processor. When the decoding program is executed by the processor, the steps of the decoding method based on multi-step context prediction as described in any one of claims 1 to 14 are implemented.

19. A coding device, characterized in that: The encoding device includes: a processor, a memory, and a decoding program and / or encoding program stored in the memory and executable on the processor. When the decoding program is executed by the processor, the steps of the decoding method based on multi-step context prediction as described in any one of claims 1 to 14 are implemented. When the encoding program is executed by the processor, the steps of the encoding method based on multi-step context prediction as described in claim 15 are implemented.

20. A storage medium, characterized in that: The storage medium stores a decoding program and / or an encoding program. When the decoding program is executed, the steps of the decoding method based on multi-step context prediction as described in any one of claims 1 to 14 are implemented. When the encoding program is executed, the steps of the encoding method based on multi-step context prediction as described in claim 15 are implemented.

Images