Decoding and encoding methods, apparatus and devices based on multi-step context prediction

Abstract

This invention belongs to the field of image processing technology and discloses a decoding and encoding method, apparatus, and device based on multi-step context prediction. The invention obtains at least one extended residual group by non-uniformly grouping the image residual data extracted from the image bitstream; it then performs residual recovery on each of the at least one extended residual group to obtain image reconstruction features corresponding to each extended residual group; finally, it performs image reconstruction based on the image reconstruction features corresponding to each extended residual group to obtain reconstructed image blocks. Because the image residual data extracted from the image bitstream is grouped first, it ensures that residual recovery processing can be performed on entire groups, thereby improving the overall residual recovery calculation efficiency and reducing time complexity. Furthermore, the non-uniform grouping based on the detailed information contained in each feature data adapts to the uneven distribution of feature data, thus improving encoding performance and reducing complexity.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a decoding and encoding method, apparatus, and device based on multi-step context prediction. Background Technology

[0002] In deep learning-based image compression schemes, the mainstream approach is to use already decoded feature points as prior information to predict the mean of the currently decoded feature points in order to reduce spatial redundancy in the image. The mainstream schemes generally use serial or wavefront encoding and decoding methods, and the serialness varies with the feature points, resulting in high overall execution time complexity.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this invention is to provide a decoding and encoding method, apparatus, and device based on multi-step context prediction, which aims to solve the technical problem of high time complexity in mean prediction during image encoding and decoding in the prior art.

[0005] To achieve the above objectives, the present invention provides a decoding method based on multi-step context prediction, the method comprising the following steps:

[0006] The image residual data extracted from the image bitstream is non-uniformly grouped to obtain at least one extended residual group;

[0007] Residual recovery is performed on each of the at least one extended residual group to obtain the image reconstruction features corresponding to each extended residual group;

[0008] Image reconstruction is performed based on the image reconstruction features corresponding to each extended residual group to obtain reconstructed image patches.

[0009] In one possible implementation of this application, the step of non-uniformly grouping the image residual data extracted from the image bitstream to obtain at least one extended residual group includes:

[0010] Extracting image residual data from the image bitstream;

[0011] Based on the feature channels corresponding to each image residual data, the image residual data is divided into at least one channel group;

[0012] Spatial domain partitioning is performed on the image residual data in the channel partitioning group to obtain at least one extended residual group.

[0013] In one possible implementation of this application, dividing the image residual data into at least one channel group based on the feature channels corresponding to each image residual data includes:

[0014] The image residual data corresponding to each feature channel is obtained by classifying the data according to the corresponding feature channels.

[0015] Determine at least one data group and the corresponding feature channel number for each data group according to the channel division rules;

[0016] Based on the feature channel labels, the image residual data corresponding to each feature channel is added to the at least one data group to obtain at least one channel division group.

[0017] In one possible implementation of this application, the channel division rule is a non-uniform division rule.

[0018] In one possible implementation of this application, the step of spatially partitioning the image residual data in the channel partitioning group to obtain at least one extended residual group includes:

[0019] The image residual data in the channel division group is phase-divided according to the phase division specification to obtain multiple feature phases corresponding to the channel division group;

[0020] Based on the airspace partitioning rules, at least one airspace partitioning group and the airspace label corresponding to each airspace partitioning group are determined.

[0021] The feature phase is spatially labeled according to the number of groups in the at least one spatial division group;

[0022] Based on the labeled characteristic phase, the image residual data in the channel partitioning group is added to the at least one spatial partitioning group to obtain at least one extended residual group.

[0023] In one possible implementation of this application, the spatial domain partitioning rule is a non-uniform partitioning rule.

[0024] In one possible implementation of this application, different channel division groups correspond to different airspace division rules.

[0025] In one possible implementation of this application, different channel division groups correspond to different phase division specifications.

[0026] In one possible implementation of this application, the step of performing residual recovery on the at least one extended residual group to obtain the image reconstruction features corresponding to each extended residual group includes:

[0027] Construct a residual recovery sequence based on the at least one extended residual set;

[0028] Based on the residual recovery sequence, residual recovery is performed on the at least one extended residual group to obtain the image reconstruction features corresponding to each extended residual group.

[0029] In one possible implementation of this application, the step of performing residual recovery on the at least one extended residual group based on the residual recovery sequence to obtain the image reconstruction features corresponding to each extended residual group includes:

[0030] The residual recovery sequence is traversed to obtain the current extended residual set;

[0031] Obtain auxiliary information output by the auxiliary coding network;

[0032] Construct prior information based on the auxiliary information;

[0033] Based on the prior information, residual recovery is performed on the current extended residual group to obtain the image reconstruction features corresponding to the current extended residual group;

[0034] At the end of the traversal, the image reconstruction features corresponding to each extended residual group are obtained.

[0035] In one possible implementation of this application, constructing prior information based on the auxiliary information includes:

[0036] The auxiliary information is processed to reduce its spatial resolution, thereby obtaining extended auxiliary information;

[0037] Detect whether the current extended residual group is the first element in the residual recovery sequence;

[0038] If it is the first element, then prior information is constructed based on the extended auxiliary information;

[0039] If it is not the first element, the extended auxiliary information is concatenated with the convolution processing result corresponding to the image reconstruction feature of the recovered extended residual group to obtain the concatenation auxiliary information, and prior information is constructed based on the concatenation auxiliary information.

[0040] In one possible implementation of this application, the step of concatenating the extended auxiliary information with the convolution processing result corresponding to the image reconstruction features of the recovered extended residual group to obtain concatenation auxiliary information includes:

[0041] Obtain the image reconstruction features corresponding to the recovered extended residual group;

[0042] The image reconstruction features are enhanced to obtain enhanced reconstruction features;

[0043] The auxiliary information is concatenated with the convolutional processing result corresponding to the enhanced reconstruction feature to obtain the concatenated auxiliary information.

[0044] In one possible implementation of this application, the step of enhancing the image reconstruction features to obtain enhanced reconstruction features includes:

[0045] Obtain the predicted mean, auxiliary information, image residual data, and / or residual data variance corresponding to the recovered extended residual group;

[0046] The image reconstruction features are enhanced by using the predicted mean, auxiliary information, image residual data, and / or residual data variance corresponding to the recovered extended residual group to obtain enhanced reconstruction features.

[0047] In one possible implementation of this application, the step of performing image reconstruction based on the image reconstruction features corresponding to each extended residual group to obtain reconstructed image patches includes:

[0048] The image reconstruction features corresponding to each extended residual group are processed to increase the spatial resolution to obtain reconstruction feature data.

[0049] Image reconstruction is performed based on the reconstructed feature data to obtain reconstructed image blocks.

[0050] Furthermore, to achieve the above objectives, this invention also proposes a coding method based on multi-step context prediction, wherein the coding method based on multi-step context prediction includes:

[0051] The image features corresponding to the image to be encoded are non-uniformly grouped to obtain at least one extended feature group;

[0052] Residual calculations are performed on each of the at least one extended feature group to obtain image residual data corresponding to each extended feature group;

[0053] An image bitstream is generated based on the image residual data, and the image bitstream is sent to the image decoding end.

[0054] Furthermore, to achieve the above objectives, the present invention also proposes a decoding device, the decoding device comprising:

[0055] The grouping module is used to perform non-uniform grouping of the image residual data extracted from the image bitstream to obtain at least one extended residual group;

[0056] The recovery module is used to perform residual recovery on the at least one extended residual group respectively to obtain the image reconstruction features corresponding to each extended residual group;

[0057] The reconstruction module is used to reconstruct images based on the image reconstruction features corresponding to each extended residual group, and obtain reconstructed image blocks.

[0058] Furthermore, to achieve the above objectives, the present invention also proposes an encoding device, the encoding device comprising:

[0059] The grouping module is used to non-uniformly group the image features corresponding to the image to be encoded to obtain at least one extended feature group.

[0060] The calculation module is used to perform residual calculation on the at least one extended feature group respectively to obtain the image residual data corresponding to each extended feature group;

[0061] The generation module is used to generate an image bitstream based on the image residual data and send the image bitstream to the image decoding end.

[0062] Furthermore, to achieve the above objectives, the present invention also proposes a decoding device, which includes: a processor, a memory, and a decoding program stored in the memory and executable on the processor. When the decoding program is executed by the processor, it implements the steps of the decoding method based on multi-step context prediction as described above.

[0063] Furthermore, to achieve the above objectives, the present invention also proposes an encoding device, the encoding device comprising: a processor, a memory, and a decoding program and / or an encoding program stored in the memory and executable on the processor, wherein when the decoding program is executed by the processor, it implements the steps of the decoding method based on multi-step context prediction as described above, and when the encoding program is executed by the processor, it implements the steps of the encoding method based on multi-step context prediction as described above.

[0064] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a decoding program and / or an encoding program, wherein the decoding program, when executed, implements the steps of the decoding method based on multi-step context prediction as described above, and the encoding program, when executed, implements the steps of the encoding method based on multi-step context prediction as described above.

[0065] This invention obtains at least one extended residual group by non-uniformly grouping the image residual data extracted from the image bitstream; residual recovery is then performed on each of the at least one extended residual group to obtain image reconstruction features corresponding to each extended residual group; finally, image reconstruction is performed based on the image reconstruction features corresponding to each extended residual group to obtain reconstructed image blocks. Because the image residual data extracted from the image bitstream is grouped first, it ensures that residual recovery processing can be performed on entire groups, thereby improving the overall residual recovery calculation efficiency and reducing time complexity. Furthermore, the non-uniform grouping based on the detailed information contained in each feature data adapts to the uneven distribution of feature data, thus improving coding performance and reducing complexity. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the structure of an electronic device in the hardware operating environment involved in the embodiments of the present invention;

[0067] Figure 2 This is a flowchart illustrating the first embodiment of the decoding method based on multi-step context prediction of the present invention;

[0068] Figure 3 This is an overall framework diagram of image compression according to an embodiment of the present invention;

[0069] Figure 4 This is a flowchart illustrating a second embodiment of the decoding method based on multi-step context prediction of the present invention.

[0070] Figure 5 This is a flowchart illustrating the third embodiment of the decoding method based on multi-step context prediction of the present invention;

[0071] Figure 6 This is a flowchart illustrating the fourth embodiment of the decoding method based on multi-step context prediction of the present invention;

[0072] Figure 7 This is a flowchart illustrating the fifth embodiment of the decoding method based on multi-step context prediction of the present invention;

[0073] Figure 8 This is a schematic diagram of non-uniform channel and uniform spatial grouping according to an embodiment of the present invention;

[0074] Figure 9 This is a schematic diagram of a channel uniform and spatial non-uniform division grouping according to an embodiment of the present invention;

[0075] Figure 10 This is a schematic diagram of non-uniform spatial grouping of a 3x3 phase partitioning specification according to an embodiment of the present invention.

[0076] Figure 11 This is a schematic diagram of a multi-network residual reconstruction structure according to an embodiment of the present invention;

[0077] Figure 12 This is a flowchart illustrating the first embodiment of the encoding method based on multi-step context prediction of the present invention;

[0078] Figure 13 This is a structural block diagram of the first embodiment of the decoding device of the present invention;

[0079] Figure 14 This is a structural block diagram of the first embodiment of the encoding device of the present invention.

[0080] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0081] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0082] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a decoding or encoding device in the hardware operating environment involved in the embodiments of the present invention.

[0083] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0084] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0085] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a decoding program and / or an encoding program.

[0086] exist Figure 1 In the illustrated electronic device, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the decoding device or the encoding device. The electronic device calls the decoding program or encoding program stored in the memory 1005 through the processor 1001 and executes the decoding method or encoding method based on multi-step context prediction provided in the embodiments of the present invention.

[0087] This invention provides a decoding method based on multi-step context prediction, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a decoding method based on multi-step context prediction according to the present invention.

[0088] In this embodiment, the decoding method based on multi-step context prediction includes the following steps:

[0089] Step S10: Non-uniformly group the image residual data extracted from the image bitstream to obtain at least one extended residual group;

[0090] It should be noted that the execution subject of this embodiment can be a decoding device when encoding image data. The decoding device can be a personal computer, server or other electronic device. Of course, it can also be other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment and the following embodiments, the decoding device is used as an example to illustrate the decoding method based on multi-step context prediction of the present invention.

[0091] Since the encoding device typically decodes the encoded image stream after encoding is completed during the image encoding process, and determines whether the parameters used in the encoding need to be adjusted based on the quality of the decoded image, the execution subject in this embodiment can also be the encoding device.

[0092] It should be noted that the image bitstream can be the bitstream generated by the encoding device after encoding the image data that needs to be compressed. When generating the image bitstream, the encoding device groups the image features, then performs mean prediction on each group, and calculates the feature residuals based on the predicted means. Finally, the encoding device directly encodes the generated image residual data into the image bitstream. At this point, the decoding device can directly extract the image residual data from the image bitstream. The decoding device can then divide the image residual data into at least one extended residual group according to the same grouping method as the encoding device, and then perform residual recovery on each group, thereby reducing the time complexity of mean prediction in the image decoding process.

[0093] It should be noted that in the process of mean prediction, the feature data used may have a small number of channels representing low-frequency information, a portion of channels representing high-frequency detail information, and the remaining channels being 0, containing no information. This non-uniformity means that using uniform feature channel and spatial domain partitioning does not achieve optimal coding performance and complexity. Therefore, when grouping image features, the coding device can use non-uniform grouping. Similarly, when non-uniformly grouping image residual data, the coding device can use non-uniform grouping. Non-uniform grouping refers to using non-uniform grouping for at least one of the feature channels and spatial domains when grouping based on feature channels and spatial domains. For example: uniform grouping for feature channels but non-uniform grouping for spatial domains; uniform grouping for feature channels but non-uniform grouping for spatial domains; or non-uniform grouping for both feature channels and spatial domains.

[0094] Technical terms involved in image encoding or decoding include: JPEG (Joint Photographic Experts Group), JPEG-AI (Joint Photographic Experts Group Artificial Intelligence), Entropy Encoding, Neural Network (NN), Convolutional Neural Network (CNN), feature, Rate-Distortion Optimized, etc., which will be explained here.

[0095] JPEG (Joint Photographic Experts Group) is a standard for compressing continuous-tone still images. Files with the extension .jpg or .jpeg are the most commonly used image file format. It primarily employs a joint coding method combining predictive coding (DPCM), discrete cosine transform (DCT), and entropy coding to remove redundant image and color data. It is a lossy compression format, capable of compressing images into a very small storage space, but this inevitably causes some damage to the image data. Especially with excessively high compression ratios, the quality of the decompressed image will decrease. Therefore, if high-quality images are desired, excessively high compression ratios should be avoided.

[0096] JPEG AI aims to create a learning-based image coding standard that provides a single-stream, compact compressed domain representation, optimized for human visualization, significantly improving compression efficiency compared to commonly used image coding standards while maintaining the same subjective quality, and offering efficient performance for image processing and computer vision tasks. JPEG AI is geared towards a wide range of applications, such as cloud storage, visual surveillance, autonomous vehicles and devices, image acquisition, storage and management, real-time monitoring of visual data, and media distribution. The goal is to design an encoding solution that significantly improves the compression efficiency of commonly used coding standards while maintaining the same subjective quality, and provides efficient compressed domain processing for machine learning-based image processing and computer vision tasks. Other key requirements include hardware / software implementation of user-friendly encoding and decoding, support for 8-bit and 10-bit depths, efficient encoding of images using text and graphics, and progressive decoding.

[0097] Entropy coding is encoding that follows the entropy principle without losing any information during the encoding process. Information entropy is the average amount of information in a source (a measure of uncertainty). Common entropy coding methods include Shannon coding, Huffman coding, and arithmetic coding.

[0098] The neural network referred to in this application is an artificial neural network, not a biological neural network. A neural network is a computational model composed of a large number of interconnected nodes (or neurons). In an artificial neural network, neurons can represent different objects, such as features, letters, concepts, or meaningful abstract patterns. There are three types of processing units in a network: input units, output units, and hidden units. Input units receive signals and data from the external world; output units output the system's processing results; hidden units are located between input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units; the representation and processing of information are reflected in the connection relationships between the network's processing units. Artificial neural networks are a non-programmed, brain-like information processing method. Essentially, they achieve parallel and distributed information processing capabilities through network transformations and dynamic behaviors, mimicking the information processing functions of the human brain's nervous system to varying degrees and levels. Currently, in the field of video processing, commonly used neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and fully connected networks.

[0099] Convolutional Neural Networks (CNNs) are a type of feedforward neural network and one of the most representative network structures in deep learning. Their artificial neurons can respond to surrounding units within a certain coverage area, exhibiting excellent performance in large-scale image processing. Generally, the basic structure of a CNN consists of two layers: a feature extraction layer (also called a convolutional layer), where the input of each neuron is connected to the local receptive field of the previous layer, extracting features from that local area. Once these local features are extracted, their positional relationship with other features is determined. The second layer is a feature mapping layer (also called an activation layer), where each computational layer of the network consists of multiple feature maps. Each feature map is a plane where all neurons have equal weights. Feature mapping structures can use functions such as the Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, and GDN function as activation functions for the convolutional network. Furthermore, since neurons on a single map share weights, the number of free parameters in the network is reduced. One of the advantages of CNNs compared to traditional image processing algorithms is that they avoid complex preprocessing steps (such as extracting artificial features) and can directly input the original image for end-to-end learning. One of the advantages of CNNs over traditional neural networks is that traditional neural networks use a fully connected approach, meaning that all neurons from the input layer to the hidden layer are connected. This results in a huge number of parameters, making network training time-consuming or even difficult. CNNs, on the other hand, avoid this difficulty by using methods such as local connectivity and weight sharing.

[0100] The feature involved in this application is a three-dimensional feature matrix of size CxWxH. C represents the number of channels, H represents the feature height, and W represents the feature width. The feature matrix can be either the input or the output of a neural network.

[0101] Metrics for evaluating coding efficiency include bitrate, PSNR, MS-SSIM, VMAF, FSIM, PSNR, and HVS, among others. More metrics can be included, but this is not a limitation. A smaller bitstream results in a higher compression ratio; a higher PSNR indicates better image coding efficiency. The discrimination formula for mode selection is essentially a comprehensive evaluation of both factors. The cost corresponding to a mode is: J(mode) = D + λ*R. Here, D represents Distortion, usually measured using the MSE metric; SSE refers to the sum of the mean squares of the differences between the reconstructed block and the source image; λ is the Lagrange multiplier; and R is the actual number of bits required for encoding the image block in that mode, including the total number of bits required for coding mode information, residuals, etc.

[0102] Step S20: Perform residual recovery on each of the at least one extended residual group to obtain the image reconstruction features corresponding to each extended residual group.

[0103] It should be noted that residual recovery of the extended residual group to obtain the image reconstruction features corresponding to the extended residual group can be achieved by predicting the mean of the extended residual group, and then summing the mean values ​​predicted from the residual data domains in the extended residual group to obtain the image reconstruction features corresponding to the extended residual group.

[0104] Step S30: Reconstruct the image based on the image reconstruction features corresponding to each extended residual group to obtain reconstructed image patches.

[0105] It is understandable that image reconstruction based on the image reconstruction features corresponding to each extended residual group can be achieved by using a pre-built synthetic transformation network to perform synthetic transformation processing on the image reconstruction features corresponding to each extended residual group, thereby obtaining the reconstructed image patch. The synthetic transformation network can be a network built based on deep learning or a neural network.

[0106] In one possible implementation of this embodiment, to ensure that image reconstruction can be performed normally, step S30 of this embodiment may include:

[0107] The image reconstruction features corresponding to each extended residual group are processed to increase the spatial resolution to obtain reconstruction feature data.

[0108] Image reconstruction is performed based on the reconstructed feature data to obtain reconstructed image blocks.

[0109] It should be noted that since the extended residual group is obtained by non-uniformly grouping the image residual data extracted from the image bitstream, and non-uniform grouping is based on feature channels and spatial domain, the spatial size and number of channels of the data may be different from the spatial size and number of channels of the image features obtained by feature extraction from the original image. Therefore, when performing image reconstruction, the spatial resolution of the image reconstruction features corresponding to each extended residual group can be enlarged first to restore the spatial size and number of channels of the image reconstruction features to be consistent with the image features obtained by feature extraction from the original image, thereby obtaining the reconstructed feature data.

[0110] Before performing spatial resolution amplification processing on the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data, feature enhancement can be performed on the image reconstruction features corresponding to each extended residual group, and then the enhanced reconstruction features corresponding to each extended residual group can be amplified in spatial resolution to obtain reconstruction feature data. This can ensure that the reliability of the final constructed reconstruction feature data is higher, and the quality of the reconstructed image blocks obtained from subsequent image reconstruction will be better.

[0111] In practical applications, feature enhancement can be performed on the image reconstruction features corresponding to the extended residual group to obtain the enhanced reconstruction features corresponding to the extended residual group. This can be achieved by using the predicted mean, auxiliary information, image residual data, and / or residual data variance of the extended residual group to enhance the image reconstruction features corresponding to the extended residual group.

[0112] To facilitate understanding, we will now combine... Figure 3 This explanation is provided, but it does not limit the scope of this solution. Figure 3 An overall framework diagram for image compression.

[0113] like Figure 3 As shown in the diagram, x and x_hat represent the input image (block) and the output reconstructed image (block), respectively. An image can be divided into one image block or multiple image blocks. The encoding and decoding process for the image blocks described below can also be directly applied to the image. y represents the latent features (latent domain features) obtained after the input image block is processed by the analysis and transformation network. y then passes through a hyperparameter encoding network to obtain the hyperparameter latent feature information z. z is quantized, encoded, and written into the first bitstream (Bitstream#1). The encoding / decoding process of the first bitstream generally uses a fixed probability density model algorithm. Based on the first bitstream, the z_hat feature is decoded. z_hat passes through a probabilistic hyperparameter decoding network to obtain the probability p required for decoding the second bitstream. Furthermore, besides obtaining the decoding probability of the second bitstream, the first feature can also be jointly reconstructed based on the first bitstream to generate the predicted value mu of the first feature, thereby obtaining the feature residual r. After quantization, the quantized feature residual r_q is obtained, and then entropy encoding is performed using probability p and encoded into the second bitstream. Based on the decoding of the second bitstream, the quantized feature residual r_q can be obtained. This residual can then be recovered to obtain the reconstructed residual r_hat. Based on mu and r_hat, the reconstructed first feature value y_hat can be obtained. y_hat can then be processed by a synthesis transform network to generate the reconstructed image patch x_hat.

[0114] This embodiment obtains at least one extended residual group by non-uniformly grouping the image residual data extracted from the image bitstream; residual recovery is then performed on each of the at least one extended residual group to obtain the image reconstruction features corresponding to each extended residual group; image reconstruction is then performed based on the image reconstruction features corresponding to each extended residual group to obtain reconstructed image blocks. Because the image residual data extracted from the image bitstream is first grouped, it ensures that residual recovery processing can be performed on entire groups, thereby improving the overall residual recovery calculation efficiency and reducing time complexity. Furthermore, the non-uniform grouping based on the detailed information contained in each feature data adapts to the uneven distribution of feature data, thus improving coding performance and reducing complexity.

[0115] refer to Figure 4 , Figure 4 This is a flowchart illustrating a second embodiment of a decoding method based on multi-step context prediction according to the present invention.

[0116] Based on the first embodiment described above, step S10 of the decoding method based on multi-step context prediction in this embodiment includes:

[0117] Step S101: Extract image residual data from the image bitstream.

[0118] It should be noted that extracting image residual data from an image bitstream can be achieved by parsing the image bitstream to obtain the image residual data.

[0119] Step S102: Based on the feature channels corresponding to each image residual data, divide the image residual data into at least one channel group.

[0120] It should be noted that, based on the feature channels corresponding to each image residual data, dividing the image residual data into at least one channel group can be achieved by grouping the image residual data corresponding to the feature channels marked as belonging to the same group into the same group, thereby obtaining at least one channel group.

[0121] In this process, the image residual data is divided into at least one channel group based on the feature channels. When the data is divided into multiple channel groups, a non-uniform method can be used for the division, that is, at least one channel group has at least one corresponding channel group with a different number of feature channels than the other channel groups.

[0122] Step S103: Spatial domain partitioning is performed on the image residual data in the channel partitioning group to obtain at least one extended residual group.

[0123] It should be noted that spatial partitioning of the image residual data in the channel partitioning group to obtain at least one extended residual group can be achieved by grouping the image residual data in the channel partitioning group that are marked as belonging to the same group into the same group, thereby obtaining the extended residual group corresponding to each channel partitioning group.

[0124] In the process of spatially partitioning the image residual data in the channel partitioning group to obtain at least one extended residual group, a non-uniform spatial partitioning method can be used, that is, to ensure that at least one extended residual group has an extended residual group whose number of corresponding spatial locations is inconsistent with other extended residual groups.

[0125] This embodiment extracts image residual data from the image bitstream; based on the feature channels corresponding to each image residual data, it divides the image residual data into at least one channel partitioning group; and it performs spatial domain partitioning on the image residual data in the channel partitioning group to obtain at least one extended residual group. Because the image residual data is grouped first based on feature channels and then based on spatial domain, the method of non-uniform grouping can be adjusted according to actual needs, ensuring that it can adapt to the uneven distribution of feature data.

[0126] refer to Figure 5 , Figure 5 This is a flowchart illustrating a third embodiment of a decoding method based on multi-step context prediction according to the present invention.

[0127] Based on the second embodiment described above, step S102 of the decoding method based on multi-step context prediction in this embodiment includes:

[0128] Step S1021: Classify according to the corresponding feature channels to obtain the image residual data corresponding to each feature channel.

[0129] It should be noted that, to facilitate grouping based on feature channels, image residual data can be categorized according to the corresponding feature channels, grouping image residual data corresponding to the same feature channel into the same category, thereby obtaining the image residual data corresponding to each feature channel. Furthermore, a corresponding label can be assigned to each feature channel, ensuring that different feature channels have different labels.

[0130] Step S1022: Determine at least one data group and the corresponding feature channel number of each data group according to the channel division rules.

[0131] It should be noted that the channel division rules can be pre-set by the administrators of the decoding or encoding devices.

[0132] In practical applications, determining at least one data group and its corresponding feature channel labels based on channel partitioning rules can be achieved by parsing the channel partitioning rules to obtain the number of groups and the set of feature channels. Determining at least one data group can also involve creating a number of data groups equal to the number of groups, thus obtaining at least one data group. Determining the feature channel labels for each data group can be done by assigning corresponding feature channel labels to each created data group based on the set of feature channel numbers.

[0133] The channel partitioning rule can be a non-uniform partitioning rule, that is, the number of feature channels corresponding to each channel partitioning group is not completely the same (i.e., there is at least one channel partitioning group whose number of feature channels is inconsistent with the number of feature channels corresponding to other channel partitioning groups). For example, assuming the number of feature channels is 4, and the labels of each feature channel are 1, 2, 3, and 4, if the channel partitioning rule is set to the number of corresponding groups is 2, and the number of corresponding feature channels is 1 and 3, then two data groups A and B can be created. The feature channel label of A is 1, and the feature channel labels of B are 2, 3, and 4.

[0134] Step S1023: Based on the feature channel labels, add the image residual data corresponding to each feature channel to the at least one data group to obtain at least one channel division group.

[0135] It should be noted that adding the image residual data corresponding to each feature channel to the at least one data group based on the feature channel label to obtain at least one channel division group can be achieved by matching the label corresponding to each feature channel with the feature channel label corresponding to each data group, and adding the image residual data corresponding to the successfully matched feature channel to the corresponding data group, thereby obtaining at least one channel division group.

[0136] This embodiment categorizes image residual data according to corresponding feature channels to obtain the image residual data for each feature channel; determines at least one data group and its corresponding feature channel label according to channel partitioning rules; and adds the image residual data for each feature channel to the at least one data group based on the feature channel labels, thus obtaining at least one channel partitioning group. Because the image residual data is first categorized according to the corresponding feature channels before grouping, batch addition can be performed directly during grouping, thereby improving grouping efficiency.

[0137] refer to Figure 6 , Figure 6 This is a flowchart illustrating the fourth embodiment of a decoding method based on multi-step context prediction according to the present invention.

[0138] Based on the second embodiment described above, step S103 of the decoding method based on multi-step context prediction in this embodiment includes:

[0139] Step S1031: Perform phase division on the image residual data in the channel division group according to the phase division specification to obtain multiple feature phases corresponding to the channel division group.

[0140] It should be noted that the phase division specification can be the size of the phase division, such as 2x2 or 3x3 or more. Each channel division group can use the same phase division specification.

[0141] In practical applications, the image residual data in the channel partitioning group is phase-divided according to the phase partitioning specification to obtain multiple feature phases corresponding to the channel partitioning group. This can be achieved by splitting the image residual data in the channel partitioning group into 1x1 sub-blocks, and then dividing the sub-blocks into multiple feature phases with the same size as the phase partitioning specification.

[0142] In one possible implementation of this embodiment, different channel division groups can correspond to different phase division specifications. For example, suppose there are two channel division groups, namely Y1 and Y2, where the phase division specification corresponding to Y1 can be 2X2 and the phase division specification corresponding to Y2 can be 3X3.

[0143] Step S1032: Determine at least one airspace division group and the airspace label corresponding to each airspace division group based on the airspace division rules.

[0144] It should be noted that the spatial domain partitioning rules can include the total number of empty leases for the total number of spatial domain partitioning groups and the phase positions corresponding to each spatial domain partitioning group. The spatial domain partitioning rules can be preset by the administrators of the image decoding device or image encoding device based on the actual location of feature information. The spatial domain partitioning rules can be pre-stored locally on the decoding device or obtained by parsing the image bitstream. This embodiment does not impose any restrictions on this. The spatial domain partitioning rules can be non-uniform partitioning rules, meaning that the number of phase positions corresponding to each spatial domain partitioning group is not exactly the same (i.e., at least one spatial domain partitioning group has a different number of phase positions than the other spatial domain partitioning groups). For example, assuming the phase partitioning specification is 2x2, each phase will include four 1x1 sub-blocks, resulting in a total of four phase positions. If there are three spatial domain partitioning groups, the number of phase positions can be set to 1, 1, and 2 respectively.

[0145] In practical applications, determining at least one airspace partitioning group and its corresponding airspace label based on airspace partitioning rules can be achieved by extracting the number of airspace groups from the airspace partitioning rules, generating airspace partitioning groups with the same number of airspace groups, obtaining at least one airspace partitioning group, and setting corresponding airspace labels for each airspace partitioning group starting from the initial label. For example, assuming that the number of airspace groups extracted from the airspace partitioning rules is 3, then 3 airspace partitioning groups can be created and assigned airspace labels (such as 1, 2, and 3) respectively.

[0146] Step S1033: Spatial label the feature phase according to the number of groups in the at least one spatial division group.

[0147] It should be noted that spatial labeling of the feature phase based on the number of groups in at least one spatial division group can be achieved by setting corresponding labels for each sub-block in the feature phase based on the number of groups in at least one spatial division group.

[0148] In practical use, when setting labels, labels can be set according to the group position mapping relationship in the spatial domain division rules. The group position mapping relationship can include the phase positions that are uniformly divided into groups.

[0149] Step S1034: Based on the labeled characteristic phase, add the image residual data in the channel partitioning group to the at least one spatial partitioning group to obtain at least one extended residual group.

[0150] It should be noted that, based on the labeled feature phase, the image residual data in the channel partitioning group is added to at least one spatial partitioning group to obtain at least one extended residual group. This can be achieved by matching the labels corresponding to each sub-block in the feature phase with the spatial labels corresponding to each extended residual group, thereby determining the corresponding spatial partitioning group for each image residual data in the channel partitioning group, and adding it to the corresponding spatial partitioning group to obtain at least one extended residual group.

[0151] This embodiment performs phase partitioning on the image residual data in the channel partitioning group according to the phase partitioning specification to obtain multiple feature phases corresponding to the channel partitioning group; determines at least one spatial partitioning group and its corresponding spatial label based on spatial partitioning rules; spatially labels the feature phases according to the number of groups in the at least one spatial partitioning group; and adds the image residual data in the channel partitioning group to the at least one spatial partitioning group based on the labeled feature phases to obtain at least one extended residual group. Because the spatial labeling of the split feature phases is based on the number of groups in the at least one spatial partitioning group during spatial grouping, it can support a wider range of group sizes and phase partitioning specifications.

[0152] refer to Figure 7 , Figure 7 This is a flowchart illustrating the fifth embodiment of a decoding method based on multi-step context prediction according to the present invention.

[0153] Based on the first embodiment described above, step S20 of the decoding method based on multi-step context prediction in this embodiment includes:

[0154] Step S201: Construct a residual recovery sequence based on the at least one extended residual set.

[0155] It should be noted that after dividing the image into at least one extended residual group, residual recovery can be performed group by group, thereby reducing the time complexity of mean prediction in the image decoding process. At this point, to determine the residual recovery order for each extended residual group, a residual recovery sequence can be constructed based on at least one extended residual group.

[0156] Step S202: Perform residual recovery on the at least one extended residual group based on the residual recovery sequence to obtain the image reconstruction features corresponding to each extended residual group.

[0157] It should be noted that residual recovery of at least one extended residual group based on the residual recovery sequence can be performed on at least one extended residual group in turn based on the sequence order in the residual recovery sequence.

[0158] In practical applications, residual recovery can be performed sequentially through a sequence traversal. In this case, step S202 of this embodiment may include:

[0159] The residual recovery sequence is traversed to obtain the current extended residual set;

[0160] Obtain auxiliary information output by the auxiliary coding network;

[0161] Construct prior information based on the auxiliary information;

[0162] Based on the prior information, residual recovery is performed on the current extended residual group to obtain the image reconstruction features corresponding to the current extended residual group;

[0163] At the end of the traversal, the image reconstruction features corresponding to each extended residual group are obtained.

[0164] It should be noted that traversing the residual recovery sequence to obtain the current extended residual set can be achieved by traversing the residual recovery sequence and using the traversed extended residual set as the current extended residual set. The auxiliary coding network can be as described above. Figure 3 The auxiliary network (AD&IQ) shown in the figure.

[0165] In one possible implementation of this embodiment, residual recovery is performed on the current extended residual group based on prior information to obtain the image reconstruction features corresponding to the current extended residual group. This can be achieved by processing the prior information with convolutional kernels to obtain features, then concatenating the auxiliary information output by the auxiliary network with the obtained features, and finally inputting the concatenated features into a Prediction Fusion Net to obtain the prediction mean. The prediction mean is then added to the residuals in the current extended residual group to achieve residual recovery and obtain the image reconstruction features corresponding to the current extended residual group.

[0166] For example, for the current extended residual group with index i in the residual recovery sequence, the mean mu_i can be obtained through the mean prediction network based on the auxiliary information z_hat and the reconstructed features y_hat_1 to y_hat_i-1 from the previous stage. Furthermore, the residual of the corresponding stage in r_hat is obtained and denoted as r_hat_i, and feature recovery is performed to obtain the image reconstruction feature y_hat_i corresponding to the current extended residual group.

[0167] In one possible implementation of this embodiment, after completing the grouping of extended residual groups, residual recovery is not performed directly based on the data within the current extended residual group. Instead, the residual corresponding to the current extended residual group is extracted from the bitstream based on the predicted parameters. At this point, residual recovery is performed on the current extended residual group based on prior information. The image reconstruction features corresponding to the current extended residual group can also be obtained by processing the prior information through convolutional kernels to obtain features. Then, the auxiliary information output by the auxiliary network is concatenated with the obtained features. After concatenation, the features are input into the Prediction Fusion Net to obtain the prediction mean and modeling parameters. Then, the quantized residual data corresponding to the current extended residual group is extracted from the image bitstream based on the modeling parameters, dequantized, and the residual data is added to the prediction mean to obtain the quantized residual data corresponding to the current extended residual group.

[0168] For example, for the current extended residual group with index i in the residual recovery sequence, the predicted mean mu_i and the modeled value sigma_i can be obtained through a mean prediction network based on the auxiliary information z_hat and the reconstructed features y_hat_1 to y_hat_i-1 from the previous stage. The quantized residual of the current extended residual group is then decoded from the image bitstream based on sigma_i, dequantized and denoted as r_hat_i, and the mean mu_i is added to it for feature recovery, resulting in the image reconstruction feature y_hat_i corresponding to the current extended residual group.

[0169] In practical use, the step of constructing prior information based on the auxiliary information described in this embodiment may include:

[0170] The auxiliary information is processed to reduce its spatial resolution, thereby obtaining extended auxiliary information;

[0171] Detect whether the current extended residual group is the first element in the residual recovery sequence;

[0172] If it is the first element, then prior information is constructed based on the extended auxiliary information;

[0173] If it is not the first element, the extended auxiliary information is concatenated with the convolution processing result corresponding to the image reconstruction feature of the recovered extended residual group to obtain the concatenation auxiliary information, and prior information is constructed based on the concatenation auxiliary information.

[0174] It should be noted that the spatial dimensions and number of feature channels of the auxiliary information output by the auxiliary coding network are actually the same as those of the original image. However, since the image residual data has been non-uniformly grouped, it may have already undergone spatial resolution reduction processing. Therefore, to ensure successful channel stitching, the auxiliary information needs to be spatially reduced first to obtain extended auxiliary information, and then prior information is constructed based on the extended auxiliary information. The spatial resolution reduction processing of the auxiliary information can be performed in the same way as the spatial resolution reduction processing caused by the non-uniform grouping of the image residual data.

[0175] When constructing prior information based on extended auxiliary information, in order to increase the accuracy of mean prediction, the prior information can also be constructed by combining the image features corresponding to the previously reconstructed extended residual group. If the current extended residual group is the first element in the residual recovery sequence, it means that the extended residual group is the first extended residual group to be recovered. At this time, there is no extended residual group that has been reconstructed. Therefore, the prior information can be constructed directly based on the extended auxiliary information.

[0176] If the current extended residual group is not the first element in the residual recovery sequence, then there exists an already reconstructed extended residual group. Therefore, the image reconstruction features of the recovered extended residual group can be convolved using a convolutional layer. Then, the extended auxiliary information and the convolutional results corresponding to the image reconstruction features of the recovered extended residual group are concatenated channel by channel. Prior information is then constructed based on the concatenated auxiliary information. When constructing the prior information and selecting recovered extended residual groups, all recovered extended residual groups can be selected, or only a portion of them can be selected.

[0177] In one possible implementation of this embodiment, when performing result stitching, feature enhancement can be performed on the image reconstruction features of the recovered extended residual group to further improve the prediction effect. The step of stitching the extended auxiliary information with the convolution processing results corresponding to the image reconstruction features of the recovered extended residual group to obtain stitching auxiliary information may include:

[0178] Obtain the image reconstruction features corresponding to the recovered extended residual group;

[0179] The image reconstruction features are enhanced to obtain enhanced reconstruction features;

[0180] The auxiliary information is concatenated with the convolutional processing result corresponding to the enhanced reconstruction feature to obtain the concatenated auxiliary information.

[0181] In practical applications, feature enhancement of image reconstruction features can be achieved by performing operations such as missing value processing and outlier processing on the image reconstruction features.

[0182] It is understandable that performing feature enhancement on the image reconstruction features before concatenating auxiliary information with image reconstruction features to obtain enhanced reconstruction features can increase the reliability of enhanced reconstruction features, thereby improving the reliability of prior information and making mean prediction more accurate when based on prior information.

[0183] In practical applications, when enhancing the image reconstruction features corresponding to the recovered extended residual group, the predicted mean, auxiliary information, image residual data, and / or residual data variance corresponding to the recovered extended residual group can be used. In this case, the step of enhancing the image reconstruction features to obtain enhanced reconstruction features as described in this embodiment can include:

[0184] Obtain the predicted mean, auxiliary information, image residual data, and / or residual data variance corresponding to the recovered extended residual group;

[0185] The image reconstruction features are enhanced by using the predicted mean, auxiliary information, image residual data, and / or residual data variance corresponding to the recovered extended residual group to obtain enhanced reconstruction features.

[0186] It should be noted that the predicted mean corresponding to the recovered extended residual group can be the value obtained when predicting the mean during residual recovery of the recovered extended residual group. The variance of the residual data can be the variance value of the image residual data corresponding to the recovered extended residual group.

[0187] To facilitate understanding, we will now combine... Figure 8 , 9 Explanation of 10 and 11. Figure 8 This is a schematic diagram of non-uniform channel and uniform spatial grouping, as shown below. Figure 8As shown, feature y contains a total of 4 feature channels. When grouping feature y, it is first grouped based on feature channels, dividing it into two channel groups, y1 and y2, in a non-uniform manner. y1 contains image residual data corresponding to one feature channel, and y2 contains image residual data corresponding to three feature channels. Then, it is spatially divided uniformly based on a 2x2 phase division specification (the spatial labels of each phase are marked by different colors), so that y1 corresponds to 4 spatial division groups and y2 corresponds to 4 spatial division groups, resulting in a total of 8 extended residual groups. After that, a residual recovery sequence is constructed based on the divided extended residual groups. Residual recovery is performed based on the residual recovery sequence to obtain the image reconstruction features corresponding to each extended residual group. The spatial resolution of these features is increased, and the spatial domain is reset to obtain the image reconstruction features y1_hat corresponding to y1 and y2_hat corresponding to y2. Finally, y1_hat and y2_hat are aggregated according to feature channels, and image reconstruction can be performed based on the aggregated features.

[0188] Figure 9 This diagram illustrates the uniform channel and non-uniform spatial partitioning of feature y. Feature y contains a total of 4 feature channels. When grouping feature y, we now perform uniform grouping based on feature channels, dividing it into two channel groups: y1 and y2. y1 and y2 each contain the image residual data corresponding to two feature channels. Figure 9 As shown, for y1 or y2, the spatial domain is non-uniformly divided using a 2x2 phase division specification. This results in three spatial domain division groups: Y_1, Y_2, and Y_3. Y_3 has more spatial domain positions than Y_1 and Y_2. Subsequently, a residual recovery sequence is constructed based on Y_1, Y_2, and Y_3. Residual recovery is then performed based on the residual recovery sequence to obtain the image reconstruction features Y_hat_1, Y_hat_2, and Y_hat_3 corresponding to Y_1, Y_2, and Y_3, respectively.

[0189] Figure 10 This is a schematic diagram of non-uniform spatial grouping based on a 3x3 phase partitioning scheme. (Example) Figure 10 As shown, a 3x3 phase is introduced for phase partitioning, dividing the 6x6 feature into 1x1 sub-blocks. Then, the 3x3 phase is used for further phase partitioning, resulting in four 3x3 phases. Each sub-block within a phase is spatially labeled (labeled 1, 2, and 3). Then, the image data is erased and added to the three spatial partition groups (Y_1, Y_2, and Y_3) according to the spatial labels. Then, a residual recovery sequence is constructed based on Y_1, Y_2, and Y_3. Residual recovery is performed based on the residual recovery sequence to obtain the image reconstruction features Y_hat_1, Y_hat_2, and Y_hat_3 corresponding to Y_1, Y_2, and Y_3, respectively.

[0190] During implementation, the spatial domain partitioning rules corresponding to each channel partitioning group may differ (i.e., the number of spatial domain partitioning groups and the corresponding phase positions of each group may be different when partitioning the spatial domain for different channel partitioning groups). Therefore, the network structure required for residual recovery of the extended residual groups corresponding to different channel partitioning groups, as well as the spatial domain partitioning rules within the network, may also differ. To ensure the normal progress of reconstruction, different networks can be used for reconstruction, such as... Figure 11 As shown, Figure 11 This is a schematic diagram of a multi-network residual reconstruction structure. Two networks are set up for residual reconstruction: Spatial4Net-1 and Spatial2Net-2. Spatial4Net-1 can be used to reconstruct the extended residual group corresponding to y1, while Spatial2Net-2 can be used to reconstruct the extended residual group corresponding to y2.

[0191] This embodiment constructs a residual recovery sequence based on at least one extended residual group; based on the residual recovery sequence, residual recovery is performed on each of the at least one extended residual group to obtain the image reconstruction features corresponding to each extended residual group. Since the residual recovery sequence is constructed based on at least one extended residual group, the order of residual recovery can be determined through the residual recovery sequence, allowing for rapid determination of whether there are recovered extended residual groups. Furthermore, if recovered extended residual groups exist, more accurate prior information can be constructed based on the image feature data corresponding to the recovered extended residual groups.

[0192] This invention provides an encoding method based on multi-step context prediction, referring to... Figure 12 , Figure 12 This is a flowchart illustrating the first embodiment of an encoding method based on multi-step context prediction according to the present invention.

[0193] In this embodiment, the encoding method based on multi-step context prediction includes the following steps:

[0194] Step S100: Non-uniformly group the image features corresponding to the image to be encoded to obtain at least one extended feature group.

[0195] When generating an image bitstream, the encoding device groups image features, performs mean prediction on each group, and calculates feature residuals based on the predicted means. This reduces the time complexity of predicting the mean during encoding. Finally, the encoding device directly encodes the generated image residual data into the image bitstream. If necessary, the encoding device can also write the grouping rules used in the non-uniform grouping process into the image bitstream for the decoding device to access. The method used by the encoding device in performing non-uniform grouping is consistent with the method used in the decoding process described above; specific implementations can be found in the embodiments of the multi-step context prediction-based decoding method described above, and will not be repeated here.

[0196] Step S200: Perform residual calculation on each of the at least one extended feature group to obtain the image residual data corresponding to each extended feature group.

[0197] It should be noted that the image residual data corresponding to the extended feature group can be obtained by performing residual calculation on each extended feature group separately, which can be achieved by predicting the mean of the extended feature group, and then subtracting the predicted mean from the image features in the extended feature group to obtain the image residual data corresponding to the extended feature group.

[0198] The process used by the encoding device when performing residual calculation is the reverse process of the decoding device when performing residual recovery. The specific implementation method can refer to the decoding method based on multi-step context prediction provided in any of the above embodiments, which will not be elaborated here.

[0199] Step S300: Generate an image bitstream based on the image residual data, and send the image bitstream to the image decoding end.

[0200] It should be noted that generating an image bitstream from image residual data can be achieved by writing the image residual data into the image bitstream through entropy encoding.

[0201] In practical applications, image encoding can also be done by writing segments into the image bitstream. For example, the original image x is processed by the corresponding analysis and transformation network to generate y, and y is processed by feature separation to obtain N stages of features to be encoded y_i (i = 1...N) (i.e. at least one extended feature group).

[0202] The subsequent encoding order of the decision components, taking the Y component as an example (generally, Y is encoded first):

[0203] 1. For the first-stage feature y_1, y is processed by a hyperparameter coding network to obtain z. z is quantized into z_hat and encoded into bitstream 1. Based on bitstream 1, z_hat is decoded. Based on z_hat (features from other stages cannot be used in the first stage), the mean mu_1 is obtained through a mean prediction network. Based on mu_1, the residual r_1 of feature y_1 is obtained. r_1 is quantized to obtain r_q_1 and encoded into bitstream 2. Based on bitstream 2, r_q_1 is decoded and dequantized to obtain r_hat_1. Combined with the mean mu_1, feature recovery is performed to obtain the reconstructed feature y_hat_1.

[0204] 2. For the i-th stage feature y_i (i>1), z_hat is obtained by decoding based on bitstream 1, and the reconstructed features y_hat_1 to y_hat_i-1 from the previous stages are used to obtain the mean mu_i through the mean prediction network. Based on mu_i, the residual r_i of feature y_i is obtained, and r_q_i is encoded into bitstream 2. Based on bitstream 2, r_hat_i is obtained by decoding, and feature recovery is performed by combining it with the mean mu_i to obtain the reconstructed feature y_hat_i.

[0205] 3. The features y_hat are obtained by clustering y_hat_i (i = 1…N). The features y_hat are then processed by a synthesis transform network to obtain the Y component reconstructed image.

[0206] This embodiment obtains at least one extended feature group by non-uniformly grouping the image features corresponding to the image to be encoded; residual calculation is performed on each of the at least one extended feature group to obtain image residual data corresponding to each extended feature group; an image bitstream is generated based on the image residual data and sent to the image decoding end. By non-uniformly grouping the image features of the image to be encoded into at least one extended residual group, residual calculation can be performed on the entire group, thereby improving the overall residual calculation efficiency and reducing the time complexity.

[0207] Furthermore, embodiments of the present invention also propose a storage medium storing a decoding program and / or an encoding program. When the decoding program is executed by a processor, it implements the steps of the decoding method based on multi-step context prediction as described above. When the encoding program is executed by a processor, it implements the steps of the encoding method based on multi-step context prediction as described above.

[0208] Reference Figure 13 , Figure 13 This is a structural block diagram of the first embodiment of the decoding device of the present invention.

[0209] like Figure 13 As shown, the decoding device proposed in this embodiment of the invention includes:

[0210] Grouping module 10 is used to perform non-uniform grouping of image residual data extracted from image bitstream to obtain at least one extended residual group.

[0211] The recovery module 20 is used to perform residual recovery on the at least one extended residual group respectively to obtain the image reconstruction features corresponding to each extended residual group.

[0212] The reconstruction module 30 is used to perform image reconstruction based on the image reconstruction features corresponding to each extended residual group to obtain reconstructed image blocks.

[0213] This embodiment obtains at least one extended residual group by non-uniformly grouping the image residual data extracted from the image bitstream; residual recovery is then performed on each of the at least one extended residual group to obtain the image reconstruction features corresponding to each extended residual group; image reconstruction is then performed based on the image reconstruction features corresponding to each extended residual group to obtain reconstructed image blocks. Because the image residual data extracted from the image bitstream is first grouped, it ensures that residual recovery processing can be performed on entire groups, thereby improving the overall residual recovery calculation efficiency and reducing time complexity. Furthermore, the non-uniform grouping based on the detailed information contained in each feature data adapts to the uneven distribution of feature data, thus improving coding performance and reducing complexity.

[0214] In one possible implementation of this embodiment, the grouping module 10 is further configured to extract image residual data from the image bitstream; divide the image residual data into at least one channel partitioning group according to the feature channel corresponding to each image residual data; and perform spatial domain partitioning on the image residual data in the channel partitioning group to obtain at least one extended residual group.

[0215] In one possible implementation of this embodiment, the grouping module 10 is further configured to classify the data according to the corresponding feature channels to obtain the image residual data corresponding to each feature channel; determine at least one data group and the feature channel number corresponding to each data group according to the channel division rules; and add the image residual data corresponding to each feature channel to the at least one data group based on the feature channel number to obtain at least one channel division group.

[0216] In one possible implementation of this embodiment, the channel division rule is a non-uniform division rule.

[0217] In one possible implementation of this embodiment, the grouping module 10 is further configured to perform phase division on the image residual data in the channel division group according to the phase division specification to obtain multiple feature phases corresponding to the channel division group; determine at least one spatial division group and the spatial domain label corresponding to each spatial division group based on the spatial domain division rule; assign spatial domain labels to the feature phases according to the number of groups in the at least one spatial division group; and add the image residual data in the channel division group to the at least one spatial division group based on the labeled feature phases to obtain at least one extended residual group.

[0218] In one possible implementation of this embodiment, the spatial domain partitioning rule is a non-uniform partitioning rule.

[0219] In one possible implementation of this embodiment, different channel division groups correspond to different spatial domain division rules.

[0220] In one possible implementation of this embodiment, different channel division groups correspond to different phase division specifications.

[0221] In one possible implementation of this embodiment, the recovery module 20 is further configured to construct a residual recovery sequence based on the at least one extended residual group; and perform residual recovery on the at least one extended residual group based on the residual recovery sequence to obtain the image reconstruction features corresponding to each extended residual group.

[0222] In one possible implementation of this embodiment, the recovery module 20 is further configured to traverse the residual recovery sequence to obtain the current extended residual group; obtain auxiliary information output by the auxiliary coding network; construct prior information based on the auxiliary information; perform residual recovery on the current extended residual group based on the prior information to obtain the image reconstruction features corresponding to the current extended residual group; and obtain the image reconstruction features corresponding to each extended residual group at the end of the traversal.

[0223] In one possible implementation of this embodiment, the recovery module 20 is further configured to perform spatial resolution reduction processing on the auxiliary information to obtain extended auxiliary information; detect whether the current extended residual group is the first element in the residual recovery sequence; if it is the first element, construct prior information based on the extended auxiliary information; if it is not the first element, concatenate the extended auxiliary information with the convolution processing result corresponding to the image reconstruction features of the recovered extended residual group to obtain concatenation auxiliary information, and construct prior information based on the concatenation auxiliary information.

[0224] In one possible implementation of this embodiment, the recovery module 20 is further configured to obtain image reconstruction features corresponding to the recovered extended residual group; perform feature enhancement on the image reconstruction features to obtain enhanced reconstruction features; and concatenate the auxiliary information with the convolution processing result corresponding to the enhanced reconstruction features to obtain concatenation auxiliary information.

[0225] In one possible implementation of this embodiment, the recovery module 20 is further configured to obtain the predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the recovered extended residual group; and to perform feature enhancement on the image reconstruction features based on the predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the recovered extended residual group to obtain enhanced reconstruction features.

[0226] In one possible implementation of this embodiment, the reconstruction module 30 is further configured to perform amplification of spatial resolution processing on the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data; and to perform image reconstruction based on the reconstruction feature data to obtain reconstructed image blocks.

[0227] Reference Figure 14 , Figure 14 This is a structural block diagram of the second embodiment of the encoding device of the present invention.

[0228] like Figure 14 As shown, the encoding device proposed in this embodiment of the invention includes:

[0229] Grouping module 10 is used to non-uniformly group the image features corresponding to the image to be encoded to obtain at least one extended feature group;

[0230] The calculation module 20 is used to perform residual calculation on the at least one extended feature group respectively to obtain the image residual data corresponding to each extended feature group;

[0231] The generation module 30 is used to generate an image bitstream based on the image residual data and send the image bitstream to the image decoding end.

[0232] This embodiment obtains at least one extended feature group by non-uniformly grouping the image features corresponding to the image to be encoded; residual calculation is performed on each of the at least one extended feature group to obtain image residual data corresponding to each extended feature group; an image bitstream is generated based on the image residual data and sent to the image decoding end. By non-uniformly grouping the image features of the image to be encoded into at least one extended residual group, residual calculation can be performed on the entire group, thereby improving the overall residual calculation efficiency and reducing the time complexity.

[0233] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0234] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0235] In addition, for technical details not described in detail in this embodiment, please refer to the decoding method or encoding method based on multi-step context prediction provided in any embodiment of the present invention, which will not be repeated here.

[0236] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0237] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0238] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0239] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A decoding method based on multi-step context prediction, characterized in that, The decoding method based on multi-step context prediction includes: The image residual data extracted from the image bitstream is non-uniformly grouped to obtain at least one extended residual group; Residual recovery is performed on each of the at least one extended residual group to obtain the image reconstruction features corresponding to each extended residual group; Image reconstruction is performed based on the image reconstruction features corresponding to each extended residual group to obtain reconstructed image patches; The step of non-uniformly grouping the image residual data extracted from the image bitstream to obtain at least one extended residual group includes: Extracting image residual data from the image bitstream; Based on the feature channels corresponding to each image residual data, the image residual data is divided into at least one channel group; Spatial domain partitioning is performed on the image residual data in the channel partitioning group to obtain at least one extended residual group.

2. The decoding method based on multi-step context prediction as described in claim 1, characterized in that, The step of dividing the image residual data into at least one channel group based on the feature channels corresponding to each image residual data includes: The image residual data corresponding to each feature channel is obtained by classifying the data according to the corresponding feature channels. Determine at least one data group and the corresponding feature channel number for each data group according to the channel division rules; Based on the feature channel labels, the image residual data corresponding to each feature channel is added to the at least one data group to obtain at least one channel division group.

3. The decoding method based on multi-step context prediction as described in claim 2, characterized in that, The channel division rule is a non-uniform division rule.

4. The decoding method based on multi-step context prediction as described in claim 1, characterized in that, The step of spatially partitioning the image residual data in the channel partitioning group to obtain at least one extended residual group includes: The image residual data in the channel division group is phase-divided according to the phase division specification to obtain multiple feature phases corresponding to the channel division group; Based on the airspace partitioning rules, at least one airspace partitioning group and the airspace label corresponding to each airspace partitioning group are determined. The feature phase is spatially labeled according to the number of groups in the at least one spatial division group; Based on the labeled characteristic phase, the image residual data in the channel partitioning group is added to the at least one spatial partitioning group to obtain at least one extended residual group.

5. The decoding method based on multi-step context prediction as described in claim 4, characterized in that, The spatial domain partitioning rule is a non-uniform partitioning rule.

6. The decoding method based on multi-step context prediction as described in claim 4, characterized in that, Different channel division groups correspond to different airspace division rules.

7. The decoding method based on multi-step context prediction as described in claim 4, characterized in that, Different channel division groups correspond to different phase division specifications.

8. The decoding method based on multi-step context prediction as described in claim 1, characterized in that, The step of performing residual recovery on each of the at least one extended residual group to obtain the image reconstruction features corresponding to each extended residual group includes: Construct a residual recovery sequence based on the at least one extended residual set; Based on the residual recovery sequence, residual recovery is performed on the at least one extended residual group to obtain the image reconstruction features corresponding to each extended residual group.

9. The decoding method based on multi-step context prediction as described in claim 8, characterized in that, The step of performing residual recovery on the at least one extended residual group based on the residual recovery sequence to obtain the image reconstruction features corresponding to each extended residual group includes: The residual recovery sequence is traversed to obtain the current extended residual set; Obtain auxiliary information output by the auxiliary coding network; Construct prior information based on the auxiliary information; Based on the prior information, residual recovery is performed on the current extended residual group to obtain the image reconstruction features corresponding to the current extended residual group; At the end of the traversal, the image reconstruction features corresponding to each extended residual group are obtained.

10. The decoding method based on multi-step context prediction as described in claim 9, characterized in that, The step of constructing prior information based on the auxiliary information includes: The auxiliary information is processed to reduce its spatial resolution, thereby obtaining extended auxiliary information; Detect whether the current extended residual group is the first element in the residual recovery sequence; If it is the first element, then prior information is constructed based on the extended auxiliary information; If it is not the first element, the extended auxiliary information is concatenated with the convolution processing result corresponding to the image reconstruction feature of the recovered extended residual group to obtain the concatenation auxiliary information, and prior information is constructed based on the concatenation auxiliary information.

11. The decoding method based on multi-step context prediction as described in claim 10, characterized in that, The step of concatenating the extended auxiliary information with the convolutional processing result corresponding to the image reconstruction features of the recovered extended residual group to obtain concatenation auxiliary information includes: Obtain the image reconstruction features corresponding to the recovered extended residual group; The image reconstruction features are enhanced to obtain enhanced reconstruction features; The auxiliary information is concatenated with the convolutional processing result corresponding to the enhanced reconstruction feature to obtain the concatenated auxiliary information.

12. The decoding method based on multi-step context prediction as described in claim 11, characterized in that, The step of enhancing the image reconstruction features to obtain enhanced reconstruction features includes: Obtain the predicted mean, auxiliary information, image residual data, and / or residual data variance corresponding to the recovered extended residual group; The image reconstruction features are enhanced by using the predicted mean, auxiliary information, image residual data, and / or residual data variance corresponding to the recovered extended residual group to obtain enhanced reconstruction features.

13. The decoding method based on multi-step context prediction as described in any one of claims 1-12, characterized in that, The process of reconstructing the image based on the image reconstruction features corresponding to each extended residual group to obtain reconstructed image patches includes: The image reconstruction features corresponding to each extended residual group are processed to increase the spatial resolution to obtain reconstruction feature data. Image reconstruction is performed based on the reconstructed feature data to obtain reconstructed image blocks.

14. A coding method based on multi-step context prediction, characterized in that, The encoding method based on multi-step context prediction includes: The image features corresponding to the image to be encoded are non-uniformly grouped to obtain at least one extended feature group; Residual calculations are performed on each of the at least one extended feature group to obtain image residual data corresponding to each extended feature group; An image bitstream is generated based on the image residual data, and the image bitstream is sent to the image decoding end; The step of non-uniformly grouping the image features corresponding to the image to be encoded to obtain at least one extended feature group includes: Obtain the image features corresponding to the image to be encoded; Based on the feature channels corresponding to each image feature, the image features are divided into at least one channel group; Spatial domain division is performed on the image features in the channel division group to obtain at least one extended feature group.

15. A decoding device, characterized in that, The decoding device includes: The grouping module is used to perform non-uniform grouping of the image residual data extracted from the image bitstream to obtain at least one extended residual group; The recovery module is used to perform residual recovery on the at least one extended residual group respectively to obtain the image reconstruction features corresponding to each extended residual group; The reconstruction module is used to reconstruct images based on the image reconstruction features corresponding to each extended residual group, and obtain reconstructed image patches. The grouping module is further configured to extract image residual data from the image bitstream; divide the image residual data into at least one channel partitioning group according to the feature channel corresponding to each image residual data; and perform spatial partitioning on the image residual data in the channel partitioning group to obtain at least one extended residual group.

16. An encoding device, characterized in that, The encoding device includes: The grouping module is used to non-uniformly group the image features corresponding to the image to be encoded to obtain at least one extended feature group. The calculation module is used to perform residual calculation on the at least one extended feature group respectively to obtain the image residual data corresponding to each extended feature group; The generation module is used to generate an image bitstream based on the image residual data and send the image bitstream to the image decoding end; The grouping module is also used to obtain image features corresponding to the image to be encoded; divide the image features into at least one channel group according to the feature channels corresponding to each image feature; and perform spatial domain division on the image features in the channel group to obtain at least one extended feature group.

17. A decoding device, characterized in that, The decoding device includes: a processor, a memory, and a decoding program stored in the memory and executable on the processor, wherein when the decoding program is executed by the processor, it implements the steps of the decoding method based on multi-step context prediction as described in any one of claims 1-13.

18. An encoding device, characterized in that, The encoding device includes: a processor, a memory, and a decoding program and / or an encoding program stored in the memory and executable on the processor. When the decoding program is executed by the processor, it implements the steps of the decoding method based on multi-step context prediction as described in any one of claims 1-13. When the encoding program is executed by the processor, it implements the steps of the encoding method based on multi-step context prediction as described in claim 14.

19. A storage medium, characterized in that, The storage medium stores a decoding program and / or an encoding program. When the decoding program is executed, it implements the steps of the decoding method based on multi-step context prediction as described in any one of claims 1-13. When the encoding program is executed, it implements the steps of the encoding method based on multi-step context prediction as described in claim 14.

Images