Image decoding and encoding method, apparatus, device, and storage medium

Abstract

This invention belongs to the field of image processing technology and discloses an image decoding and encoding method, apparatus, device, and storage medium. This application extracts image residual data or extended residual data groups from an image bitstream; based on the extracted image residual data or extended residual data, multiple extended residual groups are obtained; residual recovery is performed on each of the multiple extended residual groups to obtain image reconstruction features corresponding to each extended residual group; the spatial resolution of the image reconstruction features corresponding to each extended residual group is increased to obtain reconstruction feature data; image reconstruction is performed based on the reconstruction feature data to obtain reconstructed image blocks. Since the obtained extended residual data is residual data that has undergone spatial resolution reduction processing, residual recovery processing can be performed on groups at low resolution, thereby improving the overall residual recovery calculation efficiency and reducing time complexity.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 2023103660642, filed on March 31, 2023, entitled "Image Decoding and Encoding Method, Apparatus, Device and Storage Medium". Technical Field

[0002] This invention relates to the field of image processing technology, and in particular to an image decoding and encoding method, apparatus, device, and storage medium. Background Technology

[0003] In deep learning-based image compression schemes, the mainstream approach is to use already decoded feature points as prior information to predict the mean of the currently decoded feature points in order to reduce spatial redundancy in the image. The mainstream schemes generally use serial or wavefront encoding and decoding methods, and the seriality increases with the resolution of the features, resulting in high overall execution time complexity.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide an image decoding and encoding method, apparatus, device, and storage medium, aiming to solve the technical problem of high time complexity in mean prediction during the image encoding and decoding process in the prior art.

[0006] To achieve the above objectives, the present invention provides an image decoding method, the method comprising the following steps: Image residual data or extended residual data is extracted from the image bitstream, and multiple extended residual groups are obtained based on the extracted image residual data or extended residual data; Residual recovery is performed on each of the multiple extended residual groups to obtain the image reconstruction features corresponding to each extended residual group; The image reconstruction features corresponding to each extended residual group are processed to increase the spatial resolution to obtain reconstruction feature data. Image reconstruction is performed based on the reconstructed feature data to obtain reconstructed image blocks.

[0007] In one possible implementation of this application, the step of extracting image residual data or extended residual data from the image bitstream, and obtaining multiple extended residual groups based on the extracted image residual data or extended residual data, includes: Extract the image residual data from the image bitstream; The image residual data is processed to reduce the spatial resolution to obtain the extended residual data. The process of increasing the spatial resolution is the inverse process of reducing the spatial resolution. The extended residual data is grouped to obtain multiple extended residual groups.

[0008] In one possible implementation of this application, the step of reducing the spatial resolution of the image residual data to obtain the extended residual data includes: The spatial size corresponding to the image residual data is reduced, and / or the number of feature channels corresponding to the image residual data is increased to obtain extended residual data.

[0009] In one possible implementation of this application, reducing the spatial size corresponding to the image residual data and / or increasing the number of feature channels corresponding to the image residual data to obtain the extended residual data includes: The spatial size of the image residual data is reduced and / or the number of feature channels corresponding to the image residual data is increased based on the spatial information corresponding to the image residual data to obtain the extended residual data.

[0010] In one possible implementation of this application, the step of performing residual recovery on the plurality of extended residual groups to obtain the image reconstruction features corresponding to each extended residual group includes: Construct a residual recovery sequence based on the plurality of extended residual sets; Based on the residual recovery sequence, residual recovery is performed on the multiple extended residual groups respectively to obtain the image reconstruction features corresponding to each extended residual group.

[0011] In one possible implementation of this application, the step of performing residual recovery on the plurality of extended residual groups based on the residual recovery sequence to obtain the image reconstruction features corresponding to each extended residual group includes: The residual recovery sequence is traversed to obtain the current extended residual set; Obtain auxiliary information output by the auxiliary coding network; Construct prior information based on the auxiliary information; Based on the prior information, residual recovery is performed on the current extended residual group to obtain the image reconstruction features corresponding to the current extended residual group; At the end of the traversal, the image reconstruction features corresponding to each extended residual group are obtained.

[0012] In one possible implementation of this application, constructing prior information based on the auxiliary information includes: Obtain extended auxiliary information; Detect whether the current extended residual group is the first element in the residual recovery sequence; If it is the first element, then prior information is constructed based on the extended auxiliary information; If it is not the first element, the extended auxiliary information is concatenated with the convolution processing result corresponding to the image reconstruction feature of the recovered extended residual group to obtain the concatenation auxiliary information, and prior information is constructed based on the concatenation auxiliary information.

[0013] Furthermore, to achieve the above objectives, the present invention also proposes an image encoding method, the image encoding method comprising: The spatial resolution of the image features corresponding to the image to be encoded is reduced to obtain extended image features. The extended image features are grouped to obtain multiple extended feature groups; Residual calculations are performed on the multiple extended feature groups respectively to obtain the image residual data corresponding to each extended feature group; An image bitstream is generated based on the image residual data, and the image bitstream is sent to the image decoding end.

[0014] In one possible implementation of this application, the step of reducing the spatial resolution of the image features corresponding to the image to be encoded to obtain extended image features includes: Obtain the image features corresponding to the image to be encoded; The data in the image features are processed to reduce the spatial resolution, thereby obtaining extended image features.

[0015] In one possible implementation of this application, the step of reducing the spatial resolution of the image features corresponding to the image to be encoded to obtain extended image features includes: The spatial dimensions corresponding to the image features of the image to be encoded are reduced, and / or the number of feature channels corresponding to the image features is increased, to obtain extended image features.

[0016] In one possible implementation of this application, reducing the spatial size corresponding to the image features of the image to be encoded and / or increasing the number of feature channels corresponding to the image features to obtain extended image features includes: Based on the spatial domain information corresponding to the image features of the image to be encoded, the spatial size corresponding to the image features is reduced, and / or the number of feature channels corresponding to the image features is increased, to obtain an extended feature set.

[0017] In one possible implementation of this application, the step of performing residual calculations on the plurality of extended feature groups to obtain image residual data corresponding to each extended feature group includes: Construct a residual calculation sequence based on the multiple extended feature groups; Based on the residual calculation sequence, residual calculation is performed on the multiple extended feature groups respectively to obtain the image residual data corresponding to each extended feature group.

[0018] In one possible implementation of this application, the step of performing residual calculations on the plurality of extended feature groups based on the residual calculation sequence to obtain image residual data corresponding to each extended feature group includes: The residual calculation sequence is traversed to obtain the current extended feature group; Obtain auxiliary information output by the auxiliary coding network; Construct prior information based on the auxiliary information; Based on the prior information, residual calculation is performed on the current extended feature group to obtain the image residual data corresponding to the current extended feature group; At the end of the traversal, the image residual data corresponding to each extended feature group is obtained.

[0019] In one possible implementation of this application, grouping the extended image features to obtain multiple extended feature groups includes: Based on the feature channels corresponding to the extended feature data, the extended image features are grouped to obtain multiple extended feature groups.

[0020] In one possible implementation of this application, the step of generating an image bitstream based on the image residual data and sending the image bitstream to the image decoding end includes: The spatial resolution of the image reconstruction features corresponding to each extended feature group is increased to obtain the image residual data corresponding to the image to be encoded. The process of increasing spatial resolution is the inverse process of reducing spatial resolution. An image bitstream is generated based on the image residual data corresponding to the image to be encoded, and the image bitstream is sent to the image decoding end.

[0021] Furthermore, to achieve the above objectives, the present invention also proposes an image decoding device, the image decoding device comprising: The bitstream decoding module is used to extract image residual data or extended residual data from the image bitstream, and obtain multiple extended residual groups based on the extracted image residual data or extended residual data; The residual recovery module is used to perform residual recovery on the multiple extended residual groups respectively to obtain the image reconstruction features corresponding to each extended residual group; The data combination module is used to enlarge the spatial resolution of the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data. The image reconstruction module is used to reconstruct the image based on the reconstruction feature data to obtain reconstructed image blocks.

[0022] Furthermore, to achieve the above objectives, the present invention also proposes an image encoding device, the image encoding device comprising: The feature extraction module is used to reduce the spatial resolution of the image features corresponding to the image to be encoded, thereby obtaining extended image features. The data grouping module is used to group the extended image features to obtain multiple extended feature groups; The residual calculation module is used to perform residual calculation on the multiple extended feature groups respectively to obtain the image residual data corresponding to each extended feature group; The bitstream generation module is used to generate an image bitstream based on the image residual data and send the image bitstream to the image decoding end.

[0023] In addition, to achieve the above objectives, the present invention also proposes a decoding device, which includes: a processor, a memory, and an image decoding program stored in the memory and executable on the processor. When the image decoding program is executed by the processor, it implements the steps of the image decoding method described above.

[0024] Furthermore, to achieve the above objectives, the present invention also proposes an encoding device, the encoding device comprising: a processor, a memory, and an image decoding program and / or an image encoding program stored in the memory and executable on the processor, wherein when the image decoding program is executed by the processor, it implements the steps of the image decoding method as described above, and when the image encoding program is executed by the processor, it implements the steps of the image encoding method as described above.

[0025] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing an image decoding program and / or an image encoding program, wherein the image decoding program, when executed, implements the steps of the image decoding method as described above, and the image encoding program, when executed, implements the steps of the image encoding method as described above.

[0026] Furthermore, to achieve the above objectives, the present invention also proposes a computer program configured to, when executed by a processor having memory, implement the steps of the image decoding method described above, or implement the steps of the image encoding method described above.

[0027] Furthermore, to achieve the above objectives, the present invention also proposes a computer program product, including computer program instructions configured to, when executed by a processor having memory, implement the steps of the image decoding method as described above, or implement the steps of the image encoding method as described above.

[0028] This invention extracts image residual data or extended residual data from an image bitstream, and obtains multiple extended residual groups based on the extracted image residual data or extended residual data. Residual recovery is then performed on each of the extended residual groups to obtain image reconstruction features corresponding to each extended residual group. The spatial resolution of the image reconstruction features corresponding to each extended residual group is increased to obtain reconstruction feature data. Image reconstruction is then performed based on the reconstruction feature data to obtain reconstructed image blocks. Since the obtained extended residual data is residual data that has undergone spatial resolution reduction processing, residual recovery processing can be performed in groups at low resolution, thereby improving the overall residual recovery calculation efficiency and reducing time complexity. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the structure of an electronic device in the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the first embodiment of the image decoding method of the present invention; Figure 3 This is an overall framework diagram of image compression according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the second embodiment of the image decoding method of the present invention; Figure 5 This is a schematic diagram of the spatial resolution processing flow according to an embodiment of the present invention. Figure 6 This is a flowchart illustrating the third embodiment of the image decoding method of the present invention; Figure 7 This is a schematic diagram of the image decoding grouping execution process according to an embodiment of the present invention; Figure 8 This is a schematic diagram of a secondary grouping execution process according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the feature enhancement grouping execution process according to an embodiment of the present invention; Figure 10 This is a flowchart illustrating the first embodiment of the image encoding method of the present invention; Figure 11 This is a flowchart illustrating the second embodiment of the image encoding method of the present invention; Figure 12 This is a structural block diagram of the first embodiment of the image decoding device of the present invention; Figure 13 This is a structural block diagram of the first embodiment of the image encoding device of the present invention.

[0030] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0031] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0032] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a decoding or encoding device in the hardware operating environment involved in the embodiments of the present invention.

[0033] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0034] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0035] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and an image decoding program and / or an image encoding program.

[0036] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the decoding device or the encoding device. The electronic device calls the image decoding program or image encoding program stored in the memory 1005 through the processor 1001 and executes the image decoding method or image encoding method provided in the embodiments of the present invention.

[0037] This invention provides an image decoding method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of an image decoding method according to the present invention.

[0038] In this embodiment, the image decoding method includes the following steps: Step S10: Extract image residual data or extended residual data from the image bitstream, and obtain multiple extended residual groups based on the extracted image residual data or extended residual data.

[0039] It should be noted that the execution subject of this embodiment can be a decoding device for decoding image data. The decoding device can be a personal computer, server or other electronic device. Of course, it can also be other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment and the following embodiments, the image decoding method of the present invention is described using a decoding device as an example.

[0040] Since the encoding device typically decodes the encoded image stream after encoding is completed during the image encoding process, and determines whether the parameters used in the encoding need to be adjusted based on the image quality of the decoded image, the execution subject in this embodiment can also be the encoding device.

[0041] It should be noted that the image bitstream can be the bitstream generated by the encoding device after encoding the image data that needs to be compressed. When generating the image bitstream, the encoding device reduces the spatial resolution of the image features to reduce the time complexity of predicting the mean during the encoding process. Finally, the encoding device directly encodes the generated image residual data or extended residual data into the image bitstream. At this point, the decoding device can directly extract the image residual data or extended residual data from the image bitstream. The decoding device can then process the extracted image residual data or extended residual data to obtain multiple extended residual groups, and then perform residual recovery group by group, thereby reducing the time complexity of mean prediction in the image decoding process.

[0042] Technical terms involved in image encoding or decoding include: JPEG (Joint Photographic Experts Group), JPEG-AI (Joint Photographic Experts Group Artificial Intelligence), Entropy Encoding, Neural Network (NN), Convolutional Neural Network (CNN), feature, Rate-Distortion Optimized, etc., which will be explained here.

[0043] JPEG (Joint Photographic Experts Group) is a standard for compressing continuous-tone still images. Files with the extension .jpg or .jpeg are the most commonly used image file format. It primarily employs a joint coding method using predictive coding (e.g., Differential Pulse Code Modulation, DPCM), Discrete Cosine Transform (DCT), and entropy coding to remove redundant image and color data. It is a lossy compression format, capable of compressing images into a very small storage space, but this inevitably causes some damage to the image data. Especially with excessively high compression ratios, the quality of the decompressed image will decrease. Therefore, if high-quality images are desired, excessively high compression ratios should be avoided.

[0044] JPEG-AI aims to create a learning-based image coding standard that provides a single-stream, compact compressed domain representation for human visualization, significantly improving compression efficiency compared to commonly used image coding standards while maintaining the same subjective quality, and offering effective performance for image processing and computer vision tasks. JPEG-AI is geared towards a wide range of applications, such as cloud storage, visual surveillance, autonomous vehicles and devices, image acquisition, storage and management, real-time monitoring of visual data, and media distribution. The goal is to design an encoding solution that significantly improves the compression efficiency of commonly used coding standards while maintaining the same subjective quality, and provides effective compressed domain processing for machine learning-based image processing and computer vision tasks. Other key requirements include hardware / software implementation of user-friendly encoding and decoding, support for 8-bit and 10-bit depths, efficient encoding of images using text and graphics, and progressive decoding.

[0045] Entropy coding is encoding that follows the entropy principle without losing any information during the encoding process. Information entropy is the average amount of information in a source (a measure of uncertainty). Common entropy coding methods include Shannon coding, Huffman coding, and arithmetic coding.

[0046] The neural network referred to in this application is an artificial neural network, not a biological neural network. A neural network is a computational model composed of numerous interconnected nodes (or neurons). In an artificial neural network, neurons can represent different objects, such as features, letters, concepts, or meaningful abstract patterns. There are three types of processing units in a network: input units, output units, and hidden units. Input units receive signals and data from the external world; output units output the system's processing results; hidden units are located between input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units; the representation and processing of information are reflected in the connection relationships between the network's processing units. Artificial neural networks are a non-programmed, brain-like information processing method. Essentially, they achieve parallel and distributed information processing capabilities through network transformations and dynamic behaviors, mimicking the information processing functions of the human brain's nervous system to varying degrees and levels. Currently, in the field of video processing, commonly used neural networks include Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and fully connected networks.

[0047] Convolutional Neural Networks (CNNs) are a type of feedforward neural network and one of the most representative network structures in deep learning. Their artificial neurons can respond to surrounding units within a certain coverage area, exhibiting excellent performance in large-scale image processing. Generally, the basic structure of a CNN consists of two layers: a feature extraction layer (also called a convolutional layer), where the input of each neuron is connected to the local receptive field of the previous layer, extracting local features. Once the local feature is extracted, its positional relationship with other features is determined. The second layer is a feature mapping layer (also called an activation layer). Each computational layer of the network consists of multiple feature maps, each a plane where all neurons have equal weights. Feature mapping structures can use functions such as the Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, and GDN function as activation functions for the convolutional network. Furthermore, since neurons on a single mapping plane share weights, the number of free parameters in the network is reduced. One advantage of CNNs over traditional image processing algorithms is that they avoid complex preprocessing steps (such as extracting artificial features) and can directly input the original image for end-to-end learning. One of the advantages of CNNs over traditional neural networks is that traditional neural networks use a fully connected approach, meaning that all neurons from the input layer to the hidden layer are connected. This results in a huge number of parameters, making network training time-consuming or even difficult. CNNs, on the other hand, avoid this difficulty by using methods such as local connectivity and weight sharing.

[0048] The feature involved in this application is a three-dimensional feature matrix of CxWxH (e.g., ...). Figure 3 As shown, Figure 3 (This is a schematic diagram of the matrix structure in this embodiment). C represents the number of channels, H represents the feature height, and W represents the feature width. The feature matrix can be either the input or the output of the neural network.

[0049] Metrics for evaluating coding efficiency include bitrate, PSNR, MS-SSIM, VMAF, FSIM, PSNR, and HVS, among others. More metrics can be included, but this is not a limitation. A smaller bitstream results in a higher compression ratio; a higher PSNR indicates better image coding efficiency. The discrimination formula for mode selection is essentially a comprehensive evaluation of both factors. The cost corresponding to a mode is: J(mode) = D + λ × R. Here, D represents Distortion, typically measured using the SSE metric, which is the sum of the mean squares of the differences between the reconstructed block and the source image; λ is the Lagrange multiplier; and R is the actual number of bits required for encoding the image block in that mode, including the total number of bits needed for coding mode information, residuals, etc.

[0050] In one possible implementation of this embodiment, the obtained extended residual data is grouped according to the feature channels corresponding to each extended residual data. In this case, step S10 of this embodiment may include: Extract extended residual data from the image bitstream; The extended residual data is grouped based on the feature channels corresponding to the extended residual data to obtain multiple extended residual groups.

[0051] It should be noted that grouping the extended residual data based on the corresponding feature channels to obtain multiple extended residual groups can be achieved by uniformly dividing the extended residual data into multiple groups according to the corresponding feature channels. For example, assuming the total number of feature channels corresponding to the extended residual data is 20, then the extended residual data with feature channels 1-10 can be grouped into one group, and the extended residual data with feature channels 11-20 can be grouped into another group. The number of evenly divided groups can be preset by the administrator of the decoding device; this embodiment does not impose any restrictions on this.

[0052] Of course, the grouping can also be uneven. In this case, the extended residual data can be grouped based on the feature channels corresponding to the extended residual data to obtain multiple extended residual groups. Alternatively, the extended residual data can be divided into multiple groups according to the corresponding feature channels based on preset grouping rules. The preset grouping rules can be set in advance by the administrator of the decoding device according to actual needs. For example, the preset grouping rules can be set to group the first m / n (n is the total number of feature channels, m is a preset value, and the value range is [1,n)) of the extended residual data into one group, and the remaining extended residual data into another group.

[0053] In practical applications, when grouping extended residual data based on the feature channels corresponding to the extended residual data to obtain multiple extended residual groups, it is also possible to divide the extended residual data corresponding to one feature channel into one group. For example, assuming that the total number of feature channels corresponding to the extended residual data is 20, the extended residual data can be divided into 20 groups according to the different feature channels.

[0054] Step S20: Perform residual recovery on the multiple extended residual groups respectively to obtain the image reconstruction features corresponding to each extended residual group.

[0055] It should be noted that residual recovery of the extended residual group to obtain the image reconstruction features corresponding to the extended residual group can be achieved by predicting the mean of the extended residual group, and then summing the mean values ​​predicted from the residual data domains in the extended residual group to obtain the image reconstruction features corresponding to the extended residual group.

[0056] Step S30: Perform spatial resolution amplification processing on the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data.

[0057] It should be noted that since the extended residual data has undergone spatial resolution reduction processing, the spatial size and number of channels corresponding to each data are different from the spatial size and number of channels of the image features obtained by the encoding device from the original image feature extraction. In order to ensure the smooth execution of image reconstruction, the spatial resolution of the image reconstruction features corresponding to each extended residual group can be increased to restore the spatial size and number of channels of the image reconstruction features to be consistent with the image features obtained from the original image feature extraction.

[0058] Among them, the process of increasing spatial resolution can be the reverse process of decreasing spatial resolution in the encoding device.

[0059] Step S40: Reconstruct the image based on the reconstruction feature data to obtain a reconstructed image block.

[0060] It should be noted that after obtaining reconstructed feature data that has the same spatial size and number of channels as the image features corresponding to the original image, image reconstruction can be performed based on the reconstructed feature data to obtain reconstructed image blocks.

[0061] In this process, image reconstruction based on reconstruction feature data to obtain reconstructed image patches can be achieved by using a pre-built synthetic transformation network to perform synthetic transformation processing on the reconstruction feature data, thereby obtaining reconstructed image patches. The synthetic transformation network can be a network built based on deep learning or a neural network.

[0062] To facilitate understanding, we will now combine... Figure 3 This explanation is provided, but it does not limit the scope of this solution. Figure 3 For image compression, the overall framework diagram is as follows: Figure 3 As shown, at the encoding device: x is the input image, which is processed by the main encoder (Analysis TransformNet) to generate a latent representation y. Directly encoding y requires a large bit rate, so a context model net (Context ModelNet) and a hyperparameter encoder net (Hyper Encoder Net) and a hyperparameter decoder net (Hyper Decoder Net) are introduced for prediction to obtain the prediction result μ, and the residual is obtained. Furthermore, the distribution parameters σ of the residuals are obtained through a hyperparameter encoding network (Hyper Encoder net) and a probabilistic hyperparameter decoding network (Hyper Scale Decoder net). These distribution parameters help entropy coding encode data with a lower bit rate. Therefore, at the decoding end, the predicted mean μ and distribution parameters σ for each y are needed to correctly perform entropy decoding. The G-unit component is used to scale resi and distribution parameters σ to control the quantization loss. The quantized residuals are obtained by rounding resi to the nearest integer. Finally, lossless entropy encoding is performed.

[0063] At the decoding device: entropy decoding is performed on the image bitstream to obtain... The invG-unit component supports Scaling is performed, where the scaling factors for the G-unit and invG-unit modules are opposite. The scaled value is then combined with the predicted value μ to obtain... The input to the Synthesis Transform Net at the decoding end yields the reconstructed image patch.

[0064] This embodiment extracts image residual data or extended residual data from the image bitstream, and obtains multiple extended residual groups based on the extracted image residual data or extended residual data. Residual recovery is then performed on each of the multiple extended residual groups to obtain image reconstruction features corresponding to each extended residual group. The spatial resolution of the image reconstruction features corresponding to each extended residual group is increased to obtain reconstruction feature data. Image reconstruction is then performed based on the reconstruction feature data to obtain reconstructed image blocks. Since the obtained extended residual data is residual data that has undergone spatial resolution reduction processing, residual recovery processing can be performed in groups at low resolution, thereby improving the overall residual recovery calculation efficiency and reducing time complexity.

[0065] refer to Figure 4 , Figure 4 This is a flowchart illustrating a second embodiment of an image decoding method according to the present invention.

[0066] Based on the first embodiment described above, step S10 of the image decoding method in this embodiment includes: Step S101: Extract image residual data from the image bitstream.

[0067] It should be noted that the encoding device may process the generated extended residual data to increase the spatial resolution, restoring it to image residual data with spatial size and number of channels consistent with the image features corresponding to the original image. The image residual data is then encoded into the image bitstream. At this time, the decoding device can only extract the image residual data from the image bitstream when decoding the image bitstream.

[0068] Step S102: Perform spatial resolution reduction processing on the image residual data to obtain extended residual data.

[0069] Understandably, to facilitate subsequent grouping and reduce time complexity, after obtaining the image residual data, the spatial resolution of the image residual data can be reduced to obtain expanded residual data. To ensure decoding accuracy, the methods used by the encoding and decoding devices in reducing spatial resolution must be consistent, and increasing spatial resolution is the reverse process of reducing spatial resolution.

[0070] In this process, after reducing the spatial resolution of the image residual data, its spatial size will become smaller. Correspondingly, when processing the image residual data, a smaller convolution kernel can be used. For example, if a 5x5 convolution kernel is used on the original image residual data, it is equivalent to using a 3x3 convolution kernel after reducing the spatial resolution.

[0071] In practical applications, the step of reducing the spatial resolution of image residual data to obtain extended residual data, as described in this embodiment, may include: The spatial size corresponding to the image residual data is reduced, and / or the number of feature channels corresponding to the image residual data is increased to obtain extended residual data.

[0072] It should be noted that the reduction rate of the spatial size corresponding to the image residual data and the increase rate of the number of feature channels corresponding to the image residual data can be preset by the administrator of the decoding device, and this embodiment does not impose any restrictions on this.

[0073] In actual execution, you can set only the reduction range of the spatial size or only the increase range of the number of feature channels, and let the decoding device adaptively adjust the spatial size or the number of feature channels. Of course, you can also choose not to use device adaptation, but set the reduction range of the spatial size and the increase range of the number of feature channels, and let the device execute.

[0074] For example: Suppose the image feature corresponding to the image residual data is y∈R{H,W,C}, where H is the height of the image feature, W is the width of the image feature, and C is the number of feature channels corresponding to the image feature. In this case, H and W can be reduced to half of their original values. In order to keep the amount of data the same, the number of feature channels will become 4 times the original value. Then, the image feature corresponding to the extended residual data obtained at this time can be represented as y∈R{H / 2,W / 2,4C}.

[0075] In specific implementations, spatial resolution reduction can be performed based on the spatial or frequency information corresponding to the image residual data. Alternatively, spatial resolution reduction can be performed through a preset convolutional layer. In this case, the step of reducing the spatial size corresponding to the image residual data and / or increasing the number of feature channels corresponding to the image residual data to obtain expanded residual data, as described in this embodiment, may include: Based on the spatial information corresponding to the image residual data, the spatial size corresponding to the image residual data is reduced, and / or the number of feature channels corresponding to the image residual data is increased, in order to obtain extended residual data; or, Based on the frequency domain information corresponding to the image residual data, the spatial size corresponding to the image residual data is reduced, and / or the number of feature channels corresponding to the image residual data is increased, in order to obtain extended residual data; or, The spatial size corresponding to the image residual data is reduced and / or the number of feature channels corresponding to the image residual data is increased according to the preset convolutional layer to obtain extended residual data.

[0076] To facilitate understanding, we will now combine... Figure 5 This explanation is provided, but it does not limit the scope of this solution. Figure 5 This is a schematic diagram of the spatial resolution processing flow in this embodiment, as shown below. Figure 5 As shown in method (a), SpaceShuffle is one of the methods for reducing spatial resolution. Assuming the input of this process is a1∈R{H,W,C} and its output is a2∈R{H / 2,W / 2,4×C}, its mathematical expression is as follows: Where h represents the index value in the height dimension, ranging from 0 to H-1; w represents the index value in the width dimension, ranging from 0 to W-1; and c represents the index value in the channel dimension, ranging from 0 to C-1. Furthermore, a1[h,w,c] represents the size of the value at the corresponding h, w, or c index position within a1. Figure 5 In method (a), unSpaceShuffle is the inverse process of SpaceShuffle, which is one of the methods for amplifying spatial resolution. In this process, after the input is processed by SpaceShuffle and unSpaceShuffle, the output result is the same as the original input, so the process is lossless. The input and output of the above processing can also be other data, and the embodiments disclosed herein do not limit this.

[0077] As shown in method (b), method (b) is another way to perform spatial resolution processing based on the spatial information corresponding to the image residual data. Pixshuffle's specific sampling method is similar to method (a), but the difference is that the data is interleaved in the channel dimension. The data from the previous channel is divided into four channels according to the spatial domain and arranged sequentially.

[0078] Figure 5 Method (c) in the text describes spatial resolution processing based on the frequency domain information corresponding to the image residual data. As shown in method (c), the Wavelet transform is a two-dimensional wavelet transform that can divide the data into frequency domains and output four frequency domain sub-bands with dimensions [H / 2, W / 2, C], each representing different frequency domain characteristics. The Inv Wavelet transform is the inverse process, synthesizing the frequency domain sub-bands into the original data. This process is lossless.

[0079] Figure 5 Method (d) in the diagram involves spatial resolution processing based on a preset convolutional layer. As shown in method (d), Convolution is a convolutional transformation. The data is directly input into the convolutional layer for processing, resulting in an output of [H / 2, W / 2, 4×C], thus reducing the spatial resolution. The corresponding inverse process also uses a convolutional layer to restore the original image size. However, this method is lossy, and the data processed by both methods differs from the original data.

[0080] In one possible implementation of this embodiment, before performing spatial resolution reduction processing on the image residual data, the image residual data can be grouped first, and then the spatial resolution reduction processing can be performed on each group separately. In this case, step S102 of this embodiment includes: The image residual data is grouped according to the feature channels corresponding to the image residual data to obtain at least one image residual group; The data in the image residual group are processed to reduce the spatial resolution to obtain extended residual data.

[0081] It should be noted that when grouping data based on the feature channels corresponding to the image residual data, the same or similar methods can be used as when grouping the extended residual data. To obtain extended residual data, the data in the image residual groups can be reduced in spatial resolution separately for at least one image residual group, and then the data after reduction in spatial resolution can be aggregated to obtain the extended residual data.

[0082] For example: Suppose the image features corresponding to the image residual data are y∈R{H,W,C}. We can first divide them into two groups according to the feature channels. Then, the image features corresponding to the data in each group of image residuals can be represented as y∈R{H,W,C / 2}.

[0083] Step S103: Group the extended residual data to obtain multiple extended residual groups.

[0084] It should be noted that grouping the extended residual data to obtain multiple extended residual groups can be achieved by uniformly dividing the extended residual data into multiple groups according to the corresponding feature channels. For example, assuming the total number of feature channels corresponding to the extended residual data is 20, then the extended residual data corresponding to feature channels 1-10 can be grouped into one group, and the extended residual data corresponding to feature channels 11-20 can be grouped into another group. The number of evenly divided groups can be preset by the administrator of the decoding device; this embodiment does not impose any restrictions on this.

[0085] Of course, the grouping can also be uneven. The extended residual data can be grouped based on the feature channels corresponding to the extended residual data to obtain multiple extended residual groups. Alternatively, the extended residual data can be divided into multiple groups according to the corresponding feature channels based on preset grouping rules. The preset grouping rules can be set in advance by the administrator of the decoding device according to actual needs. For example, the preset grouping rules can be set to group the first m / n (n is the total number of feature channels, m is a preset value, and the value range is [1,n)) of the extended residual data into one group, and the remaining extended residual data into another group.

[0086] In practical applications, when grouping extended residual data based on the feature channels corresponding to the extended residual data to obtain multiple extended residual groups, it is also possible to divide the extended residual data corresponding to a single feature channel into a group. For example, assuming that the total number of feature channels corresponding to the extended residual data is 20, the extended residual data can be divided into 20 groups according to the different feature channels.

[0087] Before grouping, this embodiment will detect whether the extracted data is image residual data or extended residual data. If it is image residual data, it will first perform spatial resolution reduction processing to ensure that even if the image bitstream input from the encoding end contains image residual data, it can still be grouped and processed normally after processing, thus improving the versatility of the image decoding method in this embodiment.

[0088] refer to Figure 6 , Figure 6 This is a flowchart illustrating a third embodiment of an image decoding method according to the present invention.

[0089] Based on the first embodiment described above, step S20 of the image decoding method in this embodiment includes: Step S201: Construct a residual recovery sequence based on the plurality of extended residual groups.

[0090] It should be noted that after dividing the image into multiple extended residual groups, residual recovery can be performed group by group, thereby reducing the time complexity of mean prediction in the image decoding process. In this case, to determine the residual recovery order for each extended residual group, a residual recovery sequence can be constructed based on the multiple extended residual groups.

[0091] Step S202: Perform residual recovery on the multiple extended residual groups based on the residual recovery sequence to obtain the image reconstruction features corresponding to each extended residual group.

[0092] It should be noted that residual recovery of multiple extended residual groups based on the residual recovery sequence can be performed on the multiple extended residual groups in turn based on the sequence order in the residual recovery sequence.

[0093] In practical applications, residual recovery can be performed sequentially through a sequence traversal. In this case, step S202 of this embodiment may include: The residual recovery sequence is traversed to obtain the current extended residual set; Obtain auxiliary information output by the auxiliary coding network; Construct prior information based on the auxiliary information; Based on the prior information, residual recovery is performed on the current extended residual group to obtain the image reconstruction features corresponding to the current extended residual group; At the end of the traversal, the image reconstruction features corresponding to each extended residual group are obtained.

[0094] It should be noted that traversing the residual recovery sequence to obtain the current extended residual set can be achieved by traversing the residual recovery sequence and using the traversed extended residual set as the current extended residual set. The auxiliary coding network can be as described above. Figure 3The auxiliary network shown is either a Hyper Encoder Net or a Hyper Decoder Net.

[0095] In practical applications, residual recovery based on prior information to obtain the image reconstruction features corresponding to the current extended residual group can be achieved by processing the prior information through a prediction fusion network (Prediction Fusion Net) to obtain the prediction mean, and then adding the prediction mean to the residuals in the current extended residual group to realize residual recovery and obtain the image reconstruction features corresponding to the current extended residual group.

[0096] In practical use, the step of constructing prior information based on the auxiliary information described in this embodiment may include: Obtain extended auxiliary information; Detect whether the current extended residual group is the first element in the residual recovery sequence; If it is the first element, then prior information is constructed based on the extended auxiliary information; If it is not the first element, the extended auxiliary information is concatenated with the convolution processing result corresponding to the image reconstruction feature of the recovered extended residual group to obtain the concatenation auxiliary information, and prior information is constructed based on the concatenation auxiliary information.

[0097] It should be noted that the spatial size and number of feature channels of the auxiliary information output by the auxiliary coding network are actually the same as those of the original image. At this time, the extended residual data has already undergone spatial resolution reduction processing. Therefore, in order to ensure smooth channel stitching, the auxiliary information needs to be processed first to obtain extended auxiliary information, and then prior information is constructed based on the extended auxiliary information.

[0098] When constructing prior information based on extended auxiliary information, in order to increase the accuracy of mean prediction, the prior information can also be constructed by combining the image features corresponding to the previously reconstructed extended residual group. If the current extended residual group is the first element in the residual recovery sequence, it means that the extended residual group is the first extended residual group to be recovered. At this time, there is no extended residual group that has been reconstructed. Therefore, the prior information can be constructed directly based on the extended auxiliary information.

[0099] If the current extended residual group is not the first element in the residual recovery sequence, then there exists an already reconstructed extended residual group. Therefore, the image reconstruction features of the recovered extended residual group can be convolved using a convolutional layer. Then, the extended auxiliary information and the convolutional results corresponding to the image reconstruction features of the recovered extended residual group are concatenated channel by channel. Prior information is then constructed based on the concatenated auxiliary information. When constructing the prior information and selecting recovered extended residual groups, all recovered extended residual groups can be selected, or only a portion of them can be selected.

[0100] In one possible implementation of this embodiment, when performing result stitching, feature enhancement can be performed on the image reconstruction features of the recovered extended residual group to further improve the prediction effect. The step of stitching the extended auxiliary information with the convolution processing results corresponding to the image reconstruction features of the recovered extended residual group to obtain stitching auxiliary information may include: Obtain the image reconstruction features corresponding to the recovered extended residual group; The image reconstruction features are enhanced to obtain enhanced reconstruction features; The auxiliary information is concatenated with the convolutional processing result corresponding to the enhanced reconstruction feature to obtain the concatenated auxiliary information.

[0101] In practical applications, feature enhancement of image reconstruction features can be achieved by performing operations such as missing value processing and outlier processing on the image reconstruction features.

[0102] It is understandable that performing feature enhancement on the image reconstruction features before concatenating auxiliary information with image reconstruction features to obtain enhanced reconstruction features can increase the reliability of enhanced reconstruction features, thereby improving the reliability of prior information and making mean prediction more accurate when based on prior information.

[0103] In practical applications, when enhancing the image reconstruction features corresponding to the recovered extended residual group, the predicted mean, auxiliary information, image residual data, and / or residual data variance corresponding to the recovered extended residual group can be used. In this case, the step of enhancing the image reconstruction features to obtain enhanced reconstruction features as described in this embodiment can include: Obtain the predicted mean, auxiliary information, image residual data, and / or residual data variance corresponding to the recovered extended residual group; The image reconstruction features are enhanced by using the predicted mean, auxiliary information, image residual data, and / or residual data variance corresponding to the recovered extended residual group to obtain enhanced reconstruction features.

[0104] It should be noted that the predicted mean corresponding to the recovered extended residual group can be the value obtained when predicting the mean during residual recovery of the recovered extended residual group. The variance of the residual data can be the variance value of the image residual data corresponding to the recovered extended residual group.

[0105] In one possible implementation of this embodiment, in order to improve the reconstruction effect of image reconstruction, step S30 of this embodiment may include: Feature enhancement is performed on the image reconstruction features corresponding to each extended residual group to obtain the enhanced reconstruction features corresponding to each extended residual group; The spatial resolution of the enhanced reconstruction features corresponding to each extended residual group is magnified to obtain the reconstruction feature data.

[0106] It should be noted that feature enhancement of the image reconstruction features corresponding to the extended residual group can be achieved by using the prediction mean, auxiliary information, image residual data and / or residual data variance of the extended residual group to enhance the image reconstruction features corresponding to the extended residual group.

[0107] It is understandable that before performing spatial resolution amplification processing on the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data, feature enhancement is first performed on the image reconstruction features corresponding to each extended residual group, and then spatial resolution amplification processing is performed on the enhanced reconstruction features corresponding to each extended residual group to obtain reconstruction feature data. This can ensure that the reliability of the final constructed reconstruction feature data is higher, and the quality of the reconstructed image blocks obtained by subsequent image reconstruction will be better.

[0108] To facilitate understanding, we will now combine... Figure 7 , 8 The following points are provided for explanation, but do not limit the scope of this scheme. Figure 7 This is a schematic diagram of the image decoding grouping execution process in this embodiment. Figure 8 This is a schematic diagram of the secondary grouping execution process in this embodiment. Figure 9 This is a schematic diagram of the feature enhancement grouping execution process in this embodiment.

[0109] like Figure 7As shown, the image features corresponding to the image residual data are y∈R{H,W,C}. First, the spatial resolution is reduced, resulting in y∈R{H / 2,W / 2,4C}. This can then be divided into two groups (group1 and group2). A residual recovery sequence (group1-group2) is then constructed. The mean value mu of Group1 is obtained through a network using auxiliary (Psi) information, yielding the image reconstruction features of Group1. These features are then extracted and concatenated with Psi via channel concatenation. The mean value mu of Group2 is obtained through the network and then decoded to obtain Group2. Combining Group1 and Group2 via channel concatenation, a special spatial resolution amplification process (i.e., the reverse process of spatial resolution reduction) is performed to obtain the reconstructed feature data. Finally, image reconstruction is performed based on this reconstructed feature data.

[0110] However, if grouping is performed before reducing spatial resolution, the execution flow is as follows: Figure 8 As shown, the image features corresponding to the image residual data are y∈R{H,W,C}. First, they are divided into y1 and y2 according to the corresponding feature channels, resulting in y1∈R{H,W,C / 2}. Then, the spatial resolution is reduced, dividing y1 into part1, part2, part3, and part4. At this point, the following exists: Part1 is represented as the even-numbered row and even-numbered column of the spatial domain in y1, containing data information for all channels.

[0111] Part 2 is represented in y1, where the spatial location is in the odd-numbered row and odd-numbered column of the y1 spatial index, and contains data information for all channels.

[0112] Part 3 is represented in y1, where the spatial location is in the even-numbered row and odd-numbered column of the y1 spatial index, and contains data information for all channels.

[0113] Part 4 is represented in y1, where the spatial location is in the odd-numbered row and even-numbered column of the y1 spatial index, and contains data information for all channels.

[0114] Similarly, y2 can be divided into four parts: part5, part6, part7, and part8. The resulting residual recovery sequence is "part1-part2-part3-part4-part5-part6-part7-part8". When performing residual recovery on part1, prior information can be generated directly based on auxiliary information. When performing residual recovery on part2, prior information can be generated based on the image reconstruction features and auxiliary information of part1. When performing residual recovery on part3, prior information can be generated based on the image reconstruction features and auxiliary information of both part1 and part2. This process continues until the image reconstruction features corresponding to all parts are obtained. Of course, two residual recovery sequences can also be constructed based on y1 and y2 respectively. In this case, the two residual recovery sequences are "part1-part2-part3-part4" and "part5-part6-part7-part8". In this case, a similar process can be used to reconstruct the image of y1 based on the residual recovery sequence composed of part1-4 to obtain the image reconstruction features corresponding to each part in y1. Then, prior information is generated based on the image reconstruction features and auxiliary information corresponding to y1. Finally, the image of y2 is reconstructed based on the residual recovery sequence composed of part5-8 and the generated prior information to obtain the image reconstruction features corresponding to each part in y2.

[0115] Since y is first split into y1 and y2 and then the spatial resolution is reduced for each part, after obtaining the image reconstruction features corresponding to each part of y1 and y2, the spatial resolution of the image reconstruction features corresponding to each part of y1 and y2 can be increased before aggregation to obtain complete reconstruction feature data.

[0116] If feature enhancement is performed after grouping, the specific execution process is as follows: Figure 9As shown in (a), the image features corresponding to the image residual data are y∈R{H,W,C}. First, the spatial resolution is reduced, resulting in y∈R{H / 2,W / 2,4C}. These can then be divided into two groups (group1 and group2). The mean value mu of Group1 is obtained through a network using auxiliary (Psi) information, yielding the image reconstruction features of Group1. The predicted mean value of Group1 is used to enhance the image reconstruction features of Group1, resulting in enhanced image features Group1_E. The enhanced image features are then extracted and concatenated with Psi channels. This concatenation is then processed through a network to obtain the mean value mu of Group2, which is then decoded to obtain the image reconstruction features of Group2. The mean value mu of Group2 is then used to enhance the image reconstruction features of Group2, resulting in enhanced image features Group2_E. Group1_E and Group2_E are then concatenated, followed by a special spatial resolution amplification process (i.e., the reverse process of spatial resolution reduction), to obtain the reconstructed feature data. Finally, image reconstruction is performed based on the reconstructed feature data. Figure 9 When feature enhancement is performed in (a), the specific structure of the network (Enhance_Net) used for feature enhancement is as follows: Figure 9 As shown in (b).

[0117] This embodiment constructs a residual recovery sequence based on the multiple extended residual groups; based on the residual recovery sequence, residual recovery is performed on each of the multiple extended residual groups to obtain the image reconstruction features corresponding to each extended residual group. Since the residual recovery sequence is constructed based on multiple extended residual groups, the order of residual recovery can be determined through the residual recovery sequence, allowing for rapid determination of whether there are recovered extended residual groups. If there are recovered extended residual groups, more accurate prior information can be constructed based on the image feature data corresponding to the recovered extended residual groups.

[0118] This invention provides an image encoding method, referring to... Figure 10 , Figure 10 This is a flowchart illustrating a first embodiment of an image encoding method according to the present invention.

[0119] In this embodiment, the image encoding method includes the following steps: Step S100: Reduce the spatial resolution of the image features corresponding to the image to be encoded to obtain extended image features.

[0120] It should be noted that, in order to facilitate subsequent grouping processing and reduce time complexity, after obtaining the image features corresponding to the image to be encoded, the spatial resolution of the image residual data can be reduced to obtain extended image features. The image to be encoded is the original image mentioned in the above-described image decoding method embodiment.

[0121] In this process, after reducing the spatial resolution of image features, the spatial size becomes smaller. Correspondingly, when processing image features, a smaller convolution kernel can be used. For example, if a 5x5 convolution kernel is used on the original image residual data, it is equivalent to using a 3x3 convolution kernel on the receptive field of the convolution kernel after reducing the spatial resolution.

[0122] In one possible implementation of this embodiment, step S100 may include: The spatial dimensions corresponding to the image features of the image to be encoded are reduced, and / or the number of feature channels corresponding to the image features is increased, to obtain extended image features.

[0123] It should be noted that the reduction range of the spatial size corresponding to the image feature and the increase range of the number of feature channels corresponding to the image feature can be preset by the administrator of the decoding device, and this embodiment does not impose any restrictions on this.

[0124] In actual execution, you can set only the reduction range of the spatial size or only the increase range of the number of feature channels, and let the decoding device adaptively adjust the spatial size or the number of feature channels. Of course, you can also choose not to use device adaptation, but set the reduction range of the spatial size and the increase range of the number of feature channels, and let the device execute.

[0125] In specific implementations, spatial resolution reduction can be performed based on spatial or frequency information corresponding to image features. Alternatively, spatial resolution reduction can be performed through a preset convolutional layer. In this case, the step described in this embodiment of reducing the spatial size corresponding to the image features of the image to be encoded and / or increasing the number of feature channels corresponding to the image features to obtain expanded image features may include: Based on the spatial domain information corresponding to the image features of the image to be encoded, the spatial size corresponding to the image features is reduced, and / or the number of feature channels corresponding to the image features is increased, so as to obtain extended image features; or, Based on the frequency domain information corresponding to the image features of the image to be encoded, the spatial size corresponding to the image features is reduced, and / or the number of feature channels corresponding to the image features is increased, so as to obtain extended image features; or, The spatial dimensions corresponding to the image features are reduced and / or the number of feature channels corresponding to the image features are increased based on the preset convolutional layers to obtain extended image features.

[0126] For specific implementation details, please refer to the above-described image decoding method embodiments. Figure 5 The explanation section will not be repeated here.

[0127] Step S200: Group the extended image features to obtain multiple extended feature groups.

[0128] In a specific implementation, when grouping extended image features, the feature channels corresponding to each extended image feature can be referenced for grouping. In this case, step S200 in this embodiment may include: The extended image features are grouped based on the feature channels corresponding to the extended feature data to obtain multiple extended feature groups.

[0129] It should be noted that grouping extended image features based on the feature channels corresponding to the extended feature data to obtain multiple extended feature groups can be achieved by uniformly dividing the extended image features into multiple groups according to the corresponding feature channels. For example, assuming the total number of feature channels corresponding to the extended image features is 20, then the extended image features corresponding to feature channels 1-10 can be grouped into one group, and the extended image features corresponding to feature channels 11-20 can be grouped into another group. The number of evenly divided groups can be preset by the administrator of the decoding device; this embodiment does not impose any restrictions on this.

[0130] Of course, the grouping can also be uneven. The extended image features can be grouped based on the feature channels corresponding to the extended image features to obtain multiple extended feature groups. Alternatively, the extended image features can be divided into multiple groups according to the corresponding feature channels based on preset grouping rules. The preset grouping rules can be set in advance by the administrator of the encoding device according to actual needs. For example, the preset grouping rules can be set to group the first m / n (n is the total number of feature channels, m is a preset value, and the value range is [1,n)) of extended image features into one group, and the remaining extended image features into another group.

[0131] In practical applications, extended image features are grouped based on the feature channels corresponding to the extended image features. When multiple extended feature groups are obtained, extended image features corresponding to a single feature channel can also be divided into a group. For example, if the total number of feature channels corresponding to the extended image features is 20, then the extended image features can be divided into 20 groups according to the different feature channels.

[0132] Step S300: Perform residual calculation on the multiple extended feature groups respectively to obtain the image residual data corresponding to each extended feature group.

[0133] It should be noted that the image residual data corresponding to the extended feature group can be obtained by performing residual calculation on each extended feature group separately, which can be achieved by predicting the mean of the extended feature group, and then subtracting the predicted mean from the image features in the extended feature group to obtain the image residual data corresponding to the extended feature group.

[0134] Step S400: Generate an image bitstream based on the image residual data, and send the image bitstream to the image decoding end.

[0135] It should be noted that generating an image bitstream from image residual data can be achieved by writing the image residual data into the image bitstream through entropy encoding.

[0136] In one possible implementation of this embodiment, before performing spatial resolution reduction processing on the image features corresponding to the image to be encoded, the image features can be grouped first, and then the spatial resolution reduction processing can be performed on each group separately. In this case, step S100 of this embodiment may include: Obtain the image features corresponding to the image to be encoded; The image features are grouped according to the feature channels corresponding to the image features to obtain at least one image feature group; The data in the image feature group are processed to reduce the spatial resolution to obtain extended image features.

[0137] In one possible implementation of this embodiment, the image features corresponding to the image to be encoded are subjected to spatial resolution reduction processing to obtain extended image features. Alternatively, after obtaining the image features corresponding to the image to be encoded, the data in the image features are subjected to spatial resolution reduction processing to obtain extended image features. It should be noted that obtaining the image features corresponding to the image to be encoded can be done by extracting the image features corresponding to the image to be encoded through a preset feature extraction network. When grouping data according to the feature channels corresponding to the image features, the same or similar methods can be used as when grouping the extended image features.

[0138] In one possible implementation of this embodiment, step S400 may include: The spatial resolution of the image residual features corresponding to each extended feature group is increased to obtain the image residual data corresponding to the image to be encoded. An image bitstream is generated based on the image residual data corresponding to the image to be encoded, and the image bitstream is sent to the image decoding end.

[0139] It should be noted that the spatial resolution amplification process can be the reverse of the spatial resolution reduction process described above. After obtaining the image residual features corresponding to each extended feature group, the spatial resolution of the image residual features corresponding to each extended feature group can be amplified to restore their corresponding spatial size and number of channels to be consistent with the image features corresponding to the image to be encoded, thereby obtaining the image residual data corresponding to the image to be encoded. Then, entropy encoding is performed on the image residual data corresponding to the image to be encoded to generate an image bitstream, and the generated image bitstream is sent to the image decoding end.

[0140] This embodiment obtains extended image features by reducing the spatial resolution of the image features corresponding to the image to be encoded; the extended image features are then grouped to obtain multiple extended feature groups; residual calculations are performed on each of the multiple extended feature groups to obtain image residual data corresponding to each extended feature group; an image bitstream is generated based on the image residual data, and the image bitstream is sent to the image decoding end. Because the image features of the image to be encoded are reduced in spatial resolution and then grouped into multiple extended feature groups, residual calculations can be performed on the entire group at low resolution, thereby improving the overall efficiency of residual calculations and reducing time complexity.

[0141] refer to Figure 11 , Figure 11 This is a flowchart illustrating a second embodiment of an image encoding method according to the present invention.

[0142] Based on the first embodiment described above, step S300 of the image encoding method in this embodiment includes: Step S3001: Construct a residual calculation sequence based on the multiple extended feature groups.

[0143] It should be noted that after dividing the feature groups into multiple extended feature groups, residual calculation can be performed on the entire group. In this case, in order to determine the order of residual calculation for each extended feature group, a residual calculation sequence can be constructed based on the multiple extended feature groups.

[0144] Step S3002: Perform residual calculation on the multiple extended feature groups based on the residual calculation sequence to obtain the image residual data corresponding to each extended feature group.

[0145] It should be noted that performing residual calculations on multiple extended feature groups based on the residual calculation sequence can be based on the sequence order in the residual calculation sequence, and performing residual calculations on multiple extended feature groups one by one.

[0146] In practical applications, residual calculation can be performed sequentially through a sequence traversal. In this case, step S3002 in this embodiment may include: The residual calculation sequence is traversed to obtain the current extended feature group; Obtain auxiliary information output by the auxiliary coding network; Construct prior information based on the auxiliary information; Based on the prior information, residual calculation is performed on the current extended feature group to obtain the image residual data corresponding to the current extended feature group; At the end of the traversal, the image residual data corresponding to each extended feature group is obtained.

[0147] It should be noted that traversing the residual calculation sequence to obtain the current extended feature group can be achieved by traversing the residual calculation sequence and using the traversed extended feature group as the current extended feature group. The auxiliary coding network can be as described above. Figure 3 The auxiliary network shown is either a Hyper Encoder Net or a Hyper Decoder Net.

[0148] In practical applications, the residual calculation of the current extended feature group based on prior information can be used to obtain the image residual data corresponding to the current extended feature group. This can be achieved by processing the prior information using Prediction Fusion Net to obtain the prediction mean, and then subtracting the prediction mean from the features in the current extended feature group to calculate the residual and obtain the image residual data corresponding to the current extended feature group.

[0149] In order to increase the accuracy of mean prediction, when constructing prior information based on auxiliary information, the prior information can also be constructed by combining the image features corresponding to the extended feature group that has been previously calculated for residuals. The specific implementation method is the same as that used in the image decoding process. The specific implementation steps can refer to the method of constructing prior information based on auxiliary information provided in any of the above image decoding method embodiments.

[0150] This embodiment constructs a residual calculation sequence based on multiple extended feature groups; based on the residual calculation sequence, residual calculations are performed on each of the multiple extended feature groups to obtain image residual data corresponding to each extended feature group. Since the residual calculation sequence is constructed based on multiple extended feature groups, the order of residual calculations can be determined through the residual calculation sequence, enabling rapid determination of whether an extended feature group has been calculated. If an extended feature group has been calculated, more accurate prior information can be constructed based on the convolution processing results corresponding to the image reconstruction features of the recovered extended residual groups.

[0151] Furthermore, embodiments of the present invention also propose a storage medium storing an image decoding and / or image encoding program. When the image decoding program is executed by a processor, it implements the steps of the image decoding method described above, and when the image encoding program is executed by a processor, it implements the steps of the image encoding method described above.

[0152] Reference Figure 12 , Figure 12 This is a structural block diagram of the first embodiment of the image decoding device of the present invention.

[0153] like Figure 12 As shown, the image decoding device proposed in this embodiment of the invention includes: The bitstream decoding module 10 is used to extract image residual data or extended residual data from the image bitstream, and obtain multiple extended residual groups based on the extracted image residual data or extended residual data; The residual recovery module 20 is used to perform residual recovery on the multiple extended residual groups respectively to obtain the image reconstruction features corresponding to each extended residual group; The data combination module 30 is used to enlarge the spatial resolution of the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data. The image reconstruction module 40 is used to reconstruct the image based on the reconstruction feature data to obtain a reconstructed image block.

[0154] This embodiment obtains multiple extended residual groups based on the image residual data or extended residual data extracted from the image bitstream. Residual recovery is then performed on each of the extended residual groups to obtain image reconstruction features corresponding to each group. The spatial resolution of the image reconstruction features corresponding to each extended residual group is increased to obtain reconstruction feature data. Image reconstruction is then performed based on the reconstruction feature data to obtain reconstructed image blocks. Since the obtained extended residual data is residual data that has undergone spatial resolution reduction processing, residual recovery processing can be performed in groups at low resolution, thereby improving the overall residual recovery calculation efficiency and reducing time complexity.

[0155] In one possible implementation of this embodiment, the bitstream decoding module 10 is further configured to extract extended residual data from the image bitstream; and group the extended residual data based on the feature channels corresponding to the extended residual data to obtain multiple extended residual groups.

[0156] In one possible implementation of this embodiment, the bitstream decoding module 10 is further configured to extract image residual data from the image bitstream; perform spatial resolution reduction processing on the image residual data to obtain extended residual data, wherein the spatial resolution reduction processing is the reverse process of the spatial resolution reduction processing; and group the extended residual data to obtain multiple extended residual groups.

[0157] In one possible implementation of this embodiment, the bitstream decoding module 10 is further configured to group data according to the feature channels corresponding to the image residual data to obtain at least one image residual group; and to perform spatial resolution reduction processing on the data in the image residual group to obtain extended residual data.

[0158] In one possible implementation of this embodiment, the bitstream decoding module 10 is further configured to reduce the spatial size corresponding to the image residual data and / or increase the number of feature channels corresponding to the image residual data to obtain extended residual data.

[0159] In one possible implementation of this embodiment, the bitstream decoding module 10 is further configured to reduce the spatial size of the image residual data according to the spatial domain information corresponding to the image residual data, and / or increase the number of feature channels corresponding to the image residual data to obtain extended residual data; or, reduce the spatial size of the image residual data according to the frequency domain information corresponding to the image residual data, and / or increase the number of feature channels corresponding to the image residual data to obtain extended residual data; or, reduce the spatial size of the image residual data according to a preset convolutional layer, and / or increase the number of feature channels corresponding to the image residual data to obtain extended residual data.

[0160] In one possible implementation of this embodiment, the residual recovery module 20 is further configured to construct a residual recovery sequence based on the plurality of extended residual groups; and perform residual recovery on the plurality of extended residual groups based on the residual recovery sequence to obtain the image reconstruction features corresponding to each extended residual group.

[0161] In one possible implementation of this embodiment, the residual recovery module 20 is further configured to traverse the residual recovery sequence to obtain the current extended residual group; obtain auxiliary information output by the auxiliary coding network; construct prior information based on the auxiliary information; perform residual recovery on the current extended residual group based on the prior information to obtain the image reconstruction features corresponding to the current extended residual group; and obtain the image reconstruction features corresponding to each extended residual group at the end of the traversal.

[0162] In one possible implementation of this embodiment, the residual recovery module 20 is further configured to obtain extended auxiliary information; detect whether the current extended residual group is the first element in the residual recovery sequence; if it is the first element, construct prior information based on the extended auxiliary information; if it is not the first element, concatenate the extended auxiliary information with the convolution processing result corresponding to the image reconstruction features of the recovered extended residual group to obtain concatenation auxiliary information, and construct prior information based on the concatenation auxiliary information.

[0163] In one possible implementation of this embodiment, the residual recovery module 20 is further configured to obtain image reconstruction features corresponding to the recovered extended residual group; perform feature enhancement on the image reconstruction features to obtain enhanced reconstruction features; and concatenate the auxiliary information with the convolution processing result corresponding to the enhanced reconstruction features to obtain concatenation auxiliary information.

[0164] In one possible implementation of this embodiment, the residual recovery module 20 is further configured to acquire the predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the recovered extended residual group; and to perform feature enhancement on the image reconstruction features based on the predicted mean, auxiliary information, image residual data and / or residual data variance corresponding to the recovered extended residual group to obtain enhanced reconstruction features.

[0165] In one possible implementation of this embodiment, the data combination module 30 is further configured to perform feature enhancement on the image reconstruction features corresponding to each extended residual group to obtain enhanced reconstruction features corresponding to each extended residual group; and to perform spatial resolution amplification processing on the enhanced reconstruction features corresponding to each extended residual group to obtain reconstruction feature data.

[0166] In one possible implementation of this embodiment, the image reconstruction module 40 is further configured to perform synthetic transformation processing on the reconstructed feature data through a pre-built synthetic transformation network to achieve the image reconstruction and obtain the reconstructed image patch, wherein the synthetic transformation network is a network built based on deep learning or neural networks.

[0167] In one possible implementation of this embodiment, the bitstream decoding module 10 is further configured to uniformly group the extended residual data according to the feature channels corresponding to the extended residual data.

[0168] Reference Figure 13 , Figure 13 This is a structural block diagram of the first embodiment of the image encoding device of the present invention.

[0169] like Figure 13 As shown, the image encoding device proposed in this embodiment of the invention includes: Feature extraction module 100 is used to reduce the spatial resolution of the image features corresponding to the image to be encoded to obtain extended image features; Data grouping module 200 is used to group the extended image features to obtain multiple extended feature groups; The residual calculation module 300 is used to perform residual calculation on the multiple extended feature groups respectively to obtain the image residual data corresponding to each extended feature group; The bitstream generation module 400 is used to generate an image bitstream based on the image residual data and send the image bitstream to the image decoding end.

[0170] This embodiment obtains extended image features by reducing the spatial resolution of the image features corresponding to the image to be encoded; the extended image features are then grouped to obtain multiple extended feature groups; residual calculations are performed on each of the multiple extended feature groups to obtain image residual data corresponding to each extended feature group; an image bitstream is generated based on the image residual data, and the image bitstream is sent to the image decoding end. Because the image features of the image to be encoded are reduced in spatial resolution and then grouped into multiple extended feature groups, residual calculations can be performed on the entire group at low resolution, thereby improving the overall efficiency of residual calculations and reducing time complexity.

[0171] In one possible implementation of this embodiment, the feature extraction module 100 is further configured to acquire image features corresponding to the image to be encoded; and to perform spatial resolution reduction processing on the data in the image features to obtain extended image features.

[0172] In one possible implementation of this embodiment, the feature extraction module 100 is further used for The spatial dimensions corresponding to the image features of the image to be encoded are reduced, and / or the number of feature channels corresponding to the image features is increased, to obtain extended image features.

[0173] In one possible implementation of this embodiment, the feature extraction module 100 is further configured to reduce the spatial size corresponding to the image features based on the spatial domain information corresponding to the image features of the image to be encoded, and / or increase the number of feature channels corresponding to the image features to obtain extended image features; or, reduce the spatial size corresponding to the image features based on the frequency domain information corresponding to the image features of the image to be encoded, and / or increase the number of feature channels corresponding to the image features to obtain extended image features; or, reduce the spatial size corresponding to the image features based on a preset convolutional layer, and / or increase the number of feature channels corresponding to the image features to obtain extended image features.

[0174] In one possible implementation of this embodiment, the residual calculation module 300 is further configured to construct a residual calculation sequence based on the plurality of extended feature groups; and perform residual calculation on the plurality of extended feature groups based on the residual calculation sequence to obtain image residual data corresponding to each extended feature group.

[0175] In one possible implementation of this embodiment, the residual calculation module 300 is further configured to traverse the residual calculation sequence to obtain the current extended feature group; obtain auxiliary information output by the auxiliary coding network; construct prior information based on the auxiliary information; perform residual calculation on the current extended feature group based on the prior information to obtain the image residual data corresponding to the current extended feature group; and obtain the image residual data corresponding to each extended feature group at the end of the traversal.

[0176] In one possible implementation of this embodiment, the data grouping module 200 is further configured to group the extended image features based on the feature channels corresponding to the extended feature data to obtain multiple extended feature groups.

[0177] In one possible implementation of this embodiment, the bitstream generation module 400 is further configured to perform spatial resolution amplification processing on the image residual features corresponding to each extended feature group to obtain image residual data corresponding to the image to be encoded, wherein the spatial resolution amplification processing is the inverse process of spatial resolution reduction processing; generate an image bitstream based on the image residual data corresponding to the image to be encoded, and send the image bitstream to the image decoding end.

[0178] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0179] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0180] In addition, for technical details not described in detail in this embodiment, please refer to the image decoding method or image encoding method provided in any embodiment of the present invention, which will not be repeated here.

[0181] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0182] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0184] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. An image decoding method, characterized in that, The image decoding method includes: Image residual data is extracted from the image bitstream; the spatial resolution of the image residual data is reduced based on the spatial information corresponding to the image residual data to obtain extended residual data; the extended residual data is grouped to obtain multiple extended residual groups. Residual recovery is performed on each of the multiple extended residual groups to obtain the image reconstruction features corresponding to each extended residual group; The image reconstruction features corresponding to each extended residual group are processed to increase the spatial resolution to obtain reconstruction feature data. The process of increasing the spatial resolution is the inverse process of reducing the spatial resolution. Image reconstruction is performed based on the reconstructed feature data to obtain reconstructed image blocks.

2. The image decoding method as described in claim 1, characterized in that, The step of performing residual recovery on the plurality of extended residual groups to obtain the image reconstruction features corresponding to each extended residual group includes: Features are obtained by processing prior information through convolutional kernels; The auxiliary information output by the auxiliary network is concatenated with the obtained features, and the concatenated features are input into the Prediction Fusion Net to obtain the prediction mean. The predicted mean is added to the residuals in the current extended residual group to achieve residual recovery and obtain the image reconstruction features corresponding to the current extended residual group.

3. The image decoding method as described in claim 1, characterized in that, The step of reducing the spatial resolution of the image residual data based on the spatial information corresponding to the image residual data to obtain extended residual data includes: The spatial dimensions corresponding to the image residual data are reduced based on the spatial information corresponding to the image residual data to obtain the extended residual data; or, The number of feature channels corresponding to the image residual data is increased based on the spatial domain information corresponding to the image residual data to obtain the extended residual data; or, The spatial size of the image residual data is reduced based on the spatial information corresponding to the image residual data, and the number of feature channels corresponding to the image residual data is increased based on the spatial information corresponding to the image residual data to obtain the extended residual data.

4. The image decoding method as described in claim 1 or 3, characterized in that, When performing spatial resolution reduction processing on the image residual data The image residual data is resampled and rearranged; Resampling and arranging the image residual data includes: clustering and arranging all points with the same spatial location in the image residual data; Alternatively, the image residual data can be processed for spatial resolution based on a preset convolutional layer.

5. The image decoding method as described in claim 1 or 3, characterized in that, The spatial information corresponding to the image residual data includes the spatial size of the image residual data and / or the number of feature channels corresponding to the image residual data, wherein the spatial size includes width and height.

6. The image decoding method as described in claim 1, characterized in that, When performing spatial resolution amplification processing on the image reconstruction features corresponding to each extended residual group, the spatial resolution amplification processing is used to restore the spatial size corresponding to the image reconstruction feature to be consistent with the spatial size corresponding to the image obtained by feature extraction of the original image, and the spatial resolution amplification processing is used to restore the number of channels corresponding to the high image reconstruction feature to be consistent with the number of channels corresponding to the image obtained by feature extraction of the original image.

7. The image decoding method as described in claim 1, characterized in that, The step of performing residual recovery on the plurality of extended residual groups to obtain the image reconstruction features corresponding to each extended residual group includes: Construct a residual recovery sequence based on the multiple extended residual sets; Based on the residual recovery sequence, residual recovery is performed on the multiple extended residual groups respectively to obtain the image reconstruction features corresponding to each extended residual group.

8. An image encoding method, characterized in that, The image encoding method includes: Based on the spatial information corresponding to the image features of the image to be encoded, the spatial resolution of the image features corresponding to the image to be encoded is reduced to obtain extended image features; The extended image features are grouped to obtain multiple extended feature groups; Residual calculations are performed on the multiple extended feature groups to obtain image residual data corresponding to each extended feature group; wherein, after obtaining the image reconstruction features corresponding to each extended feature group, the image reconstruction features corresponding to each extended feature group are subjected to spatial resolution amplification processing to obtain image residual data corresponding to each extended feature group, and the spatial resolution amplification processing is the inverse process of the spatial resolution reduction processing. An image bitstream is generated based on the image residual data, and the image bitstream is sent to the image decoding end.

9. An image decoding device, characterized in that, The image decoding device includes: The bitstream decoding module is used to extract image residual data from the image bitstream; perform spatial resolution reduction processing on the image residual data according to the spatial information corresponding to the image residual data to obtain extended residual data; and group the extended residual data to obtain multiple extended residual groups. The residual recovery module is used to perform residual recovery on the multiple extended residual groups respectively to obtain the image reconstruction features corresponding to each extended residual group; The data combination module is used to perform spatial resolution amplification processing on the image reconstruction features corresponding to each extended residual group to obtain reconstruction feature data; wherein, the spatial resolution amplification processing is the inverse process of spatial resolution reduction processing; The image reconstruction module is used to reconstruct the image based on the reconstruction feature data to obtain reconstructed image blocks.

10. An image encoding device, characterized in that, The image encoding device includes: The feature extraction module is used to reduce the spatial resolution of the image features corresponding to the image features to be encoded based on the spatial information of the image features to be encoded, so as to obtain extended image features. The data grouping module is used to group the extended image features to obtain multiple extended feature groups; The residual calculation module is used to perform residual calculation on the multiple extended feature groups respectively to obtain image residual data corresponding to each extended feature group; wherein, after obtaining the image reconstruction features corresponding to each extended feature group, the residual calculation module performs spatial resolution amplification processing on the image reconstruction features corresponding to each extended feature group to obtain image residual data corresponding to each extended feature group, wherein the spatial resolution amplification processing is the inverse process of the spatial resolution reduction processing. The bitstream generation module is used to generate an image bitstream based on the image residual data and send the image bitstream to the image decoding end.

11. A decoding device, characterized in that, The decoding device includes: a processor, a memory, and an image decoding program stored in the memory and executable on the processor. When the image decoding program is executed by the processor, it implements the image decoding method as described in any one of claims 1-7.

12. An encoding device, characterized in that, The encoding device includes: a processor, a memory, and an image encoding program stored in the memory and executable on the processor, wherein the image encoding program, when executed by the processor, implements the image encoding method as described in claim 8.

13. A storage medium, characterized in that, The storage medium stores an image decoding program and / or an image encoding program. When the image decoding program is executed, it implements the image decoding method as described in any one of claims 1-7. When the image encoding program is executed, it implements the image encoding method as described in claim 8.

14. A computer program product, characterized in that, It includes computer program instructions configured to, when executed by a processor having memory, implement the image decoding method as described in any one of claims 1-7, or the image encoding method as described in claim 8.

Images