Image decoding method, encoding method and apparatus

Abstract

The application discloses an image decoding method and device, and relates to the technical field of video processing. A decoding end processes a first feature map of a reference frame based on a group of feature domain optical flows (such as an optical flow set) corresponding to an image frame to obtain a group of intermediate feature maps; the decoding end fuses the group of intermediate feature maps to obtain a predicted feature map; and the decoding end decodes the image frame according to the predicted feature map to obtain a target image. Pixel errors in the feature domain optical flow are smaller than pixel errors in the image domain optical flow, so that decoding errors caused by the intermediate feature maps determined by the feature domain optical flow are lower than decoding errors caused by the image domain optical flow in the prior art. The predicted feature map of the image frame is determined by fusing a plurality of intermediate feature maps at the decoding end, the predicted feature map contains more image information, and when the decoding end decodes the image frame based on the predicted feature map, the problem that a single intermediate feature map cannot accurately express the target image is avoided, and the accuracy of image decoding and the image quality are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and particularly relates to an image decoding method and device and an encoding method and device. BACKGROUND

[0002] In video coding technology, video compression and video enhancement technology are particularly important. A video compression system performs spatial (intra-image) prediction and / or temporal (inter-image) prediction to reduce or remove inherent redundant information in a video sequence; and a video enhancement technology is used to improve the display quality of an image. For a video decoding process corresponding to video compression or video enhancement, a decoding end uses a warping method to decode an image frame included in a video. The warping refers to that the decoding end obtains an image domain optical flow between an image frame and a reference frame, and decodes the image frame according to the optical flow. The image domain optical flow is used to indicate the motion speed and motion direction between corresponding pixels in adjacent two image frames. However, the warping is relatively sensitive to the accuracy of the optical flow, and a slight change in the accuracy of the optical flow will affect the accuracy of the warping. Since the error of the optical flow prediction between adjacent two image frames is relatively large, such as an error of 5 pixels or more, the accuracy of the decoding of the image frame by the decoding end based on the image domain optical flow is relatively low, and thus the definition of the decoded image is affected. Therefore, how to provide a more effective image decoding method has become a problem to be solved at present. SUMMARY

[0003] The present application provides an image decoding method and device and an encoding method and device, and solves the problem that the accuracy of the decoding of an image frame based on an image domain optical flow is relatively low, and thus the definition of the decoded image is affected.

[0004] The present application adopts the following technical solutions.

[0005] In a first aspect, the present disclosure provides a method for image decoding. The method is applied to a video coding system or a physical device supporting the video coding system, such as a decoder or a video decoder, which may, in some cases, include a chip system. The method for image decoding is described below by taking the decoder as an example. The method includes the following steps. First, the decoder parses a bitstream to obtain at least one optical flow set. The at least one optical flow set includes a first optical flow set corresponding to a first feature map of a reference frame of a first image frame. The first optical flow set includes one or more feature domain optical flows, and each of the one or more feature domain optical flows is used to indicate motion information between a feature map of the first image frame and the first feature map. Second, the decoder processes the first feature map based on the one or more feature domain optical flows included in the first optical flow set to obtain one or more intermediate feature maps corresponding to the first feature map. Third, the decoder fuses the one or more intermediate feature maps to obtain a first predicted feature map of the first image frame. Finally, the decoder decodes the first image frame based on the first predicted feature map to obtain the first image.

[0006] In the present embodiment, the error of pixels in a single feature domain optical flow is smaller than that of pixels in an image domain optical flow. Therefore, the decoding error caused by the intermediate feature map determined by the decoder based on the feature domain optical flow is lower than that caused by the image domain optical flow in the prior art. In other words, the decoding error caused by the image domain optical flow between two adjacent image frames is reduced by decoding the image frame based on the feature domain optical flow, and the accuracy of image decoding is improved. Moreover, the decoder fuses the plurality of intermediate feature maps determined based on the optical flow set and the feature map of the reference frame to obtain a predicted feature map of the image frame. The predicted feature map contains more image information than a single image domain optical flow, so that the problem that the information indicated by the single image domain optical flow cannot be accurately expressed in the first image is avoided when the decoder decodes the image frame based on the predicted feature map obtained by fusion, and the accuracy of image decoding and the image quality (such as image definition) are improved.

[0007] In an optional implementation, the step of processing the first feature map based on the one or more feature domain optical flows included in the first optical flow set to obtain one or more intermediate feature maps corresponding to the first feature map includes the following steps. The decoder parses the bitstream to obtain the first feature map of the reference frame. For each of the one or more feature domain optical flows included in the first optical flow set, the decoder performs optical flow wrapping on the first feature map based on a first feature domain optical flow to obtain an intermediate feature map corresponding to the first feature domain optical flow.

[0008] In the embodiment, the decoding end performs wrapping on the first feature map based on the set of feature domain optical flows, and then, after obtaining the image corresponding to the reference frame at the decoding end, the decoding end can perform interpolation on the corresponding positions of the set of feature domain optical flows in the image corresponding to the reference frame according to the set of feature domain optical flows to obtain the prediction value of the pixel value in the first image, thereby obtaining the first image, avoiding the need for the decoding end to predict all pixel values of the first image according to the image domain optical flow, reducing the amount of calculation required for image decoding at the decoding end, and improving the efficiency of image decoding.

[0009] In another optional implementation, the aforementioned first set of optical flows also corresponds to a second feature map of the reference frame. In the image decoding method provided in the embodiment, before the decoding end decodes the first image frame according to the first prediction feature map to obtain the first image, the image decoding method further includes: in a first step, the decoding end processes the second feature map based on one or more feature domain optical flows included in the first set of optical flows to obtain one or more intermediate feature maps corresponding to the second feature map. In a second step, the decoding end fuses the one or more intermediate feature maps corresponding to the second feature map to obtain a second prediction feature map of the first image frame. In this way, the decoding end can decode the first image frame to obtain the first image according to the first prediction feature map and the second prediction feature map.

[0010] In the embodiment, the plurality of feature maps (or the set of feature maps) of the reference frame can correspond to one set of optical flows (or one set of feature domain optical flows). In other words, one set of feature domain flows is shared by the plurality of feature maps belonging to the same group, the decoding end processes the feature map according to the set of feature domain optical flows corresponding to the feature map to obtain an intermediate feature map corresponding to the feature map, further, the decoding end fuses the intermediate feature map corresponding to the feature map to obtain a prediction feature map corresponding to the feature map. Finally, the decoding end decodes the image frame according to the prediction feature maps corresponding to all feature maps of the reference frame to obtain the target image. In this way, in the image decoding process, if the reference frame corresponds to a plurality of feature maps, the decoding end can divide the plurality of feature maps into one group or multiple groups, the feature maps belonging to the same group share one set of optical flows, and fuse the intermediate feature maps corresponding to the feature maps to obtain the prediction feature maps, thereby avoiding the problem that when the reference frame or the image frame has more information, the accuracy of reconstructing the image according to the feature map at the decoding end is low and the speed is slow, and improving the accuracy of image decoding. It is worth noting that the number of channels of the feature maps belonging to different groups can be different.

[0011] In another optional implementation, the decoding end fuses the one or more intermediate feature maps to obtain the first prediction feature map of the first image frame, including: the decoding end obtaining one or more weights of the one or more intermediate feature maps, wherein one intermediate feature map corresponds to one weight; and the decoding end processing the intermediate feature map corresponding to each of the one or more weights based on the one or more intermediate feature map weights, and adding all the processed intermediate feature maps to obtain the first prediction feature map. The weight is used to indicate the weight of the intermediate feature map in the first prediction feature map.

[0012] In the embodiment, the weight of each intermediate feature map corresponding to the plurality of intermediate feature maps of the first feature map can be different, in other words, the decoding end can set different weights for the intermediate feature maps according to the requirements of image decoding. For example, if the images corresponding to some intermediate feature maps are relatively blurred, the weights of these intermediate feature maps are reduced, thereby improving the clarity of the first image.

[0013] In another optional implementation, the decoding end fuses the one or more intermediate feature maps to obtain the first prediction feature map of the first image frame, including: inputting the one or more intermediate feature maps corresponding to the first feature map into a feature fusion model to obtain the first prediction feature map. The feature fusion model includes a convolution network layer for fusing the intermediate feature maps.

[0014] In the conventional technology, the decoding end obtains a plurality of image domain optical flows between the image frame and the reference frame, and obtains a plurality of images from the reference frame according to the plurality of image domain optical flows, so as to fuse the plurality of images to obtain the target image corresponding to the image frame. Therefore, the decoding end needs to predict the pixel values of a plurality of images when decoding one image frame, and fuse the plurality of images to obtain the target image, which results in a large amount of calculation resources required for image decoding, and a low efficiency of the decoding end in decoding the video according to the image domain optical flow.

[0015] In the embodiment, since the calculation resources required for feature map fusion are less than the calculation resources required for feature map decoding in the process of image decoding, the decoding end fuses the plurality of intermediate feature maps by using the feature fusion model, and then decodes the image frame based on the prediction feature map obtained by fusing the plurality of intermediate feature maps, that is, the decoding end only needs to predict the pixel values of the image position indicated by the prediction feature map in the image according to the prediction feature map, without predicting all the pixel values of the plurality of images, thereby reducing the calculation resources required for image decoding and improving the efficiency of image decoding.

[0016] In another optional implementation, the image decoding method provided by the present application further includes: first, the decoding end obtains the feature map of the first image. Second, the decoding end obtains the enhanced feature map according to the feature map of the first image, the first feature map and the first predicted feature map. Finally, the decoding end processes the first image according to the enhanced feature map to obtain the second image. The definition of the second image is higher than that of the first image.

[0017] In this embodiment, the decoding end can fuse the feature map of the first image, the first feature map and the first predicted feature map, and perform video enhancement processing on the first image based on the enhanced feature map obtained by fusion to obtain a second image with higher definition, thereby improving the definition and display effect of the decoded image.

[0018] In another optional implementation, the decoding end processes the first image according to the enhanced feature map to obtain the second image, including: the decoding end obtains the enhanced layer image of the first image according to the enhanced feature map, and reconstructs the first image based on the enhanced layer image to obtain the second image. In an example, the enhanced layer image can refer to an image determined by the decoding end according to the reference frame and the enhanced feature map, the decoding end adds part or all of the information of the enhanced layer image to the first image to obtain the second image, or the decoding end takes the enhanced layer image as the reconstructed image of the first image, that is, the aforementioned second image. In this example, the decoding end obtains multiple feature maps of the image at different stages and obtains the enhanced feature map determined by the multiple feature maps, thereby reconstructing and video enhancing the first image according to the enhanced feature map to improve the definition and display effect of the decoded image.

[0019] In a second aspect, an image encoding method is provided. The image encoding method is applied to a video coding system or a physical device supporting implementation of the video coding system, such as an encoder or a video encoder, which may include a chip system in some cases. The image encoding method is described below by taking an encoder as an example. The image encoding method includes the following steps. In a first step, the encoder obtains a feature map of a first image frame and a first feature map of a reference frame of the first image frame. In a second step, the encoder obtains at least one optical flow set based on the feature map of the first image frame and the first feature map. The at least one optical flow set includes a first optical flow set corresponding to the first feature map, and the first optical flow set includes one or more feature domain optical flows. Each of the one or more feature domain optical flows is used to indicate motion information between the feature map of the first image frame and the first feature map. In a third step, the encoder processes the first feature map based on the one or more feature domain optical flows included in the first optical flow set to obtain one or more intermediate feature maps corresponding to the first feature map. In a fourth step, the encoder fuses the one or more intermediate feature maps to obtain a first predicted feature map of the first image frame. In a fifth step, the encoder encodes the first image frame based on the first predicted feature map to obtain a bitstream.

[0020] Optionally, the bitstream includes a feature domain optical flow bitstream corresponding to the at least one optical flow set and a residual bitstream of an image region corresponding to the first predicted feature map.

[0021] In the embodiment, for a single feature domain optical flow, the error of a pixel in the feature domain optical flow is smaller than the error of a pixel in an image domain optical flow. Therefore, the encoding error caused by the intermediate feature map determined by the encoder based on the feature domain optical flow is lower than the encoding error caused by the image domain optical flow in the conventional technology. In other words, the encoding error caused by the image domain optical flow between two adjacent image frames is reduced by encoding the image frame based on the feature domain optical flow, and the accuracy of image encoding is improved. Moreover, the encoder processes the feature map of the reference frame based on the plurality of feature domain optical flows to obtain a plurality of intermediate feature maps, and fuses the plurality of intermediate feature maps to determine the predicted feature map of the image frame. In other words, the encoder fuses the plurality of intermediate feature maps determined based on the optical flow set and the feature map of the reference frame to obtain the predicted feature map of the image frame, which contains more image information. Therefore, the problem that a single intermediate feature map cannot accurately express the first image is avoided when the encoder encodes the image frame based on the fused predicted feature map, and the accuracy of image encoding and the image quality (such as image definition) are improved.

[0022] In an optional implementation, the encoding end processes the first feature map based on the one or more feature field optical flows included in the first optical flow set to obtain one or more intermediate feature maps corresponding to the first feature map, including: for the one or more feature field optical flows included in the first optical flow set, performing optical flow mapping on the first feature map based on the first feature field optical flow therein to obtain an intermediate feature map corresponding to the first feature field optical flow. The first feature map is any one of the one or more feature field optical flows included in the first optical flow set.

[0023] In the embodiment, as the computing resources required for feature map fusion are less than the computing resources required for feature map encoding in the image encoding process, the encoding end fuses the plurality of intermediate feature maps by using a feature fusion model, and then encodes the image frame based on a prediction feature map obtained by fusing the plurality of intermediate feature maps, that is, the encoding end only needs to predict the pixel values of the image positions indicated by the prediction feature map in the image according to the prediction feature map, without predicting all pixel values of the plurality of images, thereby reducing the computing resources required for image encoding and improving the efficiency of image encoding.

[0024] In another optional implementation, the first optical flow set also corresponds to a second feature map of the reference frame. Before the encoding end encodes the first image frame according to the first prediction feature map to obtain the code stream, the image encoding method provided in this embodiment further includes: a first step, the encoding end processes the second feature map based on one or more feature domain optical flows included in the first optical flow set to obtain one or more intermediate feature maps corresponding to the second feature map. A second step, the encoding end fuses the one or more intermediate feature maps corresponding to the second feature map to obtain a second prediction feature map of the first image frame. In this way, the aforementioned encoding of the first image frame by the encoding end according to the first prediction feature map to obtain the code stream can include: the encoding end encodes the first image frame according to the first prediction feature map and the second prediction feature map to obtain the code stream. In this embodiment, a plurality of feature maps (or a group of feature maps) of the reference frame can correspond to an optical flow set (or a group of feature domain optical flows). In other words, a group of feature domain flows are shared by a plurality of feature maps belonging to the same group, the encoding end processes the feature map according to the group of feature domain optical flows corresponding to the feature map to obtain an intermediate feature map corresponding to the feature map, and further, the encoding end fuses the intermediate feature map corresponding to the feature map to obtain a prediction feature map corresponding to the feature map. Finally, the encoding end encodes the image frame according to the prediction feature maps corresponding to all the feature maps of the reference frame to obtain the target code stream. In this way, in the image encoding process, if the reference frame corresponds to a plurality of feature maps, the encoding end can divide the plurality of feature maps into one group or a plurality of groups, the feature maps belonging to the same group share an optical flow set, and the intermediate feature maps corresponding to the feature maps are fused to obtain the prediction feature map, thereby avoiding the problem of more redundancy and lower accuracy when the encoding end encodes the code stream according to the feature map when the reference frame or the image frame has more information, and improving the accuracy of image encoding. It is worth noting that the number of channels of the feature maps belonging to different groups can be different.

[0025] In another optional implementation, the encoding end fuses the one or more intermediate feature maps to obtain the first prediction feature map of the first image frame, including: the encoding end obtains one or more weights of the one or more intermediate feature maps, wherein one intermediate feature map corresponds to one weight; and the encoding end processes the intermediate feature maps corresponding to the one or more weights based on the one or more weights, and adds all the processed intermediate feature maps to obtain the first prediction feature map. The weight is used to indicate the weight of the intermediate feature map in the first prediction feature map. In this embodiment, for a plurality of intermediate feature maps corresponding to the first feature map, the weight of each intermediate feature map can be different, in other words, the encoding end can set different weights for the intermediate feature maps according to the requirements of image encoding. For example, if some intermediate feature maps correspond to images that are relatively blurred, the weight of these intermediate feature maps is reduced, thereby improving the clarity of the first image.

[0026] In another optional implementation, the encoder fuses the one or more intermediate feature maps to obtain the first predicted feature map of the first image frame, including: the encoder inputs the one or more intermediate feature maps corresponding to the first feature map into a feature fusion model to obtain the first predicted feature map. The feature fusion model includes a convolution network layer used for fusing the intermediate feature maps. In the image encoding process, since the computing resources required for feature map fusion are less than the computing resources required for feature map encoding, the encoder fuses the plurality of intermediate feature maps by using the feature fusion model, and then the encoder encodes the image frame based on the predicted feature map obtained by fusing the plurality of intermediate feature maps, thereby reducing the computing resources required for image encoding and improving the efficiency of image encoding.

[0027] In a third aspect, an image decoding apparatus is provided. The image decoding apparatus can be applied to a decoder, or a video coding system supporting the implementation of the foregoing image decoding method. The image decoding apparatus includes various modules for performing the image decoding method in the first aspect or any possible implementation of the first aspect. For example, the image decoding apparatus includes a bitstream unit, a processing unit, a fusion unit, and a decoding unit.

[0028] The bitstream unit is configured to parse a bitstream to obtain at least one optical flow set. The at least one optical flow set includes a first optical flow set corresponding to a first feature map of a reference frame of a first image frame. The first optical flow set includes one or more feature domain optical flows, and each of the one or more feature domain optical flows is used to indicate motion information between the feature map of the first image frame and the first feature map.

[0029] The processing unit is configured to process the first feature map based on the one or more feature domain optical flows included in the first optical flow set to obtain one or more intermediate feature maps corresponding to the first feature map.

[0030] The fusion unit is configured to fuse the one or more intermediate feature maps to obtain a first predicted feature map of the first image frame.

[0031] The decoding unit is configured to decode the first image frame based on the first predicted feature map to obtain a first image.

[0032] The beneficial effects can be referred to the description of any of the aspects of the first aspect, which will not be repeated here. The image decoding apparatus has the functions of implementing the behaviors in the method examples of any of the aspects of the first aspect. The functions can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions.

[0033] In an optional implementation, the processing unit is specifically configured to: parse the bitstream to obtain a first feature map of the reference frame, and perform optical flow wrapping on the first feature map based on a first feature domain optical flow to obtain an intermediate feature map corresponding to the first feature domain optical flow; the first feature domain optical flow is any one of the one or more feature domain optical flows included in the first optical flow set.

[0034] In another optional implementation, the first optical flow set also corresponds to a second feature map of the reference frame. The processing unit is further configured to process the second feature map based on the one or more feature domain optical flows included in the first optical flow set to obtain one or more intermediate feature maps corresponding to the second feature map. The fusion unit is further configured to fuse the one or more intermediate feature maps corresponding to the second feature map to obtain a second prediction feature map of the first image frame. The decoding unit is specifically configured to decode the first image frame based on the first prediction feature map and the second prediction feature map to obtain the first image.

[0035] In another optional implementation, the fusion unit is specifically configured to: obtain one or more weights of the one or more intermediate feature maps, wherein one intermediate feature map corresponds to one weight, and the weight is used to indicate a weight of the intermediate feature map in the first prediction feature map; and process the intermediate feature maps corresponding to the one or more weights based on the one or more weights, and add all the processed intermediate feature maps to obtain the first prediction feature map.

[0036] In another optional implementation, the fusion unit is specifically configured to: input the one or more intermediate feature maps corresponding to the first feature map into a feature fusion model to obtain the first prediction feature map. The feature fusion model includes a convolutional network layer used to fuse the intermediate feature maps.

[0037] In another optional implementation, the image decoding apparatus further includes an obtaining unit and an enhancement unit. The obtaining unit is configured to obtain a feature map of the first image. The fusion unit is further configured to obtain an enhanced feature map based on the feature map of the first image, the first feature map, and the first prediction feature map. The enhancement unit is configured to process the first image based on the enhanced feature map to obtain a second image. The second image has a higher definition than the first image.

[0038] In another optional implementation, the enhancement unit is specifically configured to: obtain an enhanced layer image of the first image according to the enhanced feature map; and reconstruct the first image based on the enhanced layer image to obtain the second image.

[0039] In a fourth aspect, an image encoding apparatus is provided. The image encoding apparatus can be applied to an encoding end, or a video codec system supporting implementation of the aforementioned image encoding method. The image encoding apparatus comprises various modules for performing the image encoding method in the second aspect or any possible implementation of the second aspect. For example, the image encoding apparatus comprises an obtaining unit, a processing unit, a fusing unit and an encoding unit.

[0040] The obtaining unit is configured to obtain a feature map of the first image frame and a first feature map of a reference frame of the first image frame.

[0041] The processing unit is configured to obtain at least one optical flow set according to the feature map of the first image frame and the first feature map. The at least one optical flow set comprises a first optical flow set corresponding to the first feature map of the reference frame of the first image frame. The first optical flow set comprises one or more feature domain optical flows. Any one of the one or more feature domain optical flows is used to indicate motion information between the feature map of the first image frame and the first feature map.

[0042] The processing unit is further configured to process the first feature map based on the one or more feature domain optical flows included in the first optical flow set, to obtain one or more intermediate feature maps corresponding to the first feature map.

[0043] The fusing unit is configured to fuse the one or more intermediate feature maps, to obtain a first prediction feature map of the first image frame.

[0044] The encoding unit is configured to encode the first image frame according to the first prediction feature map, to obtain a bitstream.

[0045] Optionally, the bitstream comprises a feature domain optical flow bitstream corresponding to the at least one optical flow set, and a residual bitstream of an image region corresponding to the first prediction feature map.

[0046] The beneficial effects can be referred to the description of any of the second aspects, which will not be repeated here. The image encoding apparatus has the functions of implementing the behaviors in the method examples of any of the first aspects. The functions can be implemented by hardware, or by hardware executing corresponding software. The hardware or software comprises one or more modules corresponding to the functions.

[0047] In an optional implementation, the processing unit is specifically configured to: and, for the one or more feature domain optical flows included in the first optical flow set, perform optical flow wrapping on the first feature map based on a first feature domain optical flow among the one or more feature domain optical flows, to obtain an intermediate feature map corresponding to the first feature domain optical flow. The first feature domain optical flow is any one of the one or more feature domain optical flows included in the first optical flow set.

[0048] In another possible implementation manner, the first optical flow set also corresponds to a second feature map of the reference frame. The processing unit is further configured to process the second feature map based on the one or more feature domain optical flows comprised in the first optical flow set to obtain one or more intermediate feature maps corresponding to the second feature map. The fusion unit is further configured to fuse the one or more intermediate feature maps corresponding to the second feature map to obtain a second predicted feature map of the first image frame. The encoding unit is specifically configured to encode the first image frame based on the first predicted feature map and the second predicted feature map to obtain the bitstream.

[0049] In another possible implementation manner, the fusion unit is specifically configured to: obtain one or more weights of the one or more intermediate feature maps, wherein one intermediate feature map corresponds to one weight, and the weight is used to indicate a weight of the intermediate feature map in the first predicted feature map; and process the intermediate feature maps corresponding to the one or more weights based on the one or more weights, and add all the processed intermediate feature maps to obtain the first predicted feature map.

[0050] In another possible implementation manner, the fusion unit is specifically configured to: input the one or more intermediate feature maps corresponding to the first feature map into a feature fusion model to obtain the first predicted feature map. The feature fusion model comprises a convolution network layer configured to fuse the intermediate feature maps.

[0051] In a fifth aspect, an image coding apparatus is provided, which comprises at least one processor and a memory. The memory is configured to store program codes. When the processor invokes the program codes, the operation steps of the image decoding method in the first aspect or any possible implementation manner of the first aspect are performed. For example, the image coding apparatus can be a decoding end or a video decoder included in a video coding system.

[0052] Alternatively, when the processor invokes the program codes in the memory, the operation steps of the image encoding method in the second aspect or any possible implementation manner of the second aspect are performed. For example, the image coding apparatus can be an encoding end or a video encoder included in a video coding system.

[0053] In a sixth aspect, a computer readable storage medium is provided, which stores computer programs or instructions. When the computer programs or instructions are executed by an electronic device, the operation steps of the image decoding method in the first aspect or any possible implementation manner of the first aspect are performed, and / or the operation steps of the image encoding method in the second aspect or any possible implementation manner of the second aspect are performed. For example, the electronic device refers to the image coding apparatus.

[0054] In a seventh aspect, another computer-readable storage medium is provided, which stores a bitstream obtained according to the image encoding method in the second aspect or any possible implementation manner of the second aspect. For example, the bitstream can include: a feature domain optical flow bitstream corresponding to the at least one optical flow set in the second aspect, and a residual bitstream of an image region corresponding to the first predicted feature map.

[0055] In an eighth aspect, a video coding system is provided, which includes an encoding end and a decoding end. The decoding end can perform the operation steps of the image decoding method in the first aspect or any possible implementation manner of the first aspect. The encoding end can perform the operation steps of the image encoding method in the second aspect or any possible implementation manner of the second aspect. The beneficial effects can refer to the description of any aspect of the first aspect or any aspect of the second aspect, which will not be repeated here.

[0056] In a ninth aspect, a computer program product is provided. When the computer program product runs on a computer, the electronic device is caused to perform the operation steps of the method in the first aspect or any possible implementation manner of the first aspect, and / or the operation steps of the method in the first aspect or any possible implementation manner of the first aspect. For example, the electronic device refers to the image coding apparatus.

[0057] In a tenth aspect, a chip is provided, which includes a control circuit and an interface circuit. The interface circuit is used to receive a signal from another device outside the electronic device and transmit the signal to a processor, or send a signal from the processor to another device outside the electronic device. The processor is used to implement the operation steps of the method in the first aspect or any possible implementation manner of the first aspect, and / or the operation steps of the method in the second aspect or any possible implementation manner of the second aspect, through a logic circuit or an execution code instruction. For example, the electronic device refers to the image coding apparatus.

[0058] On the basis of the implementation manners of the aspects provided in the present application, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 An exemplary block diagram of a video coding system provided in the present application;

[0060] Figure 2 A mapping relationship diagram of optical flow and color provided in the present application;

[0061] Figure 3 A schematic diagram of optical flow mapping provided in the present application;

[0062] Figure 4An architecture schematic diagram of a video compression system provided in the present application;

[0063] Figure 5 A flow schematic diagram of an image encoding method provided in the present application;

[0064] Figure 6 A structure schematic diagram of a feature extraction network provided in the present application;

[0065] Figure 7 A structure schematic diagram of an optical flow estimation network provided in the present application;

[0066] Figure 8 A flow schematic diagram of optical flow mapping and feature fusion provided in the present application;

[0067] Figure 9 A flow schematic diagram of an image decoding method provided in the present application Figure 1 ;

[0068] Figure 10 A structure schematic diagram of a feature encoding network and a feature decoding network provided in the present application;

[0069] Figure 11 A flow schematic diagram of an image decoding method provided in the present application Figure 2 ;

[0070] Figure 12 A structure schematic diagram of an image reconstruction network provided in the present application;

[0071] Figure 13 A schematic diagram of a variability convolution network provided in the present application;

[0072] Figure 14 A schematic diagram of efficiency comparison provided in the present application;

[0073] Figure 15 A structure schematic diagram of an image encoding apparatus provided in the present application;

[0074] Figure 16 A structure schematic diagram of an image decoding apparatus provided in the present application;

[0075] Figure 17 A structure schematic diagram of an image decoding apparatus provided in the present application. DETAILED DESCRIPTION

[0076] Embodiments of the present application provide an image decoding (encoding) method, comprising: obtaining, by a decoding end (or an encoding end), a plurality of intermediate feature maps between a set of feature domain optical flows (or an optical flow set) of an image frame and a feature map of a reference frame, fusing the plurality of intermediate feature maps to obtain a predicted feature map of the image frame, and decoding (or encoding) the image frame according to the predicted feature map to obtain a target image (or a code stream) corresponding to the image frame. Taking an image decoding process as an example, for a single feature domain optical flow, the error of a pixel in the feature domain optical flow is less than the error of a pixel in an image domain optical flow, therefore, the decoding error caused by the intermediate feature map determined by the decoding end according to the feature domain optical flow is lower than the decoding error caused by the image domain optical flow in the prior art, in other words, the decoding end decodes the image frame according to the feature domain optical flow, reduces the decoding error caused by the image domain optical flow between two adjacent image frames, and improves the accuracy of image decoding. Moreover, the decoding end obtains a plurality of intermediate feature maps by processing the feature map of the reference frame based on a plurality of feature domain optical flows, and fuses the plurality of intermediate feature maps to determine the predicted feature map of the image frame, that is, the decoding end fuses the plurality of intermediate feature maps determined by the optical flow set and the feature map of the reference frame to obtain the predicted feature map of the image frame, and the predicted feature map contains more image information, so that when the decoding end decodes the image frame based on the fused predicted feature map, the problem that a single intermediate feature map is difficult to accurately express the first image is avoided, and the accuracy of image decoding and the image quality (such as image definition) are improved. The schemes provided by the present application will be described below in combination with embodiments. In order to describe each of the following embodiments clearly and concisely, a brief introduction to related technologies is given first.

[0077] Figure 1 An exemplary block diagram of a video coding system provided by the present application is shown in FIG. 1. As used herein, the term “video coder” generally refers to both a video encoder and a video decoder. In the present application, the term “video coding” or “coding” can generally refer to video encoding or video decoding. For example, an encoder or a decoder can be collectively referred to as an image coding device.

[0078] As Figure 1As shown in FIG. 1, a video coding system includes an encoding end 10 and a decoding end 20. The encoding end 10 generates encoded video data. Thus, the encoding end 10 can be referred to as a video encoding device. The decoding end 20 can decode the encoded video data (e.g., video including one or more image frames) generated by the encoding end 10. Thus, the decoding end 20 can be referred to as a video decoding device. Various implementations of the encoding end 10, the decoding end 20, or both, can include one or more processors and a memory coupled to the one or more processors. The memory can include, but is not limited to, random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), a floppy disk, a CD-ROM, a DVD, a Blu-ray® Disc, or any other medium that can be used to store desired program code in the form of instructions or data structures that can be accessed by a computer.

[0079] The encoding end 10 and the decoding end 20 can comprise various devices, including a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a telephone handset such as a so-called “smart” phone, a television, a camera, a display device, a digital media player, a video gaming console, an automobile computer, or similar device.

[0080] The decoding end 20 can receive encoded video data from the encoding end 10 via a link 30. The link 30 can comprise one or more media or devices capable of moving the encoded video data from the encoding end 10 to the decoding end 20. In one example, the link 30 can comprise one or more communication media to enable the encoding end 10 to transmit encoded video data directly to the decoding end 20 in real-time. In such example, the encoding end 10 can modulate encoded video data according to a communication standard, such as a wireless communication protocol, and can transmit the modulated video data to the decoding end 20. The one or more communication media can include wireless and / or wired communication media, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media can form part of a packet-based network, such as a local area network, a wide-area network, or a global network such as the Internet. The one or more communication media can include routers, switches, base stations, or other equipment that facilitates communication from the encoding end 10 to the decoding end 20.

[0081] In another example, encoded data can be output from output interface 140 to a storage device 40. Similarly, encoded data can be accessed from storage device 40 by input interface 240. Storage device 40 can include any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, Digital video disc (DVD), compact disc read-only memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.

[0082] In another example, storage device 40 can correspond to a file server or another intermediate storage device that can hold the encoded video generated by encoding end 10. Decoding end 20 can access stored video data from storage device 40 via streaming or download. The file server can be any type of server capable of storing encoded video data and transmitting encoded video data to decoding end 20. Example file servers include web servers (e.g., for a website), file transfer protocol (FTP) server, network attached storage (NAS) devices, or local disk drives. Decoding end 20 can access the encoded video data through any standard data connection, including an Internet connection. This can include a wireless channel (e.g., wireless fidelity (Wi-Fi) connection), a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both that is suitable for accessing encoded video data stored on a file server. The transmission of encoded video data from storage device 40 can be a streaming transmission, a download transmission, or a combination of both.

[0083] The decoding method of images provided by the present disclosure can be applied to video coding to support a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, streaming video transmissions (e.g., via the Internet), encoding of video data for storage on a data storage medium, decoding of video data stored on a data storage medium, or other applications. In some examples, a video coding system can be used to support one-way or two-way video transmission to support applications such as video streaming, video

[0084] Figure 1The video coding system illustrated in FIG. 1 is merely one example, and the techniques of this disclosure can be applicable to video coding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data is retrieved from local storage, streamed over a network, and so on. A video encoding device can encode data and store the data to memory, and / or a video decoding device can retrieve data from memory and decode the data. In many examples, encoding and decoding are performed by devices that do not communicate with each other, but rather only encode data to memory and / or retrieve data from memory and decode the data.

[0085] In Figure 1 In the example of FIG. 1, the encoding end 10 includes a video source 120, a video encoder 100, and an output interface 140. In some examples, the output interface 140 can include a modulator / demodulator (modem) and / or a transmitter. The video source 120 can comprise a video capture device (e.g., a video camera), a video archive containing previously captured video data, a video feed interface to receive video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of such sources of video data.

[0086] The video encoder 100 can encode video data from the video source 120. In some examples, the encoding end 10 transmits the encoded video data directly to the decoding end 20 via the output interface 140. In other examples, the encoded video data can also be stored onto the storage device 40 for later access by the decoding end 20 for decoding and / or playing back.

[0087] In Figure 1 In the example of FIG. 1, the decoding end 20 includes an input interface 240, a video decoder 200, and a display device 220. In some examples, the input interface 240 includes a receiver and / or a modem. The input interface 240 can receive encoded video data via the link 30 and / or from the storage device 40. The display device 220 can be integrated with the decoding end 20 or can be external to the decoding end 20. In general, the display device 220 displays decoded video data. The display device 220 can comprise a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0088] Although Figure 1Although not shown, in some aspects, video encoder 100 and video decoder 200 can each be integrated with an audio encoder and decoder, and can include appropriate MUX-DEMUX units, or other hardware and software, to handle encoding of both audio and video in a common data stream or separate data streams. If applicable, MUX-DEMUX units can conform to the ITU H.223 multiplexer protocol, or other protocols such as user datagram protocol (UDP).

[0089] Video encoder 100 and video decoder 200 each can be implemented as any of a variety of a circuitry, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, or any combinations thereof. If implemented partially in software, a device can store instructions for the software in any suitable non-transitory computer-readable storage medium(s) and can execute the instructions in hardware using one or more processors to

[0090] This application can generally refer to video encoder 100 as "signaling" or "transmitting" certain information to another device, such as video decoder 200. The term "signaling" or "transmitting" can generally refer to the communication of syntax elements and / or other data used to decode compressed video data. This communication can occur in real-time or near real-time. Alternatively, this communication can occur after a period of time, such as when syntax elements are stored in an encoded bitstream to a computer-readable storage medium at the time of encoding, which a decoding device can then retrieve at any time after the syntax elements are stored to this medium.

[0091] A video sequence typically includes a series of video frames or pictures. A group of pictures (GOP) illustratively comprises a series, one or more video pictures. A GOP can include syntax data in the header of the GOP, the header of one or more of the pictures, or elsewhere, that describes a number of pictures included in the GOP. Each slice of a picture can include slice syntax data that describes an encoding mode for the respective picture. Video encoder 100 generally operates on video blocks within individual video slices in order to encode the video data. A video block can correspond to a decoding node within a CU. Video blocks can have fixed or varying sizes, and can differ in size according to the designated decoding standard.

[0092] In some implementable implementations, video encoder 100 can utilize a predefined scan order to scan quantized transform coefficients to produce a serialized vector that can be entropy encoded. In other implementable implementations, video encoder 100 can perform an adaptive scan. After scanning the quantized transform coefficients to form a one-dimensional vector, video encoder 100 can entropy decode the one-dimensional vector according to context-based adaptive variable-length code (CAVLC), context-based adaptive binary arithmetic coding (CABAC), syntax-based adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy decoding methodology. Video encoder 100 can also entropy encode syntax elements associated with the encoded video data for use by video decoder 200 in decoding the video data.

[0093] To perform CABAC, video encoder 100 can assign a context within a context model to a symbol to be transmitted. The context can relate to whether an adjacent value of the symbol is non-zero or not. To perform CAVLC, video encoder 100 can select a variable length code for a symbol to be transmitted. Codewords in variable-length code (VLC) can be constructed such that relatively shorter codes correspond to more likely symbols, while longer codes correspond to less likely symbols. In this way, the use of VLC can achieve a bit rate savings over using equal length codewords for every symbol to be transmitted. The probability in CABAC can be determined based on a context assigned to a symbol.

[0094] The present application can refer to the image currently being decoded by the video decoder as a current image.

[0095] In one example, the video coding system provided by the present application can be applied to a video compression scenario, such as an artificial intelligence (AI) video encoding / decoding module.

[0096] In another example, the video coding system provided by the present application can be applied to a scenario of storing compressed video files for different services, such as data (image or video) storage of a terminal album, video monitoring, Huawei TM cloud, etc.

[0097] In yet another example, the video coding system provided by the present application can be applied to a scenario of transmitting compressed video files, such as Huawei TM cloud, video live, etc. For example, when the video coding system is applied to a live technology, the process of sending the data stream to the outside (such as a server cluster supporting the live technology) by the output interface 140 can also be referred to as a push stream, and the process of sending the data stream to the input interface 240 by the server cluster can also be referred to as a distribution.

[0098] The above examples are only possible application scenarios of the video coding system provided by the present embodiment, and should not be construed as a limitation of the present application.

[0099] In the embodiments provided by the present application, in order to reduce the data amount of the code stream and improve the transmission speed of the code stream between multiple devices, the encoding end 10 can use an optical flow mapping method to encode and compress the image frames. Correspondingly, the decoding end 20 can also use the optical flow mapping method to decode and reconstruct the image frames.

[0100] The optical flow represents the motion speed and motion direction of the pixels in the adjacent two image frames. For example, the image domain optical flow represents the motion information between the pixels in the adjacent two image frames, which can include the motion speed and motion direction between the pixels in the adjacent two image frames, and the feature domain optical flow represents the motion information between the feature maps of the adjacent two image frames, which indicates the motion speed and motion direction between the feature map of an image frame and the feature map of the adjacent image frame of the image frame. For example, the adjacent image frame (such as image frame 2) of the aforementioned image frame (such as image frame 1) can also be referred to as the reference frame of the image frame 1.

[0101] Optical flow has two directions in the time dimension: direction ①, the optical flow from the previous frame to the next frame; and direction ②, the optical flow from the next frame to the previous frame. Optical flow in one direction is usually represented digitally, such as using a three-dimensional array [2, h, w]. "2" indicates that the optical flow contains two channels, where the first channel represents the offset direction and magnitude of the image in the x-direction, and the second channel represents the offset direction and magnitude of the image in the y-direction. h is the height of the image, and w is the width of the image. In the x-direction, a positive value indicates that the object moves to the left, while a negative value indicates that the object moves to the right; in the y-direction, a positive value indicates that the object moves upward, while a negative value indicates that the object moves downward.

[0102] Figure 2 The mapping diagram of optical flow and color provided in this application Figure 2 In (A), the arrows indicate the direction of the optical flow in the image, and the length of the arrows indicates the magnitude of the optical flow in the image; Figure 2 Figure (B) shows the color and brightness of the image determined based on this optical flow. For example, color represents the direction of the optical flow, and brightness represents the magnitude of the optical flow; for instance, the greater the brightness, the greater the corresponding value of the optical flow. It is worth noting that... Figure 2 (B) uses grayscale as an example to illustrate the color of the image, but it should not be construed as a limitation of this application. For more information on the mapping relationship between optical flow and the image, please refer to the relevant content of conventional techniques, which will not be elaborated here.

[0103] The method of predicting the current frame using a reference frame and the optical flow between two frames is called optical flow mapping (Warping), usually represented as... in, For the predicted frame, x t-1 For the reference frame, v t Given the optical flow from the reference frame to the predicted frame, the decoder can infer the position of the pixel in the current frame corresponding to the reference frame based on the known reference frame and optical flow. By interpolating based on the position of the corresponding reference frame, the estimated value of the pixel in the current frame can be obtained.

[0104] Optical flow mapping includes forward wrapping and backward wrapping. For example... Figure 3 As shown, Figure 3 A schematic diagram of optical flow mapping provided for this application: Figure 3 In this context, A stands for forward mapping, which means that the decoding end predicts the next frame image (image frame 2) based on the previous frame image (image frame 1) and the optical flow between the previous and next frames. Figure 3 In this context, B stands for backward mapping, which means that the decoding end predicts the previous frame image (image frame 1) based on the image of the next frame (image frame 2) and the optical flow between the previous and next frames.

[0105] Below Figures 1 to 3Based on this, embodiments of this application provide a framework towards deep video compression in feature space (FVC), such as... Figure 4 As shown, Figure 4 The diagram below illustrates the architecture of the video compression system 400 provided in this application. The video compression system 400 includes: a motion estimation module 410, an optical flow compression module 420, a motion compensation module 430, a residual compression module 440, an entropy coding module 450, and a multi-frame feature fusion module 460. The motion estimation module 410, optical flow compression module 420, and motion compensation module 430 can be collectively referred to as variable motion compensation components.

[0106] Image frame X t After feature extraction, the feature map F of the image frame is obtained. t Reference frame X t-1 The feature map F of the reference frame is obtained after feature extraction. t-1 Reference frame X t-1 It can be stored in the decoded frame buffer, which provides data storage space for multiple frames.

[0107] Motion estimation module 410 according to F t and F t-1 Determine the characteristic domain optical flow O between the image frame and the reference frame. t .

[0108] Optical flow compression module 420 pairs O t Compression is performed to obtain the feature domain optical flow code stream O` t .

[0109] The motion compensation module 430 uses the feature map F of the reference frame. t-1 And decoding feature domain optical flow bitstream O` t Perform feature prediction to determine image frame X t The corresponding predicted feature map F t_pre .

[0110] The residual in the feature domain is R t =F t -F t_preThe residual compression module 440 obtains the feature domain residual R. t Output the compressed decoding residual R` t The predicted feature map F t_pre and decoding residual R` t It can be used to determine the initial reconstruction features F` of an image frame. t_intial .

[0111] Furthermore, the multi-frame feature fusion module 460, based on the initial reconstructed features F` t_intial Reconstructed feature maps corresponding to multiple reference frames (e.g.) Figure 4 The F shown t-1_ref F t-2_ref and F t-3_ref Determine image frame X t The final reconstructed feature map F` t_final The final reconstructed feature map F` t_final Image frame X is obtained through frame reconstruction. t The corresponding reconstructed frame X` t .

[0112] Entropy coding module 450 is used to encode the optical flow O in the feature domain. t and feature domain residual R t At least one of them is encoded to obtain a binary code stream.

[0113] Figures 1 to 4 These are merely examples provided in this application. In some examples, the encoder 10, decoder 20, video encoder 100, video decoder 200, and video encoding / decoding system may include more or fewer components or units, and this application does not limit them.

[0114] The specific implementation of the image encoding method provided in this embodiment will be described in detail below with reference to the accompanying drawings.

[0115] like Figure 5 As shown, Figure 5 This is a flowchart illustrating the image encoding method provided in this application, which can be applied to... Figure 1 The video encoding / decoding system shown or Figure 4 The video compression system shown is exemplary. The image encoding method can be executed by the encoding end 10 or the video encoder 100. Here, the image encoding method provided in this embodiment is illustrated using the encoding end 10 as an example. Figure 5 As shown, the image encoding method provided in this embodiment includes the following steps S510 to S560.

[0116] S510, the encoding end acquires the feature map of image frame 1 and the first feature map of the reference frame of image frame 1.

[0117] The image frame 1 and the reference frame can belong to a GOP included in a video. For example, the video includes one or more image frames, the image frame 1 can be any image frame in the video, and the reference frame can be an image frame adjacent to the image frame 1 in the video, for example, the reference frame is an image frame adjacent to the image frame 1 before the image frame 1, or the reference frame is an image frame adjacent to the image frame 1 after the image frame 1. It should be noted that in some cases, the image frame 1 can also be referred to as a first image frame.

[0118] In an optional example, the manner in which the encoding end obtains the feature map can include, but is not limited to, the encoding end being implemented based on a neural network model. Assuming that the size of each image frame in the video is [3, H, W], that is, the image frame has 3 channels, a height of H, and a width of W, and the size of the feature map corresponding to the image frame is [N, H / s, W / s], where s is a positive integer, and here it is assumed that s = 2 and N = 64, and the neural network model is a feature extraction network, the feature extraction process of the feature map of the image frame 1 and the first feature map of the reference frame provided in this embodiment is described as follows: Figure 6 Figure 6 The structural schematic diagram of the feature extraction network provided in this application includes 1 convolutional layer (convolutional layer, conv) and 3 residual block (residual block, resblock) processing layers, the convolution kernel of the convolutional layer is 64x5x5 / 2, and the convolution kernel size of the resblock processing layer is 64x3x3. Further, the encoding end determines that the feature map corresponding to the image frame 1 is f t , and the first feature map of the reference frame is f t-1 .

[0119] It should be noted that Figure 6 this is only an example of a feature extraction manner provided in this embodiment, and should not be construed as a limitation on the present application. When the size of the image frame and the reference frame is other, Figure 6 the parameters of each network layer in the feature extraction network shown can also be different. In some optional implementation manners, the encoding end can also extract the features of the image frame 1 and the reference frame in other manners, which are not limited by the present application.

[0120] S520, the encoding end obtains at least one optical flow set according to the feature map of the image frame 1 and the first feature map.

[0121] The first optical flow set in the at least one optical flow set corresponds to the first feature map described above, for example, the first optical flow set can be the optical flow set 1 shown in Figure 5 .

[0122] The first optical flow set can include one or more feature domain optical flows v t ​The optical flow v in this characteristic domain t This motion information is used to indicate the motion between the feature map and the first feature map of image frame 1 (or the first image frame). This motion information can be used to indicate the motion information and direction of motion between the feature map and the first feature map of image frame 1.

[0123] In the embodiments of this application, the process of acquiring the optical flow set at the encoding end is actually an optical flow estimation process. As a feasible implementation, this optical flow estimation process can be performed by... Figure 4 The motion estimation module 410 shown is used to implement this, and the motion estimation module 410 can be supported by the encoder.

[0124] In some optional examples, the encoder can utilize an optical flow estimation network to determine the aforementioned optical flow set. For example, such as Figure 7 As shown, Figure 7 The diagram below illustrates the structure of the optical flow estimation network provided in this application. This network includes an upsampling network and a downsampling network. The upsampling network comprises three network layers 1, each consisting of three residual block processing layers (with 64×3×3 kernels) and one convolutional layer (with 64×5×5 / 2 kernels). The downsampling network comprises three network layers 2, each consisting of one convolutional layer (with 64×5×5 / 2 kernels) and three residual block processing layers (with 64×3×3 kernels). Optionally, the residual block processing layer may consist of one convolutional layer (with 64×3×3 kernels), one activation layer, and one convolutional layer (with 64×3×3 kernels). The activation layer can be a rectified linear unit (ReLU) layer or a parametric rectified linear unit (PReLU), etc.

[0125] It is worth noting that, Figure 7 The optical flow estimation network provided in this application is merely an example and should not be construed as limiting the application. The kernel size, number of feature map channels in the input network layer, downsampling position, number of convolutional layers, and network activation layers of each network layer can all be adjusted. In some more complex scenarios, the optical flow estimation network can also use more complex network structures.

[0126] S530: The encoding end performs feature domain optical flow encoding on the optical flow set obtained from S520 to obtain the feature domain optical flow code stream.

[0127] The optical flow code stream in this feature domain includes the code stream corresponding to optical flow set 1.

[0128] In some cases, the feature field optical flow code stream can be a binary file, or the feature field optical flow code stream can also be other types of files conforming to multimedia transmission protocols, without limitation.

[0129] S540, the encoding end processes the first feature map based on the feature field optical flow included in the optical flow set 1, and obtains one or more intermediate feature maps corresponding to the first feature map.

[0130] Here, a feasible processing mode is provided, for one or more feature field optical flows included in the optical flow set 1, the encoding end performs optical flow mapping (wrapping) on the first feature map based on the first feature field optical flow (v1) therein, and obtains an intermediate feature map corresponding to the first feature field optical flow. It is worth noting that one feature field optical flow and the first feature map wrapping will obtain one intermediate feature map, and the number of intermediate feature maps corresponding to the first feature map is consistent with the number of feature field optical flows contained in the optical flow set 1. As shown in Figure 8 Figure 8 The flowchart of optical flow mapping and feature fusion provided in the present application, the optical flow set 1 contains multiple feature field optical flows, such as v1 to v m , m is a positive integer. For the first feature map f t , the encoding end respectively wraps each feature field optical flow contained in the optical flow set 1 with the first feature map f t , and obtains m intermediate feature maps.

[0131] S550, the encoding end fuses one or more intermediate feature maps, and obtains the first prediction feature map of the first image frame.

[0132] In general technology, the decoding end obtains multiple image field optical flows between the image frame and the reference frame, and obtains multiple images from the reference frame according to the multiple image field optical flows, so as to fuse the multiple images to obtain the target image corresponding to the image frame. Therefore, the decoding end needs to predict the pixel values of multiple images when decoding one frame of image, and fuse the multiple images to obtain the target image, which leads to a large amount of calculation resources required for image decoding, and the efficiency of the decoding end decoding the video according to the image field optical flow is low.

[0133] ​In contrast, in one possible case provided in this embodiment, the encoding end inputs the aforementioned one or more intermediate feature maps into a feature fusion model to obtain the first prediction feature map. As the feature fusion model includes a convolution network layer for fusing the intermediate feature maps, in the image decoding process, as the calculation resources required for feature map fusion are less than the calculation resources required for feature map decoding, the decoding end fuses the plurality of intermediate feature maps by using the feature fusion model, and then the decoding end decodes the image frame based on the prediction feature map obtained by fusing the plurality of intermediate feature maps, that is, the decoding end only needs to predict the pixel values of the image positions indicated by the prediction feature map in the image according to the prediction feature map, without predicting all pixel values of the plurality of images, thereby reducing the calculation resources required for image decoding and improving the efficiency of image decoding.

[0134] In another possible case provided in this embodiment, the encoding end obtains one or more weights of the aforementioned one or more intermediate feature maps, processes the intermediate feature maps corresponding to the one or more weights based on the one or more weights, and adds all the processed intermediate feature maps to obtain the first prediction feature map. The weight is used to indicate the weight of the intermediate feature map in the first prediction feature map.

[0135] In this embodiment, for the plurality of intermediate feature maps corresponding to the first feature map, the weight of each intermediate feature map can be different, in other words, the encoding end can set different weights for the intermediate feature maps according to the requirements of image encoding, for example, if some intermediate feature maps correspond to relatively blurred images, the weight of the intermediate feature map corresponding to the blurred image is reduced, thereby improving the clarity of the first image.

[0136] For example, the weight can be a mask value corresponding to each optical flow. Assuming that the optical flow set 1 includes 9 feature domain optical flows, and the weights of the feature domain optical flows are as follows: As the size of the feature domain optical flow is [2, H / s, W / s] and the size of the mask is [1, H / s, W / s], the encoding end fuses the intermediate feature maps corresponding to the 9 feature domain optical flows to obtain the first prediction feature map corresponding to the first feature map.

[0137] The above two possible cases are only examples provided in this embodiment, and should not be understood as the feature map fusion in this application being limited to the above two ways. For example, Figure 8 As shown in the figure, assuming that m=4, for the first feature map f t of the reference frame, the encoding end can wrap each feature domain optical flow included in the optical flow set 1 with the first feature map f t to obtain 4 intermediate feature maps, and then the encoding end fuses the 4 intermediate feature maps to obtain the first prediction feature map.

[0138] Please continue to refer to Figure 5 The image encoding method provided in this embodiment further includes the following step S560.

[0139] S560, the encoder encodes the residual corresponding to the first prediction feature map of the image frame 1 to obtain a residual code stream.

[0140] The residual code stream of the image region corresponding to the first prediction feature map and the feature domain optical flow code stream determined in the aforementioned S530 can be collectively referred to as the code stream corresponding to the image frame 1 (or the first image frame).

[0141] Optionally, the code stream includes the feature domain optical flow code stream corresponding to the aforementioned optical flow set and the residual code stream of the image region corresponding to the first prediction feature map.

[0142] In this embodiment, for the reference frame of the first image frame (such as the aforementioned image frame 1), the encoder determines a set of feature domain optical flows (such as the aforementioned optical flow set 1) based on the feature map of the first image frame and the first feature map of the reference frame, and processes the first feature map of the reference frame to obtain one or more intermediate feature maps; secondly, after the encoder obtains the one or more intermediate feature maps corresponding to the first feature map, the one or more intermediate feature maps are fused to obtain the first prediction feature map corresponding to the first image frame; finally, the encoder encodes the first image frame according to the first prediction feature map to obtain a code stream.

[0143] In this way, for a single feature domain optical flow, the error of the pixels in the feature domain optical flow is smaller than the error of the pixels in the image domain optical flow, so the encoding error caused by the intermediate feature map determined by the encoder based on the feature domain optical flow is lower than the encoding error caused by the image domain optical flow in the conventional technology, in other words, the encoding of the image frame by the encoder based on the feature domain optical flow reduces the encoding error caused by the image domain optical flow between the adjacent two image frames, and improves the accuracy of image encoding. Moreover, the encoder processes the feature map of the reference frame based on multiple feature domain optical flows to obtain multiple intermediate feature maps, and fuses the multiple intermediate feature maps to determine the prediction feature map of the image frame, that is, the encoder fuses the multiple intermediate feature maps determined by the optical flow set and the feature map of the reference frame to obtain the prediction feature map of the image frame, which contains more image information, so that when the encoder encodes the image frame based on the fused prediction feature map, the problem that a single intermediate feature map is difficult to accurately express the first image is avoided, and the accuracy of image encoding and the image quality (such as image definition) are improved.

[0144] As an optional implementation manner, the reference frame can correspond to multiple feature maps, such as the aforementioned first feature map and second feature map. Illustratively, the first feature map and the second feature map can be two feature maps belonging to different channels in multiple channels contained by the reference frame.

[0145] In the embodiments of the present application, the set of feature maps corresponding to the reference frame can correspond to a set of optical flows, such as the aforementioned set of optical flows 1, which can also correspond to the second feature map. In combination with the foregoing content, if the feature map f Figure 8 In combination with the foregoing content, if the feature map f t-1 is the second feature map, the encoding end can further process the second feature map based on the feature domain optical flows included in the set of optical flows 1 to obtain one or more intermediate feature maps corresponding to the second feature map (such as f Figure 8 f` t-1 f` t-m , m is a positive integer). In addition, the encoding end fuses the one or more intermediate feature maps corresponding to the second feature map to obtain the second predicted feature map of the first image frame.

[0146] Since the reference frame corresponds to multiple feature maps, such as the aforementioned first feature map and second feature map, the process of encoding the first image frame by the encoding end can include the following content: the encoding end encodes the first image frame based on the first predicted feature map and the second predicted feature map to obtain a code stream. The code stream can include residual code streams of image regions in the first image frame corresponding to the first feature map and the second feature map, and feature domain optical flow code streams corresponding to the aforementioned set of optical flows 1.

[0147] For example, assuming that 8 feature maps of the reference frame share 9 feature domain optical flows, the 8 feature maps are regarded as a group of feature maps, and in the process of image encoding, the 8 feature maps belonging to the same group can share the 9 feature domain optical flows, thereby obtaining 8x9=72 intermediate feature maps. Further, in the process of feature fusion of the intermediate feature maps by the encoding end, the 9 intermediate feature maps corresponding to one feature map are fused to obtain a predicted feature map corresponding to the one feature map. It is worth noting that in the present application, the number of shared feature domain multi-optical flows is not limited, and whether the number of feature channels of each feature map group is consistent is also not limited.

[0148] In the embodiments of the present application, the multiple feature maps (or referred to as: a set of feature maps, or a feature map group, etc.) of the reference frame can correspond to a set of optical flows (or referred to as: a set of feature domain optical flows), for example, a set of feature domain flows are shared by the multiple feature maps belonging to the same group, the encoding end processes the feature map based on the set of feature domain optical flows corresponding to the feature map, thereby obtaining an intermediate feature map corresponding to the feature map, further, the encoding end fuses the intermediate feature maps corresponding to the feature map to obtain a predicted feature map corresponding to the feature map. Finally, the encoding end encodes the image frame based on the predicted feature maps corresponding to all the feature maps of the reference frame to obtain a target code stream.

[0149] Thus, in the image encoding process, if the reference frame corresponds to multiple feature maps, the encoding end can divide the multiple feature maps into one group or multiple groups, the feature maps belonging to the same group share one optical flow set, and the intermediate feature maps corresponding to the feature maps are fused to obtain the prediction feature maps, thereby avoiding the problem that when the reference frame or the image frame has more information, the code stream obtained by the encoding end according to the feature map encoding is more redundant and has lower accuracy, and the accuracy of image encoding is improved. It is worth noting that the number of channels of the feature maps belonging to different groups can be different.

[0150] As an optional implementation manner, when the reference frame corresponds to multiple feature maps and the multiple feature maps correspond to a group of feature domain optical flows, for different feature maps of the reference frame, the encoding end can fuse the intermediate feature maps corresponding to the feature maps of the reference frame by using different feature fusion manners, so as to obtain the prediction feature maps corresponding to the feature maps of the reference frame. For example, the reference frame corresponds to feature Figure 1 and feature Figure 2 The encoding end inputs the multiple intermediate feature maps corresponding to feature Figure 1 into a feature fusion model, so as to determine the prediction feature map corresponding to feature Figure 1 The encoding end obtains the weight of each intermediate feature map corresponding to feature Figure 2 , and processes the respective intermediate feature maps according to the weights, so as to obtain the prediction feature map corresponding to feature Figure 2 In this embodiment, for different feature fusion manners, the encoding end can set the confidence of the prediction feature map output by each feature fusion manner when the encoding demand of the image is different, so as to meet different encoding demands of the user.

[0151] Corresponding to the image encoding method provided in the foregoing embodiments, to decode the code stream containing the residual code stream and the feature domain optical flow code stream, and obtain the target image or video corresponding to the code stream, the embodiments of the present application further provide an image decoding method, as shown in Figure 9 , the image decoding method provided by the present application is shown in Figure 9 the flowchart of the image decoding method provided by the present application Figure 1 , the image decoding method can be applied to Figure 1 the video coding system shown in or Figure 4 the video compression system shown in, for example, the image encoding method can be executed by the decoding end 20 or the video decoder 200, and here the decoding end 20 executing the image decoding method provided in the embodiment is taken as an example for description.

[0152] As shown in Figure 9 , the image decoding method provided in the embodiment includes the following steps S910 to S940.

[0153] S910, the decoding end parses the code stream to obtain at least one optical flow set.

[0154] The at least one optical flow set includes a first optical flow set (e.g. Figure 9 The first optical flow set 1 corresponds to the first feature map of the reference frame, and the first optical flow set 1 includes one or more feature domain optical flows, wherein any one of the one or more feature domain optical flows is used to indicate the motion information between the feature map of the first image frame and the first feature map. For more information about the optical flow set and the feature domain optical flow, refer to the relevant description of S510 above, which will not be repeated here.

[0155] Optionally, the reference frame can refer to a decoded image frame adjacent to the first image frame (an image frame before the first image frame, or an image frame after the first image frame) at the decoding end, or the reference frame is an image frame carrying complete information of an image in the code stream.

[0156] S920, the decoding end processes the first feature map based on the feature domain optical flow included in the first optical flow set 1, and obtains one or more intermediate feature maps corresponding to the first feature map.

[0157] It is worth noting that the process of processing the feature map of the reference frame according to the feature domain optical flow by the decoding end is also called feature alignment, feature prediction, or feature align, which is not limited in the present application.

[0158] Corresponding to the process of processing the feature map of the reference frame based on the first optical flow set 1 by the encoding end in S520, the decoding end can also process the first feature map in the same way as S520 above to obtain one or more intermediate feature maps corresponding to the first feature map. For example, the decoding end can wrap the first feature map based on a set of feature domain optical flows, such as the decoding end can interpolate the position of the reference frame according to a set of feature domain optical flows to obtain the predicted value of the pixel value in the first image, avoiding the decoding end to predict all pixel values in the first image according to the image domain optical flow, reducing the calculation amount required by the decoding end to decode the image, and improving the efficiency of image decoding.

[0159] S930, the decoding end fuses one or more intermediate feature maps determined in S920 to obtain a first predicted feature map of the first image frame.

[0160] Corresponding to the foregoing S550 performed at the encoding end, the decoding end can input one or more intermediate feature maps determined by S920 to a feature fusion model to obtain the first prediction feature map. If the feature fusion model includes a convolution network layer for fusing the intermediate feature maps, in the image decoding process, because the computing resources required for feature map fusion are less than the computing resources required for feature map decoding, the decoding end fuses multiple intermediate feature maps using the feature fusion model, and then decodes the image frame based on the prediction feature map obtained by fusing the multiple intermediate feature maps, thereby reducing the computing resources required for image decoding and improving the efficiency of image decoding.

[0161] Alternatively, the decoding end can also obtain one or more weights of the one or more intermediate feature maps, wherein one intermediate feature map corresponds to one weight; and the decoding end processes the intermediate feature maps based on the weights of the intermediate feature maps and adds all the processed intermediate feature maps to obtain the first prediction feature map. The weight is used to indicate the weight of the intermediate feature map in the first prediction feature map. In this embodiment, for the multiple intermediate feature maps corresponding to the first feature map, the weight of each intermediate feature map can be different, in other words, the decoding end can set different weights for the intermediate feature maps according to the requirements of image decoding. For example, if some intermediate feature maps correspond to relatively blurred images, the weights of these intermediate feature maps are reduced to improve the clarity of the first image. For the weight of the intermediate feature map, please refer to the related content of S550, which will not be described here.

[0162] S940, the decoding end decodes the first image frame according to the first prediction feature map determined by S930 to obtain the first image.

[0163] For example, the residual corresponding to the first prediction feature map in the code stream is r` t , the first prediction feature map is f` t , and the decoding end needs to reconstruct the feature map f_res = f` t + r` t to decode the first image frame, and the decoding end can decode the first image frame according to the reconstructed feature map f_res to obtain the first image.

[0164] In this embodiment, for the reference frame of the first image frame, the decoding end processes the first feature map of the reference frame based on a set of feature domain optical flows (such as the first optical flow set described above) corresponding to the first image frame to obtain one or more intermediate feature maps; secondly, after the decoding end obtains one or more intermediate feature maps corresponding to the first feature map, the decoding end fuses the one or more intermediate feature maps to obtain the first prediction feature map corresponding to the first image frame; finally, the decoding end decodes the first image frame according to the first prediction feature map to obtain the first image.

[0165] Thus, for the single feature domain optical flow, the error of the pixels in the feature domain optical flow is less than the error of the pixels in the image domain optical flow, and thus the decoding error caused by the intermediate feature map determined by the decoding end according to the feature domain optical flow is lower than the decoding error caused by the image domain optical flow in the prior art. In other words, the decoding end decodes the image frame according to the feature domain optical flow, reduces the decoding error caused by the image domain optical flow between the adjacent two image frames, and improves the accuracy of image decoding.

[0166] Further, the decoding end obtains a plurality of intermediate feature maps by processing the feature map of the reference frame based on a plurality of feature domain optical flows, and determines the predicted feature map of the image frame by fusing the plurality of intermediate feature maps, that is, the decoding end fuses the plurality of intermediate feature maps determined by the optical flow set and the feature map of the reference frame to obtain the predicted feature map of the image frame. The predicted feature map contains more image information, so that when the decoding end decodes the image frame based on the predicted feature map obtained by fusion, the problem that a single intermediate feature map is difficult to accurately express the first image is avoided, and the accuracy of image decoding and the image quality (such as image definition) are improved.

[0167] Optionally, the encoding of the feature map by the encoding end can be implemented by a feature encoding network, and the decoding of the code stream corresponding to the feature map by the decoding end can be implemented by a feature decoding network. For example, for the feature encoding network required by the encoding end and the feature decoding network required by the decoding end, a possible embodiment is provided as shown in Figure 10 . Figure 10 The structure diagram of the feature encoding network and the feature decoding network provided in the present application is shown in the figure, the feature encoding network includes three network layers 1, and the network layer 1 includes three residual block processing layers (convolution kernel is 64x3x3) and one convolution layer (convolution kernel is 64x5x5 / 2) in sequence.

[0168] The feature decoding network includes three network layers 2, and the network layer 2 includes one convolution layer (convolution kernel is 64x5x5 / 2) and three residual block processing layers (convolution kernel is 64x3x3) in sequence.

[0169] In an optional case, one residual block processing layer includes one convolution layer (convolution kernel is 64x3x3), one activation layer and one convolution layer (convolution kernel is 64x3x3) in sequence. The activation layer can be ReLU layer or PReLU, etc.

[0170] It is worth noting that, Figure 10The example of the feature encoding network and the feature decoding network provided in the embodiments of the present application should not be construed as a limitation of the present application, where the convolution kernel size of each network layer, the number of feature map channels of the input network layer, the down-sampling position, the number of convolution layers, and the network activation layer can be adjusted. In some more complex scenarios, the optical flow estimation network can also use a more complex network structure.

[0171] In an alternative implementation, the reference frame can correspond to multiple feature maps, such as the first feature map and the second feature map described above.

[0172] In the embodiments of the present application, the set of feature maps corresponding to the reference frame can correspond to a set of optical flows, such as the optical flow set 1 described above, which can also correspond to the second feature map of the reference frame. In combination with the description shown in Figure 8 As shown in the description, if the feature map f t-1 is the second feature map, the decoding end can further process the second feature map based on the feature domain optical flows included in the optical flow set 1 to obtain one or more intermediate feature maps (such as f Figure 8 as shown in the description f t-1 to f t-m , m is a positive integer). In addition, the decoding end further fuses one or more intermediate feature maps corresponding to the second feature map to obtain the second prediction feature map of the first image frame.

[0173] Since the reference frame corresponds to multiple feature maps, such as the first feature map and the second feature map described above, the decoding process of the decoding end for the first image frame can include the following content: the decoding end decodes the first image frame based on the first prediction feature map and the second prediction feature map to obtain the first image. For example, assuming that 9 feature domain optical flows are shared by every 8 feature maps of the reference frame, these 8 feature maps are regarded as a group of feature maps, and in the image decoding process, the 8 feature maps belonging to the same group can share the 9 feature domain optical flows, thereby obtaining 8x9=72 intermediate feature maps. Further, in the feature fusion process of the decoding end for the intermediate feature maps, 9 intermediate feature maps corresponding to one feature map are fused to obtain the prediction feature map corresponding to the one feature map. It is worth noting that the number of channels of the feature maps belonging to different groups can be different.

[0174] In the embodiment, the plurality of feature maps (or the group of feature maps) of the reference frame can correspond to a set of optical flow (or a group of feature domain optical flow). In other words, a group of feature domain flow is shared by the plurality of feature maps belonging to the same group. The decoding end processes the feature map according to the group of feature domain optical flow corresponding to the feature map, thereby obtaining the intermediate feature map corresponding to the feature map. Further, the decoding end fuses the intermediate feature map corresponding to the feature map to obtain the predicted feature map corresponding to the feature map. Finally, the decoding end decodes the image frame according to the predicted feature maps corresponding to all the feature maps of the reference frame to obtain the target image. In this way, in the image decoding process, if the reference frame corresponds to a plurality of feature maps, the decoding end can divide the plurality of feature maps into one group or a plurality of groups, the feature maps belonging to the same group share a set of optical flow, and the intermediate feature maps corresponding to the feature maps are fused to obtain the predicted feature map, thereby avoiding the problem that when the reference frame or the image frame has more information, the accuracy of the image reconstructed by the decoding end according to the feature map is low, and improving the accuracy of the image decoding.

[0175] In some cases, if the clarity of the first image fails to reach the expected clarity, resulting in a relatively blurred decoded first image, in order to improve the clarity of the first image and improve the video display effect, the embodiment of the present application also provides a technical solution for video enhancement, as shown in Figure 11 Figure 11 The flowchart of the image decoding method provided by the present application is shown in Figure 2 Figure 11 The image decoding method shown in the above embodiment can be combined with the image encoding method and the image decoding method provided by the above embodiment, or can be implemented independently. Here, the image decoding method shown in Figure 11 Figure 11 The image decoding method provided by the embodiment includes the following steps S1110 to S1130, as shown in

[0176] S1110, the decoding end obtains the feature map of the first image.

[0177] For example, the decoding end can obtain the feature map of the first image according to the feature extraction network shown in Figure 6

[0178] S1120, the decoding end obtains the enhanced feature map according to the feature map of the first image, the first feature map of the reference frame and the first predicted feature map.

[0179] For example, the decoding end can fuse the plurality of feature maps contained in S1120 through a feature fusion model to obtain the enhanced feature map. The feature fusion model can be the feature fusion model provided by the foregoing S550 or S930, or can be another model containing a convolution layer, such as a convolution kernel of 3x3, which is not limited by the present application.​​​​

[0180] In some possible cases, the decoding end can also set different weights for the feature map of the first image, the first feature map of the reference frame and the first predicted feature map, so as to fuse the plurality of feature maps to obtain an enhanced feature map.

[0181] In some possible examples, the enhanced feature map can be used to determine an enhanced layer image of the first image. The video quality of the enhanced layer image is higher than that of the first image, and the video quality can refer to or include at least one of signal-to-noise ratio (SNR), resolution and peak signal-to-noise ratio (PSNR). In this article, the video quality of an image can be referred to as the image quality of the image.

[0182] wherein the signal-to-noise ratio of the image refers to the ratio of the signal mean value of the image to the background standard deviation, and the "signal mean value" of the image generally refers to the average value of the gray scale of the image, and the background standard deviation can be represented by the variance of the background signal value of the image, and the variance of the background signal value refers to the noise power; for the image, the larger the signal-to-noise ratio of the image, the better the quality of the image. The resolution is the number of pixels per unit area of a single image, and the higher the resolution of the image, the better the quality of the image. PSNR is used to indicate the subjective quality of the image, and the larger the PSNR, the better the quality of the image. For more information about SNR, resolution and PSNR, please refer to the relevant description of the prior art, which will not be repeated here.

[0183] S1130, the decoding end processes the first image according to the enhanced feature map to obtain a second image.

[0184] The second image and the first image indicate the same content, but the clarity of the second image is higher than that of the first image. In some embodiments, the clarity of the image can be indicated by the peak signal-to-noise ratio (PSNR),

[0185] In this embodiment, the decoding end can fuse the feature map of the first image, the first feature map and the first predicted feature map, and perform video enhancement processing on the first image based on the enhanced feature map obtained by the fusion to obtain a second image with better clarity, thereby improving the clarity of the decoded image and the image display effect.

[0186] Optionally, the decoding end processes the first image according to the enhanced feature map to obtain a second image, including: the decoding end obtains an enhanced layer image of the first image according to the enhanced feature map, and reconstructs the first image based on the enhanced layer image to obtain the second image. In an example, the enhanced layer image can refer to an image determined by the decoding end according to the reference frame and the enhanced feature map, the decoding end adds part or all information of the enhanced layer image to the first image to obtain the second image; or, the decoding end takes the enhanced layer image as a reconstructed image of the first image, i.e., the aforementioned second image. In this example, the decoding end obtains multiple feature maps of the image at different stages, and obtains the enhanced feature map determined by the multiple feature maps, so as to reconstruct and video enhance the first image according to the enhanced feature map, and improve the clarity and display effect of the decoded image.

[0187] In a possible case, the decoding end reconstructs the first image based on an image reconstruction network, as shown in Figure 12 Figure 12 FIG. 1 shows a structural diagram of an image reconstruction network provided by the present application, which sequentially includes 3 residual block processing layers (convolution kernel is 64x3x3) and 1 deconvolution layer (convolution kernel is 3x5x5 / 2), wherein the residual block processing layer can include 1 convolution layer (convolution kernel is 64x3x3), an activation layer and 1 convolution layer (convolution kernel is 64x3x3).

[0188] It can be understood that Figure 6 Figure 7 Figure 10 Figure 12 It should be understood that the networks shown in FIGS. 1 to 5 are described by taking the size of the image frame as [3, H, W] as an example, and should not be construed as a limitation of the present application. For example, if the size of the image frame changes, the parameters of the convolution layer, the residual block processing layer and the deconvolution layer in the aforementioned networks will also change according to the needs of video coding, video enhancement, etc., which are not limited by the present application.

[0189] In the embodiments of the present application, the intermediate feature map set predicted feature map is determined by the encoding end / decoding end through optical flow mapping of the feature map of the reference frame. In some cases, the encoding end / decoding end can also process the feature map based on a deformable network (deformable convolutional networks, DCN). The DCN is based on convolution, and the following is a mathematical expression of convolution:

[0190]

[0191] where n is the size of the convolution kernel, w is the weight of the convolution kernel, F is the input feature map, p is the convolution position, and p k ​​​​This is the enumerated value of the position relative to p within the convolutional kernel. DCN is based on a network-learned offset, causing the convolutional kernel to shift its sampling points on the input feature map, concentrating on the region of interest (ROI) or the target region. Its mathematical expression is:

[0192]

[0193] The mathematical expression with a mask is as follows:

[0194]

[0195] Where Δp k For relative to p k The offset makes the convolution sampling position irregular, m(p k ) indicates position p k The mask value is for position p k Penalties. For example... Figure 13 The convolution operation shown is to obtain the center point by convolving the points in the neighborhood: (a) is the common sampling method of 3x3 convolution kernel, (b) is the change of sampling points after sampling deformable convolution plus offset, and (c) and (d) are special forms of deformable convolution. For example, in (c), multiple pixels are scaled to obtain the pixel prediction value of the corresponding position in the target image, and in (d), multiple pixels are rotated and scaled to obtain the pixel prediction value of the corresponding position in the target image.

[0196] Set the characteristic domain optical flow to the offset Δp in the aforementioned expression. k Therefore, the image encoding method and image decoding method provided in the embodiments of this application can also be implemented by DCN. It is understood that when a reference frame corresponds to multiple feature maps with different channels, each feature map can adopt the same or different DCN processing methods, such as... Figure 13 The four possible DCN processing methods are shown.

[0197] Regarding the video enhancement process provided by this solution and conventional techniques, Figure 14 A schematic diagram illustrating the efficiency comparison provided for this application. Figure 14 Two metrics are provided to compare the technical solutions provided in this application with conventional technologies: PSNR and bits per pixel (BPP). BPP refers to the number of bits used to store each pixel and is also used to indicate the resolution of an image.

[0198] Reference Figure 14It can be seen that, in multiple code points (node positions of encoding or decoding), the BPP of the scheme provided by the general technology is higher than the technical scheme provided by the embodiment of the application, that is, the storage space occupied by the code stream generated by the general technology is larger, and the network bandwidth required for transmitting the code stream is larger; but the PSNR of the scheme provided by the general technology is lower than the technical scheme provided by the embodiment of the application, that is, in the technical scheme provided by the embodiment of the application, the subjective quality of the video (or image) obtained after decoding the code stream is better.

[0199] For example, for an image with a resolution of 1080P (1920x1080), the end-to-end time of the general technical scheme for testing the frame image is 0.3206 seconds (second, s), and the end-to-end time of the technical scheme provided by the embodiment of the application for testing the frame image is 0.2188s. That is, in the video encoding and decoding process, the technical scheme provided by the embodiment of the application not only saves the code rate (reflected by BPP), but also guarantees the quality (reflected by PSNR), and reduces the coding delay of a single frame image, thereby improving the overall efficiency of video encoding and decoding.

[0200] It is worth noting that the image encoding method and the image decoding method provided by the application can not only be applied to video encoding, video enhancement, video compression and the like, but also can be applied to video prediction, video interpolation, video analysis and all video processing technology fields which need to fuse or align features between video frames.

[0201] It can be understood that, in order to realize the functions in the above-mentioned embodiments, the encoding end and the decoding end include corresponding hardware structures and / or software modules for executing various functions. Those skilled in the art should easily realize that, in combination with the units and method steps of the examples described in the embodiments disclosed in the application, the application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application scene and design constraints of the technical scheme.

[0202] Figure 15 The structure diagram of the image encoding device provided by the application is shown in the figure, and the image encoding device can be used to realize the functions of the encoding end or the video encoder in the above-mentioned method embodiments, and thus can also realize the beneficial effects possessed by the above-mentioned image encoding method embodiments. In the embodiments of the application, the image encoding device can be the encoding end 10 or the video encoder 100 as shown in Figure 1 , or a module (such as a chip) applied to the encoding end 10 or the video encoder 100.

[0203] As Figure 15As shown, the image encoding apparatus 1500 includes: an acquisition unit 1510, a processing unit 1520, a fusion unit 1530, and an encoding unit 1540. The image encoding apparatus 1500 is used to implement the above-described... Figure 5 and Figure 8 The method embodiment shown illustrates the function of the encoding end or video encoder.

[0204] When the image encoding device 1500 is used to implement Figure 5 In the method embodiment shown, the acquisition unit 1510 is used to execute S510, the processing unit 1520 is used to execute S520 and S540, the fusion unit 1530 is used to execute S550, and the encoding unit 1540 is used to execute S530 and S560.

[0205] When the image encoding device 1500 is used to implement Figure 8 In the method embodiment shown, the processing unit 1520 and the fusion unit 1530 are used to implement the functions of optical flow mapping and feature fusion.

[0206] For a more detailed description of the acquisition unit 1510, processing unit 1520, fusion unit 1530 and encoding unit 1540 mentioned above, please refer to [the relevant documentation]. Figure 5 and Figure 8 The relevant descriptions in the method embodiments shown are directly obtained and will not be repeated here.

[0207] Accordingly, embodiments of this application also provide an image decoding device, such as... Figure 16 As shown, Figure 16 This is a schematic diagram of the structure of the image decoding device provided in this application. The image decoding device can be used to implement the functions of the decoding end or video decoder in the above method embodiments, and therefore can also achieve the beneficial effects of the above image decoding method embodiments. In the embodiments of this application, the image decoding device can be as follows: Figure 1 The decoding terminal 20 or video decoder 200 shown can also be a module (such as a chip) applied to the decoding terminal 20 or video decoder 200.

[0208] like Figure 16 As shown, the image decoding device 1600 includes: a bitstream unit 1610, a processing unit 1620, a fusion unit 1630, and a decoding unit 1640. The image decoding device 1600 is used to implement the above-mentioned... Figure 8 and Figure 9 The method embodiment shown illustrates the function of the decoding end or video decoder.

[0209] When the image decoding device 1600 is used to implement Figure 8 In the method embodiment shown, the processing unit 1620 and the fusion unit 1630 are used to implement the functions of optical flow mapping and feature fusion.

[0210] When the image decoding apparatus 1600 is used to implement the method embodiments shown in Figure 9 When the image decoding apparatus 1600 is used to implement the method embodiments shown in

[0211] In some optional cases, the image decoding apparatus 1600 can further include an enhancement unit, configured to process the first image according to the enhancement feature map to obtain a second image. In a possible specific example, the enhancement unit is specifically configured to: obtain an enhancement layer image of the first image according to the enhancement feature map; and reconstruct the first image based on the enhancement layer image to obtain the second image.

[0212] For more detailed description of the above-mentioned code stream unit 1610, processing unit 1620, fusion unit 1630, decoding unit 1640 and enhancement unit, please refer to the relevant description in the method embodiments shown in Figure 8 and Figure 9 For more detailed description of the above-mentioned code stream unit 1610, processing unit 1620, fusion unit 1630, decoding unit 1640 and enhancement unit, please refer to the relevant description in the method embodiments shown in

[0213] When the image encoding (or image decoding) apparatus implements the image encoding (or image decoding) method shown in any of the foregoing figures by software, the image encoding (or image decoding) apparatus and each unit thereof can also be a software module. The image encoding (or image decoding) method described above is implemented by calling the software module by a processor. The processor can be a central processing unit (CPU), an application-specific integrated circuit (ASIC) implementation, or a programmable logic device (PLD), which can be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0214] For more detailed description of the above-mentioned image encoding (or image decoding) apparatus, please refer to the relevant description in the foregoing embodiments shown in the figures, which will not be repeated here. It can be understood that the image encoding (or image decoding) apparatus shown in the foregoing figures is only an example provided by the present embodiment, and the image encoding (or image decoding) apparatus can include more or fewer units according to the image encoding (or image decoding) process or service, which is not limited in the present application.

[0215] When an image encoding (or image decoding) device is implemented in hardware, this hardware can be implemented using a processor or a chip. The chip includes interface circuitry and control circuitry. The interface circuitry is used to receive data from other devices outside the processor and transmit it to the control circuitry, or to send data from the control circuitry to other devices outside the processor.

[0216] The control circuit uses logic circuits or executed code instructions to implement any of the possible implementation methods in the above embodiments. The beneficial effects can be found in the descriptions of any aspect of the above embodiments, and will not be repeated here.

[0217] It is understood that the processor in the embodiments of this application may be a CPU, a neural processing unit (NPU), or a graphics processing unit (GPU), or it may be other general-purpose processors, digital signal processors (DSPs), ASICs, FPGAs, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0218] The method steps in the embodiments of this application can be implemented in hardware, such as an image decoding device. Figure 17 As shown, Figure 17 This is a schematic diagram of the structure of the image decoding apparatus 1700 provided in this application. The image decoding apparatus 1700 includes a memory 1710 and at least one processor 1720. The processor 1720 can implement the image encoding method and image decoding method provided in the above embodiments. The memory 1710 is used to store the software instructions corresponding to the above image encoding method and image decoding method. As an optional implementation, in hardware implementation, the image decoding apparatus 1700 can refer to a chip or chip system that encapsulates one or more processors 1720. For example, when the image decoding apparatus 1700 is used to implement the method steps in the above embodiments, the processor 1720 included in the image decoding apparatus 1700 executes the steps of the above method and its possible sub-steps. In an optional case, the image decoding apparatus 1700 may also include a communication interface 1730, which can be used to send and receive data. For example, the communication interface 1730 is used to receive user encoding requests, decoding requests, or to send and receive code streams, etc.

[0219] In the embodiments of this application, the communication interface 1730, the processor 1720 and the memory 1710 can be connected via a bus 1740, which can be divided into an address bus, a data bus, a control bus, etc.

[0220] It is worth noting that the image coding apparatus 1700 can also perform the functions of the image encoding apparatus 1500 shown in FIG. 1, and the functions of the image decoding apparatus 1600 shown in FIG. 2, which are not described herein. Figure 15 Figure 16

[0221] The image coding apparatus 1700 provided by the embodiments can be a server, a personal computer, or other image coding apparatus 1700 having a data processing function, which is not limited in the present application. For example, the image coding apparatus 1700 can be the encoding end 10 (or the video encoder 100) or the decoding end 20 (or the video decoder 200) described above. For another example, the image coding apparatus 1700 can also have the functions of the encoding end 10 and the decoding end 20 at the same time, such as the image coding apparatus 1700 being a video coding system (or a video compression system) having a video coding and decoding function.

[0222] The method steps in the embodiments of the present application can also be implemented by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and the storage medium can also exist as discrete components in the network device or the terminal device.

[0223] In addition, the present application also provides a computer readable storage medium, which stores a code stream obtained by the image encoding method according to any one of the preceding embodiments. The computer readable storage medium can be, but is not limited to, a RAM, a flash memory, a ROM, a PROM, an EPROM, an EEPROM, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art.

[0224] ​​In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments are performed. The computer can be a general purpose computer, a special purpose computer, a computer network, a network device, a user equipment or other programmable apparatus. The computer programs or instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer programs or instructions can be transferred from one website site, computer, server or data center to another website site, computer, server or data center through wired or wireless manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center and the like integrated with one or more available media. The available media can be a magnetic medium, such as a floppy disk, a hard disk, a magnetic tape; or an optical medium, such as a digital video disc (digital video disc, DVD); or a semiconductor medium, such as a solid state drive (solid state drive, SSD).

[0225] In various embodiments of the present application, the terms and / or descriptions of different embodiments are consistent and can be referred to each other if there is no special description and logical conflict, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0226] In the present application, "at least one" means one or more, and "multiple" means two or more. The "and / or" describes the association relationship between the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In the textual description of the present application, the character " / ", generally represents that the front and rear associated objects have an "or" relationship; in the formula of the present application, the character " / ", represents that the front and rear associated objects have a "division" relationship.

[0227] It can be understood that the various numbers involved in the embodiments of the present application are only distinguished for convenience of description, and are not used to limit the scope of the embodiments of the present application. The size of the serial number of the above processes does not mean the execution order, and the execution order of the processes should be determined according to its function and inherent logic.

Claims

1. An image decoding method characterized by, The method comprises: parsing a code stream to obtain at least one optical flow set; wherein the at least one optical flow set comprises a first optical flow set, the first optical flow set comprises one or more feature domain optical flows, the first optical flow set corresponds to a first feature map of a reference frame of a first image frame, and any one of the one or more feature domain optical flows is used to indicate motion information between a feature map of the first image frame and the first feature map; processing the first feature map based on the one or more feature domain optical flows to obtain one or more intermediate feature maps corresponding to the first feature map; fusing the one or more intermediate feature maps to obtain a first prediction feature map of the first image frame; decoding the first image frame according to the first prediction feature map to obtain a first image.

2. The method of claim 1, wherein, The processing of the first feature map based on the one or more feature domain optical flows to obtain one or more intermediate feature maps corresponding to the first feature map comprises: parsing the code stream to obtain the first feature map; performing optical flow mapping wrapping on the first feature map based on a first feature domain optical flow to obtain an intermediate feature map corresponding to the first feature domain optical flow, wherein the first feature domain optical flow is any one of the one or more feature domain optical flows.

3. The method according to claim 1 or 2, characterized in that, The first optical flow set also corresponds to a second feature map of the reference frame; Before the decoding of the first image frame according to the first prediction feature map to obtain a first image, the method further comprises: processing the second feature map based on the one or more feature domain optical flows to obtain one or more intermediate feature maps corresponding to the second feature map; fusing the one or more intermediate feature maps corresponding to the second feature map to obtain a second prediction feature map of the first image frame; The decoding of the first image frame according to the first prediction feature map to obtain a first image comprises: decoding the first image frame according to the first prediction feature map and the second prediction feature map to obtain a first image.

4. The method according to any one of claims 1 to 3, characterized in that, The fusing of the one or more intermediate feature maps to obtain a first prediction feature map of the first image frame comprises: obtaining one or more weights of the one or more intermediate feature maps, wherein one intermediate feature map corresponds to one weight, and the weight is used to indicate a weight of the intermediate feature map in the first prediction feature map; processing the intermediate feature maps corresponding to the one or more weights based on the one or more weights, and adding all the processed intermediate feature maps to obtain the first prediction feature map.

5. The method according to any one of claims 1 to 3, characterized in that, The fusing of the one or more intermediate feature maps to obtain a first prediction feature map of the first image frame comprises: inputting the one or more intermediate feature maps corresponding to the first feature map into a feature fusion model to obtain a first prediction feature map; and the feature fusion model comprises a convolution network layer used for fusing intermediate feature maps.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: obtaining a feature map of the first image; obtaining an enhanced feature map according to the feature map of the first image, the first feature map, and the first prediction feature map; According to the enhanced feature map, the first image is processed to obtain a second image, and the second image has a higher definition than the first image.

7. The method of claim 6, wherein, The processing of the first image according to the enhanced feature map to obtain a second image comprises: An enhanced layer image of the first image is obtained according to the enhanced feature map; The first image is reconstructed based on the enhanced layer image to obtain the second image.

8. An image coding method characterized by, The method comprises: obtaining a feature map of a first image frame and a first feature map of a reference frame of the first image frame; at least one optical flow set is obtained according to the feature map of the first image frame and the first feature map, the at least one optical flow set comprising a first optical flow set, the first optical flow set comprising one or more feature domain optical flows, the first optical flow set corresponding to the first feature map, and any one of the one or more feature domain optical flows being used to indicate motion information between the feature map of the first image frame and the first feature map; the first feature map is processed based on the one or more feature domain optical flows to obtain one or more intermediate feature maps corresponding to the first feature map; the one or more intermediate feature maps are fused to obtain a first predicted feature map of the first image frame; the first image frame is encoded according to the first predicted feature map to obtain a code stream.

9. The method of claim 8, wherein, The processing of the first feature map based on the one or more feature domain optical flows to obtain one or more intermediate feature maps corresponding to the first feature map comprises: the first feature map is wrapped based on a first feature domain optical flow to obtain an intermediate feature map corresponding to the first feature domain optical flow, wherein the first feature domain optical flow is any one of the one or more feature domain optical flows.

10. The method according to claim 8 or 9, characterized in that, The first optical flow set also corresponds to a second feature map of the reference frame. Before the first image frame is encoded according to the first predicted feature map to obtain a code stream, the method further comprises: the second feature map is processed based on the one or more feature domain optical flows to obtain one or more intermediate feature maps corresponding to the second feature map; the one or more intermediate feature maps corresponding to the second feature map are fused to obtain a second predicted feature map of the first image frame; The encoding of the first image frame according to the first predicted feature map to obtain a code stream comprises: the first image frame is encoded according to the first predicted feature map and the second predicted feature map to obtain a code stream.

11. The method according to any one of claims 8 to 10, characterized in that, The fusion of the one or more intermediate feature maps to obtain a first predicted feature map of the first image frame comprises: one or more weights of the one or more intermediate feature maps are obtained, wherein one intermediate feature map corresponds to one weight, and the weight is used to indicate a weight of the intermediate feature map in the first predicted feature map; the intermediate feature maps corresponding to the one or more weights are processed based on the one or more weights, and all the processed intermediate feature maps are added to obtain the first predicted feature map.

12. The method according to any one of claims 8 to 11, characterized in that, The fusion of the one or more intermediate feature maps to obtain a first predicted feature map of the first image frame comprises: The one or more intermediate feature maps corresponding to the first feature map are input into a feature fusion model to obtain a first predicted feature map; the feature fusion model comprises a convolution network layer, and the convolution network layer is configured to fuse the intermediate feature maps.

13. An image decoding apparatus characterized by comprising: The device comprises: A code stream unit is configured to parse a code stream to obtain at least one optical flow set; the at least one optical flow set comprises a first optical flow set, and the first optical flow set comprises one or more feature domain optical flows; the first optical flow set corresponds to a first feature map of a reference frame of a first image frame; and any one of the one or more feature domain optical flows is configured to indicate motion information between the first feature map and a feature map of the first image frame. A processing unit is configured to process the first feature map based on the one or more feature domain optical flows to obtain one or more intermediate feature maps corresponding to the first feature map. A fusion unit is configured to fuse the one or more intermediate feature maps to obtain a first predicted feature map of the first image frame. A decoding unit is configured to decode the first image frame based on the first predicted feature map to obtain a first image.

14. The apparatus of claim 13, wherein, The processing unit is specifically configured to parse the code stream to obtain the first feature map, and perform optical flow wrapping on the first feature map based on a first feature domain optical flow to obtain an intermediate feature map corresponding to the first feature domain optical flow; the first feature domain optical flow is any one of the one or more feature domain optical flows.

15. The apparatus of claim 13 or 14, wherein, The first optical flow set also corresponds to a second feature map of the reference frame. The processing unit is further configured to process the second feature map based on the one or more feature domain optical flows to obtain one or more intermediate feature maps corresponding to the second feature map. The fusion unit is further configured to fuse the one or more intermediate feature maps corresponding to the second feature map to obtain a second predicted feature map of the first image frame. The decoding unit is specifically configured to decode the first image frame based on the first predicted feature map and the second predicted feature map to obtain a first image.

16. The apparatus of any one of claims 13-15, wherein, The fusion unit is specifically configured to obtain one or more weights of the one or more intermediate feature maps, wherein one intermediate feature map corresponds to one weight, and the weight is configured to indicate a weight of the intermediate feature map in the first predicted feature map; and process the intermediate feature maps corresponding to the one or more weights based on the one or more weights, and add all the processed intermediate feature maps to obtain the first predicted feature map.

17. The apparatus of any one of claims 13-15, wherein, The fusion unit is specifically configured to input the one or more intermediate feature maps corresponding to the first feature map into a feature fusion model to obtain a first predicted feature map; the feature fusion model comprises a convolution network layer, and the convolution network layer is configured to fuse the intermediate feature maps.

18. The apparatus of any one of claims 13-17, wherein, The device further comprises an acquisition unit and an enhancement unit. The acquisition unit is configured to acquire a feature map of the first image. The fusion unit is further configured to obtain an enhanced feature map based on the feature map of the first image, the first feature map, and the first predicted feature map. The enhancement unit is configured to process the first image according to the enhancement feature map to obtain a second image, and the second image has a higher definition than the first image.

19. The apparatus of claim 18, wherein, The enhancement unit is specifically configured to obtain an enhancement layer image of the first image according to the enhancement feature map, and reconstruct the first image based on the enhancement layer image to obtain the second image.

20. An image coding apparatus characterized by comprising: The device comprises: The acquisition unit is configured to acquire a feature map of a first image frame and a first feature map of a reference frame of the first image frame; The processing unit is configured to obtain at least one optical flow set according to the feature map of the first image frame and the first feature map, wherein the at least one optical flow set comprises a first optical flow set, the first optical flow set comprises one or more feature domain optical flows, the first optical flow set corresponds to the first feature map, and any one of the one or more feature domain optical flows is used to indicate motion information between the feature map of the first image frame and the first feature map. The processing unit is further configured to process the first feature map based on the one or more feature domain optical flows to obtain one or more intermediate feature maps corresponding to the first feature map. The fusion unit is configured to fuse the one or more intermediate feature maps to obtain a first prediction feature map of the first image frame. The encoding unit is configured to encode the first image frame according to the first prediction feature map to obtain a bitstream.

21. The apparatus of claim 20, wherein, The processing unit is specifically configured to perform optical flow mapping wrapping on the first feature map based on a first feature domain optical flow to obtain an intermediate feature map corresponding to the first feature domain optical flow, wherein the first feature domain optical flow is any one of the one or more feature domain optical flows.

22. The apparatus of claim 20 or 21, wherein, The first optical flow set also corresponds to a second feature map of the reference frame. The processing unit is further configured to process the second feature map based on the one or more feature domain optical flows to obtain one or more intermediate feature maps corresponding to the second feature map. The fusion unit is further configured to fuse the one or more intermediate feature maps corresponding to the second feature map to obtain a second prediction feature map of the first image frame. The encoding unit is specifically configured to encode the first image frame according to the first prediction feature map and the second prediction feature map to obtain a bitstream.

23. The apparatus of any one of claims 20-22, wherein, The fusion unit is specifically configured to obtain one or more weights of the one or more intermediate feature maps, wherein one intermediate feature map corresponds to one weight, the weight is used to indicate a weight of the intermediate feature map in the first prediction feature map, process the intermediate feature map corresponding to each of the one or more weights based on the one or more weights, and add all the processed intermediate feature maps to obtain the first prediction feature map.

24. The apparatus of any one of claims 20-22, wherein, The fusion unit is specifically configured to input the one or more intermediate feature maps corresponding to the first feature map into a feature fusion model to obtain a first prediction feature map, and the feature fusion model comprises a convolution network layer configured to fuse intermediate feature maps.

25. An apparatus for image decoding, the apparatus comprising: The device comprises: The memory and the processor; The memory is configured to store program code, and the processor is configured to invoke the program code to implement the image decoding method in any one of claims 1 to 7, or implement the image encoding method in any one of claims 8 to 12.

26. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, and when the computer program or instructions are executed by the electronic device, the image decoding method in any one of claims 1 to 7 is implemented, or the image encoding method in any one of claims 8 to 12 is implemented.

27. A computer-readable storage medium, characterized in that, The storage medium stores a code stream obtained according to the image encoding method in any one of claims 8 to 12.

Images