Inter-frame prediction method, device, encoding / decoding method, codec, and electronic device

Abstract

The present invention discloses an inter-frame prediction method, apparatus, encoding and decoding method, codec, and electronic device. After obtaining a current frame image and a corresponding reference frame image, two methods are used to determine the motion information of the current frame image. In the pixel domain, an optical flow estimation algorithm is used to determine the first motion information of the current frame image; in the feature space domain, the current frame feature map and the reference frame feature map are input into a trained motion information detection model, and based on the motion information detection model, the second motion information is determined. Then, the first motion information and the second motion information are fused to obtain the target motion information. Embodiments of the present invention start from two dimensions, namely the pixel domain and the feature space domain, to obtain the final target motion information, making the determined target motion information more accurate, and then performing motion compensation based on the target motion information, making the compensated image more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of video coding and decoding, and particularly to an inter-frame prediction method, apparatus, coding and decoding method, codec, and electronic device. Background Art

[0002] A video coding and decoding system mainly includes three parts: encoding, transmission, and decoding. Since the amount of data in video images is relatively large, the main function of video encoding is to compress video pixel data (such as RGB color space, luminance signal, chrominance signal YUV in color space, etc.) into a video bitstream, thereby reducing the amount of video data and achieving the purpose of reducing network bandwidth during transmission and reducing storage space.

[0003] Inter-frame prediction in video coding and decoding technology generally means that the luminance and chrominance signal values of pixel points in adjacent frames in time are relatively close and have strong correlation. Inter-frame prediction searches for the most similar matching block to the current block in the reference frame through methods such as motion search, and estimates motion information between the current block and the matching block, such as motion vector MV (motion vector) and reference frame index. Encoding the motion information and transmitting it to the decoding end. At the decoding end, as long as the decoder parses the motion information of the current block through corresponding syntax elements, it can find the matching block of the current block. And copying the pixel values of the matching block to the current block is the inter-frame prediction value of the current block.

[0004] When estimating the motion information between the reference frame and the current frame in the prior art, it is generally only carried out in the pixel domain. Only estimating the motion information in the pixel domain will result in relatively low accuracy of the estimated motion information, and thus relatively low accuracy of inter-frame prediction. Summary of the Invention

[0005] Embodiments of the present invention provide an inter-frame prediction method, apparatus, coding and decoding method, codec, and electronic device, to solve the problem in the prior art that the accuracy of inter-frame prediction is relatively low due to relatively low accuracy of the estimated motion information.

[0006] Embodiments of the present invention provide an inter-frame prediction method, the method including:

[0007] Obtain a current frame image and a corresponding reference frame image;

[0008] Determine first motion information according to the current frame image, the reference frame image, and an optical flow estimation algorithm;

[0009] Extract features from the current frame image and the reference frame image respectively to obtain respective corresponding current frame feature maps and reference frame feature maps; input the current frame feature maps and the reference frame feature maps into a trained motion information detection model, and determine second motion information based on the motion information detection model;

[0010] Fuse the first motion information and the second motion information to obtain target motion information, and perform motion compensation based on the target motion information to obtain a compensated image.

[0011] Further, obtaining the corresponding reference frame image includes:

[0012] Select at least one reference frame image from the forward reference list of the current frame image; or select at least one reference frame image from the backward reference list of the current frame image; or select at least one reference frame image from the forward and backward reference lists of the current frame image respectively.

[0013] Further, the respectively extracting features from the current frame image and the reference frame image to obtain the corresponding current frame feature map and reference frame feature map includes:

[0014] Input the current frame image and the reference frame image into a feature extraction network model respectively, and perform feature extraction on the current frame image and the reference frame image based on the feature extraction network model to obtain the corresponding current frame feature map and reference frame feature map respectively; wherein, the feature extraction network model includes a first convolutional neural network structure and at least one first residual block connected in sequence.

[0015] Further, the inputting the current frame feature map and the reference frame feature map into a trained motion information detection model and determining the second motion information based on the motion information detection model includes:

[0016] The trained motion information detection model includes a trained first neural network model, and the first neural network model includes a second convolutional neural network structure;

[0017] Input the current frame feature map and the reference frame feature map into the trained first neural network model, and determine the second motion information based on the first neural network model.

[0018] Further, the respectively extracting features from the current frame image and the reference frame image to obtain the corresponding current frame feature map and reference frame feature map includes:

[0019] Input the current frame image and the reference frame image into a pyramid network model respectively, and perform feature extraction on the current frame image and the reference frame image based on the pyramid network model to obtain the current frame feature map and the reference frame feature map in each layer of the pyramid; wherein, the pyramid network model includes at least two feature extraction network structures connected in sequence.

[0020] Further, inputting the current frame feature map and the reference frame feature map into the trained motion information detection model, and determining the second motion information based on the motion information detection model includes:

[0021] The trained motion information detection model includes a trained second neural network model, and the second neural network model includes a warping layer, a cost calculation layer, an optical flow prediction layer, and an optical flow optimization layer connected in sequence;

[0022] Based on the warping layer, the cost calculation layer, the optical flow prediction layer, and the optical flow optimization layer in the pyramid network model, respectively determine the initial motion information corresponding to the current frame feature map of each layer in the pyramid;

[0023] The initial motion information of the low-resolution layer is upsampled layer by layer to obtain the second motion information; or the initial motion information of each layer is fused to obtain the second motion information.

[0024] Further, the optical flow prediction layer includes a second residual block, or a depthwise separable convolution structure, or a spatial separable convolution structure;

[0025] The optical flow optimization layer includes a dilated convolution structure and a filtering layer.

[0026] Further, the motion compensation based on the target motion information to obtain the compensated image includes:

[0027] Input the target motion information and the reference frame feature map into the predicted frame feature map module in the motion information compensation network to obtain the predicted frame feature map; input the predicted frame feature map into the reconstructed image module in the motion information compensation network to obtain the compensated image.

[0028] Further, the predicted frame feature map module includes a deformable convolution layer, a concatenation layer, and a convolution layer connected in sequence;

[0029] The inputting the target motion information and the reference frame feature map into the predicted frame feature map module in the motion information compensation network to obtain the predicted frame feature map includes:

[0030] Input the target motion information and the reference frame feature map into the deformable convolution layer;

[0031] Input the output result of the deformable convolution layer and the reference frame feature map into the concatenation layer;

[0032] Input the output result of the concatenation layer into the convolution layer;

[0033] Fuse the output result of the convolution layer and the output result of the deformable convolution layer to obtain the predicted frame feature map.

[0034] Further, the reconstructed image module includes a third residual block and a transposed convolution layer connected in sequence;

[0035] The obtaining of the compensated image by inputting the predicted frame feature map into the reconstructed image module in the motion information compensation network includes:

[0036] Inputting the predicted frame feature map into the third residual block;

[0037] Inputting the output result of the third residual block into the transposed convolution layer to obtain the compensated image.

[0038] Further, if there are at least two reference frame feature maps; the obtaining of the predicted frame feature map includes:

[0039] Grouping at least two reference frame feature maps, and inputting each group of reference frame feature maps and the corresponding target motion information into their respective deformable convolution layers; inputting the output results of each deformable convolution layer into the temporal attention module;

[0040] Inputting the output result of each group of temporal attention modules into the concatenation layer;

[0041] Inputting the output result of the concatenation layer into the spatial attention module, and obtaining the predicted frame feature map based on the spatial attention module.

[0042] Further, the processing process of the temporal attention module for the output results of each deformable convolution layer includes;

[0043] Respectively inputting the output results of each deformable convolution layer into the corresponding convolution layer in the temporal attention module;

[0044] Performing dot product and activation function operations on the output result of each convolution layer; multiplying the result of the activation function operation by the output result of any one deformable convolution layer to obtain the output result of the temporal attention module.

[0045] Further, the processing process of the spatial attention module for the output result of the concatenation layer includes;

[0046] Inputting the output result of the concatenation layer into the convolution layer in the spatial attention module;

[0047] Inputting the output result of the convolution layer into a multi-scale network structure; the multi-scale network structure includes fourth residual blocks of different scales;

[0048] Performing activation function operation on the output result of the multi-scale network structure; multiplying the result of the activation function operation by the output result of the convolution layer to obtain the predicted frame feature map.

[0049] Further, the method further includes:

[0050] Sending the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image to a decoding end, so that the decoding end determines a predicted frame image according to the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image; the reference direction information of the reference frame image includes forward reference information and / or backward reference information.

[0051] Further, the sending the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image to the decoding end includes:

[0052] If there are multiple reference frame images, performing fusion or preselection processing on the target motion information corresponding to each reference frame image to obtain one or two target motion information after fusion or preselection processing; sending the target motion information after fusion or preselection processing, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end determines a predicted frame image according to the target motion information after fusion or preselection processing, the reference direction information of each reference frame image, and the index number of each reference frame image; or

[0053] If there are multiple reference frame images, sending the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end respectively determines candidate predicted frame images according to the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image, and performing fusion processing on each candidate predicted frame image to obtain a predicted frame image; or

[0054] If there are multiple reference frame images, sending the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end performs alignment and fusion processing according to each reference frame image and the corresponding target motion information thereof during a motion compensation stage to determine a predicted frame image.

[0055] On the other hand, an embodiment of the present invention provides an inter-frame prediction device, and the device includes:

[0056] A first obtaining unit, configured to obtain a current frame image and a corresponding reference frame image;

[0057] A first determining unit, configured to determine first motion information according to the current frame image, the reference frame image, and an optical flow estimation algorithm;

[0058] A second determination unit, configured to perform feature extraction on the current frame image and the reference frame image respectively to obtain corresponding current frame feature maps and reference frame feature maps; input the current frame feature maps and the reference frame feature maps into a trained motion information detection model, and determine second motion information based on the motion information detection model.

[0059] A prediction unit, configured to perform fusion processing on the first motion information and the second motion information to obtain target motion information, and perform motion compensation based on the target motion information to obtain a compensated image.

[0060] The first acquisition unit is specifically configured to select at least one reference frame image from the forward reference list of the current frame image; or select at least one reference frame image from the backward reference list of the current frame image; or select at least one reference frame image from the forward and backward reference lists of the current frame image respectively.

[0061] The second determination unit is specifically configured to input the current frame image and the reference frame image into a feature extraction network model respectively, and perform feature extraction on the current frame image and the reference frame image respectively based on the feature extraction network model to obtain corresponding current frame feature maps and reference frame feature maps; wherein, the feature extraction network model includes a first convolutional neural network structure and at least one first residual block connected in sequence.

[0062] The second determination unit is specifically configured to the trained motion information detection model includes a trained first neural network model, and the first neural network model includes a second convolutional neural network structure; input the current frame feature maps and the reference frame feature maps into the trained first neural network model, and determine second motion information based on the first neural network model.

[0063] The second determination unit is specifically configured to input the current frame image and the reference frame image into a pyramid network model respectively, and perform feature extraction on the current frame image and the reference frame image respectively based on the pyramid network model to obtain current frame feature maps and reference frame feature maps in each layer of the pyramid; wherein, the pyramid network model includes at least two feature extraction network structures connected in sequence.

[0064] The second determination unit is specifically configured that the trained motion information detection model includes a trained second neural network model, and the second neural network model includes a warping layer, a cost calculation layer, an optical flow prediction layer, and an optical flow optimization layer that are connected in sequence; based on the warping layer, the cost calculation layer, the optical flow prediction layer, and the optical flow optimization layer in the pyramid network model, respectively determine the initial motion information corresponding to the current frame feature map of each layer in the pyramid; the initial motion information of the low-resolution layer is upsampled layer by layer to obtain the second motion information; or fuse the initial motion information of each layer to obtain the second motion information.

[0065] The optical flow prediction layer includes a second residual block, or a depthwise separable convolution structure, or a spatial separable convolution structure; the optical flow optimization layer includes a dilated convolution structure and a filtering layer.

[0066] The prediction unit is specifically configured to input the target motion information and the reference frame feature map into the predicted frame feature map module in the motion information compensation network to obtain a predicted frame feature map; input the predicted frame feature map into the reconstructed image module in the motion information compensation network to obtain a compensated image.

[0067] The predicted frame feature map module includes a deformable convolution layer, a concatenation layer, and a convolution layer that are connected in sequence;

[0068] The prediction unit is specifically configured to input the target motion information and the reference frame feature map into the deformable convolution layer; input the output result of the deformable convolution layer and the reference frame feature map into the concatenation layer; input the output result of the concatenation layer into the convolution layer; fuse the output result of the convolution layer and the output result of the deformable convolution layer to obtain a predicted frame feature map.

[0069] The reconstructed image module includes a third residual block and a transposed convolution layer that are connected in sequence;

[0070] The prediction unit is specifically configured to input the predicted frame feature map into the third residual block; input the output result of the third residual block into the transposed convolution layer to obtain a compensated image.

[0071] If there are at least two reference frame feature maps, the prediction unit is specifically configured to group the at least two reference frame feature maps, input the reference frame feature maps of each group and the corresponding target motion information into their respective deformable convolution layers; input the output results of each deformable convolution layer into the temporal attention module; input the output results of each temporal attention module into the concatenation layer; input the output result of the concatenation layer into the spatial attention module, and obtain a predicted frame feature map based on the spatial attention module.

[0072] The apparatus further includes:

[0073] A sending unit, configured to send the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image to a decoding end, so that the decoding end determines a predicted frame image according to the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image; the reference direction information of the reference frame image includes forward reference information and / or backward reference information.

[0074] Specifically, if there are multiple reference frame images, the sending unit performs fusion or preselection processing on the target motion information corresponding to each reference frame image to obtain one or two pieces of target motion information after fusion or preselection processing; and sends the target motion information after fusion or preselection processing, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end determines a predicted frame image according to the target motion information after fusion or preselection processing, the reference direction information of each reference frame image, and the index number of each reference frame image; or

[0075] If there are multiple reference frame images, the sending unit sends the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end respectively determines candidate predicted frame images according to the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image, and performs fusion processing on each candidate predicted frame image to obtain a predicted frame image; or

[0076] If there are multiple reference frame images, the sending unit sends the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end performs alignment and fusion processing according to each reference frame image and the corresponding target motion information thereof during a motion compensation stage to determine a predicted frame image.

[0077] On the other hand, an embodiment of the present invention provides an encoding method, and the method includes:

[0078] Obtain a current frame image; [[ID=##]]

[0079] Encode the current frame image to obtain an encoded image corresponding to the current frame image; wherein, during the encoding process, the above-mentioned inter-frame prediction method is used to perform inter-frame prediction on the current frame image.

[0080] On the other hand, an embodiment of the present invention provides an encoder, and the encoder includes:

[0081] A second obtaining unit, configured to obtain a current frame image;

[0082] An encoding unit for encoding the current frame image to obtain an encoded image corresponding to the current frame image.

[0083] In another aspect, an embodiment of the present invention provides a decoding method, the method comprising:

[0084] Receiving target motion information, reference direction information of a reference frame image, and an index number of the reference frame image sent by an encoder, wherein the target motion information is obtained by the encoder using the inter-frame prediction method described in any one of the above;

[0085] Determining a predicted frame image according to the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image.

[0086] In another aspect, an embodiment of the present invention provides a decoder, the decoder comprising:

[0087] A receiving unit for receiving the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image sent by the encoder;

[0088] A decoding unit for determining a predicted frame image according to the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image.

[0089] In another aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0090] The memory is used for storing a computer program;

[0091] The processor is used for implementing the method steps described in any one of the above when executing the program stored in the memory.

[0092] In another aspect, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored, and the computer program realizes the method steps described in any one of the above when being executed by a processor.

[0093] Embodiments of the present invention provide an inter-frame prediction method, apparatus, encoding and decoding method, codec, and electronic device. The method includes: obtaining a current frame image and a corresponding reference frame image; determining first motion information according to the current frame image, the reference frame image, and an optical flow estimation algorithm; respectively performing feature extraction on the current frame image and the reference frame image to obtain a corresponding current frame feature map and reference frame feature map; inputting the current frame feature map and the reference frame feature map into a trained motion information detection model, and determining second motion information based on the motion information detection model; performing fusion processing on the first motion information and the second motion information to obtain target motion information, and performing motion compensation based on the target motion information to obtain a compensated image.

[0094] The above technical solution has the following advantages or beneficial effects:

[0095] Since in the embodiments of the present invention, after obtaining the current frame image and the corresponding reference frame image, two methods are used to respectively determine the motion information of the current frame image and the corresponding reference frame image. That is, the first method uses an optical flow estimation algorithm in the pixel domain to determine the first motion information of the current frame image and the corresponding reference frame image; the second method is in the feature space domain, respectively performing feature extraction on the current frame image and the reference frame image to obtain a corresponding current frame feature map and reference frame feature map; inputting the current frame feature map and the reference frame feature map into a trained motion information detection model, and determining second motion information based on the motion information detection model. Then, fusion processing is performed on the first motion information and the second motion information to obtain target motion information, and finally, motion compensation is performed according to the target motion information to obtain a compensated image. The embodiments of the present invention start from two dimensions, the pixel domain and the feature space domain, to obtain the final target motion information, making the determined target motion information more accurate, and then performing motion compensation based on the target motion information, making the compensated image more accurate. Description of the Drawings

[0096] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0097] Figure 1 Schematic diagram of the inter-frame prediction process provided in Embodiment 1 of the present invention;

[0098] Figure 2 Schematic diagram of obtaining a reference frame image corresponding to a current frame image provided in Embodiment 1 of the present invention;

[0099] Figure 3Schematic diagram of the feature extraction network model structure provided in Embodiment 2 of the present invention;

[0100] Figure 4 Schematic diagram of the structure of the first neural network model provided in Embodiment 2 of the present invention;

[0101] Figure 5 Schematic diagram of the pyramid network model structure provided in Embodiment 3 of the present invention;

[0102] Figure 6 Schematic diagram of another pyramid network model structure provided in Embodiment 3 of the present invention;

[0103] Figure 7 Basic principle diagram of deformable convolution provided in Embodiment 4 of the present invention;

[0104] Figure 8 Schematic diagram of the structure of the predicted frame feature map module provided in Embodiment 4 of the present invention;

[0105] Figure 9 Schematic diagram of the structure of the reconstructed image module provided in Embodiment 4 of the present invention;

[0106] Figure 10 Framework diagram for obtaining the predicted frame feature map provided in Embodiment 5 of the present invention;

[0107] Figure 11 Schematic diagram of the processing process of the output results of each deformable convolution layer by the temporal attention module provided in Embodiment 5 of the present invention;

[0108] Figure 12 Schematic diagram of the processing process of the output results of the concatenation layer by the spatial attention module provided in Embodiment 5 of the present invention;

[0109] Figure 13 Schematic diagram of the multi-scale network structure provided in Embodiment 5 of the present invention;

[0110] Figure 14 Schematic diagram of the structure of the inter-frame prediction device provided in Embodiment 7 of the present invention;

[0111] Figure 15 Schematic diagram of the encoding process provided in Embodiment 8 of the present invention;

[0112] Figure 16 Schematic diagram of the structure of the encoder provided in Embodiment 8 of the present invention;

[0113] Figure 17 Schematic diagram of the decoding process provided in Embodiment 9 of the present invention;

[0114] Figure 18 Schematic diagram of the structure of the decoder provided in Embodiment 9 of the present invention;

[0115] Figure 19 Schematic diagram of the electronic device structure provided in Embodiment 10 of the present invention. Detailed implementation manners

[0116] The present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0117] Embodiment 1:

[0118] Figure 1 Schematic diagram of the inter-frame prediction process provided in the embodiment of the present invention. This process includes the following steps:

[0119] S101: Obtain the current frame image and the corresponding reference frame image.

[0120] S102: Determine the first motion information according to the current frame image, the reference frame image, and the optical flow estimation algorithm.

[0121] S103: Respectively perform feature extraction on the current frame image and the reference frame image to obtain the corresponding current frame feature map and reference frame feature map; input the current frame feature map and the reference frame feature map into the trained motion information detection model, and based on the motion information detection model, determine the second motion information.

[0122] S104: Perform fusion processing on the first motion information and the second motion information to obtain the target motion information, and perform motion compensation based on the target motion information to obtain the compensated image.

[0123] The inter-frame prediction method provided in the embodiment of the present invention is applied to an electronic device, and this electronic device is an encoding end device. The encoding end device first obtains the current frame image and the reference frame image corresponding to the current frame image. In the prior art, when obtaining the reference frame image corresponding to the current frame image, generally a certain frame of historical reconstructed frame image is selected as the reference frame image. Only determining the motion information based on one frame of reference frame image corresponding to the current frame image results in poor accuracy of the obtained motion information. In the embodiment of the present invention, to solve the above problem, obtaining the corresponding reference frame image includes: selecting at least one frame of reference frame image from the forward reference list of the current frame image; or selecting at least one frame of reference frame image from the backward reference list of the current frame image; or respectively selecting at least one frame of reference frame image from the forward and backward reference lists of the current frame image.

[0124] Such as Figure 2As shown, when the embodiment of the present invention obtains the reference frame image corresponding to the current frame image, at least one reference frame image can be selected from the N frame images in the forward reference list, or at least one reference frame image can be selected from the N frame images in the backward reference list, or at least one reference frame image can be selected from the L frame images in the forward reference list respectively, and at least one reference frame image can be selected from the M frame images in the backward reference list. The types of reference frame images in the embodiment of the present invention: in addition to historical reconstructed frames, the reference frame images can also be of other types, such as a new reference frame image formed by fusing multiple reconstructed frames, a background frame obtained through background modeling, or designating a certain frame (such as a key frame) as a long-term reference frame; the number and structure of reference frame images: it can be changed from the existing single-frame reference to multi-frame reference, and can also be changed from one-way reference to two-way reference. The reference frame list can be composed of N (N>=1) reference frames, and the reference frame includes one or more of historical reconstructed frames, fused reference frames, background frames, designated frames, etc. The structure of the reference frame image can be Figure 2 One of the examples.

[0125] In the prior art, motion information estimation is generally performed in the pixel domain using an optical flow estimation algorithm. This method may have the problem of inaccurate motion information estimation, which further affects the accuracy of subsequent inter-frame prediction. The embodiment of the present invention starts from two perspectives to improve the accuracy of motion information estimation.

[0126] Perspective 1: Determine the first motion information according to the current frame image, the reference frame image, and the optical flow estimation algorithm. Use a more effective optical flow estimation algorithm to estimate motion information in the pixel domain, such as the Spynet algorithm, the scale-space optical flow estimation algorithm, the Flownet2 algorithm, etc.

[0127] Perspective 2: Perform feature extraction on the current frame image and the reference frame image respectively to obtain the corresponding current frame feature map and reference frame feature map; input the current frame feature map and the reference frame feature map into the trained motion information detection model, and based on the motion information detection model, determine the second motion information. Estimate motion information in the feature space, and a convolutional neural network model or a pyramid network model can be used to determine the second motion information.

[0128] After obtaining the first motion information and the second motion information, perform fusion processing on the first motion information and the second motion information to obtain the target motion information, and perform inter-frame prediction based on the target motion information. When performing fusion processing on the first motion information and the second motion information, the first motion information and the second motion information can be first concatenated in the channel dimension, and then the target motion information can be obtained through convolutional processing. Or different weights can be assigned to the first motion information and the second motion information as needed, and the target motion information can be obtained through weighted fusion of the first motion information and the second motion information.

[0129] Since in the embodiments of the present invention, after obtaining the current frame image and the corresponding reference frame image, two methods are used to respectively determine the motion information of the current frame image and the corresponding reference frame image. That is, the first method uses an optical flow estimation algorithm in the pixel domain to determine the first motion information of the current frame image and the corresponding reference frame image; the second method is in the feature space domain, and feature extraction is respectively performed on the current frame image and the reference frame image to obtain the corresponding current frame feature map and reference frame feature map; the current frame feature map and the reference frame feature map are input into a trained motion information detection model, and based on the motion information detection model, the second motion information is determined. Then, the first motion information and the second motion information are fused to obtain the target motion information, and finally, frame interpolation prediction is performed according to the target motion information. The embodiments of the present invention start from two dimensions, the pixel domain and the feature space domain, to obtain the final target motion information, making the determined target motion information more accurate, and then performing motion compensation based on the target motion information, making the compensated image more accurate.

[0130] Embodiment 2:

[0131] Based on the above embodiments, in the embodiments of the present invention, the feature extraction of the current frame image and the reference frame image respectively to obtain the corresponding current frame feature map and reference frame feature map includes:

[0132] The current frame image and the reference frame image are respectively input into a feature extraction network model, and based on the feature extraction network model, feature extraction is respectively performed on the current frame image and the reference frame image to obtain the corresponding current frame feature map and reference frame feature map; wherein, the feature extraction network model includes a first convolutional neural network structure and at least one first residual block connected in sequence.

[0133] The trained feature extraction network model is stored in the electronic device. As Figure 3 shown, the feature extraction network model includes a first convolutional neural network structure and at least one first residual block connected in sequence. The current frame image and the reference frame image are respectively input into the feature extraction network model, and based on the first convolutional neural network structure and at least one first residual block in the feature extraction network model, feature extraction is respectively performed on the current frame image and the reference frame image to obtain the corresponding current frame feature map and reference frame feature map.

[0134] As Figure 3As shown, the current frame image or the reference frame image is input, and the final current frame feature map or reference frame feature map is obtained through a first convolutional neural network structure and several first residual blocks. The first convolutional neural network structure is composed of several convolutional layers and activation layers. The convolutional layers extract features and downsample, and the activation layers can use activation functions such as GDN and Relu. After the convolutional layers extract features, several residual blocks are used to further extract features, and the residual blocks do not change the size of the feature map.

[0135] In the embodiment of the present invention, the inputting the current frame feature map and the reference frame feature map into the trained motion information detection model, and determining the second motion information based on the motion information detection model includes:

[0136] The trained motion information detection model includes a trained first neural network model, and the first neural network model includes a second convolutional neural network structure;

[0137] Input the current frame feature map and the reference frame feature map into the trained first neural network model, and determine the second motion information based on the first neural network model.

[0138] The trained motion information detection model is stored in the electronic device. In the embodiment of the present invention, the trained motion information detection model includes a trained first neural network model, as Figure 4 shown. The first neural network model includes a second convolutional neural network structure. Input the current frame feature map and the reference frame feature map into the trained first neural network model, and determine the second motion information based on the second convolutional neural network structure of the first neural network model. As Figure 4 shown, input the current frame feature map and the reference frame feature map into the trained first neural network model. First, the current frame feature map and the reference frame feature map are concatenated in the channel dimension through a concatenation layer, and then the concatenated feature map is input into the second convolutional neural network structure to determine the second motion information.

[0139] After feature extraction, the original image is converted to the feature space. At this time, the information of the current frame feature map and the reference frame feature map is concatenated, and then a second convolutional neural network structure is used to learn the motion offset between the two frames, that is, the motion information is obtained, as Figure 4 shown. Similarly, the second convolutional neural network structure here is also composed of several convolutional layers and activation layers, and the convolutional layers still have the two functions of feature extraction and downsampling.

[0140] Embodiment 3:

[0141] Based on the above embodiments, in the embodiment of the present invention, the respectively performing feature extraction on the current frame image and the reference frame image to obtain the corresponding current frame feature map and reference frame feature map includes:

[0142] Input the current frame image and the reference frame image into the pyramid network model respectively, and perform feature extraction on the current frame image and the reference frame image respectively based on the pyramid network model to obtain the current frame feature map and the reference frame feature map in each layer of the pyramid; wherein, the pyramid network model includes at least two feature extraction network structures connected in sequence.

[0143] In the embodiment of the present invention, feature extraction is performed on the current frame image and the reference frame image respectively based on the pyramid network model to obtain the current frame feature map and the reference frame feature map in each layer of the pyramid. As Figure 5 shown, the first layer feature, the second layer feature, the Lth layer feature, and the (L + 1)th layer feature are obtained respectively based on the pyramid network model, and each layer of feature includes a current frame feature map and a reference frame feature map respectively. Among them, the scale of the first layer feature is the largest, and the scale of the (L + 1)th layer feature is the smallest.

[0144] In the embodiment of the present invention, inputting the current frame feature map and the reference frame feature map into the trained motion information detection model, and determining the second motion information based on the motion information detection model includes:

[0145] The trained motion information detection model includes a trained second neural network model, and the second neural network model includes a warping layer, a cost calculation layer, an optical flow prediction layer, and an optical flow optimization layer connected in sequence;

[0146] Based on the warping layer, the cost calculation layer, the optical flow prediction layer, and the optical flow optimization layer in the pyramid network model, determine the initial motion information corresponding to the current frame feature map in each layer of the pyramid respectively;

[0147] The initial motion information of the low-resolution layer is upsampled layer by layer to obtain the second motion information; or the initial motion information of each layer is fused to obtain the second motion information.

[0148] First, determine the initial motion information corresponding to the current frame feature map and the reference frame feature map of the (L + 1)th layer. Specifically, input the reference frame feature map of the (L + 1)th layer into the warping layer; input the output result of the warping layer and the current frame feature map of the (L + 1)th layer into the cost calculation layer; input the output result of the cost calculation layer and the current frame feature map of the (L + 1)th layer into the optical flow prediction layer; input the output result of the optical flow prediction layer into the optical flow optimization layer to obtain the second motion information. The initial motion information corresponding to the current frame feature map and the reference frame feature map of the (L + 1)th layer.

[0149] Furthermore, based on the initial motion information of layer L+1, the initial motion information corresponding to the current frame feature map and the reference frame feature map of layer L is obtained. Specifically, the initial motion information of layer L+1 is upsampled, and the scale of the upsampled initial motion information is the same as the scales of the current frame feature map and the reference frame feature map of layer L. Then, the upsampled initial motion information and the reference frame feature map of layer L are input into the warping layer; the output result of the warping layer and the current frame feature map of layer L are input into the cost calculation layer. The output result of the cost calculation layer, the upsampled initial motion information of layer L+1, and the current frame feature map of layer L are input into the optical flow prediction layer; the output result of the optical flow prediction layer is input into the optical flow optimization layer to obtain the initial motion information corresponding to the current frame feature map and the reference frame feature map of layer L.

[0150] And so on, based on the warping layer, cost calculation layer, optical flow prediction layer, and optical flow optimization layer in the pyramid network model, the initial motion information corresponding to the current frame feature map and the reference frame feature map of each layer in the pyramid can be determined respectively. Then, the initial motion information of each layer is fused to obtain the second motion information. The process of fusing the initial motion information of each layer can be to splice the initial motion information of each layer in the channel dimension, and then perform convolution processing on the initial motion information after channel dimension splicing to obtain the fusion processing result. Or the initial motion information finally obtained by upsampling the initial motion information of the low-resolution layer layer by layer is used as the second motion information.

[0151] Figure 6 It is a schematic diagram of the pyramid network model structure provided by the embodiment of the present invention, and Figure 6 the process of determining the initial motion information corresponding to the current frame feature map and the reference frame feature map of layer L is schematically shown.

[0152] The basic pyramid network architecture can be the PWC-Net network architecture, and this network includes the following several modules:

[0153] Feature input layer: Downsample and convolve the input image to form a six-layer pyramid structure, for example;

[0154] Image warping layer: Upsample the initial motion information predicted by layer L+1 of the pyramid and perform an optical flow warp with the reference frame feature map of layer L to make it closer to the current frame feature map;

[0155] Cost calculation layer: Perform associated pixel matching on the two processed feature maps;

[0156] Optical flow prediction layer: Use a convolutional neural network for optical flow estimation;

[0157] Optical flow optimization layer: Use dilated convolution to improve the estimation accuracy of large displacement optical flow;

[0158] In the embodiments of the present invention, the following improvements are made on the basis of PWC-Net:

[0159] Since pyramid optical flow estimation is performed in the feature space, the number of layers N of the pyramid can be less than that of the original network, for example, N < 6, so as to construct a lighter pyramid network.

[0160] Compared with using ordinary convolutional layers in the optical flow prediction layer, consider using a residual block structure to prevent gradient disappearance, or consider using a depthwise separable convolutional structure, a spatial separable convolutional structure, etc., to greatly reduce the number of model parameters and further lighten the network structure.

[0161] Add a filtering layer after the dilated convolutional structure in the optical flow optimization layer. The filtering layer can be a bilateral filtering layer or a Gaussian filtering layer to remove high-frequency noise in the image while retaining the boundary information of the image. That is, the optical flow prediction layer includes a second residual block, or a depthwise separable convolutional structure, or a spatial separable convolutional structure; the optical flow optimization layer includes a dilated convolutional structure and a filtering layer.

[0162] The above networks are only shown as several feasible examples. In specific applications, the above networks can also exist in the form of other network models.

[0163] Embodiment 4:

[0164] On the basis of the above embodiments, in the embodiments of the present invention, performing motion compensation based on the target motion information to obtain a compensated image includes:

[0165] Inputting the target motion information and the reference frame feature map into the predicted frame feature map module in the motion information compensation network to obtain a predicted frame feature map; inputting the predicted frame feature map into the reconstructed image module in the motion information compensation network to obtain a compensated image.

[0166] In the embodiments of the present invention, the specific process of performing inter-frame prediction based on target motion information is to input the target motion information and the reference frame feature map into the predicted frame feature map module in the motion information compensation network, obtain a predicted frame feature map based on the predicted frame feature map module, and then input the predicted frame feature map into the reconstructed image module in the motion information compensation network to obtain a compensated image based on the reconstructed image module.

[0167] In the prior art, motion compensation is generally implemented using a convolutional neural network. The input of this network is the reference frame feature map and the motion information obtained by motion estimation. This method is prone to inaccurate prediction results. Moreover, in an ordinary CNN, each layer uses a fixed and regular convolutional kernel, that is, the range of the receptive field is fixed, which easily limits the network's ability to model geometric transformations.

[0168] For the above reasons, in the motion compensation process of the embodiments of the present invention, deformable convolution is introduced to improve the accuracy of motion compensation. Figure 7 FIG. Figure 7 is a basic schematic diagram of the deformable convolution provided by the embodiments of the present invention, that is, a learnable offset is introduced for the position of each pixel in the receptive field, and this offset makes the receptive field no longer a rigid square, but close to the actual shape of the object.

[0169] Therefore, in the embodiments of the present invention, as Figure 8 shown, the predicted frame feature map module includes a deformable convolutional layer, a concatenation layer, and a convolutional layer connected in sequence. The number of convolutional layers can be multiple, Figure 8 FIG. Figure 8 shows two convolutional layers.

[0170] Input the target motion information and the reference frame feature map into the deformable convolutional layer; input the output result of the deformable convolutional layer and the reference frame feature map into the concatenation layer; the concatenation layer is used to concatenate the output result of the deformable convolutional layer and the reference frame feature map in the channel dimension, and then input the output result of the concatenation layer into the first convolutional layer; input the output result of the first convolutional layer into the second convolutional layer, and fuse the output result of the second convolutional layer and the output result of the deformable convolutional layer to obtain the predicted frame feature map. Fusing the output result of the second convolutional layer and the output result of the deformable convolutional layer can be adding the feature values at the corresponding positions of the output result of the second convolutional layer and the output result of the deformable convolutional layer to obtain the fusion result, or assigning different weights to the output result of the second convolutional layer and the output result of the deformable convolutional layer, and adding the feature values at the corresponding positions of the output result of the second convolutional layer and the output result of the deformable convolutional layer after weighting to obtain the fusion result.

[0171] As Figure 8 shown, concatenate the feature map after deformable convolution and the reference frame feature map as the input of the subsequent convolutional layer, and refer to the structure of the residual network for motion compensation. The form of the ordinary CNN network connected after the deformable convolution here can be various. For example, an activation function can be added between the convolutional layers, and a residual block can be connected after the example network structure, etc.

[0172] In addition, since the motion compensation process is also carried out in the feature space, after the motion compensation is completed, the predicted frame feature map needs to be deconvolved back to the pixel space. As Figure 9 shown, the reconstructed image module includes a third residual block and a deconvolutional layer connected in sequence; input the predicted frame feature map into the third residual block; input the output result of the third residual block into the deconvolutional layer to obtain the compensated image. The number of third residual blocks can be multiple, Figure 9 FIG. Figure 9 shows 2 third residual blocks, Figure 9 which is only for illustration, and the number of third residual blocks can be set according to needs.

[0173] Embodiment 5:

[0174] Based on the above embodiments, in the embodiments of the present invention, if there are at least two reference frame feature maps, the obtaining of the predicted frame feature map includes:

[0175] Group at least two reference frame feature maps, and input each group of reference frame feature maps and corresponding target motion information into their respective deformable convolutional layers; input the output results of each deformable convolutional layer into the temporal attention module;

[0176] Input the output results of each group of temporal attention modules into the concatenation layer;

[0177] Input the output result of the concatenation layer into the spatial attention module, and obtain the predicted frame feature map based on the spatial attention module.

[0178] Figure 10 The framework diagram for obtaining the predicted frame feature map provided by the embodiments of the present invention is as Figure 10 shown. If the reference frame feature map includes reference frame feature Figure 1 , reference frame feature Figure 2 , reference frame feature map n - 1, and reference frame feature map n, and each reference frame feature map corresponds to target motion information, as Figure 10 shown, reference frame feature Figure 1 corresponds to target motion information 1, reference frame feature Figure 2 corresponds to target motion information 2, reference frame feature map n - 1 corresponds to target motion information n - 1, and reference frame feature map n corresponds to target motion information n.

[0179] Take reference frame feature Figure 1 and reference frame feature Figure 2 as a group, and take reference frame feature map n - 1 and reference frame feature map n as a group. Input reference frame feature Figure 1 and target motion information 1 into the deformable convolutional layer, input reference frame feature Figure 2 and target motion information 2 into another deformable convolutional layer, and then input the output results of the two deformable convolutional layers into the temporal attention module. Input reference frame feature map n - 1 and target motion information n - 1 into the deformable convolutional layer, input reference frame feature map n and target motion information n into another deformable convolutional layer, and then input the output results of the two deformable convolutional layers into another temporal attention module. Input the output results of the two temporal attention modules into the concatenation layer, perform concatenation in the channel dimension on the output results of the two temporal attention modules, and then input the output result of the concatenation layer into the spatial attention module, and obtain the predicted frame feature map based on the spatial attention module.

[0180] The feature maps of each reference frame and the corresponding target motion information are input into the deformable convolutional layer to obtain aligned feature maps. The feature maps are combined in pairs and input into the temporal attention module for feature extraction. Then, the extracted features of each group are concatenated and sent into the spatial attention module for feature fusion, and finally, the predicted frame feature map is obtained.

[0181] Figure 11 FIG. is a schematic diagram of the processing process of the temporal attention module provided by the embodiment of the present invention for the output results of each deformable convolutional layer. The processing process of the temporal attention module for the output results of each deformable convolutional layer includes: respectively inputting the output results of each deformable convolutional layer into the corresponding convolutional layer in the temporal attention module; performing dot product and activation function operations on the output results of each convolutional layer; multiplying the result of the activation function operation by the output result of any deformable convolutional layer to obtain the output result of the temporal attention module.

[0182] Figure 12 FIG. is a schematic diagram of the processing process of the spatial attention module provided by the embodiment of the present invention for the output results of the concatenation layer. The processing process of the spatial attention module for the output results of the concatenation layer includes: inputting the output results of the concatenation layer into the convolutional layer in the spatial attention module; inputting the output results of the convolutional layer into the multi-scale network structure; as Figure 13 shown, the multi-scale network structure includes fourth residual blocks of different scales, Figure 13 and 5 fourth residual blocks are shown in ; performing activation function operation on the output results of the multi-scale network structure; multiplying the result of the activation function operation by the output result of the convolutional layer to obtain the predicted frame feature map.

[0183] It should be noted that the above-mentioned networks are only shown as several feasible examples. In specific applications, the above networks may also exist in the form of other network models.

[0184] Embodiment 6:

[0185] Based on the above embodiments, in the embodiment of the present invention, the method further includes:

[0186] Sending the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image to the decoding end, so that the decoding end determines the predicted frame image according to the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image; the reference direction information of the reference frame image includes forward reference information and / or backward reference information.

[0187] When the reference frame image is only one frame, the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image are sent to the decoding end, so that the decoding end determines the predicted frame image according to the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image.

[0188] When the reference frame images are multiple frames, the embodiments of the present invention provide the following methods:

[0189] Method 1: Perform fusion or preselection processing on the target motion information corresponding to each reference frame image to obtain one or two pieces of target motion information after fusion or preselection processing; send the target motion information after fusion or preselection processing, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end determines the predicted frame image according to the target motion information after fusion or preselection processing, the reference direction information of each reference frame image, and the index number of each reference frame image.

[0190] Method 2: Send the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end respectively determines candidate predicted frame images according to the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image, and performs fusion processing on each candidate predicted frame image to obtain the predicted frame image.

[0191] Method 3: Send the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end performs alignment and fusion processing according to each reference frame image and its corresponding target motion information in the motion compensation stage to determine the predicted frame image.

[0192] In the case of multiple-frame reference, three pieces of information, namely size, direction, and reference frame index, need to be transmitted for the motion information. Specifically, it is divided into the following situations:

[0193] (1) First perform fusion or preselection processing on multiple reference frames. The preselection processing can be performed according to the distance between the reference frame and the current frame, the coding quality of the reference frame, etc. Finally, 1 or 2 reference frames are selected for unidirectional or bidirectional motion estimation. The index of the reference frame, the motion vector, and the reference direction need to be transmitted.

[0194] (2) Perform fusion processing on multiple motion vectors and their corresponding reference frames respectively at the encoding end to obtain a fused motion vector and a fused reference frame. The index of the reference frame and the fused motion vector need to be transmitted.

[0195] (3) Multiple frames generate multiple motion information through motion estimation, and there are the following processing methods for the above multiple motion information:

[0196] (i) Transmit the multiple motion information to the decoding end, use each motion information to obtain multiple predicted frames, and obtain the predicted frame of the current frame by taking a weighted average of each prediction result. It is necessary to transmit the indexes of each reference frame, motion vectors, and reference directions.

[0197] (ii) Transmit the multiple motion information to the decoding end. It is necessary to transmit the indexes of each reference frame, motion vectors, and reference directions, and perform operations such as alignment and fusion in the motion compensation stage to finally obtain the predicted frame.

[0198] Embodiment 7:

[0199] Figure 14 The structural schematic diagram of the inter-frame prediction device provided by the embodiment of the present invention. The device includes:

[0200] The first acquisition unit 141 is used to acquire the current frame image and the corresponding reference frame image;

[0201] The first determination unit 142 is used to determine the first motion information according to the current frame image, the reference frame image, and the optical flow estimation algorithm;

[0202] The second determination unit 143 is used to perform feature extraction on the current frame image and the reference frame image respectively to obtain the corresponding current frame feature map and reference frame feature map; input the current frame feature map and the reference frame feature map into the trained motion information detection model, and determine the second motion information based on the motion information detection model;

[0203] The prediction unit 144 is used to perform fusion processing on the first motion information and the second motion information to obtain the target motion information, and perform motion compensation based on the target motion information to obtain the compensated image.

[0204] The first acquisition unit 141 is specifically used to select at least one reference frame image from the forward reference list of the current frame image; or select at least one reference frame image from the backward reference list of the current frame image; or select at least one reference frame image from the forward and backward reference lists of the current frame image respectively.

[0205] The second determination unit 143 is specifically used to input the current frame image and the reference frame image into the feature extraction network model respectively, and perform feature extraction on the current frame image and the reference frame image respectively based on the feature extraction network model to obtain the corresponding current frame feature map and reference frame feature map; wherein, the feature extraction network model includes a first convolutional neural network structure and at least one first residual block connected in sequence.

[0206] The second determination unit 143 is specifically configured such that the trained motion information detection model includes a trained first neural network model, and the first neural network model includes a second convolutional neural network structure; input the current frame feature map and the reference frame feature map into the trained first neural network model, and based on the first neural network model, determine the second motion information.

[0207] The second determination unit 143 is specifically configured to input the current frame image and the reference frame image into a pyramid network model respectively, perform feature extraction on the current frame image and the reference frame image respectively based on the pyramid network model, and obtain the current frame feature map and the reference frame feature map in each layer of the pyramid; wherein, the pyramid network model includes at least two feature extraction network structures connected in sequence.

[0208] The second determination unit 143 is specifically configured such that the trained motion information detection model includes a trained second neural network model, and the second neural network model includes a warping layer, a cost calculation layer, an optical flow prediction layer, and an optical flow optimization layer connected in sequence; based on the warping layer, the cost calculation layer, the optical flow prediction layer, and the optical flow optimization layer in the pyramid network model, determine the initial motion information corresponding to the current frame feature map in each layer of the pyramid respectively; the initial motion information of the low-resolution layer is upsampled layer by layer to obtain the second motion information; or fuse the initial motion information of each layer to obtain the second motion information.

[0209] The optical flow prediction layer includes a second residual block, or a depthwise separable convolution structure, or a spatial separable convolution structure; the optical flow optimization layer includes a dilated convolution structure and a filtering layer.

[0210] The prediction unit 144 is specifically configured to input the target motion information and the reference frame feature map into a prediction frame feature map module in a motion information compensation network to obtain a prediction frame feature map; input the prediction frame feature map into a reconstructed image module in the motion information compensation network to obtain a compensated image.

[0211] The prediction frame feature map module includes a deformable convolution layer, a concatenation layer, and a convolution layer connected in sequence;

[0212] The prediction unit 144 is specifically configured to input the target motion information and the reference frame feature map into the deformable convolution layer; input the output result of the deformable convolution layer and the reference frame feature map into the concatenation layer; input the output result of the concatenation layer into the convolution layer; fuse the output result of the convolution layer and the output result of the deformable convolution layer to obtain a prediction frame feature map.

[0213] The reconstruction image module includes a third residual block and a transposed convolution layer connected in sequence;

[0214] The prediction unit 144 is specifically configured to input the predicted frame feature map into the third residual block; input the output result of the third residual block into the transposed convolution layer to obtain a compensated image.

[0215] If there are at least two reference frame feature maps, the prediction unit is specifically configured to group the at least two reference frame feature maps, input each group of reference frame feature maps and corresponding target motion information into their respective deformable convolution layers; input the output results of each deformable convolution layer into the temporal attention module; input the output results of each group of temporal attention modules into the concatenation layer; input the output result of the concatenation layer into the spatial attention module, and obtain the predicted frame feature map based on the spatial attention module.

[0216] The device further includes:

[0217] A sending unit 145, configured to send the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image to the decoding end, so that the decoding end determines the predicted frame image according to the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image; the reference direction information of the reference frame image includes forward reference information and / or backward reference information.

[0218] The sending unit 145 is specifically configured to, if there are multiple reference frame images, perform fusion or preselection processing on the target motion information corresponding to each reference frame image to obtain one or two pieces of target motion information after fusion or preselection processing; send the target motion information after fusion or preselection processing, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end determines the predicted frame image according to the target motion information after fusion or preselection processing, the reference direction information of each reference frame image, and the index number of each reference frame image; or

[0219] If there are multiple reference frame images, send the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end respectively determines candidate predicted frame images according to the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image, and performs fusion processing on each candidate predicted frame image to obtain the predicted frame image; or

[0220] If there are multiple reference frame images, send the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end can perform alignment and fusion processing during the motion compensation stage according to each reference frame image and its corresponding target motion information to determine the predicted frame image.

[0221] Embodiment 8:

[0222] Figure 15 It is a schematic diagram of the encoding process provided by an embodiment of the present invention. The encoding process includes the following steps:

[0223] S151: Obtain the current frame image.

[0224] S152: Encode the current frame image to obtain the encoded image corresponding to the current frame image; wherein, during the encoding process, use the inter-frame prediction method described in any one of the above to perform inter-frame prediction on the current frame image.

[0225] The encoding method provided by an embodiment of the present invention is applied to an encoder. Figure 16 It is a schematic diagram of the structure of an encoder provided by an embodiment of the present invention. The encoder includes:

[0226] A second acquisition unit 161, configured to acquire the current frame image;

[0227] An encoding unit 162, configured to encode the current frame image to obtain the encoded image corresponding to the current frame image.

[0228] Embodiment 9:

[0229] Figure 17 It is a schematic diagram of the decoding process provided by an embodiment of the present invention. The decoding process includes the following steps:

[0230] S171: Receive the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image sent by the encoder, wherein the target motion information is obtained by the encoder using the inter-frame prediction method described in any one of the above;

[0231] S172: Determine the predicted frame image according to the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image.

[0232] The decoding method provided by an embodiment of the present invention is applied to a decoder. Figure 18 It is a schematic diagram of the structure of a decoder provided by an embodiment of the present invention. The decoder includes:

[0233] A receiving unit 181, configured to receive the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image sent by the encoder;

[0234] A decoding unit 182, configured to determine a predicted frame image according to the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image.

[0235] Embodiment 10:

[0236] Based on the above embodiments, an electronic device is further provided in an embodiment of the present invention. As Figure 19 shown, it includes: a processor 301, a communication interface 302, a memory 303, and a communication bus 304. Among them, the processor 301, the communication interface 302, and the memory 303 complete mutual communication through the communication bus 304;

[0237] A computer program is stored in the memory 303. When the program is executed by the processor 301, the processor 301 is caused to execute the following steps:

[0238] Obtain a current frame image and a corresponding reference frame image;

[0239] According to the current frame image, the reference frame image, and an optical flow estimation algorithm, determine first motion information;

[0240] Extract features from the current frame image and the reference frame image respectively to obtain a corresponding current frame feature map and a reference frame feature map; input the current frame feature map and the reference frame feature map into a trained motion information detection model, and based on the motion information detection model, determine second motion information;

[0241] Perform fusion processing on the first motion information and the second motion information to obtain target motion information, and perform motion compensation based on the target motion information to obtain a compensated image.

[0242] Based on the same inventive concept, an electronic device is further provided in an embodiment of the present invention. Since the principle of solving problems by the above electronic device is similar to that of the inter-frame prediction method, the implementation of the above electronic device can refer to the implementation of the method, and the repeated parts will not be described again.

[0243] The electronic device provided in the embodiment of the present invention may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (Personal Digital Assistant, PDA), a network-side device, etc.

[0244] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0245] The communication interface 302 is used for communication between the above electronic device and other devices.

[0246] The memory may include a Random Access Memory (RAM), or may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0247] The above processor may be a general-purpose processor, including a central processing unit, a Network Processor (NP), etc.; it may also be a Digital Signal Processing (DSP), an application-specific integrated circuit, a field programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0248] In the embodiment of the present invention, when the processor executes the program stored on the memory, it realizes obtaining the current frame image and the corresponding reference frame image; determining the first motion information according to the current frame image, the reference frame image, and the optical flow estimation algorithm; respectively performing feature extraction on the current frame image and the reference frame image to obtain the corresponding current frame feature map and reference frame feature map; inputting the current frame feature map and the reference frame feature map into the trained motion information detection model, and determining the second motion information based on the motion information detection model; performing fusion processing on the first motion information and the second motion information to obtain the target motion information, and performing motion compensation based on the target motion information to obtain the compensated image.

[0249] In the embodiments of the present invention, after obtaining the current frame image and the corresponding reference frame image, two methods are adopted to respectively determine the motion information of the current frame image and the corresponding reference frame image. That is, in the first method, an optical flow estimation algorithm is used in the pixel domain to determine the first motion information of the current frame image and the corresponding reference frame image; in the second method, in the feature space domain, feature extraction is respectively performed on the current frame image and the reference frame image to obtain the corresponding current frame feature map and reference frame feature map; the current frame feature map and the reference frame feature map are input into the trained motion information detection model, and based on the motion information detection model, the second motion information is determined. Then, the first motion information and the second motion information are fused to obtain the target motion information, and finally, frame interpolation prediction is performed according to the target motion information. The embodiments of the present invention start from two dimensions, namely the pixel domain and the feature space domain, to obtain the final target motion information, making the determined target motion information more accurate, and then performing motion compensation based on the target motion information, making the compensated image more accurate.

[0250] Embodiment 11:

[0251] Based on the above embodiments, the embodiments of the present invention further provide a computer-readable storage medium, in which a computer program executable by an electronic device is stored. When the program runs on the electronic device, the electronic device is caused to execute the following steps when executed:

[0252] Obtain the current frame image and the corresponding reference frame image;

[0253] Determine the first motion information according to the current frame image, the reference frame image, and the optical flow estimation algorithm;

[0254] Respectively perform feature extraction on the current frame image and the reference frame image to obtain the corresponding current frame feature map and reference frame feature map; input the current frame feature map and the reference frame feature map into the trained motion information detection model, and based on the motion information detection model, determine the second motion information;

[0255] Fuse the first motion information and the second motion information to obtain the target motion information, and perform motion compensation based on the target motion information to obtain the compensated image.

[0256] Based on the same inventive concept, the embodiments of the present invention further provide a computer-readable storage medium. Since the principle of the processor solving problems when executing the computer program stored on the above computer-readable storage medium is similar to that of the frame interpolation prediction method, the implementation of the processor executing the computer program stored on the above computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0257] The above computer-readable storage medium may be any available medium or data storage device accessible by a processor in an electronic device, including but not limited to magnetic memories such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc., optical memories such as CDs, DVDs, BDs, HVDs, etc., and semiconductor memories such as ROM, EPROM, EEPROM, non-volatile memories (NANDFLASH), solid-state drives (SSD), etc.

[0258] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a means for implementing the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0259] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including an instruction means that implements the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0260] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the specified functions in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0261] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0262] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these modifications and variations.

Claims

1. An inter-frame prediction method, characterized in that, The method includes: Obtaining a current frame image and a corresponding reference frame image; Determining first motion information according to the current frame image, the reference frame image, and an optical flow estimation algorithm; Performing feature extraction on the current frame image and the reference frame image respectively to obtain a corresponding current frame feature map and a reference frame feature map; inputting the current frame feature map and the reference frame feature map into a trained motion information detection model, and determining second motion information based on the motion information detection model; Performing fusion processing on the first motion information and the second motion information to obtain target motion information, inputting the target motion information and the reference frame feature map into a predicted frame feature map module in a motion information compensation network to obtain a predicted frame feature map; inputting the predicted frame feature map into a reconstructed image module in the motion information compensation network to obtain a compensated image; If there are at least two reference frame feature maps, the obtaining of the predicted frame feature map includes: Grouping at least two reference frame feature maps, inputting each group of reference frame feature maps and corresponding target motion information into respective deformable convolutional layers; inputting the output results of each deformable convolutional layer into a temporal attention module; Inputting the output results of each group of temporal attention modules into a concatenation layer; Inputting the output result of the concatenation layer into a spatial attention module, and obtaining a predicted frame feature map based on the spatial attention module.

2. The method according to claim 1, wherein The obtaining of the corresponding reference frame image includes: Selecting at least one reference frame image from the forward reference list of the current frame image; or selecting at least one reference frame image from the backward reference list of the current frame image; or respectively selecting at least one reference frame image from the forward and backward reference lists of the current frame image.

3. The method according to claim 1, characterized in that, The performing feature extraction on the current frame image and the reference frame image respectively to obtain a corresponding current frame feature map and a reference frame feature map includes: Inputting the current frame image and the reference frame image into a feature extraction network model respectively, and performing feature extraction on the current frame image and the reference frame image respectively based on the feature extraction network model to obtain a corresponding current frame feature map and a reference frame feature map; wherein, the feature extraction network model includes a first convolutional neural network structure and at least one first residual block connected in sequence.

4. The method according to claim 3, wherein The inputting the current frame feature map and the reference frame feature map into a trained motion information detection model, and determining second motion information based on the motion information detection model includes: The trained motion information detection model includes a trained first neural network model, and the first neural network model includes a second convolutional neural network structure; Inputting the current frame feature map and the reference frame feature map into the trained first neural network model, and determining second motion information based on the first neural network model.

5. The method according to claim 1, characterized in that, The performing feature extraction on the current frame image and the reference frame image respectively to obtain a corresponding current frame feature map and a reference frame feature map includes: Input the current frame image and the reference frame image into the pyramid network model respectively, and perform feature extraction on the current frame image and the reference frame image based on the pyramid network model to obtain the current frame feature map and the reference frame feature map in each layer of the pyramid; wherein, the pyramid network model includes at least two sequentially connected feature extraction network structures.

6. The method according to claim 5, wherein Input the current frame feature map and the reference frame feature map into the trained motion information detection model, and based on the trained motion information detection model, determining the second motion information includes: The trained motion information detection model includes a trained second neural network model, and the second neural network model includes a warping layer, a cost calculation layer, an optical flow prediction layer, and an optical flow optimization layer connected in sequence; Based on the warping layer, the cost calculation layer, the optical flow prediction layer, and the optical flow optimization layer in the pyramid network model, determine the initial motion information corresponding to the current frame feature map in each layer of the pyramid respectively; The initial motion information of the low-resolution layer is upsampled layer by layer to obtain the second motion information; or the initial motion information of each layer is fused to obtain the second motion information.

7. The method according to claim 6, wherein The optical flow prediction layer includes a second residual block, or a depthwise separable convolution structure, or a spatial separable convolution structure; The optical flow optimization layer includes a dilated convolution structure and a filtering layer.

8. The method according to claim 1, wherein The reconstruction image module includes a third residual block and a transposed convolution layer connected in sequence; Inputting the predicted frame feature map into the reconstruction image module in the motion information compensation network to obtain the compensated image includes: Input the predicted frame feature map into the third residual block; Input the output result of the third residual block into the transposed convolution layer to obtain the compensated image.

9. The method according to claim 1, wherein The processing process of the temporal attention module for the output results of the respective deformable convolutional layers includes; Input the output results of the respective deformable convolutional layers into the respective corresponding convolutional layers in the temporal attention module; Perform dot product and activation function operations on the output results of each convolutional layer; multiply the result of the activation function operation by the output result of any one of the deformable convolutional layers to obtain the output result of the temporal attention module.

10. The method according to claim 1, wherein The processing process of the spatial attention module for the output result of the concatenation layer includes; Input the output result of the concatenation layer into the convolutional layer in the spatial attention module; Input the output result of the convolutional layer into the multi-scale network structure; The multi-scale network structure includes fourth residual blocks of different scales; Perform activation function operations on the output results of the multi-scale network structure; multiply the result of the activation function operation by the output result of the convolutional layer to obtain the predicted frame feature map.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: Send the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image to the decoding end, so that the decoding end determines the predicted frame image according to the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image; the reference direction information of the reference frame image includes forward reference information and / or backward reference information.

12. The method according to claim 11, wherein, Sending the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image to the decoding end includes: If there are multiple reference frame images, performing fusion or preselection processing on the target motion information corresponding to each reference frame image to obtain one or two target motion information after fusion or preselection processing; sending the target motion information after fusion or preselection processing, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end determines a predicted frame image according to the target motion information after fusion or preselection processing, the reference direction information of each reference frame image, and the index number of each reference frame image; or If there are multiple reference frame images, sending the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end respectively determines candidate predicted frame images according to the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image, and performs fusion processing on each candidate predicted frame image to obtain a predicted frame image; or If there are multiple reference frame images, sending the target motion information corresponding to each reference frame image, the reference direction information of each reference frame image, and the index number of each reference frame image to the decoding end, so that the decoding end performs alignment and fusion processing according to each reference frame image and the corresponding target motion information in the motion compensation stage to determine a predicted frame image.

13. An inter-frame prediction device, characterized in that, The device includes: A first acquisition unit, configured to acquire a current frame image and a corresponding reference frame image; A first determination unit, configured to determine first motion information according to the current frame image, the reference frame image, and an optical flow estimation algorithm; A second determination unit, configured to respectively perform feature extraction on the current frame image and the reference frame image to obtain a corresponding current frame feature map and reference frame feature map; input the current frame feature map and the reference frame feature map into a trained motion information detection model, and determine second motion information based on the motion information detection model; A prediction unit, configured to perform fusion processing on the first motion information and the second motion information to obtain target motion information, input the target motion information and the reference frame feature map into a predicted frame feature map module in a motion information compensation network to obtain a predicted frame feature map; input the predicted frame feature map into a reconstructed image module in the motion information compensation network to obtain a compensated image; Among them, the prediction unit is specifically configured to, if there are at least two reference frame feature maps, group the at least two reference frame feature maps, and input each group of reference frame feature maps and the corresponding target motion information into a corresponding deformable convolutional layer; input the output results of each deformable convolutional layer into a temporal attention module; input the output results of each group of temporal attention modules into a concatenation layer; input the output result of the concatenation layer into a spatial attention module, and obtain a predicted frame feature map based on the spatial attention module.

14. A coding method, characterized in that, The method includes: Acquiring a current frame image; Encode the current frame image to obtain the encoded image corresponding to the current frame image; wherein, during the encoding process, use the inter-frame prediction method described in any one of claims 1-12 to perform inter-frame prediction on the current frame image.

15. An encoder based on the inter-frame prediction method according to any one of claims 1-12, characterized in that, The encoder includes: A second acquisition unit, configured to acquire a current frame image; An encoding unit, configured to encode the current frame image to obtain the encoded image corresponding to the current frame image.

16. A decoding method, characterized in that, The method includes: Receive the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image sent by the encoder, wherein the target motion information is obtained by the encoder using the inter-frame prediction method described in any one of claims 1-12; Determine the predicted frame image according to the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image.

17. A decoder for an inter prediction method according to any one of claims 1-12, characterized in that, The decoder includes: A receiving unit, configured to receive the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image sent by the encoder; A decoding unit, configured to determine the predicted frame image according to the target motion information, the reference direction information of the reference frame image, and the index number of the reference frame image.

18. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus; The memory is used to store computer programs; The processor, when executing the programs stored on the memory, implements the method steps described in any one of claims 1-12.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method steps described in any one of claims 1-12.

Images