Optical flow-based video encoding method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide a video encoding method and device based on optical flow, equipment, medium and program product. The method takes multiple video frames in front of the to-be-predicted video frame as reference frames based on a forward optical flow estimation module, captures the optical flow change characteristics of the reference frames through optical flow estimation, and gradually approximates the true value of the to-be-predicted video frame through a multi-round loop processing mode, and finally generates a target predicted video frame. The method gradually approximates the true value of the to-be-predicted video frame through the multi-round loop optical flow estimation method, and provides the accuracy of the prediction estimation. At the same time, only the video frames in the front direction are taken as reference frames, which avoids the influence of introducing backward reference frames on the encoding efficiency, and meets the needs of low-latency time-domain encoding mode.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of video coding, and in particular, to a video coding method and device based on optical flow, an electronic device, a storage medium, and a computer program product. BACKGROUND

[0002] A Video Coding for Machines (VCM) working group is established by a video coding standard organization to study video coding for machine processing. At present, a temporal frame filling network adopted by the working group is a method of generating an intermediate frame by prediction based on forward and backward reference frames. This method performs well in terms of frame filling effect, can effectively improve the frame rate and smoothness of a video, and is suitable for various video processing scenarios. However, this method of bidirectional reference frames has certain limitations, especially in a Low Delay (LD) mode, because it needs to rely on future frames as reference, which cannot meet the requirements of application scenarios with high real-time requirements. SUMMARY

[0003] The present disclosure provides a video coding method and device based on optical flow, an electronic device, a storage medium, and a computer program product.

[0004] Other features and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0005] According to one aspect of the present disclosure, a video coding method based on optical flow is provided, the method comprising: splicing a first video frame, a second video frame, and a first change frame, a second change frame, and a mask obtained in an i-1th round to obtain a first feature; wherein the first change frame and the second change frame are obtained by change processing based on the first video frame and the second video frame respectively; based on a scaling factor of an i th round, performing optical flow estimation on an optical flow of an i-1th round and the first feature by a forward optical flow estimation module to obtain residual information of the optical flow and the mask of the i th round; fusing the optical flow, the mask obtained in the i-1th round, and the residual information of the optical flow and the mask of the i th round to obtain the optical flow and the mask of the i th round; based on the optical flow of the i th round, performing change processing on the first video frame and the second video frame to obtain a first change frame and a second change frame of the i th round; in response to a round number i not being equal to a preset round number N, increasing the round number i by 1 and repeating the foregoing steps; and in response to the round number i being equal to the preset round number N, generating a target predicted video frame based on the first change frame, the second change frame, and the mask obtained in the i th round.

[0006] In an example embodiment, the first video frame, the second video frame, and the target predicted video frame are time-continuous video frames.

[0007] In an example embodiment, the scaling factor is associated with a wheel number i.

[0008] In an example embodiment, the generating the target predicted video frame based on the first change frame, the second change frame and the mask obtained in the i-th wheel includes:

[0009] The generating the target predicted video frame based on the first change frame, the second change frame and the mask obtained in the i-th wheel includes a weighted sum, and a corresponding formula is as follows:

[0010]

[0011] wherein, denotes the first change frame obtained in the i-th wheel, denotes the second change frame obtained in the i-th wheel, and I2 denotes the target predicted video frame, and M i denotes the mask obtained in the i-th wheel.

[0012] In an example embodiment, the generating the optical flow and the residual information of the mask in the i-th wheel based on the scaling factor in the i-th wheel includes: performing optical flow estimation on the optical flow in the (i-1)-th wheel and the first feature based on the scaling factor in the i-th wheel by a forward optical flow estimation module to obtain the optical flow in the i-th wheel and the residual information of the mask; performing optical flow change estimation on the optical flow in the (i-1)-th wheel and the first feature by an optical flow change module based on the scaling factor in the i-th wheel to obtain an optical flow change feature; performing spatial change estimation on the optical flow in the (i-1)-th wheel and the first feature by a spatial change module to obtain a spatial change feature; splicing the optical flow change feature and the spatial change feature to obtain an optical flow estimation feature; and performing deconvolution operation on the optical flow estimation feature to obtain the optical flow in the i-th wheel and the residual information of the mask.

[0013] In an example embodiment, the generating the optical flow change feature based on the scaling factor in the i-th wheel by the optical flow change module includes: splicing the optical flow in the (i-1)-th wheel and the first feature to obtain a second feature; performing channel alignment operation on the second feature to obtain a third feature; and performing optical flow feature learning on the third feature by a residual feature learning module to obtain the optical flow change feature.

[0014] In the example embodiment, the splicing the optical flow and the first feature of the i-1th round to obtain the second feature comprises: performing downsampling operation on the optical flow and the first feature of the i-1th round to obtain downsampled optical flow and first feature; splicing the downsampled optical flow and the first feature to obtain the second feature; the optical flow feature learning on the third feature by the residual feature learning module to obtain the optical flow change feature comprises: performing optical flow feature learning on the third feature by the residual feature learning module to obtain a fourth feature; performing channel alignment operation on the fourth feature to obtain a fifth feature; performing upsampling operation on the fifth feature to obtain the optical flow change feature; wherein the sampling rate of the downsampling and the upsampling is associated with the scaling factor.

[0015] In the example embodiment, the channel alignment operation on the second feature to obtain the third feature comprises: performing at least one convolution operation on the second feature to obtain the third feature aligned with a preset channel number.

[0016] In the example embodiment, the spatial change estimation on the optical flow and the first feature of the i-1th round by the spatial change module to obtain the spatial change feature comprises: splicing the optical flow and the first feature of the i-1th round to obtain a sixth feature; performing channel alignment operation on the sixth feature to obtain a seventh feature; performing spatial attention learning on the seventh feature by a group-based spatial attention (GSA) module to obtain the spatial attention feature; performing feature learning on the spatial attention feature to obtain the spatial change feature.

[0017] In the example embodiment, the spatial attention learning on the seventh feature by the group-based spatial attention (GSA) module to obtain the spatial attention feature comprises: performing first hole convolution operation on the seventh feature to obtain an eighth feature; dividing the eighth feature into at least one channel feature according to channels; performing hole convolution operation on the at least one channel feature respectively to obtain at least one channel spatial feature; splicing the at least one channel spatial feature to obtain a ninth feature; performing feature learning on the seventh feature and the ninth feature respectively to obtain a tenth feature and a thirteenth feature; performing activation processing on the tenth feature and the thirteenth feature, and performing point multiplication on the activated tenth feature and the thirteenth feature to obtain an eleventh feature; performing feature learning on the eleventh feature to obtain a twelfth feature; performing feature addition on the seventh feature and the twelfth feature to obtain the spatial attention feature.

[0018] In an example embodiment, the dividing the eighth feature into at least one channel feature according to a channel comprises: dividing the eighth feature into the at least one channel feature according to a preset channel number ratio.

[0019] In an example embodiment, the performing a hole convolution operation on the at least one channel feature respectively to obtain at least one channel spatial feature further comprises: different convolution kernel sizes of the hole convolution operation corresponding to different channel features.

[0020] In an example embodiment, the further comprising: judging whether a number of the generated target prediction video frames is greater than a remaining frame number of the video to be encoded; and in response to the number of the target prediction video frames being greater than the remaining frame number, discarding the target prediction video frames according to the remaining frame number.

[0021] According to yet another aspect of the present disclosure, there is provided a video encoding device based on optical flow, comprising: a video frame splicing module configured to splice a first video frame, a second video frame, and a first change frame, a second change frame and a mask obtained in an i-1th round to obtain a first feature; wherein the first change frame and the second change frame are obtained based on change processing of the first video frame and the second video frame respectively; a forward optical flow estimation module configured to obtain an i-th round optical flow and residual information of a mask by performing optical flow estimation on an optical flow in an i-1th round and the first feature through the forward optical flow estimation module based on a scaling factor in the i-th round; a fusion module configured to fuse the optical flow in the i-1th round, the mask, and the residual information of the i-th round optical flow and the mask to obtain the i-th round optical flow and the mask; a change processing module configured to perform change processing on the first video frame and the second video frame based on the i-th round optical flow to obtain a first change frame and a second change frame in the i-th round; and a prediction frame module configured to, in response to a round number i not being equal to a preset round number N, increase the round number i by 1 and repeat the foregoing steps, and in response to the round number i being equal to the preset round number N, generate a target prediction video frame based on the first change frame, the second change frame and the mask obtained in the i-th round.

[0022] According to yet another aspect of the present disclosure, there is provided an electronic device, comprising: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a video encoding method based on optical flow as described in the above embodiments.

[0023] According to yet another aspect of the present disclosure, there is provided a computer readable storage medium storing a computer program, which, when executed by a processor, implements a video encoding method based on optical flow as described in the above embodiments.

[0024] According to yet another aspect of the present disclosure, a computer program product is provided, comprising a computer program / signaling, characterized in that the computer program / signaling, when executed by a processor, implements the optical flow based video encoding method as described in the above embodiments.

[0025] The optical flow based video encoding method provided by the embodiments of the present disclosure takes the video frames in the forward direction of the video frame to be predicted as reference frames, captures the optical flow change characteristics of the reference frames through optical flow estimation, and gradually approximates the true value of the video frame to be predicted through multi-round loop processing, and finally generates the target predicted video frame. The method gradually approximates the true value of the video frame to be predicted through multi-round loop optical flow estimation, and provides the accuracy of prediction and estimation. At the same time, only the video frames in the forward direction are taken as reference frames, which avoids the influence of introducing backward reference frames on the encoding efficiency, and meets the needs of low latency time domain encoding mode.

[0026] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an undue limitation on the present disclosure.

[0028] Figure 1 A flowchart of an optical flow based video encoding method according to an embodiment of the present disclosure is shown;

[0029] Figure 2 A schematic diagram of an optical flow based video encoding process according to an embodiment of the present disclosure is shown;

[0030] Figure 3 A flowchart of an optical flow estimation process according to an embodiment of the present disclosure is shown;

[0031] Figure 4 A flowchart of an optical flow change estimation process according to an embodiment of the present disclosure is shown;

[0032] Figure 5 A flowchart of a spatial change estimation process according to an embodiment of the present disclosure is shown;

[0033] Figure 6 A structural schematic diagram of a grouping spatial attention module according to an embodiment of the present disclosure is shown;

[0034] Figure 7 A result schematic diagram of a forward optical flow estimation module according to an embodiment of the present disclosure is shown;

[0035] Figure 8 FIG. 1 shows a structural schematic diagram of a video coding device based on optical flow according to an example embodiment of the present disclosure;

[0036] Figure 9 FIG. 1 shows a structural schematic diagram of an electronic device suitable for implementing the example embodiments of the present disclosure. DETAILED DESCRIPTION

[0037] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these example implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive aspects to those skilled in the art. Features, structures, or characteristics described in connection with one example implementation can be combined in any suitable manner with features, structures or characteristics of other example implementations.

[0038] In addition, the drawings are to be considered in all respects as illustrative and not restrictive; identical reference numerals have been used, where possible, to denote identical or similar features, and thus repetition of the description thereof will be omitted. Some of the blocks in the drawings are functional blocks, and thus can not necessarily correspond to physical or logical entities. These functional blocks can be implemented in software, hardware, or a combination thereof, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0039] It should be noted that the ordinal terms "first", "second", and the like in the embodiments of the present disclosure are used only to distinguish different objects, and do not imply sequence, time, priority, or importance of the objects, and the description of "first", "second" does not necessarily mean that the objects are different.

[0040] To solve the above problems, the embodiments of the present disclosure provide a video coding method based on optical flow, which improves the existing frame interpolation algorithm based on optical flow estimation, avoids the dependence on backward reference frames, and meets the low delay mode requirement in video coding. This method significantly reduces the processing delay without sacrificing frame prediction quality.

[0041] Figure 1 FIG. 1 shows a structural schematic diagram of a video coding device based on optical flow according to an example embodiment of the present disclosure; Figure 2 FIG. 1 shows a structural schematic diagram of a video coding device based on optical flow according to an example embodiment of the present disclosure; Figure 1 、 2 As shown in FIG. 1, the video coding method based on optical flow can include the following steps.

[0042] In step S110, the first video frame, the second video frame, and the first change frame, the second change frame and the mask obtained in the i-1th round are spliced to obtain a first feature; wherein the first change frame and the second change frame are respectively obtained based on the first video frame and the second video frame.

[0043] In the embodiment of the present disclosure, the video encoding method predicts the future video frame based on the optical flow estimation of the existing video frame, thereby realizing the encoding of the inserted video frame. Wherein, the optical flow video frame prediction is a technology for predicting future frames based on the optical flow field between adjacent frames. Optical flow refers to the pixel-level motion vector of an object, surface or edge in a sequence of consecutive image frames. By calculating the optical flow, the changes between frames can be captured, and then used to predict the content of the subsequent frame. This method is usually used to predict one frame from several consecutive frames, and generates multiple frames through rolling prediction, which has the characteristics of high computational efficiency and good prediction accuracy in the short term.

[0044] In the embodiment of the present disclosure, based on the optical flow video frame prediction method, a step-by-step approximation method is adopted, and the optical flow estimation is performed on the existing video frame in a cycle to gradually approximate the prediction of the future video frame. Therefore, steps S110 to S140 in this scheme are a cycle estimation process of step-by-step approximation. According to the pre-set cycle estimation process step-by-step approximation, until the pre-set cycle jump-out condition is met.

[0045] In the embodiment of the present disclosure, as shown in Figure 2 The first video frame I0, the second video frame I1, and the first change frame The second change frame and the mask M i-1 are spliced to obtain a first feature X1.

[0046] Wherein, the first video frame I0 and the second video frame I1 are video frames that exist before the video frame to be predicted.

[0047] The first change frame The second change frame is the first change frame and the second change frame obtained in the i-1th round in the aforementioned cycle process. Wherein, the first change frame and the second change frame are obtained based on the first video frame and the second video frame through warping. Warping is an optical flow change technology in computer vision and image processing. It mainly refers to changing a given image to match the position and perspective of another image according to the given optical flow field. Specifically, warping compensates for the displacement between images by moving the pixels in the image, so that one image can be aligned with another image.

[0048] The mask Mi-1 is a mask representing the change of the optical flow obtained by the optical flow estimation.

[0049] In the example embodiment, the first video frame I0, the second video frame I1 and the target predicted video frame are adjacent video frames which are continuous in the time domain.

[0050] It should be noted that the example embodiment given in the present disclosure is based on predicting one to-be-predicted video frame based on two existing video frames. In actual application, a plurality of to-be-predicted video frames can be predicted based on a plurality of existing video frames as needed. For example, two to-be-predicted video frames can be predicted based on three existing video frames.

[0051] In the example embodiment, predicting a plurality of to-be-predicted video frames simultaneously based on a plurality of existing video frames can further adopt a rolling prediction manner. Rolling prediction refers to the process of predicting a series of continuous frames one by one, starting from the current frame and predicting future frames in time sequence, and realizing long-sequence frame prediction by constantly updating the reference frame. In the rolling prediction process, the reference frame can be an original video frame or a predicted frame obtained by prior prediction. The rolling prediction is performed backward based on the obtained predicted frame.

[0052] In step S120, based on the scaling factor of the i-th round, the optical flow of the (i-1)-th round and the first feature are subjected to optical flow estimation by a forward flow estimation module to obtain the residual information of the optical flow and the mask of the i-th round.

[0053] In the embodiment of the present disclosure, as shown in Figure 2 , the first feature X1 obtained by splicing the aforementioned step S110 and the optical flow F i-1 of the (i-1)-th round are respectively input into a forward flow estimation (FFE) module. The optical flow F i-1 is the flow information (Flow) obtained by optical flow estimation. The forward flow estimation module is a technology for estimating the pixel motion vector from the current video frame to the next video frame. The module predicts the flow field by analyzing the changes between adjacent frames, thereby supporting the interpolation or prediction of video frames. The forward flow estimation module is pre-provided with a scaling factor. The scaling factor is used to adjust the scaling ratio of the related input features. Based on the pre-provided scaling factor, the forward flow estimation module performs optical flow estimation on the input first feature X1 and the optical flow F i-1 of the (i-1)-th round to obtain the residual information of the optical flow and the mask of the i-th round, i.e., ΔF i , ΔM i . The residual information refers to the difference between the actual value and the predicted value in signal processing, which is used to represent the information that cannot be captured or the error.

[0054] In the example embodiment, the scaling factor is associated with the current round i, i.e. the scaling factor changes with the increase of the round, so as to realize the step-by-step approximation of the actual value. For example, the scaling factor is updated once every three rounds. The scaling factor becomes smaller with the increase of the round.

[0055] In the example embodiment, the scaling factor is not a fixed value preset in advance, but is calculated according to the parameter value obtained in the previous round in the loop process. How to select the parameter in the previous round and how to design the related calculation formula are not specifically limited in the disclosure.

[0056] In step S130, the optical flow obtained in the i-1th round, the mask and the residual information of the optical flow and the mask of the ith round are fused to obtain the optical flow and the mask of the ith round.

[0057] In the example embodiment of the disclosure, the optical flow F i-1 obtained in the i-1th round is fused with the residual information ΔF i of the optical flow of the ith round obtained in the foregoing to obtain the optical flow F i of the ith round. Similarly, the mask M i-1 obtained in the i-1th round is fused with the residual information ΔM i of the mask of the ith round obtained in the foregoing to obtain the mask M i of the ith round. For details, refer to the following formula:

[0058] F i = F i-1 + ΔF i

[0059] M i = M i-1 + ΔM i

[0060] In step S140, the first video frame and the second video frame are processed based on the optical flow of the ith round to obtain the first changed frame and the second changed frame of the ith round.

[0061] In the example embodiment of the disclosure, in order to perform the next round of loop processing, the first video frame I0 and the second video frame I1 are processed based on the optical flow F i of the ith round to obtain the first changed frame I0 and the second changed frame I1 of the ith round. Through the above processing, the first changed frame I0 and the second changed frame I1 needed for the feature splicing in step S110 of the next round are obtained.

[0062] In the embodiments of the present disclosure, the steps S110 to S140 are a loop process. Through the loop process, the first change frame Ii-1, the second change frame Ii-1, and the mask M The second change frame The mask M i-1 And the optical flow F i-1 The optical flow F i And the mask M i And the first change frame Ii-1 of the i-th round The second change frame In this way, with the increase of the round number i, the real value of the video frame to be predicted is gradually approached.

[0063] In step S150, in response to the round number i not being equal to the preset round number N, the round number i is increased by 1, and the foregoing steps are cycled.

[0064] In the embodiments of the present disclosure, the exit condition of the loop process is set as the preset round number N. In response to the current round number i not being equal to the preset round number N, that is, the exit condition is not met, the loop process is continued. Based on the loop process, the round number i is increased by 1, and the process of the foregoing steps S110 to S140 is cycled.

[0065] In step S160, in response to the round number i being equal to the preset round number N, the target predicted video frame is generated based on the first change frame, the second change frame, and the mask obtained in the i-th round.

[0066] In the embodiments of the present disclosure, the exit condition of the loop process is set as the preset round number N. In response to the current round number i being equal to the preset round number N, that is, the exit condition is met, the loop process is exited. Based on the first change frame Ii-1, the second change frame Ii-1, and the mask M The second change frame And the mask M i The target predicted video frame I2 is generated.

[0067] In the exemplary embodiments, the target predicted video frame can be obtained by weighted summation based on the first change frame Ii-1, the second change frame Ii-1, and the mask M The second change frame And the mask M i Specifically, the following formula can be referred to:

[0068]

[0069] Wherein, Ii-1 represents the first change frame obtained in the i-th round, Ii-1 represents the second change frame obtained in the i-th round, and I2 represents the target predicted video frame, M iMask obtained in the i-th round.

[0070] In the example embodiment, before generating the target prediction video frame based on the first change frame and the second change frame obtained in the i-th round, the first change frame and the second change frame also need to be subjected to a sigmoid operation. Based on the first change frame and the second change frame after the sigmoid operation and the mask, the target prediction video frame is generated.

[0071] In the example embodiment, in the video encoding process, the total number of video frames after the inserted video frames are supplemented should be consistent with the total number of video frames of the original video. Therefore, when the scheme provided by the present disclosure is used to simultaneously predict multiple inserted video frames based on the existing video frames, the total number of video frames of the original video may not be consistent due to the redundancy of the inserted video frames. Therefore, special processing is needed for the remaining frames at the tail of the video frames to ensure that the total number of video frames after encoding remains consistent. Specifically, the following steps can be included.

[0072] It is determined whether the number of generated target prediction video frames is greater than the number of remaining frames of the video to be encoded.

[0073] In response to the number of target prediction video frames being greater than the number of remaining frames, the target prediction video frames are discarded according to the number of remaining frames.

[0074] After the target prediction video frames are generated based on step S160, the number of generated target prediction video frames is compared with the number of remaining frames of the current video to be encoded. When the number of target prediction video frames is less than or equal to the number of remaining frames, the prediction of the inserted video frames is performed normally according to the method provided by the present disclosure. When the number of target prediction video frames is greater than the number of remaining frames, it indicates that there is redundancy in the generated target prediction video frames, and part of the generated target prediction video frames are discarded according to the number of remaining frames. For example, two target prediction video frames are generated based on three existing video frames, and the number of remaining frames of the current video to be encoded is one. Based on this, one of the two generated target prediction video frames is discarded according to the number of remaining frames, so that the total number of video frames after encoding remains consistent.

[0075] The video encoding method based on optical flow provided by the embodiment of the present disclosure gradually approaches the true value of the video frame to be predicted through the forward optical flow estimation module based on the multiple video frames in front of the video frame to be predicted as reference frames, captures the optical flow change characteristics through optical flow estimation, and gradually approaches the true value of the video frame to be predicted through the multi-round loop processing method, and finally generates the target prediction video frame. This method gradually approaches the true value of the video frame to be predicted through the multi-round loop optical flow estimation method, providing the accuracy of the prediction estimation. At the same time, only the forward video frames are used as reference frames, avoiding the influence of the introduction of backward reference frames on the encoding efficiency, meeting the needs of the low-latency time-domain encoding mode.

[0076] Figure 3 A flowchart of an optical flow estimation process of an embodiment of the present disclosure is shown. Figure 7 A result schematic diagram of a forward optical flow estimation module of an embodiment of the present disclosure is shown. As shown in Figure 3 , 7 based on the optical flow based video encoding method shown in Figure 1 , this step S120 can include the following steps.

[0077] In step S310, based on the scaling factor of the i-th round, the optical flow change module performs optical flow change estimation on the optical flow of the i-1-th round and the first feature to obtain optical flow change features.

[0078] In an embodiment of the present disclosure, as shown in Figure 7 , the forward optical flow estimation module includes an optical flow change module and a spatial change module. The optical flow change module, based on the scaling factor of the i-th round, performs optical flow change estimation on the optical flow F i-1 of the i-1-th round and the first feature X1 input to the forward optical flow estimation module to obtain optical flow change features X f . The optical flow change estimation is to predict the change of the optical flow field by analyzing the optical flow field and the feature map of the previous round, given the scaling factor of the current round. The optical flow change features X f are used to represent the related features of the optical flow change between the video frames.

[0079] In step S320, the spatial change module performs spatial change estimation on the optical flow of the i-1-th round and the first feature to obtain spatial change features.

[0080] In an embodiment of the present disclosure, as shown in Figure 7 , the forward optical flow estimation module includes an optical flow change module and a spatial change module. The spatial change module, based on the scaling factor of the i-th round, performs spatial change estimation on the optical flow F i-1 of the i-1-th round and the first feature X1 input to the forward optical flow estimation module to obtain spatial change features X s . The spatial change estimation is to predict the change of the image in the spatial domain by analyzing the optical flow and the feature map of the previous round, given the scaling factor of the current round. The spatial change features X s are used to represent the related features of the spatial change between the video frames.

[0081] In step S330, the optical flow change features and the spatial change features are spliced to obtain optical flow estimation features.

[0082] In an embodiment of the present disclosure, the optical flow change features X f and the spatial change features Xs The splicing is performed to obtain the optical flow estimation feature X ffe .

[0083] In an exemplary embodiment, the optical flow change feature X f and the spatial change feature X s are spliced based on feature channels. The feature channels are used to represent different dimensional feature information in a video frame.

[0084] In step S340, the optical flow estimation feature is deconvoluted to obtain residual information of the optical flow and the mask of the i-th round.

[0085] In the embodiments of the present disclosure, the optical flow estimation feature X ffe is deconvoluted to obtain residual information of the optical flow and the mask of the i-th round, i.e., ΔF i and ΔM i . As described above, the optical flow estimation feature X ffe is generated based on feature channel splicing. Therefore, after the optical flow estimation feature X ffe is deconvoluted, the residual information of the optical flow ΔF i and the residual information of the mask ΔM i can be split based on the correspondence of the feature channels.

[0086] Figure 4 A flowchart of an optical flow change estimation process of the embodiments of the present disclosure is shown. Figure 7 A result schematic diagram of a forward optical flow estimation module of the embodiments of the present disclosure is shown. As shown in Figure 4 , 7 , on the basis of the optical flow-based video encoding method shown in Figure 3 , the step S310 can include the following steps.

[0087] In step S410, the optical flow of the i-1-th round and the first feature are spliced to obtain a second feature.

[0088] In the embodiments of the present disclosure, as shown in Figure 7 , the optical flow F i-1 of the i-1-th round input into the forward optical flow estimation module and the first feature X1 are spliced to obtain a second feature X2.

[0089] In an exemplary embodiment, as shown in Figure 7 , in order to improve the efficiency of data processing, the optical flow F i-1 and the first feature X1 are respectively down-sampled before the above-mentioned feature splicing is performed to obtain down-sampled optical flow and first feature. The down-sampled optical flow and the first feature are spliced to obtain the second feature X2.

[0090] In an example embodiment, the sampling rate of the downsampling processing is Scale_list[i]. The sampling rate of the downsampling processing is associated with the scaling factor of the current round, and the sampling rate is adjusted according to the scaling factor.

[0091] In step S420, a channel alignment operation is performed on the second feature to obtain a third feature.

[0092] In the embodiments of the present disclosure, as shown in Figure 7 a channel alignment operation is performed on the second feature X2 to obtain a third feature X3. The channel alignment operation refers to adjusting the number of feature channels of the feature to align with a target number of channels. The target number of channels may be different according to the needs of subsequent processing. The specific target number of channels is not limited herein. Meanwhile, the channel alignment operation is a conventional technical means in the art, which will not be further introduced herein.

[0093] In an example embodiment, at least one convolution operation is performed on the second feature to obtain a third feature aligned with a preset number of channels. In this example, the channel alignment operation is achieved by multiple convolution operations. For example, stride = 2, and M is a set number of feature channels. Therefore, two convolution operations are performed on the second feature to obtain a third feature aligned with the set number of feature channels M. After the first convolution operation, an M / 2 channel feature is output. After the second convolution operation, an M channel feature is output.

[0094] In step S430, a residual feature learning module is used to perform optical flow feature learning on the third feature to obtain the optical flow change feature.

[0095] In the embodiments of the present disclosure, as shown in Figure 7 the residual feature learning module is used to perform optical flow feature learning on the third feature X3 to obtain an optical flow change feature X f . The residual feature learning module is a network module for learning residual information from input features, aiming to capture the difference between the input features and the target features. For example, ResBlock network. Optical flow feature learning refers to a process of estimating pixel-level optical flow motion vectors by analyzing image sequences.

[0096] In an example embodiment, as shown in Figure 7 corresponding to the downsampling operation in the aforementioned step S410, the residual feature learning module is used to perform optical flow feature learning on the third feature to obtain the optical flow change feature, including:

[0097] performing optical flow feature learning on the third feature by using the residual feature learning module to obtain a fourth feature;

[0098] performing a channel alignment operation on the fourth feature to obtain a fifth feature;

[0099] Upsampling the fifth feature to obtain the optical flow change feature.

[0100] In an exemplary embodiment, the optical flow feature learning on the third feature by the residual feature learning module first obtains a fourth feature X4. Then, a channel alignment operation is performed on the fourth feature X4 to obtain a fifth feature X5. Finally, an upsampling operation is performed on the fifth feature X5 to obtain the optical flow change feature X f . The upsampling process is the inverse process of the aforementioned downsampling in step S410. Therefore, the sampling rate of the upsampling process is the same as the aforementioned downsampling, which is Scale_list[i]. Similarly, the sampling rate of the upsampling process is also associated with the scaling factor of the current round, and the sampling rate is adjusted according to the scaling factor.

[0101] Figure 5 A flowchart of a spatial variation estimation process of an embodiment of the present disclosure is shown. Figure 7 A result schematic diagram of a forward optical flow estimation module of an embodiment of the present disclosure is shown. As shown in Figure 5 、 7 Based on the optical flow-based video encoding method shown in Figure 3 , the step S320 can include the following steps.

[0102] In step S510, the (i-1)th round optical flow and the first feature are spliced to obtain a sixth feature.

[0103] In an embodiment of the present disclosure, as shown in Figure 7 , the (i-1)th round optical flow F i-1 and the first feature X1 input into the forward optical flow estimation module are spliced to obtain a sixth feature X6. The sixth feature X6 can be the same as the aforementioned second feature X2, or can be different from the second feature X2 due to different splicing methods, which is not limited here.

[0104] In step S520, a channel alignment operation is performed on the sixth feature to obtain a seventh feature.

[0105] In an embodiment of the present disclosure, based on the need for channel alignment, a channel alignment operation is performed on the sixth feature X6 to obtain a seventh feature X7.

[0106] In step S530, a spatial attention learning is performed on the seventh feature by a group-based spatial attention (GSA) module to obtain a spatial attention feature.

[0107] In the embodiments of the present disclosure, the seventh feature X7 is subjected to spatial attention learning through a pre-designed grouping spatial attention module, so as to obtain a spatial attention feature X A .

[0108] The grouping spatial attention module is a module for extracting attention features of different scales in the spatial domain of a video frame. Different sizes of the hole convolution are applied to the module to extract features of different scales in the spatial domain of the video frame. Through the extraction of features of different scales, the ability to capture different levels of details is enhanced, so that the spatial change features of the video frame are better captured. The spatial attention feature X A obtained is used to represent the change features of the video frame in the spatial domain.

[0109] In the example embodiments, different weights can be added to the grouping spatial attention modules in different rounds according to the foregoing cycle process. The weights are associated with the foregoing scaling factors. According to the scaling factors corresponding to the cycle rounds, the grouping spatial attention modules with the same scaling factor have the same weight, and the grouping spatial attention modules with different scaling factors have different weights.

[0110] In step S540, the spatial attention feature is subjected to feature learning, so as to obtain the spatial change feature.

[0111] In the embodiments of the present disclosure, the spatial attention feature X A is subjected to feature learning, so as to obtain the spatial change feature X s .

[0112] In the example embodiments, the spatial attention feature X A is subjected to feature learning through at least one convolution operation. Specifically, the spatial attention feature X A is subjected to convolution operation through a 1X1 convolution kernel, so as to obtain the spatial change feature X s .

[0113] Figure 6 A structural schematic diagram of the grouping spatial attention module in the embodiments of the present disclosure is shown. As Figure 6 shown, based on the optical flow-based video encoding method shown in Figure 5 , the step S530 can include the following steps.

[0114] Step 1, the seventh feature is subjected to first hole convolution operation, so as to obtain an eighth feature.

[0115] In the embodiments of the present disclosure, the seventh feature X7 is subjected to first hole convolution operation, so as to obtain an eighth feature X8.

[0116] Dilated convolution is a convolution technique proposed to address the problem of image semantic segmentation, where downsampling reduces image resolution and causes information loss. It introduces a new parameter called the dilation rate, which defines the spacing between values ​​when the convolution kernel processes data. The dilation rate determines the spacing between elements in the convolution kernel, thereby increasing its receptive field (i.e., the area covered by the convolution kernel) while maintaining the same kernel size.

[0117] In an exemplary embodiment, the dilated convolution operation can be adjusted by multiple parameters. The adjustment of the dilated convolution operation can be represented by DWConv(a,b,c,d). Where a represents the number of input feature channels, b represents the number of output feature channels, c represents the convolution kernel size, and d represents the step size. In addition, Dilation represents the dilation rate of the dilated convolution. Figure 6 As shown, in this example, the first dilated convolution is set to DWConv(C, C, 3, 1), with Dilation = 1.

[0118] Step 2: Divide the eighth feature into at least one channel feature according to the channel; perform a dilated convolution operation on the at least one channel feature to obtain at least one channel space feature.

[0119] In the embodiment of the present disclosure, the eighth feature X8 is divided into a plurality of different channel features X according to the feature channel. C . For the divided multiple channel features X C Then perform the hole convolution operation separately to obtain the features X of each channel C Corresponding multiple channel space features X DW .

[0120] In an exemplary embodiment, during the channel feature division process, the eighth feature X8 may be divided into at least one channel feature X according to the number of channels according to a preset ratio of the number of channels. C .like Figure 6 As shown, in this example, the eighth feature X8 is evenly divided into four channel features X according to the number of channels. C1 、X C2 、X C3 、X C4 For example, the eighth feature X8 includes 16 feature channels, and the features of channels 0-3 are divided into channel features X C1 , divide the features of 4-7 channels into channel features X C2 , divide the features of channels 8-11 into channel features X C3 , divide the features of channels 12-15 into channel features X C4Of course, according to actual application needs, the number of channels of each channel feature can be divided not evenly, but according to the preset channel number ratio. For example, the eighth feature X8 includes 16 feature channels, which are divided into three channel features according to 2:1:1. Then the features of channels 0-7 are divided into channel features X C1 , divide the features of channels 8-11 into channel features X C2 , divide the features of channels 12-15 into channel features X C3 .

[0121] In an exemplary embodiment, as mentioned above, the dilated convolution operation can be adjusted by multiple parameters. Based on this, in this example, the convolution kernel sizes corresponding to the dilated convolution operation are different for different channel features, thereby performing dilated convolution operations of different scales. Figure 6 As shown, in this example, for the channel feature X C1 The feature is retained without performing the hole convolution operation, and the channel space feature X is obtained. DW1 ; For channel feature X C2 The convolution kernel scale corresponding to the dilated convolution operation is 3X3, Dilation=3, and the channel space feature X is obtained. DW2 ; For channel feature X C3 The convolution kernel scale corresponding to the dilated convolution operation is 5X5, Dilation=3, and the channel space feature X is obtained. DW3 ; For channel feature X C4 The convolution kernel scale corresponding to the dilated convolution operation is 7X7, Dilation=3, and the channel space feature X is obtained. DW4 By applying dilated convolutions of different sizes, we can extract features of different scales in the spatial domain of video frames, thereby enhancing the ability to capture different levels of detail and better capturing the spatial variation characteristics of video frames.

[0122] It should be pointed out that not performing a dilated convolution operation on channel features to retain the original features should also be regarded as a special dilated convolution operation, as one of the different dilated convolution operations available.

[0123] Step 3: splice the at least one channel space feature to obtain a ninth feature.

[0124] In the embodiment of the present disclosure, the multiple channel space features X obtained above are DW After splicing, we get the ninth feature X9.

[0125] Step 4: Perform feature learning on the seventh feature and the ninth feature respectively to obtain the tenth feature and the thirteenth feature.

[0126] In the embodiments of the present disclosure, the seventh feature X7 and the ninth feature X9 are respectively subjected to feature learning to obtain a tenth feature X 10 and a thirteenth feature X 13 . As shown in Figure 6 , the feature learning can be performed by one convolutional layer. The convolution operation can be set as Conv(C, C, 3, 1).

[0127] Step 5, the tenth feature and the thirteenth feature are subjected to activation processing, and the tenth feature and the thirteenth feature after the activation processing are subjected to point multiplication to obtain an eleventh feature.

[0128] In the embodiments of the present disclosure, the tenth feature X 10 and the thirteenth feature X 13 are subjected to activation processing, and the tenth feature X 10 and the thirteenth feature X 13 after the activation processing are subjected to point multiplication to obtain an eleventh feature X 11 .

[0129] The activation processing can be processed by different activation functions such as Sigmoid, ReLU or SiLU.

[0130] Step 6, the eleventh feature is subjected to feature learning to obtain a twelfth feature.

[0131] In the embodiments of the present disclosure, the eleventh feature X 11 is subjected to feature learning to obtain a twelfth feature X 12 . As shown in Figure 6 , the feature learning can be performed by one convolutional layer. The convolution operation can be set as Conv(C, C, 3, 1).

[0132] Step 7, the seventh feature and the twelfth feature are subjected to feature addition to obtain the spatial attention feature.

[0133] In the embodiments of the present disclosure, the seventh feature X7 and the twelfth feature X 12 are subjected to feature addition to obtain a spatial attention feature X A .

[0134] The method for video coding based on optical flow provided by the embodiments of the present disclosure designs an optical flow change module and a spatial change module in the forward optical flow estimation module. The optical flow change module is used to capture the optical flow change features between reference frames. The spatial change module is used to capture the spatial change features between reference frames. By combining the optical flow change features and the spatial change features, the change features between reference frames can be better captured, so that the prediction of future video frames can be accurately performed. In the optical flow change module, a scaling factor is introduced, and the optical flow change features are captured through a residual feature learning module. In the spatial change module, different sizes of hole convolution are applied to extract features of different scales in the spatial domain of the video frames, so that the capture ability of different detail levels is enhanced, and the spatial change features of the video frames can be better captured.

[0135] Based on the same inventive concept, the embodiments of the present disclosure provide a device for video coding based on optical flow, as described in the following embodiments. Since the principle of solving problems of the device embodiments is similar to the above-mentioned method embodiments, the implementation of the real-time communication authentication device embodiments can be referred to the implementation of the above-mentioned method embodiments, and the repeated parts will not be described here.

[0136] Figure 8 The structure schematic diagram of a device for video coding based on optical flow is shown. As shown in the figure, the device for video coding based on optical flow 800 can include a video frame splicing module 810, a forward optical flow estimation module 820, a fusion module 830, a change processing module 840 and a predicted frame module 850. Figure 8 The video frame splicing module 810 is configured to splice the first video frame, the second video frame, the first change frame and the second change frame obtained by the i-1th round of change processing, and the mask to obtain the first feature; wherein the first change frame and the second change frame are obtained based on the change processing of the first video frame and the second video frame respectively.

[0137] The forward optical flow estimation module 820 is configured to perform optical flow estimation on the optical flow of the i-1th round and the first feature through the forward optical flow estimation module based on the scaling factor of the i th round, to obtain the optical flow of the i th round and the residual information of the mask.

[0138] The fusion module 830 is configured to fuse the optical flow of the i-1th round, the mask and the residual information of the optical flow and the mask of the i th round to obtain the optical flow of the i th round and the mask.

[0139] The change processing module 840 is configured to perform change processing on the first video frame and the second video frame based on the optical flow of the i th round to obtain the first change frame and the second change frame of the i th round.

[0140]

[0141] ​The prediction frame module 850 is configured to, in response to the wheel number i not being equal to the preset wheel number N, add 1 to the wheel number i, and cycle the foregoing steps; and in response to the wheel number i being equal to the preset wheel number N, generate a target prediction video frame based on the first change frame, the second change frame and the mask obtained in the i-th wheel.

[0142] In an example embodiment, the first video frame, the second video frame and the target prediction video frame are time-continuous video frames.

[0143] In an example embodiment, the scaling factor is associated with the wheel number i.

[0144] In an example embodiment, the prediction frame module 850 is further configured to generate the target prediction video frame by weighted summation based on the first change frame, the second change frame and the mask obtained in the i-th wheel, and the corresponding formula is as follows:

[0145]

[0146] wherein, denotes the first change frame obtained in the i-th wheel, denotes the second change frame obtained in the i-th wheel, and I2 denotes the target prediction video frame, and M i denotes the mask obtained in the i-th wheel.

[0147] In an example embodiment, the forward optical flow estimation module 820 is further configured to, based on the scaling factor of the i-th wheel, perform optical flow change estimation on the optical flow and the first feature of the i-1-th wheel by an optical flow change module to obtain an optical flow change feature; perform spatial change estimation on the optical flow and the first feature of the i-1-th wheel by a spatial change module to obtain a spatial change feature; splice the optical flow change feature and the spatial change feature to obtain an optical flow estimation feature; and perform deconvolution operation on the optical flow estimation feature to obtain residual information of the optical flow and the mask of the i-th wheel.

[0148] In an example embodiment, the forward optical flow estimation module 820 is further configured to splice the optical flow and the first feature of the i-1-th wheel to obtain a second feature; perform channel alignment operation on the second feature to obtain a third feature; and perform optical flow feature learning on the third feature by a residual feature learning module to obtain the optical flow change feature.

[0149] In an example embodiment, the forward optical flow estimation module 820 is further configured to down-sample the optical flow and the first feature of the i-1th round to obtain a down-sampled optical flow and a down-sampled first feature; and stitch the down-sampled optical flow and the down-sampled first feature to obtain the second feature; and the optical flow feature learning on the third feature by the residual feature learning module to obtain the optical flow change feature comprises: performing optical flow feature learning on the third feature by the residual feature learning module to obtain a fourth feature; performing channel alignment on the fourth feature to obtain a fifth feature; and performing up-sampling on the fifth feature to obtain the optical flow change feature; wherein the sampling rates of the down-sampling and the up-sampling are associated with the scaling factor.

[0150] In an example embodiment, the forward optical flow estimation module 820 is further configured to perform at least one convolution operation on the second feature to obtain the third feature aligned with a preset channel number.

[0151] In an example embodiment, the forward optical flow estimation module 820 is further configured to stitch the optical flow and the first feature of the i-1th round to obtain a sixth feature; perform channel alignment on the sixth feature to obtain a seventh feature; perform spatial attention learning on the seventh feature by a group-based spatial attention (GSA) module to obtain the spatial attention feature; and perform feature learning on the spatial attention feature to obtain the spatial change feature.

[0152] In an example embodiment, the forward optical flow estimation module 820 is further configured to perform a first dilated convolution operation on the seventh feature to obtain an eighth feature; divide the eighth feature into at least one channel feature according to channels; perform dilated convolution operations on the at least one channel feature respectively to obtain at least one channel spatial feature; stitch the at least one channel spatial feature to obtain a ninth feature; perform feature learning on the seventh feature and the ninth feature respectively to obtain a tenth feature and a thirteenth feature; perform activation processing on the tenth feature and the thirteenth feature, and perform point multiplication on the activated tenth feature and the thirteenth feature to obtain an eleventh feature; perform feature learning on the eleventh feature to obtain a twelfth feature; and perform feature addition on the seventh feature and the twelfth feature to obtain the spatial attention feature.

[0153] In an example embodiment, the forward optical flow estimation module 820 is further configured to divide the eighth feature into the at least one channel feature according to a preset channel number ratio.

[0154] In an example embodiment, the forward optical flow estimation module 820 is further configured to have different kernel sizes for the deconvolution operation corresponding to different channel features.

[0155] In an example embodiment, the predicted frame module 850 is further configured to determine whether a number of generated target predicted video frames is greater than a number of remaining frames of the video to be encoded; and in response to the number of target predicted video frames being greater than the number of remaining frames, discard the target predicted video frames according to the number of remaining frames.

[0156] Figure 9 A structural schematic diagram of an electronic device suitable for implementing the example embodiments of the present disclosure is shown. The electronic device 900 according to this implementation of the present disclosure is described below with reference to Figure 9 Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0157] As shown in Figure 9 , the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 can include, but are not limited to, the at least one processing unit 910 described above, the at least one storage unit 920 described above, a bus 930 connecting different system components, including the storage unit 920 and the processing unit 910, and a display unit 940.

[0158] The storage unit 920 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 9201 and / or a cache memory unit 9202, and can further include a read-only memory (ROM) 9203.

[0159] The storage unit 920 can further include a program / utility 9204 having a set of program modules 9205, including but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which can include implementation of a network environment, alone or in some combination.

[0160] The bus 930 can represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit bus using any of a variety of bus structures, or a local bus.

[0161] ​The electronic device 900 can also communicate with one or more external devices 970 such as a keyboard or pointing device, using one or more communication interfaces 950. Communication interfaces 950 can include, for example, a modem, a network interface card (e.g. an Ethernet card), a television

[0162] In the example embodiments of the present disclosure, a computer readable storage medium is also provided, on which a program product capable of implementing the above-mentioned method of the present specification is stored.

[0163] In some possible implementation manners, various aspects of the present application can also be implemented as a program product in the form of a program code, which is used to cause the terminal device to perform the steps described in the above-mentioned "example method" section of the present specification according to various example embodiments of the present application when the program product is run on the terminal device.

[0164] The program product for implementing the above-mentioned method according to the embodiments of the present application can be in the form of a portable compact disc read-only memory (CD-ROM) and include program codes, and can be run on a terminal device such as a personal computer. However, the program product of the present application is not limited to this, and in the present document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in conjunction with a signaling execution system, device or apparatus.

[0165] The program product can take any combination of one or more computer-readable media. The computer-readable media can be a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0166] The computer-readable signal medium can include a computer-readable storage medium that is configured to store and deliver a computer-readable program code. The computer-readable program code can be propagated as a computer-readable signal medium.

[0167] The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the foregoing.

[0168] The program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, etc., or conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0169] It should be noted that, although several modules or units of the device for action execution are mentioned in the foregoing detailed description, such a division is not mandatory. Indeed, according to an embodiment of the present disclosure, features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, features and functions of one module or unit described above can be further divided into a plurality of modules or units.

[0170] Moreover, although individual steps of the methods in the present disclosure are described in a particular order in the drawings, this is not required or implied as to the order of the steps, nor is it required that all of the steps be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, one step can be broken into multiple steps, etc.

[0171] From the above description of the embodiments, those skilled in the art will readily perceive that the example embodiments described herein can be implemented by software and / or by hardware coupled with software. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or on a network, and includes a number of instructions for causing a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0172] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure that come within the scope of the following claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0173] It should be understood that the present disclosure is not limited to the precise structures described and shown in the drawings, and that various modifications and changes can be made without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A method for optical flow based video frame interpolation, characterized in that, The method comprises: splicing the first video frame, the second video frame, and the first change frame, the second change frame, and the mask obtained in the i-1th round to obtain a first feature; wherein the first change frame and the second change frame are respectively obtained by change processing based on the first video frame and the second video frame; based on the scaling factor of the i th round, performing optical flow estimation on the optical flow of the i-1th round and the first feature by a forward optical flow estimation module to obtain residual information of the optical flow and the mask of the i th round; fusing the optical flow, the mask obtained in the i-1th round, and the residual information of the optical flow and the mask of the i th round to obtain the optical flow and the mask of the i th round; based on the optical flow of the i th round, performing change processing on the first video frame and the second video frame to obtain the first change frame and the second change frame of the i th round; in response to the number of rounds i not being equal to the preset number of rounds N, increasing the number of rounds i by 1, and repeating the foregoing steps; in response to the number of rounds i being equal to the preset number of rounds N, generating a target predicted video frame based on the first change frame, the second change frame, and the mask obtained in the i th round.

2. The method of claim 1, wherein, The first video frame, the second video frame, and the target predicted video frame are time-continuous video frames.

3. The method of claim 1, wherein, The scaling factor is associated with the number of rounds i.

4. The method of claim 1, wherein, The generating of the target predicted video frame based on the first change frame, the second change frame, and the mask obtained in the i th round comprises: performing weighted summation on the first change frame, the second change frame, and the mask obtained in the i th round to generate the target predicted video frame, and the corresponding formula is as follows: wherein, denotes the first changed frame obtained in the i-th round, denotes the second changed frame obtained in the i-th round, I2 denotes the target predicted video frame, M i denotes the mask obtained in the i-th round.

5. The method of claim 1, wherein, The performing of optical flow estimation on the optical flow of the i-1th round and the first feature by the forward optical flow estimation module based on the scaling factor of the i th round to obtain the residual information of the optical flow and the mask of the i th round comprises: based on the scaling factor of the i th round, performing optical flow change estimation on the optical flow of the i-1th round and the first feature by an optical flow change module to obtain an optical flow change feature; performing spatial change estimation on the optical flow of the i-1th round and the first feature by a spatial change module to obtain a spatial change feature; splicing the optical flow change feature and the spatial change feature to obtain an optical flow estimation feature; performing deconvolution operation on the optical flow estimation feature to obtain the residual information of the optical flow and the mask of the i th round.

6. The method of claim 5, wherein, The performing of optical flow change estimation on the optical flow of the i-1th round and the first feature by the optical flow change module based on the scaling factor of the i th round to obtain the optical flow change feature comprises: splicing the optical flow of the i-1th round and the first feature to obtain a second feature; performing channel alignment operation on the second feature to obtain a third feature; performing optical flow feature learning on the third feature by a residual feature learning module to obtain the optical flow change feature.

7. The method of claim 6, wherein the splicing of the optical flow of the i-1th round and the first feature to obtain the second feature comprises: performing downsampling operation on the optical flow of the i-1th round and the first feature to obtain downsampled optical flow and first feature; splicing the downsampled optical flow and the first feature to obtain the second feature; The optical flow feature learning on the third feature through the residual feature learning module comprises: The optical flow feature learning on the third feature through the residual feature learning module comprises: The channel alignment operation on the fourth feature comprises: The up-sampling operation on the fifth feature comprises: The sampling rate of the down-sampling and the up-sampling is associated with the scaling factor.

8. The method of claim 6, wherein, The channel alignment operation on the second feature comprises: The at least one convolution operation on the second feature comprises:

9. The method of claim 5, wherein, The spatial change estimation on the optical flow and the first feature of the i-1th round through the spatial change module comprises: The channel alignment operation on the sixth feature comprises: The spatial attention learning on the seventh feature through a group-based spatial attention (GSA) module comprises: The spatial change estimation on the optical flow and the first feature of the i-1th round through the spatial change module comprises: The spatial attention learning on the seventh feature through a group-based spatial attention (GSA) module comprises:

10. The method of claim 9, wherein, The spatial change estimation on the optical flow and the first feature of the i-1th round through the spatial change module comprises: The first hole convolution operation on the seventh feature comprises: The at least one channel spatial feature is obtained by performing a hole convolution operation on the at least one channel feature, respectively. The at least one channel spatial feature is obtained by performing a hole convolution operation on the at least one channel feature, respectively. The activation processing on the tenth feature and the thirteenth feature comprises: The feature addition on the seventh feature and the twelfth feature comprises: The channel number ratio of the eighth feature is divided into the at least one channel feature according to a preset channel number ratio. The hole convolution operation on the at least one channel feature is different in the kernel size.

11. The method of claim 10, wherein, Further comprising:

12. The method of claim 10, wherein, It is judged whether the number of the generated target prediction video frames is greater than the remaining frame number of the video to be encoded.

13. The method of claim 1, wherein, In response to the number of the target prediction video frames being greater than the remaining frame number, the target prediction video frames are discarded according to the remaining frame number. Further comprising: ​ 14. An optical flow based video frame interpolation apparatus, characterized by, ​ a video frame splicing module configured to splice the first video frame, the second video frame, the first changed frame, the second changed frame and the mask obtained in the i-1th round to obtain first features; wherein the first changed frame and the second changed frame are obtained based on change processing of the first video frame and the second video frame, respectively; a forward optical flow estimation module configured to, based on a scaling factor in the i th round, perform optical flow estimation on the optical flow in the i-1th round and the first features by the forward optical flow estimation module to obtain optical flow in the i th round and residual information of the mask; a fusion module configured to fuse the optical flow, the mask obtained in the i-1th round and the residual information of the optical flow and the mask in the i th round to obtain optical flow and a mask in the i th round; a change processing module configured to perform change processing on the first video frame and the second video frame based on the optical flow in the i th round to obtain a first changed frame and a second changed frame in the i th round; a predicted frame module configured to, in response to the number of rounds i not being equal to a preset number of rounds N, add 1 to the number of rounds i and cycle the foregoing steps; and in response to the number of rounds i being equal to the preset number of rounds N, generate a target predicted video frame based on the first changed frame, the second changed frame and the mask obtained in the i th round.

15. An electronic device, comprising: comprising: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1 to 13.

16. A computer-readable storage medium, the computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 13.

17. A computer program product comprising a computer program / signalling, characterized in that, The computer program / signaling is executed by the processor to implement the method of any one of claims 1 to 13.

Images