Video coding method and device based on optical flow, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video coding method and device based on optical flow, equipment, a medium and a program product. According to the method, a plurality of forward video frames of a to-be-predicted video frame are taken as reference frames through a forward optical flow estimation module, optical flow change characteristics of the reference frames are captured through optical flow estimation, a true value of the to-be-predicted video frame is gradually approached through a multi-round circular processing mode, and finally a target prediction video frame is generated. According to the method, the true value of the video frame to be predicted is gradually approached through a multi-round cyclic optical flow estimation method, and the accuracy of prediction and estimation is improved. Meanwhile, only a forward video frame is used as a reference frame, so that the influence of introduction of a backward reference frame on the coding efficiency is avoided, and the requirement of a low-delay time domain coding mode is met.

Description

Technical Field

[0001] The present disclosure relates to the field of video coding and decoding technology, and in particular to a video coding method, device, electronic device, storage medium and computer program product based on optical flow. Background Art

[0002] The Video Coding Standards Organization has established a Video Coding for Machines (VCM) working group to study video coding for machine processing. At present, the time domain interpolation network adopted by the working group is a method of predicting and generating intermediate frames based on the time domain interpolation network of forward and backward reference frames. This method performs well in the interpolation effect, can effectively improve the frame rate and smoothness of the video, and is suitable for a variety of video processing scenarios. However, this bidirectional reference frame method has certain limitations, especially in the low delay (LowDelay, LD) mode, because it needs to rely on future frames as references, which makes it impossible to meet application scenarios with high real-time requirements. Summary of the invention

[0003] Embodiments of the present disclosure provide a video encoding method, device, electronic device, storage medium, and computer program product based on optical flow.

[0004] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0005] According to one aspect of the present disclosure, a video encoding method based on optical flow is provided, the method comprising: splicing a first video frame, a second video frame, a first change frame, a second change frame and a mask obtained in an i-1th round to obtain a first feature; wherein the first change frame and the second change frame are obtained based on the change processing of the first video frame and the second video frame respectively; based on the scaling factor of the i-th round, performing optical flow estimation on the optical flow of the i-1th round and the first feature through a forward optical flow estimation module to obtain residual information of the optical flow and the mask of the i-th round; The optical flow and mask obtained in the i-1th round and the residual information of the optical flow and mask in the i-th round are fused to obtain the optical flow and mask in the i-th round; the first video frame and the second video frame are changed based on the optical flow in the i-th round to obtain the first changed frame and the second changed frame in the i-th round; in response to the round number i not being equal to the preset round number N, the round number i is increased by 1, and the above steps are repeated; in response to the round number i being equal to the preset round number N, a target predicted video frame is generated based on the first changed frame, the second changed frame and the mask obtained in the i-th round.

[0006] In an exemplary embodiment, the first video frame, the second video frame and the target prediction video frame are temporally consecutive video frames.

[0007] In an exemplary embodiment, the scaling factor is associated with the round number i.

[0008] In an exemplary embodiment, the generating of a target predicted video frame based on the first changed frame, the second changed frame and the mask obtained in the i-th round includes:

[0009] Based on the weighted sum of the first change frame, the second change frame and the mask obtained in the i-th round, the target predicted video frame is generated. The corresponding formula is as follows:

[0010]

[0011] in, represents the first change frame obtained in the i-th round, represents the second change frame obtained in the i-th round, I 2 represents the target predicted video frame, M i Represents the mask obtained in the i-th round.

[0012] In an exemplary embodiment, the optical flow estimation is performed on the optical flow of the i-1th round and the first feature through a forward optical flow estimation module based on the scaling factor of the i-th round to obtain residual information of the optical flow and mask of the i-th round, including: based on the scaling factor of the i-th round, the optical flow change estimation is performed on the optical flow and the first feature of the i-1th round through an optical flow change module to obtain an optical flow change feature; the spatial change estimation is performed on the optical flow and the first feature of the i-1th round through a spatial change module to obtain a spatial change feature; the optical flow change feature and the spatial change feature are spliced ​​to obtain an optical flow estimation feature; and a deconvolution operation is performed on the optical flow estimation feature to obtain residual information of the optical flow and mask of the i-th round.

[0013] In an exemplary embodiment, based on the scaling factor of the i-th round, an optical flow change estimation is performed on the optical flow of the i-1th round and the first feature through an optical flow change module to obtain an optical flow change feature, including: splicing the optical flow of the i-1th round and the first feature to obtain a second feature; performing a channel alignment operation on the second feature to obtain a third feature; and performing optical flow feature learning on the third feature through a residual feature learning module to obtain the optical flow change feature.

[0014] In an exemplary embodiment, the step of splicing the optical flow of the i-1th round and the first feature to obtain the second feature comprises: performing a downsampling operation on the optical flow of the i-1th round and the first feature to obtain the downsampled optical flow and the first feature; splicing the downsampled optical flow and the first feature to obtain the second feature; the step of performing optical flow feature learning on the third feature through the residual feature learning module to obtain the optical flow change feature comprises: performing optical flow feature learning on the third feature through the residual feature learning module to obtain a fourth feature; performing a channel alignment operation on the fourth feature to obtain a fifth feature; and performing an upsampling operation on the fifth feature to obtain the optical flow change feature; wherein the sampling ratios of the downsampling and upsampling are associated with the scaling factor.

[0015] In an exemplary embodiment, performing a channel alignment operation on the second feature to obtain a third feature includes: performing at least one convolution operation on the second feature to obtain the third feature aligned with a preset number of channels.

[0016] In an exemplary embodiment, the spatial variation module performs spatial variation estimation on the optical flow and the first feature of the i-1th round to obtain the spatial variation feature, including: splicing the optical flow and the first feature of the i-1th round to obtain the sixth feature; performing channel alignment on the sixth feature to obtain the seventh feature; performing spatial attention learning on the seventh feature through a group-based spatial attention module (Group-based Spatial Attention, GSA) to obtain the spatial attention feature; and performing feature learning on the spatial attention feature to obtain the spatial variation feature.

[0017] In an exemplary embodiment, the spatial attention learning of the seventh feature through the grouped spatial attention module to obtain the spatial attention feature includes: performing a first dilated convolution operation on the seventh feature to obtain an eighth feature; dividing the eighth feature into at least one channel feature according to the channel; performing a dilated convolution operation on the at least one channel feature to obtain at least one channel spatial feature; concatenating the at least one channel spatial feature to obtain a ninth feature; performing feature learning on the seventh feature and the ninth feature to obtain a tenth feature and a thirteenth feature respectively; activating the tenth feature and the thirteenth feature, and performing dot multiplication on the activated tenth feature and the thirteenth feature to obtain an eleventh feature; performing feature learning on the eleventh feature to obtain a twelfth feature; and performing feature addition on the seventh feature and the twelfth feature to obtain the spatial attention feature.

[0018] In an exemplary embodiment, dividing the eighth feature into at least one channel feature according to channels includes: dividing the eighth feature into the at least one channel feature according to the number of channels according to a preset ratio of the number of channels.

[0019] In an exemplary embodiment, performing a dilated convolution operation on the at least one channel feature to obtain at least one channel space feature further includes: the convolution kernel sizes of the dilated convolution operations corresponding to different channel features are different.

[0020] In an exemplary embodiment, the method further includes: determining whether the number of generated target prediction video frames is greater than the number of remaining frames of the video to be encoded; in response to the number of target prediction video frames being greater than the number of remaining frames, discarding the target prediction video frames according to the number of remaining frames.

[0021] According to another aspect of the present disclosure, a video encoding device based on optical flow is provided, including: a video frame splicing module, configured to splice a first video frame, a second video frame, a first change frame, a second change frame and a mask obtained in the i-1th round to obtain a first feature; wherein the first change frame and the second change frame are obtained based on the change processing of the first video frame and the second video frame respectively; a forward optical flow estimation module, configured to perform optical flow estimation on the optical flow of the i-1th round and the first feature through the forward optical flow estimation module based on the scaling factor of the i-th round to obtain residual information of the optical flow of the i-1th round and the mask; a fusion module The block is configured to fuse the optical flow and mask obtained in the i-1th round and the residual information of the optical flow and mask in the i-th round to obtain the optical flow and mask in the i-th round; the change processing module is configured to perform change processing on the first video frame and the second video frame based on the optical flow in the i-th round to obtain the first change frame and the second change frame in the i-th round; the prediction frame module is configured to, in response to the round number i not being equal to the preset round number N, increase the round number i by 1 and repeat the above steps; in response to the round number i being equal to the preset round number N, generate the target predicted video frame based on the first change frame, the second change frame and the mask obtained in the i-th round.

[0022] According to another aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a storage device configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the optical flow-based video encoding method as described in the above embodiments.

[0023] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the optical flow-based video encoding method as described in the above embodiments is implemented.

[0024] According to another aspect of the present disclosure, a computer program product is provided, including a computer program / signaling, characterized in that when the computer program / signaling is executed by a processor, the optical flow-based video encoding method as described in the above embodiments is implemented.

[0025] The optical flow-based video encoding method provided by the embodiment of the present disclosure uses a forward optical flow estimation module to use multiple video frames forward of the video frame to be predicted as reference frames, captures the optical flow change characteristics through optical flow estimation, and gradually approaches the true value of the video frame to be predicted through multiple rounds of cyclic processing, and finally generates a target predicted video frame. This method gradually approaches the true value of the video frame to be predicted through multiple rounds of cyclic optical flow estimation, thereby providing accuracy of prediction estimation. At the same time, based only on the forward video frame as a reference frame, it avoids the impact of introducing backward reference frames on coding efficiency, and meets the requirements of low-latency time domain coding mode.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0028] Figure 1 A flowchart of a video encoding method based on optical flow according to an embodiment of the present disclosure is shown;

[0029] Figure 2 A schematic diagram showing a video encoding process based on optical flow according to an embodiment of the present disclosure is shown;

[0030] Figure 3 A flowchart of an optical flow estimation process according to an embodiment of the present disclosure is shown;

[0031] Figure 4 A flowchart of an optical flow change estimation process according to an embodiment of the present disclosure is shown;

[0032] Figure 5 A flowchart of a spatial variation estimation process according to an embodiment of the present disclosure is shown;

[0033] Figure 6 A schematic diagram of the structure of a grouped spatial attention module according to an embodiment of the present disclosure is shown;

[0034] Figure 7 A schematic diagram showing the result of a forward optical flow estimation module according to an embodiment of the present disclosure is shown;

[0035] Figure 8 A schematic diagram of the structure of a video encoding device based on optical flow according to an embodiment of the present disclosure is shown;

[0036] Fig. 9 A schematic diagram of the structure of an electronic device suitable for implementing an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0038] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0039] It should be noted that the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present disclosure are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects, and the descriptions of "first" and "second" do not limit the objects to be different.

[0040] In order to solve the above problems, the embodiment of the present disclosure proposes a video coding method based on optical flow, which improves the existing frame interpolation algorithm based on optical flow estimation, avoids the dependence on backward reference frames, and meets the low-latency mode requirements in video coding. This method significantly reduces processing delay without sacrificing frame prediction quality.

[0041] Figure 1 A flowchart of a video encoding method based on optical flow according to an embodiment of the present disclosure is shown. Figure 2 FIG. 2 is a schematic diagram showing a video encoding process based on optical flow according to an embodiment of the present disclosure. Figure 1 , 2 As shown, the optical flow-based video encoding method may include the following steps.

[0042] In step S110, the first video frame, the second video frame, the first changed frame, the second changed frame and the mask obtained in the i-1th round are spliced ​​to obtain a first feature; wherein the first changed frame and the second changed frame are obtained based on the change processing of the first video frame and the second video frame respectively.

[0043] In the disclosed embodiment, the video encoding method predicts future video frames based on the optical flow estimation method according to the existing video frames, thereby realizing the encoding of the inserted video frames. Among them, optical flow video frame prediction is a technology that predicts future frames based on the optical flow field between adjacent frames. Optical flow refers to the pixel-level motion vector of an object, surface or edge in a sequence of continuous image frames. By calculating the optical flow, the changes between frames can be captured and then used to predict the content of subsequent frames. This method is usually used to predict one frame from several consecutive frames, and to generate multiple frames through rolling prediction. It has the characteristics of high computational efficiency and good prediction accuracy in the short term.

[0044] In the disclosed embodiment, based on the optical flow video frame prediction method, a step-by-step approximation method is used to gradually approximate the prediction of future video frames by cyclically performing optical flow estimation on existing video frames. Therefore, in this scheme, steps S110 to S140 are a cyclic estimation process of step-by-step approximation. Step-by-step approximation is performed according to a preset cyclic estimation process until a preset cyclic exit condition is met.

[0045] In the present disclosure, Figure 2 As shown, the first video frame I 0 , the second video frame I 1 and the first change frame obtained in round i-1 Second change frame and mask M i-1 Splice and get the first feature X 1 .

[0046] Among them, the first video frame I 0 , the second video frame I 1 It is a video frame that exists before the video frame to be predicted.

[0047] The first change frame Second change frame are the first changed frame and the second changed frame obtained in the i-1th round in the aforementioned cycle process. The first changed frame and the second changed frame are obtained by warping the first video frame and the second video frame respectively. Warping is an optical flow warping technique in computer vision and image processing. It mainly refers to changing an image to match the position and perspective of another image according to a given optical flow field. Specifically, warping compensates for the displacement between images by moving pixels in the image so that one image can be aligned with another image.

[0048] The mask M i-1 It is a mask image that represents the change of optical flow obtained through optical flow estimation.

[0049] In an exemplary embodiment, the first video frame I 0 , the second video frame I 1 and the target prediction video frames are adjacent video frames that are continuous in time domain.

[0050] It should be noted that the exemplary embodiment provided in the present disclosure predicts one video frame to be predicted based on two existing video frames. In practical applications, multiple video frames to be predicted may also be predicted based on multiple existing video frames as required. For example, two video frames to be predicted may be predicted based on three existing video frames.

[0051] In an exemplary embodiment, a rolling prediction method can be further used to simultaneously predict multiple video frames to be predicted through multiple existing video frames. Rolling prediction refers to the process of predicting a series of consecutive frames in sequence, starting from the current frame, predicting future frames frame by frame in chronological order, and realizing long sequence frame prediction by continuously updating the reference frame. In the rolling prediction process, the reference frame can be the original video frame or the predicted frame obtained by the previous prediction. Rolling prediction is performed backward through the obtained predicted frame.

[0052] In step S120, based on the scaling factor of the i-th round, the optical flow of the i-1th round and the first feature are estimated by a forward optical flow estimation module to obtain residual information of the optical flow of the i-th round and the mask.

[0053] In the present disclosure, Figure 2 As shown, the first feature X obtained by splicing in the above step S110 is 1 and the optical flow F of the i-1th round i-1 The optical flow F i-1is the optical flow information (Flow) obtained through optical flow estimation. The forward optical flow estimation module is a technology for estimating the pixel motion vector from the current video frame to the next video frame. The module predicts the optical flow field by analyzing the changes between adjacent frames, thereby supporting the interpolation or prediction of video frames. A scaling factor is pre-set in the forward optical flow estimation module. The scaling factor is used to adjust the scaling ratio of the relevant input features. The forward optical flow estimation module, based on the pre-set scaling factor, 1 and the optical flow F of the i-1th round i-1 Perform optical flow estimation to obtain the residual information ΔF of the optical flow and mask of the i-th round i , ΔM i Residual information refers to the difference between the actual value and the predicted value in signal processing, which is used to represent the information or error that cannot be captured.

[0054] In an exemplary embodiment, the scaling factor is associated with the current round number i, that is, the scaling factor changes with the increase of the round number, so as to achieve a step-by-step approach to the actual value. For example, the scaling factor is updated once every three rounds. The scaling factor becomes smaller with the increase of the round number.

[0055] In an exemplary embodiment, the scaling factor is not a pre-set fixed value, but is calculated by adjusting the parameter values ​​obtained in the previous rounds during the cycle. The present disclosure does not specifically limit how to select the parameters of the previous rounds and how to design the relevant calculation formulas.

[0056] In step S130, the optical flow and mask obtained in the (i-1)th round are fused with the residual information of the optical flow and mask in the i-th round to obtain the optical flow and mask in the i-th round.

[0057] In the disclosed embodiment, the optical flow F obtained in the i-1th round is i-1 Compared with the above-mentioned optical flow residual information ΔF of the i-th round i Through the fusion module, the optical flow F of the i-th round is obtained. i Similarly, the mask M obtained in the i-1th round i-1 The mask residual information ΔM obtained above for the i-th round i Through the fusion module, the mask M of the i-th round is obtained. i . For details, please refer to the following formula:

[0058] F i =F i-1 +ΔF i

[0059] M i =M i-1 +ΔMi

[0060] In step S140, the first video frame and the second video frame are subjected to change processing based on the i-th round of optical flow to obtain a first changed frame and a second changed frame of the i-th round.

[0061] In the embodiment of the present disclosure, in order to perform the next round of cyclic processing, based on the optical flow F of the i-th round i For the first video frame I 0 and the second video frame I 1 Perform warping to obtain the first change frame of the i-th round Second change frame Through the above change processing, the first change frame required for the next round of feature splicing in step S110 is obtained. Second change frame

[0062] In the embodiment of the present disclosure, the above steps S110 to S140 are a cycle process. Through this cycle process, based on the first video frame I0, the second video frame I 1 and the first change frame obtained in round i-1 Second change frame Mask M i-1 and optical flow F i-1 , the optical flow F of the i-th round can be obtained through the optical flow estimation of the forward optical flow estimation module. i and mask M i , and the first change frame of the i-th round Second change frame This cycle continues, and as the number of rounds i increases, the true value of the video frame to be predicted is gradually approached.

[0063] In step S150, in response to the round number i not being equal to the preset round number N, the round number i is increased by 1, and the above steps are repeated.

[0064] In the embodiment of the present disclosure, the exit condition of the loop process is set to a preset round number N. In response to the current round number i not being equal to the preset round number N, that is, not satisfying the exit condition, the loop process is continued. Based on the loop process, the round number i is increased by 1, and the process of the aforementioned steps S110 to S140 is looped.

[0065] In step S160 , in response to the round number i being equal to the preset round number N, a target predicted video frame is generated based on the first change frame, the second change frame and the mask obtained in the i-th round.

[0066] In the embodiment of the present disclosure, the exit condition of the loop process is set to the preset round number N. In response to the current round number i being equal to the preset round number N, that is, the exit condition is met, the loop process is exited. Second change frame and mask M i , generate target prediction video frame I 2 .

[0067] In an exemplary embodiment, the target prediction video can be based on the first change frame obtained in the i-th round Second change frame and mask M i The weighted sum is obtained. For details, please refer to the following formula:

[0068]

[0069] in, represents the first change frame obtained in the i-th round, represents the second change frame obtained in the i-th round, I 2 represents the target predicted video frame, M i Represents the mask obtained in the i-th round.

[0070] In an exemplary embodiment, before generating a target predicted video frame based on the first change frame and the second change frame obtained in the i-th round, an activation (sigmoid) operation needs to be performed on the first change frame and the second change frame. Based on the first change frame, the second change frame and the mask after the activation operation, the target predicted video frame is generated.

[0071] In an exemplary embodiment, during the video encoding process, the total number of video frames supplemented by the inserted video frames should be consistent with the total number of video frames of the original video. Therefore, when multiple inserted video frames are predicted backward based on the existing video frames based on the solution provided by the present disclosure, the inserted video frames may be redundant and inconsistent with the total number of video frames of the original video. Therefore, it is necessary to perform special processing on the remaining frames at the end of the video frames to ensure that the total number of video frames after encoding remains consistent. Specifically, the following steps may be included.

[0072] It is determined whether the number of generated target predicted video frames is greater than the number of remaining frames of the video to be encoded.

[0073] In response to the number of the target predicted video frames being greater than the number of remaining frames, the target predicted video frames are discarded according to the number of remaining frames.

[0074] After the target prediction video frames are generated based on step S160, the number of generated target prediction video frames is compared with the number of remaining frames of the current video to be encoded. When the number of target prediction video frames is less than or equal to the number of remaining frames, the prediction of the inserted video frames is performed normally according to the method provided in the present disclosure. When the number of target prediction video frames is greater than the number of remaining frames, it means that there is redundancy in the generated target prediction video frames, and some of the generated target prediction video frames are discarded according to the remaining number of frames. For example, two target prediction video frames are generated based on three existing video frames, and the number of remaining frames of the current video to be encoded is one. Based on this, one of the two generated target prediction video frames is discarded according to the remaining number of frames to keep the total number of encoded video frames consistent.

[0075] The optical flow-based video encoding method provided by the embodiment of the present disclosure uses a forward optical flow estimation module to use multiple video frames forward of the video frame to be predicted as reference frames, captures the optical flow change characteristics through optical flow estimation, and gradually approaches the true value of the video frame to be predicted through multiple rounds of cyclic processing, and finally generates a target predicted video frame. This method gradually approaches the true value of the video frame to be predicted through multiple rounds of cyclic optical flow estimation, thereby providing accuracy of prediction estimation. At the same time, based only on the forward video frame as a reference frame, it avoids the impact of introducing backward reference frames on coding efficiency, and meets the requirements of low-latency time domain coding mode.

[0076] Figure 3 A flowchart of an optical flow estimation process according to an embodiment of the present disclosure is shown. Figure 7 FIG. 4 is a schematic diagram showing the result of the forward optical flow estimation module of the embodiment of the present disclosure. Figure 3 , 7 As shown, in Figure 1 Based on the optical flow-based video encoding method, step S120 may include the following steps.

[0077] In step S310, based on the scaling factor of the i-th round, an optical flow change module is used to estimate the optical flow change of the (i-1)-th round optical flow and the first feature to obtain an optical flow change feature.

[0078] In the present disclosure, Figure 7 As shown, the forward optical flow estimation module includes: an optical flow variation module and a spatial variation module. The optical flow variation module, based on the scaling factor of the i-th round, transforms the optical flow F of the i-1th round input to the forward optical flow estimation module. i-1 and the first feature X 1 Perform optical flow change estimation to obtain the optical flow change feature X f The optical flow change estimation is based on the current round of scaling factors, by analyzing the previous round of optical flow field and feature map to predict the change of the optical flow field. Optical flow change feature Xf , which is used to characterize the relevant features of the optical flow changes between the previous and next video frames.

[0079] In step S320, a spatial variation module is used to perform spatial variation estimation on the optical flow and the first feature of the (i-1)th round to obtain a spatial variation feature.

[0080] In the present disclosure, Figure 7 As shown, the forward optical flow estimation module includes: an optical flow variation module and a spatial variation module. The spatial variation module, based on the scaling factor of the i-th round, performs the i-1th round of optical flow F input to the forward optical flow estimation module. i-1 and the first feature X 1 Perform spatial variation estimation to obtain spatial variation feature X s The spatial variation estimation is to predict the change of the image in the spatial domain by analyzing the optical flow and feature map of the previous round given the current round scaling factor. Spatial variation feature X s , used to characterize the relevant features of spatial changes between video frames.

[0081] In step S330, the optical flow variation feature and the spatial variation feature are concatenated to obtain an optical flow estimation feature.

[0082] In the embodiment of the present disclosure, the optical flow change feature X obtained in the above steps S310 and S320 is f and spatial variation characteristics X s Splice to get the optical flow estimation feature X ffe .

[0083] In an exemplary embodiment, the optical flow variation feature X f and spatial variation characteristics X s Splicing is performed based on feature channels. Feature channels are used to represent feature information of different dimensions in video frames.

[0084] In step S340, a deconvolution operation is performed on the optical flow estimation feature to obtain residual information of the optical flow and mask of the i-th round.

[0085] In the embodiment of the present disclosure, the optical flow estimation feature X ffe Perform deconvolution operation to obtain the residual information ΔF of the optical flow and mask of the i-th round i , ΔM i As mentioned above, the optical flow estimation feature X ffe It is generated based on the splicing of feature channels. Therefore, the optical flow estimation feature X ffe After the deconvolution operation, the residual information ΔF of the optical flow can be split based on the correspondence between the feature channels. i And the residual information of the mask ΔMi .

[0086] Figure 4 A flowchart of an optical flow change estimation process according to an embodiment of the present disclosure is shown. Figure 7 FIG. 4 is a schematic diagram showing the result of the forward optical flow estimation module of the embodiment of the present disclosure. Figure 4 , 7 As shown, in Figure 3 Based on the optical flow-based video encoding method, step S310 may include the following steps.

[0087] In step S410, the optical flow of the (i-1)th round and the first feature are concatenated to obtain a second feature.

[0088] In the present disclosure, Figure 7 As shown, the optical flow F of the i-1th round is input into the forward optical flow estimation module. i-1 and the first feature X 1 Splice and get the second feature X 2 .

[0089] In an exemplary embodiment, if Figure 7 As shown, in order to improve the efficiency of data processing, the optical flow F is also processed before the above feature splicing. i-1 and the first feature X 1 Perform a downsampling operation to obtain the downsampled optical flow and the first feature. Then concatenate the downsampled optical flow and the first feature to obtain the second feature X 2 .

[0090] In an exemplary embodiment, the sampling magnification of the downsampling process is Scale_list[i]. The sampling magnification of the downsampling process is associated with the scaling factor of the current round, and the sampling magnification is adjusted according to the scaling factor.

[0091] In step S420, a channel alignment operation is performed on the second feature to obtain a third feature.

[0092] In the present disclosure, Figure 7 As shown, for the second feature X 2 Perform channel alignment to obtain the third feature X 3 The channel alignment operation refers to adjusting the number of feature channels of a feature to align with the number of target channels. The number of target channels may be different depending on the needs of subsequent processing. The specific number of target channels is not limited here. At the same time, the channel alignment operation is a conventional technical means in this field and will not be further introduced here.

[0093] In an exemplary embodiment, at least one convolution operation is performed on the second feature to obtain a third feature aligned with a preset number of channels. In this example, the channel alignment operation is implemented by multiple convolution operations. For example, where stride=2, M is the set number of feature channels. Therefore, by performing two convolution operations on the second feature, a third feature aligned with the set number of feature channels M can be obtained. After the first convolution operation, the features of the M / 2 channels are output. After the second convolution operation, the features of the M channels are output.

[0094] In step S430, optical flow feature learning is performed on the third feature through a residual feature learning module to obtain the optical flow change feature.

[0095] In the present disclosure, Figure 7 As shown, the third feature X is learned by the residual feature learning module 3 Perform optical flow feature learning to obtain the optical flow change feature X f The residual feature learning module is a network module for learning residual information from input features, aiming to capture the difference between input features and target features. For example, the ResBlock network. Optical flow feature learning refers to the process of estimating pixel-level optical flow motion vectors by analyzing image sequences.

[0096] In an exemplary embodiment, if Figure 7 As shown, corresponding to the downsampling operation in the aforementioned step S410, the optical flow feature learning is performed on the third feature by the residual feature learning module to obtain the optical flow change feature, including:

[0097] Performing optical flow feature learning on the third feature through the residual feature learning module to obtain a fourth feature;

[0098] Performing a channel alignment operation on the fourth feature to obtain a fifth feature;

[0099] An upsampling operation is performed on the fifth feature to obtain the optical flow change feature.

[0100] In an exemplary embodiment, the residual feature learning module performs optical flow feature learning on the third feature to first obtain the fourth feature X 4 Then the fourth feature X 4 Perform channel alignment to obtain the fifth feature X 5 Finally, for the fifth feature X 5 Perform upsampling operation to obtain the optical flow change feature X fThe upsampling process is the reverse process of the downsampling in step S410. Therefore, the sampling magnification of the upsampling process is the same as the downsampling process, which is Scale_list[i]. Similarly, the sampling magnification of the upsampling process is also associated with the scaling factor of the current round, and the sampling magnification is adjusted according to the scaling factor.

[0101] Figure 5 A flowchart of a spatial variation estimation process according to an embodiment of the present disclosure is shown. Figure 7 FIG. 4 is a schematic diagram showing the result of the forward optical flow estimation module of the embodiment of the present disclosure. Figure 5 , 7 As shown, in Figure 3 Based on the optical flow-based video encoding method, step S320 may include the following steps.

[0102] In step S510, the optical flow of the (i-1)th round and the first feature are concatenated to obtain a sixth feature.

[0103] In the present disclosure, Figure 7 As shown, the optical flow F of the i-1th round is input into the forward optical flow estimation module. i-1 and the first feature X 1 After splicing, we get the sixth feature X 6 The sixth characteristic X 6 Can be combined with the aforementioned second feature X 2 The same, but also due to different splicing methods, and the second feature X 2 Different, not limited here.

[0104] In step S520, a channel alignment operation is performed on the sixth feature to obtain a seventh feature.

[0105] In the embodiment of the present disclosure, based on the channel alignment requirement, the sixth feature X 6 Perform channel alignment to obtain the seventh feature X 7 .

[0106] In step S530, spatial attention learning is performed on the seventh feature through a group-based spatial attention module (GSA) to obtain the spatial attention feature.

[0107] In the embodiment of the present disclosure, the seventh feature X is treated by a pre-designed grouping space attention module. 7 Perform spatial attention learning to obtain spatial attention feature X A .

[0108] The grouped spatial attention module is a module that extracts attention features at different scales in the spatial domain of video frames. The module applies dilated convolutions of different sizes to extract features of different scales in the spatial domain of video frames. By extracting features of different scales, the ability to capture different levels of detail is enhanced, thereby better capturing the spatial variation characteristics of video frames. The obtained spatial attention feature X A , used to characterize the changing characteristics of video frames in the spatial domain.

[0109] In an exemplary embodiment, different weights may be added to the grouped spatial attention modules in different rounds according to the aforementioned loop process. The weight is associated with the aforementioned scaling factor. According to the scaling factor corresponding to the loop round, the grouped spatial attention modules with the same scaling factor have the same weight, and the grouped spatial attention modules with different scaling factors have different weights.

[0110] In step S540, feature learning is performed on the spatial attention feature to obtain the spatial variation feature.

[0111] In the disclosed embodiment, the spatial attention feature X A Perform feature learning to obtain the spatial variation feature X s .

[0112] In an exemplary embodiment, the spatial attention feature X is subjected to at least one layer of convolution operation. A Specifically, the spatial attention feature X can be learned through a 1X1 convolution kernel. A Perform convolution operation to obtain the spatial variation feature X s .

[0113] Figure 6 FIG. 4 is a schematic diagram showing the structure of the grouping spatial attention module of the embodiment of the present disclosure. Figure 6 As shown, in Figure 5 Based on the optical flow-based video encoding method, step S530 may include the following steps.

[0114] Step 1: Perform a first dilated convolution operation on the seventh feature to obtain an eighth feature.

[0115] In the embodiment of the present disclosure, the seventh feature X 7 Perform the first dilated convolution operation to obtain the eighth feature X 8 .

[0116] The dilated convolution is a convolution idea proposed for the problem of image semantic segmentation, where downsampling will reduce image resolution and lose information. It introduces a new parameter called "dilation rate" to define the spacing between values ​​when the convolution kernel processes data. The dilation rate determines the spacing between elements in the convolution kernel, thereby increasing its receptive field (i.e., the area that the convolution kernel can cover) while keeping the size of the convolution kernel unchanged.

[0117] In an exemplary embodiment, the dilated convolution operation can be adjusted by multiple parameters. The adjustment of the dilated convolution operation can be represented by DWConv(a,b,c,d). Where a represents the number of input feature channels, b represents the number of output feature channels, c represents the convolution kernel size, and d represents the step size. In addition, Dilation represents the dilation rate of the dilated convolution. Figure 6 As shown, in this example, the first dilated convolution is set to DWConv(C,C,3,1), Dilation=1.

[0118] Step 2: Divide the eighth feature into at least one channel feature according to the channel; perform a dilated convolution operation on the at least one channel feature to obtain at least one channel space feature.

[0119] In the disclosed embodiment, the eighth feature X 8 Divide into multiple different channel features X according to the feature channel C . For the divided multiple channel features X C Then perform the dilated convolution operation separately to obtain the features X of each channel C Corresponding multiple channel space features X DW .

[0120] In an exemplary embodiment, during the channel feature division process, the eighth feature X can be divided into 8 Divide into at least one channel feature X according to the number of channels C .like Figure 6 As shown, in this example, the eighth feature X 8 Divide into four channel features X according to the number of channels C1 , X C2 , X C3 , X C4 For example, the eighth feature X 8 Including 16 feature channels, the features of channels 0-3 are divided into channel features X C1 , divide the features of channels 4-7 into channel features X C2 , divide the features of channels 8-11 into channel features XC3 , divide the features of channels 12-15 into channel features X C4 Of course, according to actual application needs, the number of channels of each channel feature may not be evenly divided, but divided according to a preset channel number ratio. For example, the eighth feature X 8 It includes 16 feature channels, which are divided into three channel features according to 2:1:1. Then the features of channels 0-7 are divided into channel features X C1 , divide the features of channels 8-11 into channel features X C2 , divide the features of channels 12-15 into channel features X C3 .

[0121] In an exemplary embodiment, as described above, the dilated convolution operation can be adjusted by multiple parameters. Based on this, in this example, the convolution kernel sizes corresponding to the dilated convolution operation for different channel features are different, and dilated convolution operations of different scales are performed. Figure 6 As shown, in this example, for the channel feature X C1 The feature is retained without performing the dilated convolution operation, and the channel space feature X is obtained. DW1 ; For channel feature X C2 The convolution kernel scale corresponding to the dilated convolution operation is 3X3, Dilation=3, and the channel space feature X is obtained. DW2 ; For channel feature X C3 The convolution kernel scale corresponding to the dilated convolution operation is 5X5, Dilation=3, and the channel space feature X is obtained. DW3 ; For channel feature X C4 The convolution kernel scale corresponding to the dilated convolution operation is 7X7, Dilation=3, and the channel space feature X is obtained. DW4 By applying dilated convolutions of different sizes, features of different scales are extracted from video frames in the spatial domain, which enhances the ability to capture different levels of detail and better captures the spatial variation characteristics of video frames.

[0122] It should be pointed out that not performing a dilated convolution operation on channel features to retain the original features should also be regarded as a special dilated convolution operation, as one of the different dilated convolution operations available.

[0123] Step 3: splice the at least one channel space feature to obtain a ninth feature.

[0124] In the embodiment of the present disclosure, the multiple channel space features X obtained above are DW After splicing, we get the ninth feature X 9 .

[0125] Step 4: Perform feature learning on the seventh feature and the ninth feature respectively to obtain the tenth feature and the thirteenth feature.

[0126] In the embodiment of the present disclosure, the seventh feature X 7 and the ninth feature X 9 Perform feature learning separately and get the tenth feature X 10 and the thirteenth characteristic X 13 .like Figure 6 As shown, the feature learning can be performed through a convolution layer. The convolution operation can be set to Conv(C, C, 3, 1).

[0127] Step 5: Activate the tenth feature and the thirteenth feature, and perform dot multiplication on the activated tenth feature and the thirteenth feature to obtain the eleventh feature.

[0128] In the embodiment of the present disclosure, the tenth feature X 10 and the thirteenth characteristic X 13 Perform activation processing and convert the activated tenth feature X 10 and the thirteenth characteristic X 13 Perform dot multiplication to obtain the eleventh feature X 11 .

[0129] The activation process may be performed using different activation functions such as Sigmoid, ReLU or SiLU.

[0130] Step 6: Perform feature learning on the eleventh feature to obtain the twelfth feature.

[0131] In the embodiment of the present disclosure, the eleventh feature X 11 Perform feature learning to obtain the twelfth feature X 12 .like Figure 6 As shown, the feature learning can be performed through a convolution layer. The convolution operation can be set to Conv(C, C, 3, 1).

[0132] Step 7, adding the seventh feature and the twelfth feature to obtain the spatial attention feature.

[0133] In the embodiment of the present disclosure, the seventh feature X 7 and the twelfth feature X 12 Add the features to get the spatial attention feature X A .

[0134] In the optical flow-based video encoding method provided by the embodiment of the present disclosure, an optical flow change module and a spatial change module are designed in the forward optical flow estimation module. The optical flow change module is used to capture the optical flow change characteristics between reference frames. The spatial change module is used to capture the spatial change characteristics between reference frames. By combining the optical flow change characteristics and the spatial change characteristics, the change characteristics between reference frames can be better captured, thereby accurately predicting future video frames. Among them, a scaling factor is introduced in the optical flow change module, and the optical flow change characteristics are captured through the residual feature learning module. In the spatial change module, by applying hole convolutions of different sizes, features of different scales are extracted from video frames in the spatial domain, thereby enhancing the ability to capture different levels of details, thereby better capturing the spatial change characteristics of video frames.

[0135] Based on the same inventive concept, the embodiment of the present disclosure provides a video encoding device based on optical flow, as described in the following embodiment. Since the principle of solving the problem in the embodiment of the device is similar to that in the above method embodiment, the implementation of the communication authentication device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0136] Figure 8 FIG. 2 shows a schematic diagram of the structure of a video encoding device based on optical flow according to an embodiment of the present disclosure. Figure 8 As shown, the optical flow-based video encoding device 800 may include: a video frame splicing module 810 , a forward optical flow estimation module 820 , a fusion module 830 , a change processing module 840 and a prediction frame module 850 .

[0137] The video frame splicing module 810 is configured to splice the first video frame, the second video frame, the first changed frame, the second changed frame and the mask obtained in the i-1th round to obtain a first feature; wherein the first changed frame and the second changed frame are respectively obtained based on the change processing of the first video frame and the second video frame.

[0138] The forward optical flow estimation module 820 is configured to perform optical flow estimation on the optical flow of the i-1th round and the first feature based on the scaling factor of the i-th round, and obtain residual information of the optical flow of the i-th round and the mask.

[0139] The fusion module 830 is configured to fuse the optical flow and mask obtained in the (i-1)th round with the residual information of the optical flow and mask in the i-th round to obtain the optical flow and mask in the i-th round.

[0140] The change processing module 840 is configured to perform change processing on the first video frame and the second video frame based on the i-th round of optical flow to obtain the i-th round of first changed frame and second changed frame.

[0141] The prediction frame module 850 is configured to, in response to the round number i not being equal to the preset round number N, increase the round number i by 1 and repeat the aforementioned steps; in response to the round number i being equal to the preset round number N, generate a target predicted video frame based on the first change frame, the second change frame and the mask obtained in the i-th round.

[0142] In an exemplary embodiment, the first video frame, the second video frame and the target prediction video frame are temporally consecutive video frames.

[0143] In an exemplary embodiment, the scaling factor is associated with the round number i.

[0144] In an exemplary embodiment, the prediction frame module 850 is further configured to perform weighted summation based on the first change frame, the second change frame and the mask obtained in the i-th round to generate the target predicted video frame, and the corresponding formula is as follows:

[0145]

[0146] in, represents the first change frame obtained in the i-th round, represents the second change frame obtained in the i-th round, I 2 represents the target predicted video frame, M i Represents the mask obtained in the i-th round.

[0147] In an exemplary embodiment, the forward optical flow estimation module 820 is also configured to perform optical flow change estimation on the optical flow and the first feature of the i-1th round based on the scaling factor of the i-th round through an optical flow change module to obtain an optical flow change feature; perform spatial change estimation on the optical flow and the first feature of the i-1th round through a spatial change module to obtain a spatial change feature; splice the optical flow change feature and the spatial change feature to obtain an optical flow estimation feature; perform a deconvolution operation on the optical flow estimation feature to obtain residual information of the optical flow and mask of the i-th round.

[0148] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to concatenate the optical flow of the i-1th round and the first feature to obtain a second feature; perform a channel alignment operation on the second feature to obtain a third feature; and perform optical flow feature learning on the third feature through a residual feature learning module to obtain the optical flow change feature.

[0149] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to perform a downsampling operation on the optical flow and the first feature of the i-1th round to obtain the downsampled optical flow and the first feature; concatenate the downsampled optical flow and the first feature to obtain the second feature; perform optical flow feature learning on the third feature through the residual feature learning module to obtain the optical flow change feature, including: performing optical flow feature learning on the third feature through the residual feature learning module to obtain a fourth feature; performing a channel alignment operation on the fourth feature to obtain a fifth feature; and upsampling the fifth feature to obtain the optical flow change feature; wherein the sampling ratios of the downsampling and upsampling are associated with the scaling factor.

[0150] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to perform at least one convolution operation on the second feature to obtain the third feature aligned with a preset number of channels.

[0151] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to concatenate the optical flow of the i-1th round and the first feature to obtain a sixth feature; perform a channel alignment operation on the sixth feature to obtain a seventh feature; perform spatial attention learning on the seventh feature through a group-based spatial attention module (Group-based Spatial Attention, GSA) to obtain the spatial attention feature; perform feature learning on the spatial attention feature to obtain the spatial change feature.

[0152] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to perform a first dilated convolution operation on the seventh feature to obtain an eighth feature; divide the eighth feature into at least one channel feature according to the channel; perform a dilated convolution operation on the at least one channel feature to obtain at least one channel space feature; concatenate the at least one channel space feature to obtain a ninth feature; perform feature learning on the seventh feature and the ninth feature to obtain a tenth feature and a thirteenth feature; activate the tenth feature and the thirteenth feature, and perform dot multiplication on the activated tenth feature and the thirteenth feature to obtain an eleventh feature; perform feature learning on the eleventh feature to obtain a twelfth feature; and perform feature addition on the seventh feature and the twelfth feature to obtain the spatial attention feature.

[0153] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to divide the eighth feature into the at least one channel feature according to the number of channels according to a preset ratio of the number of channels.

[0154] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to have different convolution kernel sizes for dilated convolution operations corresponding to different channel features.

[0155] In an exemplary embodiment, the prediction frame module 850 is also configured to determine whether the number of generated target prediction video frames is greater than the remaining number of frames of the video to be encoded; in response to the number of target prediction video frames being greater than the remaining number of frames, the target prediction video frames are discarded according to the remaining number of frames.

[0156] Fig. 9 The structure diagram of the electronic device suitable for implementing the exemplary embodiment of the present disclosure is shown. Fig. 9 An electronic device 900 according to this embodiment of the present invention will be described. Fig. 9 The electronic device 900 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0157] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include, but are not limited to: the at least one processing unit 910, the at least one storage unit 920, a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910), and a display unit 940.

[0158] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .

[0159] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0160] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0161] The electronic device 900 may also communicate with one or more external devices 970 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0162] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is further provided, on which a program product capable of implementing the above method of the present specification is stored.

[0163] In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present invention described in the above “Exemplary Methods” section of this specification.

[0164] The program product for implementing the above method according to the embodiment of the present invention may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with a signaling execution system, apparatus, or device.

[0165] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0166] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a signaling execution system, apparatus, or device.

[0167] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0168] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0169] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0170] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0171] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several signalings to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation methods of the present disclosure.

[0172] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0173] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video encoding method based on optical flow, characterized in that: The method comprises: The first video frame, the second video frame, the first changed frame obtained in the (i-1)th round, the second changed frame, and the mask are spliced ​​to obtain a first feature; wherein the first changed frame and the second changed frame are obtained based on the change processing of the first video frame and the second video frame respectively; Based on the scaling factor of the i-th round, the optical flow of the i-1th round and the first feature are estimated by the forward optical flow estimation module to obtain residual information of the optical flow of the i-th round and the mask; The optical flow and mask obtained in the (i-1)th round are combined with the residual information of the optical flow and mask in the i-th round to obtain the optical flow and mask in the i-th round; Performing change processing on the first video frame and the second video frame based on the i-th round of optical flow to obtain a first changed frame and a second changed frame of the i-th round; In response to the round number i not being equal to the preset round number N, the round number i is increased by 1, and the aforementioned steps are repeated; In response to the round number i being equal to the preset round number N, a target predicted video frame is generated based on the first change frame, the second change frame and the mask obtained in the i-th round.

2. The method according to claim 1, characterized in that The first video frame, the second video frame and the target predicted video frame are temporally continuous video frames.

3. The method according to claim 1, characterized in that The scaling factor is associated with the round number i.

4. The method according to claim 1, characterized in that: The generating a target predicted video frame based on the first changed frame, the second changed frame and the mask obtained in the i-th round includes: Based on the weighted sum of the first change frame, the second change frame and the mask obtained in the i-th round, the target predicted video frame is generated. The corresponding formula is as follows: in, represents the first change frame obtained in the i-th round, represents the second change frame obtained in the i-th round, I2 represents the target predicted video frame, M i Represents the mask obtained in the i-th round.

5. The method according to claim 1, characterized in that The step of performing optical flow estimation on the optical flow of the i-1th round and the first feature based on the scaling factor of the i-th round by a forward optical flow estimation module to obtain residual information of the optical flow of the i-th round and the mask includes: Based on the scaling factor of the i-th round, an optical flow change module is used to estimate the optical flow change of the optical flow of the i-1th round and the first feature to obtain an optical flow change feature; Performing spatial variation estimation on the optical flow and the first feature of the (i-1)th round by using a spatial variation module to obtain a spatial variation feature; The optical flow variation feature and the spatial variation feature are concatenated to obtain an optical flow estimation feature; A deconvolution operation is performed on the optical flow estimation feature to obtain residual information of the optical flow and mask of the i-th round.

6. The method according to claim 5, characterized in that The step of estimating the optical flow change of the optical flow of the i-1th round and the first feature based on the scaling factor of the i-th round by using an optical flow change module to obtain an optical flow change feature includes: The optical flow of the (i-1)th round and the first feature are concatenated to obtain a second feature; Performing a channel alignment operation on the second feature to obtain a third feature; The optical flow feature learning is performed on the third feature through the residual feature learning module to obtain the optical flow change feature.

7. The method according to claim 6, characterized in that The step of concatenating the optical flow of the i-1th round and the first feature to obtain the second feature includes: Performing a downsampling operation on the optical flow and the first feature of the (i-1)th round to obtain the downsampled optical flow and the first feature; Concatenate the downsampled optical flow and the first feature to obtain the second feature; The performing optical flow feature learning on the third feature through the residual feature learning module to obtain the optical flow change feature includes: Performing optical flow feature learning on the third feature through the residual feature learning module to obtain a fourth feature; Performing a channel alignment operation on the fourth feature to obtain a fifth feature; Performing an upsampling operation on the fifth feature to obtain the optical flow change feature; The sampling ratios of the downsampling and upsampling are associated with the scaling factor.

8. The method according to claim 6, characterized in that The performing a channel alignment operation on the second feature to obtain a third feature includes: Perform at least one convolution operation on the second feature to obtain the third feature aligned with a preset number of channels.

9. The method according to claim 5, characterized in that The spatial variation module is used to perform spatial variation estimation on the optical flow and the first feature of the i-1th round to obtain the spatial variation feature, including: The optical flow of the (i-1)th round and the first feature are concatenated to obtain a sixth feature; Performing a channel alignment operation on the sixth feature to obtain a seventh feature; Performing spatial attention learning on the seventh feature through a group-based spatial attention module (GSA) to obtain the spatial attention feature; Feature learning is performed on the spatial attention feature to obtain the spatial variation feature.

10. The method according to claim 9, characterized in that The performing spatial attention learning on the seventh feature by the grouping spatial attention module to obtain the spatial attention feature includes: Performing a first dilated convolution operation on the seventh feature to obtain an eighth feature; Dividing the eighth feature into at least one channel feature according to the channel; performing a dilated convolution operation on the at least one channel feature to obtain at least one channel space feature; splicing the at least one channel space feature to obtain a ninth feature; Performing feature learning on the seventh feature and the ninth feature respectively to obtain a tenth feature and a thirteenth feature; Activate the tenth feature and the thirteenth feature, and perform dot multiplication on the activated tenth feature and the thirteenth feature to obtain an eleventh feature; Performing feature learning on the eleventh feature to obtain a twelfth feature; The seventh feature and the twelfth feature are added to obtain the spatial attention feature.

11. The method according to claim 10, characterized in that The dividing the eighth feature into at least one channel feature according to channels includes: dividing the eighth feature into the at least one channel feature according to the number of channels according to a preset channel number ratio.

12. The method according to claim 10, characterized in that The at least one channel feature is subjected to a dilated convolution operation to obtain at least one channel space feature, and further includes: the convolution kernel sizes of the dilated convolution operations corresponding to different channel features are different.

13. The method according to claim 1, characterized in that Said also includes: Determining whether the number of generated target predicted video frames is greater than the number of remaining frames of the video to be encoded; In response to the number of the target predicted video frames being greater than the number of remaining frames, the target predicted video frames are discarded according to the number of remaining frames.

14. A video encoding device based on optical flow, characterized in that: include: A video frame splicing module is configured to splice the first video frame, the second video frame, the first changed frame, the second changed frame and the mask obtained in the (i-1)th round to obtain a first feature; wherein the first changed frame and the second changed frame are obtained based on the change processing of the first video frame and the second video frame respectively; A forward optical flow estimation module is configured to perform optical flow estimation on the optical flow of the i-1th round and the first feature based on the scaling factor of the i-th round, so as to obtain residual information of the optical flow of the i-th round and the mask; A fusion module is configured to fuse the optical flow and mask obtained in the (i-1)th round with the residual information of the optical flow and mask in the i-th round to obtain the optical flow and mask in the i-th round; a change processing module, configured to perform change processing on the first video frame and the second video frame based on the i-th round of optical flow to obtain a first changed frame and a second changed frame of the i-th round; The prediction frame module is configured to, in response to the round number i not being equal to the preset round number N, increase the round number i by 1 and repeat the aforementioned steps; in response to the round number i being equal to the preset round number N, generate a target predicted video frame based on the first change frame, the second change frame and the mask obtained in the i-th round.

15. An electronic device, characterized in that: include: one or more processors; A storage device configured to store one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement the method according to any one of claims 1 to 13.

16. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

17. A computer program product, comprising a computer program / signaling, characterized in that When the computer program / signaling is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images