Video encoding method, system, electronic device and storage medium

Abstract

The application discloses a video coding method, system, electronic equipment and storage medium, and relates to the technical field of data processing, and comprises the following steps: dividing an original video sequence into multiple groups of images; determining a forward cumulative optical flow and a backward cumulative optical flow for any video frame in any group of images; determining a forward inter-frame motion intensity based on the forward cumulative optical flow and a backward inter-frame motion intensity based on the backward cumulative optical flow; determining a forward motion estimation result based on the forward inter-frame motion intensity and a backward motion estimation result based on the backward inter-frame motion intensity; and coding the video frame based on the forward motion estimation result and the backward motion estimation result to obtain video coding data corresponding to the video frame. The application can improve the accuracy of motion estimation during bidirectional video coding by introducing the forward and backward cumulative optical flows and the inter-frame motion intensity calculation based on the optical flows, thereby improving the coding efficiency and quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a video coding method and system, an electronic device and a storage medium. BACKGROUND

[0002] Intelligent video coding refers to analyzing and compressing video content to improve compression efficiency while maintaining video quality. According to different use scenarios, video coding mainly has two common configurations: low delay (LD) and random access (RA). The LD configuration emphasizes minimizing coding delay and is suitable for real-time applications such as video calls and live broadcasts. In the LD configuration, predictive frames (P frames) are encoded in a single direction, that is, only previously encoded frames are referred to during encoding. The RA configuration allows viewers to freely browse videos and is suitable for applications that require random access, such as video on demand. The key frame type of the RA configuration is the bi-directional frame (B frame), which takes past and future frames as references.

[0003] However, the existing intelligent bi-directional video coding method has the problem of inaccuracy in processing long-term motion estimation between distant frames with large motion scenes. For example, some methods are trained on small groups of pictures (GoP) with small motion intensity, but are tested on large GoP with various motion intensities. The domain shift problem between the training and testing scenes causes the motion estimation module to be unable to effectively process long-term motion, thereby affecting the performance of bi-directional video coding.

[0004] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide a video coding method, system, electronic device and storage medium, which aims to solve the technical problem of inaccurate long-term motion estimation during bi-directional video coding, which affects the coding performance.

[0006] To achieve the above purpose, the present application provides a video coding method, which comprises:

[0007] dividing an original video sequence into multiple groups of pictures, for any video frame in any group of pictures, determining a forward cumulative optical flow between the video frame and a forward reference frame, and determining a backward cumulative optical flow between the video frame and a backward reference frame;

[0008] determine a forward inter-frame motion strength between the video frame and the forward reference frame based on the forward accumulated optical flow, and determine a backward inter-frame motion strength between the video frame and the backward reference frame based on the backward accumulated optical flow;

[0009] determine a forward motion estimation result between the video frame and the reference frame based on the forward inter-frame motion strength, and determine a backward motion estimation result between the video frame and the reference frame based on the backward inter-frame motion strength;

[0010] encode the video frame based on the forward motion estimation result and the backward motion estimation result to obtain video encoding data corresponding to the video frame.

[0011] In an embodiment, the step of determining the forward accumulated optical flow between the video frame and the forward reference frame comprises:

[0012] perform a warping operation on a second local optical flow between the first neighboring frame and a second neighboring frame based on a first local optical flow between the video frame and the first neighboring frame to obtain a warped optical flow, and superimpose the warped optical flow and the second local optical flow to obtain an accumulated optical flow between the video frame and the second neighboring frame, wherein the first neighboring frame is a forward video frame adjacent to the video frame, and the second neighboring frame is a forward video frame adjacent to the first neighboring frame.

[0013] update the accumulated optical flow between the video frame and the second neighboring frame as the first local optical flow, and update a third local optical flow between the second neighboring frame and a third neighboring frame as the second local optical flow, and perform the step of performing a warping operation on a second local optical flow between the first neighboring frame and a second neighboring frame based on a first local optical flow between the video frame and the first neighboring frame to obtain a warped optical flow and subsequent steps until the forward accumulated optical flow between the video frame and the forward reference frame is obtained, wherein the third neighboring frame is a forward video frame adjacent to the second neighboring frame.

[0014] In an embodiment, after the step of determining the forward accumulated optical flow between the video frame and the forward reference frame, the method further comprises:

[0015] perform a warping operation on the forward reference frame to obtain a predicted reference frame;

[0016] input the predicted reference frame, the forward reference frame, and the forward accumulated optical flow into a preset convolution layer to obtain a first processing result;

[0017] superimposing the first processing result and the forward accumulated optical flow to obtain a second processing result, updating the forward accumulated optical flow by the second processing result, and performing the step of determining a forward inter-frame motion strength between the video frame and the forward reference frame based on the forward accumulated optical flow.

[0018] In an embodiment, the step of determining a forward inter-frame motion strength between the video frame and the forward reference frame based on the forward accumulated optical flow comprises:

[0019] mapping the forward accumulated optical flow to a motion strength factor;

[0020] if the motion strength factor is greater than or equal to a preset strength threshold, determining that the forward inter-frame motion strength between the video frame and the forward reference frame is high strength;

[0021] if the motion strength factor is less than the preset strength threshold, determining that the forward inter-frame motion strength is low strength.

[0022] In an embodiment, the step of mapping the forward accumulated optical flow to a motion strength factor comprises:

[0023] determining a horizontal accumulated offset and a vertical accumulated offset of each pixel point in the video frame based on the forward accumulated optical flow;

[0024] traversing each of the pixel points, calculating a Euclidean norm of the horizontal accumulated offset and the vertical accumulated offset to obtain an accumulated offset length of the pixel point between the video frame and the forward reference frame;

[0025] obtaining a total offset length based on the accumulated offset length of each of the pixel points, dividing the total offset length by a number of pixels of the video frame to obtain an average optical flow strength of each of the pixel points, and determining the average optical flow strength as the motion strength factor.

[0026] In an embodiment, the step of determining a forward motion estimation result between the video frame and the forward reference frame based on the forward inter-frame motion strength comprises:

[0027] if the forward inter-frame motion strength is low strength, inputting the video frame and the forward reference frame into a preset optical flow network to obtain a forward direct optical flow, and determining the forward direct optical flow as the forward motion estimation result between the video frame and the forward reference frame;

[0028] if the forward inter-frame motion strength is high strength, determining the forward accumulated optical flow as the forward motion estimation result.

[0029] In an embodiment, the encoding the video frame based on the forward motion estimation result and the backward motion estimation result to obtain video encoding data corresponding to the video frame comprises:

[0030] encoding based on the forward motion estimation result and the backward motion estimation result to obtain encoding optical flow data, and decoding the encoding optical flow data to obtain decoding optical flow data;

[0031] generating a multi-scale temporal context based on the decoding optical flow data, forward reference features of the forward reference frame, and backward reference features of the backward reference frame;

[0032] calculating residual information by taking the multi-scale temporal context as a condition and the difference between the video frame;

[0033] determining the forward motion estimation, the backward motion estimation, and the residual information as the video encoding data corresponding to the video frame.

[0034] In addition, to achieve the above object, the present application further provides a video encoding system, which comprises:

[0035] an adaptive motion estimation module, configured to divide an original video sequence into multiple image groups, and for any video frame in any image group, determine a forward cumulative optical flow between the video frame and a forward reference frame, and determine a backward cumulative optical flow between the video frame and a backward reference frame;

[0036] the adaptive motion estimation module is further configured to determine a forward inter-frame motion strength between the video frame and the forward reference frame based on the forward cumulative optical flow, and determine a backward inter-frame motion strength between the video frame and the backward reference frame based on the backward cumulative optical flow;

[0037] the adaptive motion estimation module is further configured to determine a forward motion estimation result between the video frame and the reference frame based on the forward inter-frame motion strength, and determine a backward motion estimation result between the video frame and the reference frame based on the backward inter-frame motion strength;

[0038] a video encoding module, configured to encode the video frame based on the forward motion estimation result and the backward motion estimation result to obtain video encoding data corresponding to the video frame.

[0039] In addition, to achieve the above object, the present application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the video encoding method as described above.

[0040] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the steps of the video encoding method.

[0041] In the application, the original video sequence is divided into multiple groups of images, for any video frame in any group of images, forward accumulated optical flow between the video frame and a forward reference frame is determined, and backward accumulated optical flow between the video frame and a backward reference frame is determined; forward inter-frame motion strength between the video frame and the forward reference frame is determined based on the forward accumulated optical flow, and backward inter-frame motion strength between the video frame and the backward reference frame is determined based on the backward accumulated optical flow. By calculating the forward accumulated optical flow between the video frame and the forward reference frame, and the backward accumulated optical flow between the video frame and the backward reference frame, the motion information of the video frame in different directions can be captured, so that the motion relationship between the video frame and the reference frame can be estimated, which provides a basis for subsequent motion estimation and reduces the problem of inaccurate long-term motion estimation caused by domain offset.

[0042] The forward motion estimation result between the video frame and the reference frame is determined based on the forward inter-frame motion strength, and the backward motion estimation result between the video frame and the reference frame is determined based on the backward inter-frame motion strength; the video frame is encoded based on the forward motion estimation result and the backward motion estimation result to obtain video encoding data corresponding to the video frame. By more accurate long-term motion estimation, the correlation between video frames can be utilized to improve the accuracy of subsequent video encoding, thereby improving the video encoding quality.

[0043] In summary, by introducing forward and backward accumulated optical flow and inter-frame motion strength calculation based on optical flow, the application can improve the accuracy of motion estimation in bidirectional video encoding, thereby improving the encoding efficiency and quality. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and serve to explain the principles of the application, together with the description.

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0046] Figure 1Flowchart provided by the embodiment one of the video encoding method of the present application;

[0047] Figure 2 Bidirectional video encoding framework provided by the embodiment of the present application;

[0048] Figure 3 Direct optical flow estimation framework provided by the embodiment of the present application;

[0049] Figure 4 Cumulative optical flow estimation process provided by the embodiment of the present application;

[0050] Figure 5 Cumulative optical flow estimation framework provided by the embodiment of the present application;

[0051] Figure 6 Optical flow fine-tuning framework provided by the embodiment of the present application;

[0052] Figure 7 Adaptive motion estimation flowchart provided by the embodiment of the present application;

[0053] Figure 8 Effect comparison diagram of optical flow estimation under small motion provided by the embodiment of the present application;

[0054] Figure 9 Effect comparison diagram of optical flow estimation under large motion provided by the embodiment of the present application;

[0055] Figure 10 Module structure diagram of the video encoding system of the embodiment of the present application;

[0056] Figure 11 Device structure diagram of the hardware running environment involved in the video encoding method in the embodiment of the present application.

[0057] The purpose implementation, functional features and advantages of the present application will be further explained in combination with the embodiments and with reference to the drawings. DETAILED DESCRIPTION

[0058] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0059] In order to better understand the technical solutions of the present application, the following will be described in detail in combination with the drawings of the specification and the specific embodiments.

[0060] The main solution of the embodiment of the present application is: dividing an original video sequence into multiple groups of pictures, determining, for any video frame in any group of pictures, a forward accumulated optical flow between the video frame and a forward reference frame, and determining a backward accumulated optical flow between the video frame and a backward reference frame; determining a forward inter-frame motion strength between the video frame and the forward reference frame based on the forward accumulated optical flow, and determining a backward inter-frame motion strength between the video frame and the backward reference frame based on the backward accumulated optical flow; determining a forward motion estimation result between the video frame and the reference frame based on the forward inter-frame motion strength, and determining a backward motion estimation result between the video frame and the reference frame based on the backward inter-frame motion strength; and encoding the video frame based on the forward motion estimation result and the backward motion estimation result to obtain video encoding data corresponding to the video frame.

[0061] In the embodiment, the following is described with the electronic device as the execution subject for the convenience of description.

[0062] Currently, the performance of the intelligent bidirectional video coding method is far behind the traditional video coding standard H.266 / VVC under the RA configuration. One of the main reasons for the poor performance of the intelligent bidirectional video coding method is that the long-term motion estimation between distant frames is not accurate, especially in large motion scenes. The existing intelligent bidirectional video coding method is trained on small groups of pictures (GoP) with small motion strength, but is tested on large GoP with various motion strengths. The domain shift problem between the training and testing scenes causes the motion estimation module to fail to handle long-term motion. Therefore, the existing intelligent video coding method performs well under the LD configuration, but performs poorly under the RA configuration. Currently, a coding scheme combining B frames and P frames is proposed to solve this problem. First, the P frame is encoded with a time step of 2, and then the intermediate B frame is encoded using a video interpolation method. Although this method keeps the distance from the reference frame unchanged and reduces the motion variability, it does not fully utilize the temporal correlation of the video, resulting in a decrease in compression performance. Another method is to select the most suitable frame type for each GoP. B frames are used for small motion GoP encoding, and P frames are used for large motion GoP encoding. The disadvantage is that the decision for each GoP introduces a time delay and has a high computational complexity. In addition, these two methods do not fundamentally improve the accuracy of long-term motion estimation.

[0063] The application provides a solution, which can more comprehensively capture the motion information of a video frame by calculating the forward accumulated optical flow between the video frame and a forward reference frame and the backward accumulated optical flow between the video frame and a backward reference frame, further determines the forward and backward inter-frame motion strength based on the optical flow information, and thus more accurately estimates the long-term motion relationship between the video frame and the reference frame, thereby improving the accuracy of long-term motion estimation.

[0064] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone or the like, or an electronic device capable of realizing the above functions. The embodiment and the following embodiments will be described below by taking an electronic device as an example.

[0065] Based on this, the embodiment of the application provides a video encoding method, which refers to Figure 1 , Figure 1 The flowchart of the first embodiment of the video encoding method of the application is shown in FIG. 1.

[0066] In the embodiment, the video encoding method includes steps S10-S40:

[0067] In step S10, the original video sequence is divided into multiple groups of images, and for any video frame in any group of images, the forward accumulated optical flow between the video frame and a forward reference frame is determined, and the backward accumulated optical flow between the video frame and a backward reference frame is determined;

[0068] The original video sequence refers to a set of continuous video frames without compression or encoding. The group of images refers to a set of continuous video frames divided from the original video sequence. In video encoding, the group of images is taken as the basic unit of encoding. Each group of images contains one or more key frames (I frames) and non-key frames (such as P frames and B frames). The key frames can be decoded independently, while the non-key frames need to be decoded by relying on other frames (such as the previous I frames or P frames). In the present embodiment, any video frame in any group of images is a non-key frame in the group of images. It should be noted that the length and composition of the group of images can be adjusted according to the encoding strategy and video content, which is not limited herein. Exemplarily, in a group of images including 32 frames of images, the process of frame encoding is as follows: the 1st frame in the group of images is taken as the forward reference frame, the 1st frame in the next adjacent group of images (i.e. the 33rd frame in the video sequence) is taken as the backward reference frame, and the 17th frame is encoded, wherein the 1st frame and the 33rd frame are taken as key frames for encoding; the 1st frame in the group of images is taken as the forward reference frame, and the 9th frame is encoded by taking the 17th frame as the backward reference frame; the 25th frame is encoded by taking the 17th frame as the forward reference frame and the 33rd frame as the backward reference frame; and so on, until the 32 frames of video frames are all encoded. It should be noted that for the last group of images, there is no next adjacent group, and the last frame of the group of images can be encoded in one direction by taking the first frame of the group of images as the reference frame, and then the above-mentioned encoding process is performed by taking the encoded last frame as the reference frame.

[0069] The forward reference frame refers to a frame used for forward prediction of a video frame in the video encoding process. The forward accumulated optical flow can reflect the overall motion trend of the pixels in the image from the forward reference frame to the video frame. The backward reference frame refers to a frame used for backward prediction of a video frame in the video encoding process. The backward accumulated optical flow can reflect the overall motion trend of the pixels in the image from the backward reference frame to the video frame.

[0070] In step S20, the forward inter-frame motion intensity between the video frame and the forward reference frame is determined based on the forward accumulated optical flow, and the backward inter-frame motion intensity between the video frame and the backward reference frame is determined based on the backward accumulated optical flow.

[0071] The forward inter-frame motion intensity refers to the motion intensity or the degree of motion change between the video frame and the forward reference frame. The forward inter-frame motion intensity includes high intensity and low intensity. The backward inter-frame motion intensity refers to the motion intensity or the degree of motion change between the video frame and the backward reference frame. The backward inter-frame motion intensity includes high intensity and low intensity.

[0072] According to the amplitude or speed of the forward cumulative optical flow, the forward inter-frame motion intensity between the video frame and the forward reference frame is calculated, and similarly, according to the amplitude or speed of the backward cumulative optical flow, the backward inter-frame motion intensity between the video frame and the backward reference frame is calculated. By calculating the inter-frame motion intensity, a basis can be provided for motion estimation and encoding in subsequent steps.

[0073] In step S30, the forward motion estimation result between the video frame and the reference frame is determined based on the forward inter-frame motion intensity, and the backward motion estimation result between the video frame and the reference frame is determined based on the backward inter-frame motion intensity.

[0074] The forward motion estimation result refers to the optical flow output by the optical flow estimation network (such as SpyNet) based on the forward inter-frame motion intensity, and the backward motion estimation result refers to the optical flow output by the optical flow estimation network based on the backward inter-frame motion intensity.

[0075] In step S40, the video frame is encoded based on the forward motion estimation result and the backward motion estimation result to obtain the video encoding data corresponding to the video frame.

[0076] According to the forward and backward motion estimation results and information such as forward and backward reference frames, the video frame is encoded, and the specific encoding process is not described here. After encoding, the video encoding data corresponding to the video frame is generated for subsequent storage and transmission. By encoding based on the motion estimation result, the redundancy information between video frames can be removed more effectively, thereby improving the encoding efficiency and compression ratio.

[0077] In an embodiment, the step S40 of encoding the video frame based on the forward motion estimation result and the backward motion estimation result to obtain the video encoding data corresponding to the video frame comprises:

[0078] In step S401, the encoding is performed based on the forward motion estimation result and the backward motion estimation result to obtain encoded optical flow data, and the decoded optical flow data is obtained by decoding the encoded optical flow data.

[0079] According to the motion vectors obtained by the forward and backward motion estimation, the optical flow field data is generated, and the encoded optical flow data is obtained by compressing the optical flow field data. Then, the encoded optical flow data is decoded to restore the original or approximately original optical flow field information to obtain the decoded optical flow data. By encoding and decoding the optical flow data, the motion information between video frames can be efficiently transmitted during transmission to meet the needs of video data transmission and processing.

[0080] This embodiment gives an original video sequence X = {x0, x1, …, x t ,…}, wherein x tLet X represent the frame at time t. Divide X into several GoPs and compress them separately. Each GoP contains 32 video frames. The Kth GoP unit X... K All video frames are taken as input, and the video frame x is estimated. t With two original reference frames x t-i ,x t+i bidirectional optical flow For the estimated original motion Perform joint encoding and transmission to decode and obtain the reconstructed motion. The optical flow codec is based on an autoencoder.

[0081] Step S402: Based on the decoded optical flow data, the forward reference features of the forward reference frame, and the backward reference features of the backward reference frame, a multi-scale temporal context is generated.

[0082] Feature information, such as edges, textures, and color histograms, is extracted from forward and backward reference frames. Combined with decoded optical flow data and reference frame features, multi-scale analysis methods (such as pyramid structures and wavelet transforms) are employed to construct contextual information at different time scales. Specifically, based on the decoded motion... Twisted Transformation Reference Features To generate multi-scale temporal context

[0083]

[0084] Step S403: Calculate the difference between the multi-scale temporal context and the video frame to obtain residual information;

[0085] With multi-scale temporal context As a condition, video frame x t Compressed and decoded by B-frame codec into intermediate feature F t and reconstructed frames Among them, F t It is referenced when encoding or decoding the next frame.

[0086] Step S404: The forward motion estimation, the backward motion estimation, and the residual information are determined as the video encoded data corresponding to the video frame.

[0087] Forward motion estimation, backward motion estimation, and residual information are used as the video encoded data corresponding to the video frame. It is understood that residual information reflects information loss and reconstruction errors during the encoding process. Using encoded video data and residual information together as video encoded data can improve the quality of the decoded video and encoding efficiency.

[0088] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and the subsequent will not be described. On this basis, the step S10: determining the forward accumulated optical flow between the video frame and the forward reference frame, comprising:

[0089] Step S101, based on the first local optical flow between the video frame and the first adjacent frame, the second local optical flow between the first adjacent frame and the second adjacent frame is twisted to obtain the twisted optical flow, and the twisted optical flow and the second local optical flow are superimposed to obtain the accumulated optical flow between the video frame and the second adjacent frame, wherein the first adjacent frame is a forward video frame adjacent to the video frame, and the second adjacent frame is a forward video frame adjacent to the first adjacent frame;

[0090] Based on the local optical flow information between the video frame and its directly adjacent forward video frame (first adjacent frame), the local optical flow between the first adjacent frame and its previous video frame (second adjacent frame) is twisted to simulate the propagation of optical flow between consecutive frames. Then, the twisted optical flow and the original local optical flow between the second adjacent frame are superimposed to obtain the accumulated optical flow between the video frame and the second adjacent frame.

[0091] Step S102, the accumulated optical flow between the video frame and the second adjacent frame is updated as the first local optical flow, and the third local optical flow between the second adjacent frame and the third adjacent frame is updated as the second local optical flow, the step of performing the first local optical flow between the video frame and the first adjacent frame, and the second local optical flow between the first adjacent frame and the second adjacent frame is twisted to obtain the twisted optical flow and the subsequent step are executed until the forward accumulated optical flow between the video frame and the forward reference frame is obtained, wherein the third adjacent frame is a forward video frame adjacent to the second adjacent frame.

[0092] At the beginning of each iteration, the local optical flow between the current video frame and the adjacent frame is updated as the last part of the accumulated optical flow calculated in the last iteration (i.e. the local optical flow of the previous frame directly adjacent to the current video frame), and the local optical flow between the next adjacent frame and the next adjacent frame is updated as the local optical flow between the adjacent frames considered in the current iteration. Based on the updated local optical flow information, the twisting and superimposing operations are repeated to calculate the accumulated optical flow between the current video frame and the next adjacent frame, until the forward reference frame is reached, at which time the obtained accumulated optical flow is the forward accumulated optical flow between the video frame and the forward reference frame.

[0093] To estimate long-term motion accurately, the embodiment proposes a cumulative optical flow estimation method, i.e. recursively accumulating local optical flow between adjacent frames. For example, directly estimate local optical flow {v 1→0 ,v 2→1 ,v 3→2 ,v 4→3} between original adjacent frames {x0, x1, x2, x3, x4}. 2→0 In step 1, to obtain the optical flow v 1→0 between x2 and x0, v 2→1 is warped to v 2→0 by:

[0094] v 1→0 = W(v 2→1 , v 2→1

[0095] where W represents the warping operation. 4→0 In step 2 and step 3, recursively accumulate to generate the target optical flow v 3→0 :

[0096] v 2→0 = W(v 3→2 , v 3→2

[0097] v 4→0 = W(v 3→0 , v 4→3 )+v 4→3

[0098] In a feasible embodiment, after the step S10 of determining the forward cumulative optical flow between the video frame and the forward reference frame, the method further comprises:

[0099] Step S50, performing a warping operation based on the forward reference frame to obtain a predicted reference frame.

[0100] Due to the influence of occlusion, the optical flow accumulation process will bring cumulative error, resulting in inaccurate optical flow estimation in non-occluded areas. To solve this problem, the embodiment designs an optical flow refinement (OFR) module to correct the error in each accumulation process.

[0101] Specifically, based on the forward cumulative optical flow information, a warping operation is performed on the forward reference frame to generate a predicted reference frame. The predicted reference frame is an attempt to simulate the possible state of the video frame on the forward time axis, providing an approximate representation of the video frame based on forward motion for subsequent motion estimation and encoding.

[0102] Step S60, input the prediction reference frame, the forward reference frame and the forward accumulated optical flow into a preset convolution layer to obtain a first processing result;

[0103] The prediction reference frame, the forward reference frame and the forward accumulated optical flow are input and processed through a preset convolution layer to extract features and generate a first processing result. Feature fusion can capture the internal relationship between the prediction reference frame, the forward reference frame and the forward accumulated optical flow, and improve the accuracy of subsequent processing.

[0104] Step S70, superimpose the first processing result and the forward accumulated optical flow to obtain a second processing result, update the forward accumulated optical flow through the second processing result, and perform the step of determining the forward inter-frame motion intensity between the video frame and the forward reference frame based on the forward accumulated optical flow.

[0105] The first processing result is pixel-level superimposed with the forward accumulated optical flow, and the superimposed result is output as a second processing result. The forward accumulated optical flow is updated according to the second processing result to reflect more accurate motion information. The updated forward accumulated optical flow is used to perform subsequent steps to improve overall encoding efficiency and video quality.

[0106] Specifically, based on the accumulated optical flow First, x t-i is warped to obtain the prediction frame t of the video frame x Then, a convolution network composed of 3 3x3 convolution layers is designed to refine the initial accumulated optical flow:

[0107]

[0108] Wherein, Conv represents a convolution layer.

[0109] In an embodiment, the step S20 of determining the forward inter-frame motion intensity between the video frame and the forward reference frame based on the forward accumulated optical flow comprises:

[0110] Step S201, map the forward accumulated optical flow to a motion intensity factor;

[0111] The forward accumulated optical flow (i.e. the accumulated motion information between the video frame and the forward reference frame) is converted into a quantitative index, i.e. a motion intensity factor, which is used to measure the motion intensity of the video frame relative to the forward reference frame. The specific mapping rule can be set according to actual needs, which is not limited here.

[0112] Step S202, if the motion intensity factor is greater than or equal to a preset intensity threshold, determining that the forward inter-frame motion intensity between the video frame and the forward reference frame is high intensity.

[0113] The motion intensity factor is compared with a preset intensity threshold. If the motion intensity factor is large, it is considered that the forward inter-frame motion intensity between the video frame and the forward reference frame is high intensity. The intensity threshold can be determined according to application scenarios and coding requirements, which is not limited here.

[0114] Step S203, if the motion intensity factor is less than the preset intensity threshold, determining that the forward inter-frame motion intensity is low intensity.

[0115] If the motion intensity factor is small, it is considered that the forward inter-frame motion intensity between the video frame and the forward reference frame is low intensity. It can be understood that by quantifying the motion intensity, comparing the preset threshold and making a judgment, an important basis is provided for the selection of subsequent coding strategies.

[0116] In a feasible embodiment, the step S201 of mapping the forward cumulative optical flow into a motion intensity factor comprises:

[0117] Step S2011, based on the forward cumulative optical flow, determining the cumulative horizontal offset and the cumulative vertical offset of each pixel point in the video frame;

[0118] The motion vector of each pixel point is extracted from the forward cumulative optical flow data, and the motion vector of each pixel point is decomposed into a horizontal component and a vertical component, representing the offset of the pixel point in the horizontal direction and the vertical direction, respectively. For each pixel point, the offsets of all frames in the forward time axis are accumulated to obtain the cumulative horizontal offset and the cumulative vertical offset.

[0119] Step S2012, traversing each pixel point, calculating the Euclidean norm of the cumulative horizontal offset and the cumulative vertical offset to obtain the cumulative offset length of the pixel point between the video frame and the forward reference frame;

[0120] Each pixel point in the video frame is traversed in turn. For each pixel point, the cumulative offset length is calculated using the Euclidean norm formula according to its cumulative horizontal offset and cumulative vertical offset. The cumulative offset length can reflect the overall motion distance of each pixel point relative to the forward reference frame, which provides a basis for subsequent calculation of the average optical flow intensity.

[0121] Step S2013, based on the cumulative offset length of each pixel point, obtaining a total offset length, and dividing the total offset length by the number of pixels of the video frame to obtain the average optical flow intensity of each pixel point, and determining the average optical flow intensity as the motion intensity factor.

[0122] The total offset length is obtained by adding the accumulated offset lengths of all the pixels, and the average optical flow intensity is obtained by dividing the total offset length by the number of pixels of the video frame. The calculated average optical flow intensity is output as the motion intensity factor.

[0123] That is, the motion intensity factor calculation method is defined as:

[0124]

[0125] wherein m x and m y represent the horizontal and vertical offsets of the accumulated optical flow, and H and W represent the height and width of the frame. If the average MIF of the bidirectional optical flow is less than a given threshold T, it is defined as small motion, and the direct optical flow estimation method is used, otherwise, it is defined as large motion and the accumulated-based optical flow estimation method is used. Exemplarily, the threshold T can be set to 10.

[0126] In an implementable embodiment, the step S30 of determining the forward motion estimation result between the video frame and the forward reference frame based on the forward inter-frame motion intensity comprises:

[0127] In step S301, if the forward inter-frame motion intensity is low intensity, the video frame and the forward reference frame are input into a preset optical flow network to obtain a forward direct optical flow, and the forward direct optical flow is determined as the forward motion estimation result between the video frame and the forward reference frame.

[0128] The video frame and the forward reference frame are input as input data into a preset optical flow network. The optical flow network extracts and matches the features of the input two frames of images through internal convolutional layers, activation functions and the like, and finally outputs the optical flow information between the two frames of images, i.e. the forward direct optical flow. As shown in FIG. 2, the direct estimation method directly estimates the optical flow between two frames of images x t ,x t-i and x Figure 6 The method performs well in small motion scenes.

[0129] In step S302, if the forward inter-frame motion intensity is high intensity, the forward accumulated optical flow is determined as the forward motion estimation result.

[0130] The forward accumulated optical flow that has been calculated is obtained, and the forward accumulated optical flow is directly used as the forward motion estimation result between the video frame and the forward reference frame. For high-intensity motion scenes, using the accumulated optical flow can more comprehensively capture the motion information and avoid errors that may occur in the optical flow network under complex motion.

[0131] ​It can be understood that different forward motion estimation methods are selected according to different forward inter-frame motion intensities. In a low-intensity scene, an optical flow network is used for accurate estimation; in a high-intensity scene, accumulated optical flow is directly used for estimation. According to the characteristics of different scenes, the appropriate motion estimation method is flexibly selected, which can improve the performance and accuracy of video coding.

[0132] Exemplarily, in order to help understand the implementation process of the video coding method based on the above embodiments, please refer to Figure 2 The intelligent bidirectional video coding method framework based on the present embodiment is shown in Figure 2 , and a given original video sequence X = {x0, x1, …, x t , …} is given, where x t represents the frame at time t. X is divided into several GoPs and compressed respectively, and one GoP contains 32 video frames. The intelligent bidirectional video coding method mainly consists of the following units:

[0133] Adaptive motion estimation (AME). The AME module takes all video frames of the Kth GoP unit X K as input, and estimates the bidirectional optical flow t between the current frame x t-i and two original reference frames x t+i In this implementation, SpyNet is used as the optical flow estimation network.

[0134] Optical flow codec. The estimated original motion is jointly encoded and transmitted to obtain the reconstructed motion where the optical flow codec is implemented based on the autoencoder.

[0135] Temporal context mining. Based on the decoded motion , the reference features are warped to generate multi-scale temporal contexts

[0136] B-frame codec. The current frame x is compressed and decoded into intermediate features F t and reconstructed frame t by the B-frame codec conditioned on the multi-scale temporal contexts where F t is referenced when encoding or decoding the next frame.

[0137] The AME method proposed in the present embodiment consists of two motion estimation methods: direct optical flow estimation and accumulated-based optical flow estimation, as shown in Figure 3 ​​As shown, the direct estimation method directly estimates the x values ​​of the two frames. t ,x t-i Input optical flow estimation network to estimate optical flow This method performs well in scenes with minimal motion. To accurately estimate long-term motion, this embodiment proposes an accumulated optical flow estimation method, which recursively accumulates the local optical flow between adjacent frames. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 An example of the optical flow accumulation process for estimating the optical flow between x4 and x0 is provided. First, the local optical flow {v} between the original adjacent frames {x0, x1, x2, x3, x4} is directly estimated. 1→0 ,v 2→1 ,v 3→2 ,v 4→3 In step 1, in order to obtain the optical flow v between x2 and x0... 2→0 v 1→0 Twisted to v 2→1 :

[0138] v 2→0 =W(v) 1→0 ,v 2→1 )+v 2→1

[0139] Here, W represents the warping operation. In steps 2 and 3, the target optical flow v is generated recursively by accumulating the following steps: 4→0 :

[0140] v 3→0 =W(v) 2→0 ,v 3→2 )+v 3→2

[0141] v 4→0 =W(v) 3→0 ,v 4→3 )+v 4→3

[0142] The framework diagram of the cumulative optical flow estimation method is as follows: Figure 5 As shown in the diagram. Due to the influence of occlusion, the optical flow accumulation process introduces accumulation errors, leading to inaccurate optical flow estimation in unoccluded areas. To address this issue, this embodiment also designs an Optical Flow Refinement (OFR) module to correct the errors in each accumulation process. The detailed network structure of the optical flow refinement module is shown in the diagram. Figure 6 As shown. Based on the accumulated optical flow First, regarding x t-i Perform warping to obtain the current frame x. t Predicted frames Then, a convolutional network consisting of 3 3x3 convolutional layers is designed to refine the initial accumulated optical flow:

[0143]

[0144] where Conv denotes a convolutional layer.

[0145] Meanwhile, a motion intensity factor (MIF) is defined to evaluate the motion intensity between two frames, which is calculated as:

[0146]

[0147] where m x and m y denote the horizontal and vertical offsets of the accumulated optical flow, and H and W denote the height and width of the frame. If the average MIF of the bi-directional optical flow is less than a given threshold T, it is defined as small motion and the direct optical flow estimation method is used. Otherwise, it is defined as large motion and the accumulated-based optical flow estimation method is used. In this embodiment, the threshold T is set to 10, Figure 7 A flowchart of the adaptive motion estimation method is shown.

[0148] However, as shown in Figure 8 , with the increase of the time interval, the accuracy of optical flow estimation drops severely due to the domain shift problem and even becomes unacceptable, as shown in Figure 9 , the AME module proposed in this embodiment achieves accurate motion estimation and prediction results in both small and large motion scenarios.

[0149] Ablation experiments were conducted on the UVG, MCL-JCV, and HEVC datasets to demonstrate the effectiveness of this embodiment, which contain videos with different motion intensities and resolutions and are widely used to evaluate the performance of intelligent video coding methods. Table 1 calculates the BD-Rate (%) with PSNR as the indicator. Method A uses direct optical flow estimation and is set as the anchor. This embodiment replaces the optical flow estimation method of method A with adaptive motion estimation. By comparison, it can be found that the AME method proposed in this embodiment achieves an average code rate saving of 7.9%, which proves the effectiveness of the AME method proposed in this embodiment. Since the AME method uses direct estimation for small motion, no gain is obtained on the HEVC E dataset, which only contains small motion.

[0150] Table 1: Ablation experiments on different datasets

[0151]

[0152] In addition, the embodiment also conducts an ablation study on video sequences of different motion types. Among them, the direct estimation method is set as the anchor point. As shown in Table 2, the cumulative-based optical flow estimation method brings performance improvement in large motion scenes, but suffers performance loss in small motion scenes. Adding the optical flow refinement module to the cumulative-based method can bring obvious performance improvement. This is because the OFR module can effectively correct the cumulative error. The comparative experiment results show that the AME method proposed in the embodiment can effectively handle different intensity of motion.

[0153] Table 2: Ablation experiment on data sets of different motion intensity

[0154]

[0155]

[0156] In order to find the key factors that cause the low compression performance of the existing intelligent bidirectional video compression method, the motion estimation and prediction results of the prior art B-CANF are visualized in Figure 1 From the figure, it can be seen that the motion estimation of B-CANF performs well in small motion scenes. However, when the frame distance increases, B-CANF cannot effectively handle large motion between distant frames, resulting in inaccurate prediction.

[0157] It should be noted that the above examples are only for understanding the present application and do not constitute a limitation on the video encoding method of the present application. Further simple transformations based on this technical concept are within the scope of protection of the present application.

[0158] The present application also provides a video encoding system, please refer to Figure 10 , the video encoding system comprises:

[0159] An adaptive motion estimation module 10 is configured to divide an original video sequence into multiple groups of images, and for any video frame in any group of images, determine a forward cumulative optical flow between the video frame and a forward reference frame, and determine a backward cumulative optical flow between the video frame and a backward reference frame;

[0160] The adaptive motion estimation module 10 is further configured to determine a forward inter-frame motion intensity between the video frame and the forward reference frame based on the forward cumulative optical flow, and determine a backward inter-frame motion intensity between the video frame and the backward reference frame based on the backward cumulative optical flow;

[0161] The adaptive motion estimation module 10 is further configured to determine a forward motion estimation result between the video frame and the reference frame based on the forward inter-frame motion intensity, and determine a backward motion estimation result between the video frame and the reference frame based on the backward inter-frame motion intensity;

[0162] a video encoding module 20, configured to encode the video frame based on the forward motion estimation result and the backward motion estimation result to obtain video encoding data corresponding to the video frame.

[0163] Optionally, the adaptive motion estimation module 10 is further configured to:

[0164] based on a first local optical flow between the video frame and a first neighboring frame, performing a warping operation on a second local optical flow between the first neighboring frame and a second neighboring frame to obtain a warped optical flow, and superimposing the warped optical flow and the second local optical flow to obtain a cumulative optical flow between the video frame and the second neighboring frame, wherein the first neighboring frame is a forward video frame adjacent to the video frame, and the second neighboring frame is a forward video frame adjacent to the first neighboring frame.

[0165] updating the cumulative optical flow between the video frame and the second neighboring frame as the first local optical flow, and updating a third local optical flow between the second neighboring frame and a third neighboring frame as the second local optical flow, performing the step of based on a first local optical flow between the video frame and a first neighboring frame, performing a warping operation on a second local optical flow between the first neighboring frame and a second neighboring frame to obtain a warped optical flow and subsequent steps until a forward cumulative optical flow between the video frame and the forward reference frame is obtained, wherein the third neighboring frame is a forward video frame adjacent to the second neighboring frame.

[0166] Optionally, the adaptive motion estimation module 10 is further configured to:

[0167] performing a warping operation based on the forward reference frame to obtain a predicted reference frame;

[0168] inputting the predicted reference frame, the forward reference frame and the forward cumulative optical flow into a preset convolution layer to obtain a first processing result;

[0169] superimposing the first processing result and the forward cumulative optical flow to obtain a second processing result, updating the forward cumulative optical flow through the second processing result, and performing the step of determining a forward inter-frame motion strength between the video frame and the forward reference frame based on the forward cumulative optical flow.

[0170] the adaptive motion estimation module 10 is further configured to:

[0171] determining a forward inter-frame motion strength between the video frame and the forward reference frame based on the second processing result.

[0172] Optionally, the adaptive motion estimation module 10 is further configured to:

[0173] mapping the forward accumulated optical flow as a motion intensity factor;

[0174] if the motion intensity factor is greater than or equal to a preset intensity threshold, determining that a forward inter-frame motion intensity between the video frame and the forward reference frame is high intensity;

[0175] if the motion intensity factor is less than the preset intensity threshold, determining that the forward inter-frame motion intensity is low intensity.

[0176] Optionally, the adaptive motion estimation module 10 is further configured to:

[0177] determine respective accumulated horizontal and vertical offsets of each pixel point in the video frame based on the forward accumulated optical flow;

[0178] traverse each of the pixel points, calculate a Euclidean norm of the accumulated horizontal and vertical offsets to obtain an accumulated offset length of the pixel point between the video frame and the forward reference frame;

[0179] obtain a total offset length based on the accumulated offset lengths of each of the pixel points, divide the total offset length by a number of pixels of the video frame to obtain an average optical flow intensity of each of the pixel points, and determine the average optical flow intensity as the motion intensity factor.

[0180] Optionally, the adaptive motion estimation module 10 is further configured to:

[0181] if the forward inter-frame motion intensity is low intensity, input the video frame and the forward reference frame into a preset optical flow network to obtain a forward direct optical flow, and determine the forward direct optical flow as a forward motion estimation result between the video frame and the forward reference frame;

[0182] if the forward inter-frame motion intensity is high intensity, determine the forward accumulated optical flow as the forward motion estimation result.

[0183] Optionally, the video encoding module 20 comprises:

[0184] an optical flow encoding and decoding unit configured to encode based on the forward motion estimation result and the backward motion estimation result to obtain encoded optical flow data, and decode the encoded optical flow data to obtain decoded optical flow data;

[0185] a temporal context mining unit configured to generate a multi-scale temporal context based on the decoded optical flow data, a forward reference feature of the forward reference frame, and a backward reference feature of the backward reference frame;

[0186] a bidirectional frame encoding and decoding unit configured to calculate a difference between the video frame and the multi-scale temporal context to obtain residual information.

[0187] a video frame encoding unit configured to determine the forward motion estimation, the backward motion estimation and the residual information as video encoding data corresponding to the video frame.

[0188] The video encoding system provided by the present application adopts the video encoding method in the above embodiments, and can solve the technical problem that long-term motion estimation is inaccurate during bidirectional video encoding, and the encoding performance is affected. Compared with the prior art, the video encoding system provided by the present application has the same beneficial effects as the video encoding method provided by the above embodiments, and other technical features in the video encoding system are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0189] The present application provides an electronic device, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the video encoding method in the above embodiments.

[0190] Reference will now be made to the drawings, and specific examples thereof will be described hereinbelow. Figure 11 which shows a structural schematic diagram of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application can include, but is not limited to, mobile terminals such as notebook computers, digital broadcast receivers, PAD (Portable Application Description: tablet computer), PMP (Portable Media Player: portable multimedia player) and the like, and fixed terminals such as digital TV, desktop computers and the like. Figure 11 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0191] As shown in FIG. 1, the electronic device 100 can include a bus 110, a processor 120, a memory 130, an input device 140, an output device 150, a display device 160, a communication device 170 and the like. Figure 11As shown, the electronic device can include a processing device 1001 (e.g., a central processor, a graphics processor, etc.) that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for operation of the electronic device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate wirelessly or wired with other devices to exchange data. Although the electronic device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0192] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0193] The electronic device provided in the present application adopts the video coding method in the above-mentioned embodiments, which can solve the technical problem that long-term motion estimation is inaccurate during bidirectional video coding, resulting in the coding performance being affected. Compared with the prior art, the electronic device provided in the present application has the same beneficial effects as the video coding method provided in the above-mentioned embodiments, and other technical features in the electronic device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0194] It is to be understood that the various parts of the disclosure can be implemented in hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0195] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any variations and modifications that can be made by any person skilled in the art within the spirit and scope of the application are intended to be encompassed by the application. The scope of the application is defined by the appended claims.

[0196] The application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e., a computer program) for performing the video encoding method in the above embodiments.

[0197] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or any combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), etc., or any suitable combination thereof.

[0198] The above computer readable storage medium can be contained in an electronic device or can exist separately without being assembled into an electronic device.

[0199] The above computer readable storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to implement the above embodiments of the video encoding method.

[0200] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0201] The flow diagrams and the block diagrams in the drawings are meant as methodological and functional description of implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0202] The modules involved in the embodiments of the present application can be implemented in software or hardware. In some cases, the names of the modules do not limit the modules themselves.

[0203] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the video encoding method described above, and can solve the technical problem that long-term motion estimation is inaccurate during bidirectional video encoding, and the encoding performance is affected. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the video encoding method provided by the above embodiments, and will not be described here.

[0204] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the video encoding method as described above.

[0205] The computer program product provided by the application can solve the technical problem that long-term motion estimation is inaccurate during bidirectional video encoding, thereby affecting the encoding performance. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the video encoding method provided by the above-described embodiments, and are not described herein.

[0206] The above only describes some embodiments of the application, and does not limit the patent scope of the application. Any equivalent structural transformation, direct / indirect application in other related technical fields, or the like made by using the content of the application specification and drawings within the technical concept of the application is included in the patent protection scope of the application.

Claims

1. A video encoding method, characterized in that, The video encoding method includes: The original video sequence is divided into multiple image groups. For any video frame in any image group, the forward cumulative optical flow between the video frame and the forward reference frame is determined, and the backward cumulative optical flow between the video frame and the backward reference frame is determined. The forward inter-frame motion intensity between the video frame and the forward reference frame is determined based on the forward cumulative optical flow, and the backward inter-frame motion intensity between the video frame and the backward reference frame is determined based on the backward cumulative optical flow. The forward motion estimation result between the video frame and the reference frame is determined based on the forward inter-frame motion intensity, and the backward motion estimation result between the video frame and the reference frame is determined based on the backward inter-frame motion intensity. The video frame is encoded based on the forward motion estimation result and the backward motion estimation result to obtain the video encoded data corresponding to the video frame; The step of determining the forward inter-frame motion intensity between the video frame and the forward reference frame based on the forward accumulated optical flow includes: The forward cumulative optical flow is mapped to a motion intensity factor; If the motion intensity factor is greater than or equal to a preset intensity threshold, then the forward inter-frame motion intensity between the video frame and the forward reference frame is determined to be high intensity. If the motion intensity factor is less than the preset intensity threshold, then the forward inter-frame motion intensity is determined to be low. The step of determining the forward motion estimation result between the video frame and the reference frame based on the forward inter-frame motion intensity includes: If the forward inter-frame motion intensity is low, the video frame and the forward reference frame are input into a preset optical flow network to obtain forward direct optical flow, and the forward direct optical flow is determined as the forward motion estimation result between the video frame and the forward reference frame. If the forward inter-frame motion intensity is high, then the forward cumulative optical flow is determined as the forward motion estimation result.

2. The video encoding method as described in claim 1, characterized in that, The step of determining the forward cumulative optical flow between the video frame and the forward reference frame includes: Based on the first local optical flow between the video frame and the first adjacent frame, a distortion operation is performed on the second local optical flow between the first adjacent frame and the second adjacent frame to obtain a distorted optical flow. The distorted optical flow and the second local optical flow are superimposed to obtain the cumulative optical flow between the video frame and the second adjacent frame. The first adjacent frame is a forward video frame adjacent to the video frame, and the second adjacent frame is a forward video frame adjacent to the first adjacent frame. The cumulative optical flow between the video frame and the second adjacent frame is updated to the first local optical flow, and the third local optical flow between the second adjacent frame and the third adjacent frame is updated to the second local optical flow. The steps of performing a distortion operation on the second local optical flow between the first adjacent frame and the second adjacent frame based on the first local optical flow between the video frame and the first adjacent frame to obtain a distorted optical flow and subsequent steps are performed until the forward cumulative optical flow between the video frame and the forward reference frame is obtained, wherein the third adjacent frame is a forward video frame adjacent to the second adjacent frame.

3. The video encoding method as described in claim 2, characterized in that, After the step of determining the forward cumulative optical flow between the video frame and the forward reference frame, the method further includes: The prediction reference frame is obtained by performing a warping operation based on the forward reference frame. The predicted reference frame, the forward reference frame, and the forward cumulative optical flow are input into a preset convolutional layer to obtain the first processing result; The first processing result and the forward cumulative optical flow are superimposed to obtain the second processing result. The forward cumulative optical flow is updated using the second processing result, and the step of determining the forward inter-frame motion intensity between the video frame and the forward reference frame based on the forward cumulative optical flow is performed.

4. The video encoding method as described in claim 1, characterized in that, The step of mapping the forward cumulative optical flow to a motion intensity factor includes: Based on the forward cumulative optical flow, the cumulative horizontal offset and cumulative vertical offset of each pixel in the video frame are determined. Traverse each pixel, calculate the Euclidean norm of the cumulative horizontal offset and the cumulative vertical offset, and obtain the cumulative offset length of the pixel between the video frame and the forward reference frame; The total offset length is obtained based on the cumulative offset length of each pixel. The average optical flow intensity of each pixel is obtained by dividing the total offset length by the number of pixels in the video frame. The average optical flow intensity is then determined as the motion intensity factor.

5. The video encoding method according to any one of claims 1 to 4, characterized in that, The step of encoding the video frame based on the forward motion estimation result and the backward motion estimation result to obtain the video encoded data corresponding to the video frame includes: Encoded optical flow data is obtained by encoding the forward motion estimation results and the backward motion estimation results, and decoded optical flow data is obtained by decoding the encoded optical flow data. Based on the decoded optical flow data, the forward reference features of the forward reference frame, and the backward reference features of the backward reference frame, a multi-scale temporal context is generated. The residual information is obtained by calculating the difference between the multi-scale temporal context and the video frame; The forward motion estimation, the backward motion estimation, and the residual information are determined as the video encoded data corresponding to the video frame.

6. A video encoding system, characterized in that, The video encoding system includes: An adaptive motion estimation module is used to divide the original video sequence into multiple image groups, and for any video frame in any image group, determine the forward cumulative optical flow between the video frame and the forward reference frame, and determine the backward cumulative optical flow between the video frame and the backward reference frame. The adaptive motion estimation module is further configured to determine the forward inter-frame motion intensity between the video frame and the forward reference frame based on the forward cumulative optical flow, and to determine the backward inter-frame motion intensity between the video frame and the backward reference frame based on the backward cumulative optical flow. The adaptive motion estimation module is further configured to determine the forward motion estimation result between the video frame and the reference frame based on the forward inter-frame motion intensity, and to determine the backward motion estimation result between the video frame and the reference frame based on the backward inter-frame motion intensity. The video encoding module is used to encode the video frame based on the forward motion estimation result and the backward motion estimation result to obtain the video encoded data corresponding to the video frame; The adaptive motion estimation module is further configured to: map the forward cumulative optical flow to a motion intensity factor; if the motion intensity factor is greater than or equal to a preset intensity threshold, determine that the forward inter-frame motion intensity between the video frame and the forward reference frame is high intensity; if the motion intensity factor is less than the preset intensity threshold, determine that the forward inter-frame motion intensity is low intensity. The adaptive motion estimation module is further configured to: if the forward inter-frame motion intensity is low, input the video frame and the forward reference frame into a preset optical flow network to obtain forward direct optical flow, and determine the forward direct optical flow as the forward motion estimation result between the video frame and the forward reference frame; if the forward inter-frame motion intensity is high, determine the forward cumulative optical flow as the forward motion estimation result.

7. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video encoding method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the video encoding method as described in any one of claims 1 to 5.

Images