Deep learning video compression method and system based on global-to-local motion estimation

Abstract

The invention discloses a deep learning video compression method and system based on global-to-local motion estimation. The method comprises the steps of efficiently compressing global motion generated by view angle transformation in flight of an unmanned aerial vehicle through homography transformation; integrating the obtained global motion as prior into a multi-scale structure so as to estimate and refine local object motion; carrying out compression coding on the local motion information, and synthesizing the local motion information with global motion for motion compensation to generate a prediction frame; the residual error of the prediction frame and the current frame is calculated in the feature domain, and after the residual error is compressed and coded, the residual error and the prediction frame are recovered into an uncompressed frame through the feature domain; and the obtained homography transformation parameter and the compressed and coded local motion information and residual information jointly form a video compressed code stream. According to the method, the global motion and the local motion are separated and cooperatively processed, so that the motion estimation precision of the unmanned aerial vehicle visual angle video is remarkably improved, and the bit rate consumption of residual information and the overall code stream is effectively reduced while the compression quality is ensured.

Description

Technical Field

[0001] This invention relates to the field of video compression technology, and specifically to a deep learning video compression method and system based on global-to-local motion estimation. Background Technology

[0002] The core of current video compression technology lies in eliminating spatiotemporal redundancy to achieve efficient bitrate control. Traditional coding standards, such as H.264 / AVC, HEVC / H.265, and VVC / H.266, generally employ block-matching motion estimation and transform coding techniques. However, these methods face fundamental limitations when processing complex dynamic scenes captured by unmanned aerial vehicles (UAVs). The uniqueness of UAV video lies in the simultaneous presence of global perspective motion caused by changes in the UAV platform's attitude (such as pitch, yaw, and roll), and local displacements formed by independent movements of objects within the scene. These two types of motion are coupled, constituting an extremely complex motion pattern. Traditional block-matching methods, based on the assumption of local rigid motion, struggle to accurately capture this complex interaction between global and local motion, resulting in insufficient motion modeling capabilities, significantly increased residuals after motion compensation, and consequently, a substantial increase in bitrate consumption. Especially when the UAV undergoes drastic attitude changes, the performance of traditional methods deteriorates sharply.

[0003] In recent years, deep learning-based video compression methods, such as DVC, DCVC, and various end-to-end optimized neural network compression schemes, have made some progress in compression performance. However, these methods still have serious shortcomings in motion estimation. Existing deep learning schemes typically use optical flow networks directly to estimate overall motion, lacking the distinction between global motion dominance and local motion details, resulting in the model's inability to effectively learn and differentiate between these two different motion modes. This single motion estimation strategy means that even powerful deep learning models still suffer from insufficient motion estimation capabilities when dealing with videos from moving perspectives, ultimately leading to unnecessary bit consumption. For example, a few independently moving objects in a scene can easily be mistaken for part of the global motion, resulting in redundant encoded information. Furthermore, optical flow estimation itself is prone to inaccurate estimation in scenes with high texture loss or drastic lighting changes, further affecting compression performance.

[0004] Homography transformation, a core tool in computer vision for modeling planar projection motion, effectively describes global motion caused by changes in camera viewpoint. Traditional implementations rely on feature point matching and direct linear transformation (DLT) algorithms, but their performance degrades drastically and robustness is poor in scenarios with sparse features, simple textures, or occlusion. In recent years, deep learning-enabled homography estimation methods, which learn the homography matrix between images through convolutional neural networks, have demonstrated strong performance and robustness in image registration, panoramic stitching, and camera pose estimation, and have been widely applied. However, regrettably, homography transformation has not yet been effectively incorporated into video compression frameworks to address the problem of global motion modeling. Introducing it into video compression can accurately capture global motion in drone videos, reduce residuals, and significantly improve compression efficiency. Combining homography estimation with optical flow estimation, and distinguishing and processing global and local motion separately, promises to achieve more efficient video compression.

[0005] The invention disclosed in CN111656401A is a video compression method using motion warping based on learning-based motion segmentation. It mentions using a homography-based global motion model to warp the texture-coded region in a reference frame to synthesize the texture-coded region in the current frame. However, this invention only uses the homography model as an optional encoding tool for specific "texture-coded regions." Its core idea is to divide image regions based on classification and apply different encoding strategies, but it does not solve the problem of distinguishing and co-estimating global and local motion in UAV scenarios. This method cannot effectively separate the global motion caused by the UAV's own flight attitude changes from the local motion of independent objects within the scene, resulting in inaccurate modeling of complex composite motion patterns in UAV-view videos. Therefore, this invention naturally cannot solve the core problems of inaccurate motion estimation, high residual information energy, and high bit rate consumption in UAV-view video compression. Summary of the Invention

[0006] To address the problems of inaccurate motion estimation, high bit rate consumption, and low compression efficiency caused by the large number of complex motions in drone videos when processing videos from a drone's perspective, this invention provides a deep learning-based video compression method and system based on global-to-local motion estimation. This method can achieve accurate inter-frame motion estimation for videos from a moving perspective, effectively reducing bit rate consumption while improving compression quality.

[0007] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0008] A deep learning-based video compression method based on global-to-local motion estimation, the method comprising the following steps:

[0009] S1, perform homography transformation estimation on the input video frame sequence to obtain the global optical flow caused by the change in the UAV's viewpoint;

[0010] S2, construct a four-scale pyramid optical flow estimation network, downsample the global optical flow, the previous reconstructed frame, and the current frame step by step, and fuse the downsampled global optical flow as prior information with the local optical flow estimation of the previous scale at each scale layer. After multi-scale iterative optimization, the local motion optical flow caused by the motion of objects in the picture is obtained.

[0011] S3, the local motion optical flow is encoded and decoded to reconstruct it, and the reconstructed local motion optical flow is combined with the global optical flow to obtain the motion optical flow;

[0012] S4. Based on the obtained motion optical flow, the previous reconstructed frame is distorted to obtain a distorted frame, and then motion compensation is performed on the distorted frame to obtain the predicted frame.

[0013] S5: Extract features from the current frame and the predicted frame respectively, subtract the features of the current frame and the predicted frame to obtain residual features, encode and decode the residual features to obtain reconstructed residual features, add the reconstructed residual features to the predicted frame features to obtain reconstructed frame features, and obtain the current reconstructed frame based on the reconstructed frame features.

[0014] S6. The obtained homography transformation parameters, compressed local motion information, and residual information are combined to form the video compressed bitstream.

[0015] Furthermore, in step S1, a homography transformation network is used to estimate the homography transformation of the input video frame sequence;

[0016] The homography transformation network consists of a multi-scale Transformer network. First, a lightweight CNN converts the input frames into shallow features, followed by a multi-scale encoder to construct a feature pyramid. At each scale, a self-attention encoder captures cross-frame feature correspondences, and a learnable attention-based decoder aggregates global information using learnable weight tokens. Finally, it outputs eight basic flow field coefficients of the homography matrix describing camera viewpoint changes, thus obtaining the global optical flow. :

[0017]

[0018] in This represents the global optical flow, which is composed of eight basis vectors. This represents the homography transformation network function.

[0019] Furthermore, in step S2, the process of obtaining the local optical flow caused by the motion of objects in the image after multi-scale iterative optimization includes the following steps:

[0020] Global optical flow is added as an initial value to each scale, and then applied to the current frame using an average pooling layer. Previous Reconstructed Frame and global optical flow By downsampling at a ratio of 1 / 2 layer by layer, a four-layer optical flow pyramid is constructed, named from 1 to 4 from high scale to low scale.

[0021] For scale i, the global optical flow downsampling result at the current scale. Local optical flow at the previous scale Combined to obtain flow Then use the obtained stream Downsampling results of reconstructed frames at the current scale Twisting, and relative to Refine the local flow at the current scale to obtain the local optical flow at the current scale. :

[0022]

[0023] in This indicates a twist operation. It is an optical flow estimation module based on SpyNet; The value is 0; for scales i=1, 2, 3, the obtained Upsampled data is then used as input for the next scale; the local optical flow of the last scale is then... As a localized motion optical flow caused by the movement of objects in the image.

[0024] Step S3 further includes:

[0025] The local motion optical flow is encoded and decoded. Specifically, a latent representation of the local motion optical flow is extracted through a cascaded three convolutional layers. The latent representation is then passed through a super-prior encoder to generate corresponding probability distribution parameters. Finally, the probability distribution parameters of the latent representation are passed through a symmetrical deconvolution structure to recover the local motion optical flow. ;

[0026] The reconstructed local motion optical flow and global optical flow Synthesized into moving optical flow :

[0027] .

[0028] Step S5 further includes:

[0029] From the current frame respectively With the predicted frame Extract current frame features With predicted frame features The residual characteristics are calculated using the following formula: ;

[0030] For residual characteristics Encoding and decoding processes are performed. Latent representations of residual features are extracted through a cascaded three-layer convolutional architecture. These residual features are then processed by a hyper-prior encoder to generate corresponding probability distribution parameters. Finally, these probability distribution parameters are passed through a symmetrical deconvolutional structure to reconstruct the reconstructed residual features. ;

[0031] Reconstructing residual features after decoding With predicted frame features Adding them together yields the reconstructed frame features. :

[0032] ;

[0033] Based on reconstructed frame features Recover the reconstructed frame .

[0034] Secondly, the present invention discloses a deep learning video compression system based on global to local motion estimation, the system comprising a global motion estimation module, a local motion estimation module, a motion encoding / decoding module, a motion compensation module, a feature extraction module, a residual encoding / decoding module, and a video reconstruction module;

[0035] The global motion estimation module performs homography transformation estimation on the input video frame sequence to obtain the global optical flow caused by the change in the UAV's viewpoint;

[0036] The local motion estimation module constructs a four-scale pyramid optical flow estimation network, which downsamples the global optical flow, the previous reconstructed frame, and the current frame at each scale level. At each scale level, the downsampled global optical flow is used as prior information and fused with the local optical flow estimation of the previous scale. After multi-scale iterative optimization, the local motion optical flow caused by the motion of objects in the picture is obtained.

[0037] The motion encoding and decoding module encodes and decodes the local motion optical flow to reconstruct the local motion optical flow. The reconstructed local motion optical flow is combined with the global optical flow to obtain the motion optical flow. Based on the obtained motion optical flow, the previous reconstructed frame is distorted to obtain a distorted frame. Then, the motion compensation module is called to perform motion compensation on the distorted frame to obtain the prediction frame.

[0038] The feature extraction module extracts features from the current frame and the predicted frame respectively, to obtain the current frame features and the predicted frame features;

[0039] The residual encoding / decoding module subtracts the features of the current frame and the predicted frame to obtain residual features, and then performs encoding / decoding processing on the residual features to obtain reconstructed residual features. After adding the reconstructed residual features to the predicted frame features to obtain the reconstructed frame features, the video reconstruction module is called to obtain the current reconstructed frame.

[0040] The video reconstruction module is used to reconstruct high-quality video frames from compressed features. It fuses predicted frame features and residual features using a scalable UNet structure, employs an adaptive weighting mechanism to balance the contributions of different feature sources, and finally reconstructs high-quality feature representations step-by-step through feature refinement.

[0041] Furthermore, the motion compensation module employs feature distortion and feature fusion techniques. It uses optical flow to distort the features of the previous frame, aligns the features to the current frame position, and then concatenates the distorted features with the motion-compensated frame. Feature fusion is then performed through a convolutional network. Simultaneously, an attention mechanism is used to adaptively adjust the channel weights to highlight important feature information. Finally, a residual compensation signal is generated and added to the initial result to obtain the prediction frame.

[0042] Furthermore, the video reconstruction module fuses predicted frame features and residual features through a scalable UNet structure, adopts an adaptive weighting mechanism to balance the contributions of different feature sources, and finally reconstructs high-quality feature representations step by step through feature refinement, thereby reconstructing high-quality video frames from compressed features.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] First, the deep learning video compression method and system based on global-to-local motion estimation of this invention, targeting UAV motion view scenarios, divides the traditional single-step motion estimation of deep learning into two parts: global and local. The global motion estimation module is used to estimate the viewpoint change caused by the viewpoint motion, while the local motion estimation module is used to estimate the motion caused by the movement of other individuals in the scene. Furthermore, since the global motion estimation module uses homography estimation, its estimated motion only requires eight parameters to describe, and its bit rate consumption during encoding and decoding is negligible. This invention can improve motion estimation performance while reducing bit rate consumption for the same compression effect.

[0045] Second, the deep learning video compression method and system based on global to local motion estimation of the present invention utilizes a multi-scale Transformer network to accurately model the motion of the UAV from the perspective and transforms the homography matrix into an 8-parameter base flow field representation. The 8-parameter representation of global motion has almost zero bit overhead. Combined with multi-scale estimation of local motion, it effectively reduces the consumption of residual bit rate and achieves an average bit rate saving of 4.68% under the same compression quality. Global motion compression can be completed with almost no consumption of transmission bit rate. While effectively estimating the global motion caused by the viewpoint movement, it reduces the transmission burden of the communication link.

[0046] Third, the deep learning video compression method and system based on global-to-local motion estimation of this invention constructs a four-scale pyramid network. At each level, downsampled global optical flow is used as the motion prior. Local optical flow is iteratively optimized layer by layer across multiple scales, simultaneously enhancing object displacement details (such as ocean wave ripples) and correcting projection distortion in non-planar scenes. The separation estimation mechanism from global to local motion enhances motion estimation performance and maintains stable performance across various scenarios. More image details are preserved in scenes with vegetation cover and ocean ripples. Attached Figure Description

[0047] Figure 1 This is a schematic diagram illustrating an application scenario of the UAV video compression method based on global-to-local motion estimation according to the present invention.

[0048] Figure 2a A simplified diagram of a conventional motion estimation-based video compression method;

[0049] Figure 2b This is a simplified diagram of the UAV video compression method based on global-to-local motion estimation according to the present invention.

[0050] Figure 3 This is a system architecture diagram of the UAV video compression method based on global-to-local motion estimation of the present invention;

[0051] Figure 4 This is a schematic diagram of the local motion estimation module structure of the UAV video compression method based on global to local motion estimation of the present invention. Detailed Implementation

[0052] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0053] A deep learning-based video compression method based on global-to-local motion estimation, the method comprising the following steps:

[0054] S1, perform homography transformation estimation on the input video frame sequence to obtain the global optical flow caused by the change in the UAV's viewpoint;

[0055] S2, construct a four-scale pyramid optical flow estimation network, downsample the global optical flow, the previous reconstructed frame, and the current frame step by step, and fuse the downsampled global optical flow as prior information with the local optical flow estimation of the previous scale at each scale layer. After multi-scale iterative optimization, the local motion optical flow caused by the motion of objects in the picture is obtained.

[0056] S3, encode and decode the local optical flow to reconstruct the local motion optical flow; combine the reconstructed local motion optical flow with the global optical flow to obtain the motion optical flow;

[0057] S4. Based on the obtained motion optical flow, the previous reconstructed frame is distorted to obtain a distorted frame, and then motion compensation is performed on the distorted frame to obtain the predicted frame.

[0058] S5: Extract features from the current frame and the predicted frame respectively, subtract the features of the current frame and the predicted frame to obtain residual features, encode and decode the residual features to obtain reconstructed residual features, add the reconstructed residual features to the predicted frame features to obtain reconstructed frame features, and obtain the current reconstructed frame based on the reconstructed frame features.

[0059] Based on this, the present invention also discloses a deep learning video compression system based on global to local motion estimation, the system including a global motion estimation module, a local motion estimation module, a motion encoding and decoding module, a motion compensation module, a feature extraction module, a residual encoding and decoding module, and a video reconstruction module;

[0060] The global motion estimation module performs homography transformation estimation on the input video frame sequence to obtain the global optical flow caused by the change in the UAV's viewpoint;

[0061] The local motion estimation module constructs a four-scale pyramid optical flow estimation network, which downsamples the global optical flow, the previous reconstructed frame, and the current frame at each scale level. At each scale level, the downsampled global optical flow is used as prior information and fused with the local optical flow estimation of the previous scale. After multi-scale iterative optimization, the local motion optical flow caused by the motion of objects in the picture is obtained.

[0062] The motion encoding and decoding module encodes and decodes the local motion optical flow to reconstruct the local motion optical flow; after the reconstructed local motion optical flow is combined with the global optical flow to obtain the motion optical flow, the previous reconstructed frame is distorted based on the obtained motion optical flow to obtain a distorted frame, and then the motion compensation module is called to perform motion compensation on the distorted frame to obtain the prediction frame.

[0063] The motion compensation module employs feature warping and feature fusion techniques. It uses optical flow to warp the features from the previous frame, aligning them to the current frame position. The warped features are then concatenated with the motion-compensated frame, and feature fusion is performed through a convolutional network. Simultaneously, an attention mechanism is used to adaptively adjust channel weights, highlighting important feature information. Finally, a residual compensation signal is generated and added to the initial result to obtain the predicted frame.

[0064] The feature extraction module extracts features from the current frame and the predicted frame respectively, to obtain the current frame features and the predicted frame features;

[0065] The residual encoding / decoding module subtracts the features of the current frame and the predicted frame to obtain residual features, and then performs encoding / decoding processing on the residual features to obtain reconstructed residual features. After adding the reconstructed residual features to the predicted frame features to obtain the reconstructed frame features, the video reconstruction module is called to obtain the current reconstructed frame.

[0066] The video reconstruction module is used to reconstruct high-quality video frames from compressed features. It fuses predicted frame features and residual features using a scalable UNet structure, employs an adaptive weighting mechanism to balance the contributions of different feature sources, and finally reconstructs high-quality feature representations step-by-step through feature refinement.

[0067] Figure 2a This is a simplified diagram of a conventional motion estimation-based video compression method. Figure 2b This is a simplified diagram of the UAV video compression method based on global to local motion estimation according to the present invention. The core innovation of the present invention lies in the construction of a global to local motion separation estimation framework. Figure 3 This is a schematic diagram of the UAV video compression structure based on global-to-local motion estimation of this invention. By separating the viewpoint motion from the motion of objects in the video, a global motion estimation module estimates the viewpoint motion, and a local motion estimation module estimates the motion of objects in the scene, achieving efficient motion estimation. This end-to-end video compression framework can be divided into a compression end and a decompression end. The compression end includes all the compression processes shown in the diagram, while the decompression end only includes the process after obtaining the binary bitstream of motion information.

[0068] Specifically, using on the compression end and Representing the current frame and the previous reconstructed frame respectively, this method first... and Input homography transformation subnetwork Obtain global optical flow This network, based on multi-view geometry, predicts the homography matrix describing camera viewpoint changes through a multi-scale Transformer architecture. This process can be expressed by the following formula:

[0069]

[0070] in It represents a global optical flow composed of eight basis vectors. In actual transmission, only the parameters of the eight basis vectors need to be transmitted. Therefore, it does not need to be converted into binary code for transmission through encoding and decoding, which effectively controls the transmission bit rate consumption.

[0071] Subsequently, local motion estimation is performed to obtain the local motion. The optical flow estimation network of the pyramid Implementation. The input to this network is the current frame. Previous Reconstructed Frame and global optical flow This process can be expressed by the following formula:

[0072]

[0073] Specifically, at each scale level of the subnet, the local flow estimated from the previous scale is integrated with the global flow at the current scale for pre-warping. This integrated motion field is then refined using local optical flow, enhancing finer-grained details. Note that due to the inherent imperfections of monotonic estimation, this refinement also implicitly serves as a further correction to the global motion.

[0074] After motion estimation is completed, due to the eight-parameter characteristic of homography estimation, only the local motion needs to be fed into the motion encoding / decoding module for encoding. A binary bitstream is obtained through convolutional layers and a hyper-prior encoder. Simultaneously, while the binary bitstream is being transmitted to the decompression end, the local motion is recovered at the compression end through a deconvolution structure symmetrical to the encoding part. Used for subsequent compression processes. After this, Can be with Synthesized into the final motion optical flow :

[0075]

[0076] Based on the obtained motion optical flow For the previous reconstructed frame Perform a warp operation to obtain warp frames And generate prediction frames through the motion compensation module. The predicted frame obtained at this time Compared to the current actual frames There is still a large error, so more image details need to be corrected through residual correction.

[0077] Because the residuals themselves consume a large number of bits, this invention performs residual prediction within the feature domain. First, a feature extraction module is used to extract features from the current frame... With the predicted frame Extract the corresponding features, respectively using and This indicates that the residual is... The residual is obtained. This represents a comparison with the original image. Predicted frames The error message still exists. Subsequently, the residual is used to generate residual binary code using a residual codec with a structure similar to a motion codec, which is then transmitted to the decompression end. The compression end uses the decoded residual. With predicted frame features Adding them together yields the reconstructed frame features. :

[0078]

[0079] Reconstructing frame features The reconstructed frames can be recovered using the video reconstruction module. .

[0080] The decompression end only includes a portion of the compression end's process. Specifically, after obtaining the global motion, local motion bitstream, and residual feature bitstream from the compression end, the local motion bitstream and residual feature bitstream are respectively processed by the corresponding decoding networks to recover the local motion. With residual characteristics Subsequently, similar to the compression end, localized motion occurs. With global motion Synthesized into moving optical flow Based on the obtained motion optical flow For the previous reconstructed frame Perform a warp operation to obtain warp frames And generate prediction frames through the motion compensation module. .

[0081] Since the decoding end has already obtained the residual features after decoding, the feature extraction stage only needs to obtain the prediction frame through the feature extraction module. Features Then, the predicted features are used directly. With residual characteristics Adding them together yields the reconstructed frame features. Then let the reconstructed frame features The reconstructed frames can be obtained through the video reconstruction module. .

[0082] The various modules in the compression process are described in detail below:

[0083] 1. Global Motion Estimation Module: The compression framework employs HomoGAN, a deep learning-based homography estimation method. This network first converts input frames into shallow features using a lightweight CNN, then constructs a feature pyramid via a multi-scale encoder (gradually downsampling from the original resolution of 512×384 to 128×96). At each scale, a self-attention encoder captures cross-frame feature correspondences, while an attention-like decoder aggregates global information using learnable "weight tokens," ultimately outputting eight basic flow field coefficients describing the homography matrix. To overcome interference from multi-plane scenes, a coplanarity-aware adversarial mechanism is employed: a soft mask is generated for the network to predict aligned regions, and the network verifies whether the masked regions satisfy the homography coplanarity condition, forcing the model to focus on the dominant plane (such as the ground or building facade).

[0084] 2. Local Motion Estimation Module: Local motion estimation is performed by the pyramid optical subnetwork. The network structure is as follows: Figure 4 As shown, its basic structure is a pyramid-shaped optical flow estimation network based on SpyNet. Its core lies in continuously refining the motion optical flow as the scale decreases after predicting the optical flow at a high scale, thereby improving the motion estimation effect. To implement a motion estimation module oriented towards local motion, this invention incorporates global motion into each scale as an initial value. Specifically, it first uses an average pooling layer to estimate the current frame... Previous Reconstructed Frame and global motion By downsampling at a ratio of 50% layer by layer, a four-layer optical flow pyramid is constructed, named from 1 to 4 from high to low scale. For scale i, the global motion of the current scale is... Local motion at the previous scale Combined to obtain Then use the obtained stream For the previous frame To distort, and on this basis, relative to Refine the local flow at the current scale. The entire process can be represented as follows:

[0085]

[0086] in This indicates a twist operation. This is an optical flow estimation module based on SpyNet. It's important to note that for scales i=1, 2, and 3, the obtained... Upsampling is performed to ensure the scale meets the size requirements for use as input to the next scale, but this is not necessary for i=4 (the last scale). Meanwhile, since the scale at i=1 cannot obtain higher scale data... Therefore, Set to 0.

[0087] 3. Motion / Residual Encoding / Decoding Module: The motion encoding / decoding module and residual encoding / decoding module of this invention employ a probabilistic model based on a priori information to achieve efficient compression. For motion information (local optical flow)... ) / Residual characteristics The encoder part first extracts latent representations through a cascade of three convolutional layers. The subsequent super-prior module generates latent representations. The probability distribution parameters (mean) with standard deviation The prior module models the latent representation using a conditional Gaussian entropy model. The probability distribution is shown. The decoder uses a symmetric deconvolution structure to gradually recover the motion field.

[0088] The training process of this invention employs a two-stage collaborative optimization strategy: a pre-training stage and a fine-tuning stage. It's important to note that for the first part of both the pre-training and fine-tuning stages, to stabilize the training process, we do not initially introduce a local motion estimation network; instead, we use a SpyNet-based motion estimation network. During pre-training, we first train the global motion estimation network and the video compression network using natural image datasets and the Vimeo90k dataset, respectively. The pre-training stage primarily serves to provide a starting point for the subsequent integration of networks, simplifying the single-step training process.

[0089] After pre-training the two modules mentioned above, the global motion estimation module is combined with the video compression module, followed by a fine-tuning phase. The fine-tuning phase also employs a two-stage fine-tuning strategy to optimize model performance. The main goal in the initial fine-tuning phase is to simplify the training process while adapting parts of the network, except for the local optical flow prediction network, to the global-to-local motion separation estimation framework. During this fine-tuning step, the local optical flow prediction module is kept frozen. To enable the optical flow prediction network to initially predict local optical flow, a special frame is introduced in this example. The frame is the global motion obtained by the global motion estimation module. Compared with the previous reconstructed frame The direct combination yields the result, and the process can be expressed by the following formula:

[0090]

[0091] After this, the motion estimation module in the video compression network directly estimates... With the current frame The movement between them. Due to The text already includes global motion. ,therefore and The motion between these points constitutes the remaining local motion, and the optical flow obtained by the motion estimation module in this way is the relatively coarse local motion. The following loss function is used in this process:

[0092]

[0093] in and They represent local optical flow. With residual characteristics Bit rate consumed after encoding This represents the distortion between the current frame and the reconstructed frame. This represents the distortion between the current frame and the reconstructed frame. Parameters and As a rate-distortion tradeoff factor, in this example, Randomly selected from the set [1840, 920, 420, 180]. Pick One-tenth of that, i.e., [184, 92, 42, 18]. The learning rate for this stage is set to... .

[0094] After the first stage of fine-tuning adapts other modules in the framework to the global motion-to-local motion separation estimation framework, the next stage of fine-tuning aims to improve the accuracy of local optical flow prediction, effectively enhancing the overall compression performance. In this stage, the local motion estimation module is added to the fine-tuning process, while the special frames used in the previous stage are discontinued. And all modules except the local motion estimation module are frozen and trained separately for the local motion estimation module. The learning rate is set to [value missing] during this phase. Use the following loss function:

[0095]

[0096] In this stage, parameters Set as the previous stage The values ​​are the same, i.e., [1840, 920, 420, 180]. This loss function allows the network parameters to focus on the local motion extraction module, thereby maximizing the performance of the local motion extraction module in extracting local motion.

[0097] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0098] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A deep learning-based video compression method based on global-to-local motion estimation, characterized in that, The method includes the following steps: S1, perform homography transformation estimation on the input video frame sequence to obtain the global optical flow caused by the change in the UAV's viewpoint; S2, construct a four-scale pyramid optical flow estimation network, downsample the global optical flow, the previous reconstructed frame, and the current frame step by step, and fuse the downsampled global optical flow as prior information with the local optical flow estimation of the previous scale at each scale layer. After multi-scale iterative optimization, the local motion optical flow caused by the motion of objects in the picture is obtained. S3, the local motion optical flow is encoded and decoded to reconstruct it, and the reconstructed local motion optical flow is combined with the global optical flow to obtain the motion optical flow; S4. Based on the obtained motion optical flow, the previous reconstructed frame is distorted to obtain a distorted frame, and then motion compensation is performed on the distorted frame to obtain a predicted frame. S5: Extract features from the current frame and the predicted frame respectively, subtract the features of the current frame and the predicted frame to obtain residual features, encode and decode the residual features to obtain reconstructed residual features, add the reconstructed residual features to the predicted frame features to obtain reconstructed frame features, and obtain the current reconstructed frame based on the reconstructed frame features. S6. The obtained homography transformation parameters, compressed local motion information, and residual information are combined to form the video compressed bitstream.

2. The deep learning video compression method based on global-to-local motion estimation according to claim 1, characterized in that, In step S1, a homography transformation network is used to estimate the homography transformation of the input video frame sequence; The homography transformation network consists of a multi-scale Transformer network. First, a lightweight CNN converts the input frames into shallow features, followed by a multi-scale encoder to construct a feature pyramid. At each scale, a self-attention encoder captures cross-frame feature correspondences, and a learnable attention-based decoder aggregates global information using learnable weight tokens. Finally, it outputs eight basic flow field coefficients of the homography matrix describing camera viewpoint changes, thus obtaining the global optical flow. : ， in This represents the global optical flow, which is composed of eight basis vectors. This represents the homography transformation network function.

3. The deep learning video compression method based on global-to-local motion estimation according to claim 1, characterized in that, In step S2, the process of obtaining the local optical flow caused by the motion of objects in the image after multi-scale iterative optimization includes the following steps: Global optical flow is added as an initial value to each scale, and then applied to the current frame using an average pooling layer. Previous Reconstructed Frame and global optical flow By downsampling at a ratio of 1 / 2 layer by layer, a four-layer optical flow pyramid is constructed, named from 1 to 4 from high scale to low scale. For scale i, the global optical flow downsampling result at the current scale. Local optical flow at the previous scale Combined to obtain flow Then use the obtained stream Downsampling results of reconstructed frames at the current scale Twisting, and relative to Refine the local flow at the current scale to obtain the local optical flow at the current scale. : ， in This indicates a twist operation. It is an optical flow estimation module based on SpyNet; The value is 0; for scales i=1, 2, 3, the obtained Upsampled data is then used as input for the next scale; the local optical flow of the last scale is then... As a localized motion optical flow caused by the movement of objects in the image.

4. The deep learning video compression method based on global-to-local motion estimation according to claim 1, characterized in that, Step S3 further includes: The local motion optical flow is encoded and decoded. Specifically, a latent representation of the local motion optical flow is extracted through a cascaded three convolutional layers. The latent representation is then passed through a super-prior encoder to generate corresponding probability distribution parameters. Finally, the probability distribution parameters of the latent representation are passed through a symmetrical deconvolution structure to recover the local motion optical flow. ; The reconstructed local motion optical flow and global optical flow Synthesized into moving optical flow : 。 5. The deep learning video compression method based on global-to-local motion estimation according to claim 1, characterized in that, Step S5 further includes: From the current frame respectively With the predicted frame Extract current frame features With predicted frame features The residual characteristics are calculated using the following formula: ; For residual characteristics Encoding and decoding processes are performed. Latent representations of residual features are extracted through a cascaded three-layer convolutional architecture. These residual features are then processed by a hyper-prior encoder to generate corresponding probability distribution parameters. Finally, these probability distribution parameters are passed through a symmetrical deconvolutional structure to reconstruct the reconstructed residual features. ; Reconstructing residual features after decoding With predicted frame features Adding them together yields the reconstructed frame features. : ； Based on reconstructed frame features Recover the reconstructed frame .

6. A deep learning-based video compression system based on global-to-local motion estimation, characterized in that, The system includes a global motion estimation module, a local motion estimation module, a motion encoding / decoding module, a motion compensation module, a feature extraction module, a residual encoding / decoding module, and a video reconstruction module; The global motion estimation module performs homography transformation estimation on the input video frame sequence to obtain the global optical flow caused by the change in the UAV's viewpoint; The local motion estimation module constructs a four-scale pyramid optical flow estimation network, which downsamples the global optical flow, the previous reconstructed frame, and the current frame at each scale level. At each scale level, the downsampled global optical flow is used as prior information and fused with the local optical flow estimation of the previous scale. After multi-scale iterative optimization, the local motion optical flow caused by the motion of objects in the picture is obtained. The motion encoding and decoding module encodes and decodes the local motion optical flow to reconstruct the local motion optical flow. The reconstructed local motion optical flow is combined with the global optical flow to obtain the motion optical flow. Based on the obtained motion optical flow, the previous reconstructed frame is distorted to obtain a distorted frame. Then, the motion compensation module is called to perform motion compensation on the distorted frame to obtain the prediction frame. The feature extraction module extracts features from the current frame and the predicted frame respectively, to obtain the current frame features and the predicted frame features; The residual encoding / decoding module subtracts the features of the current frame and the predicted frame to obtain residual features, and then performs encoding / decoding processing on the residual features to obtain reconstructed residual features. After adding the reconstructed residual features to the predicted frame features to obtain the reconstructed frame features, the video reconstruction module is called to obtain the current reconstructed frame. The video reconstruction module is used to reconstruct high-quality video frames from compressed features. It fuses predicted frame features and residual features using a scalable UNet structure, employs an adaptive weighting mechanism to balance the contributions of different feature sources, and finally reconstructs high-quality feature representations step-by-step through feature refinement.

7. The deep learning video compression system based on global-to-local motion estimation according to claim 6, characterized in that, The motion compensation module employs feature distortion and feature fusion techniques. It uses optical flow to distort the features of the previous frame, aligning the features to the current frame position. Then, it concatenates the distorted features with the motion-compensated frame and performs feature fusion through a convolutional network. Simultaneously, it uses an attention mechanism to adaptively adjust channel weights to highlight important feature information. Finally, it generates a residual compensation signal, which is added to the initial result to obtain the predicted frame.

8. The deep learning video compression system based on global-to-local motion estimation according to claim 6, characterized in that, The video reconstruction module fuses predicted frame features and residual features through a scalable UNet structure, adopts an adaptive weighting mechanism to balance the contributions of different feature sources, and finally reconstructs high-quality feature representations step by step through feature refinement, thereby reconstructing high-quality video frames from compressed features.

Images