A video post-half frame real-time prediction method based on optical flow method
Patent Information
- Application Number
- CN202610412963.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-31
- Publication Date
- 2026-11-03
AI Technical Summary
The time delay in the wavefront correction process in airborne imaging systems leads to a decrease in correction accuracy, and existing video frame prediction methods struggle to predict the second half of the image at a high frame rate and with high quality within a single cycle.
A real-time prediction method for the second half of a video frame based on optical flow is adopted. By defining an optical flow estimation module and a thinning reconstruction module, image prediction is performed using multi-layer depthwise separable convolution and channel attention enhancement blocks. The network is trained by a combined loss function and the Adam optimization algorithm to output a high-quality second half-frame image.
It achieves high frame rate and high quality second-half frame image prediction, effectively reduces errors caused by computational latency, and improves the accuracy of fast mirror correction.
Smart Images
Figure CN123035303A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing and real-time control technology, specifically relating to a method for predicting the second half of a video frame to reduce the control delay of a fast-reflection mirror in an airborne imaging system. Background Technology
[0002] Airborne imaging systems typically employ deformable mirrors combined with highly sensitive, high-response sensors to achieve high-precision wavefront correction of the target area. However, there is a time delay in measuring wavefront error, calculating deformation correction amounts in software, and controlling the deformable mirror to complete the entire closed-loop process of deformation. This affects the system's correction accuracy. For the error at time t, the deformable mirror at time t... Only then is the correction applied. The inter-frame gap, which is greater than zero but less than a complete frame period, places demands on image processing speed.
[0003] Existing prediction-based compensation methods typically predict multiple consecutive frames. However, the prediction accuracy of such methods decreases significantly with increasing step size, and the output is inherently always lagging behind the actual time by a delay period, making it difficult to fundamentally eliminate the impact of the delay. Therefore, how to directly calculate and output the second half-frame image corresponding to the actual control activation time within a single processing cycle, thereby effectively reducing the error caused by computational delay, has become a key technical challenge for improving the accuracy of fast-reflection mirror correction. Summary of the Invention
[0004] The present invention aims to solve the technical problems in the prior art, such as the decrease in correction accuracy of airborne imaging systems due to time delay in the wavefront correction process, and the difficulty of existing video frame prediction methods in predicting the second half of the image at a high frame rate and with high quality within a single period.
[0005] Technical solution
[0006] To address the aforementioned technical problems, the present invention aims to provide a real-time prediction method for the second half of a video frame based on optical flow. This method calculates and outputs the second half-frame image corresponding to the actual effective time within a single processing cycle, effectively reducing the error caused by the calculation delay of the adaptive real-time controller in the airborne imaging system and improving the accuracy of fast-reflection mirror correction.
[0007] To achieve the above objectives, the technical solution adopted by this invention is: a real-time prediction method for the second half-frame of a video based on optical flow, comprising the following steps:
[0008] Step 1: Define a second-half-frame prediction network model based on optical flow. The network model includes an optical flow estimation module and a thinning reconstruction module. The optical flow estimation module employs a multi-layer depthwise separable convolutional structure, introducing channel attention enhancement blocks. It consists of three cascaded depthwise separable convolutional blocks (AttBlock) with channel attention, used to predict the forward optical flow field. and reverse optical flow field The refinement and reconstruction module is used to stitch the two sets of spatial alignment results with the corresponding optical flow fields. It achieves feature fusion and texture compensation through multi-layer separable convolution and channel attention mechanism, and introduces residual connections to aggregate spatiotemporal information.
[0009] Step 2: Randomly initialize the network parameters in the optical flow estimation module and the thinning reconstruction module;
[0010] Step 3: Define the loss function. A combined loss function is used to simultaneously constrain pixel accuracy and structural consistency. The expression is as follows:
[0011] (1)
[0012] (2)
[0013] (3)
[0014] (4)
[0015] in, Measuring pixel-level error This is based on Laplacian pyramid loss and is used to enhance high-frequency detail reconstruction. Structural similarity loss is used to improve the perceptual quality of predicted frames. and These are the weighting coefficients for each loss.
[0016] Step 4: Train the second half-frame prediction network. During the training process, the network parameters are updated using the Adam optimization algorithm.
[0017] Step 5: Combine two consecutive frames of images and Input the trained second-half-frame prediction network to directly predict the second-half-frame image. By comparing the generated image with the corresponding real second half-frame image, peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and frame rate (FPS) are calculated to measure the quality and real-time performance of the generated image.
[0018] Furthermore, in the optical flow estimation module, based on the short-time linear motion assumption, a time encoding parameter t is introduced, and the parameter t is set to 1.5 to correspond to the relative time position of the second half of the frame in the input sequence; the predicted bidirectional optical flow is linearly scaled to obtain the pixel displacement at the second half of the frame; and the scaled optical flow field is then used to... and Bilinear inverse sampling is performed to obtain two sets of spatial alignment results for the second half of the frame. The intermediate results obtained by bidirectional inverse deformation are input into the reconstruction network, and the network adaptively learns the fusion weights of optical flow in different directions at the texture details during the end-to-end training process.
[0019] Furthermore, the refinement and reconstruction module performs channel-dimensional stitching of the two sets of alignment results with the corresponding optical flow fields, achieving fusion through multi-layer separable convolution and channel attention mechanisms. It also introduces local convolution and residual connections to aggregate spatiotemporal information, mitigating the blurring and artifact problems caused by optical flow errors, and finally outputs the predicted image of the second half frame. .
[0020] Furthermore, in the combined loss function, For L1 loss, For the loss based on the Laplace pyramid, For SSIM loss, the weighting coefficients are set to , .
[0021] Furthermore, the expression for PSNR is as follows:
[0022] (5)
[0023] Where: n=8 represents the number of binary bits for each pixel in the image, and MSE is the mean square error between the real image and the generated image.
[0024] Wherein, MSE is the mean square error between the original image and the output image, and the definition of MSE is shown in the following formula:
[0025] (6)
[0026] in, and Reference regions for a given frame of the original image and the output image, respectively. The pixel value at the location is given, where m and n are the length and width of the image, respectively.
[0027] Furthermore, the expression for SSIM is as follows:
[0028] (7)
[0029] in: , This represents two images used for evaluation. express The average value, express The average value, yes variance yes variance yes and covariance; and It is a constant used to maintain stability.
[0030] Furthermore, the expression for FPS is as follows:
[0031] (8)
[0032] Where N represents the number of image frames generated or processed within time T, and T is the time taken, usually in seconds. Frame rate is a direct quantification of the number of frames processed within a unit of time, reflecting the model's processing speed or generation capability.
[0033] Beneficial effects: The above-mentioned real-time prediction method for the second half of video frames based on optical flow enables high frame rate and high quality prediction of the second half of video frames, providing key technical support for real-time wavefront correction of airborne imaging systems. Attached Figure Description
[0034] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0035] Figure 1 This is a diagram of the overall architecture of the video second-half frame prediction network based on optical flow in this embodiment of the invention.
[0036] Figure 2 for Figure 1 A schematic diagram of the specific structure of AttBlock used in the optical flow estimation module;
[0037] Figure 3 This is a diagram showing the prediction results of the second half of the frame on a visible light dataset in Embodiment 1 of the present invention;
[0038] Figure 4 This is a diagram showing the prediction result of the second half of the frame on an infrared simulation dataset in Embodiment 2 of the present invention. Detailed Implementation
[0039] The invention will now be further described with reference to the accompanying drawings.
[0040] This invention provides a real-time prediction model for the second half of a video frame based on optical flow, which can achieve high frame rate and high quality prediction of the second half of the image. The invention is illustrated in more detail below with examples.
[0041] This application provides a real-time prediction method for the second half of a video frame based on optical flow. This method is based on an optical flow-based prediction model for the second half of a video frame, which includes an optical flow estimation module and a refinement and reconstruction module. This network is applicable to visible light and near-infrared datasets.
[0042] The implementation of the above model, as described in this application, specifically includes the following steps:
[0043] Step 1: Define the network structure: An optical flow estimation module based on multi-layer depthwise separable convolutions is adopted as the core structure. Three-layer feature extraction blocks (AttBlocks) progressively extract inter-frame motion information. Each AttBlock uses depthwise separable convolutions to improve model speed and introduces channel attention enhancement blocks (SEBlocks) to achieve adaptive recalibration between feature channels to capture detailed features. The refinement and reconstruction module concatenates the two sets of Warp results with the corresponding optical flow fields. Through multi-layer separable convolutions and channel attention mechanisms, spatial features are re-fused and texture compensation is achieved. Within the module, local convolutions and residual connections complete the re-aggregation of spatiotemporal information.
[0044] Step 2: Initialize network parameters: Randomly initialize parameters such as network weights in the optical flow estimation module and the refinement reconstruction module.
[0045] Step 3: Define the loss function: A combined loss function is used to simultaneously constrain pixel accuracy and structural consistency. The specific formula is as follows:
[0046] (1)
[0047] (2)
[0048] (3)
[0049] (4)
[0050] in, Measuring pixel-level error This is based on Laplacian pyramid loss and is used to enhance high-frequency detail reconstruction. Structural similarity loss is used to improve the perceptual quality of predicted frames. and These are the weighting coefficients for each loss.
[0051] Step 4: Define the optimization algorithm: Use the ADAM optimizer, with weight decay set to 1×10⁻ 4 To prevent overfitting, the initial learning rate is set to 3×10⁻ 4 Furthermore, a cosine annealing strategy is used to dynamically adjust the learning rate in the later stages of training.
[0052] Step 5, Training Process: The ADAM optimizer is used, with weight decay set to 1×10⁻ 4 The initial learning rate is set to 3 × 10⁻ 4 Furthermore, a cosine annealing strategy is used to dynamically adjust the learning rate in the later stages of training. During training, two consecutive frames of images are... and The time-encoded input network at t=1.5 seconds is used to predict the bidirectional optical flow field via the optical flow estimation module. The optical flow is then linearly scaled down. and Backsampling is performed, and the spatial alignment result is stitched together with the optical flow field and then input into the thinning and reconstruction module to output the second half-frame predicted image. Calculate the combined loss between the predicted image and the real second half-frame image, and simultaneously update the parameters of the optical flow estimation module and the thinning reconstruction module using the backpropagation algorithm. Iterate training until the loss converges.
[0053] Step Six: Testing and Evaluation Process: Input two consecutive frames of images into the trained network to generate predicted images for the second half of the frame. Compare these predicted images with the corresponding real second half-frame images, and calculate PSNR and SSIM to measure the quality of the generated images. Simultaneously, test the network's prediction frame rate.
[0054] Furthermore, in the optical flow estimation module, based on the short-time linear motion assumption, a time encoding parameter t is introduced, and the parameter t is set to 1.5 to correspond to the relative time position of the second half of the frame in the input sequence; the predicted bidirectional optical flow is linearly scaled to obtain the pixel displacement at the second half of the frame; and the scaled optical flow field is then used to... and Bilinear inverse sampling is performed to obtain two sets of spatial alignment results for the second half of the frame. The intermediate results obtained by bidirectional inverse deformation are input into the reconstruction network, and the network adaptively learns the fusion weights of optical flow in different directions at the texture details during the end-to-end training process.
[0055] Furthermore, the refinement and reconstruction module performs channel-dimensional stitching of the two sets of alignment results with the corresponding optical flow fields, achieving fusion through multi-layer separable convolution and channel attention mechanisms. It also introduces local convolution and residual connections to aggregate spatiotemporal information, mitigating the blurring and artifact problems caused by optical flow errors, and finally outputs the predicted image of the second half frame. .
[0056] Furthermore, in the combined loss function, For L1 loss, For the loss based on the Laplace pyramid, For SSIM loss, the weighting coefficients are set to... , .
[0057] Furthermore, the expression for PSNR is as follows:
[0058] (5)
[0059] Where: n=8 represents the number of binary bits for each pixel in the image, and MSE is the mean square error between the real image and the generated image.
[0060] Wherein, MSE is the mean square error between the original image and the output image, and the definition of MSE is shown in the following formula:
[0061] (6)
[0062] in, and Reference regions for a given frame of the original image and the output image, respectively. The pixel value at the location is given, where m and n are the length and width of the image, respectively.
[0063] Furthermore, the expression for SSIM is as follows:
[0064] (7)
[0065] in: , This represents two images used for evaluation. express The average value, express The average value, yes variance yes variance yes and covariance; and It is a constant used to maintain stability.
[0066] Furthermore, the expression for FPS is as follows:
[0067] (8)
[0068] Where N represents the number of image frames generated or processed within time T, and T is the time taken, usually in seconds. Frame rate is a direct quantification of the number of frames processed within a unit of time, reflecting the model's processing speed or generation capability.
[0069] Example 1:
[0070] This invention provides a real-time prediction network for the second half of a video frame based on optical flow, the overall framework of which is as follows: Figure 1 As shown, it comprises two main components: an optical flow estimation module and a thinning and reconstruction module. This is illustrated using two consecutive frames of images. and As input, the forward optical flow field is predicted by the optical flow estimation module. and reverse optical flow field After introducing time coding at t=1.5 and scaling the bidirectional optical flow linearly over time, the following steps were taken: and Bilinear backsampling is performed to obtain spatial alignment results. The two sets of results are then concatenated with the corresponding optical flow field features and input into the thinning and reconstruction module for feature re-fusion and texture compensation. Finally, the predicted image of the second half frame is output. .
[0071] Figure 2 The diagram shows the specific structure of the channel attention enhancement block AttBlock. This module employs a multi-layer depthwise separable convolutional structure to significantly improve processing speed. The channel attention enhancement block enables adaptive recalibration between feature channels, effectively capturing image detail features. Based on the assumption that pixel motion is approximately linear over short timescales, a continuous-time encoding parameter t=1.5 is introduced, corresponding to the relative time position of the target frame in the input frame. This linearly scales the optical flow vector to obtain the pixel displacement at the target time position. The intermediate results obtained from bidirectional inverse deformation are input into the reconstruction network, which adaptively learns the fusion weights of optical flow in different directions at texture details during end-to-end training.
[0072] The Visible Light Dataset (VIS) was used for training and testing. This dataset was extracted frame-by-frame from visible light video at a frame rate of 30 FPS, containing high-speed motion scenes of aircraft against a sky background in real-world scenarios, with image sizes of 640×512. To extend the model's applicability across different imaging bands, the number of input and output channels was uniformly set to 3.
[0073] Based on the above steps, the network structure is defined, and a combined loss function is used to simultaneously constrain pixel accuracy and structural consistency. This includes pixel-level L1 loss, high-frequency detail reconstruction loss based on the Laplacian pyramid, and structural similarity loss. , As weighting coefficients. The ADAM optimizer is used, with weight decay set to 1×10⁻. 4 The initial learning rate is set to 3 × 10⁻ 4In the later stages of training, a cosine annealing strategy was used to dynamically adjust the learning rate, and the batch size was set to 4. The loss converged around the 200th epoch.
[0074] After training, the dataset was tested on the Visible Light System (VIS), and the results are as follows. Figure 3 As shown, the corresponding SSIM and PSNR metrics and frame rates are detailed in Table 1.
[0075] Table 1
[0076]
[0077] Example 2:
[0078] This invention provides a real-time prediction network for the second half of a video frame based on optical flow, and its network structure is the same as that in Example 1.
[0079] The near-infrared simulation datasets NIR1 and NIR2 were used for training and testing. These datasets were extracted frame-by-frame from videos at a frame rate of 100 FPS. The main difference between the two datasets lies in the target scale: NIR1 represents large-scale target scenes, while NIR2 represents smaller-scale targets. The video images are all 640×512 pixels and cover various typical motion modes, including translation and acceleration. The background environments are primarily static and weakly dynamic scenes, and the original imaging noise characteristics are preserved during generation to simulate real near-infrared imaging conditions.
[0080] The network structure and parameter settings are the same as in Example 1. The single-channel infrared image is expanded into a three-channel input through channel replication. This process does not introduce redundant information or cause information loss.
[0081] After training, tests were performed on the infrared simulation datasets NIR1 and NIR2, and the results are as follows. Figure 4 As shown, the corresponding SSIM and PSNR metrics and frame rates are detailed in Table 2.
[0082] Table 2
[0083]
[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of the present invention. Several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A real-time prediction method for the second half of a video frame based on optical flow, characterized in that, The method includes the following steps: Step 1: Obtain two consecutive frames of images, i.e., the previous frame image. and the next frame image ; Step 2: Build a lightweight optical flow estimation module for prediction of the second half of the frame. Train the optical flow estimation module using two consecutive frames as input to predict the optical flow from... arrive Forward optical flow field And from arrive Reverse optical flow field The bidirectional optical flow is obtained; wherein, the optical flow estimation module adopts a multi-layer depthwise separable convolutional structure to improve the network running speed, and introduces a channel attention enhancement block to realize adaptive recalibration between feature channels to capture detailed features; Step 3: Based on the short-time linear motion assumption, a time-coding parameter t is introduced to linearly scale the predicted bidirectional optical flow, obtaining the pixel displacement corresponding to the target time, and then... and Bilinear backsampling is performed to obtain two sets of spatially aligned results for the second half of the frame. Step 4: The two sets of spatial alignment results are stitched together with the corresponding optical flow fields. The spatial features are then re-fused and texture compensation is performed through the refinement and reconstruction module, finally outputting the predicted image of the second half frame. .
2. The method according to claim 1, characterized in that, The training employs a combined loss function to simultaneously constrain pixel accuracy and structural consistency, as shown in the following equation: (1) (2) (3) (4) in, Measuring pixel-level error This is based on Laplacian pyramid loss and is used to enhance high-frequency detail reconstruction. Structural similarity loss is used to improve the perceptual quality of predicted frames. and These are the weighting coefficients for each loss; The frame prediction network was tested using three parameters to evaluate its performance in predicting frames: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and frame rate (FR).
3. The method according to claim 2, characterized in that, Peak Signal-to-Noise Ratio (PSNR) is calculated based on the error between corresponding pixels in the original reference image and the generated image. PSNR is expressed in dB, and a higher value indicates a better result. The specific calculation method is shown in the following formula: (5) Wherein, MSE is the mean square error between the original image and the output image, and the definition of MSE is shown in the following formula: (6) in, and Reference regions for a given frame of the original image and the output image, respectively. The pixel value at the specified location, where m and n are the length and width of the image, respectively; Structural Similarity (SSIM) measures the structural properties of an object by evaluating three features: brightness, contrast, and structure. A higher SSIM value (closer to 1) indicates a better result. The SSIM value is calculated as follows: (7) in: , This represents two images used for evaluation. express The average value, express The average value, yes variance yes variance yes and covariance; and It is a constant used to maintain stability.
4. The method according to claim 2, characterized in that, use , As a weighting coefficient, pixel-level reconstruction is the primary focus, with structural and detail constraints as secondary considerations.
5. The method according to claim 1, characterized in that, The number of input and output channels is uniformly set to 3. Single-channel infrared images are expanded into three channels through channel replication to achieve unified modeling of visible light and infrared datasets.
6. The method according to claim 1, characterized in that, The time encoding parameter t is set to 1.5, which corresponds to the target frame being located half a frame later than the middle of the two input frames.
7. The method according to claim 1, characterized in that, The optical flow estimation module extracts inter-frame motion information step by step through three-layer feature extraction blocks AttBlock; each AttBlock adopts a multi-layer depthwise separable convolutional structure and introduces channel attention enhancement blocks in it to improve the frame rate while ensuring prediction quality.
8. The method according to claim 1, characterized in that, The refinement and reconstruction module performs feature stitching between the two sets of Warp results and the corresponding optical flow field. It achieves spatial feature re-fusion and texture compensation through multi-layer separable convolution and channel attention mechanism. Structurally, this module completes the re-aggregation of spatiotemporal information through local convolution and residual connection, which alleviates the blurring and artifact problems caused by optical flow error.