Video frame insertion time downscaling method for coupling data assimilation and electronic equipment

The video interpolation temporal downscaling method with data assimilation addresses the limitations of traditional methods by using deep learning to transform low-resolution weather data into high-resolution data, enhancing forecast precision and timeliness through site observation data assimilation.

CN119094817APending Publication Date: 2024-12-06CSIC HAIZHUANG WINDPOWER CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411055449.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Traditional time downscaling methods struggle to accurately capture the dynamic characteristics of complex weather systems due to reliance on statistical models or simple interpolation techniques, leading to errors in weather forecast precision and timeliness.

Method used

A video interpolation temporal downscaling method coupled with data assimilation, utilizing deep learning to transform low temporal resolution simulation data into high resolution data by incorporating site observation data loss into the video interpolation algorithm's loss function, enhancing accuracy through deep learning-based video interpolation and data assimilation.

Benefits of technology

The method achieves high-precision temporal downscaling by learning field changes over time and considering spatial field influences, improving the accuracy and timeliness of weather forecasts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094817A_ABST
    Figure CN119094817A_ABST
Patent Text Reader

Abstract

The invention provides a video frame insertion time downscaling method and device for coupling data assimilation and a storage medium, and belongs to the technical field of atmospheric science. The method comprises the following steps: acquiring simulation data with high time resolution and observation data with the same high time resolution in a corresponding region in the same time period; performing down-sampling on the analog data with the high time resolution to obtain data with a low time resolution; inputting the data into a video frame insertion model, constructing a loss function, and performing deep learning training on the video frame insertion model through an Adam optimization strategy; and performing frame insertion on the low-time-resolution data by using the trained video frame insertion model to obtain high-precision time-resolution data. According to the method, the video frame insertion technology is applied to the time downscaling, meanwhile, the loss direct to the site observation data is added into the loss function of the video frame insertion algorithm to assimilate and correct the time downscaling result, and the time downscaling precision is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of atmospheric science, and in particular to a video frame insertion time downscaling method, device and storage medium coupled with data assimilation. Background Art

[0002] With the intensification of global climate change and the complexity of social and economic activities, the reliance on weather forecasts is increasing, especially in the fields of disaster warning, agricultural management, urban planning and renewable energy scheduling, which poses unprecedented challenges to the accuracy and timeliness of weather forecasts. Meteorological data with high temporal resolution can not only capture more subtle weather change characteristics, but also significantly shorten the forecast response time, providing a more timely and accurate reference for decision-making. However, directly obtaining such data through large-scale numerical simulation is not only computationally expensive, but also limited by the accuracy and uncertainty of the model itself, making it difficult to fully meet the needs of actual applications.

[0003] In this context, time downscaling technology has emerged as an effective means to convert lower time resolution simulation data into higher resolution data through algorithm optimization, thereby improving the precision and timeliness of forecasts without significantly increasing simulation costs. However, traditional time downscaling methods often rely on statistical models or simple interpolation techniques, which make it difficult to accurately capture the dynamic characteristics of complex weather systems, resulting in errors in the downscaling results. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a video interpolation temporal downscaling method coupled with data assimilation. The present invention transforms low temporal resolution simulation data into high temporal resolution simulation results based on deep learning video interpolation technology, and deep learning can learn the characteristics of field changes over time; at the same time, in the deep learning algorithm, the time series information of discrete sites is extracted, and the loss is calculated with the real detection data, and added to the total loss of the video interpolation algorithm according to a certain ratio, thereby achieving temporal downscaling while having an assimilation correction effect; the video interpolation model obtained by training can achieve high-precision temporal downscaling.

[0005] The technical solution adopted by the present invention is:

[0006] In a first aspect, a video frame interpolation temporal downscaling method coupled with data assimilation is provided, comprising:

[0007] Obtain high-temporal-resolution simulation data and high-temporal-resolution observation data of the corresponding area in the same time period;

[0008] Downsampling the high time resolution analog data to obtain low time resolution data;

[0009] Inputting the low temporal resolution data, the high temporal resolution simulation data, and the high temporal resolution observation data into an existing video interpolation model, constructing a loss function, and performing deep learning training on the video interpolation model through an Adam optimization strategy;

[0010] The trained video interpolation model is used to interpolate low temporal resolution data to obtain high-precision temporal resolution data.

[0011] Furthermore, the acquired observation data are sampled to obtain the same time resolution as the simulation data.

[0012] Furthermore, the low temporal resolution data is converted into high temporal resolution data in a video interpolation model through a video interpolation technique.

[0013] Furthermore, the loss function includes:

[0014] The loss loss1 comes from the high temporal resolution data generated by video interpolation technology and the high temporal resolution simulated data;

[0015] The loss loss2 comes from the loss between the high-temporal-resolution data generated by video interpolation technology and the observational data corresponding to the sequence information of the site.

[0016] Furthermore, the loss1 and loss2 are added at a certain ratio to obtain the total loss Loss, and the proportional coefficient a is a model hyperparameter, and the a is obtained by a hyperparameter tuning method.

[0017] Furthermore, the hyperparameter tuning method is: grid search, Bayesian parameter tuning or random parameter tuning.

[0018] Furthermore, high temporal resolution simulation data were obtained through simulation using the Weather Research and Forecasting Model WRF.

[0019] In a second aspect, an electronic device is provided, which, in a second possible implementation manner, includes:

[0020] one or more processors;

[0021] A storage device for storing one or more programs;

[0022] When one or more programs are executed by one or more processors, the one or more processors implement the video frame insertion time downscaling method coupled with data assimilation described in the first aspect.

[0023] In a third aspect, a computer-readable storage medium storing a computer program is provided. In a third achievable manner, when the computer program is executed by a processor, a video frame interpolation temporal downscaling method for coupled data assimilation as described in the first aspect is implemented.

[0024] It can be seen from the above technical solution that the beneficial technical effects of the present invention are as follows:

[0025] 1. The present invention applies the video interpolation technology to time downscaling. At the same time, the loss directly related to the site observation data is added to the loss function of the video interpolation algorithm to assimilate and correct the time downscaling results, thereby further improving the accuracy of time downscaling.

[0026] 2. The present invention transforms low-temporal-resolution simulation data into high-temporal-resolution simulation results through deep learning-based video interpolation technology. Deep learning can learn the characteristics of field changes over time. At the same time, in the deep learning algorithm, the time series information of discrete sites is extracted and loss is calculated with the real detection data. The loss is added to the total loss of the video interpolation algorithm according to a certain ratio, thereby achieving time downscaling and assimilation correction effect. The video interpolation model obtained by training can achieve high-precision time downscaling.

[0027] 3. The present invention combines the idea of ​​video interpolation technology with data assimilation and sets up a new time downscaling method; video interpolation can consider the motion characteristics and insert intermediate frames more reasonably. At the same time, compared with the time downscaling of a single point, video interpolation can also consider the influence of the spatial field, so the time downscaling result obtained is more reasonable; at the same time, the difference in site observation data is considered in the function, so that the data assimilation of the time downscaling result can be realized, thereby achieving the purpose of improving the accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the prior art description. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0029] Figure 1 This is a schematic diagram of the technical route of an embodiment of the present invention;

[0030] Figure 2 This is a regional and internal site distribution diagram of an embodiment of the present invention;

[0031] Figure 3 This is a diagram showing the effect of the video frame insertion technology according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0033] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the invention belongs.

[0034] Example

[0035] In one embodiment, a video frame insertion time downscaling method coupled with data assimilation is provided, and the technical route is as follows: Figure 1 As shown, in the first possible implementation method;

[0036] This embodiment is illustrated by the hourly WRF simulation in Zhangbei area of ​​my country in June 2018. The steps of the method are as follows:

[0037] Step 1: Use WRF simulation to obtain hourly high temporal resolution simulation data for Zhangbei area in my country in June 2018;

[0038] The 1-hour high-time-resolution simulation data can be sampled in the time dimension to obtain 2-hour low-time-resolution data;

[0039] At the same time, the meteorological data of the stations in this area can be obtained through Wyoming WeatherWeb. The time resolution of the station observation data is often higher than the time resolution of the simulation, so the observation data is often sampled to obtain the same time resolution as the simulation. The station location is as follows: Figure 2 As shown;

[0040] Step 2: Input 2h low temporal resolution data, 1h high temporal resolution simulation data, and 1h high temporal resolution observation data into the video interpolation model, construct a loss function, and use the optimization strategy of the adaptive moment estimation optimizer Adam to train the deep learning model of the video interpolation technology. In this embodiment, the trained video interpolation model can downscale the 2h low temporal resolution data to obtain 1h high temporal resolution data, such as Figure 3 shown.

[0041] In some embodiments, the existing video frame insertion model includes:

[0042] (1) Super SloMo (super slow motion model) is a video interpolation method based on bidirectional optical flow estimation and deep learning. It is suitable for videos that need to generate slow motion effects and can significantly improve the smoothness and viewing experience of the video.

[0043] (2) DAIN (Depth-Aware Video Frame Interpolation) combines scene depth information to improve the accuracy of video interpolation. It can better handle problems such as occlusion and perspective transformation, and performs particularly well in video interpolation tasks in complex motion scenes.

[0044] (3) CAIN (Channel Attention for Efficient Video Interpolation) is a video interpolation model based on convolutional neural network (CNN). It pays special attention to the channel attention mechanism and is suitable for application scenarios that require efficient and high-quality interpolation.

[0045] (4) IFRNet (Efficient Video Frame Interpolation via Feature Refinement) is an efficient video frame interpolation network that uses a network architecture that only contains an encoder-decoder structure. By jointly optimizing the intermediate optical flow estimation and intermediate feature reconstruction process, it can achieve fast and high-quality intermediate frame synthesis, which is suitable for application scenarios such as real-time video processing.

[0046] (5) RIFE (Real-Time Intermediate Flow Estimation for Video Frame Interpolation) uses a lightweight network structure to quickly estimate the optical flow information between adjacent frames and synthesize intermediate frames based on this information. It has broad application prospects in real-time video communication, online video editing and other fields.

[0047] (6) IconVSR model. IconVSR is further expanded and optimized on the basis of BasicVSR model. Two new components are introduced on the basis of BasicVSR: Information-refill Mechanism and Coupled Propagation, thus forming the IconVSR model.

[0048] The video interpolation model used in this embodiment is the IconVSR model of CVPR2021. In the video interpolation model, 2h low-temporal-resolution data is converted into 1h high-temporal-resolution data through video interpolation technology;

[0049] The loss function constructed includes a part of the loss from the loss between the 1h high-resolution data generated by the video interpolation model and the 1h data label obtained by the simulation data, which reflects the temporal variation characteristics of the field. The other part of the loss comes from the loss between the 1h high-resolution data generated by the video interpolation model and the 1h data label obtained by the simulation data. Figure 1 The sequence information corresponding to the site is used to obtain the loss loss2 between the observed data, which is used for assimilation correction to make the video interpolation output more consistent with the observation of the real world. The total loss Loss is the result of adding loss1 and loss2 at a certain ratio, where the proportional coefficient a of the addition of the two is the model hyperparameter, which is obtained by the Bayesian parameter adjustment method.

[0050] In one embodiment, an electronic device is provided, which, in a second possible implementation, includes:

[0051] one or more processors;

[0052] A storage device for storing one or more programs;

[0053] When one or more programs are executed by one or more processors, the one or more processors implement the video frame insertion time downscaling method coupled with data assimilation described in the first aspect.

[0054] In one embodiment, a computer-readable storage medium storing a computer program is provided. In a third achievable manner, when the computer program is executed by a processor, the video frame interpolation temporal downscaling method coupled with data assimilation described in the first aspect is implemented.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A video frame interpolation temporal downscaling method coupled with data assimilation, characterized in that: include: Obtain high-temporal-resolution simulation data and high-temporal-resolution observation data of the corresponding area in the same time period; Downsampling the high time resolution analog data to obtain low time resolution data; Inputting the low temporal resolution data, the high temporal resolution simulation data, and the high temporal resolution observation data into an existing video interpolation model, constructing a loss function, and performing deep learning training on the video interpolation model through an optimization strategy of an adaptive moment estimation optimizer Adam; The trained video interpolation model is used to interpolate low temporal resolution data to obtain high-precision temporal resolution data.

2. The video frame insertion temporal downscaling method coupled with data assimilation according to claim 1, characterized in that: The acquired observational data are sampled to obtain the same temporal resolution as the simulated data.

3. The video frame insertion temporal downscaling method coupled with data assimilation according to claim 1, characterized in that: The low temporal resolution data is converted into high temporal resolution data through video interpolation technology in a video interpolation model.

4. The video frame insertion temporal downscaling method coupled with data assimilation according to claim 3, characterized in that: The loss function includes: The loss loss1 comes from the high temporal resolution data generated by video interpolation technology and the high temporal resolution simulated data; The loss between the high temporal resolution data generated by video interpolation technology and the observational data corresponding to the sequence information of the site is loss2.

5. The video frame insertion temporal downscaling method coupled with data assimilation according to claim 4, characterized in that: The loss1 and loss2 are added at a certain ratio to obtain the total loss Loss. The proportional coefficient a is a model hyperparameter, and a is obtained by a hyperparameter tuning method.

6. The video frame insertion temporal downscaling method coupled with data assimilation according to claim 5, characterized in that: The hyperparameter tuning method is: grid search, Bayesian parameter tuning or random parameter tuning.

7. The video frame insertion temporal downscaling method coupled with data assimilation according to claim 1, characterized in that: The simulation data with high temporal resolution are obtained through simulation by the Weather Research and Forecasting Model WRF.

8. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video frame insertion time downscaling method for coupled data assimilation as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video frame insertion time downscaling method for coupled data assimilation according to any one of claims 1 to 7 is implemented.