Meteorological data processing method based on improved moving average filtering, medium and equipment
By improving the sliding average filtering method, including data cleaning, weighted sliding average filtering and trend analysis, the problems of high boundary effect and calculation complexity in the existing technology are solved, and efficient and accurate meteorological data processing is achieved, suitable for a variety of meteorological conditions. Data analysis and prediction scenarios.
Patent Information
- Application Number
- CN202411242847.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-11-26
AI Technical Summary
Existing meteorological data filtering methods such as Lanczos filtering, Kalman filtering and wavelet transform filtering have boundary effects, high computational complexity, complex parameter tuning, and limited non-stationary signal processing capabilities when processing meteorological data, resulting in data processing not being done. Accurate and inefficient.
Improved sliding average filtering methods are adopted, including data cleaning, weighted sliding average filtering, trend analysis and boundary processing. Through dynamic sliding windows and weight adjustments, combined with zero filling or mirror expansion, the data continuity and integrity are ensured.
It improves the accuracy and reliability of meteorological data, reduces the computational complexity, adapts to different frequency components and non-stationary signals, significantly reduces the boundary effect, and is suitable for real-time processing of large-scale meteorological data.
Smart Images

Figure CN119025828A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of meteorological data technology, and in particular to the field of meteorological data processing methods, media and equipment based on improved sliding average filtering. Background Art
[0002] Currently, meteorological data filtering is mostly performed using bandpass filtering, such as Lanczos filtering. Lanczos filtering is a commonly used resampling and signal processing technique, widely used in meteorological data processing due to its excellent frequency domain characteristics. However, despite its relatively superior performance, Lanczos filtering still has the following significant problems when processing meteorological data:
[0003] 1. Boundary Issues: When applying a Lanczos filter, the filter cannot fully cover all data points at the beginning and end of the data sequence, resulting in ineffective filtering of data at the boundaries. This boundary effect can lead to partial data loss or inaccurate processing, especially when processing meteorological data of limited length. Because there is insufficient adjacent data for interpolation around boundary data points, the filter weight distribution cannot evenly cover all data points, resulting in increased interpolation error at the boundaries. This error introduces significant uncertainty in high-precision meteorological data processing (such as short-term weather forecasts).
[0004] 2. Edge Effect: The Lanczos filter is based on a weighted sum of sinc functions. Data near the edges are affected by the filter kernel, which can easily cause ringing. This ringing effect produces unnatural fluctuations near the edges, distorting the true characteristics of meteorological data and affecting data interpretation and application. Due to the presence of edge effects, the filtered data may exhibit localized signal distortion. This distortion is particularly noticeable when processing meteorological data with strong mutations (such as sudden changes in wind speed and precipitation), and can lead to the loss or misunderstanding of critical meteorological information.
[0005] 3. Frequency domain characteristic limitations: Although the Lanczos filter has good low-pass filtering characteristics, its effectiveness in processing high-frequency noise is still limited. In meteorological data, high-frequency noise is often caused by sensor errors or environmental interference. If improperly handled, it will affect the accuracy of subsequent data analysis and prediction models. The frequency response function of the Lanczos filter has specific bandwidth limitations and cannot flexibly adapt to meteorological data with different frequency components. This means that when processing complex meteorological data with multiple frequency components (such as radar reflectivity data), some frequency components cannot be effectively filtered, affecting the comprehensiveness and accuracy of the data.
[0006] 4. Computational Complexity: The Lanczos filter requires calculating large weighted sums. This complexity is high when processing large-scale meteorological data (such as high-resolution satellite imagery or long-term meteorological observations), making it difficult to meet real-time processing requirements. This can lead to processing delays in frequently updated weather forecasting systems, impacting the timeliness of forecasts. High computational complexity also requires significant computing resources. In resource-constrained computing environments (such as embedded systems or mobile devices), the Lanczos filter's application is limited, making it difficult to operate efficiently.
[0007] Currently, common meteorological data filtering methods include Kalman filtering and wavelet transform filtering. While these methods are effective in denoising and dealing with outliers, they still face challenges in practical applications, such as high computational complexity and difficulty in optimizing parameters. Therefore, a simple, efficient, and innovative improved sliding average filtering method is proposed for meteorological data processing, which has important practical significance. Summary of the Invention
[0008] In view of the above-mentioned shortcomings of the prior art, the object of the present invention is to provide a meteorological data processing method, medium and device based on improved sliding average filtering, so as to improve the quality and reliability of data.
[0009] To achieve the above-mentioned purpose and other related purposes, the present invention provides a meteorological data processing method based on improved sliding average filtering, including: collecting meteorological raw data, and cleaning the meteorological raw data to obtain meteorological data to be processed; performing weighted sliding average filtering on the meteorological data to be processed to obtain meteorological filtered data; performing trend analysis and boundary processing on the meteorological filtered data; and reconstructing the meteorological filtered data after trend analysis and boundary processing to ensure that the meteorological filtered data has continuity and integrity.
[0010] In one embodiment of the present application, the cleaning of the meteorological raw data to obtain the meteorological data to be processed includes: processing the error data of the meteorological raw data to obtain the initial cleaning data; the processing of the error data of the meteorological raw data includes boundary value detection processing and time consistency detection processing; normalizing the initial cleaning data to obtain the meteorological data to be processed.
[0011] In one embodiment of the present application, the boundary value detection processing includes eliminating data that does not fall within the boundary value range; the time consistency detection processing includes detecting whether the data is missing in time, and filling in the missing data, and detecting whether the meteorological element sampling value exceeds the range of change within a certain time, and eliminating the data that exceeds the range of change.
[0012] In one embodiment of the present application, the weighted sliding average filtering of the meteorological data to be processed to obtain meteorological filtered data includes: determining an initial sliding window; dynamically adjusting the size of the initial sliding window based on the volatility and periodicity of the data to form the required sliding window; assigning a weight to each data point in the required sliding window; and obtaining meteorological filtered data based on the meteorological data to be processed and the weight.
[0013] In one embodiment of the present application, the method of obtaining meteorological filtering data based on the meteorological data to be processed and the weight is:
[0014]
[0015] Among them, x i is the original data, ω i is the weight coefficient, y t is the meteorological filtering data, W is the initial sliding window, t is the sequence number of the data point, and i is the sequence number of each data point in the sliding window.
[0016] In one embodiment of the present application, the trend analysis method includes a fitting model of linear regression or polynomial regression.
[0017] In one embodiment of the present application, the boundary processing method includes zero padding or mirror extension.
[0018] To achieve the above-mentioned purpose and other related purposes, the present application also provides a meteorological data processing device based on improved sliding average filtering, including: a data preprocessing module, used to collect meteorological raw data, and clean the meteorological raw data to obtain meteorological data to be processed; a weighted sliding average filtering module, used to perform weighted sliding average filtering on the meteorological data to be processed to obtain meteorological filtered data; a trend analysis module, used to perform trend analysis on the meteorological filtered data; a boundary processing module, used to perform boundary processing on the meteorological filtered data; a reconstruction module, used to reconstruct the meteorological filtered data after trend analysis and boundary processing, so that the meteorological filtered data has continuity and integrity.
[0019] To achieve the above-mentioned object and other related objects, the present invention also provides a computer storage medium storing program instructions, which, when executed, implement the steps of the meteorological data processing method based on improved sliding average filtering as described above.
[0020] To achieve the above-mentioned purpose and other related purposes, the present invention also provides an electronic device, including a memory for storing a computer program; and a processor for running the computer program to implement the steps of the meteorological data processing method based on improved sliding average filtering as described above.
[0021] As described above, the meteorological data processing method, medium and device based on improved sliding average filtering of the present invention have the following advantages:
[0022] Beneficial effects:
[0023] This application can provide higher accuracy and reliability in the process of meteorological data processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0025] Figure 1 Shown is a schematic diagram of the overall flow of a meteorological data processing method based on an improved sliding average filter in one embodiment of the present application;
[0026] Figure 2 Shown is a flow chart of data preprocessing in a meteorological data processing method based on improved sliding average filtering in one embodiment of the present application;
[0027] Figure 3 Shown is a schematic diagram of the principle process of data preprocessing in a meteorological data processing method based on improved sliding average filtering in one embodiment of the present application;
[0028] Figure 4 Shown is a schematic diagram of jump data in a meteorological data processing method based on improved sliding average filtering in one embodiment of the present application;
[0029] Figure 5 Shown is a flow chart of obtaining meteorological filtered data in a meteorological data processing method based on improved sliding average filtering in one embodiment of the present application;
[0030] Figure 6 Shown is a comparison diagram of the filtering processing method in the meteorological data processing method based on the improved sliding average filter in one embodiment of the present application;
[0031] Figure 7 Shown is a principle block diagram of a meteorological data processing device based on improved sliding average filtering in one embodiment of the present application;
[0032] Figure 8 1 is a functional block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0033] The following describes the implementation of this embodiment through specific examples. Those skilled in the art can easily understand the other advantages and effects of this embodiment from the content disclosed in this specification. This embodiment can also be implemented or applied through different specific implementations. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this embodiment. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless there is a conflict.
[0034] Filtering originated in radio technology, where interference during radio wave propagation causes noise in the received signal. This can be expressed as y(t) = x(t) + σ(t), where y(t) is the observed value, x(t) is the useful information, and σ(t) is the noise. Thus, filtering is the technique for eliminating interference and extracting useful information.
[0035] Against the backdrop of global climate change and frequent extreme weather events, accurate acquisition and processing of meteorological data are crucial for weather forecasting and environmental monitoring. With the continuous advancement of meteorological observation and sensor technologies, the variety and volume of meteorological data have increased dramatically. However, this data often contains significant noise and outliers. The atmosphere contains a variety of fluctuations, including acoustic, gravity, and inertial waves—fast, long, and ultra-long slow waves. Forecasting large-scale motion requires filtering out acoustic, gravity, and even inertial waves. Otherwise, these fast waves rapidly grow due to observational errors, leading to significant forecast bias. These fast waves, which degrade forecasts, are essentially noise. Depending on the forecast timeframe and project, it is necessary to filter out different minor fluctuations while preserving the primary fluctuations that govern the weather. Effective filtering and processing methods are required to extract useful information for accurate weather forecasts and climate analysis.
[0036] The importance of meteorological data filtering is mainly reflected in the following aspects:
[0037] 1. Improve data accuracy:
[0038] Meteorological data often contains various interferences and noises, which can affect data accuracy and lead to biased analysis results. Filtering can effectively remove noise, improve data accuracy, and ensure the reliability of subsequent analysis and predictions.
[0039] 2. Remove outliers:
[0040] Outliers may appear in meteorological data due to sensor failure, environmental interference, or other factors. If left unaddressed, these outliers can negatively impact data analysis and model training. Filtering methods can identify and remove these outliers, resulting in more realistic meteorological data.
[0041] 3. Enhance the smoothness of data:
[0042] The smoothness of meteorological data is crucial for capturing long-term trends and cyclical changes. Filtering can eliminate short-term fluctuations and make the data smoother, making it easier to identify and analyze long-term trends.
[0043] 4. Improve data continuity:
[0044] During meteorological data collection, data may be missing or discontinuous. Appropriate filtering methods can be used to interpolate or complete missing data and enhance data continuity.
[0045] 5. Improve data processing efficiency:
[0046] In the process of processing large-scale meteorological data, filtering can reduce noise and irrelevant information, reduce data complexity, and improve processing efficiency. It is especially important in application scenarios of real-time data processing and rapid response.
[0047] Currently, commonly used meteorological data filtering methods include Kalman filtering and wavelet transform filtering. These methods have certain effects in denoising and outlier processing, but there are still some problems in practical applications:
[0048] 1. High computational complexity: For example, Kalman filtering requires a large number of matrix operations, and wavelet transform filtering involves complex transformation and inverse transformation processes, which consumes a lot of computing resources when processing large-scale data.
[0049] 2. Difficulty in parameter tuning: These filtering methods typically involve the selection and tuning of multiple parameters, such as the state transition matrix and measurement noise covariance matrix of the Kalman filter, and the selection of the mother wavelet and the number of decomposition layers of the wavelet transform filter. Improper parameter selection can affect filtering effectiveness, and the tuning process is complex.
[0050] 3. Significant boundary effects: The filtering process often produces boundary effects at the beginning and end of the signal, resulting in inaccurate boundary data processing and the introduction of artifacts and distortion.
[0051] 4. Limited ability to process non-stationary signals: Meteorological data are often non-stationary. Traditional filtering methods are not very effective in processing non-stationary signals and cannot accurately capture the dynamic changes in meteorological data.
[0052] Therefore, this embodiment provides a meteorological data processing method, medium, and device based on an improved sliding average filter, which is of great practical significance for meteorological data processing. By introducing technologies such as weighted sliding average, dynamic sliding window, and trend analysis adjustment, this embodiment aims to address issues such as boundary effects, poor noise removal, and high computational complexity in existing technologies, providing an efficient and reliable solution for meteorological data processing.
[0053] Lanczos filtering is a digital signal processing technique widely used in image processing, signal reconstruction, and data smoothing. It is a highly efficient low-pass filter that removes noise by smoothing the signal in the frequency domain. The core concept of Lanczos filtering is to use the Lanczos kernel function to perform a weighted average of the signal, thereby achieving smoothing.
[0054] The Lanczos kernel function is a window function that filters the signal by convolving it with the kernel function. The mathematical expression of the Lanczos kernel function is as follows:
[0055]
[0056] Among them, a is the parameter of the filter, which determines the size of the window and the frequency response characteristics. Usually the value of a ranges from 2 to 5.
[0057] Advantages: Good frequency selectivity: Lanczos filter can achieve good frequency selectivity through Lanczos window function, which can effectively reduce spectrum leakage.
[0058] Good smoothness: It can effectively remove noise while retaining the main characteristics of the signal, making the signal smoother.
[0059] Disadvantages: High computational complexity: Lanczos filter involves Fourier transform and window function operations, which requires a large amount of calculation and is less efficient when processing large-scale data; complex parameter tuning: the selection of window function and parameter setting require experience and debugging, which affects the filtering effect; boundary problems: when applying Lanczos filtering, the filter cannot completely cover all data points at the beginning and end of the data sequence, resulting in the data at the boundary cannot be effectively filtered; edge effect: the filtered data may have local signal distortion.
[0060] The Kalman filter is a recursive filter used to estimate the state of a dynamic system. The basic idea is to use prior information and current observation data to update the state estimate. The formula is:
[0061] Status prediction:
[0062] Kalman gain:
[0063] Status Update:
[0064] in, is the state prediction at time k, P k|k-1 is the forecast error covariance matrix, K k is the Kalman gain, z k is the observed value, F k is the state transition matrix, B k is the control matrix, u k is the control input, H k is the observation matrix, Q k is the process noise covariance matrix, R k is the observation noise covariance matrix.
[0065] Advantages: Good dynamics: Kalman filtering can update state estimates in real time and is suitable for processing dynamically changing data; Strong noise processing capabilities: By modeling system noise and measurement noise, it can effectively filter out noise and improve estimation accuracy; Strong adaptability: It can process multidimensional data and complex systems and has strong adaptability.
[0066] Disadvantages: High computational complexity: Involves matrix operations and inversion, with a large amount of calculation, especially in high-dimensional systems, the efficiency is low; complex parameter setting: requires accurate noise statistical characteristics and system models, and improper parameter settings will lead to poor filtering effects; sensitive to linear assumptions: Kalman filtering assumes that the system is linear, and nonlinear systems require extended Kalman filtering (EKF) or unscented Kalman filtering (UKF), which is more complex.
[0067] Wavelet transform filtering uses wavelet transform to decompose the signal into different scales, processes the coefficients of each scale, and then reconstructs the signal through inverse wavelet transform to achieve the filtering effect. The formula is expressed as:
[0068] Discrete Wavelet Transform (DWT):
[0069] Threshold processing (denoising): Assume the threshold is λ, c j,k Threshold the wavelet coefficients:
[0070]
[0071] Inverse Discrete Wavelet Transform (IDWT):
[0072] Through the above steps, the wavelet coefficients are thresholded and then the signal is reconstructed to achieve the filtering effect.
[0073] Advantages: Multi-scale analysis: It can perform signal analysis in both the time domain and the frequency domain, and is suitable for processing non-stationary signals; Good denoising effect: Wavelet transform filtering can remove noise at different scales while retaining the main features of the signal; High flexibility: Different wavelet basis functions and decomposition levels can adapt to different types of signals and application requirements.
[0074] Disadvantages: High computational complexity: The wavelet transform and inverse transform processes are complex, computationally intensive, and inefficient when processing large-scale data; Significant boundary effects: Wavelet transforms easily introduce artifacts and distortion in boundary processing, requiring special boundary processing strategies; Complex parameter tuning: The selection of wavelet basis functions and the setting of the number of decomposition layers require experience and debugging, which affects the filtering effect.
[0075] Therefore, the meteorological data processing method based on the improved moving average filter proposed in this embodiment has significant advantages in terms of computational complexity, ease of implementation and optimization, boundary effect processing, adaptability to non-stationary signals, denoising effectiveness, and real-time processing capability. Therefore, this embodiment is more suitable for real-time processing and rapid response of large-scale meteorological data, providing an efficient and reliable solution for meteorological data processing.
[0076] The principles and implementation methods of the meteorological data processing method, medium and equipment based on improved sliding average filtering of this embodiment will be explained in detail below, so that those skilled in the art can understand the meteorological data processing method, medium and equipment based on improved sliding average filtering of this embodiment without creative work.
[0077] First, this embodiment provides a meteorological data processing method based on improved sliding average filtering. Specifically, Figure 1 As shown, the meteorological data processing method based on the improved sliding average filter in this embodiment includes:
[0078] Step S100: collecting raw meteorological data and cleaning the raw meteorological data to obtain meteorological data to be processed;
[0079] Step S200, performing weighted sliding average filtering on the meteorological data to be processed to obtain meteorological filtered data;
[0080] Step S300, performing trend analysis and boundary processing on the meteorological filtering data;
[0081] Step S400 : reconstructing the meteorological filtering data after trend analysis and boundary processing, so as to make the meteorological filtering data continuous and complete.
[0082] The following specifically describes steps S100 to S400 in the meteorological data processing method based on the improved sliding average filter of this embodiment.
[0083] Step S100: collecting raw meteorological data and cleaning the raw meteorological data to obtain meteorological data to be processed.
[0084] Figure 2 Shown is a flow chart of data preprocessing in a meteorological data processing method based on improved sliding average filtering in one embodiment of the present application; Figure 2 As shown, in one embodiment of the present application, the cleaning of the raw meteorological data to obtain the meteorological data to be processed includes the following steps S110 to S120.
[0085] Step S110 , processing the error data of the meteorological raw data to obtain initial cleansing data; the processing of the error data of the meteorological raw data includes boundary value detection processing and time consistency detection processing.
[0086] Step S120 , normalizing the initial cleaned data to obtain meteorological data to be processed.
[0087] Figure 3 The figure shows a schematic diagram of the principle process of data preprocessing in the meteorological data processing method based on improved sliding average filtering in one embodiment of the present application; the meteorological raw data required for filtering is collected, and the original observation data has problems such as missing values. In order to improve the quality of the data, enhance the value of the data, reduce redundancy, and accelerate data analysis, this embodiment cleans the raw data. The specific process includes data quality control (i.e., processing the error data of the meteorological raw data to obtain the initial cleaned data) and data homogenization.
[0088] Specifically, in one embodiment of the present application, the boundary value detection processing includes eliminating data that does not fall within the boundary value range; the time consistency detection processing includes detecting whether the data is missing in time, and filling in the missing data, and detecting whether the meteorological element sampling value exceeds the range of change within a certain time, and eliminating the data that exceeds the range of change.
[0089] Meteorological data often suffer from errors during observation. Depending on the nature and cause of the error, meteorological observation data may contain three types of errors: random error (mean is zero), systematic error (mean is not zero, mainly due to habitual errors, instrument deviation, etc.), and negligent error (observation, calculation, transmission, etc.). Random error is mainly eliminated through multiple sampling, while quality control focuses on reducing the impact of the latter two types of errors on the data.
[0090] Specifically, in one embodiment, in the boundary value detection process, the boundary value check is divided into climatological boundary value checks according to its characteristics. Data that does not fall within the boundary value range is treated as abnormal data. Whether the upper limit of the boundary value range is set reasonably determines the effectiveness of the boundary value check. If the upper limit is set too high, erroneous data may be treated as correct data, resulting in the use of erroneous data in the application; if the upper limit is set too low, individual correct data with excessively large values may be treated as erroneous data, which is precisely the data that is very helpful for studying extreme weather and meteorological warnings.
[0091] The time consistency detection process includes time continuity check and meteorological element characteristic check. The time continuity check mainly checks whether the data is missing in time and processes the missing data.
[0092] When missing data is relatively rare, consider directly deleting records with missing values. This approach is suitable when there is little missing data and the amount of retained data is sufficiently large. If the missing percentage of a variable is too high, consider deleting the entire variable. This is usually appropriate when the variable has little impact on the analysis results.
[0093] This embodiment can also fill missing values, and the filling methods include but are not limited to constant filling, interpolation filling, and model prediction filling.
[0094] Constant filling uses a fixed value to fill missing values, such as using statistical values such as the mean, median, and mode. This method is simple and fast, but may introduce additional bias, especially when the filled value differs significantly from the actual value. Interpolation methods predict and fill missing values by fitting existing data, such as linear interpolation and polynomial interpolation. This method takes into account the trend of the data, but has high computational complexity and may not be accurate when the data fluctuates significantly. Model prediction filling uses other complete data to build a model (such as linear regression, decision tree, KNN, etc.) to predict and fill missing values. This method can effectively utilize the correlation between data, but requires sufficient computing resources, and the choice and accuracy of the model have a significant impact on the filling effect.
[0095] Meteorological element characteristic detection and processing utilizes the fact that meteorological elements exhibit regular patterns over time, avoiding significant jumps within relatively short periods of time. This method verifies whether changes in meteorological element values conform to these patterns. The sampled values of each meteorological element must not exceed a certain range of variation within a certain timeframe; data outside this range is considered suspicious. Figure 4 It shows a schematic diagram of jump data in a meteorological data processing method based on improved sliding average filtering in an embodiment of the present application. Figure 4 The data circled in the middle are suspicious and need to be eliminated.
[0096] Because meteorological variables fluctuate widely over a year, using these data directly without correction can introduce significant errors in calculations and assessments. Data normalization involves scaling data to bring data from different distributions into a common range. It is often used in the processing of indicators for comparison and evaluation to eliminate inconsistencies in data unit magnitudes and convert the data into dimensionless, pure values, facilitating comparison and weighting of indicators with different units and magnitudes.
[0097] The data standardization process can not only unify the dimensions and smooth the gradients between different batches and different layers of data, but also prevent the model gradient from exploding or spreading. After the data is standardized, the "gradient gap" of the original data will be narrowed as a whole, effectively preventing a certain layer of data in the model from experiencing large fluctuations, and having the effect of smoothing the gradient. The most typical data standardization operation is normalization. This embodiment adopts minimum and maximum normalization, and the formula is as follows:
[0098]
[0099] Step S200 , performing weighted sliding average filtering on the meteorological data to be processed to obtain meteorological filtered data.
[0100] In this embodiment, after cleaning the raw meteorological data to obtain the meteorological data to be processed, the meteorological data to be processed is subjected to weighted sliding average filtering to obtain meteorological filtered data. Specifically, the meteorological filtered data is obtained by defining a sliding window size and performing weighted sliding average filtering on the data. Figure 5 Shown is a flow chart of obtaining meteorological filtered data in a meteorological data processing method based on improved sliding average filtering in one embodiment of the present application; Figure 5 As shown, in one embodiment of the present application, the weighted sliding average filtering is performed on the meteorological data to be processed, and obtaining the meteorological filtered data includes the following steps S210 to S240.
[0101] Step S210: Determine the initial sliding window.
[0102] The initial sliding window W can be determined according to the characteristics of meteorological data and specific processing requirements.
[0103] Step S220: dynamically adjust the size of the initial sliding window based on the volatility and periodicity of the data to form a required sliding window.
[0104] This embodiment dynamically adjusts the sliding window size to adapt to the volatility and periodicity of the data; for example, the window is reduced when the data fluctuates greatly, and the window is expanded when the data is relatively stable.
[0105] Step S230: assigning a weight to each data point in the required sliding window.
[0106] This embodiment assigns a weight ωi to each data point in the sliding window, where the weight can be set according to factors such as the time position of the data point and the size of the data value.
[0107] Step S240: Acquire meteorological filtering data based on the meteorological data to be processed and the weight.
[0108] In one embodiment of the present application, the method of obtaining meteorological filtering data based on the meteorological data to be processed and the weight is:
[0109]
[0110] Among them, x i is the original data, ω i is the weight coefficient, y t is the meteorological filtering data, W is the initial sliding window, t is the sequence number of the data point, and i is the sequence number of each data point in the sliding window.
[0111] Step S300: performing trend analysis and boundary processing on the meteorological filtering data.
[0112] In one embodiment of the present application, the trend analysis method includes a fitting model of linear regression or polynomial regression.
[0113] This embodiment further adjusts the filtering results through trend analysis:
[0114] This embodiment performs trend analysis on the filtered meteorological filter data: for example, linear regression, polynomial regression, or other suitable fitting models are used to perform trend detection on the results after weighted sliding filtering. For example, it is assumed that the fitting model is T(t), which represents the overall trend of the data;
[0115] Calculate the residual: Calculate the residual r(t) = yt - T(t) between the result after the initial weighted moving average filter and the fitted model, where yt is the data after the initial filtering. This step of calculating the residual is to compare the result of the initial filtering data with the result of the trend line. The residual represents the difference between the data and the trend line, revealing high-frequency fluctuations and noise in the data;
[0116] Adjust the residual: Smooth the residual to remove the influence of high-frequency noise and obtain the smoothed residual rs(t);
[0117] Reconstruct data: Add the smoothed residual to the fitted model to obtain the adjusted data yt2 = T(t) + rs(t), so that it more accurately reflects the trends and changes of the actual data.
[0118] Based on the above analysis, the filtering results are further adjusted to better capture the trend changes in the data. For example, the weight coefficients are adjusted to strengthen the trend signal and reduce the influence of noise.
[0119] In one embodiment of the present application, the boundary processing method includes zero padding or mirror extension.
[0120] This embodiment reduces the boundary effect by processing the data boundary: for example, using strategies such as zero padding and mirror extension to process the data boundary to reduce the boundary effect.
[0121] Among them, zero padding is to fill zero values at both ends of the data sequence to extend the sequence length; mirror extension is to copy the mirror part of the original sequence at both ends of the data sequence to extend the sequence length.
[0122] Performing weighted sliding average calculation on the data after boundary processing can ensure the accuracy of boundary data processing.
[0123] Step S400 : reconstructing the meteorological filtering data after trend analysis and boundary processing, so as to make the meteorological filtering data continuous and complete.
[0124] This embodiment reconstructs the filtered data to ensure the continuity and integrity of the data.
[0125] Specifically, the filtered, trend-adjusted, and boundary-processed data is reconstructed to ensure its continuity and integrity. The reconstructed data can also be verified and validated to check for outliers or discontinuities. Finally, the processed data is stored in a database for subsequent analysis and forecasting.
[0126] According to the above steps, the temperature data of a certain place in a certain month is used for filtering, and the results are as follows Figure 6 As shown: Figure 6 This figure shows a comparison of the filtering methods in the meteorological data processing method based on the improved sliding average filter in one embodiment of the present application. The figure above shows the original data, the data after weighted sliding average filtering, and the data after trend analysis, mirror expansion, and adjustment. The following is a description of each curve:
[0127] Original data (blue curve): contains noise and outliers;
[0128] Cleaned Data (green curve): shows the original meteorological data after cleaning, removing obvious outliers (such as 10.0℃);
[0129] Weighted Moving Average Filter (red curve Weighted Moving Data): The data after applying the weighted moving average filter smoothes the original data and removes some noise;
[0130] Combined with trend analysis and adjustment (purple curve Trend-AdjustedData): Based on the weighted moving average filter, combined with the trend analysis method, the filtering results are further adjusted to better capture the trend changes in the data, and the effect is smoother and more continuous;
[0131] Mirrored Moving Average (orange curve): processes boundary data and reduces boundary effects;
[0132] Adjusted mirrored data (brown curve Adjusted MirroredData): The final data after combining trend analysis and boundary processing is the smoothest and most reliable.
[0133] Lanczos filtering result (pink curve Lanczos Filtered Data): It can be seen that the boundary data is set to missing values due to the limitations of the filter.
[0134] Specific comparison:
[0135] Data smoothing and trend retention: Lanczos filtering and weighted moving average filtering can both smooth data and remove high-frequency noise. Weighted moving average filtering, combined with trend analysis and adjustment, can better capture data trend changes.
[0136] Boundary processing: Lanczos filtering can cause data loss at boundaries, resulting in missing values. This embodiment effectively solves the boundary effect problem through methods such as mirror expansion and trend adjustment, ensuring data continuity.
[0137] Frequency response: Lanczos filtering has a clear bandpass characteristic and mainly targets low-frequency components. The weighted moving average filtering of this embodiment can adapt to different frequency components by flexibly adjusting the weights, and the frequency response is more flexible.
[0138] In terms of frequency band characteristics, the Lanczos filter and the weighted moving average filter method of this embodiment have certain consistency in data smoothing and high-frequency noise removal. However, by combining weighted moving average filtering, trend analysis, and boundary processing, this embodiment can better address boundary effects, ensure data continuity and integrity, and thus achieve better results in practical applications. Therefore, this embodiment exhibits greater advantages in comprehensive filtering effects and boundary processing.
[0139] This method, based on the improved sliding average filter, effectively removes noise and outliers from meteorological data, improving data quality and reliability. This method, based on the traditional sliding average filter, incorporates innovative improvements, including weighted sliding average, dynamic sliding window, and trend analysis adjustments. It overcomes existing issues such as boundary effects and poor noise removal, while maintaining low computational complexity and ease of implementation. It is suitable for real-time processing of large-scale meteorological data and can be widely used in fields such as meteorological data processing, weather forecasting, disaster warning, and environmental monitoring.
[0140] The beneficial effects of the embodiment are specifically manifested in the following aspects:
[0141] 1. Effectively remove noise and outliers: By using weighted moving average filtering and dynamic sliding window technology, noise and outliers in meteorological data can be filtered out more accurately. Compared with traditional Kalman filtering and wavelet transform filtering methods, the filtering method of this embodiment has a more significant denoising effect, especially when dealing with abnormal data introduced by sensor failures and environmental interference;
[0142] 2. Reduce boundary effects: By adopting specific boundary processing strategies (such as zero padding, mirror extension, etc.), the signal distortion caused by boundary effects in the filtering process is effectively reduced, the smoothness and continuity of the filtering results are ensured, and the accuracy of meteorological data processing is improved;
[0143] 3. Improved computational efficiency: While retaining the computational simplicity of sliding average filtering, the method of this embodiment optimizes the computational process through weighted processing and dynamic adjustment of the sliding window size. While ensuring efficient denoising, it maintains low computational complexity, making it suitable for real-time processing of large-scale meteorological data and highly practical.
[0144] 4. Enhanced trend capture capabilities: Combined with trend analysis methods, further adjustments to filtering results can better capture trend changes in meteorological data, improving the analysis and prediction capabilities of meteorological data, and facilitating applications in weather forecasting, disaster warning, and environmental monitoring.
[0145] 5. Adaptability to multi-source data: The method of this embodiment can process meteorological data collected from multiple sources such as weather stations, satellites, radars, and sensors. It has a wide range of applications and can achieve consistent excellent performance in different data sources and application scenarios.
[0146] It can be seen that this embodiment provides an efficient meteorological data processing method, which significantly improves the smoothness and accuracy of the data through steps such as weighted sliding average filtering, trend analysis and adjustment, boundary processing and data reconstruction, solves the boundary effect and data missing problems existing in the existing technology, and is suitable for a variety of meteorological data analysis and prediction scenarios.
[0147] This embodiment also provides a meteorological data processing device based on improved sliding average filtering, Figure 7 Shown is a principle block diagram of a meteorological data processing device based on improved sliding average filtering in one embodiment of the present application; Figure 7 As shown, the meteorological data processing device based on improved sliding average filtering of this embodiment includes: a data preprocessing module, a weighted sliding average filtering module, a trend analysis module, a boundary processing module and a reconstruction module.
[0148] Specifically, the data preprocessing module is used to collect raw meteorological data and clean the raw meteorological data to obtain meteorological data to be processed. The data preprocessing module removes obvious outliers and missing values by cleaning and normalizing the collected raw data. The weighted sliding average filtering module is used to perform weighted sliding average filtering on the meteorological data to be processed to obtain meteorological filtered data. The weighted sliding average filtering module defines the sliding window size and performs weighted sliding average filtering on the data. The trend analysis module is used to perform trend analysis on the meteorological filtered data. The trend analysis module combines the trend analysis method to further adjust the filtering results to better capture the trend changes of the data. The boundary processing module is used to perform boundary processing on the meteorological filtered data to reduce boundary effects. The reconstruction module is used to reconstruct the meteorological filtered data after trend analysis and boundary processing to make the meteorological filtered data continuous and complete.
[0149] In this embodiment, the implementation principles of each module of the meteorological data processing device based on the improved sliding average filter are the same as the implementation principles of each step of the above-mentioned meteorological data processing method based on the improved sliding average filter. The implementation principles and specific implementation methods of the meteorological data processing device based on the improved sliding average filter will not be repeated here.
[0150] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is cached. When the computer program is executed by a processor, the meteorological data processing method based on improved sliding average filtering provided in any embodiment of the present application is implemented.
[0151] In the embodiment of the present application, any combination of one or more storage media can be used. The storage medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium that contains or caches a program that can be used by or in combination with an instruction execution system, device or device.
[0152] An embodiment of the present application also provides an electronic device. Figure 8 Shown is a structural diagram of an electronic device 10 provided in an embodiment of the present application. In some embodiments, the electronic device 10 can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) and other terminal devices. In addition, the meteorological data processing method based on the improved sliding average filter provided in the present application can also be applied to databases, servers and service response systems based on terminal artificial intelligence. The embodiment of the present application does not impose any restrictions on the specific application scenarios of the meteorological data processing method based on the improved sliding average filter.
[0153] like Figure 8 As shown, the electronic device 10 provided in an embodiment of the present application includes a memory 101 and a processor 102 .
[0154] The memory 101 is used to cache computer programs; preferably, the memory 101 includes: ROM, RAM, disk, USB flash drive, cache card or optical disk, etc., various media that can cache program codes.
[0155] Specifically, the memory 101 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory. The electronic device 10 may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 101 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the various embodiments of the present application.
[0156] The processor 102 is connected to the memory 101 and is used to execute the computer program cached in the memory 101 so that the electronic device 10 executes the meteorological data processing method based on improved sliding average filtering provided in any embodiment of the present application.
[0157] Optionally, the processor 102 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0158] Optionally, the electronic device 10 in this embodiment may further include a display 103. The display 103 is communicatively connected to the memory 101 and the processor 102, and is configured to display a GUI interface related to the meteorological data processing method based on the improved sliding average filter.
[0159] Furthermore, the electronic device 100 also includes other components such as a firewall, a load balancer, a communication component, and a power supply component. Figure 8 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 8 Components shown.
[0160] The above embodiments can be implemented in whole or in part through software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. A computer program product includes multiple computer instructions. When the computer program instructions are loaded or executed on a computer, the processes or functions according to the present embodiment are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.
[0161] In summary, the meteorological data processing method based on the improved moving average filter proposed in this embodiment overcomes the shortcomings of Lanczos filtering, wavelet transform filtering, and Kalman filtering in processing meteorological data by comprehensively using multiple methods such as weighted moving average filtering, trend analysis and adjustment, and boundary processing, and has the following significant advantages:
[0162] 1) Boundary effect processing: effectively solve the problem of data boundary effect and ensure data continuity and integrity;
[0163] 2) Simple calculation: low computational complexity, suitable for real-time processing applications;
[0164] 3) Flexible adaptability: by adjusting weights and window sizes, it can adapt to different frequency components and data characteristics;
[0165] 4) Strong robustness: It has a significant effect on processing noise and abnormal data and is highly robust;
[0166] Through these advantages, this embodiment can provide higher accuracy and reliability in the process of meteorological data processing. Therefore, this embodiment effectively overcomes various shortcomings of the prior art and has high industrial application value.
[0167] The above embodiments are merely illustrative of the principles and effects of the present embodiments and are not intended to limit the present embodiments. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present embodiments. Therefore, all equivalent modifications or alterations made by persons skilled in the art, including those of ordinary skill in the art, without departing from the spirit and technical concepts disclosed in the present embodiments shall be covered by the claims of the present embodiments.
Claims
1. A meteorological data processing method based on improved sliding average filtering, characterized in that: include: Collecting raw meteorological data, and cleaning the raw meteorological data to obtain meteorological data to be processed; Performing weighted sliding average filtering on the meteorological data to be processed to obtain meteorological filtered data; Performing trend analysis and boundary processing on the meteorological filtering data; The meteorological filter data after trend analysis and boundary processing is reconstructed so that the meteorological filter data has continuity and integrity.
2. The meteorological data processing method based on improved sliding average filtering according to claim 1 is characterized in that: The cleaning of the raw meteorological data to obtain the meteorological data to be processed comprises: Processing the error data of the original meteorological data to obtain initial cleansing data; the processing of the error data of the original meteorological data includes boundary value detection processing and time consistency detection processing; The initial cleaning data is normalized to obtain the meteorological data to be processed.
3. The meteorological data processing method based on improved sliding average filtering according to claim 1 is characterized in that: The boundary value detection process includes eliminating data that does not fall within the boundary value range; the time consistency detection process includes detecting whether the data is missing in time, and filling in the missing data, as well as detecting whether the meteorological element sampling value exceeds the range of change within a certain time, and eliminating the data that exceeds the range of change.
4. The meteorological data processing method based on improved sliding average filtering according to claim 3 is characterized in that: The step of performing weighted sliding average filtering on the meteorological data to be processed to obtain meteorological filtered data comprises: Determine the initial sliding window; Dynamically adjust the size of the initial sliding window based on the volatility and periodicity of the data to form a required sliding window; Assigning a weight to each data point within the desired sliding window; Acquire meteorological filtering data based on the meteorological data to be processed and the weight.
5. The meteorological data processing method based on improved sliding average filtering according to claim 4 is characterized in that: The method of obtaining meteorological filtering data based on the meteorological data to be processed and the weight is: Among them, x i is the original data, ω i is the weight coefficient, y t is the meteorological filtering data, W is the initial sliding window, t is the sequence number of the data point, and i is the sequence number of each data point in the sliding window.
6. The meteorological data processing method based on improved sliding average filtering according to claim 1 is characterized in that: The trend analysis method includes a linear regression or polynomial regression fitting model.
7. The meteorological data processing method based on improved sliding average filtering according to claim 1 or 6, characterized in that: The boundary processing method includes zero filling or mirror extension.
8. A meteorological data processing device based on improved sliding average filtering, characterized in that: include: The data preprocessing module is used to collect raw meteorological data, clean the raw meteorological data, and obtain the meteorological data to be processed; A weighted moving average filtering module is used to perform weighted moving average filtering on the meteorological data to be processed to obtain meteorological filtering data; A trend analysis module, used for performing trend analysis on the meteorological filtering data; A boundary processing module, used for performing boundary processing on the meteorological filtering data; The reconstruction module is used to reconstruct the meteorological filter data after trend analysis and boundary processing, so that the meteorological filter data has continuity and integrity.
9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to run the computer program to implement the steps of the meteorological data processing method based on improved sliding average filtering as described in any one of claims 1 to 7.
10. A computer storage medium storing program instructions, characterized in that: When the program instructions are executed, the steps of the meteorological data processing method based on improved sliding average filtering as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Robust smooth filtering method, system and device and storage medium
CN113569686A
Underground water reserve prediction method
CN118095573A
Meteorological equipment state big data analysis and intelligent maintenance system
CN118445563A
Method and device for detecting pulling resistance of anchoring device
CN118467933A
