Method and system for analyzing multi-source heterogeneous real-time meteorological data

Abstract

This application provides a method and system for analyzing multi-source heterogeneous real-time meteorological data, relating to the field of data processing technology. The method includes: real-time acquisition of meteorological data; handling missing values ​​and / or outlier detection in the meteorological data to determine clean meteorological data; standardizing the clean meteorological data to determine standard meteorological data; storing the standard meteorological data in three levels; and performing trend forecasting based on data analysis needs and the standard meteorological data to issue early warnings based on the forecast results. This application effectively solves the problem of integrating and utilizing multi-source and heterogeneous data in the meteorological field. Through a unified processing flow, it transforms disorganized raw data into standardized, usable data assets, breaking down data silos and improving the efficiency and accuracy of meteorological forecasting and early warning.

Description

Technical Field

[0001] This application relates to the field of data processing technology, for example to a method and system for analyzing multi-source heterogeneous real-time meteorological data. Background Technology

[0002] With the continuous upgrading of meteorological monitoring equipment and sensors, the volume of meteorological data has exploded, encompassing multiple sources including ground meteorological stations, satellite remote sensing, and meteorological sensors. While current meteorological data governance technologies have addressed basic needs such as data collection, storage, and analysis to some extent, they still face numerous challenges in areas such as data quality control, real-time processing, and intelligent analysis. This, in turn, limits the development and practical application effectiveness of meteorological data governance technologies.

[0003] In related technologies, traditional data cleaning and standardization methods are generally used to analyze meteorological data. Traditional data cleaning methods, such as mean imputation and linear interpolation, have limited effectiveness when dealing with complex, nonlinear, or periodically fluctuating data, and struggle to accurately fill in missing values ​​or correct outliers. Data standardization methods rely on fixed rules, lacking flexibility and scalability, and are ill-suited for handling large-scale heterogeneous data. Furthermore, most data processing platforms rely on batch processing, which cannot meet the demands of high real-time scenarios such as extreme weather warnings. The inefficiency of multi-source data fusion leads to poor performance in meteorological forecasting and decision support, impacting the efficiency and accuracy of meteorological warnings, climate monitoring, and other fields. Summary of the Invention

[0004] This application aims to provide a method and system for analyzing multi-source heterogeneous real-time meteorological data.

[0005] According to one aspect of this application, a method for analyzing multi-source heterogeneous real-time meteorological data is proposed, comprising:

[0006] Real-time meteorological data collection; Perform missing value processing and / or outlier detection processing on meteorological data to determine clean meteorological data; Standardize the clean meteorological data to determine the standard meteorological data; Standard meteorological data is stored in three levels, and trend forecasts are made based on data analysis needs and standard meteorological data, so as to issue early warnings based on the forecast results.

[0007] According to one aspect of this application, a system for analyzing multi-source heterogeneous real-time meteorological data is proposed, comprising: The data acquisition module is used to collect meteorological data in real time. The data preprocessing module is used to process meteorological data for missing values ​​and / or outlier detection to determine clean meteorological data. The data standardization module is used to standardize clean meteorological data in order to determine standard meteorological data; The data management module is used to store standard meteorological data in three levels and to perform trend forecasting based on data analysis needs and standard meteorological data, so as to issue early warnings based on the forecast results.

[0008] According to one aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing a computer program that, when executed by the processor, causes the processor to perform the method described above.

[0009] According to one aspect of this application, a non-transitory computer-readable medium is proposed, on which readable instructions are stored, which, when executed by a processor, cause the processor to perform the method described above.

[0010] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application.

[0011] Beneficial effects: Through the embodiments provided in this application, the system effectively addresses issues such as missing data, errors, and inconsistent formats in raw meteorological data through the collaborative work of the data preprocessing module and the data standardization module. This provides a high-precision and highly consistent data foundation for subsequent analysis and decision-making, greatly improving the reliability and credibility of the analysis results. The system adopts a streaming processing approach, performing real-time cleaning, standardization, and storage on the collected data. This pipeline-style processing significantly reduces data latency, ensuring the shortest possible time interval from data acquisition to availability, thus meeting the real-time and timely requirements necessary for meteorological analysis, especially for severe weather warnings. The data management module employs a three-tiered storage strategy, achieving an optimal balance between storage cost and access efficiency. The system can quickly schedule relevant data according to data analysis needs, significantly improving the efficiency of historical data querying, batch analysis, and model training. The system extends beyond data management, integrating analysis and prediction functions. The data management module can perform analysis and trend prediction based on standardized, high-quality data and automatically trigger warnings. This changes the traditional passive data viewing model, realizing a shift from "people looking at data" to "data finding people," significantly improving the foresight and proactiveness in responding to meteorological disasters and providing users with valuable lead time. This application effectively solves the challenge of integrating and utilizing multi-source (from different sensors, satellites, radars, etc.) and heterogeneous (different formats, protocols, and precision) data in the meteorological field. Through a unified processing workflow, it transforms disorganized raw data into standardized, usable data assets, breaking down data silos and improving the efficiency and accuracy of weather forecasting and early warning. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0013] Figure 1 A flowchart illustrating the method for analyzing multi-source heterogeneous real-time meteorological data provided in this application embodiment; Figure 2 A block diagram of a multi-source heterogeneous real-time meteorological data analysis system provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0015] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0016] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0017] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0018] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0019] For specific implementation details, please refer to the following examples.

[0020] Figure 1 This is a flowchart illustrating the method for analyzing multi-source heterogeneous real-time meteorological data provided in an embodiment of this application. Figure 1 As shown, the method includes steps S10, S11, S12 and S13.

[0021] In step S10, meteorological data is collected in real time.

[0022] The meteorological data collected in this application may come from sources such as ground weather stations, satellites, radar, and radiosondes. Meteorological data can be collected in real time from the aforementioned various meteorological devices and sensors.

[0023] In step S11, missing value processing and / or outlier detection processing are performed on the meteorological data to determine the clean meteorological data.

[0024] In this application, meteorological data can be preprocessed according to a preset cleaning and preprocessing method.

[0025] In some implementations, a data collection table can be pre-defined, and data can be collected according to the requirements in the table. If a certain data point in the table is not collected, the corresponding position in the table will be empty, indicating missing data. The data format required for different positions in the collection table can be pre-defined. If the format is incorrect, it can be cleaned or outlier handling can be performed.

[0026] In other implementations, if missing value handling and outlier detection need to be performed simultaneously, the order is to perform outlier detection first, followed by missing value handling.

[0027] In step S12, the clean meteorological data is standardized to determine standard meteorological data.

[0028] In this application, in order to ensure data compatibility between different data sources, Z-score (StandardScore) combined with seasonal sliding window correction can be used to standardize the clean meteorological data to obtain standard meteorological data.

[0029] In step S13, standard meteorological data is stored in three levels, and trend prediction is performed based on data analysis needs and standard meteorological data, so as to issue early warnings based on the prediction results.

[0030] In this application, multiple data storage methods can be pre-configured. In some implementations, three methods can be selected to store standard meteorological data to address different situations. Storage methods may include: using Hadoop's HDFS (Hadoop Distributed File System) as a foundation, dividing standard meteorological data into small files by station-time (e.g., weather station A, time a), and using MapReduce (Hadoop's computing framework) for parallel writing and redundant backup to solve the problems of high throughput and high fault tolerance for massive data; pre-building a columnar database on top of HDFS, storing elements such as temperature, humidity, and air pressure by column, and using dictionary encoding and bit compression algorithms to achieve a storage compression rate of 3–10 times; introducing a Redis (Remote Dictionary Server) distributed memory cache cluster to store hourly automatic station data that is frequently queried by the business in key-value pairs, where _surf1h_ station number is the key and 210 elements are hash fields, residing in memory, supporting millisecond-level retrieval, and avoiding single-node failures and memory overflows through master-slave replication and expiration strategies.

[0031] In some implementations, data analysis requirements can be used to characterize user forecasting needs, such as short-term (hourly) trend forecasting. Specifically, for short-term trend forecasting, an autoregressive integral moving average model can be pre-set, using standard meteorological data from the 24 hours prior to the current moment as model input, and outputting hourly element forecasts for the next six hours, i.e., forecasts for different types of meteorological data. For long-term trend forecasting, a linear regression model can be pre-set, incorporating seasonal features. Standard meteorological data is processed to extract multi-feature data containing core features, which are then used as model input. These core features include: month (1-12, representing the season), holiday status (0 / 1, excluding human interference), and historical statistical features, which may include: the average daily humidity over the past 30 days and the maximum / minimum humidity over the past 7 days. In some implementations, each sample contains 3-5 features. The model outputs the predicted average daily humidity for the next seven days, etc.

[0032] The above methods are used to make predictions, and then the corresponding early warning methods are matched with the prediction results.

[0033] This application utilizes a collaborative data preprocessing module and a data standardization module to effectively address issues such as missing data, errors, and inconsistent formats in raw meteorological data. This provides a high-precision, highly consistent data foundation for subsequent analysis and decision-making, significantly enhancing the reliability and credibility of the analysis results. The system employs a streaming processing approach, instantly cleaning, standardizing, and storing real-time acquired data. This pipeline-like processing significantly reduces data latency, minimizing the time interval from data acquisition to usability, thus meeting the real-time and timely requirements necessary for meteorological analysis, particularly for severe weather warnings. The data management module adopts a three-tiered storage strategy, achieving an optimal balance between storage costs and access efficiency. The system can quickly schedule relevant data according to data analysis needs, greatly improving the efficiency of historical data queries, batch analysis, and model training. Beyond data management, the system integrates analysis and prediction functions. The data management module can perform analysis and trend prediction based on standardized, high-quality data and automatically trigger warnings. This changes the traditional passive data viewing model, shifting from "people looking at data" to "data finding people," significantly improving the foresight and proactiveness in responding to meteorological disasters and providing users with valuable lead time. This application effectively solves the challenge of integrating and utilizing multi-source (from different sensors, satellites, radars, etc.) and heterogeneous (different formats, protocols, and precision) data in the meteorological field. Through a unified processing workflow, it transforms disorganized raw data into standardized, usable data assets, breaking down data silos and improving the efficiency and accuracy of weather forecasting and early warning.

[0034] According to some embodiments, after meteorological data is collected, it can be filtered and compressed according to key data requirements to determine key meteorological data.

[0035] In this application, the collected meteorological data may contain redundancy, duplication, and interference. The key data requirements are the meteorological data needed for subsequent analysis, and which data is unnecessary and needs to be processed.

[0036] In some implementations, the collected meteorological data can be searched according to key data requirements, and data that does not meet the requirements can be filtered out. Pre-set compression algorithms, such as shift threshold algorithms or lossless compression algorithms, are then used to compress the data and obtain the key meteorological data.

[0037] In other implementations, lightweight edge computing software can be pre-deployed on the meteorological data source device to perform filtering.

[0038] Based on this, missing value processing and / or outlier detection processing can be performed on key meteorological data to determine pure meteorological data.

[0039] This application significantly reduces the amount of data that needs to be transmitted and stored over the network from the source of data generation through compression and selective collection, thereby reducing bandwidth and storage costs and preventing system resources from being occupied by a large amount of low-value data, thus improving overall economy and efficiency.

[0040] According to some embodiments, when performing outlier detection processing on meteorological data, the specific steps are as follows: Based on meteorological industry standards, a meteorological threshold range is determined, and the meteorological data is marked with a first outlier based on the meteorological threshold range to obtain first meteorological labeled data; Based on a preset confidence interval, the first meteorological labeled data is marked with a second outlier to obtain second meteorological labeled data; Based on a preset multivariate outlier detection method, the second meteorological labeled data is marked with a third outlier to obtain third meteorological labeled data; Based on a preset residual test method, the third meteorological labeled data is marked with a fourth outlier to obtain fourth meteorological labeled data; Based on a preset unsupervised outlier detection algorithm, the fourth meteorological labeled data is marked with a fifth outlier to obtain fifth meteorological labeled data; According to a preset neighborhood weighted median replacement method, the first, second, third, fourth, and fifth outliers are corrected, and the fifth meteorological labeled data is marked with an anomaly correction to obtain clean meteorological data.

[0041] In this application, the meteorological industry standard can be the World Meteorological Organization standard or a national meteorological industry standard. Meteorological threshold ranges, such as temperature threshold ranges, can be directly extracted from these standards. Data outside these ranges can be considered abnormal, and the abnormal data corresponds to the first outlier. The first outlier is marked according to a preset marking method, such as adding preset characters, and the marked meteorological data is used as the first meteorological marked data.

[0042] This application allows for pre-setting of the cluster number selection method corresponding to different data characteristics. The data characteristics may include data size, diversity of meteorological elements, etc., and the cluster number selection method may include elbow rule, silhouette coefficient method, etc., to determine the appropriate number of clusters for K-Means clustering.

[0043] Outlier detection based on statistical distribution can be used, assuming that meteorological data of the same type (hereinafter referred to as elements) approximately follow a normal distribution in the same climate zone and the same season. , with 3 Principles for constructing confidence intervals: .in, The mean is the location parameter of the distribution; σ is the standard deviation, which is the scale parameter of the distribution; For variance; This is used to characterize different elements. Meteorological data outside this range are considered outliers and labeled as second outliers. The first labeled meteorological data is then used as the second labeled meteorological data. In some implementations, and The system updates online with a sliding window of data from the past 30 days to adapt to seasonal changes.

[0044] In this application, to further capture multivariate anomalies, a multivariate anomaly detection method based on Mahalanobis distance can be introduced. This method is specifically designed for the sample vector corresponding to the second meteorological label data. .in, The set of data used to characterize the absence of anomalies in the second meteorological labeling data. Data used to characterize the anomalies in the second meteorological label data The number of features in the data without anomalies used to characterize the second meteorological label data.

[0045] calculate: .

[0046] in, For the second meteorological marker data, a multivariate anomaly indicator, This is the mean vector of the current cluster. Let be the covariance matrix.

[0047] if If it is, then it is judged as abnormal. The third outlier is marked to obtain the third meteorological label data.

[0048] In this application, considering the characteristics of the time series, a residual test is used after seasonal-trend decomposition, i.e., a preset residual test method is used to process the third meteorological label data, and the residual series obtained from the decomposition is analyzed. Establish an autoregressive moving average model, if If the value is 0, it indicates that the corresponding observation point is an anomaly, and the data corresponding to that observation point is the fourth anomaly value. This is then marked to obtain the overall fourth meteorological labeling data. This is a time anomaly indicator for the third meteorological marker data; This is an empirical value for time, typically 3.5; This is for estimating the residual standard deviation. The standard deviation is bound to the "spatial location" of the observation point; that is, different locations (such as the Jianghuai Plain / Qinghai-Tibet Plateau grid) have different climatic backgrounds and corresponding specific parameters. k is bound to the "meteorological element type" of the observation point. The spatiotemporal binding specifically involves: each observation point (unique identifier: spatial location + time + element) calls the corresponding spatial location and corresponding element to achieve "one-to-one" anomaly detection. The observation point is the location where meteorological data is acquired.

[0049] In this application, an unsupervised anomaly detection algorithm can also be pre-set. In some implementations, the unsupervised anomaly detection algorithm can be a density-based Local Outlier Factor (LOF) detection method. This method is used to perform secondary verification on the spatial grid data in the fourth meteorological labeling data. In the specific implementation process, for any spatial point in the fourth meteorological labeling data... ,calculate:

[0050] in, Spatial anomaly indicators used to characterize fourth meteorological label data; Candidate outliers in the LOF algorithm Any sample point in the nearest neighbor set; for Nearest neighbor set; Used to characterize locally reachable density. The nearest neighbor set is obtained through two steps: "spatial range limitation + meteorological background filtering".

[0051] The core relationship with meteorological data is "adapting to the spatial continuity of meteorological elements": the spatial distribution of meteorological data (such as temperature and precipitation) is affected by topography, climate zones, etc. Neighbors selected solely by distance may not have meteorological similarities (e.g., a mountaintop and its foot may be close but have large temperature differences), while data selected based on meteorological background... The nearest neighbor set can ensure that the meteorological patterns of the nearest points are consistent with those of the target point (such as similar temperature / precipitation distribution trends in the same region or terrain), providing a reliable reference for subsequent judgment on whether the target point violates local meteorological patterns.

[0052] After processing the meteorological data, it is used as input to an algorithm. The algorithm uses mathematical and physical models to extract meteorological patterns from the data and then obtains prediction results. If... Then mark Points represent spatial anomalies; the anomalous data can be considered the fifth anomaly, and the overall data yields the fifth meteorological labeling data. As a spatial empirical threshold, in some implementations, .

[0053] After completing the above anomaly detection steps, all detected outliers—namely, the first, second, third, fourth, and fifth outliers—are uniformly corrected using a neighborhood-weighted total bit replacement method. In some implementations, each outlier can be considered an anomaly observation point, and nine normal values ​​are selected centered on this observation point, including two time points before and after it, and four spatially adjacent grid points. It is a composite distance weighted by time interval and spatial distance.

[0054] The calculation steps are as follows: 1. Assume the collection time for missing / outlier data is... Candidate sample time is Calculate the time interval . , Coordinates of missing / abnormal data collection points. , The coordinates of the candidate sample collection points, , Calculate spatial distance Among them, missing data time directly corresponds to "timestamp of no observation record"; abnormal data time corresponds to "observation timestamp of abnormality detected"; candidate sample time refers to observation timestamps around T0 whose corresponding data is "normal" (determined to be problem-free after anomaly detection). In some implementations, the time range is initially delineated based on "time proximity", and then the observation rhythm of the two times (e.g., both are on the hour, both are every six hours) is checked to see if they are consistent, ultimately providing "normal reference samples in the time dimension" for outlier correction.

[0055] 2. Regarding and Standardization can be implemented in some ways, allowing for... Standardize to [0, 1], Standardize to [0, 1] to eliminate dimensional differences.

[0056] 3. Calculate the composite distance according to the preset weights. Time weight (can be 0.6). For spatial weights (0.4 can be selected), and This weight can be adjusted based on the type of meteorological element. For "time-sensitive" meteorological elements (such as elements that change rapidly in a short period of time, like thunderstorms), the time-sensitive element α is increased, making the time factor more important. For "spatial-sensitive" meteorological elements (such as elements with large regional differences, like wind speed affected by topography), the spatial-sensitive element β is increased, making the spatial factor more important.

[0057] If Then mark The weighting formula for outlier correction when points represent spatial anomalies. Then calculate the weighted median. , here Used to characterize composite distance in time and space. Characterizing the first Clusters. The weighted median is used as the replacement value for the corresponding data, and the fifth meteorological label data is marked with an "anomaly correction" label and weight to ensure traceability and obtain the final clean meteorological data.

[0058] In some implementations, for scenarios involving missing values, it is also necessary to calculate the weight for handling missing values. Regarding the formula mentioned above, [the following is a more detailed explanation:] Replace with , Euclidean distance is used to characterize "non-missing meteorological elements".

[0059] The system in this application does not rely on a single detection method, but integrates five major categories of methods: rule-based (industry standards), statistical (confidence intervals), correlation (multivariate detection), time-series features (residual testing), and machine learning (unsupervised algorithms). This "five-fold filtering" mechanism ensures that various types of anomalies (such as sudden instrument errors, slow drift errors, and multivariate correlation errors) can be captured from different dimensions and levels, greatly reducing the false negative and false positive rates of outliers, thereby producing clean meteorological data with ultra-high accuracy and reliability.

[0060] According to some embodiments, when handling missing values ​​in meteorological data, the specific steps may include: detecting the meteorological data and determining the missing proportion of the missing data; and determining the corresponding missing value handling method based on the missing proportion.

[0061] In this application, the percentage of missing data can be determined based on the missing locations and total data locations in the collection table set above, and this percentage can be used as the missing ratio.

[0062] In some implementations, different missing value handling methods corresponding to different missing ratios can be pre-set, and corresponding processing can be carried out during the specific implementation process.

[0063] The system in this application does not use a single, one-size-fits-all imputation method (such as global mean imputation) for all missing data. Instead, it first diagnoses the severity of the missing data (missing percentage) and then dynamically selects the most suitable processing algorithm based on the diagnosis. This mechanism ensures that the processing strategy is highly matched with the actual situation of missing data, avoids introducing new biases due to inappropriate method selection, and thus significantly improves the scientific nature and accuracy of missing value repair.

[0064] According to some embodiments, in the process of determining the missing value handling method based on the missing ratio, if the missing ratio is less than a preset ratio threshold, the missing value handling method can be determined to be deleting missing data; if the missing ratio is greater than or equal to the preset ratio threshold, the missing value handling method can be determined to be filling missing data.

[0065] In some implementations, if the missing percentage is low (e.g., less than 5%), a deletion method is used to remove missing data. If the missing percentage is high (5%-30%), K-Means clustering is used for data imputation. If the missing percentage is greater than 30%, it may be due to a malfunction in the data acquisition device; a high percentage of missing data renders the data less valuable, so it is not considered.

[0066] This application employs a clearly defined preset proportion threshold as the decision-making basis, making the missing value handling process completely objective and rule-based. This avoids fluctuations in processing results caused by subjective judgment, greatly enhancing the stability and repeatability of the data processing workflow. The decision logic only requires comparison, resulting in extremely fast execution. For data with a low missing proportion, direct deletion is the fastest processing method, incurring almost no computational overhead. For data with a high missing proportion, although filling requires computation, the simple rules make the decision itself extremely quick, improving the overall efficiency of the module.

[0067] According to some embodiments, when the missing data ratio is greater than or equal to a preset threshold, the cluster number selection method can be determined based on the characteristics of the meteorological data; the target cluster number can be determined based on the cluster number selection method and the meteorological data; the meteorological data can be clustered according to the target cluster number and the K-Means clustering method to determine the meteorological cluster data; the cluster to which the missing data in the meteorological cluster data belongs can be obtained, and non-missing sample points can be selected from the clusters according to a preset non-missing sample selection method; the similarity between the non-missing sample points and the corresponding missing data can be determined according to a preset similarity weight correspondence, so as to determine the imputation value based on the similarity and the weight of the cluster; the corresponding missing data can be replaced according to the imputation value to determine the clean meteorological data.

[0068] In this application, different cluster selection methods can be preset for different data characteristics. The data characteristics may include data size, diversity of meteorological elements, etc. The cluster selection methods may include elbow rule, silhouette coefficient method, etc., to determine the appropriate number of clusters for K-Means clustering.

[0069] Randomly select M sample points as initial cluster centers, or use the K-Means++ algorithm to select initial centers to improve the stability of clustering results.

[0070] Calculate the distance from each sample point to each cluster center. (Usually using Euclidean distance), the sample points are assigned to the cluster containing the nearest cluster center; then the center point of each cluster is recalculated, and the mean of all sample points in the cluster is taken as the new cluster center point.

[0071]

[0072] in, The number of meteorological element variables, The total number of meteorological element variables. The value ranges from 1 to Between; k is the total number of clusters, It is the number of sample points; For the first Each sample point in the element The value at the sample point. The Kth cluster center .

[0073] The sum of squared errors within clusters is used to evaluate cluster quality and help determine the optimal number of clusters. :

[0074] in, Let K represent the sample set of the Kth cluster.

[0075] In this application, the elbow rule selects the K value with a significantly slower rate of SSE decrease by plotting the sum of squared errors (SSE) curves for different K values; the silhouette coefficient rule comprehensively considers intra-cluster compactness and inter-cluster separation, selecting the K value with the largest silhouette coefficient as the optimal number of clusters. .

[0076] The change in the value of K directly affects the magnitude of SSE (sum of squared errors), while SSE, in turn, helps determine the optimal value of K, i.e., the optimal number of clusters. .

[0077] Perform cluster center updates, the first The new cluster centers for each cluster are selected from the mean of the samples within the cluster. :

[0078] The steps of sample allocation and centroid update are repeated until the change in cluster centroids is less than a preset threshold, or the preset number of iterations is reached. At this point, the algorithm converges, the clustering process is completed, and the target number of clusters is obtained. Meteorological clustering data, which consists of multiple target clusters and meteorological data within each cluster.

[0079] Based on the labeling of missing data, the cluster to which the missing data belongs is detected, and non-missing sample points are selected from that cluster according to the non-missing sample selection method. In some implementations, the non-missing sample selection method can be a selection method based on temporal proximity.

[0080] Among them, the definition of temporal adjacency is: based on the time series continuity of meteorological data, it is believed that "the changes of meteorological elements at adjacent time points are correlated", and samples with the shortest time interval with missing / abnormal data are selected first.

[0081] Time granularity alignment: Determine the time unit based on the data collection frequency (e.g., hourly data in "hours", minute-level data in "minutes"). For example, if the missing data time is at the hour level, then adjacent samples should have their intervals calculated with that timestamp in "hour" granularity.

[0082] Time interval quantization: Refer to the above. The calculation method. Usually, the following is selected. The N smallest samples are designated as "temporally adjacent samples".

[0083] Threshold constraint: If If the data exceeds a preset threshold (e.g., the threshold for hourly data is set to 6 hours), the sample is considered to have weak temporal correlation and is excluded from the "time adjacent" range.

[0084] Definition of spatial proximity: Based on the spatial correlation of meteorological elements (such as small differences in temperature and humidity between adjacent areas), the "spatial proximity" is quantified by physical coordinate distance, and samples closest to the missing / abnormal data collection points are selected first.

[0085] Spatial coordinate acquisition: Each meteorological data sample is bound to the latitude and longitude coordinates (Lat, Lon) of the collection point (such as the coordinates of the ground meteorological station or the coordinates of the center point of the satellite grid).

[0086] Spatial distance calculation: The spherical distance between two sampling points is calculated using the Haversine formula (applicable to Earth surface coordinates), as follows: ,

[0087] Where R is the average radius of the Earth. Coordinates of missing / abnormal data collection points ; represents the coordinates of the candidate sample collection points, which are pre-set to select collection points that are spatially close to the target collection point and have a strong correlation with meteorological elements. , The magnitude of the difference between latitude and longitude directly affects the length of the spherical distance, and the trigonometric function relationship in the formula realizes the quantitative mapping from the "angle difference" to the "actual distance on the ground".

[0088] Proximity threshold and filtering: A preset spatial distance threshold (e.g., 10km) is used to select only samples where d ≤ the threshold. Here, d is the spherical distance between the target sampling point and the candidate sample sampling point.

[0089] The choice between "temporally adjacent" and "spatially similar" samples depends primarily on the dominant factors influencing meteorological changes (temporal continuity takes precedence / spatial correlation takes precedence) and data availability. Specific scenario divisions are as follows: 1. Scenarios that prioritize time-adjacent samples When the temporal correlation of meteorological elements is significantly stronger than that of spatial elements, or when the spatial sample missing rate is high, temporally adjacent samples should be used to fill / correct the gaps. Typical scenarios include: Rapidly changing meteorological elements: These include wind speed, precipitation, and air pressure, which change significantly over a short period (hourly), but spatially they may vary greatly due to terrain (such as mountains or rivers). For example, if a station lacks wind speed data at 14:00, wind speed data at adjacent times (13:00, 15:00) can more accurately reflect the wind speed change trend at that station, while wind speed data at a station 10km away may not be representative of the situation at that station due to local terrain differences.

[0090] Scenarios with insufficient spatial samples: If there are few (less than 3) weather stations / sensors around the missing data collection point (such as in remote areas), or if spatially adjacent samples are also missing, then temporally adjacent samples should be selected first (the ΔT threshold constraint must be met) to avoid filling errors caused by poor spatial sample quality.

[0091] Single-point continuous observation data: such as long-term temperature time series data of a single station. If data is missing at a certain moment, the temperature change pattern of the station can be directly continued by the adjacent time samples (1 hour before and 1 hour after). The reference value of spatial samples is low.

[0092] 2. Prioritize scenarios with spatially similar samples. When the spatial correlation of meteorological elements is significantly stronger than that of the temporal correlation, or when temporally adjacent samples are missing or abnormal, spatially similar samples should be used to fill or correct the gaps. Typical scenarios include: Slowly changing meteorological elements, such as daily average temperature, monthly average humidity, and average air pressure, exhibit gradual changes over long timescales (days and above), but show consistency over large spatial areas (e.g., daily average temperature differences within the same climate zone are less than 2°C). For example, if the daily average temperature of a certain station is missing, and data for the day immediately preceding or following that station is also missing, the daily average temperatures of three stations within a 10km radius can be used as a filler, resulting in less error.

[0093] Scenarios with severe time sample missing: If the missing data is within the "preceding and following time window" (e.g.) If there are many missing data points (≤6 hours) (e.g., equipment failure causing 4 consecutive hours of missing data), and the availability of time-adjacent samples is low, then samples with spatial proximity should be used instead. For example, if data from 12:00 to 15:00 is missing at a certain site and cannot be filled by time-adjacent samples, then data from surrounding sites at 12:00 should be used instead.

[0094] Meteorological elements with strong regional consistency, such as cloud cover retrieved by satellite remote sensing and humidity in vegetation cover areas, have small spatial differences within the same region (e.g., cloud cover difference within 10km is less than 10%), but may change over time due to the movement of weather systems. However, if there is no obvious weather system passing through at the missing time, spatially similar samples are more reliable.

[0095] 3. Mixed selection scenarios When both time and space samples are available, a "time-space composite weighting" method can be used (such as 9 samples in outlier correction: 4 temporally adjacent + 4 spatially similar + 1 normal sample at the current time). The weights are determined by calculating the composite distance (time interval + spatial distance weighting), which takes into account the correlation between time and space and is applicable to most common meteorological elements (such as hourly temperature and humidity).

[0096] After selecting non-missing sample points using the method described above, the similarity and corresponding weights can be determined based on the similarity weight correspondence. This similarity weight correspondence includes the weights corresponding to different types of meteorological elements, as well as the similarity corresponding to pre-set weights. The weights are determined by the similarity (higher similarity means higher weight). Samples In meteorological elements There are missing values. Its cluster is Select L nodes within the cluster that are related to The most similar non-missing sample The fill value is:

[0097] in, For extremely small positive numbers (to prevent division by zero), weight It is inversely proportional to similarity; For the sample With sample The feature distance between two samples is used to quantify the similarity between them. Characterizing the first cluster.

[0098] Replace the missing data with filler values ​​to obtain clean meteorological data.

[0099] This application uses clustering to divide the entire dataset into multiple local groups with similar meteorological characteristics (such as "high temperature and high humidity cluster" and "low temperature and dry cluster"). When imputing missing values, reference samples are only searched within the local cluster to which the missing value belongs, ensuring that the imputed value comes from the other data points that are most similar in a meteorological sense, rather than from the entire dataset, thereby greatly improving the accuracy and contextual relevance of the imputed value.

[0100] According to some embodiments, the filling accuracy can also be determined based on the filling value, meteorological data, and a preset accuracy quantification method.

[0101] This application allows for pre-setting the precision quantization method using mean squared error (MSE) and mean absolute error (MAE):

[0102]

[0103] in, This represents the total number of missing values ​​in the preset validation set. To verify the true values ​​in the set, This corresponds to the fill value. Based on the above method, fill accuracy at different angles can be obtained.

[0104] This application assigns a clear quality label (i.e., filling accuracy) to the filled values. The system no longer outputs filling results in a "black box," but can self-assess and quantify the reliability of each filling operation. This greatly enhances the transparency and interpretability of the data processing, allowing users to clearly understand the reliability of the filling results and providing crucial decision-making basis for subsequent data use.

[0105] The following describes an apparatus embodiment of this application, which can be used to perform the above-described method. For details not disclosed in the method embodiments of this application, please refer to the apparatus embodiments of this application.

[0106] Figure 2 A block diagram of a multi-source heterogeneous real-time meteorological data analysis system provided in an embodiment of this application. Figure 2 As shown, the analysis system 200 for multi-source heterogeneous real-time meteorological data includes: a data acquisition module 201, a data preprocessing module 202, a data standardization module 203, and a data management module 204.

[0107] Data acquisition module 201 is used to collect meteorological data in real time; Data preprocessing module 202 is used to perform missing value processing and / or outlier detection processing on meteorological data to determine clean meteorological data; Data standardization module 203 is used to standardize clean meteorological data to determine standard meteorological data; The data management module 204 is used to store standard meteorological data in three levels and to perform trend forecasting based on data analysis needs and standard meteorological data, so as to issue early warnings based on the forecast results.

[0108] Optionally, the data acquisition module 201 is also used to filter and compress meteorological data according to key data requirements in order to determine key meteorological data.

[0109] Optionally, when performing outlier detection processing on meteorological data, the data preprocessing module 202 is specifically used for: determining a meteorological threshold range based on meteorological industry standards, marking the meteorological data for first outliers based on the meteorological threshold range, and obtaining first meteorological labeled data; marking the first meteorological labeled data for second outliers based on a preset confidence interval, and obtaining second meteorological labeled data; marking the second meteorological labeled data for third outliers based on a preset multivariate outlier detection method, and obtaining third meteorological labeled data; marking the third meteorological labeled data for fourth outliers based on a preset residual test method, and obtaining fourth meteorological labeled data; marking the fourth meteorological labeled data for fifth outliers based on a preset unsupervised outlier detection algorithm, and obtaining fifth meteorological labeled data; correcting the first, second, third, fourth, and fifth outliers according to a preset neighborhood weighted median replacement method, and marking the fifth meteorological labeled data for outlier correction, so as to obtain clean meteorological data.

[0110] Optionally, when performing missing value processing on meteorological data, the data preprocessing module 202 is specifically used to: detect meteorological data and determine the missing proportion of missing data; and determine the corresponding missing value processing method based on the missing proportion.

[0111] Optionally, when determining the corresponding missing value handling method based on the missing ratio, the data preprocessing module 202 is specifically used to: determine the missing value handling method as deleting missing data when the missing ratio is less than a preset ratio threshold; and determine the missing value handling method as filling missing data when the missing ratio is greater than or equal to the preset ratio threshold.

[0112] Optionally, when the missing data ratio is greater than or equal to a preset threshold, the data preprocessing module 202 specifically performs the following steps: determining the cluster number selection method based on the characteristics of the meteorological data; determining the target cluster number based on the cluster number selection method and the meteorological data; clustering the meteorological data according to the target cluster number and the K-Means clustering method to determine the meteorological cluster data; obtaining the cluster to which the missing data in the meteorological cluster data belongs, and selecting non-missing sample points from the clusters according to a preset non-missing sample selection method; determining the similarity between the non-missing sample points and the corresponding missing data according to a preset similarity weight correspondence, and determining the imputation value based on the similarity and the weight of the cluster; replacing the corresponding missing data according to the imputation value to determine the clean meteorological data.

[0113] Optionally, the analysis system 200 for multi-source heterogeneous real-time meteorological data also includes a precision quantization module 205, which is used to determine the filling precision based on the filling value, meteorological data and a preset precision quantization method.

[0114] The device performs similar functions to the methods described above; other functions can be found in the previous descriptions and will not be repeated here.

[0115] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 3 As shown, the electronic device 300 of this embodiment may include a memory 301 and a processor 302.

[0116] The memory 301 stores a computer program, which, when executed by the processor 302, causes the processor 302 to perform the method described in the above embodiments.

[0117] The processor 302 and the memory 301 are connected, for example, via a bus.

[0118] Optionally, the electronic device 300 may also include a transceiver. It should be noted that in practical applications, the transceiver is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of this application.

[0119] Processor 302 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 302 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0120] A bus can include a pathway for transmitting information between the aforementioned components. The bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one thick line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0121] The memory 301 can be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD. ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0122] The memory 301 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 302. The processor 302 is used to execute the application code stored in the memory 301 to implement the content shown in the foregoing method embodiments.

[0123] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0124] The electronic device in this embodiment can be used to execute the method of any of the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0125] This application also provides a non-transitory computer-readable storage medium storing computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform the method as described in the above embodiments.

[0126] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a non-transitory computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0127] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, and on the specific implementation methods and application scope of this application, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for analyzing multi-source heterogeneous real-time meteorological data, characterized in that, include: Real-time meteorological data collection; The meteorological data is processed to handle missing values ​​and / or detect outliers in order to determine the clean meteorological data; The purified meteorological data is standardized to determine standard meteorological data; The standard meteorological data is stored in three levels, and trend prediction is performed based on data analysis needs and the standard meteorological data, so as to issue early warnings based on the prediction results.

2. The method according to claim 1, characterized in that, The real-time meteorological data collected includes: Based on key data requirements, the meteorological data is filtered and compressed to identify critical meteorological data.

3. The method according to claim 1, characterized in that, In the case of outlier detection processing of the meteorological data, the missing value processing and / or outlier detection processing of the meteorological data to determine clean meteorological data includes: Based on meteorological industry standards, a meteorological threshold range is determined, and the meteorological data is marked with a first outlier based on the meteorological threshold range to obtain first meteorological labeled data; Based on a preset confidence interval, the first meteorological labeled data is labeled with second outliers to obtain second meteorological labeled data; Based on a preset multivariate anomaly detection method, the second meteorological labeled data is labeled with a third anomaly value to obtain the third meteorological labeled data; Based on a preset residual test method, the third meteorological labeled data is labeled with a fourth outlier to obtain the fourth meteorological labeled data; Based on a preset unsupervised anomaly detection algorithm, the fourth meteorological labeled data is labeled with a fifth outlier to obtain the fifth meteorological labeled data; According to the preset neighborhood weighted median replacement method, the first outlier, the second outlier, the third outlier, the fourth outlier and the fifth outlier are corrected, and the fifth meteorological data is marked as an anomaly to obtain the clean meteorological data.

4. The method according to claim 1, characterized in that, In the case of missing value processing of the meteorological data, the missing value processing and / or outlier detection processing of the meteorological data to determine clean meteorological data includes: Analyze the meteorological data to determine the proportion of missing data; Based on the missing percentage, determine the corresponding missing value handling method.

5. The method according to claim 4, characterized in that, The step of determining the corresponding missing value handling method based on the missing ratio includes: If the missing proportion is less than a preset proportion threshold, the missing value processing method is determined to be deleting the missing data; If the missing proportion is greater than or equal to the preset proportion threshold, the missing value processing method is determined to be filling the missing data.

6. The method according to claim 5, characterized in that, When the missing proportion is greater than or equal to the preset proportion threshold, determining the missing value processing method as filling the missing data includes: Based on the characteristics of the meteorological data, the method for selecting the number of clusters is determined; The target number of clusters is determined based on the cluster number selection method and the meteorological data. The meteorological data are clustered according to the target cluster number and the K-Means clustering method to determine the meteorological cluster data; Obtain the cluster to which the missing data in the meteorological clustering data belongs, and select non-missing sample points from the cluster according to the preset non-missing sample selection method; Based on the preset similarity weight correspondence, the similarity between the non-missing sample points and the corresponding missing data is determined, so as to determine the filling value based on the similarity and the weight of the cluster. The missing data is replaced with the fill value to determine the clean meteorological data.

7. The method according to claim 6, characterized in that, Also includes: The filling accuracy is determined based on the filling value, the meteorological data, and the preset accuracy quantification method.

8. A system for analyzing multi-source heterogeneous real-time meteorological data, characterized in that, include: The data acquisition module is used to collect meteorological data in real time. The data preprocessing module is used to perform missing value processing and / or outlier detection processing on the meteorological data to determine clean meteorological data; The data standardization module is used to standardize the purified meteorological data to determine standard meteorological data. The data management module is used to store the standard meteorological data in three levels and to perform trend prediction based on data analysis needs and the standard meteorological data, so as to issue early warnings based on the prediction results.

9. An electronic device, characterized in that, include: processor; A memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-7.

Images