A meteorological data quality management and control method

Abstract

The application relates to the technical field of meteorological data quality control, and discloses a meteorological data quality control method, which comprises the following steps: extracting statistical features of pre-acquired meteorological data and identifying a current meteorological event context, generating a feature vector data set containing an event context identifier, calculating a quality score by using a dynamic decision algorithm, separating a high-quality data set and a low-quality data set, calculating adaptive weights that dynamically change with the event context, adaptively weighting and fusing the high-quality data set, generating a preliminary target meteorological data estimate, combining external multi-source data to perform multi-source correction on residual data, integrating low-quality correction data into the preliminary target meteorological data estimate to obtain final target meteorological data, performing consistency verification on the final target meteorological data, and feeding back and adjusting parameters of the dynamic decision algorithm according to a verification result; and the application can improve the efficiency of meteorological data quality control.

Description

Technical Field

[0001] This invention relates to the field of meteorological data quality control technology, and in particular to a method for meteorological data quality management. Background Technology

[0002] In the field of meteorological data processing, traditional data quality control methods often adopt a fixed threshold judgment model, which does not fully consider the contextual differences of different meteorological events, such as rainstorms, typhoons, and sunny days. These methods apply a uniform standard to all scenarios, which leads to high-quality data being easily misjudged as low-quality data when data fluctuations are large under extreme weather conditions, while it is difficult to effectively filter noisy data under stable weather conditions, which seriously affects the accuracy of data screening. At the same time, existing methods rely on a single dimension feature to assess data quality, without combining the statistical characteristics of meteorological data and event attributes to build a comprehensive evaluation system. This makes the quality score unable to truly reflect the actual reliability of the data, thus affecting the effectiveness of subsequent data applications.

[0003] Furthermore, traditional meteorological data quality control lacks a complete closed-loop optimization mechanism, and the data processing flow is unidirectional. On the one hand, the correction of low-quality data often relies on a single data source and does not fully integrate external multi-source data such as satellite remote sensing, radar, and nearby meteorological stations, resulting in limited correction accuracy and difficulty in compensating for the defects of the original data. On the other hand, after data processing, no effective verification and feedback channels are established, making it impossible to dynamically adjust the judgment parameters based on the consistency of the final data. This leads to poor algorithm adaptability, and the efficiency of quality control gradually declines over long-term use, making it difficult to meet the actual needs of real-time processing and high-precision applications of meteorological data. Therefore, accurately reflecting the actual reliability of data and improving the efficiency of quality control have become urgent problems to be solved. Summary of the Invention

[0004] This invention provides a meteorological data quality control method to solve the problems mentioned in the background section.

[0005] To achieve the above objectives, the present invention provides a meteorological data quality control method, comprising: Extract statistical features from the pre-acquired meteorological data and identify the current meteorological event context to generate a feature vector dataset containing event context identifiers; A dynamic judgment algorithm is used to calculate the quality score of each data point in the feature vector dataset, and the judgment threshold is adjusted according to the event type to separate the high-quality dataset and low-quality dataset of the meteorological data from the quality score. Based on the quality score and the event context identifier in the feature vector dataset, an adaptive weight that dynamically changes with the event context is calculated to generate a weight value dataset of the meteorological data. An adaptive weighted fusion is performed on the weighted dataset and the high-quality dataset to generate a preliminary target meteorological data estimate of the meteorological data. By combining external multi-source data, residual data multi-source correction is performed on the low-quality dataset to generate low-quality corrected data of the meteorological data. The low-quality corrected data is then integrated into the preliminary target meteorological data estimate to obtain the final target meteorological data. The final target meteorological data is subjected to consistency verification, and the parameters of the dynamic judgment algorithm are adjusted based on the verification results.

[0006] In a preferred embodiment, the step of extracting statistical features from the pre-acquired meteorological data and identifying the current meteorological event context to generate a feature vector dataset containing event context identifiers includes: Statistical features are extracted from the pre-acquired meteorological data to obtain a statistical feature dataset of the meteorological data; Perform event context recognition on the statistical feature dataset to obtain the event context identifier of the meteorological data; Based on the statistical feature dataset and event context identifiers, feature vectors are constructed to obtain a feature vector dataset containing event context identifiers.

[0007] In a preferred embodiment, the step of using a dynamic judgment algorithm to calculate the quality score of each data point in the feature vector dataset includes: Load the feature weights from the feature vector dataset to obtain the feature weight vector; The quality scores of the feature vector dataset and the feature weight vector are weighted and summed to obtain the quality score of each data point in the feature vector dataset.

[0008] In a preferred embodiment, the step of adjusting the judgment threshold according to the event type to separate the high-quality dataset and the low-quality dataset of the meteorological data from the quality score includes: Based on the event context identifier, a baseline threshold loading operation is performed to obtain the baseline judgment threshold; Based on the event context identifier, the baseline judgment threshold is dynamically adjusted and calculated to obtain the dynamic judgment threshold; Based on the dynamic judgment threshold, a dataset separation operation is performed on the quality score to obtain a high-quality dataset and a low-quality dataset.

[0009] In a preferred embodiment, the step of calculating adaptive weights that dynamically change with the event context based on the quality score and the event context identifiers in the feature vector dataset to generate the weight value dataset of the meteorological data includes: Calculate the base weight value of the quality score; Weight adjustment factors are matched for event context identifiers in the feature vector dataset to obtain weight adjustment factors; Adaptive weight calculation is performed on the base weight values ​​and the weight adjustment factors to obtain a weight value dataset.

[0010] In a preferred embodiment, the step of performing weight adjustment factor matching on the event context identifiers in the feature vector dataset to obtain the weight adjustment factor includes: The adjustment factor is obtained by searching the predefined adjustment factor mapping table for the adjustment factor value corresponding to the event context identifier in the feature vector dataset. The adjustment factor for rainstorm events is set to 1.2, the adjustment factor for sunny events is set to 0.8, and the adjustment factor for typhoon events is set to 1.5.

[0011] In a preferred embodiment, the adaptive weighted fusion of the weighted dataset and the high-quality dataset to generate a preliminary target meteorological data estimate of the meteorological data includes: The weighted data points of the meteorological data are obtained by performing a weighted product calculation on the weighted value dataset and the high-quality dataset. The weighted sum of the weighted data points is obtained by summing the weighted data points. The total weight of the meteorological data is obtained by summing the weights of the dataset. The total weight and the weighted sum are normalized to obtain a preliminary estimate of the target meteorological data.

[0012] In a preferred embodiment, the step of combining external multi-source data to perform residual data multi-source correction on the low-quality dataset to generate low-quality corrected meteorological data includes: Based on a predefined external data source, load external data to obtain an external data set; Perform a spatiotemporal alignment operation on the low-quality dataset and the external dataset to obtain the aligned low-quality dataset and the external dataset. Residual correction is performed on the aligned low-quality dataset and the external dataset to obtain low-quality corrected data.

[0013] In a preferred embodiment, integrating the low-quality corrected data into the preliminary target meteorological data estimate to obtain the final target meteorological data includes: Based on the low-quality correction data and the preliminary target meteorological data estimate, an integrated weight calculation operation is performed to obtain the preliminary estimate weight and the correction data weight. Based on the preliminary valuation weights, correction data weights, preliminary target meteorological data valuations, and low-quality correction data, a weighted fusion operation is performed to obtain the final target meteorological data.

[0014] In a preferred embodiment, the step of performing consistency verification on the final target meteorological data, and adjusting the parameters of the dynamic judgment algorithm based on the verification result, includes: The reliability and consistency level of the data are assessed by calculating the differences between data points or the deviation from the expected pattern, in order to perform a consistency verification operation on the final target meteorological data and obtain the verification results. The mathematical expression used in the consistency verification operation is as follows: ; In the formula, To verify the results, The final target is the number of data points in the meteorological data. For indexing data points, Let i be the value of the i-th data point. The average value of the final target meteorological data; The verification results are then adjusted to obtain the adjusted dynamic judgment algorithm parameters.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention effectively overcomes the limitations of traditional fixed threshold judgment in terms of dynamic quality judgment and threshold adjustment mechanism based on event context, significantly improving the accuracy and scenario adaptability of meteorological data quality screening. On the one hand, by extracting statistical features of meteorological data and identifying the current meteorological event context, a feature vector dataset containing event context identifiers is constructed, so that quality assessment no longer relies on a single dimension, but combines data statistical characteristics and event attributes to form a comprehensive judgment basis, ensuring that the quality score can truly reflect the actual reliability of data under different scenarios. On the other hand, the judgment threshold is dynamically adjusted based on event type. For example, a higher dynamic threshold is set for extreme weather such as rainstorms and typhoons to filter abnormal fluctuation data, while the threshold is appropriately reduced for stable weather such as sunny days to retain more effective data. This avoids the misjudgment problem caused by uniform standards, and makes the separation of high-quality datasets and low-quality datasets more in line with the needs of actual meteorological scenarios, laying a high-quality foundation for subsequent data processing.

[0016] 2. This invention achieves comprehensive improvement and continuous optimization of meteorological data quality through a closed-loop quality control process that integrates multi-source data and feedback optimization. It significantly enhances data integrity, accuracy, and the long-term adaptability of the algorithm. In the data correction stage, it combines external multi-source data such as satellite remote sensing, radar, and nearby meteorological stations to first perform spatiotemporal alignment on low-quality datasets. Then, it generates high-quality low-quality corrected data through residual correction, which is then weighted and fused with the preliminary target meteorological data estimate. This not only compensates for the insufficient accuracy of correction from a single data source but also fully utilizes the effective information from low-quality data, improving the integrity and accuracy of the final target meteorological data. In the algorithm optimization stage, it quantifies the reliability level of the final data through consistency verification and adjusts the dynamic judgment algorithm parameters based on the verification results, forming a closed-loop mechanism of "data processing - verification - parameter optimization - reprocessing." This enables the algorithm to continuously adapt to different meteorological data characteristics and scene changes, maintaining efficient quality control capabilities over the long term and meeting the needs of real-time meteorological data processing and high-precision applications. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a meteorological data quality control method according to an embodiment of the present invention. The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] This application provides a method for meteorological data quality control. The executing entity of this meteorological data quality control method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the meteorological data quality control method can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0020] Reference Figure 1 The diagram shown is a flowchart illustrating a meteorological data quality control method according to an embodiment of the present invention. In this embodiment, the meteorological data quality control method includes: S1, extract the statistical features of the pre-acquired meteorological data and identify the current meteorological event context to generate a feature vector dataset containing event context identifiers; In this embodiment of the invention, the step of extracting statistical features from the pre-acquired meteorological data and identifying the current meteorological event context to generate a feature vector dataset containing event context identifiers includes: Statistical features are extracted from the pre-acquired meteorological data to obtain a statistical feature dataset of the meteorological data; Perform event context recognition on the statistical feature dataset to obtain the event context identifier of the meteorological data; Based on the statistical feature dataset and event context identifiers, feature vectors are constructed to obtain a feature vector dataset containing event context identifiers.

[0021] It should be noted that the statistical feature extraction operation is a process of performing multi-dimensional statistical calculations on the pre-acquired meteorological data, including calculating the average, standard deviation, extreme values, and skewness of meteorological elements such as temperature, humidity, air pressure, and wind speed, in order to quantify the central tendency, dispersion, and distribution pattern of the meteorological data. The pre-acquired meteorological data includes temperature, humidity, air pressure, and wind speed.

[0022] Furthermore, the statistical feature extraction operation processes time-series meteorological data through a sliding window mechanism, recursively updating statistics within a fixed time interval to ensure real-time feature accuracy and reduce computational redundancy. The fixed time interval is set to 1 hour.

[0023] It should be noted that the statistical feature dataset is a numerical set containing multiple statistics, used to characterize the overall behavior pattern and fluctuation characteristics of meteorological data, providing a quantitative basis for event identification.

[0024] It should be noted that the event context recognition operation is based on the statistical feature dataset to identify the current meteorological event type. First, the statistical feature dataset is multiplied by the weight vector of each event category and a bias term is added to obtain the score of each event category. Then, the scores of all event categories are compared and the event category with the highest score is selected as the event context identifier. The weight vector of each event category is obtained by means of the average distribution, which is one-tenth of the total number of event categories.

[0025] Furthermore, the bias term, obtained from training the event classification model using historical meteorological data, is used to train the model on historical meteorological data and their corresponding event category labels. The weight vector and bias term are optimized using gradient descent to minimize the classification error. The initial value of the bias term is usually set to zero, and it is adjusted during training based on backpropagation of the error. The final value depends on the distribution of the training data and the complexity of the model. For example, the bias term may converge to 0.2 in a rainstorm event and to -0.1 in a sunny event. The bias term is used to adjust the baseline of the event category score, eliminate the inherent bias of the feature data, and improve the accuracy and robustness of event context recognition. For example, in a statistical feature dataset, the bias term compensates for the prior probability differences of different event categories to ensure that the event context labeling is more consistent with the actual meteorological pattern.

[0026] It should be noted that the event context label is a classification label that indicates the type of the current meteorological event. It is used to identify the event category to which the data belongs in the feature vector, such as rainstorm, sunny day or typhoon, to enhance the context awareness of data processing.

[0027] It should be noted that feature vector construction involves concatenating and normalizing the values ​​of each statistic in the statistical feature dataset with the event context identifier to form a unified feature vector.

[0028] Furthermore, the feature vector generation operation uses one-hot encoding to convert the event context identifier into a binary vector and combines it with statistical feature values ​​by dimension. The feature vector dataset containing the event context identifier is the input data for subsequent quality assessment. It combines statistical characteristics and event context, improving the adaptability and accuracy of the quality judgment algorithm.

[0029] S2, using a dynamic judgment algorithm, calculate the quality score of each data point in the feature vector dataset, adjust the judgment threshold according to the event type, and separate the high-quality dataset and low-quality dataset of the meteorological data from the quality score; In this embodiment of the invention, the step of using a dynamic judgment algorithm to calculate the quality score of each data point in the feature vector dataset includes: Load the feature weights from the feature vector dataset to obtain the feature weight vector; The quality scores of the feature vector dataset and the feature weight vector are weighted and summed to obtain the quality score of each data point in the feature vector dataset.

[0030] It should be noted that the feature weight loading operation reads the weight value corresponding to each feature from the pre-stored weight configuration file to form a feature weight vector. The weight values ​​are set based on historical meteorological data quality assessment experience and are used to quantify the degree of influence of different features on data quality. For the four features of temperature, humidity, air pressure and wind speed, the initial default weights adopt an equal distribution strategy, each set to 0.25, indicating that each feature contributes equally to the quality score. In practical applications, the weight values ​​can be adjusted by the operator based on domain knowledge or data characteristics. For example, if historical analysis shows that the temperature feature has a more significant impact on data quality, the weight can be set to 0.3, and the weights of other features can be adjusted to around 0.23 accordingly to ensure that the feature weight vector is aligned with the feature vector dimension, providing a stable and reliable input basis for weighted summation calculation.

[0031] Furthermore, the feature weight loading operation loads weight values ​​in the order of feature dimensions by parsing the key-value pair structure in the weight configuration file, ensuring that the weight vector is aligned with the feature vector dimensions.

[0032] It should be noted that the feature weight vector is a set of values ​​containing multiple weight values. Each weight value corresponds to a feature dimension in the feature vector. The higher the weight value, the greater the contribution of that feature to the quality score, providing a quantitative basis for weighted calculation.

[0033] It should be noted that the weighted summation calculation involves multiplying each feature value in the feature vector by its corresponding weight, and then summing all the products to obtain the quality score for each data point.

[0034] It should be noted that the quality score represents the quality level of each data point. The higher the score, the better the data quality, providing an input basis for subsequent threshold separation.

[0035] In this embodiment of the invention, the step of adjusting the judgment threshold according to the event type to separate the high-quality dataset and the low-quality dataset of the meteorological data from the quality score includes: Based on the event context identifier, a baseline threshold loading operation is performed to obtain the baseline judgment threshold; Based on the event context identifier, the baseline judgment threshold is dynamically adjusted and calculated to obtain the dynamic judgment threshold; Based on the dynamic judgment threshold, a dataset separation operation is performed on the quality score to obtain a high-quality dataset and a low-quality dataset.

[0036] It should be noted that the baseline threshold loading operation reads the baseline threshold corresponding to each event type from the pre-stored threshold configuration file to form the baseline judgment threshold. The baseline threshold is set based on historical meteorological data quality statistics and is used to represent the default quality requirement level for different event types.

[0037] Furthermore, the baseline threshold for each event type in the stored threshold configuration file is set based on historical meteorological data quality statistics. The specific setting method is as follows: First, historical meteorological data is collected and its event type and quality label are marked. Then, for each event type, its quality score percentile is calculated, and the 75th percentile is used as the baseline threshold to ensure that the threshold can cover most high-quality data. It can be adjusted in specific situations. For example, for rainstorm events, the baseline threshold is set to 0.7 because rainstorm data usually fluctuates greatly and requires a higher threshold to filter out outliers. The baseline threshold for sunny events is set to 0.5 because sunny data is highly stable, and the threshold can be appropriately reduced to retain more data. The baseline threshold for typhoon events is set to 0.8 to cope with data noise under extreme weather conditions.

[0038] Furthermore, the baseline threshold loading operation parses the mapping relationship between event types and thresholds in the threshold configuration file, matches the corresponding baseline threshold according to the event context identifier, and ensures that the threshold is aligned with the current event type. The baseline judgment threshold is a reference value used for initial quality separation. Its value reflects the basic requirements of different event types for data quality. For example, a higher threshold may be set for rainstorm events to cope with data fluctuations, while a lower threshold may be set for sunny events to adapt to stable data patterns.

[0039] It should be noted that the dynamic adjustment calculation is based on the event context identifier. The baseline judgment threshold is scaled proportionally by the adjustment coefficient to obtain the dynamic judgment threshold. The adjustment coefficient is dynamically determined based on the sensitivity parameter of the event type. First, the corresponding adjustment coefficient is found in the adjustment coefficient table according to the event context identifier. Then, the baseline judgment threshold is multiplied by the adjustment coefficient to obtain the dynamic judgment threshold.

[0040] Furthermore, the adjustment coefficients in the adjustment coefficient table are dynamically determined based on the sensitivity parameter of the event type. The sensitivity parameter is obtained through the correlation analysis of event type and data quality fluctuation in historical data. The adjustment coefficient table is stored in key-value pair form. For example, for rainstorm events, the adjustment coefficient is 1.2, which is used to amplify the threshold to enhance adaptability to data fluctuations. For sunny events, the adjustment coefficient is 0.8, which is used to narrow the threshold to improve data utilization. For typhoon events, the adjustment coefficient is 1.5, which further tightens the threshold to deal with high-risk scenarios, ensuring that the dynamic judgment threshold can be flexibly adjusted according to the event context, and improving the accuracy of separating high-quality datasets and low-quality datasets.

[0041] It should be noted that the dynamic judgment threshold is the quality score separation boundary value after event type adjustment. It is used to adapt to the fluctuation characteristics of data quality under different event types and improve the separation accuracy. Its value changes dynamically with the event type to ensure the contextual adaptability of quality assessment.

[0042] It should be noted that the threshold comparison and dataset separation operation compares the quality score of each data point in the quality score dataset with the dynamic judgment threshold, and classifies the data points into high-quality datasets or low-quality datasets based on the comparison results.

[0043] Furthermore, the specific process of threshold comparison and dataset separation is as follows: For each data point, if the quality score is greater than or equal to the dynamic judgment threshold, the data point is added to the high-quality dataset. If the quality score is less than the dynamic judgment threshold, the data point is added to the low-quality dataset.

[0044] It should be noted that a high-quality dataset is a collection of all data points with a quality score not lower than the dynamic judgment threshold, representing data that meets the quality requirements of the current event type and can be directly used for subsequent data processing. A low-quality dataset is a collection of all data points with a quality score lower than the dynamic judgment threshold, representing data that needs further correction or processing and providing an input basis for residual data correction.

[0045] S3, based on the quality score and the event context identifier in the feature vector dataset, calculate the adaptive weight that dynamically changes with the event context, and generate the weight value dataset of the meteorological data; In this embodiment of the invention, the step of calculating adaptive weights that dynamically change with the event context based on the quality score and the event context identifiers in the feature vector dataset, and generating the weight value dataset of the meteorological data, includes: Calculate the base weight value of the quality score; Weight adjustment factors are matched for event context identifiers in the feature vector dataset to obtain weight adjustment factors; Adaptive weight calculation is performed on the base weight values ​​and the weight adjustment factors to obtain a weight value dataset.

[0046] It should be noted that the calculation of the base weight value involves normalizing the quality score, converting it into an initial weight value within the range of 0 to 1. This normalization is achieved using a min-max scaling method to ensure that the weight values ​​are within a standard range and positively correlated with the quality score. Furthermore, the specific process for calculating the base weight value is as follows: First, determine the minimum and maximum scores in the quality score dataset. Then, apply a linear transformation to each quality score, mapping it to the 0-1 interval. It's important to note that the base weight value is a normalized representation of the quality score, reflecting the initial quality level of the data point. A higher score results in a larger base weight value, indicating a higher contribution of the data point to the final result, providing the basic input for subsequent adaptive adjustments.

[0047] It should be noted that adaptive weight calculation involves multiplying the base weight value by the weight adjustment factor to obtain the final adaptive weight for each data point. The multiplication operation ensures that the weight changes dynamically with the event context.

[0048] Furthermore, the adaptive weight calculation process is as follows: For each data point, its basic weight value is multiplied by a weight adjustment factor to obtain the adaptive weight of that data point. The adaptive weights of all data points constitute the weight value dataset. It should be noted that the weight value dataset is a set of values ​​containing the adaptive weight of each data point, which is used for subsequent adaptive weighted fusion operations. Its value is dynamically adjusted according to the event context and quality score to improve the accuracy and context adaptability of data fusion and ensure that high-quality data occupies a higher weight in the fusion.

[0049] In this embodiment of the invention, the step of performing weight adjustment factor matching on the event context identifiers in the feature vector dataset to obtain the weight adjustment factor includes: The adjustment factor is obtained by searching the predefined adjustment factor mapping table for the adjustment factor value corresponding to the event context identifier in the feature vector dataset. The adjustment factor for rainstorm events is set to 1.2, the adjustment factor for sunny events is set to 0.8, and the adjustment factor for typhoon events is set to 1.5.

[0050] It should be noted that the predefined adjustment factor mapping table stores the correspondence between event context identifiers and adjustment factors in key-value pairs. The specific mapping relationship is set based on the importance and sensitivity of the event type. The adjustment factor values ​​are determined by analyzing the impact of different event types on data quality in historical meteorological data: the adjustment factor for rainstorm events is set to 1.2 because rainstorm data usually fluctuates greatly, requiring a higher adjustment factor to emphasize the importance of the data in the fusion and avoid underestimating high-quality data; the adjustment factor for sunny events is set to 0.8 because sunny data is highly stable, and the adjustment factor is appropriately reduced to improve data utilization and prevent overweighting; the adjustment factor for typhoon events is set to 1.5 to cope with the high-risk characteristics of data under extreme weather conditions, further strengthen the data weight, and ensure the reliability of the fusion results. These adjustment factor values ​​are optimized through historical data regression analysis, such as using the least squares method to fit the correlation between event type and data quality, and can be dynamically updated according to real-time data characteristics in practical applications. The adjustment factor mapping table uses a hash table structure to achieve fast lookup, ensuring the efficiency of matching event context identifiers with adjustment factors and improving the real-time performance and accuracy of adaptive weight calculation.

[0051] It should be noted that the adjustment factor calculation is based on the event context identifier, and the corresponding adjustment factor value is looked up from the predefined adjustment factor mapping table. The adjustment factor value is set based on the importance and sensitivity of the event type. For example, a rainstorm event may be set with a higher adjustment factor to emphasize the importance of the data, while a sunny day event may be set with a lower adjustment factor.

[0052] Furthermore, the adjustment factor matching quickly matches the adjustment factor value corresponding to the current event type by parsing the key-value pair relationship between the event context identifier and the adjustment factor, ensuring that the factor value is consistent with the event context. The weight adjustment factor is a coefficient used to dynamically adjust the basic weight value according to the event type. Different event types correspond to different adjustment factors to reflect the influence of the event context on the data weight. Its value reflects the relative importance of the event type in data fusion.

[0053] S4, perform adaptive weighted fusion of the weight value dataset and the high-quality dataset to generate a preliminary target meteorological data estimate of the meteorological data; In this embodiment of the invention, the step of adaptively weighting and fusing the weighted dataset and the high-quality dataset to generate a preliminary target meteorological data estimate of the meteorological data includes: The weighted data points of the meteorological data are obtained by performing a weighted product calculation on the weighted value dataset and the high-quality dataset. The weighted sum of the weighted data points is obtained by summing the weighted data points. The total weight of the meteorological data is obtained by summing the weights of the dataset. The total weight and the weighted sum are normalized to obtain a preliminary estimate of the target meteorological data.

[0054] It should be noted that the weighted product calculation multiplies each data point in the high-quality dataset with the corresponding adaptive weight in the weight value dataset to obtain a weighted data point. The multiplication operation ensures that each data point is scaled according to its weight, and the higher the weight value, the greater the contribution of the data point in the fusion.

[0055] Furthermore, the weighted product calculation performs element-wise multiplication by traversing the corresponding elements of the high-quality dataset and the weight value dataset to generate a set of weighted data points. The weighted data points are the product of the high-quality data points and the adaptive weights, used to represent the adjusted data values. Their numerical values ​​reflect the quality weights of the data points and the influence of the event context, providing the input basis for subsequent summation calculations.

[0056] It should be noted that the summation of the weighted data points involves adding all weighted data points together to obtain a weighted sum. This summation integrates the contributions of all weighted data points to form a cumulative weighted data value. By accumulating the values ​​of all weighted data points, a single value is obtained, representing the weighted data sum. The weighted sum is the cumulative result of all weighted data points and is used to represent the total amount of data after weighted fusion. Its value integrates the weighted contributions of all high-quality data points, providing the numerator input for normalization calculation.

[0057] It should be noted that the calculation of the total weight is to add up all the adaptive weights in the weight value dataset to obtain the total weight. The addition operation calculates the sum of all weight values ​​and is used to normalize the denominator. The total weight is calculated by traversing the weight value dataset and accumulating all adaptive weight values ​​to obtain the total weight value.

[0058] It should be noted that the total weight is the sum of all weight values, used to represent the overall magnitude of all weights. Its value ensures the correctness of the denominator in the normalization calculation and prevents weight bias from affecting the fusion result.

[0059] It should be noted that the normalization calculation is to divide the weighted sum by the total weight to obtain the preliminary target meteorological data. The division operation ensures that the fusion result is not affected by the size of the weighted sum, thus achieving standardized output.

[0060] It should be noted that the preliminary target meteorological data is a standardized result after adaptive weighted fusion, used to represent the meteorological data estimate based on high-quality data and dynamic weights. Its value integrates data quality, event context and weight adjustment, providing a reliable preliminary data foundation for subsequent residual data correction.

[0061] S5, combine external multi-source data to perform residual data multi-source correction on the low-quality dataset to generate low-quality corrected data of the meteorological data, and integrate the low-quality corrected data into the preliminary target meteorological data estimate to obtain the final target meteorological data; In this embodiment of the invention, the step of combining external multi-source data to perform residual data multi-source correction on the low-quality dataset to generate low-quality corrected meteorological data includes: Based on a predefined external data source, load external data to obtain an external data set; Perform a spatiotemporal alignment operation on the low-quality dataset and the external dataset to obtain the aligned low-quality dataset and the external dataset. Residual correction is performed on the aligned low-quality dataset and the external dataset to obtain low-quality corrected data.

[0062] It should be noted that the external data loading operation obtains multi-source meteorological data from predefined external data sources, including satellite remote sensing data, radar data, and data from nearby meteorological stations, to form an external data set. This set is used to provide additional data references to correct low-quality data. The external data loading operation accesses external databases or real-time data streams through data interfaces to ensure the timeliness and diversity of the data and avoid bias from a single data source.

[0063] It should be noted that the external dataset is a collection of multiple external data points, each with a timestamp and spatial coordinates, used for comparison and correction with the low-quality dataset. Its diverse data sources improve the reliability and coverage of the correction, providing a multi-source input basis for residual correction.

[0064] It should be noted that spatiotemporal alignment matches low-quality datasets with external datasets in both time and space, ensuring that data points have the same time reference and spatial location for direct comparison and correction. The specific method for matching low-quality datasets with external datasets in time and space is as follows: Time alignment uses linear interpolation to uniformly align the data to a standard time point. First, a time reference sequence is determined. Then, for the timestamp of each data point, linear interpolation is performed using the data values ​​from the nearest preceding and following time points. The formula is as follows: ,in, The value after alignment. and Values ​​at adjacent time points. For the target time, and Adjacent timestamps; Spatial alignment: An inverse distance weighted interpolation method is used to uniformly align the data to a standard spatial grid. For each target grid point, its spatial distance to surrounding data points is calculated, and a weighted average is performed using the inverse of the distance as the weight. The formula is as follows: ,in, The value after alignment. For the value of the i-th data point, For distance, As an exponential parameter, let's set it to 2; This method ensures that low-quality datasets and external datasets are perfectly matched in the spatiotemporal dimensions, providing accurate input for residual correction.

[0065] Furthermore, the spatiotemporal alignment operation adjusts the data to a unified time point and spatial grid through temporal and spatial interpolation methods. Linear interpolation is used to eliminate the bias caused by spatiotemporal inconsistency. The aligned low-quality dataset and the external dataset are spatiotemporally matched datasets, in which each low-quality data point has a corresponding external data point, ensuring data comparability and providing accurate aligned input for residual correction calculation, thereby reducing correction errors.

[0066] It should be noted that the residual correction calculation is based on the aligned low-quality dataset and external datasets. The correction algorithm is used to calculate the correction value of the low-quality data to generate low-quality corrected data.

[0067] Furthermore, the residual correction calculation process is as follows: For each aligned data point, the difference between the low-quality data value and the external data value is calculated. Then, this difference is multiplied by the correction coefficient, and the product is added to the low-quality data value to obtain the corrected data value. The correction coefficient ranges from 0 to 1, with a default value of 0.6, representing the contribution of the external data to the correction. In practical applications, it can be dynamically adjusted according to the data quality. For example, when the confidence level of the external data is high, the correction coefficient can be set to 0.8; when the residual error of the low-quality data is small, the correction coefficient can be set to 0.4. The correction coefficient is used to balance the weight of low-quality data and external data in residual correction. The larger the coefficient, the stronger the influence of the external data; the smaller the coefficient, the more original low-quality data is retained. It adjusts the residual through linear combination to avoid over-correction or under-correction.

[0068] It should be noted that low-quality corrected data is a dataset that has undergone multi-source correction, representing the corrected meteorological data values. These values ​​combine information from the original low-quality data and external data. By adjusting the residuals through correction coefficients, the accuracy and reliability of the data are improved, providing reliable correction input for subsequent integration into the preliminary target meteorological data estimation.

[0069] In this embodiment of the invention, integrating the low-quality correction data into the preliminary target meteorological data estimate to obtain the final target meteorological data includes: Based on the low-quality correction data and the preliminary target meteorological data estimate, an integrated weight calculation operation is performed to obtain the preliminary estimate weight and the correction data weight. Based on the preliminary valuation weights, correction data weights, preliminary target meteorological data valuations, and low-quality correction data, a weighted fusion operation is performed to obtain the final target meteorological data.

[0070] It should be noted that the integrated weight calculation operation is based on the number of data points in the low-quality corrected data and the number of source data points in the preliminary target meteorological data estimate. The weight value is calculated through a proportional allocation method, which ensures that the weight is positively correlated with the number of data points; the more data points, the higher the corresponding weight.

[0071] Furthermore, the specific process for integrating weight calculation is as follows: First, count the number of data points in the low-quality correction data and the number of source data points in the preliminary target meteorological data estimate. Then, add these two data to calculate the total number of data points. Finally, obtain the preliminary estimate weight by dividing the number of source data points in the preliminary target meteorological data estimate by the total number of data points. Obtain the correction data weight by dividing the number of data points in the low-quality correction data by the total number of data points.

[0072] It should be noted that the preliminary valuation weight is a coefficient that represents the relative importance of the preliminary target meteorological data valuation in the final fusion. Its value is based on the number of high-quality data points; the more high-quality data points there are, the higher the weight, reflecting the reliability of the preliminary valuation.

[0073] It should be noted that the correction data weight is a coefficient that represents the relative importance of low-quality correction data in the final fusion. Its value is based on the number of correction data points; the more points there are, the higher the weight, reflecting the supplementary role of the correction data.

[0074] It should be noted that the weighted fusion operation is to linearly combine the preliminary target meteorological data estimate with the low-quality correction data through weights to obtain the final target meteorological data. The linear combination ensures that the result integrates the contributions of the preliminary estimate and the correction data.

[0075] Furthermore, the weighted fusion operation first calculates the average value of the low-quality correction data, then multiplies the preliminary estimate weight by the preliminary target meteorological data estimate to obtain the weighted preliminary value, and simultaneously multiplies the correction data weight by the average value of the low-quality correction data to obtain the weighted correction value. Finally, the weighted preliminary value and the weighted correction value are added together to obtain the final target meteorological data.

[0076] It should be noted that the final target meteorological data is the result of integrating and weighting the meteorological data. It is used to represent the final estimate that combines the preliminary estimate of high-quality data and the supplementary nature of low-quality correction data. Its value balances the reliability of the preliminary estimate and the supplementary nature of the correction data, improves the overall accuracy and completeness of the meteorological data, and provides an input basis for subsequent consistency verification.

[0077] S6, perform consistency verification on the final target meteorological data, and adjust the parameters of the dynamic judgment algorithm based on the verification result of the consistency verification.

[0078] In this embodiment of the invention, the step of performing consistency verification on the final target meteorological data, and adjusting the parameters of the dynamic judgment algorithm based on the verification result, includes: The reliability and consistency level of the data are assessed by calculating the differences between data points or the deviation from the expected pattern, in order to perform a consistency verification operation on the final target meteorological data and obtain the verification results. The mathematical expression used in the consistency verification operation is as follows: ; In the formula, To verify the results, The final target is the number of data points in the meteorological data. For indexing data points, Let i be the value of the i-th data point. The average value of the final target meteorological data; The verification results are then adjusted to obtain the adjusted dynamic judgment algorithm parameters.

[0079] It should be noted that the consistency verification operation is to perform an internal consistency check on the final target meteorological data. By calculating the differences between data points or the deviation from the expected model, the reliability and consistency level of the data are assessed.

[0080] Furthermore, the consistency verification process is as follows: First, calculate the average value of the final target meteorological data. Then, calculate the absolute deviation of each data point from the average value. Finally, sum all the absolute deviations and divide by the number of data points to obtain the average absolute deviation as a consistency index. The verification result is a consistency index, which is used to represent the degree of internal consistency of the final target meteorological data. The smaller the index value, the higher the data consistency. The larger the index value, the more serious the data inconsistency, providing a basis for parameter adjustment.

[0081] It should be noted that the parameter adjustment operation is based on the verification results. It modifies the parameters of the dynamic judgment algorithm by adjusting the rules to optimize the subsequent data quality judgment process. The parameter adjustment operation uses a parameter adjustment algorithm, the mathematical expression of which is as follows: ; In the formula, These are the adjusted parameters for the dynamic determination algorithm. These are the parameters for the current dynamic determination algorithm. C is the learning rate, and C is the validation result. Furthermore, the parameter adjustment algorithm process is as follows: First, obtain the current dynamic judgment algorithm parameters and learning rate. Then, multiply the learning rate by the verification result to obtain the adjustment amount. Finally, subtract the adjustment amount from the current parameters to obtain the adjusted parameters. The learning rate is set to 0.05 by default.

[0082] It should be noted that the adjusted dynamic judgment algorithm parameters are parameter values ​​that have been optimized through feedback. They are used to improve the performance of the dynamic judgment algorithm in subsequent data quality control, and to improve the accuracy and adaptability of the quality score. The values ​​are dynamically adjusted based on the consistency verification results to ensure continuous optimization of the algorithm.

[0083] In the several embodiments provided by this invention, it should be understood that the disclosed method can be implemented in other ways.

[0084] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0085] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, and technology that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for meteorological data quality control, characterized in that, The method includes: Extract statistical features from the pre-acquired meteorological data and identify the current meteorological event context to generate a feature vector dataset containing event context identifiers; A dynamic judgment algorithm is used to calculate the quality score of each data point in the feature vector dataset, and the judgment threshold is adjusted according to the event type to separate the high-quality dataset and low-quality dataset of the meteorological data from the quality score. Based on the quality score and the event context identifier in the feature vector dataset, an adaptive weight that dynamically changes with the event context is calculated to generate a weight value dataset of the meteorological data. An adaptive weighted fusion is performed on the weighted dataset and the high-quality dataset to generate a preliminary target meteorological data estimate of the meteorological data. By combining external multi-source data, residual data multi-source correction is performed on the low-quality dataset to generate low-quality corrected data of the meteorological data. The low-quality corrected data is then integrated into the preliminary target meteorological data estimate to obtain the final target meteorological data. The final target meteorological data is subjected to consistency verification, and the parameters of the dynamic judgment algorithm are adjusted based on the verification results. The process of using a dynamic evaluation algorithm to calculate the quality score for each data point in the feature vector dataset includes: Load the feature weights from the feature vector dataset to obtain the feature weight vector; The quality scores of the feature vector dataset and the feature weight vector are weighted and summed to obtain the quality score of each data point in the feature vector dataset. The step of adjusting the judgment threshold according to the event type to separate the high-quality dataset and low-quality dataset of the meteorological data from the quality score includes: Based on the event context identifier, a baseline threshold loading operation is performed to obtain the baseline judgment threshold; Based on the event context identifier, the baseline judgment threshold is dynamically adjusted and calculated to obtain the dynamic judgment threshold; Based on the dynamic judgment threshold, a dataset separation operation is performed on the quality score to obtain a high-quality dataset and a low-quality dataset. The step of calculating adaptive weights that dynamically change with the event context based on the quality score and the event context identifiers in the feature vector dataset, and generating a weighted dataset of the meteorological data, includes: Calculate the base weight value of the quality score; The event context identifiers in the feature vector dataset are matched with weight adjustment factors to obtain weight adjustment factors; Adaptive weight calculation is performed on the base weight values ​​and the weight adjustment factors to obtain a weight value dataset; The step of performing weight adjustment factor matching on the event context identifiers in the feature vector dataset to obtain the weight adjustment factor includes: The adjustment factor is obtained by searching the predefined adjustment factor mapping table for the adjustment factor value corresponding to the event context identifier in the feature vector dataset. The adjustment factor for rainstorm events is set to 1.2, the adjustment factor for sunny events is set to 0.8, and the adjustment factor for typhoon events is set to 1.

5.

2. The meteorological data quality control method as described in claim 1, characterized in that, The process of extracting statistical features from pre-acquired meteorological data and identifying the current meteorological event context to generate a feature vector dataset containing event context identifiers includes: Statistical features are extracted from the pre-acquired meteorological data to obtain a statistical feature dataset of the meteorological data; Perform event context recognition on the statistical feature dataset to obtain the event context identifier of the meteorological data; Based on the statistical feature dataset and event context identifiers, feature vectors are constructed to obtain a feature vector dataset containing event context identifiers.

3. The meteorological data quality control method as described in claim 1, characterized in that, The adaptive weighted fusion of the weighted dataset and the high-quality dataset to generate a preliminary target meteorological data estimate of the meteorological data includes: The weighted data points of the meteorological data are obtained by performing a weighted product calculation on the weighted value dataset and the high-quality dataset. The weighted sum of the weighted data points is obtained by summing the weighted data points. The total weight of the meteorological data is obtained by summing the weights of the dataset. The total weight and the weighted sum are normalized to obtain a preliminary estimate of the target meteorological data.

4. The meteorological data quality control method as described in claim 1, characterized in that, The process of combining external multi-source data to perform residual data multi-source correction on the low-quality dataset to generate low-quality corrected meteorological data includes: Based on a predefined external data source, load external data to obtain an external data set; Perform a spatiotemporal alignment operation on the low-quality dataset and the external dataset to obtain the aligned low-quality dataset and the external dataset. Residual correction is performed on the aligned low-quality dataset and the external dataset to obtain low-quality corrected data.

5. The meteorological data quality control method as described in claim 1, characterized in that, The process of integrating the low-quality corrected data into the preliminary target meteorological data estimate to obtain the final target meteorological data includes: Based on the low-quality correction data and the preliminary target meteorological data estimate, an integrated weight calculation operation is performed to obtain the preliminary estimate weight and the correction data weight. Based on the preliminary valuation weights, correction data weights, preliminary target meteorological data valuations, and low-quality correction data, a weighted fusion operation is performed to obtain the final target meteorological data.

6. The meteorological data quality control method as described in claim 1, characterized in that, The process of performing consistency verification on the final target meteorological data, and adjusting the parameters of the dynamic judgment algorithm based on the verification results, includes: The reliability and consistency level of the data are assessed by calculating the differences between data points or the deviation from the expected pattern, in order to perform a consistency verification operation on the final target meteorological data and obtain the verification results. The mathematical expression used in the consistency verification operation is as follows: ; In the formula, To verify the results, The final target is the number of data points in the meteorological data. For indexing data points, Let i be the value of the i-th data point. The average value of the final target meteorological data; The verification results are then adjusted to obtain the adjusted dynamic judgment algorithm parameters.

Images