An artificial intelligence weather forecast model evaluation method, system, device and medium

Abstract

The application discloses an artificial intelligence weather forecast model evaluation method, system, device and medium, and relates to the technical fields of meteorological data processing and artificial intelligence prediction; the method comprises the following steps: acquiring grid meteorological forecast data of a target prediction model in a specified period and corresponding actual observed meteorological data; based on a preset meteorological event threshold, a two-class contingency table is constructed by using the grid meteorological forecast data and the actual observed meteorological data; based on the two-class contingency table, a symmetric extreme dependence index and a frequency bias are calculated; according to the frequency bias, a Gaussian penalty function is constructed in a logarithmic deviation space; the symmetric extreme dependence index is weighted and corrected by using the Gaussian penalty function to obtain a smoothed penalty extreme dependence index; and the performance of the target prediction model is evaluated based on the smoothed penalty extreme dependence index to obtain an evaluation result. The application can objectively evaluate the real physical performance and distribution integrity of an artificial intelligence weather forecast model in an extreme event.

Description

Technical Field

[0001] This invention relates to the field of meteorological data processing and artificial intelligence forecasting technology, and in particular to an artificial intelligence weather forecasting model evaluation method, system, equipment and medium. Background Technology

[0002] In recent years, data-driven weather prediction (DDWP) models such as AIFS and Pangu-Weather have developed rapidly, surpassing traditional dynamic models based on physical fluid dynamics equations (such as the European Centre for Medium-Range Weather Forecasts' IFS) on many conventional test metrics. However, current mainstream AI weather prediction models generally use loss functions based on minimizing grid point errors, such as mean squared error, for optimization. This leads to a strong "smoothing effect" and systematic spatial variance suppression when predicting extreme weather events. This spatial variance suppression further manifests as variance collapse in the forecast field, meaning that the spatial variance of the forecast field is severely compressed, and extreme peak values ​​cannot be reflected.

[0003] Existing extreme event validation metrics, such as the Symmetric Extremal Dependence Index (SEDI), primarily focus on measuring the model's taxonomic resolution, i.e., its ability to distinguish whether an event has occurred. However, traditional metrics have a serious diagnostic blind spot when evaluating AI weather forecasting models: they cannot identify statistical artifacts where AI weather forecasting models use "large-area spatial smoothing" to obtain high scores for grid points. This smoothing strategy leads to a collapse in the forecast field variance and causes severe frequency distortion, resulting in extreme wet biases in AI weather forecasting models at low thresholds (such as light rain) and extreme dry biases at extreme thresholds (such as heavy rainstorms) because peak intensity is completely suppressed. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides an evaluation method, system, device, and medium for artificial intelligence weather forecast models, which can accurately quantify and penalize frequency distortion caused by spatial variance suppression, thereby objectively evaluating the real physical performance and distribution integrity of artificial intelligence weather forecast models in extreme events.

[0005] To achieve the above objectives, the present invention provides an artificial intelligence weather forecasting model evaluation method, comprising: Acquire gridded meteorological forecast data of the target forecast model for a specified time period, as well as the corresponding actual observed meteorological data; Based on preset meteorological event thresholds, a binary classification contingency table is constructed using the gridded meteorological forecast data and actual observed meteorological data; Based on the binary classification contingency table, the symmetric extreme value dependence index and frequency deviation are calculated. Based on the frequency deviation, a Gaussian penalty function is constructed in the logarithmic deviation space; The symmetric extremum dependence index is weighted and corrected using the Gaussian penalty function to obtain the smoothed penalty extremum dependence index. The performance of the target prediction model is evaluated based on the smoothing penalty extreme value dependence index, and the evaluation results are obtained.

[0006] Optionally, the method further includes: By mapping the frequency deviation to polar coordinates and the symmetric extremum dependence index to polar radius coordinates, a resolution-reliability polar coordinate phase space is constructed. In the resolution-reliability polar phase space, the evolution trajectory of the target forecast model under different meteorological event thresholds or different forecast lead times is plotted.

[0007] Optionally, the method further includes: The ray with a polar angle of 45° in the resolution-reliability polar coordinate phase space is defined as an unbiased physical axis; If the evolution trajectory, as the meteorological event threshold increases, spans from a region with a polar angle greater than 45° to a region with a polar angle less than 45°, and the change in polar angle exceeds a preset threshold, then the target forecast model is determined to have frequency distortion and variance collapse defects caused by spatial smoothing.

[0008] Optionally, the formula for calculating the symmetric extremum dependence index is: In the formula, SEDI represents the symmetric extremum dependence index. Indicates hit rate. This indicates the false alarm rate.

[0009] Optionally, the formula for calculating the frequency deviation is: In the formula, Indicates frequency deviation. The number that is hit. Empty number reporting This is an underreported number.

[0010] Optionally, the formula for calculating the smoothing penalty extreme value dependence index is: In the formula, The SEDI index represents the smoothing penalty extremum dependency index, while the symmetric extremum dependency index represents the symmetric extremum dependency index. Indicates frequency deviation. This is the preset deviation tolerance constant.

[0011] Optionally, .

[0012] This invention also provides an artificial intelligence weather forecast model evaluation system, comprising: The data acquisition unit is used to acquire gridded meteorological forecast data of the target forecast model within a specified time period, as well as the corresponding actual observed meteorological data; The contingency table construction unit is used to construct a binary classification contingency table based on a preset meteorological event threshold using the gridded meteorological forecast data and actual observed meteorological data. The basic index calculation unit is used to calculate the symmetric extreme value dependence index and frequency deviation based on the binary classification contingency table. The penalty correction unit is used for: Based on the frequency deviation, a Gaussian penalty function is constructed in the logarithmic deviation space; The symmetric extremum dependence index is weighted and corrected using the Gaussian penalty function to obtain the smoothed penalty extremum dependence index. The evaluation output unit is used to evaluate the performance of the target prediction model based on the smoothing penalty extreme value dependence index, and obtain the evaluation result.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the evaluation method described above.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the evaluation method described above.

[0015] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: The AI ​​weather forecasting model evaluation method provided by this invention constructs a Gaussian penalty function in the logarithmic bias space and uses this penalty function to weight and correct the symmetric extreme value dependency index, resulting in a smoothing penalty extreme value dependency index. This effectively solves the problem that traditional evaluation indicators cannot diagnose the frequency distortion defects caused by systematic spatial smoothing in AI weather forecasting models. Traditional symmetric extreme value dependency indices can only measure the model's taxonomic resolution and cannot identify statistical artifacts where models use large-area spatial smoothing to manipulate grid point scores. This invention introduces frequency bias into the evaluation framework, so that when the model produces extreme wet bias at low thresholds or extreme dry bias at high thresholds due to spatial smoothing, the frequency bias will deviate significantly from the unbiased state, and the Gaussian penalty term will be greatly reduced. This accurately combats the cheating behavior of models that excessively smooth the weather field to minimize grid point errors.

[0016] This invention achieves a decoupled evaluation of the taxonomic resolution and distribution reliability of artificial intelligence weather forecasting models. By independently calculating the symmetric extreme value dependence index and frequency bias, and unifying them into a comprehensive index—the smoothing penalty extreme value dependence index—the evaluation results reflect both the model's ability to distinguish whether extreme events occur and its physical honesty in terms of frequency distribution. Only when a model possesses both high resolution and good frequency distribution reliability can it obtain a high smoothing penalty extreme value dependence index score, thus providing an objective and reliable quantitative basis for the screening and diagnosis of extreme weather event forecasting models. Attached Figure Description

[0017] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same parts.

[0018] Figure 1 This is a schematic diagram of the method flow for evaluating an artificial intelligence weather forecast model according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the evolution of SP-EDI trajectories under different meteorological event thresholds, as shown in an embodiment of the present invention. Figure 3 This is a schematic diagram of the module structure of an artificial intelligence weather forecast model evaluation system according to an embodiment of the present invention; Figure 4 This is a schematic diagram of an electronic device shown in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating the methodology for evaluating artificial intelligence weather forecasting models.

[0021] Methods for evaluating artificial intelligence weather forecasting models include: S101: Obtain gridded meteorological forecast data of the target forecast model within a specified time period, as well as the corresponding actual observed meteorological data.

[0022] In application, the first step is to acquire the foundational data for evaluation. The target forecast model refers to the AI-powered weather forecast model to be evaluated, including various data-driven weather forecast models such as AIFS and Pangu-Weather, which are atmospheric forecast models based on deep learning architectures. It can also be a traditional numerical weather prediction model that needs to be tested for performance under extreme events. Model evaluation requires real observational data as a benchmark, therefore, two sets of data need to be acquired simultaneously: one is the gridded meteorological forecast data generated by the target forecast model within a specified time period, and the other is the actual observed meteorological data that strictly corresponds to that time period.

[0023] Gridded weather forecast data refers to the gridded forecast values ​​of meteorological elements, such as precipitation, temperature, and wind speed, output by the target forecast model on a specified spatial grid. The specified time period can be a continuous period of several days, months, or longer, or it can be a representative period selected based on typical weather events. In order to match with observational data, the spatial extent and temporal resolution of the forecast data need to be consistent with the observational data used subsequently, or they can be spatiotemporally aligned through interpolation or other methods.

[0024] Actual meteorological observation data can come from ground-based meteorological observation stations, radar quantitative precipitation estimates, etc. The actual observation data needs to cover the same spatial range and time period as the forecast data, and meet the requirements for constructing a binary classification contingency table in terms of spatiotemporal resolution. When the observation data is station data, it can be mapped to forecast grid points using nearest-neighbor matching or spatial interpolation methods; when the observation data itself is a gridded product, a direct comparison of each grid point can be performed.

[0025] After obtaining the aforementioned basic data, these data can be preprocessed, such as removing missing values, standardizing units, and performing spatiotemporal matching verification, to ensure that the forecast data and the actual observation data have consistent data dimensions and valid numerical ranges before participating in subsequent calculations.

[0026] S102: Based on preset meteorological event thresholds, construct a binary classification contingency table using gridded meteorological forecast data and actual observed meteorological data.

[0027] After acquiring gridded meteorological forecast data (hereinafter referred to as forecast data) and corresponding actual observed meteorological data (hereinafter referred to as observation data), it is necessary to convert the continuous forecast data and observation data into binary judgment results of whether an event has occurred based on preset meteorological event thresholds, and construct a binary classification contingency table accordingly. Meteorological event thresholds refer to the critical values ​​used to define whether a certain meteorological event has occurred, and they can be determined according to the actual assessment objectives. For example, in precipitation forecast assessment, several precipitation level thresholds can be set according to hourly or daily amounts, such as 0.1 mm, 10 mm, 50 mm, and 100 mm, corresponding to different intensity levels such as light rain, moderate rain, heavy rain, and rainstorm, respectively. The selection of meteorological event thresholds can be flexibly set according to assessment needs; extremely high thresholds can be set to examine the model's ability to forecast heavy precipitation events, while low thresholds can be set to examine the model's ability to distinguish weak precipitation events.

[0028] For each selected meteorological event threshold, at each spatiotemporal point (i.e., forecast grid point), the forecast data and observed data are binarized separately. When the forecast value of the forecast data reaches or exceeds the threshold, the forecast event is considered to have occurred and recorded as "yes"; when the forecast value is below the threshold, the forecast event is considered not to have occurred and recorded as "no". Similarly, when the observed value of the observed data reaches or exceeds the threshold, the observed event is considered to have occurred; when the observed value is below the threshold, the observed event is considered not to have occurred. By cross-referencing the binarized judgment results of the forecast data and the binarized judgment results of the observed data grid point by grid point, all sample points can be classified into four mutually exclusive classification cases.

[0029] The four classification scenarios are: both the predicted event and the observed event occur, referred to as the hit number, which can be denoted as... When a predicted event occurs but the observed event does not, it is called a false alarm, which can be denoted as . b When a predicted event does not occur but an observed event does, the number of missed reports is called the number of false negatives (Misses), which can be denoted as _____. c When both the predicted and observed events fail to occur, the number of correct negatives is called a correct negative, which can be denoted as: dAs shown in Table 1, filling the above four statistics into a 2x2 table will form a binary classification contingency table.

[0030] Table 1

[0031] Taking the forecasting and assessment of precipitation classification as an example, all valid grid points and all valid time periods are traversed under the corresponding meteorological event threshold, and statistics are collected respectively. , b , c , d The numerical value. This process can be completed automatically by a computer program, which generates a corresponding binary mask matrix from the forecast data matrix and the observation data matrix according to the meteorological event threshold, and then quickly accumulates the matrix through logical operations to obtain four statistics. The hit count is then used to determine the statistical value. Empty reporting b Underreporting c and correct negation number d The relative magnitudes of these values ​​can provide a preliminary indication of the model's forecast tendency under the threshold of that meteorological event. For example, when the airborne data... b Significantly greater than the number of underreported cases c When the model tends to over-predict the weather event, it shows signs of wet bias; conversely, when the model misses a number of cases... c Significantly greater than the number of empty reports b At that time, the model tends to miss the meteorological event, indicating a dry bias.

[0032] Understandably, for different meteorological event thresholds, it is necessary to construct separate binary classification contingency tables, with each meteorological event threshold corresponding to an independent binary classification contingency table.

[0033] S103: Calculate the symmetric extreme value dependence index and frequency deviation based on the binary classification contingency table.

[0034] Based on the constructed binary classification contingency table, the two fundamental evaluation parameters, symmetric extreme value dependence index and frequency bias, are calculated. These two parameters characterize the performance of the AI ​​weather forecasting model from the dimensions of taxonomic resolution and distribution reliability, respectively.

[0035] The symmetric extreme value dependence index is a statistic that measures a model's ability to distinguish between the occurrence and non-occurrence of events. A higher value indicates that the model is more effective in taxonomically in differentiating regions from those where extreme events have occurred. The formula for calculating the symmetric extreme value dependence index is: In the formula, SEDI represents the symmetric extremum dependence index. Indicates hit rate. This indicates the false alarm rate. Among them, the hit rate... It is obtained by dividing the number of hits by the sum of the number of hits and the number of missed calls, i.e. = / ( + ); The false alarm rate is calculated by dividing the number of false alarms by the sum of the number of false alarms and the number of correct negative alarms. = / ( + ).

[0036] Frequency deviation measures the ratio of predicted frequency to observed frequency and is a fundamental parameter describing the reliability of a model's distribution. The formula for calculating frequency deviation is: In the formula, Indicates frequency deviation. The number that is hit. Empty number reporting This is an underreported number.

[0037] when When the value equals 1, it indicates that the total frequency of predicted events is completely consistent with the total frequency of observed events, and the model is unbiased in frequency distribution; when... A value greater than 1 indicates a wet bias in the model forecast, meaning the predicted event occurs more frequently than the observed event; when When the value is less than 1, it indicates that there is a bias in the model forecast, meaning that the frequency of the forecast event is lower than that of the observed event.

[0038] S104: Construct a Gaussian penalty function in the logarithmic deviation space based on the frequency deviation.

[0039] The purpose of this Gaussian penalty function is to mathematically penalize and correct the frequency distortion caused by spatial smoothing in data-driven weather forecasting models when suppressing grid point errors. The smoothing effect exhibited by data-driven models when forecasting extreme weather events leads to a systematic suppression of the spatial variance of the forecast field. Specifically, at low thresholds, the model tends to predict events over a large area, resulting in inflated false negatives and a significant deviation of the frequency bias from the unbiased state. At high thresholds, due to the weakening of peak intensity through smoothing, the model is almost unable to trigger events, leading to a sharp increase in false negatives and a significant deviation of the frequency bias from the unbiased state again. To quantify and penalize this bias, this invention transforms the frequency bias into logarithmic space for processing.

[0040] The mathematical form of the Gaussian penalty function is: in, For frequency deviation, This is the preset deviation tolerance constant. This function... The maximum value of 1 is achieved when the value is equal to 1, indicating that no penalty is imposed when the model is completely unbiased in the frequency distribution; as... As the deviation from 1 increases, regardless of whether the deviation is wet or dry, the penalty weight decays smoothly in the form of a Gaussian function. This decay characteristic means that slight frequency deviations are only slightly penalized, while severe frequency distortions are severely penalized, thus effectively suppressing the statistical artifact of obtaining high symmetric extremum dependent scores through spatial smoothing in the evaluation results.

[0041] Deviation tolerance constant It controls the severity of the punishment. In a preferred embodiment, Of course, depending on the actual needs of different application scenarios, The value can also be adjusted. In extreme event early warning assessments with high reliability requirements, a smaller value can be set. This allows for the imposition of stricter frequency deviation constraints; in typical evaluation scenarios, relatively lenient constraints can be set. value.

[0042] S105: The symmetric extremum dependence index is weighted and corrected using the Gaussian penalty function to obtain the smoothed penalty extremum dependence index.

[0043] In application, the evaluation information of the two dimensions of taxonomic resolution and distribution reliability is unified into a comprehensive index through weighted correction. This allows the symmetric extreme value dependence index, which originally could only measure the model’s discriminative ability, to reflect the physical honesty of the model in the frequency distribution after being constrained by the frequency bias penalty.

[0044] Smoothing penalty extreme value dependence index The calculation formula is: In the formula, SEDI represents the symmetric extremum dependence index. Indicates frequency deviation. This is the preset deviation tolerance constant.

[0045] The physical significance of this penalty mechanism lies in precisely targeting the behavior of data-driven AI weather forecasting models that excessively smooth the meteorological field in order to minimize grid point errors. Under low meteorological event thresholds (such as light rain), the model achieves high hit rates through large-area spatial smoothing, thus maintaining a high symmetric extreme value dependency index score. However, this also results in severe frequency wet bias due to inflated false alarm numbers. Under the smoothing penalty extreme value dependency index framework of this invention, although its symmetric extreme value dependency index is high, the severe deviation in frequency bias will cause a significant decay of the Gaussian penalty term, thereby reducing the final smoothing penalty extreme value dependency index score. Similarly, under high meteorological event thresholds (such as heavy rain), the model can hardly trigger event forecasts because the peak intensity is completely suppressed. Although it may still maintain a certain symmetric extreme value dependency index value from a contingency table statistical perspective, its extreme frequency wet bias will also trigger the corresponding penalty.

[0046] Through this weighted correction mechanism, the smoothing penalty extreme value dependence index achieves a decoupled evaluation of the model's taxonomic resolution and distribution reliability, effectively filling the diagnostic blind spot where traditional indicators such as the symmetric extreme value dependence index cannot identify the statistical artifact of large-area spatial smoothing deceiving grid points to obtain high scores.

[0047] Furthermore, the smoothing penalty extremum dependence index has the mathematical property of being continuously differentiable. It can not only serve as a benchmark index for post-model evaluation, but also has the potential to be directly integrated into the loss function of data-driven artificial intelligence weather forecasting models as a differentiable constraint term, thereby guiding the next generation of artificial intelligence weather forecasting models to perform physical heuristic optimization.

[0048] In practical applications, the smoothing penalty extreme value dependence index can be used as an auxiliary training constraint. It can be introduced into the loss function of data-driven weather forecasting models in a differentiable approximation form as a regularization term or auxiliary loss term, participating in the backpropagation update of model parameters to suppress frequency bias in extreme precipitation forecasts. Furthermore, this index can also be applied to the post-processing stage of the model as an objective function for post-processing optimization, reducing frequency distortion by adjusting the statistical distribution of forecast results. It should be noted that the smoothing penalty extreme value dependence index is constructed based on a thresholded contingency table. When using it as a training constraint, its differentiable approximation form must be constructed, and it is preferable to use it in combination with traditional continuous loss functions (such as mean squared error, quantile loss, etc.) and spatial structure diagnostic constraints (such as spatial gradient consistency constraints) to balance grid-by-grid accuracy, spatial distribution integrity, and reliability of extreme event frequency.

[0049] S106: The performance of the target prediction model is evaluated based on the smoothing penalty extreme value dependence index, and the evaluation results are obtained.

[0050] After calculating the smoothing penalty extreme value dependence index, the performance of the target prediction model is evaluated based on this index, yielding the final evaluation result. The smoothing penalty extreme value dependence index itself is a comprehensive quantitative measure of model performance, and its value directly reflects the model's overall performance in balancing taxonomic resolution and distribution reliability. A higher smoothing penalty extreme value dependence index indicates that while maintaining high event resolution, the deviation between predicted and observed frequencies is smaller, and the prediction results are closer to physical reality. A lower smoothing penalty extreme value dependence index indicates that the model either has insufficient taxonomic resolution, serious frequency distortion, or both.

[0051] The evaluation results can be used to characterize whether the target forecast model has frequency distortion defects caused by spatial smoothing. This diagnostic capability is not available in traditional evaluation metrics. The traditional symmetric extreme value dependency index only measures the model's taxonomic resolution. When an AI weather forecast model maintains a high hit rate at grid points through a large-area spatial smoothing strategy, the symmetric extreme value dependency index may still give a high score, thus masking the serious frequency distortion caused by the smoothing effect. The smoothing-penalized extreme value dependency index introduces a Gaussian penalty based on frequency deviation on top of the symmetric extreme value dependency index, effectively deducting points for any forecast that deviates from the frequency unbiased state, thereby exposing the hidden defect of frequency distortion in the evaluation results.

[0052] The evaluation results can be output in the form of numerical reports, including but not limited to the name of the target forecast model, the evaluation period, the list of meteorological event thresholds used, the values ​​of the smoothing penalty extreme value dependence index corresponding to each threshold, and the values ​​of the original symmetric extreme value dependence index and frequency deviation, providing direct quantitative basis for evaluators to identify frequency distortion defects in the model.

[0053] In one embodiment, the above method further includes: By mapping frequency deviation to polar coordinates and symmetric extremum dependence exponent to polar radius coordinates, a resolution-reliability polar coordinate phase space is constructed. In the resolution-reliability polar phase space, plot the evolution trajectory of the target forecast model under different meteorological event thresholds or different forecast lead times.

[0054] In application, after completing the calculation of the smoothing penalty extreme value dependence index, in order to further reveal the physical causes behind the performance of the target prediction model, especially to intuitively present whether the model has frequency distortion and distribution drift phenomena across thresholds, this invention further introduces the topological diagnostic tool of resolution-reliability polar coordinate phase space.

[0055] When constructing this resolution-reliability polar phase space, the frequency deviation is mapped to polar coordinates, and the mapping formula is: Simultaneously, the symmetric extremum dependence index is mapped to polar radius coordinates, and the mapping formula is: because , After mapping, the polar radius polar angle The entire mapping process ensures the compactness of the polar coordinate phase space.

[0056] Through the above mapping, the frequency deviation, a measure of the reliability of the distribution, is converted into an angular position in polar phase space. The symmetry extremum dependence index, a measure of taxonomic resolution, is converted into the radial distance from the origin in polar phase space. Each set of frequency bias and symmetric extreme value dependence index corresponds to a unique point in the polar phase space. The performance of the model under a specific meteorological event threshold and a specific forecast lead time is compactly represented as a two-dimensional coordinate point. By connecting the forecast state points of the target forecast model under different meteorological event thresholds, such as from light rain threshold to heavy rain threshold, in ascending order of threshold, or by connecting the state points of the model under different forecast lead times, such as from 6-hour forecast to 72-hour forecast, in ascending order of lead time, an evolution trajectory can be plotted in the polar phase space.

[0057] See Figure 2 , Figure 2 The paper presents the evolution trajectories of two AI weather forecasting models, Model A and Model B, under different meteorological event thresholds of 0.1 mm, 10.0 mm, 25.0 mm, 50.0 mm, and 100.0 mm. The evolution trajectories comprehensively record the dynamic evolution of model performance with varying meteorological event thresholds or lead times. The characteristic patterns of model performance can be clearly identified from the direction and shape of the evolution trajectories.

[0058] Furthermore, the above methods also include: In the resolution-reliability polar coordinate phase space, a ray with a polar angle of 45° is defined as an unbiased physical axis. If the evolution trajectory spans from a region with a polar angle greater than 45° to a region with a polar angle less than 45° as the meteorological event threshold increases, and the change in polar angle exceeds the preset threshold, then the target forecast model is determined to have frequency distortion and variance collapse defects caused by spatial smoothing.

[0059] In applications, based on the constructed resolution-reliability polar phase space and the evolutionary trajectories plotted within it, automated diagnosis of frequency distortion defects in target prediction models can be achieved. First, a ray with a polar angle of 45° in the polar phase space can be defined as an unbiased physical axis. At this point, the prediction frequency of the prediction model is completely consistent with the observation frequency, and the model is unbiased in its frequency distribution. Therefore, the ray with a polar angle of 45° constitutes a benchmark reference line for measuring the physical honesty of the model throughout the phase space. Trajectory points located above this ray, i.e., in the region with a polar angle greater than 45°, indicate a frequency deviation greater than 1, indicating a wet bias in the model; trajectory points located below this ray, i.e., in the region with a polar angle less than 45°, indicate a frequency deviation less than 1, indicating a dry bias in the model.

[0060] Based on this, an automated diagnosis of the evolution trajectory is performed. If the evolution trajectory of the target forecast model crosses from a region with a polar angle greater than 45° to a region with a polar angle less than 45° as the meteorological event threshold increases, this means that the model exhibits a wet bias with more forecast frequencies at low thresholds, and a dry bias with fewer forecast frequencies at high thresholds. This deflection across the unbiased physical axis directly reflects a reversal in the model's frequency behavior between events of different intensities. Furthermore, the change in polar angle of the trajectory is calculated, i.e., the difference between the polar angle at the high threshold end and the polar angle at the low threshold end. This change in polar angle is compared with a preset threshold; if it exceeds the preset threshold, the target forecast model is determined to have frequency distortion and variance collapse defects caused by spatial smoothing.

[0061] Through the aforementioned automated diagnostics, it is possible to automatically identify and warn of frequency distortion and variance collapse defects caused by spatial smoothing in target forecast models without the need for manual interpretation. This provides an efficient and feasible technical means for large-scale, multi-model operational evaluation.

[0062] Corresponding to the aforementioned application function implementation method embodiments, the present invention also provides an artificial intelligence weather forecast model evaluation system and corresponding embodiments.

[0063] Please see Figure 3 , Figure 3 This is a schematic diagram of the module structure of an artificial intelligence weather forecast model evaluation system.

[0064] An AI-powered weather forecasting model evaluation system includes: The data acquisition unit 31 is used to acquire gridded meteorological forecast data of the target forecast model within a specified time period, as well as the corresponding actual observed meteorological data; The contingency table construction unit 32 is used to construct a binary classification contingency table based on a preset meteorological event threshold using gridded meteorological forecast data and actual observed meteorological data. Basic index calculation unit 33 is used to calculate the symmetric extreme value dependence index and frequency deviation based on the binary classification contingency table; Penalty correction unit 34, used for: Based on the frequency deviation, a Gaussian penalty function is constructed in the logarithmic deviation space; By applying a Gaussian penalty function to the symmetric extremum dependency index with weights, a smoothed penalty extremum dependency index is obtained. The evaluation output unit 35 is used to evaluate the performance of the target prediction model based on the smoothing penalty extreme value dependence index and obtain the evaluation results.

[0065] In one embodiment, the system further includes: The evolution trajectory drawing unit is used for: By mapping the frequency deviation to polar coordinates and the symmetric extremum dependence index to polar radius coordinates, a resolution-reliability polar coordinate phase space is constructed. In the resolution-reliability polar phase space, the evolution trajectory of the target forecast model under different meteorological event thresholds or different forecast lead times is plotted.

[0066] In one embodiment, the system further includes: The decision unit is used for: The ray with a polar angle of 45° in the resolution-reliability polar coordinate phase space is defined as an unbiased physical axis; If the evolution trajectory, as the meteorological event threshold increases, spans from a region with a polar angle greater than 45° to a region with a polar angle less than 45°, and the change in polar angle exceeds a preset threshold, then the target forecast model is determined to have frequency distortion and variance collapse defects caused by spatial smoothing.

[0067] Regarding the system in the above embodiments, the specific manner in which each unit module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0068] This application also provides an electronic device, please refer to [link to relevant documentation]. Figure 4 The electronic device 4000 includes a memory 4010 and a processor 4020.

[0069] The processor 4020 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0070] Memory 4010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 4020 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 4010 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 4010 may include removable storage devices that are readable and / or writable, such as laser discs (CDs), read-only digital multifunction optical discs (e.g., DVD-ROMs, dual-layer DVD-ROMs), read-only Blu-ray discs, ultra-high density optical discs, flash memory cards (e.g., SD cards, mini SD cards, Micro-SD cards, etc.), magnetic floppy disks, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0071] The memory 4010 stores executable code, which, when processed by the processor 1020, can cause the processor 4020 to execute part or all of the methods described above.

[0072] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0073] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0074] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for evaluating artificial intelligence weather forecasting models, characterized in that, include: Acquire gridded meteorological forecast data of the target forecast model for a specified time period, as well as the corresponding actual observed meteorological data; Based on preset meteorological event thresholds, a binary classification contingency table is constructed using the gridded meteorological forecast data and actual observed meteorological data; Based on the binary classification contingency table, the symmetric extreme value dependence index and frequency deviation are calculated. Based on the frequency deviation, a Gaussian penalty function is constructed in the logarithmic deviation space; The symmetric extremum dependence index is weighted and corrected using the Gaussian penalty function to obtain the smoothed penalty extremum dependence index. The performance of the target prediction model is evaluated based on the smoothing penalty extreme value dependence index, and the evaluation results are obtained.

2. The method for evaluating an artificial intelligence weather forecast model according to claim 1, characterized in that, The method further includes: By mapping the frequency deviation to polar coordinates and the symmetric extremum dependence index to polar radius coordinates, a resolution-reliability polar coordinate phase space is constructed. In the resolution-reliability polar phase space, the evolution trajectory of the target forecast model under different meteorological event thresholds or different forecast lead times is plotted.

3. The method for evaluating an artificial intelligence weather forecast model according to claim 2, characterized in that, The method further includes: The ray with a polar angle of 45° in the resolution-reliability polar coordinate phase space is defined as an unbiased physical axis; If the evolution trajectory, as the meteorological event threshold increases, spans from a region with a polar angle greater than 45° to a region with a polar angle less than 45°, and the change in polar angle exceeds a preset threshold, then the target forecast model is determined to have frequency distortion and variance collapse defects caused by spatial smoothing.

4. The method for evaluating an artificial intelligence weather forecast model according to claim 1, characterized in that, The formula for calculating the symmetric extremum dependence index is as follows: In the formula, SEDI represents the symmetric extremum dependence index. Indicates hit rate. This indicates the false alarm rate.

5. The method for evaluating an artificial intelligence weather forecast model according to claim 1, characterized in that, The formula for calculating the frequency deviation is: In the formula, Indicates frequency deviation. The number that is hit. The number is empty. This is an underreported number.

6. The method for evaluating an artificial intelligence weather forecast model according to claim 1, characterized in that, The formula for calculating the smoothing penalty extreme value dependence index is as follows: In the formula, The SEDI index represents the smoothing penalty extremum dependency index, while the symmetric extremum dependency index represents the symmetric extremum dependency index. Indicates frequency deviation. This is the preset deviation tolerance constant.

7. The method for evaluating an artificial intelligence weather forecast model according to claim 6, characterized in that, 。 8. An artificial intelligence weather forecasting model evaluation system, characterized in that, include: The data acquisition unit is used to acquire gridded meteorological forecast data of the target forecast model within a specified time period, as well as the corresponding actual observed meteorological data; The contingency table construction unit is used to construct a binary classification contingency table based on a preset meteorological event threshold using the gridded meteorological forecast data and actual observed meteorological data. The basic index calculation unit is used to calculate the symmetric extreme value dependence index and frequency deviation based on the binary classification contingency table. The penalty correction unit is used for: Based on the frequency deviation, a Gaussian penalty function is constructed in the logarithmic deviation space; The symmetric extremum dependence index is weighted and corrected using the Gaussian penalty function to obtain the smoothed penalty extremum dependence index. The evaluation output unit is used to evaluate the performance of the target prediction model based on the smoothing penalty extreme value dependence index, and obtain the evaluation result.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed, implements the method as described in any one of claims 1 to 7.

Images