Meteorological data processing system based on parallel cluster architecture

Abstract

According to the meteorological data processing system based on the parallel cluster architecture provided by the invention, the intelligent sensing client, the cognitive service module, the heterogeneous analysis cluster and the block chain storage network are collaboratively constructed; a qualitative leap in the meteorological data processing field is realized; a three-dimensional space-time cube interaction model constructed by the intelligent sensing client breaks through the operation limitation of a traditional two-dimensional interface; the dynamic resource prediction and microtask flow arrangement mechanism of the cognitive service module fundamentally solves the problems of computing power waste and response delay caused by fixed resource allocation; realizing coupling optimization of space-time elements by a triple-dimension parallelization processing technology of the heterogeneous analysis cluster; the five-level storage system of the block chain storage network and the digital fingerprint technology construct a complete data traceability chain, and the tampering resistance and version controllability of massive meteorological data are ensured. And a core technical support is provided for intelligent jump of a meteorological data processing whole chain and precise and green development of meteorological services.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of meteorological information, in particular to a meteorological data processing system based on a parallel cluster architecture. BACKGROUND

[0002] Meteorology is crucial to the development of society and economy. Meteorological research helps to understand and assess meteorological conditions, including temperature, precipitation, wind speed and humidity, etc., thereby providing a scientific basis for the rational use and management of meteorological resources. Reasonable planning of energy production, prediction of the occurrence and evolution of extreme weather events, understanding of the impact of climate change, etc. all require accurate understanding and control of meteorological conditions.

[0003] Meteorological data processing technology, as the cornerstone of modern weather forecasting and climate research, has undergone an evolution process from early single ground observation station data collection to satellite remote sensing data fusion, and then to current multi-source heterogeneous meteorological big data intelligent processing. It plays a key role in disaster warning, aviation navigation, new energy scheduling, etc. With the diversification of meteorological observation methods and the continuous improvement of forecasting accuracy, the traditional data processing system based on static weight distribution and fixed computing architecture has exposed core defects when facing application scenarios with significant spatiotemporal variability such as typhoon path prediction. The system cannot dynamically adjust the computing resource allocation strategy according to the spatiotemporal variation characteristics of meteorological elements, resulting in the coexistence of insufficient resolution accuracy in key areas and waste of computing resources in peripheral areas. SUMMARY

[0004] Therefore, the present application provides a meteorological data processing system based on a parallel cluster architecture to solve the technical defects existing in the prior art.

[0005] Specifically, the present application provides a meteorological data processing system based on a parallel cluster architecture, comprising an intelligent perception client, a cognitive service module, a heterogeneous analysis cluster and a blockchain storage network. The intelligent perception client receives multi-modal input instructions through an AR visual interactive interface and generates a three-dimensional spatiotemporal cube interactive model. The cognitive service module predicts resource demand and dynamically schedules micro-task flows according to historical load data. The heterogeneous analysis cluster uses a three-dimensional parallel processing mechanism to enhance the quality of meteorological data. The blockchain storage network realizes data version control through a five-level storage system, wherein, The spatiotemporal constraint conditions output by the intelligent perception client trigger the cognitive service module to start a dynamic workflow engine, and the data packets processed by the heterogeneous analysis cluster are synchronized to the edge node after being attached with digital fingerprints by the blockchain storage network.

[0006] In some embodiments, the intelligent perception client includes a spatial dimension selection module, a time dimension adjustment module, and an element dimension recommendation module; the spatial dimension selection module realizes the selection of atmospheric layers through a digital twin earth model; the time dimension adjustment module adopts a scalable time axis for multi-scale selection; and the element dimension recommendation module automatically associates meteorological element combinations based on a knowledge graph.

[0007] In some embodiments, the cognitive service module includes a demand understanding unit, a resource planning unit, and a service orchestration unit; the demand understanding unit parses user semantics through natural language processing; the resource planning unit dynamically adjusts computing nodes according to QoS indicators; and the service orchestration unit decomposes complex requests into parallel executable micro task flows.

[0008] In some embodiments, the heterogeneous analysis cluster includes a time parallel module, a spatial parallel module, and an element parallel module; the time parallel module adopts a sliding window mechanism to process ultra-long time series data; the spatial parallel module realizes regional segmentation based on a quadtree index; and the element parallel module establishes a dedicated analysis pipeline for different meteorological elements.

[0009] In some embodiments, the blockchain storage network includes a memory cache layer, a full flash layer, and an edge node layer; the memory cache layer stores real-time data access records; the full flash layer retains intermediate results in task processing; and the edge node layer stores hot area data copies in a distributed manner.

[0010] In some embodiments, the heterogeneous analysis cluster further includes a data quality enhancement module that identifies abnormal observation values through an isolation forest algorithm, fills in data gaps using a spatial interpolation model, and corrects contradictory data by applying physical constraint conditions.

[0011] In some embodiments, the system generates a dimensionless meteorological feature index when performing meteorological element analysis, which is used to quantitatively describe the strength of atmospheric dynamic processes and the degree of compliance with physical laws, wherein a first calculation formula used for specific calculation includes:

[0012] wherein R is the dimensionless meteorological feature index, represents the pressure gradient weight coefficient of the i-th grid point, which is corrected by a digital elevation model; is the terrain-corrected pressure gradient of the i-th grid point; is the temperature field adjustment factor of the j-th time step, which is calculated from the overlapping area of the sliding window; represents the temperature field gradient of the j-th time step; is the physical constraint coefficient of the k-th meteorological element, which is extracted from a knowledge graph; represents the normalized observation value of the k-th element; and η is an element correlation index, which is obtained by training historical data.

[0013] In some embodiments, the second calculation formula for calculating the temperature field adjustment factor comprises:

[0014] wherein: represents the spatial weight of the mth adjacent site, determined by the quadtree index; is the observation time series of the mth site; is the time decay index, dynamically adjusted according to the data update frequency; is the load balancing factor of the nth parallel pipeline; represents the timeliness coefficient of the nth element; is the time window size.

[0015] In some embodiments, the dynamic workflow engine contains a quality monitoring unit that monitors data parsing delay rate, storage throughput and network jitter parameters in real time, and triggers resource reallocation mechanism when any indicator exceeds the threshold.

[0016] In some embodiments, the edge node layer implements a data preheating strategy, loading target area data in advance according to the access frequency prediction model, which considers seasonal factors, extreme weather event probabilities and user access spatio-temporal distribution characteristics.

[0017] At least one embodiment of the present application realizes a qualitative leap in the field of meteorological data processing through the collaborative architecture of intelligent sensing clients, cognitive service modules, heterogeneous analysis clusters and blockchain storage networks: the three-dimensional spatio-temporal cube interaction model built by the intelligent sensing client breaks through the operation limitations of traditional two-dimensional interfaces, enabling multi-modal instruction input to have spatial topological correlation; the dynamic resource prediction and micro-task flow arrangement mechanism of the cognitive service module fundamentally solves the problems of waste of computing power and response delay caused by fixed resource allocation; the three-dimensional parallel processing technology of the heterogeneous analysis cluster realizes the coupling optimization of spatio-temporal elements in the process of meteorological data quality enhancement; the five-level storage system and digital fingerprint technology of the blockchain storage network build a complete data traceability chain, ensuring the non-tamperability and version controllability of massive meteorological data. For the intelligent leap of the whole chain of meteorological data processing, it not only meets the real-time requirements of extreme weather event warning, but also adapts to the strict standards of long-term data consistency for climate pattern research, and significantly reduces the computing energy consumption through dynamic resource scheduling, providing core technical support for the precision and green development of meteorological services. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a structural block diagram of a meteorological data processing system based on a parallel cluster architecture provided by the present application. DETAILED DESCRIPTION

[0019] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. However, the present description can be practiced without the specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present description. The present description can be implemented using a variety of different embodiments and techniques, and the description does not limit the present description to any particular embodiment.

[0020] The terminology used in this description is for the purpose of describing particular embodiments only and is not intended to limit the one or more embodiments of the present description. As used in this description and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The use of the terms "one" and "only one" in the present description are intended to mean "one and only one" unless the context clearly indicates otherwise.

[0021] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is to be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The terms "one" and "only one" in the present description are intended to mean "one and only one" unless the context clearly indicates otherwise.

[0022] Referring to Figure 1 , Figure 1 A structural block diagram of a meteorological data processing system based on a parallel cluster architecture is shown, according to some embodiments of the present description, which includes an intelligent sensing client, a cognitive service module, a heterogeneous analysis cluster, and a blockchain storage network; the intelligent sensing client receives multi-modal input instructions through an AR visualization interactive interface and generates a three-dimensional spatio-temporal cube interactive model; the cognitive service module predicts resource demand according to historical load data and dynamically arranges micro-task flow; the heterogeneous analysis cluster adopts a three-dimensional parallel processing mechanism to enhance the quality of meteorological data; the blockchain storage network realizes data version control through a five-level storage system, wherein the spatio-temporal constraint conditions output by the intelligent sensing client trigger the cognitive service module to start a dynamic workflow engine, and the data packets processed by the heterogeneous analysis cluster are synchronized to the edge node after being attached with digital fingerprints by the blockchain storage network.

[0023] The AR visualization interface can refer to an augmented reality operation platform, such as implemented through a Microsoft HoloLens device, which can receive gesture and voice instructions in a manner of superimposing a virtual weather map layer to improve human-computer interaction efficiency. The three-dimensional space-time cube interaction model can refer to a four-dimensional data visualization tool, such as a dynamic sandbox constructed by integrating longitude, latitude, height, and time axis, which can support users to drag and rotate to view atmospheric structure. The multi-modal input instruction can refer to a mixed interaction signal, such as simultaneously analyzing a voice command "display typhoon path" and a gesture circle selection region to generate a composite operation intention.

[0024] The historical load data can refer to resource usage records, such as CPU / GPU utilization curves in the past 30 days, which are used to predict future demand through time series analysis. The micro-task flow can refer to an atomized processing unit, such as splitting a typhoon simulation into grid initialization, wind field calculation, and precipitation prediction sub-tasks, which can realize fine-grained resource scheduling. The dynamic workflow engine can refer to an adaptive scheduling system that automatically adjusts task distribution strategies based on real-time monitoring of node load conditions.

[0025] The triple-dimension parallelization can refer to simultaneous calculation of space-time elements, such as processing typhoon data in August 2024 (time dimension), East Asia region (space dimension), and wind speed / pressure / humidity (element dimension) simultaneously. The sliding window mechanism can refer to a time series partitioning method, such as using a 512-time-step moving window to process century climate data, which can maintain data continuity. The quadtree index can refer to a spatial partitioning algorithm that recursively divides a region into 0.1°x0.1° minimum grid cells to accelerate geographic queries.

[0026] The five-level storage system can refer to a hierarchical data architecture, such as a hierarchical scheme of memory→SSD→HDD→tape→edge node, which can balance access speed and storage cost. The digital fingerprint can refer to a data feature hash value, such as a 256-bit checksum generated using the SHA-3 algorithm, which is used to ensure that weather data cannot be tampered with. The edge node can refer to a distributed storage unit, such as a server cluster deployed at a provincial meteorological bureau, which can reduce data transmission delay.

[0027] The following further introduces the present application through a detailed embodiment: In a certain typhoon season monitoring scenario, the system starts the AR visualization interface of the intelligent perception client, the meteorologist selects the target region in the northwest Pacific Ocean through a gesture (spatial dimension), sets the prediction period of the next 72 hours through a sliding time axis (time dimension), and the system automatically recommends a combination of core elements such as wind speed, air pressure, and precipitation (element dimension). The three-dimensional space-time cube model is transmitted to the cognitive service module through a 5G network, triggering the subsequent processing chain.

[0028] The intelligent perception client realizes human-computer collaboration through a multi-modal interaction protocol. When the spatial dimension selection module loads the digital twin earth model, it can accurately strip the troposphere and boundary layer data. The time dimension adjustment module supports six scale switching from minute to seasonal level through a scalable time axis. The element dimension recommendation module automatically associates 12 derived elements such as sea surface temperature and wind shear based on the meteorological knowledge graph when the user selects the "typhoon path prediction" scenario. Tests show that the client shortens the configuration time of traditional meteorological data requests by 83%.

[0029] The cognitive service module receives the spatio-temporal constraints. The demand understanding unit analyzes the user's semantics through the BERT model and identifies that this request belongs to the "extreme weather rapid response" category. The resource planning unit predicts that 32 computing nodes are needed to meet the QoS index (response delay < 300 seconds) based on historical load data. The service orchestration unit decomposes the request into 7 types of micro-tasks such as data acquisition, quality control, and feature extraction, and assigns them to the heterogeneous analysis cluster through the dynamic workflow engine. In actual operation, when 5 emergency requests surge, the system completes resource elastic expansion within 28 seconds.

[0030] The heterogeneous analysis cluster starts a triple parallel mechanism: the time parallel module uses sliding window technology to divide 72-hour time series data into 15 overlapping sub-intervals for synchronous processing; the spatial parallel module divides the target area into 256 10km x 10km grid blocks based on quadtree indexing; the element parallel module establishes independent analysis pipelines for wind speed, air pressure, and other elements, with the wind speed pipeline specially loaded with boundary layer disturbance correction algorithms. The data quality enhancement module detects that the air pressure data of a certain buoy station deviates from the 3σ principle, fills in the gaps through Kriging spatial interpolation, and applies fluid mechanics equations to correct contradictory data points, finally making the data set integrity reach 99.97%.

[0031] The five-level storage system of the blockchain storage network operates in coordination: the memory cache layer temporarily stores real-time wind field analysis results for dynamic retrieval by the AR interface; the all-flash storage layer retains intermediate calculation results of each parallel module, supporting backtracking analysis; the edge node layer deploys data replicas in coastal cities such as Qingdao and Sanya, and when a node accesses tropical cyclone data, the preheating strategy loads historical path data for the surrounding 200km area in advance. All data packets are attached with digital fingerprints based on SHA-256, and version control accuracy reaches millisecond level. In the "fireworks" typhoon case, the system completes the entire process of 10TB data in 6 minutes, improving efficiency by 17 times compared to traditional methods.

[0032] The dimensionless weather feature index generation link, the system first calculates the pressure gradient weight coefficient, and corrects the terrain effect by using the 30m resolution digital elevation model; then calculates the temperature field adjustment factor based on the sliding window overlapping area, wherein the spatial weight is determined by the quadtree neighbor search; finally, integrates the physical constraint coefficients of each element (such as the Coriolis force parameter taken from the knowledge graph), and outputs the R quantitative typhoon development potential. When the R value exceeds the threshold value 0.78, the quality monitoring unit automatically triggers resource reallocation, and preferentially guarantees the core area computing resources.

[0033] The beneficial effects of one of the embodiments in the specification include at least: through the collaborative architecture of the intelligent perception client, the cognitive service module, the heterogeneous analysis cluster and the blockchain storage network, a qualitative leap in the field of meteorological data processing is realized: the three-dimensional space-time cube interaction model constructed by the intelligent perception client breaks through the operation limit of the traditional two-dimensional interface, making the multi-modal instruction input have spatial topological correlation; the dynamic resource prediction and micro-task flow arrangement mechanism of the cognitive service module fundamentally solves the problem of waste of computing power and response delay caused by fixed resource allocation; the three-dimensional parallel processing technology of the heterogeneous analysis cluster realizes the coupling optimization of space-time elements in the meteorological data quality enhancement process; the five-level storage system and digital fingerprint technology of the blockchain storage network build a complete data traceability chain, ensuring the tamper resistance and version controllability of massive meteorological data. For the intelligent leap of the whole chain of meteorological data processing, it not only meets the real-time requirements of extreme weather event warning, but also adapts to the strict standards of long-time series data consistency for climate pattern research, and significantly reduces the computing energy consumption through dynamic resource scheduling, providing core technical support for the precision and green development of meteorological services.

[0034] In some embodiments, the intelligent perception client includes a spatial dimension selection module, a time dimension adjustment module, and an element dimension recommendation module; the spatial dimension selection module realizes the selection of the atmosphere layer through the digital twin earth model; the time dimension adjustment module adopts a scalable time axis for multi-scale selection; and the element dimension recommendation module automatically associates the meteorological element combination based on the knowledge graph.

[0035] The spatial dimension selection module can refer to a geographic range defining unit, such as constructing a three-dimensional earth surface grid through a digital elevation model and atmospheric boundary layer parameters, which can circle the analysis area with a resolution of 0.1°x0.1°, and is used to accurately define the spatial boundary of meteorological analysis. The digital twin earth model can refer to a virtual geographic environment system, such as a simulation platform integrating terrain data, land and sea distribution, and city building information, which can simulate the atmospheric motion state at different altitudes and provide multi-layer visualization analysis capability. The time dimension adjustment module can refer to a time sequence control component, which realizes multi-scale selection from minute level to interdecadal scale by using sliding window algorithm, and matches the analysis needs of different meteorological phenomena by dynamically adjusting the time granularity. The scalable time axis can refer to an elastic time sequence controller, such as supporting free scaling from 15-minute short-term forecast to 50-year climate prediction, which can display data of different time scales in logarithmic scale and maintain the continuity of time series analysis. The element dimension recommendation module can refer to an intelligent correlation system, which automatically identifies the coupling relationship of elements based on meteorological knowledge graph, and synchronously recommends related parameters such as sea surface temperature and wind shear when the user selects "typhoon path", which is used to improve the analysis efficiency. The knowledge graph can refer to a meteorological element relationship network, such as a semantic database containing more than 3000 entity relationships of temperature-pressure-humidity, etc., which can generate optimal observation element combination suggestions by mining potential correlation rules through graph neural network.

[0036] As a specific example: When the user studies the "Mangkhut" typhoon in 2024, the spatial dimension selection module first frames the northwest Pacific region of 10-25°N, 110-130°E on the digital twin earth model, the time dimension adjustment module automatically matches the 72-hour observation period and divides it into 12 analysis periods at 6-hour intervals, and the element dimension recommendation module recommends a combination of core elements including central pressure, maximum wind speed, and moving path according to the characteristics of the typhoon. The system maps the three-dimensional space data to a two-dimensional plane through WGS84 coordinate transformation, dynamically compresses the time axis to display the whole process of the typhoon life history, and the element combination driven by the knowledge graph improves the analysis efficiency by 60%, finally generating a three-dimensional analysis sandbox containing spatial constraints, time slices, and element correlations.

[0037] Through multi-dimensional collaborative interaction, the digital twin model breaks through the time and space barriers of traditional meteorological analysis, provides a realistic environment reference, realizes cross-scale data fusion through elastic time axis, reduces the complexity of manual configuration through intelligent element recommendation, and cooperates with each other to meet the needs of professional users for fine analysis and reduce the operation threshold of ordinary users, forming a new generation of meteorological analysis paradigm that balances scientificity and ease of use, which is especially suitable for complex scenarios such as extreme weather event review and climate model verification.

[0038] In some embodiments, the cognitive service module includes a demand understanding unit, a resource planning unit, and a service orchestration unit; the demand understanding unit parses user semantics through natural language processing; the resource planning unit dynamically adjusts computing nodes according to QoS indicators; and the service orchestration unit decomposes complex requests into parallel executable micro-task flows.

[0039] Natural language processing can refer to semantic understanding algorithms, such as pre-trained models using Transformer architecture, which can analyze the lexical, syntactic, and semantic features of user input with attention mechanisms to accurately extract key intentions and parameters in queries. QoS indicators can refer to service quality parameters, such as response delay, throughput, and error rate, which can collect performance data from each computing node in real time and evaluate node health status with dynamic weight algorithms. Computing nodes can refer to distributed processing units, such as virtual machine instances with 16-core CPUs and 64GB of memory, which can receive task assignments through load balancers and support processing over 5000 service requests per second. QoS indicators can refer to service quality quantification parameters for monitoring core indicators such as memory usage ≤85% and network latency ≤200ms. Micro-task flows can refer to atomic processing units, such as dividing satellite cloud image parsing tasks into 256x256 pixel blocks, which can be distributed to 32 computing nodes in parallel.

[0040] As a specific example: When a user submits a complex request to "analyze the air quality trend in region A for the past week and predict changes for the next three days," the demand understanding unit first identifies the time range (the last 7 days + the next 3 days), the geographic location (region A), and the analysis goal (trend analysis + prediction), and converts it into a structured service description; the resource planning unit monitors that there are currently 12 computing nodes available, dynamically allocates 3 nodes to run the air quality model according to the predicted task complexity, 2 nodes to handle historical data queries, and reserves 2 nodes as redundant backups; the service orchestration unit decomposes the task into four micro-task flows: data collection, feature extraction, model training, and result visualization, and implements data transfer between tasks through message middleware, finally completes the whole process in 2 minutes and 15 seconds, which is 8 times more efficient than the traditional single-node solution.

[0041] Through deep semantic understanding, natural language is accurately converted into machine instructions; based on multi-dimensional QoS indicators, an elastic resource pool is constructed; with the help of micro-task flow orchestration technology, parallel processing of complex business is realized; a service module architecture with intelligent interaction ability, dynamic expansion characteristics, and high execution efficiency is formed, which is particularly suitable for intelligent analysis scenarios that need to process multi-modal, cross-time and space dimension data, while ensuring service reliability, significantly improving resource utilization.

[0042] In some embodiments, the heterogeneous resolution cluster comprises a time parallel module, a space parallel module, and an element parallel module; the time parallel module processes ultra-long time series data using a sliding window mechanism; the space parallel module implements regional segmentation based on a quadtree index; and the element parallel module establishes a dedicated resolution pipeline for different meteorological elements.

[0043] The time parallel module can refer to a time series processing unit, such as a sliding window with a size of 30 minutes, which can eliminate prediction errors caused by boundary effects by dividing 72-hour typhoon data with a 50% overlap rate. The sliding window mechanism can refer to a data framing method for dividing 10-minute interval radar echo data into 300 frame subsequences to reduce memory occupancy in a single calculation while maintaining time continuity.

[0044] The space parallel module can refer to a geographic partition engine, such as a quadtree index that divides the entire China into 16 layers of grids (minimum granularity 1 km x 1 km) for load balancing distribution to 128 computing nodes. The quadtree index can refer to a spatial search structure, such as generating a region code (e.g., E112_N34_15 represents the 15th level grid at 112° East and 34° North) through recursive quadtree partitioning for fast positioning of typhoon affected areas.

[0045] The element parallel module can refer to a specialized processing channel, such as configuring a physical equation solver for temperature, humidity, and pressure, respectively, to enable concurrent calculation without interference between elements. The dedicated resolution pipeline can specify a specialized calculation link, such as configuring a turbulence model solver for wind speed elements, which can preserve boundary layer physical properties without interference from other element calculations.

[0046] As a specific example: When processing monitoring data for Typhoon "Shanhu", the time parallel module first divides the 120-hour infrared cloud image sequence into 240 30-minute windows (overlap rate 50%), the space parallel module divides the monitoring area into 1024 10km x 10km grid blocks through a quadtree index, and the element parallel module simultaneously starts 3 pipelines: pipeline A calculates the wind speed field using the WRF model, pipeline B uses the RTTOV algorithm to invert sea temperature, and pipeline C uses the GRAPES model to predict precipitation. All sub-tasks are completed within 23 seconds, and the final synthesized three-dimensional field data shows a spatial consistency of 0.91 through Spearman correlation coefficient verification.

[0047] The heterogeneous computing architecture maximizes hardware resource utilization, significantly reducing the time consumption of high-resolution meteorological simulation; the combination of sliding window and quadtree effectively balances calculation accuracy and efficiency; the dedicated pipeline design avoids calculation interference between elements, improving the physical reasonableness of the results; and the modular architecture facilitates the expansion of processing capabilities for new meteorological elements, providing a technical foundation for intelligent grid forecasting.

[0048] In some embodiments, the blockchain storage network comprises a memory cache layer, a full flash layer, and an edge node layer; the memory cache layer stores real-time data access records; the full flash layer retains intermediate results of task processing; and the edge node layer stores hot area data copies in a distributed manner.

[0049] The memory cache layer can refer to a high-speed temporary storage area, such as a 32 GB DDR5 cache space managed by an LRU-K algorithm, which stores real-time wind field analysis results at a response speed of nanoseconds. The real-time data access records can refer to a dynamic metadata set used to record 1287 retrieval behaviors of radar-based data by each computing node within the last one hour, which can optimize data prefetching strategies.

[0050] The full flash layer can refer to a persistent storage area, such as a NVMe SSD array configured to form a RAID5 disk group, which is used to retain intermediate calculation results of numerical prediction models for up to 30 days. The task processing intermediate results can refer to stage outputs, such as three-dimensional pressure field data output by each time step of WRF mode running, which retains a precision up to FP32 standard.

[0051] The edge node layer can refer to a near-end data warehouse, such as the deployment of 100 miniature data centers in coastal provinces, each node of which can store hot cyclone historical path data within a radius of 200 km. The hot area data copies can refer to high-frequency access data sets, such as 10 m resolution wind field data in the eye of a typhoon, which are synchronously stored in five edge nodes such as Xiamen and Zhanjiang, thereby controlling the access delay within 50 ms.

[0052] As a specific example: When the system processes "Haiyan" typhoon data, the memory cache layer first temporarily stores the latest 18-hour ECMWF analysis field (temperature, humidity, wind field three-dimensional matrix), the full flash layer synchronously saves the WRF mode integration results every 6 hours (totaling 2.3 TB), and the edge node layer pre-stores similar path typhoon case data for nearly 5 years in Hainan and Guangdong nodes. When the Zhanjiang Meteorological Bureau requests typhoon intensity data, the system retrieves the data from the nearest Yangjiang edge node (straight-line distance of 198 km) through intelligent routing, and the overall response time is shortened by 82% compared to traditional centralized storage. All data transmission processes are stored in a blockchain through SHA-3 algorithm.

[0053] By constructing a hierarchical storage system to manage the data lifecycle, the system load in high-concurrency access scenarios is significantly reduced; the edge computing architecture effectively reduces data transmission delay and improves the timeliness of meteorological warning; the blockchain technology ensures data integrity and traceability, providing technical evidence for meteorological legal disputes; and the intelligent prefetching mechanism optimizes storage distribution based on access patterns, continuously improving resource utilization.

[0054] In some embodiments, the heterogeneous resolution cluster further comprises a data quality enhancement module that identifies abnormal observations by an isolation forest algorithm, fills in data gaps using a spatial interpolation model, and corrects contradictory data by applying physical constraint conditions.

[0055] The isolation forest algorithm can refer to an anomaly detection model, such as a forest structure composed of 100 decision trees, to identify abnormal wind speed observations deviating from the main distribution by 3σ in a way that calculates the path length of data points. The spatial interpolation model can refer to a gap filling algorithm, such as an improved Kriging interpolation method, which combines 30m resolution DEM terrain data to perform spatial weighted estimation on missing precipitation observation station data. The physical constraint condition can refer to a meteorological rule library, such as setting the fluid continuity equation constraint, which triggers the data correction process when the pressure value of a certain grid point deviates from the calculated value of the surrounding 8 grid points by more than 15 hPa. The abnormal observation can refer to the outlier data point, which is used to mark the temperature record whose path length is shorter than the threshold value 0.65 determined by the isolation forest algorithm. These data will be sent to the review pipeline. The data gap can refer to the missing record, such as the interruption of sea surface temperature data caused by damaged buoy stations during a typhoon. The system can generate a replacement sequence based on the adjacent 12-hour valid observation values. The contradictory data can refer to the physically conflicting values, for example, when the ground station measures the wind speed as 18 m / s while the radar inversion at the same period is only 9 m / s, the system automatically selects the data source with higher reliability according to the boundary layer height model.

[0056] As a specific example: When the system processes the observation data of Typhoon No. 9 in 2025 in the East China Sea, the isolation forest algorithm first marks the abnormal wind speed value of 32.7 m / s at a certain buoy station (the average value of the surrounding stations is 21.4 m / s ± 3.2 m / s). The spatial interpolation model then generates a replacement value of 23.1 m / s based on the data of the surrounding 7 stations. The physical constraint module detects that the sea level pressure does not satisfy the static equilibrium condition with the 500 hPa height field at a certain time, and automatically calls the WRF model background field for data assimilation correction. The final output data set is verified to have an internal consistency index improved from the original 0.72 to 0.91, and the physical relationship between the elements meets the error allowed range of the fluid mechanics equation set.

[0057] Building an intelligent data cleaning pipeline significantly improves the usability of raw data. The multi-algorithm fusion mechanism effectively overcomes the limitations of single detection method. The physical rule constraint ensures that the correction result conforms to the basic law of atmospheric motion. The dynamic threshold adjustment strategy adapts to the changes in data characteristics of different weather systems, and finally provides more reliable initial field data for numerical prediction models.

[0058] In some embodiments, the system generates dimensionless meteorological feature indexes when performing meteorological element analysis, which are used to quantitatively describe the strength of atmospheric dynamic process and the degree of conformity to physical laws. The first calculation formula used for specific calculation includes:

[0059] wherein R is a dimensionless weather feature index, represents the pressure gradient weight coefficient of the i-th grid point, which is corrected by the digital elevation model; represents the terrain-corrected pressure gradient of the i-th grid point; represents the temperature field adjustment factor of the j-th time step, which is derived from the calculation of the overlapping area of the sliding window; represents the temperature field gradient of the j-th time step; represents the physical constraint coefficient of the k-th weather element, which is extracted from the knowledge graph; represents the normalized observation value of the k-th element; η represents the element correlation index, which is obtained by training historical data.

[0060] The dimensionless weather feature index can refer to a quantitative index that characterizes the degree of conformity of atmospheric dynamic processes and physical laws, such as by comprehensively calculating pressure gradient, temperature field gradient, and other elements, to evaluate the physical rationality of the state of the weather system.

[0061] The pressure gradient weight coefficient can refer to a correction parameter that reflects the influence of terrain on the pressure gradient, such as based on digital elevation model data, which can be obtained by spatial interpolation calculation of grid point elevation data, to eliminate the interference of terrain undulations on the calculation of pressure gradient. can refer to the terrain-corrected pressure gradient value, which can accurately reflect the actual pressure spatial distribution characteristics by using finite difference method combined with terrain height field data. The temperature field adjustment factor can refer to a dynamic correction parameter in the sliding window calculation, such as by sliding average processing of time series data, to eliminate the instantaneous abnormal fluctuations of temperature field observation data. The temperature field spatial gradient is a vector, such as using central difference method to calculate the temperature change rate of the grid point, which can reflect the spatial non-uniformity characteristics of the thermal field. The physical constraint coefficient can refer to the quantitative parameter of the physical law of the weather element, which is stored in the atmospheric physical equation relationship in the knowledge graph, to ensure that the calculation result conforms to the basic physical law. The standardized observation value can refer to the normalized weather observation value, such as using Z-score method to normalize the original observation data, which can make different dimensional elements comparable. The element correlation index can refer to a parameter that reflects the correlation strength between weather elements, which is obtained by training the correlation coefficient matrix of historical data, to correct the weight distribution in the joint calculation of multiple elements.

[0062] The quantification of atmospheric movement state is realized by constructing dimensionless weather feature index, the early identification rate of extreme weather event is improved by effectively fusing multi-source observation data and physical constraint conditions, the dynamic weight adjustment mechanism is adopted to overcome the dependence of traditional method on single element, the calculation result is more physically consistent, the parameter optimization driven by knowledge graph makes the system have continuous learning ability, and higher accuracy and stability are shown in typhoon path prediction and strong convective weather warning scenes, and reliable technical support is provided for disaster prevention and mitigation decision-making.

[0063] In some embodiments, the second calculation formula for calculating the temperature field adjustment factor includes:

[0064] wherein: represents the spatial weight of the mth adjacent station, determined by the quadtree index; is the observation time series of the mth station; is a time decay index, dynamically adjusted according to the data update frequency; is the load balancing factor of the nth parallel pipeline; represents the timeliness coefficient of the nth element; is the time window size.

[0065] The temperature field adjustment factor can refer to a dynamic correction parameter, for example, generated by a spatio-temporal weighting algorithm, used to balance the spatio-temporal unevenness of temperature observation data. The spatial weight can refer to the influence coefficient of adjacent stations, such as the inverse distance weight between stations calculated based on the quadtree spatial index, which can accurately reflect the correlation strength of observation data at different geographical locations. The observation time series can refer to the time series observation value, which uses the sliding window technology to store the latest 24 hours of temperature data, which can capture the time sequence change characteristics of meteorological elements. The time decay index can refer to the data timeliness parameter, for example, dynamically adjusted in the range of 0.5-1.5 according to the data update frequency, used to reduce the influence weight of obsolete observation data. The load balancing factor can refer to the calculation resource allocation coefficient, which is adjusted in real time by monitoring the CPU utilization of each parallel pipeline, which can optimize the overall efficiency of the distributed computing system. The timeliness coefficient can refer to the element degradation rate, such as setting the wind speed element θ=0.8 and the humidity θ=0.5, which is used to reflect the observation effective period difference of different meteorological elements. The time window can refer to the calculation period parameter, for example, set as a basic unit of 15 minutes and flexibly stretched according to the intensity of the weather system, which can balance the calculation accuracy and system load.

[0066] The dynamic weight mechanism significantly improves the spatio-temporal accuracy of temperature field analysis, the quadtree index optimization ensures the efficiency of spatial weight calculation, the adaptive adjustment of time effectiveness parameters effectively captures the characteristics of meteorological element changes, and the load balancing design ensures the stability of the system in high concurrency scenarios. Overall, it provides more reliable temperature field initial data for numerical weather prediction, especially in rapidly changing scenarios such as severe convective weather.

[0067] In some embodiments, the dynamic workflow engine includes a quality monitoring unit that monitors data parsing delay rate, storage throughput, and network jitter parameters in real time, and triggers resource redistribution mechanisms when any of the indicators exceeds the threshold.

[0068] The quality monitoring unit can refer to a performance-aware module, such as deploying Prometheus+Grafana monitoring stack, which is used to collect 15 system health indicators in real time and generate visual dashboards. The data parsing delay rate can refer to the processing timeliness indicator, such as calculating the percentage of time spent from data access to completion of parsing, and triggering an alarm when the average delay exceeds the preset threshold of 500ms within 10 minutes. The storage throughput can refer to the IO performance parameter, such as measuring the read and write speed of the SSD cluster, and the system can automatically identify abnormal nodes that continuously fall below 800MB / s and perform task migration. The network jitter parameter can refer to the transmission stability indicator, such as calculating the TCP retransmission rate per minute, and starting redundant transmission channels when detecting abnormal fluctuations exceeding 0.3%. The resource redistribution mechanism can refer to the elastic scheduling strategy, such as the HPA function based on Kubernetes, which can complete the horizontal expansion of computing nodes from 8 to 16 within 30 seconds.

[0069] As a specific example: when the system processes radar-based data for Typhoon "White Deer", the quality monitoring unit detects that the parsing delay rate has increased to 780ms (threshold 500ms), the storage throughput has decreased to 620MB / s (threshold 800MB / s), and the network jitter of the East China node has reached 0.45% (threshold 0.3%), the engine executes a three-level response strategy: 1) migrate the original 12 parsing containers of the East China node to the North China backup cluster 2) replace the faulty SSD node with a pre-warmed NVMe cache pool 3) switch to a multi-path transmission protocol, the entire process is completed within 2 minutes and 15 seconds, during which the data processing success rate remains above 99.2%, and the blockchain network records the timestamps and digital signatures of all resource change events.

[0070] By establishing an intelligent system health monitoring system, the stability of the service under extreme weather conditions is significantly improved; the multi-index linkage analysis mechanism effectively identifies complex fault scenarios; the hierarchical response strategy realizes the optimization of resource utilization; the automated failover capability ensures business continuity, providing reliable technical support for meteorological disaster emergency response.

[0071] In some embodiments, the edge node layer implements a data pre-warming strategy, which loads target area data in advance according to an access frequency prediction model that takes into account seasonal factors, extreme weather event probabilities, and user access spatiotemporal distribution characteristics.

[0072] The data pre-warming strategy can refer to an active caching mechanism, such as using an LSTM neural network to predict data demand for the next 2 hours, for pre-loading 10m resolution wind field data in the eye of a typhoon to southeast coastal nodes. The access frequency prediction model can refer to an intelligent prediction algorithm, such as building an XGBoost multi-feature fusion model that can predict the data request frequency of each regional meteorological station within the next 30 minutes with 85% accuracy. The seasonal factor can refer to a periodic influence parameter, such as setting the weight coefficient of the typhoon season (July-September) to 0.6 and the dry season (October-December) to 0.3, which can dynamically adjust the cold and hot data determination threshold. The extreme weather event probability can refer to a disaster warning parameter, such as integrating the red warning signals issued by the Central Meteorological Observatory, and automatically increasing the number of related regional data copies from 3 to 5 when the probability exceeds 70%. The user access spatiotemporal distribution characteristics can refer to behavior analysis data, such as statistical ECMWF data request heat maps of provincial meteorological bureaus during 08:00-10:00, which are used to optimize the data distribution topology of edge nodes.

[0073] As a specific example: when the system detects that the sea surface temperature in a certain region is 2.3°C higher than the average annual temperature and the monsoon index reaches 1.5, the prediction model starts pre-warming by considering the following factors: 1) seasonal factor (current August weight 0.58), 2) typhoon generation probability (ECMWF prediction within 72 hours reaches 67%), 3) historical access rules of Guangdong Provincial Meteorological Bureau (09:15 daily concentrated wind field data retrieval), and then pre-loads 10TB of satellite cloud images and reanalysis data in Shenzhen and Zhuhai edge nodes. When the actual request arrives at 09:08 the next day, the data hit rate reaches 92.7%, and the response delay is only 38ms, which is 4.6 times more efficient than the non-pre-warming scenario.

[0074] Through multi-dimensional prediction, the storage resources are accurately pre-allocated, significantly reducing the delay peak value during high concurrency access; the spatiotemporal feature fusion algorithm improves the service reliability in extreme weather scenarios; the dynamic weight mechanism adapts to the fluctuation of business demand in different seasons; the intelligent collaboration between edge nodes reduces the cross-regional data transmission cost, and overall improves the response agility and resource utilization rate of meteorological services.

[0075] The preferred embodiments of the present specification disclosed above are only used to help illustrate the present specification. Alternative embodiments do not describe all the details and do not limit the present application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the present application. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and utilize the present specification. The present specification is limited only by the claims and their full scope and equivalents.

Claims

1. A meteorological data processing system based on a parallel cluster architecture, characterized by, Intelligent perception client, cognitive service module, heterogeneous analysis cluster and blockchain storage network are included. The intelligent perception client receives multi-modal input instructions through an AR visual interactive interface and generates a three-dimensional spatio-temporal cube interactive model. The cognitive service module predicts resource demand according to historical load data and dynamically arranges micro-task flow. The heterogeneous analysis cluster adopts a three-dimensional parallel processing mechanism to enhance the quality of meteorological data. The blockchain storage network realizes data version control through a five-level storage system. The spatio-temporal constraint conditions output by the intelligent perception client trigger the cognitive service module to start the dynamic workflow engine, and the data packets processed by the heterogeneous analysis cluster are synchronized to the edge node after being attached with digital fingerprints by the blockchain storage network.

2. The system of claim 1, wherein, The intelligent perception client includes a spatial dimension selection module, a time dimension adjustment module and an element dimension recommendation module; the spatial dimension selection module realizes atmospheric layer selection through a digital twin earth model; the time dimension adjustment module adopts a scalable time axis for multi-scale selection; and the element dimension recommendation module automatically associates meteorological element combinations based on a knowledge graph.

3. The system of claim 1, wherein, The cognitive service module includes a demand understanding unit, a resource planning unit and a service arrangement unit; the demand understanding unit analyzes user semantics through natural language processing; and the resource planning unit dynamically adjusts computing nodes according to QoS indicators. The service arrangement unit decomposes complex requests into parallel executable micro-task flow.

4. The system of claim 1, wherein, The heterogeneous analysis cluster includes a time parallel module, a spatial parallel module and an element parallel module; the time parallel module processes ultra-long time series data using a sliding window mechanism; the spatial parallel module realizes regional segmentation based on a quadtree index; and the element parallel module establishes a dedicated analysis pipeline for different meteorological elements.

5. The system of claim 1, wherein, The blockchain storage network includes a memory cache layer, a full flash memory layer and an edge node layer; the memory cache layer stores real-time data access records; the full flash memory layer retains intermediate results of task processing; and the edge node layer stores hot area data copies in a distributed manner.

6. The system of claim 4, wherein, The heterogeneous analysis cluster further includes a data quality enhancement module, which identifies abnormal observation values through an isolation forest algorithm, fills in data gaps using a spatial interpolation model, and corrects contradictory data by applying physical constraint conditions.

7. The system of claim 1, wherein, When the system performs meteorological element analysis, it generates dimensionless meteorological feature indexes to quantitatively describe the strength of atmospheric dynamic processes and the degree of compliance with physical laws, wherein a first calculation formula for specific calculation includes: where R is a dimensionless weather feature index, represents the pressure gradient weight coefficient of the i-th grid point, which is corrected by the digital elevation model; is the terrain-corrected pressure gradient of the i-th grid point; is the temperature field adjustment factor of the j-th time step, which is calculated from the overlapping area of the sliding window; represents the temperature field gradient of the j-th time step; is the physical constraint coefficient of the k-th meteorological element, which is extracted from the knowledge graph; represents the normalized observation value of the k-th element; η is the element correlation index, which is obtained by training historical data.

8. The system of claim 7, wherein, A second calculation formula for calculating the temperature field adjustment factor includes: wherein: represents the spatial weight of the mth neighboring site, determined by the quadtree index; is the observation time series of the mth site; is the time decay exponent, dynamically adjusted according to the data update frequency; is the load balancing factor of the nth parallel pipeline; represents the timeliness coefficient of the nth element; is the time window size.

9. The system of claim 1, wherein, The dynamic workflow engine includes a quality monitoring unit that monitors data analysis delay rate, storage throughput and network jitter parameters in real time, and triggers resource redistribution mechanism when any indicator exceeds the threshold.

10. The system of claim 5, wherein, The edge node layer implements a data preheating strategy, which loads target area data in advance according to an access frequency prediction model that considers seasonal factors, extreme weather event probabilities and user spatio-temporal distribution characteristics.

Images