Meteorological metadata storage method and storage system based on machine learning

Abstract

The present invention provides a meteorological metadata storage method and storage system based on machine learning. The method comprises: obtaining a meteorological metadata dataset for a target area; inputting the meteorological metadata dataset into a pre-trained temporal feature extraction model to generate temporal distribution features for each meteorological file; and inputting the temporal distribution features into a spatial topology mapping model to generate spatial correlation topology features for each meteorological element; constructing a hierarchical index structure for meteorological metadata based on the cross-fusion results of the temporal distribution features and the spatial correlation topology features; invoking a meteorological storage optimization model to generate a storage node allocation strategy for meteorological file; writing each meteorological file in the meteorological metadata set to a corresponding distributed storage node according to the storage node allocation strategy; and dynamically adjusting the physical location and index mapping relationship of the stored files based on a real-time update mechanism of the hierarchical index. The present invention can improve meteorological metadata storage efficiency and retrieval accuracy, and reduce storage resource waste and operation and maintenance costs.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a meteorological metadata storage method and storage system based on machine learning. Background Art

[0002] In the current meteorological data storage system, meteorological monitoring agencies manage massive unstructured meteorological files through distributed storage nodes, and generally adopt indexing technology based on spatiotemporal dimensions to improve data retrieval efficiency. In related technologies, fixed rules are usually used to divide the storage hierarchy structure, such as pre-setting vertical classification indexes by meteorological element types or dividing static spatial indexes by geographic grids. Although such methods can achieve basic data organization, they are difficult to dynamically adapt to the spatiotemporal evolution characteristics of meteorological data: on the one hand, the periodic fluctuations of meteorological elements and sudden events cause data access hotspots to change dramatically over time, and the static index structure cannot dynamically adjust the storage priority according to the time series characteristics, causing high-frequency access data to be retained in low-performance storage nodes, and retrieval delays to increase significantly; on the other hand, the spatial correlation between meteorological sites and cross-regional data dependencies have not been effectively modeled, resulting in a mismatch between the spatial index density distribution and actual business needs, serious waste of storage resources and insufficient disaster recovery capabilities. Summary of the Invention

[0003] In view of this, an embodiment of the present invention provides a meteorological metadata storage method and storage system based on machine learning. The technical solution of the embodiment of the present invention is implemented as follows:

[0004] In one aspect, an embodiment of the present invention provides a meteorological metadata storage method based on machine learning. The method comprises: obtaining a meteorological metadata dataset for a target area, the meteorological metadata dataset comprising a plurality of unstructured meteorological file files, each meteorological file file containing monitoring data of at least one meteorological element and corresponding spatiotemporal identification information; inputting the meteorological metadata dataset into a pre-trained temporal feature extraction model to generate temporal distribution features for each meteorological file file, and simultaneously inputting the meteorological metadata dataset into a spatial topology mapping model to generate spatial correlation topological features for each meteorological element; constructing a hierarchical index structure for meteorological metadata based on the cross-fusion results of the temporal distribution features and the spatial correlation topological features, the hierarchical index structure comprising a vertical hierarchy divided by meteorological element type and a horizontal hierarchy divided by spatiotemporal dimensions; invoking a meteorological storage optimization model to generate a storage node allocation strategy for meteorological file files based on the hierarchical weight distribution of the hierarchical index structure, the storage node allocation strategy comprising a compression rate threshold and a storage location priority for each meteorological element data; writing each meteorological file file in the meteorological metadata set to a corresponding distributed storage node according to the storage node allocation strategy, and dynamically adjusting the physical location and index mapping relationship of the stored files based on a real-time update mechanism of the hierarchical index.

[0005] On the other hand, an embodiment of the present invention provides a storage system including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps in the above method when executing the program.

[0006] The meteorological metadata storage method based on machine learning provided by the present invention constructs a dynamic hierarchical index structure by cross-analyzing the integration of temporal distribution features and spatial correlation topological features, and uses the meteorological storage optimization model to generate an adaptive storage strategy, thereby solving the core problems of spatiotemporal correlation separation and rigid storage resource allocation in meteorological data storage: the temporal distribution features provide dynamic weight factors for the index hierarchy by capturing the periodic fluctuations and mutation events of meteorological elements, so that high-frequency access data is preferentially allocated to high-performance storage nodes, while the spatial correlation topological features generate a spatial density grid by depicting the spatial conduction relationship between meteorological sites to optimize the geographical distribution density of data. The cross-fusion mechanism of the two converts the nonlinear correlation of spatiotemporal features into the hierarchical weight distribution of the hierarchical index, thereby realizing dynamic adaptation of the storage structure to the meteorological business scenario; at the same time, the hierarchical index structure enables meteorological file files to be quickly located based on compound index tags through bidirectional mapping between the vertical hierarchy (meteorological element type) and the horizontal hierarchy (space-time grid), and the meteorological storage optimization model locates meteorological files based on the hierarchical weight distribution and the node Point load fluctuation data generates compression rate thresholds, redundant copy strategies and storage location priority queues to ensure that high-frequency data is stored preferentially in high-bandwidth nodes and low-frequency data is stored with a high compression rate. The dynamic adjustment mechanism avoids storage node overload and ensures the retrievability of historical data by migrating low-temperature data in real time and updating index mapping relationships. In addition, in view of the strong timeliness of meteorological data, the storage priority of expired data is automatically reduced through the dynamic weight factor decay mechanism to free up storage space. Combined with the core storage area division of the spatial density grid, the storage redundancy of data in disaster-prone areas is prioritized to improve disaster recovery capabilities. At the same time, the model-driven adaptive capability dynamically adjusts the compression algorithm and storage location allocation strategy to adapt to the differentiated needs of meteorological data centers of different sizes. Ultimately, through the technical collaboration of spatiotemporal feature fusion, dynamic index construction, model strategy generation and closed-loop feedback mechanism, the storage efficiency and retrieval accuracy of meteorological metadata are significantly improved, the waste of storage resources and operation and maintenance costs are reduced, and high-reliability and high-adaptability data management support is provided for meteorological services. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A schematic diagram of the implementation flow of a meteorological metadata storage method based on machine learning provided in an embodiment of the present invention.

[0008] Figure 2 A hardware entity diagram of a storage system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0009] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention are further elaborated in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0010] An embodiment of the present invention provides a method for storing meteorological metadata based on machine learning, which can be executed by a processor of a storage system, wherein the storage system can refer to a device with data processing capabilities, such as a server or desktop computer.

[0011] Figure 1 A schematic diagram of the implementation flow of a meteorological metadata storage method based on machine learning provided in an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes the following steps:

[0012] Step S100: Acquire a meteorological metadata set of a target area. The meteorological metadata set includes a plurality of unstructured meteorological file files. Each meteorological file file contains monitoring data of at least one meteorological element and corresponding spatiotemporal identification information.

[0013] The process of obtaining a meteorological metadata set for a target area involves collecting a collection of raw meteorological data from a specific geographic area. The meteorological metadata set consists of multiple unstructured meteorological files. Unstructured meteorological files are, for example, raw data files not organized according to a unified format or standard. Their content may include text, numerical values, charts, and other formats. Each meteorological file contains monitoring data for at least one meteorological element. Meteorological elements are, for example, fundamental physical quantities describing atmospheric conditions, such as temperature, humidity, wind speed, and precipitation. Monitoring data refers to real-time or historical measurements collected by sensors, satellites, or ground-based observation equipment. Spatiotemporal identification information, for example, includes the time and space tags associated with meteorological element monitoring data. The time tag includes the specific time point or time period of data collection (e.g., October 1, 2023, 12:00 to 18:00). The spatial tag includes the geographic coordinates of the data collection location (e.g., longitude 116.40°E, latitude 39.90°N) or the regional code (e.g., meteorological zone code A01 for City B). Specifically, meteorological metadata datasets can be acquired through interfaces with data sources such as meteorological observation stations, remote sensing satellites, or weather radar. Unstructured meteorological files are stored in various formats, such as CSV, JSON, and NetCDF. For example, a meteorological file might contain temperature monitoring data for City B during the summer of 2023, with its spatiotemporal identifiers labeled as "June 1 to August 31, 2023" and "the entire geographic area of ​​City B." During this process, the integrity and originality of the meteorological metadata dataset must be ensured to ensure accurate spatiotemporal feature extraction in subsequent processing stages.

[0014] As an embodiment, after obtaining the meteorological metadata set of the target area in step S100, a process of performing data preprocessing on the meteorological metadata set may be further included, wherein the data preprocessing process may include:

[0015] Step S110: identifying the timestamp intervals of missing monitoring data in the meteorological file, and generating interpolated data based on the fluctuation trend of monitoring data of adjacent meteorological stations;

[0016] Identifying the timestamp intervals where monitoring data is missing in a meteorological file is, for example, by analyzing the time series of monitoring data recorded in the meteorological file and locating the continuous time periods in which no valid data was collected. The timestamp interval is defined by a starting time point and an ending time point. For example, a certain meteorological file does not record any humidity monitoring data in the time period from 14:00 to 16:00 on July 10, 2023. Adjacent meteorological stations are, for example, other stations in the target area that are geographically adjacent to the current meteorological station and whose meteorological element monitoring data have spatial continuity. For example, the Chaoyang District Meteorological Station and the Haidian District Meteorological Station in City B are adjacent stations to each other. The fluctuation trend of monitoring data is, for example, the changing pattern of meteorological element data at adjacent stations in the same time period. For example, the temperature data of the Chaoyang District Meteorological Station from 14:00 to 16:00 on July 10, 2023 showed a trend of rising by 1°C per hour, while the temperature data of the Haidian District Meteorological Station during the same period showed a trend of rising by 0.8°C per hour. The process of generating interpolated data based on this fluctuation trend involves, for example, using linear regression, time series forecasting, or spatial kriging interpolation algorithms to infer the meteorological element values ​​for the current station within the missing timestamp interval. For example, if the Chaoyang District weather station is missing temperature data from 2:00 PM to 4:00 PM, while the Haidian District data for the same period shows a linear increase, interpolated data can be generated based on the historical correlation between the two stations (e.g., a temperature change slope ratio of 1:0.8). The generation of interpolated data must ensure spatial continuity constraints with adjacent stations to avoid interpolation results deviating from the actual observation range due to local terrain or microclimate differences.

[0017] Step S120: detecting abnormal jump points in the meteorological element monitoring data, and using a sliding window algorithm to smooth and correct the data sequence within the window;

[0018] Detecting abnormal transition points in meteorological monitoring data involves, for example, using statistical analysis methods to identify sudden changes in the data sequence that do not conform to meteorological laws or the normal measurement range of the equipment. An abnormal transition point manifests as a dramatic fluctuation in the value of a meteorological element within a short period of time, such as a sudden increase in wind speed from 3 m / s to 20 m / s within 10 minutes, or a sudden increase in precipitation from 0 mm to 50 mm within five consecutive time points (collected every 10 minutes). A sliding window algorithm, for example, divides the data sequence into continuous subsequences (windows) of fixed length and performs statistical analysis on the data within each window. For example, for a temperature data series, a window length of 24 hours is set, with a sliding step of 1 hour. The standard deviation and mean of the temperature are calculated within each window. If a data point deviates from the mean by more than three standard deviations, it is considered an abnormal transition point. Smoothing correction, for example, involves replacing or adjusting the values ​​of abnormal transition points to conform to the natural variation of meteorological elements using filtering algorithms (such as moving average, median filtering, or exponential smoothing). For example, for the above-mentioned wind speed anomaly, the sliding window algorithm can replace the anomaly with the average of the data within the window for one hour before and after the jump point is detected, thereby eliminating the noise caused by equipment failure or transient interference.

[0019] Step S130: Convert meteorological file files of different formats into a unified binary encoding format, and add metadata tags containing spatiotemporal identification information to the file header; wherein the preprocessed meteorological metadata set meets the input format specifications of the temporal feature extraction model and the spatial topology mapping model.

[0020] Meteorological archive files of various formats, such as raw files from heterogeneous data sources, may include text formats (e.g., CSV, TXT) and scientific data formats (e.g., NetCDF, HDF5). A unified binary encoding format, for example, converts all files into a fixed-byte-length binary stream, using encoding rules defined by Protocol Buffers or Apache Avro. The numeric type (e.g., floating-point, integer), byte order, and data length of each field are strictly standardized. For example, the file header is a specific byte area at the beginning of a binary file that stores metadata tags describing the file's contents. Metadata tags for spatiotemporal identification information include the geographic range of data collection (e.g., longitude 116.20°E to 116.60°E, latitude 39.80°N to 40.20°N), the time interval (e.g., August 1, 2023, 00:00 to August 31, 2023, 23:59), and the meteorological element type code (e.g., temperature code TEMP_01, humidity code HUMI_02). For example, after conversion, the first 128 bytes of a binary-encoded header of a meteorological file in raw CSV format record the metadata tag "City B Entire Area - August 2023 - Temperature Data," and the subsequent bytes store hourly temperature values ​​as floating-point numbers. The preprocessed meteorological metadata dataset must meet the input format specifications of the time series feature extraction model and the spatial topology mapping model, including data dimension alignment (such as sorting timestamps by UTC time), numerical range normalization (such as uniformly converting temperatures to degrees Celsius), and missing value placeholders (such as NaN filling). This standardized format ensures that subsequent models can efficiently parse the data, avoiding parsing errors or feature extraction biases caused by format differences.

[0021] Step S200: Input the meteorological metadata dataset into the pre-trained temporal feature extraction model to generate temporal distribution features of each meteorological file, and synchronously input the meteorological metadata dataset into the spatial topology mapping model to generate spatial correlation topology features of each meteorological element.

[0022] A pre-trained time series feature extraction model is a machine learning model trained on historical meteorological data, such as a recurrent neural network. Its function is to extract representative patterns or regularities from time series data. When the meteorological metadata dataset is input into this model, it first parses the monitoring data and corresponding timestamps in each meteorological file to identify the temporal trends of meteorological elements. For example, temporal distribution features are vectors or matrices output by the model that characterize the temporal evolution of meteorological elements, such as periodic fluctuations, sudden events, or long-term trends. For example, for temperature monitoring data, temporal distribution features may include the amplitude of daily temperature changes, the seasonal rate of temperature rise, or the duration of extreme high temperature events. A spatial topological mapping model, which is synchronously fed with the meteorological metadata dataset, is another independently trained model that analyzes the spatial correlation of meteorological elements. Spatial correlation topological features are, for example, structures generated by the model that characterize the mutual influence of meteorological elements between different geographical locations, such as the data similarity between adjacent meteorological stations, wind propagation paths, or the movement of precipitation clouds. For example, for wind speed data, spatial correlation topological features may represent a network structure of wind direction transmission, where nodes represent meteorological stations and edge weights represent the correlation coefficient of wind speed changes between stations. During this process, the temporal feature extraction model and the spatial topology mapping model need to run in parallel to ensure the independence and complementarity of temporal and spatial features, thereby providing a multi-dimensional analysis basis for subsequent fusion.

[0023] As an implementation method, the training process of the above-mentioned time series feature extraction model may include:

[0024] Step S201: Collect continuous time series monitoring data of different meteorological elements in the historical meteorological data set as training samples. The time series monitoring data includes the numerical fluctuation sequence of the meteorological elements at a preset time granularity and the corresponding timestamp mark.

[0025] The process of collecting historical meteorological datasets, for example, involves filtering continuous time series data for multiple meteorological elements within a target area from long-term stored meteorological archives. These elements include independent physical quantities such as temperature, humidity, wind speed, and precipitation. Continuous time series monitoring data requires that data points are uninterrupted in time and arranged at a fixed time granularity, such as hourly, daily, or monthly. The preset time granularity is, for example, the minimum time unit predefined during the model training phase. For example, at hourly granularity, time series monitoring data consists of 24 data points per day, each corresponding to a measurement value at the hour. A numerical fluctuation series is, for example, a curve showing the numerical changes of meteorological elements at a specific time granularity. For example, the daily average temperature series for a region from January 1, 2020, to December 31, 2023, contains 1,461 data points. The timestamp indicates the precise time associated with each data point, such as "2023-07-15 14:00:00 UTC." Training samples should encompass data from different seasons, climate events (such as El Niño), and geographic regions to ensure the model can learn the multi-scale variations of meteorological elements. For example, a temperature training sample could include hourly temperature monitoring data from City B over the past decade, with timestamps covering all valid collection moments. The numerical fluctuation sequence should reflect temperature fluctuations at hourly, daily, monthly, and annual scales.

[0026] Step S202: Input the time series monitoring data into the forward propagation path of the bidirectional recurrent neural network to extract the long-term trend characteristics of the meteorological elements. The long-term trend characteristics represent the periodic change patterns of the meteorological elements on the seasonal scale or the interannual scale.

[0027] The forward propagation path of a bidirectional recurrent neural network, for example, is the computational path that processes input data in chronological order (from past to future). Time series monitoring data is arranged in chronological order by timestamp and fed into the recurrent units (such as LSTM or GRU) in the forward propagation path, where the hidden state is updated time-step by time step. Long-term trend features refer to macroscopic patterns of change that the model extracts from the input sequence, such as a slow upward trend in temperature on an interannual scale (e.g., the increasing average annual temperature due to global warming) or cyclical fluctuations on a seasonal scale (e.g., high temperatures in summer and low temperatures in winter). For example, for ten years of temperature data from City B, the forward propagation path can capture the long-term trend of an average temperature increase of 0.1°C each July compared to the previous year, as well as the periodic peak temperatures from June to August within each year. Extracting long-term trend features relies on the memory capacity of the recurrent neural network. By retaining the cumulative effect of historical information, it can ignore short-term noise and enhance cross-year or cross-seasonal correlations.

[0028] Step S203: Input the time series monitoring data into the back propagation path of the bidirectional recurrent neural network to extract the short-term fluctuation characteristics of the meteorological elements. The short-term fluctuation characteristics represent the correlation of sudden changes in meteorological elements at the hourly or daily scale.

[0029] The backpropagation path of a bidirectional recurrent neural network, for example, is the computational path by which the model processes input data in reverse chronological order (from the future to the past). After the time series monitoring data is fed back into the recurrent unit, the model iteratively computes the hidden state from the end of the sequence back to the beginning. Short-term fluctuation features capture the rapid changes in meteorological elements within a fine-grained time window, such as a sudden drop in temperature within a few hours (e.g., a cold front passing through) or a sudden increase in wind speed on a daily scale (e.g., an approaching typhoon). For example, for temperature data from the same city B, the backpropagation path can identify a short-term event, such as a sudden drop of 5°C between 2:00 PM and 4:00 PM on a certain day in July 2023, and analyze its correlation with the daily change in air pressure. Extraction of short-term fluctuation features focuses on data abrupt changes in a local time window. The sensitivity of the backpropagation path enhances the model's responsiveness to recent events, thereby complementing the long-term trend analysis of the forward path.

[0030] Step S204: Perform multi-scale time window fusion on the long-term trend characteristics and the short-term fluctuation characteristics to generate a fused time series feature vector. The first dimension of the fused time series feature vector corresponds to the intensity coefficient of the periodic change law, and the second dimension corresponds to the confidence score of the correlation of the mutation event.

[0031] Multi-scale time window fusion, for example, combines long-term trend features (such as interannual trends) with short-term fluctuation features (such as hourly sudden changes) through feature concatenation, weighted summation, or convolution. The fused time series feature vector is a fixed-dimensional numerical vector. Its first dimension quantifies the strength of the meteorological element's periodicity, such as the annual cycle amplitude coefficient extracted through Fourier transform. The second dimension assesses the statistical significance of sudden changes, such as the p-value calculated from a hypothesis test or the probability score of a model prediction. For example, after fusing the long-term trend and short-term fluctuations of city B's temperature data, the feature vector might be expressed as [0.85, 0.93], where 0.85 represents the strength of the annual periodicity (1.0 is the strongest) and 0.93 represents the confidence level in the association between the sudden temperature drop event on that day and similar historical events. The fusion process must preserve the independence of multi-scale features to avoid information aliasing, while normalization ensures that the numerical ranges of different dimensions are consistent.

[0032] Step S205: Introduce an adaptive attention allocation mechanism in the output layer of the neural network to dynamically adjust the feature weight distribution of meteorological elements in different time windows according to the product relationship between the intensity coefficient of the fused time series feature vector and the confidence score.

[0033] For example, the adaptive attention allocation mechanism uses learnable weight parameters to automatically allocate the contribution of features from different time windows to the final output. The attention weight is generated by multiplying the strength coefficient and the confidence score. For example, the product of a strength coefficient of 0.85 and a confidence score of 0.93 is 0.7905, indicating that the features in this time window have a higher weight. The dynamic adjustment process assigns weights based on this product. For example, in windows with strong interannual periodicity and high confidence in sudden events, the weight is increased to 0.9; in windows with weak periodicity or low confidence in sudden events, the weight is reduced to 0.3. For example, for city B's summer 2023 data, the model might assign a high weight to the sudden temperature drop on July 15th (due to the high intensity periodicity and high confidence in sudden events) while assigning a low weight to the stable autumn data. The attention mechanism adjusts the numerical distribution of feature vectors through scaling and translation operations, allowing the model to focus more on critical time windows in prediction or classification tasks.

[0034] Step S206: Optimize the network parameters through the supervised learning algorithm so that the time series distribution characteristics output by the time series feature extraction model can simultaneously reflect the periodic stability and mutation sensitivity of meteorological elements, and use the optimized time series distribution characteristics as the basis for generating the dynamic weight factor of the vertical level main index node in the hierarchical index structure.

[0035] Supervised learning algorithms use labeled training data, measure the difference between model output and true values ​​using a loss function (such as mean squared error or cross entropy), and adjust network parameters through backpropagation. Cyclic stability requires that the long-term trend characteristics of the model output are consistent with the cyclical patterns of historical data, for example, the prediction error of annual temperature fluctuations is less than 1%. Mutation sensitivity requires that the model detect short-term events with an accuracy rate exceeding 95%. The optimized time series distribution feature is a vector that combines cyclical and mutational information. For example, [0.85, 0.93] becomes [0.88, 0.95] after parameter adjustment, enhancing its ability to represent actual meteorological changes. In a hierarchical index structure, the dynamic weight factor of the primary index node in the vertical hierarchy (such as the temperature feature node) is generated by weighting the dimensions of the time series distribution feature, for example, 0.88 × 0.6 (cyclical weight factor) + 0.95 × 0.4 (mutation weight factor) = 0.904. This gives features with high cyclic stability and high mutation sensitivity higher storage and retrieval priority. The dynamic weight factor real-time update mechanism ensures that the index structure can adapt to changes in meteorological data. For example, the weight factor of the wind speed factor in the typhoon season increases significantly, triggering the priority adjustment of the storage node.

[0036] Step S300: Based on the cross-fusion results of temporal distribution features and spatial correlation topology features, a hierarchical index structure of meteorological metadata is constructed. The hierarchical index structure includes vertical levels divided by meteorological element types and horizontal levels divided by spatiotemporal dimensions.

[0037] The cross-integration of temporal distribution features and spatial correlation topology features combines temporal and spatial features through mathematical operations or model integration, such as feature concatenation, weighted overlay, or attention mechanisms. The goal of constructing a hierarchical index structure is to establish a multidimensional retrieval and storage framework for meteorological metadata. The vertical hierarchy is divided by meteorological element type, such as temperature, humidity, and wind speed. Each category serves as an independent primary index node, storing the element's temporal distribution features and associated spatial topology. The horizontal hierarchy is divided by spatiotemporal dimensions. The temporal dimension can be subdivided into hourly, daily, and monthly granularities, while the spatial dimension can be based on geographic grids (e.g., 1 km × 1 km cells) or administrative divisions (e.g., provinces, cities, and counties). For example, in the vertical hierarchy, the primary index node for the temperature element contains its periodic variation characteristics and its transmission relationship with surrounding areas. In the horizontal hierarchy, the temperature data for City B in June 2023 might be classified into the "North China - Summer - Core Grid" cell. The cross-fusion process must ensure dynamic linkage between the vertical and horizontal levels. For example, when a meteorological element exhibits a spatial anomaly within a specific time period, the weight of the corresponding vertical primary index node is automatically increased, and the storage priority of the relevant spatiotemporal grids in the horizontal level is simultaneously adjusted. The construction of a hierarchical index structure relies on feature fusion algorithms (such as tensor decomposition or graph neural networks) to achieve efficient mapping of cross-dimensional features and dynamic updating of index nodes.

[0038] As an implementation method, the above step S300, based on the cross-fusion result of the temporal distribution feature and the spatial correlation topology feature, constructs a hierarchical index structure of meteorological metadata, which may include:

[0039] Step S310: Decompose the time series distribution characteristics into periodic characteristics and mutation characteristics of meteorological elements, wherein the periodic characteristics represent the fluctuation pattern of meteorological elements that recurs in historical monitoring data, and the mutation characteristics represent the abnormal fluctuation range that exceeds the preset threshold.

[0040] The decomposition process for time series distribution features involves, for example, separating independent characteristic components of periodic and abrupt changes from the fused time series feature vector output by the model. Periodic features are extracted by analyzing recurring regular fluctuation patterns of meteorological elements in historical monitoring data, such as annual seasonal variations in temperature data (high temperatures in summer, low temperatures in winter) or monthly average distribution patterns in precipitation data (such as concentrated precipitation during the plum rain season). Sudden changes are identified by detecting abnormal fluctuation intervals in the numerical sequence of meteorological elements that exceed a preset threshold. The preset threshold can be dynamically set based on the statistical distribution of historical data (such as three standard deviations) or domain knowledge (such as typhoon wind speed thresholds). For example, for wind speed data, if the wind speed suddenly increases from 5 m / s to 25 m / s within a certain time period and exceeds a preset threshold of 20 m / s, this interval is marked as a sudden change feature interval. During the decomposition process, periodic features can be quantified by Fourier transform or autocorrelation analysis to quantify their fluctuation amplitude and frequency, while sudden changes are located using extreme value detection algorithms (such as peak recognition) within a sliding window combined with threshold comparison. The separation of periodic and sudden changes provides a basis for differentiated weighting in subsequent hierarchical divisions.

[0041] Step S320: Decompose the spatial correlation topological features into geographic correlation features and cross-regional conduction features, wherein the geographic correlation features represent the spatial continuity of the monitoring data of adjacent meteorological stations, and the cross-regional conduction features represent the correlation strength of meteorological elements between non-adjacent regions.

[0042] The decomposition of spatial correlation topological features aims to distinguish between local correlation and long-range transmission of meteorological elements in spatial dimensions. Geographic correlation features are generated by calculating spatial continuity indicators (such as the Pearson correlation coefficient or covariance matrix) of monitoring data from adjacent meteorological stations. For example, the temperature data from meteorological stations in Chaoyang District and Haidian District of City B show high synchronization in diurnal trends (correlation coefficient of 0.92), indicating strong geographic correlation between the two stations. Cross-regional transmission features are extracted by analyzing the statistical dependence of meteorological elements or physical transmission pathways (such as wind fields and ocean currents) between non-adjacent regions. For example, when a typhoon moves from Sea Area C toward the southeast coast, its wind speed and precipitation data show a lagged correlation (transmission strength of 0.75) between meteorological stations in the non-adjacent cities of Xiamen and Fuzhou. The decomposition process uses a graph neural network or spatial lag model to divide the edge weights in the spatial topological structure into local adjacent edges (geographic correlation features) and long-range connection edges (cross-regional transmission features). For example, in the wind speed transmission network, geographic correlation features correspond to edge weights between adjacent stations (0.92), while cross-regional transmission features correspond to edge weights between inter-provincial stations (0.68). The independent extraction of geographic correlation features and cross-regional transmission features provides a multi-scale spatial relationship basis for the division of spatial density grids.

[0043] Step S330: In the vertical hierarchical division, a main index node is created according to the meteorological element type, and a dynamic weight factor is assigned to each main index node. The dynamic weight factor is determined by the ratio of the periodic feature to the mutation feature, so that the main index node of the high-frequency mutation element obtains a higher vertical hierarchical priority.

[0044] Vertical hierarchical division constructs independent primary index nodes based on meteorological element types (such as temperature, humidity, and wind speed). Each node represents a data set for a specific meteorological element. A dynamic weight factor is calculated based on the ratio of periodicity to mutation. For example, if the strength of a wind speed element's periodicity is 0.6 (weak annual seasonal fluctuations) and its mutation strength is 0.9 (frequent sudden high wind events), the ratio is 0.9 / 0.6 = 1.5. This ratio reflects the dominance of a factor's mutation frequency over its cyclic stability; a higher ratio indicates a higher priority. Vertical hierarchical priority is ranked based on the ratio. For example, if the ratio for wind speed is 1.5 and the ratio for temperature is 0.8, the wind speed primary index node is assigned a higher position in the hierarchy, ensuring that high-frequency mutation data is prioritized during retrieval and storage. The calculation of the dynamic weight factor requires real-time data updates. For example, if the strength of the wind speed mutation strength increases to 1.2 during typhoon season, the ratio becomes 1.2 / 0.6 = 2.0, further consolidating its hierarchical priority. The creation and weight allocation of master index nodes are implemented through the database index management module, which supports the dynamic adjustment of node positions and the linkage of storage resource allocation strategies.

[0045] Step S340: In the horizontal hierarchical division, a spatial density grid is generated based on the superposition results of geographic correlation features and cross-regional conduction features. Each grid cell records the statistical distribution characteristics of meteorological elements within the corresponding geographic range, and the grid cells are divided into core storage areas and edge cache areas according to the spatial density values.

[0046] The generation of spatial density grid is achieved by superimposing the spatial weights of geographic correlation features and cross-regional transmission features. The spatial weight matrix of geographic correlation features (such as 0.92 corresponding to adjacent station areas) and the spatial weight matrix of cross-regional transmission features (such as 0.68 corresponding to remote transmission areas) are linearly weighted and fused to generate a comprehensive spatial density value. For example, the comprehensive spatial density value of area B in the city is , while the density of the suburbs of city T is . Each grid cell (such as 1km×1km) records its density value and the statistical distribution characteristics of the corresponding meteorological elements (such as mean, variance, and extreme values). The division of the core storage area and the edge cache area is based on a preset density threshold. For example, grid cells with a density value ≥0.8 are divided into core storage areas (such as city B area) to store high-access frequency or high-priority data; cells with a density value <0.8 are divided into edge cache areas (such as city T suburbs) to store low-frequency or archived data. The division of spatial density grids needs to be updated periodically. For example, during the passage of a typhoon, the conduction characteristic weight of the affected area is increased, and the core area expansion is triggered when the density value exceeds the threshold.

[0047] Step S350: Establish a bidirectional mapping rule between the main index node of the vertical level and the spatial density grid of the horizontal level. When the meteorological file matches the dynamic weight factor threshold of the main index node and the core storage area condition of the spatial density grid at the same time, the cross-level joint index mark is triggered to generate a composite index tag containing the spatiotemporal dimension and element attributes.

[0048] Bidirectional mapping rules are implemented by associating primary index node attributes (such as dynamic weight factors) at the vertical level with spatial density grid attributes (such as core storage area identifiers) at the horizontal level. Cross-level joint indexing is triggered when a meteorological file meets the following conditions: 1) the dynamic weight factor of the primary index node for its meteorological feature type exceeds a preset threshold (e.g., ≥1.0); and 2) the spatial density grid cell corresponding to its geographic extent is a core storage area. For example, a typhoon wind speed file is marked as a cross-level joint index object because its primary index node dynamic weight factor is 2.0 (exceeding the threshold of 1.0) and it is located in the core storage area (density value 0.85). Composite index labels are generated by concatenating spatiotemporal dimension codes (e.g., "2023-07-15_City B Core Area") with feature attribute codes (e.g., "Wind Speed_High-Frequency Sudden Change"), for example, "ARPU_WIND_20230715_BJ_CORE." After label generation, the index management system optimizes storage locations (e.g., assigning them to high-bandwidth nodes) and search paths (e.g., prioritizing matching data in the core area) based on the label content. Bidirectional mapping rules are implemented through database triggers or rule engines to ensure that labeling and resource allocation logic are automatically executed when data is written or updated.

[0049] Step S400: Calling the meteorological storage optimization model, generating a storage node allocation strategy for meteorological file files according to the hierarchical weight distribution of the hierarchical index structure, the storage node allocation strategy includes the compression rate threshold and storage location priority of each meteorological element data.

[0050] The meteorological storage optimization model is a decision-making model trained based on machine learning. Its input is the weight distribution data of each level in a hierarchical index structure, and its output is the allocation rules for storage resources. Optionally, it can be implemented as a hybrid model combining deep learning and reinforcement learning. A bidirectional LSTM (BiLSTM) is used to extract the long-term trend characteristics and short-term fluctuation characteristics of meteorological elements. A graph convolutional network (GCN) is used to construct a spatial topology model to capture the geographical correlation and cross-regional transmission characteristics of adjacent meteorological stations. A multi-head self-attention mechanism is then used to fuse temporal and spatial features to generate a multidimensional encoding. The policy generation module is based on the actor-critic reinforcement learning framework. The actor network generates storage allocation policies based on the real-time hierarchical weight distribution and node load status, and the critic network optimizes decisions by evaluating the effectiveness of the policies. The feedback learning module adjusts model parameters based on changes in node capacity utilization after migration, and dynamically updates feature fusion weights through meta-learning. Index consistency checks (such as hash value comparisons) and atomic locking mechanisms ensure data integrity and query continuity during the migration process, ultimately achieving efficient storage, real-time migration, and intelligent retrieval of meteorological data.

[0051] The hierarchical weight distribution includes the popularity score of meteorological element types in the vertical hierarchy (e.g., elements with high access frequency or high mutability receive higher weights) and the access frequency prediction value of the spatiotemporal grid in the horizontal hierarchy. The storage node allocation strategy specifically consists of two parts: a compression rate threshold determines the compression algorithm selected for different meteorological element data (e.g., lossless compression for high-precision temperature data, lossy compression for historical wind speed data), and a storage location priority determines the distribution of data across distributed storage nodes (e.g., high-bandwidth nodes for real-time typhoon data, low-cost nodes for archived precipitation data). For example, for core grid data with high weight, the model might specify a low compression rate (preserving high precision) and allocate it to high-performance storage nodes; whereas, for historical humidity data on edge grids, a high compression rate is used and stored on cold backup nodes. The training of the meteorological storage optimization model requires combining historical storage load data, node hardware performance parameters, and the changing trends of the hierarchical weights in the index structure. Through supervised learning, the accuracy and resource utilization of the optimization strategy generation process are optimized.

[0052] As an embodiment, in step S400, the meteorological storage optimization model is called to generate a storage node allocation strategy for meteorological file files according to the hierarchical weight distribution of the hierarchical index structure, which may specifically include:

[0053] Step S410: extracting the dynamic weight factor of each main index node in the vertical hierarchy, and generating a feature type heat weight sequence, wherein the heat weight is exponentially positively correlated with the dynamic weight factor.

[0054] The dynamic weight factor of each main index node in the vertical hierarchy is, for example, a meteorological element type priority score calculated based on the fusion of temporal distribution characteristics and spatial correlation topological characteristics. For example, the dynamic weight factor of the temperature element is 1.5, and the dynamic weight factor of the wind speed element is 2.3. The generation process of the element type heat weight sequence maps the dynamic weight factor to the storage priority weight through an exponential function. The specific formula is heat weight = e (k×动态权重因子) , where k is the scaling factor determined during the model training phase (e.g., k=0.8). The exponential positive correlation means that the higher the dynamic weight factor, the exponentially greater the heat weight, which significantly increases the storage resource allocation weight of high-priority elements. For example, when the dynamic weight factor of the wind speed element is 2.3, its heat weight is e (0.8×2.3) ≈8.17; and the dynamic weight factor of the temperature element is 1.5, which corresponds to the heat weight e (0.8×1.5) ≈3.32. The heat weight sequence is sorted by element type to form a list, such as [wind speed: 8.17, precipitation: 5.43, temperature: 3.32], which provides a quantitative basis for subsequent storage node type matching.

[0055] Step S420: extracting the boundary threshold between the core storage area and the edge cache area of ​​the spatial density grid in the horizontal layer, and generating a regional storage density gradient map, wherein the gradient map reflects the data access frequency prediction values ​​of different geographic grid units.

[0056] The threshold for demarcating the core storage area and edge cache area of ​​the spatial density grid is determined based on historical data access frequency statistics. For example, the core storage area threshold is set to a spatial density value ≥ 0.8, and the edge cache area threshold is set to a spatial density value < 0.8. The regional storage density gradient map is generated by mapping the spatial density value of each geographic grid cell (e.g., 1 km × 1 km) into a color gradient or numerical matrix. For example, a grid cell with a density value of 0.9 is represented by dark red (highly accessed core area), and a grid cell with a density value of 0.6 is represented by light blue (lowly accessed edge area). Data access frequency predictions are based on historical access logs using time series analysis models (such as ARIMA or LSTM). For example, the average number of visits to the core grid cell in city B over the next week is predicted to be 1000 per day, while the average number of visits to the edge grid cell in city T is predicted to be 200 per day. The gradient map records the density value and predicted access frequency of each grid cell through visualization or matrix encoding. For example, the cell in row i and column j of the matrix stores the value 0.9 (density value) and the predicted value 1000 (number of visits), providing spatial input for the geographic storage strategy.

[0057] Step S430: Input the element type heat weight sequence into the first feature processing layer of the meteorological storage optimization model to generate an element priority queue, which arranges the best storage node type for each meteorological element type in descending order of heat weight.

[0058] The first feature processing layer of the meteorological storage optimization model consists of a fully connected neural network. Its input is a numerical vector of feature type heat weight sequences (e.g., [8.17, 5.43, 3.32]). Its output is a storage node type label (e.g., high-bandwidth node, medium-performance node, cold storage node) matching each feature type. The feature priority queue is generated using a sorting algorithm (e.g., quick sort) to rank feature types from high to low heat weight and assign each type a predefined node type rule. For example, features with heat weights ≥ 5.0 (e.g., wind speed and precipitation) are assigned to high-bandwidth storage nodes (SSD arrays), features with weights between 2.0 and 5.0 (e.g., temperature) are assigned to medium-performance nodes (SATA hard drives), and features with weights < 2.0 (e.g., historical humidity) are assigned to cold storage nodes (tape libraries). The queue can take the form of a list structure, such as [wind speed: high-bandwidth node, precipitation: high-bandwidth node, temperature: medium-performance node], ensuring that high-heat features prioritize high-performance storage resources.

[0059] Step S440: Inputting the regional storage density gradient map into the second feature processing layer of the meteorological storage optimization model to generate a geographic storage strategy matrix, where each cell in the matrix records the storage position adjustment frequency of the corresponding grid within a preset time window;

[0060] The second feature processing layer uses a convolutional neural network to extract features from the regional storage density gradient map and output a geographic storage strategy matrix. Each cell in the matrix corresponds to a geographic grid and records the number of times that grid requires storage relocation within a preset time window (e.g., the next 24 hours). For example, due to the high predicted access frequency (1,000 times / day), the relocation frequency for the core grid cell in City B is set to once per hour (i.e., 24 times per day), while the relocation frequency for the edge grid cells in City T is set to once per day. The relocation frequency is calculated by combining spatial density, predicted access frequency, and node load balancing strategies. For example, when the grid density is ≥ 0.8 and the predicted access frequency exceeds the node throughput threshold, the relocation frequency is automatically increased to twice per hour. The geographic storage strategy matrix is ​​stored in a two-dimensional array or database table. For example, the cell in row i and column j in the matrix stores the value 24 (number of relocations), providing a basis for the execution frequency of dynamic migration operations.

[0061] Step S450: The feature priority queue and the geographic storage strategy matrix are integrated to generate a storage node allocation strategy, wherein meteorological file files of feature types with heat weights greater than a preset value are allocated to the target bandwidth storage node, and based on the grid cells whose adjustment frequency exceeds the threshold in the geographic storage strategy matrix, the number of redundant copies and the cross-node backup frequency of the associated files are increased.

[0062] The fusion process logically overlays the rules of the feature priority queue and the geographic storage strategy matrix through a policy generation algorithm. Meteorological file files for feature types with a popularity weight greater than a preset value (i.e., high-popularity feature types, such as wind speed and precipitation) are allocated to target bandwidth storage nodes based on the queue instructions. These target bandwidth storage nodes are high-bandwidth storage nodes. For example, typhoon monitoring data can be stored in SSD arrays to ensure low-latency access. Furthermore, grid cells in the geographic storage strategy matrix whose adjustment frequency exceeds a preset threshold (e.g., once per hour) trigger a redundant replication strategy. For example, three replicas are created for typhoon data in the core grid of City B, with cross-node backups performed every 30 minutes (e.g., synchronizing from Node A to Nodes B and C). The storage node allocation strategy output includes the storage location path (e.g., " / ssd_node1 / typhoon_data"), the number of replicas (e.g., 3), and the backup period (e.g., 30 minutes). These are then distributed to the distributed storage system for execution via configuration files or API commands. For example, a meteorological file belongs to the wind speed element (high heat) and is located in the core grid with a high adjustment frequency. Its allocation strategy is marked as "high-bandwidth node, 3 replicas, 30-minute backup" to meet both performance and disaster recovery requirements.

[0063] As an implementation method, the training process of the meteorological storage optimization model may include:

[0064] Step S401: Acquire historical meteorological metadata storage records, which include hierarchical weight distribution samples of a hierarchical index structure, storage node allocation strategy labels, and node load fluctuation time series data.

[0065] The acquisition process for historical meteorological metadata storage records involves extracting historical operation records related to meteorological file storage from the log database of the distributed storage system. The hierarchical weight distribution samples for the hierarchical index structure include the dynamic weight factor change sequence of each primary index node in the vertical hierarchy (e.g., the dynamic weight factor of the wind speed factor node increases from 1.8 to 2.5 during typhoon season) and the core storage area coverage ratio of the spatial density grid in the horizontal hierarchy (e.g., the coverage of the core grid in City B increases from 60% to 85%). Storage node allocation strategy labels are manually annotated or automatically generated storage rules. For example, high-dynamic-weight wind speed data is allocated to high-bandwidth nodes, and three redundant copies are created for the frequently adjusted core grid. Node load fluctuation time series data records the time-varying curves of metrics such as CPU utilization, disk throughput, and network latency for each storage node. For example, the CPU utilization of an SSD node peaked at 90% on July 15, 2023, while the disk throughput remained at 500 MB / s during the same period. The collection of historical data must cover different meteorological events (such as typhoons and rainstorms) and hardware load scenarios (such as node expansion and fault switching) to ensure the diversity of training samples and the generalization ability of the model.

[0066] Step S402: Convert the hierarchical weight distribution samples into a multidimensional training feature vector, wherein the first component of the multidimensional training feature vector corresponds to the dynamic weight factor change trend of the main index node in the vertical hierarchy, the second component corresponds to the core storage area coverage ratio of the spatial density grid in the horizontal hierarchy, and the third component corresponds to the compatibility parameter between the meteorological element type and the storage node hardware configuration.

[0067] The construction process of multidimensional training feature vectors achieves multidimensional quantization of hierarchical weight distribution samples through mathematical coding. The first component, the dynamic weight factor trend, is generated by calculating the mean, variance, and slope of the dynamic weight factor for the primary index node within a time window. For example, the dynamic weight factor for the wind speed node during typhoon season has a mean of 2.3, a variance of 0.5, and a daily growth rate of 0.2. The second component, the core storage area coverage ratio, is calculated by counting the proportion of grid cells in the horizontal hierarchy whose spatial density exceeds a preset threshold (e.g., ≥0.8) to the total number of grid cells. For example, if the core grid coverage ratio of City B increases from 60% to 85%, this component is 0.85. The third component, the compatibility parameter, is defined based on the matching rules between meteorological element type and storage node hardware performance. For example, wind speed data, due to its high real-time requirements, is rated 0.9 for compatibility with the low latency characteristics of SSD nodes, while historical temperature data is rated 0.7 for compatibility with high-capacity mechanical hard drives. Multidimensional training feature vectors are normalized to unify their numerical range. For example, the mean of the dynamic weight factor is mapped to the interval [0, 1], and the compatibility parameter is converted by percentage, so as to avoid the interference of feature scale differences on model training.

[0068] Step S403: In the model initialization stage, the multi-dimensional training feature vector is input into the convolutional feature extraction layer of the meteorological storage optimization model, and the local correlation pattern of the hierarchical weight distribution is captured through the sliding window to generate the initial feature code.

[0069] The convolutional feature extraction layer uses a one-dimensional convolution kernel sliding over time to capture local correlations between the dynamic weight factor's changing trend and the core storage area's coverage ratio. The sliding window length is set based on the data period, for example, a 24-hour window to capture daily load fluctuations, or a 30-day window to analyze monthly weight changes. The output of each convolution kernel corresponds to a local correlation pattern. For example, a kernel might identify a correlation where the core storage area coverage ratio increases by 5% when the daily growth rate of the dynamic weight factor exceeds 0.1. The initial feature encoding is generated by concatenating the results of multiple convolutions. For example, an input feature vector with 3 dimensions (dynamic weight, coverage ratio, compatibility) is processed by 32 convolution kernels to generate a 32-dimensional encoding vector. This encoding process preserves key patterns in the hierarchical weight distribution, such as the strong correlation between the surge in wind speed node weights and the expansion of the core grid during typhoon events, providing a local feature foundation for subsequent attention allocation.

[0070] Step S404: Input the initial feature code into the multi-head attention allocation layer of the meteorological storage optimization model, calculate the influence weights of different hierarchical dimensions on the storage node allocation strategy, and generate a global feature code with attention weights.

[0071] The multi-head attention allocation layer uses multiple parallel self-attention mechanisms to focus on the contribution of features at different levels of dimension. Each attention head independently calculates the query vector, key vector, and value vector. For example, the first attention head analyzes the impact of the dynamic weight factor trend on the storage strategy (e.g., for every 0.1 increase in the weight factor slope, the probability of high-bandwidth node allocation increases by 15%). The second attention head focuses on the relationship between the core storage area coverage ratio and the number of redundant replicas (e.g., for every 10% increase in coverage ratio, the number of replicas needs to be increased by 1). Attention weights are normalized using a softmax function to generate a contribution distribution for each feature, for example, a dynamic weight factor feature contribution of 0.6, a core storage feature contribution of 0.3, and a compatibility feature contribution of 0.1. A global feature encoding is generated by weighted summing the outputs of all attention heads. For example, the 32-dimensional initial encoding is multiplied by the attention weight matrix and then reduced to 16 dimensions, thereby incorporating the global correlation of multi-level features.

[0072] Step S405: input the global feature code into the strategy generation layer of the meteorological storage optimization model, and generate a candidate storage node allocation strategy set in combination with the periodic law of the node load fluctuation time series data.

[0073] The policy generation layer consists of a fully connected neural network and a rules engine. Its input is a global feature encoding and time-series data on node load fluctuations. Periodic patterns in this data are extracted using Fourier transforms or period detection algorithms. For example, a node's CPU utilization may exhibit periodic peaks from 9:00 AM to 12:00 PM daily, while disk throughput may periodically decrease at the end of the month due to data archiving operations. Based on these periodic patterns, the policy generation layer predicts load conditions for future time windows (e.g., CPU utilization is expected to reach 85% at 9:00 AM tomorrow) and generates appropriate candidate policies. For example, during periods of predicted high CPU load, a policy might be generated to limit new data writes to this node to 50 GB; during periods of low disk throughput, a policy might be generated to initiate cross-node backups to alleviate I / O bottlenecks. The candidate strategy set is generated by enumerating possible node allocation rules (such as the number of replicas, storage location, and backup frequency) and screening solutions that meet the load constraints. For example, three strategies are generated: Strategy A (high-bandwidth nodes, 2 replicas, hourly backups), Strategy B (medium-performance nodes, 3 replicas, daily backups), and Strategy C (cold storage nodes, 1 replica, no backups).

[0074] As an implementation method, in step S405, the process of generating a set of candidate storage node allocation strategies based on the periodicity of the node load fluctuation time series data may include:

[0075] Step S4051: input the global feature code and node load fluctuation time series data into the load balancing prediction submodule of the meteorological storage optimization model to predict the CPU occupancy curve and disk throughput change trend of each storage node in the future time window.

[0076] The load balancing prediction submodule uses a time series prediction model (such as LSTM or Transformer). Its input is a global feature code (representing the correlation between hierarchical weights and storage policies) and time series data on node load fluctuations (historical CPU utilization and disk throughput). The prediction process analyzes the periodicity (such as daily peaks and weekly troughs) and trends (such as long-term hardware performance degradation) of historical load data to output a CPU utilization curve (e.g., 85% predicted at 9:00 AM and 60% predicted at 15:00 PM) and disk throughput trend (e.g., 500 MB / s in the morning and 200 MB / s in the evening) for each node within a future time window (e.g., the next 24 hours). For example, for a specific SSD node, the model predicts a CPU utilization of 88% and a disk throughput of 550 MB / s at 10:00 AM the next day, triggering a high-voltage node flag.

[0077] Step S4052: Generate storage node load pressure level identifiers based on the predicted CPU occupancy rate curve. The identifiers are divided into three categories: high-voltage nodes, medium-voltage nodes, and low-voltage nodes.

[0078] For example, load stress level labels can be generated by dividing the predicted CPU usage curve by thresholds. For example, a CPU usage ≥80% is a high-pressure node (red label), 50%-80% is a medium-pressure node (yellow label), and <50% is a low-pressure node (green label). During the forecast period, a node with a CPU usage of 88% at 10:00 AM is labeled a high-pressure node, a 75% usage at 2:00 PM is labeled a medium-pressure node, and a 45% usage at 8:00 PM is labeled a low-pressure node. After labels are generated, storage policies need to be dynamically adjusted based on stress levels. For example, high-pressure nodes can be restricted from writing data to prevent overload.

[0079] Step S4053: Generate a storage node data transmission efficiency score based on the predicted disk throughput change trend. The score is positively correlated with the slope value of the throughput increase trend.

[0080] The data transfer efficiency score is calculated by quantifying the trend of disk throughput. For example, if a node's throughput increases from 300MB / s to 600MB / s in the morning, the slope is (600 - 300) / 3 hours = 100MB / s², resulting in a score of 0.9 (out of a maximum score of 1.0). Another node's throughput decreases from 200MB / s to 150MB / s, resulting in a negative slope and a score of 0.3. The scoring formula can be defined as score = tanh(k × slope), where k is a scaling factor (e.g., k = 0.01), ensuring the score falls within the [-1, 1] range and is mapped to [0, 1] using an offset. Efficient nodes (score ≥ 0.7) are prioritized for real-time data transfer tasks, while inefficient nodes (score < 0.4) are subject to backup or migration.

[0081] Step S4054: In the strategy generation layer, the load pressure level identifier and the data transmission efficiency score are jointly constrained and analyzed: for high-pressure nodes, the maximum capacity threshold for allocating new data is limited; for low-pressure nodes, the weight coefficient of receiving target priority data is increased; for nodes with a data transmission efficiency score lower than the preset standard, the cross-node replica backup mark is triggered.

[0082] Joint constraint analysis uses a rules engine to combine load pressure and transmission efficiency conditions to generate policy constraints. For example: 1) The new data capacity threshold for high-voltage nodes (CPU ≥ 80%) is set to 30% of the current remaining capacity (e.g., if 1TB is remaining, 300GB is allowed to be written); 2) The weight coefficient for target priority data (i.e., high-priority data such as wind speed) on low-voltage nodes (CPU < 50%) is increased from 0.6 to 0.9, giving it priority during allocation; 3) Nodes with a transmission efficiency score < 0.4 are required to create at least two cross-node replicas for stored data. The analysis process balances load balancing with storage efficiency. For example, while restricting writes to high-voltage nodes, the weight of low-voltage nodes can be increased to ensure that high-priority data can still be stored in a timely manner.

[0083] Step S4055: Based on the results of the joint constraint analysis, a set of candidate storage node allocation strategies is generated. Each strategy in the set contains the target node's load pressure compatibility parameters, data transmission efficiency guarantee parameters, and redundant backup execution conditions. The normalization processing of the strategy generation layer is used to ensure the consistency of parameter scales between different strategies.

[0084] The candidate strategy set is generated by enumerating all feasible node allocation schemes and applying constraints to filter them. Each strategy's parameters include: 1) load compatibility (e.g., a maximum write volume of 300GB allowed for high-load nodes); 2) data transmission efficiency guarantee (e.g., a minimum throughput of 500MB / s for high-load nodes); and 3) redundancy requirements (e.g., two replicas for low-load nodes). Normalization is performed using Min-Max scaling or Z-score standardization to unify the parameter scales. For example, write volume from 0GB to 1000GB is mapped to the range of 0-1, and throughput from 0MB / s to 1000MB / s is mapped to the range of 0-1. The final strategy set is sorted by a comprehensive score (e.g., compatibility × efficiency × redundancy guarantee), and the top-K (e.g., top 5) optimal strategies are output for system execution. For example, if strategy A has a comprehensive score of 0.92, strategy B has a score of 0.85, and strategy C has a score of 0.78, the system will prioritize strategy A for storage allocation.

[0085] Step S406: In the model optimization stage, the candidate storage node allocation strategy set is compared with the storage node allocation strategy label in the storage record, the strategy offset loss value is calculated, and the network parameters of the convolutional feature extraction layer, the multi-head attention allocation layer and the strategy generation layer are adjusted through error back propagation until the matching degree between the allocation strategy output by the model and the storage node allocation strategy label reaches the preset threshold.

[0086] Policy drift loss is measured by comparing the candidate policy with the policy label. For example, a cross-entropy loss function is used to compare the policy's probability distribution, or a mean squared error function is used to compare numerical parameters such as the number of replicas and storage locations. For example, if the policy label specifies "high-bandwidth nodes, 3 replicas" and candidate policy A specifies "high-bandwidth nodes, 2 replicas," the loss due to the replica number difference is 1. The error backpropagation process uses chained derivatives to adjust the parameters of each network layer, such as the convolution kernel weights, the query matrix of the attention head, and the bias terms of the fully connected layers. Optimization iterations continue until the policy output by the model meets a preset threshold (e.g., accuracy ≥ 95% or loss ≤ 0.05) on the validation set. After training, the model generates a storage allocation plan that is highly consistent with the historical optimal policy based on real-time layer weights and node load data.

[0087] Step S500: According to the storage node allocation strategy, each meteorological file in the meteorological metadata set is written to the corresponding distributed storage node, and based on the real-time update mechanism of the hierarchical index, the physical location and index mapping relationship of the stored files is dynamically adjusted.

[0088] Distributed storage nodes are networked storage systems comprised of multiple physical or virtual storage devices, potentially distributed across different geographic locations or data centers. The write process adheres to the compression thresholds and location priorities specified in the storage node allocation policy. For example, high-priority weather archive files are written to SSD storage arrays via dedicated transmission channels, while the files are encoded using the Zstandard or GZIP algorithm at a specified compression ratio. The real-time update mechanism for hierarchical indexes continuously monitors changes in index weights (e.g., a new weather event causing a rise in the weight of a spatiotemporal grid) and the load status of storage nodes (e.g., insufficient node capacity or increased access latency), triggering data migration or index reconstruction. For example, if heavy rain monitoring data in a particular area is frequently accessed due to real-time warning requirements, the system automatically migrates it from edge cache nodes to core storage nodes and updates the index of the corresponding grid in the horizontal hierarchy to "high priority." This dynamic adjustment process must ensure the atomicity of data migration (to avoid query failures during migration) and the consistency of index mappings (to ensure that both old and new paths can be resolved after migration). Furthermore, the real-time update mechanism must be integrated with the meteorological storage optimization model to optimize the efficiency of subsequent allocation policy generation and the overall performance of the storage system through a feedback loop.

[0089] As an implementation method, in step S500, the process of dynamically adjusting the mapping relationship between the physical location and index of the stored file includes:

[0090] Step S510: Collect the capacity utilization rate and data access frequency of the distributed storage nodes in real time to generate a node load state vector. The node load state vector includes the remaining capacity percentage of the storage node, the number of input and output operations per unit time, and the average response delay time.

[0091] Real-time collection of distributed storage node capacity utilization and data access frequency is performed periodically by a monitoring agent deployed on the storage node. The monitoring agent collects node hardware resource metrics at preset intervals (e.g., every minute). Capacity utilization refers to the ratio of the node's storage media's used capacity to its total capacity. For example, if a node has a total capacity of 10TB and 7TB is used, the remaining capacity percentage is 30%. Data access frequency is calculated by counting the number of data read and write operations (IOPS) per unit of time (e.g., per second) on the storage node. For example, if a node processes 1000 read operations and 500 write operations per second, the total number of I / O operations is 1500 / second. Average response latency refers to the average time from receiving a request to returning a result. For example, if the average latency for read operations on a node is 20 milliseconds and the average latency for write operations is 50 milliseconds, the overall average response latency is 35 milliseconds. The node load status vector encapsulates the above metrics in a structured data format (such as JSON or Protocol Buffers). For example, the vector representation is [remaining capacity percentage: 30%, number of input and output operations: 1500, average response delay time: 35ms], providing real-time input for subsequent migration decisions.

[0092] Step S520: Input the node load state vector into the real-time decision submodule of the meteorological storage optimization model, and calculate the heat priority score of the file to be migrated in combination with the heat decay curve of the hierarchical weight distribution in the hierarchical index structure. The heat priority score is determined by the decay rate of the dynamic weight factor of the main index node of the vertical hierarchy to which the file belongs and the downward trend of the access frequency of the horizontal spatial density grid.

[0093] Exemplarily, the real-time decision-making submodule processes the node load state vector using a machine learning model (such as a random forest or gradient boosting tree) and incorporates a heat decay curve from a hierarchical index structure. This heat decay curve is modeled based on a time decay function (such as exponential decay) and reflects the rate at which the dynamic weight factor of a meteorological element decreases over time. For example, the dynamic weight factor of a typhoon wind speed element decreases by 5% daily after a typhoon passes. The downward trend in access frequency of a horizontal spatial density grid is calculated by analyzing the sliding window mean of historical access logs. For example, the access frequency of a core grid cell decreases from 1,000 to 200 per day, a rate of 80 per day. The heat priority score is calculated as follows: score = dynamic weight factor decay rate × α + access frequency decline trend × β, where α and β are weighting coefficients determined during the model training phase (e.g., α = 0.6, β = 0.4). For example, if the dynamic weight factor of a typhoon file decays at a rate of 5% per day and its access frequency decreases by 80 times per day, the score is 5 × 0.6 + 80 × 0.4 = 3 + 32 = 35. A lower score indicates a faster decline in file popularity and a higher migration priority.

[0094] Step S530: Generate a migration task queue based on the heat priority score, sort the files to be migrated in ascending order of the score, and mark the hardware performance matching parameters of the target migration node;

[0095] For example, the migration task queue can be generated by using a sorting algorithm to sort the files to be migrated from low to high priority based on their popularity, ensuring that low-scoring files are migrated first. Hardware performance matching parameters are calculated based on the target node's remaining capacity, I / O operation capabilities, and latency characteristics. For example, if target node A has 40% remaining capacity, an I / O operation limit of 2000 / second, and an average latency of 25ms, its matching score is 0.8 (out of a maximum of 1.0). Node B has 20% remaining capacity, an I / O operation limit of 1200 / second, and an average latency of 50ms, and a matching score of 0.5. Each entry in the queue records the source node path, target node path, migration file size, and matching parameters, for example, [Source node: / node1 / typhoon_20230715.data, Target node: / node3 / archive / , File size: 50GB, Matching: 0.8]. After the queue is generated, the migration scheduler selects the optimal target node based on the matching parameters, for example, migrating files to nodes with a score ≥ 0.7 to ensure performance.

[0096] Step S540: When performing the incremental migration operation, the meteorological file with the lowest score in the queue and the continuous inaccessibility time exceeding the preset threshold is migrated first, and the spatial density grid association mark of the hierarchical index structure is synchronously updated after creating a migration copy in the target node.

[0097] Exemplarily, the incremental migration operation only migrates the newly added or modified parts of the file, reducing network bandwidth consumption. The preset threshold is set according to business needs. For example, a file that has not been accessed for 30 consecutive days triggers migration. For example, the heat priority score of a historical humidity file is 10 (the lowest score), and the last access time is June 1, 2023, which exceeds the 30-day threshold and is included in the first migration task. When the migration is executed, the source node transfers the file blocks to the target node, and after the target node completes the copy verification, it updates the spatial density grid mark associated with the file in the hierarchical index structure. For example, after a file originally marked as "Core Storage Area-City B Grid A01" is migrated to the edge cache area, the mark is changed to "Edge Cache Area-City T City Grid B02". The update operation ensures atomicity through database transactions to avoid inconsistent index status.

[0098] Step S550: During the migration process, an atomic locking mechanism is implemented on the index mapping relationship between the source node and the target node to ensure that the query request always points to the valid storage location during the migration, and triggers the index consistency check after the migration is completed. When a cross-node index path conflict is detected, it automatically rolls back to the pre-migration state and regenerates the migration task queue.

[0099] Exemplarily, the atomic lock mechanism can be implemented through a distributed lock service (such as ZooKeeper or Etcd). When the migration starts, the source node file is locked to prohibit concurrent write or delete operations. The query request is redirected to the source node or the target node replica during the lock period. For example, if the migration is not completed, the request is still responded by the source node; if the migration is completed, the request is switched to the target node. The index consistency check detects inconsistent items by comparing the file hash values, index labels and path mapping relationships of the source node and the target node. For example, when it is detected that the hash value of the target node replica is inconsistent with the source node, it is determined to be a conflict and triggers an automatic rollback: the target node replica is deleted, the source node index mark is restored, and the file is reinserted into the head of the migration queue. The rollback log records the exception information for subsequent analysis, and the migration task queue is regenerated according to the latest node load status. For example, after a rollback due to a target node failure, the backup node C is selected as the new target.

[0100] As an implementation method, the meteorological metadata retrieval process includes the following steps:

[0101] Step S600: Receive a search request submitted by a user, parse the spatiotemporal range constraints and meteorological element type combination conditions in the search request, and generate a multi-dimensional search condition vector. The vector includes a geographic boundary coordinate set, a time interval stamp, and an element type code sequence.

[0102] Exemplarily, the process of receiving a user-submitted search request is implemented through a front-end interface or API gateway. The parsing process extracts the spatiotemporal range constraints and meteorological element type combination conditions in the request. The spatiotemporal range constraints include a set of geographic boundary coordinates and a time interval stamp. The geographic boundary coordinates define the target search area as a sequence of longitude and latitude coordinate pairs (for example, the boundary coordinates of City B are [116.20°E, 39.80°N], [116.60°E, 39.80°N], [116.60°E, 40.20°N], [116.20°E, 40.20°N]). The time interval stamp defines the target time period with a start and end timestamp (for example, July 1, 2023, 00:00:00 to July 31, 2023, 23:59:59). Combined meteorological feature type conditions are represented by predefined coding sequences (e.g., temperature is coded as TEMP_01, wind speed is coded as WIND_02). For example, when a user requests to retrieve both temperature and wind speed data, the feature type coding sequence is [TEMP_01, WIND_02]. Multidimensional search condition vectors encapsulate these parameters using structured data. For example, the vector format is {geographic boundary coordinates: [[116.20,39.80], [116.60,39.80], [116.60,40.20], [116.20,40.20]], time interval stamps: [1688169600, 1690847999], feature type coding sequence: [TEMP_01, WIND_02]}, providing standardized input for subsequent joint queries.

[0103] Step S700: Input the multi-dimensional retrieval condition vector into the joint query engine of the hierarchical index structure, match all main index nodes whose dynamic weight factors are greater than the preset threshold in the vertical layer, and simultaneously filter the geographic grid cells covered by the core storage area of ​​the spatial density grid in the horizontal layer.

[0104] Exemplarily, the federated query engine for a hierarchical index structure processes index conditions at both the vertical and horizontal levels in parallel. The vertical level matches the dynamic weight factor of primary index nodes based on meteorological element types. A dynamic weight factor threshold is set based on real-time business needs (e.g., a threshold of 1.0), filtering out primary index nodes with weight factors exceeding that threshold (e.g., a dynamic weight factor of 2.3 for a wind speed node during typhoon season meets the threshold). The horizontal level filters based on the core storage area identifier of the spatial density grid. The core storage area consists of geographic grid cells with spatial density values ​​exceeding a preset threshold (e.g., ≥0.8) (e.g., a density of 0.9 for core grid cell A01 in city B and a density of 0.6 for edge grid cell B02 in city T). The federated query engine matches the feature type code sequence in the search condition vector with the type of primary index node in the vertical level (e.g., TEMP_01 matches the temperature node and WIND_02 matches the wind speed node). The federated query engine also intersects the geographic boundary coordinate set with the geographic extent of the horizontal level grid cells (e.g., the boundary of city B covers grid cells A01, A02, and A03). The matching result is the intersection set that meets the longitudinal weight condition and the horizontal core area condition. For example, the intersection of the wind speed main index node (dynamic weight 2.3) and the core grid unit A01 of city B is identified as the valid search range.

[0105] Step S800: Generate a candidate meteorological file set based on the intersection result of the matched main index node and the geographic grid unit. Each file in the set is marked with the compression algorithm type and storage location path recorded in the storage node allocation strategy.

[0106] For example, a set of candidate meteorological file files is extracted using a federated query engine from the bidirectional mapping rules of a hierarchical index structure. The meteorological file corresponding to the intersection result must meet both the feature type weight and core storage area location requirements. For example, the July 2023 typhoon wind speed file (feature type WIND_02) stored in core grid cell A01 of City B is included in the candidate set because it meets the dynamic weight threshold and core area requirements. Each file's metadata tag includes the compression algorithm type defined in the storage node allocation policy (e.g., Zstandard compression algorithm identifier ZSTD_01) and the storage location path (e.g., distributed node path / node_ssd_01 / typhoon_wind_202307.data). The candidate set is generated using a database query statement, such as the SQL query "WHERE feature type IN (TEMP_01, WIND_02) AND grid cell IN (A01, A02, A03)." The result set is returned as a list, annotated with compression and storage information.

[0107] Step S900: calling the real-time decompression interface of the distributed storage node, performing parallel decompression operations on the meteorological file files in the candidate set based on the compression algorithm type, and generating a standardized meteorological element monitoring data stream.

[0108] The real-time decompression interface of the distributed storage nodes calls the corresponding decoding library based on the file's compression algorithm. For example, the Zstandard compression algorithm calls the libzstd library, and the GZIP compression algorithm calls the zlib library. Parallel decompression operations are distributed to multiple compute nodes through a task scheduler, allowing simultaneous processing of files in the candidate set. For example, 10 wind speed file files can be assigned to 10 CPU cores for parallel decompression. Standardized meteorological element monitoring data streams require that the decompressed data be formatted in a predefined format (such as NetCDF or Parquet), containing fields such as timestamps, geographic coordinates, and feature values. For example, the data stream generated by decompressing a typhoon wind speed file contains hourly timestamps (2023-07-15T12:00:00), latitude and longitude coordinates (116.40°E, 39.90°N), and wind speed values ​​(15m / s), arranged in chronological order as a time series data matrix. Standardizing the data stream ensures that subsequent relevance scoring models can consistently process monitoring data from different sources.

[0109] Step S1000: Input the standardized data stream into the factor correlation scoring model established in the meteorological storage optimization model training phase, calculate the spatiotemporal coverage completeness, factor matching degree and historical access popularity weight of each meteorological file and the retrieval condition vector, generate the final retrieval result sorting list and return it to the user end.

[0110] For example, the feature relevance scoring model uses machine learning algorithms (such as random forests or neural networks) to calculate multidimensional scoring metrics. Spatiotemporal coverage assesses the geographic and temporal coverage of the file data relative to the search criteria (e.g., a file covers 80% of the area in City B and 90% of the time period in July). Feature matching assesses whether the number of data points and their fluctuations in value match the search feature type (e.g., wind speed data points account for 95% of the total number during the search period and their fluctuations are within ±2σ of the historical mean). Historical access popularity weights are based on the file's access frequency and recent attenuation trends in storage logs (e.g., a file was accessed 100 times in the past 30 days but only 5 times in the last 7 days). The model then weights and sums these scores to generate an overall relevance score. For example, file A scores 0.92 (coverage 0.8 × 0.4 + matching 0.95 × 0.3 + popularity 0.7 × 0.3), while file B scores 0.85. The sorted list is sorted in descending order by total score (e.g., File A > File B > File C) and returned in JSON or table format through the user interface, supporting paging and highlighting of key data.

[0111] As an implementation method, the calculation process of the factor relevance scoring model includes:

[0112] Step S1001: extracting the spatiotemporal identification information of the meteorological file, calculating the overlap ratio between its geographic coverage and the geographic boundary coordinate set in the search request, and generating a first dimension coverage score;

[0113] Spatiotemporal identification information includes the coordinate sets of the polygon vertices of the geographic area recorded in the file and the time interval stamp. The percentage of overlapping geographic coverage is calculated using spatial overlay analysis using a geographic information system (GIS). For example, if the geographic scope of file A is the Dongcheng District of City B (an area of ​​41.84 square kilometers), and the search request covers the entire area of ​​City B (an area of ​​16,410.54 square kilometers), the overlap area is 41.84 square kilometers, resulting in a percentage of 41.84 / 16,410.54, which is approximately 0.255%. The coverage score is normalized to map the percentage to the range of 0–1 (e.g., 0.255% corresponds to a score of 0.00255), or logarithmic scaling is used to mitigate bias in the percentage of small areas (e.g., log(1+0.255)=0.23). If a file covers the entire search area (100%), the coverage score is 1.0; if there is no overlap, the score is 0.

[0114] Step S1002: Analyze the number of data points in the meteorological element monitoring data stream that match the retrieval request element type coding sequence, and generate a second dimension element matching score based on the deviation between the numerical fluctuation range of the data points and the historical average value.

[0115] The data point quantity match is calculated by counting the total number of data points in the monitoring data stream that match the search feature type code (e.g., WIND_02) relative to the expected number of points within the search period. For example, if the search period is 720 hours (30 days), with an expected number of one data point per hour, and a file actually contains 700 wind speed data points, the match ratio is 700 / 720, which is approximately 0.972. The value fluctuation range is determined by calculating the standard deviation between the data point value and the historical mean (e.g., the average wind speed for the same period over the past five years is 12 m / s). The deviation score is calculated as 1 - (actual standard deviation / historical standard deviation). For example, if the actual standard deviation is 3 m / s and the historical standard deviation is 4 m / s, the score is 1 - 3 / 4 = 0.25. The total feature match score is the weighted average of the quantity ratio and the deviation score (e.g., 0.972 × 0.7 + 0.25 × 0.3 = 0.785).

[0116] Step S1003: Obtain the dynamic weight factor attenuation curve of the main index node in the hierarchical index structure, and generate a third-dimensional heat attenuation compensation coefficient in combination with the access frequency decrease rate of the file in the historical storage log.

[0117] The dynamic weight factor attenuation curve is fitted by an exponential function. For example, the dynamic weight factor of the wind speed node decays by 5% every day after the typhoon passes. The attenuation function is W(t)=W0×e (-0.05t) The rate of decline in file access frequency is determined through linear regression analysis of daily access counts in historical logs. For example, if a file's access count dropped from 10 to 2 per day over the past 30 days, the rate of decline is Δ = -0.27 times / day. The popularity decay compensation coefficient is calculated as the absolute value of the product of the decay curve slope (e.g., -0.05) and the access decline rate (-0.27) (|(-0.05) × (-0.27)| = 0.0135), which is then mapped to the range of 0 to 1 using a sigmoid function (e.g., 0.0135 → 0.503). A higher coefficient indicates a slower decline in file popularity, and this compensation should be applied to the score to slow down the ranking decline.

[0118] Step S1004: Input the first dimension coverage score, the second dimension element matching score and the third dimension heat attenuation compensation coefficient into the weight distribution function defined in the meteorological storage optimization model training phase. The function dynamically adjusts the weight ratio of each dimension score according to the business priority of each category in the element type coding sequence.

[0119] For example, the weight assignment function dynamically assigns weights to each dimension using a service priority table encoded by feature type. For example, during the typhoon warning phase, the wind speed feature (WIND_02) has a priority of 0.6, and the temperature feature (TEMP_01) has a priority of 0.4. The function defines coverage weight as 0.4 × priority, matching weight as 0.3 × priority, and heat weight as 0.3 × priority. For the wind speed feature, the weights for each dimension are: coverage 0.24 (0.4 × 0.6), matching 0.18 (0.3 × 0.6), and heat 0.18 (0.3 × 0.6). For the temperature feature, the weights are 0.16, 0.12, and 0.12, respectively. The weighted total score is calculated as coverage score × 0.24 + matching score × 0.18 + heat coefficient × 0.18, ensuring that high-priority features are prioritized in the ranking.

[0120] Step S1005: normalize the weighted sum total score to generate a standardized relevance score value, and sort the candidate meteorological file files in descending order according to the score value to form a final search result sorting list.

[0121] For example, normalization uses Min-Max scaling to map the total score to a range of 0–1. For example, the highest total score of 0.92 is mapped to 1.0, and the lowest score of 0.75 is mapped to 0.75 / 0.92≈0.815. Normalized scores are rounded to two decimal places (e.g., 0.92 → 0.92, 0.85 → 0.85), and the sorted list is sorted in descending order (0.92, 0.85, 0.78). List entries contain the file name, relevance score, storage path, and key data summary (e.g., "Typhoon Wind Speed_20230715.data, score 0.92, path / node_ssd_01, maximum wind speed 25 m / s"). Users can directly access or download highly relevant data by clicking on an entry. Sorted results are cached in a distributed in-memory database (e.g., Redis), enabling fast response and result reuse for subsequent search requests.

[0122] In an optional derivative implementation, after dynamically adjusting the mapping relationship between the physical location and index of the stored file in step S500, the method provided by the embodiment of the present invention may further include:

[0123] Step S1100: Perform cross-node data integrity verification on the migrated meteorological file. The verification process includes:

[0124] For example, cross-node data integrity verification involves comparing the hash codes of meteorological archive files after they are migrated from a source storage node to a target storage node, ensuring that the data has not been damaged or tampered with during transmission. The hash code is generated using a cryptographic hash function (such as SHA-256 or MD5). The source node calculates the hash value of the original file before migration (e.g., the SHA-256 result is "a1b2c3d4..."). After receiving the file, the target node performs the same hash calculation on the migrated copy. The consistency verification result is generated by comparing the hash value strings of the two nodes to ensure they are completely consistent. For example, if the source node hash value is "a1b2c3d4..." and the target node hash value is "a1b2c3d4...", the verification passes; if the target node hash value is "e5f6g7h8...", the verification fails. The verification process must be performed in an independent and secure environment to prevent temporary files or network caches from interfering with the results during the migration process. For example, after the 2023 typhoon wind speed file of city B is migrated from node A to node B, the system calls the SHA-256 algorithm to calculate the hash values ​​of the two node files respectively, and returns the consistency verification result as "success" or "failure".

[0125] Step S1200: extracting the hash check code of the original file in the source storage node and comparing it with the hash check code of the migration copy to generate a consistency verification result.

[0126] Exemplarily, the hash check code extraction of the source storage node and the target storage node is implemented through a file system interface or a dedicated verification tool. When the migration task is triggered, the source node generates a hash value of the original file and temporarily stores it in the log database. After the migration is completed, the target node calls the same hash function to generate a copy hash value. The comparison process is completed by matching the strings character by character. For example, when the hash value of the original file "a1b2c3d4" and the copy hash value "a1b2c3d4" are exactly the same, the consistency verification result is "consistent"; if there is any character difference (such as "a1b2c3d4" and "a1b2c3d5"), the result is "inconsistent". The verification result is recorded in the migration task log and triggers subsequent index updates or exception retry processes. For example, when a temperature history data file is migrated from node C to node D, the hash check code comparison finds inconsistency, the system marks the task as "failed", and triggers the exception handling mechanism.

[0127] Step S1300: When the consistency verification result passes, the storage location path of the migration copy and the spatial density grid association mark of the hierarchical index structure are synchronously updated, and the migration effective timestamp is marked in the index mapping relationship table.

[0128] For example, after the consistency verification is passed, the system needs to update the spatial density grid association tag in the hierarchical index structure to reflect the new storage location of the migrated copy. The spatial density grid association tag includes the geographic grid unit code (such as the core grid A01 of city B) and the storage node path (such as / node_ssd_02 / typhoon_wind_202307.data). The index mapping relationship table updates the migration effective timestamp (such as 2023-07-15T14:30:00 UTC) through database transactions to ensure that query requests can be accurately routed to the target node. For example, after the typhoon wind speed file is migrated to node B, its association tag is updated from " / node_ssd_01 / typhoon_wind_202307.data" to " / node_ssd_02 / typhoon_wind_202307.data", and the effective timestamp is recorded in the mapping table. Subsequent retrieval requests directly access the data copy of node B.

[0129] Step S1400: When the consistency verification result fails, the abnormal retry mechanism of the migration task queue is triggered. The mechanism automatically increases the heat priority score of the failed migration file to a preset level and reinserts it to the head of the queue to wait for the next migration scheduling.

[0130] Exemplarily, the abnormal retry mechanism handles files that fail verification by modifying the priority and retry strategy of the migration task queue. The heat priority score is increased to a preset level, for example, by multiplying the original score by a coefficient (such as 1.5) or increasing a fixed value (such as +20). For example, a file with an original score of 35 is increased to 52.5 (35×1.5), so that its ranking in the queue moves up. Reinserting the file into the head of the queue ensures that the file is processed first in the next migration scheduling cycle. For example, a humidity history file fails the hash check due to a network interruption. The system increases its score from 10 to 15 and inserts it into the first place of the queue, and attempts to transfer it first during the next migration. The upper limit of the number of retries is configurable (such as a maximum of 3 times). If the upper limit is exceeded, it will be marked as "permanent failure" and a manual intervention alarm will be triggered.

[0131] Step S1500: Input the storage node load state vector of the migration copy that finally passes the verification into the feedback learning module of the meteorological storage optimization model to generate a storage node load distribution feature vector, which is used to update the third component compatibility parameter of the multidimensional training feature vector in the model training phase.

[0132] For example, the storage node load state vector includes the capacity utilization, number of input / output operations, and average response latency of the target node after migration. For example, after migration, the remaining capacity of node B drops from 40% to 35%, the number of input / output operations increases from 1500 / second to 1800 / second, and the average latency increases from 25ms to 30ms. The feedback learning module analyzes the impact of load changes on storage policies, generates a load distribution feature vector (e.g., [capacity change: -5%, operation change: +300, latency change: +5ms]), and adjusts the compatibility parameter of the third component of the multi-dimensional training feature vector. The compatibility parameter reflects the compatibility between the meteorological element type and the storage node hardware. For example, the compatibility parameter of a high-bandwidth node with real-time wind speed data is adjusted from 0.9 to 0.85 (due to performance degradation caused by increased load), thereby reducing the node's priority for high-demand elements in subsequent policy generation.

[0133] As an implementation method, the process of updating the storage node load distribution feature vector may include:

[0134] Step S1501: According to the change range of the capacity utilization rate of the target node where the migration copy is located, the weight coefficient of the load balancing prediction submodule of the strategy generation layer in the meteorological storage optimization model is adjusted.

[0135] Exemplarily, the change in capacity utilization is obtained by calculating the difference in the remaining capacity of the target node before and after migration. For example, the remaining capacity of node B drops from 40% to 35%, with a change of -5%. The weight coefficient of the load balancing prediction submodule is used to balance the relationship between capacity, performance and heat priority. For example, the original weight coefficient is capacity: 0.4, performance: 0.4, heat: 0.2; if the capacity utilization deteriorates (such as three consecutive migrations resulting in a decrease in capacity), the capacity weight coefficient is adjusted to 0.5, the performance weight is reduced to 0.3, and the heat is maintained at 0.2. The adjustment process is optimized through the gradient descent algorithm, so that the model pays more attention to capacity constraints when predicting and avoids node overload. For example, a node is short of capacity due to repeated reception of high-heat wind speed data. The model increases the capacity weight coefficient and gives priority to nodes with higher remaining capacity when generating subsequent strategies.

[0136] Step S1502: Based on the correlation between the migration effective timestamp and the access frequency of similar files in the historical storage log, the slope parameter of the dynamic weight factor attenuation curve of the primary index node in the hierarchical index structure is recalculated.

[0137] For example, correlation analysis between the migration effective timestamp and historical access logs is performed through time series alignment. For example, after a typhoon wind speed file is migrated to node B on July 15, 2023, its access frequency drops from 100 times per day to 50 times per day between July 16 and July 20. The slope parameter of the dynamic weight factor decay curve reflects the decay rate of the element's popularity. The original slope parameter was -0.05 (5% daily decay), and it was recalculated and adjusted to -0.06 (6% daily decay) to more quickly reduce the storage priority of inactive files. The slope parameter is calculated using linear regression to fit the decreasing trend of access frequency. For example, slope = Δ number of accesses / Δ time = (50-100) / (5 days) = -10 times / day, which maps to a dynamic weight factor decay rate of -0.06 / day.

[0138] Step S1503: input the updated dynamic weight factor attenuation curve into the core storage area coverage ratio calculation process of the spatial density grid, and re-divide the boundary threshold between the core storage area and the edge cache area of ​​the geographic grid unit.

[0139] Exemplarily, the demarcation threshold between the core storage area and the edge cache area is adjusted according to the slope of the dynamic weight factor attenuation curve. For example, the original threshold density value is 0.8. When the slope parameter changes from -0.05 to -0.06, the density value threshold is increased to 0.82 to more strictly screen high-heat grid cells. The re-division process is analyzed through density distribution histogram. For example, the density value of the core grid cell A01 of city B drops from 0.9 to 0.85 (due to the acceleration of dynamic weight decay). If the new threshold of 0.82 is still higher than this value, it will be retained as the core storage area; if the density value of a grid cell drops from 0.81 to 0.79, it will be downgraded to the edge cache area. The threshold adjustment ensures that the core area only retains the current highly active meteorological data, optimizing the allocation of storage resources.

[0140] Step S1504: triggering the horizontal hierarchical reconstruction operation of the hierarchical index structure according to the re-divided demarcation threshold, updating the composite index tag of the meteorological file, and simultaneously optimizing the number of redundant copies and the cross-node backup frequency in the storage node allocation strategy.

[0141] For example, horizontal hierarchical reconstruction is performed by batch updating the storage zone identifiers of spatial density grids. For example, A08 and A09 in the original core grids A01 to A10 are downgraded to edge zones, while A11 is added as a core zone. The composite index tag is updated to "core storage zone-wind speed-A11" or "edge cache zone-temperature-A08." The number of redundant replicas is adjusted based on the new thresholds. For example, the number of core zone replicas is reduced from 3 to 2 (due to the reduction in the number of core zones after the density threshold is increased), while the number of edge zone replicas is increased from 1 to 2 (to improve disaster recovery). The backup frequency across nodes is also optimized simultaneously. For example, the core zone backup frequency is increased from once an hour to every 30 minutes, and the edge zone backup frequency is increased from once a day to every 12 hours. The reconstruction operation ensures atomicity through database transactions, preventing query errors caused by inconsistent index states. For example, if a typhoon file is downgraded to an edge zone due to grid A09 being downgraded, the number of its replicas is adjusted from 3 to 2, triggering an additional backup task from node B to node C.

[0142] Figure 2 A hardware entity diagram of a storage system provided by an embodiment of the present invention is as follows Figure 2 As shown, the hardware entity of the storage system 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.

[0143] The memory 1002 stores computer programs that can be run on the processor. The memory 1002 is configured to store instructions and applications executable by the processor 1001. It can also cache data to be processed or processed by the processor 1001 and various modules in the storage system 1000 (for example, image data, audio data, voice communication data, and video communication data). This can be implemented through flash memory (FLASH) or random access memory (RAM).

[0144] When the processor 1001 executes the program, the steps of any of the above-mentioned meteorological metadata storage methods based on machine learning are implemented. The processor 1001 generally controls the overall operation of the storage system 1000.

[0145] An embodiment of the present invention provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the meteorological metadata storage method based on machine learning in any of the above embodiments.

[0146] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0147] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0148] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0149] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0150] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0151] The above description is only an embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A meteorological metadata storage method based on machine learning, characterized in that: The method comprises: Acquire a meteorological metadata set for a target area, wherein the meteorological metadata set includes a plurality of unstructured meteorological file files, each meteorological file file including monitoring data of at least one meteorological element and corresponding spatiotemporal identification information; Inputting the meteorological metadata set into a pre-trained temporal feature extraction model to generate temporal distribution features of each meteorological file, and synchronously inputting the meteorological metadata set into a spatial topology mapping model to generate spatial correlation topology features of each meteorological element; Based on the cross-fusion results of the temporal distribution characteristics and the spatial correlation topological characteristics, a hierarchical index structure of meteorological metadata is constructed, wherein the hierarchical index structure includes a vertical level divided by meteorological element type and a horizontal level divided by spatiotemporal dimensions; Invoking a meteorological storage optimization model to generate a storage node allocation strategy for meteorological file files based on the hierarchical weight distribution of the hierarchical index structure, wherein the storage node allocation strategy includes a compression rate threshold and a storage location priority for each meteorological element data; According to the storage node allocation strategy, each meteorological file in the meteorological metadata set is written to the corresponding distributed storage node, and based on the real-time update mechanism of the hierarchical index, the physical location and index mapping relationship of the stored files is dynamically adjusted; The step of constructing a hierarchical index structure of meteorological metadata based on the cross-fusion results of the temporal distribution features and the spatial correlation topology features includes: Decomposing the time series distribution characteristics into periodic characteristics and mutation characteristics of meteorological elements, wherein the periodic characteristics represent the fluctuation pattern of meteorological elements that recurs in historical monitoring data, and the mutation characteristics represent the abnormal fluctuation range that exceeds the preset threshold; Decomposing the spatial correlation topological features into geographic correlation features and cross-regional conduction features, wherein the geographic correlation features represent the spatial continuity of monitoring data of adjacent meteorological stations, and the cross-regional conduction features represent the correlation strength of meteorological elements between non-adjacent regions; In the vertical hierarchical division, primary index nodes are created according to meteorological element types, and a dynamic weight factor is assigned to each primary index node. The dynamic weight factor is determined by the ratio of periodic characteristics to mutation characteristics, so that the primary index nodes of high-frequency mutation elements obtain higher vertical hierarchical priority. In the horizontal hierarchical division, a spatial density grid is generated based on the superposition of geographic correlation characteristics and cross-regional conduction characteristics. Each grid cell records the statistical distribution characteristics of meteorological elements within the corresponding geographic range, and the grid cells are divided into core storage areas and edge cache areas according to the spatial density value. A bidirectional mapping rule is established between the main index node of the vertical level and the spatial density grid of the horizontal level. When the meteorological file matches the dynamic weight factor threshold of the main index node and the core storage area condition of the spatial density grid at the same time, the cross-level joint index tag is triggered to generate a composite index tag containing spatiotemporal dimensions and feature attributes.

2. The method according to claim 1, characterized in that After obtaining the meteorological metadata set of the target area, the method further includes: Performing data preprocessing on the meteorological metadata set, the data preprocessing comprising: Identify the timestamp intervals of missing monitoring data in meteorological files and generate interpolated data based on the fluctuation trend of monitoring data from adjacent meteorological stations; Detect abnormal jump points in meteorological element monitoring data and use sliding window algorithm to smooth and correct the data sequence within the window; Convert meteorological file files of different formats into a unified binary encoding format and add metadata tags containing spatiotemporal identification information to the file header; The preprocessed meteorological metadata dataset meets the input format specifications of the temporal feature extraction model and the spatial topology mapping model.

3. The method according to claim 2, characterized in that The calling of the meteorological storage optimization model to generate a storage node allocation strategy for meteorological file files according to the hierarchical weight distribution of the hierarchical index structure includes: Extract the dynamic weight factor of each main index node in the vertical hierarchy and generate a feature type heat weight sequence, where the heat weight is exponentially positively correlated with the dynamic weight factor; Extracting the boundary threshold between the core storage area and the edge cache area of ​​the spatial density grid in the horizontal layer, and generating a regional storage density gradient map, wherein the gradient map reflects the predicted value of data access frequency of different geographic grid cells; Inputting the element type heat weight sequence into the first feature processing layer of the meteorological storage optimization model to generate an element priority queue, wherein the queue arranges the optimal storage node type of each meteorological element type in descending order of heat weight; Inputting the regional storage density gradient map into the second feature processing layer of the meteorological storage optimization model to generate a geographic storage strategy matrix, wherein each cell in the matrix records the storage position adjustment frequency of the corresponding grid within a preset time window; The feature priority queue and the geographic storage strategy matrix are integrated to generate a storage node allocation strategy, in which meteorological file files of feature types with heat weights greater than a preset value are allocated to the target bandwidth storage node, and based on the grid cells whose adjustment frequency exceeds the threshold in the geographic storage strategy matrix, the number of redundant copies and the cross-node backup frequency of the associated files are increased.

4. The method according to claim 1, wherein The training process of the meteorological storage optimization model includes: Acquire historical meteorological metadata storage records, wherein the historical meteorological metadata storage records include hierarchical weight distribution samples of a hierarchical index structure, storage node allocation strategy labels, and node load fluctuation time series data; Converting the hierarchical weight distribution sample into a multidimensional training feature vector, wherein the first component of the multidimensional training feature vector corresponds to the dynamic weight factor change trend of the main index node in the vertical hierarchy, the second component corresponds to the core storage area coverage ratio of the spatial density grid in the horizontal hierarchy, and the third component corresponds to the compatibility parameter between the meteorological element type and the storage node hardware configuration; In the model initialization stage, the multidimensional training feature vector is input into the convolutional feature extraction layer of the meteorological storage optimization model, and the local correlation pattern of the hierarchical weight distribution is captured through a sliding window to generate an initial feature code; Input the initial feature code into the multi-head attention allocation layer of the meteorological storage optimization model, calculate the influence weights of different hierarchical dimensions on the storage node allocation strategy, and generate a global feature code with attention weights; Inputting the global feature code into the strategy generation layer of the meteorological storage optimization model, and combining it with the periodicity of the node load fluctuation time series data to generate a set of candidate storage node allocation strategies; During the model optimization phase, the candidate storage node allocation strategy set is compared with the storage node allocation strategy labels in the storage records, the strategy offset loss value is calculated, and the network parameters of the convolutional feature extraction layer, the multi-head attention allocation layer, and the strategy generation layer are adjusted through error back propagation until the matching degree between the allocation strategy output by the model and the storage node allocation strategy label reaches a preset threshold.

5. The method according to claim 4, characterized in that The generation of a candidate storage node allocation strategy set based on the periodicity of the node load fluctuation time series data includes: Input the global feature code and node load fluctuation time series data into the load balancing prediction submodule of the meteorological storage optimization model to predict the CPU occupancy rate curve and disk throughput change trend of each storage node in the future time window; Generate a storage node load pressure level identifier based on the predicted CPU occupancy rate curve, wherein the storage node load pressure level identifier includes a high-voltage node, a medium-voltage node, and a low-voltage node; Generate a storage node data transmission efficiency score based on the predicted disk throughput change trend, where the score is positively correlated with the slope value of the throughput increase trend; In the strategy generation layer, the load pressure level identifier and the data transmission efficiency score are jointly constrained and analyzed: for high-pressure nodes, the maximum capacity threshold for allocating new data is limited; for low-pressure nodes, the weight coefficient of their receiving target priority data is increased; for nodes with data transmission efficiency scores below the preset standard, cross-node replica backup marks are triggered; Based on the results of the joint constraint analysis, a set of candidate storage node allocation strategies is generated. Each strategy in the set contains the load pressure compatibility parameters of the target node, data transmission efficiency guarantee parameters and redundant backup execution conditions. The parameter scale consistency between different strategies is ensured through normalization processing at the strategy generation layer.

6. The method according to claim 1, characterized in that The process of dynamically adjusting the mapping relationship between the physical location and index of the stored files includes: Collect the capacity utilization rate and data access frequency of distributed storage nodes in real time to generate a node load state vector, which includes the remaining capacity percentage of the distributed storage node, the number of input and output operations per unit time, and the average response delay time; The node load state vector is input into the real-time decision submodule of the meteorological storage optimization model. The heat priority score of the file to be migrated is calculated by combining the heat decay curve of the hierarchical weight distribution in the hierarchical index structure. The heat priority score is determined by the decay rate of the dynamic weight factor of the main index node of the vertical hierarchy to which the file belongs and the access frequency decline trend of the horizontal spatial density grid. Generate a migration task queue based on the popularity priority score, sort the files to be migrated in ascending order of score, and mark the hardware performance matching parameters of the target migration node; When performing incremental migration operations, priority is given to migrating the meteorological file with the lowest score in the queue and the time it has not been accessed for exceeding the preset threshold. After creating the migration copy in the target node, the spatial density grid association mark of the hierarchical index structure is updated synchronously. During the migration process, an atomic locking mechanism is implemented on the index mapping relationship between the source node and the target node, so that query requests always point to valid storage locations during the migration. An index consistency check is triggered after the migration is completed. When a cross-node index path conflict is detected, the system rolls back to the pre-migration state and regenerates the migration task queue.

7. The method according to claim 1, characterized in that The meteorological metadata retrieval process includes: Receive a search request submitted by a user, parse the spatiotemporal range constraint and meteorological element type combination condition in the search request, and generate a multi-dimensional search condition vector, the vector including a geographic boundary coordinate set, a time interval stamp, and an element type code sequence; Inputting the multi-dimensional search condition vector into a joint query engine of a hierarchical index structure, matching all primary index nodes whose dynamic weight factors are greater than a preset threshold in the vertical layer, and simultaneously screening the geographic grid cells covered by the core storage area of ​​the spatial density grid in the horizontal layer; Generate a candidate meteorological file set based on the intersection of the matched primary index node and the geographic grid unit, wherein each file in the set is annotated with the compression algorithm type and storage location path recorded in the storage node allocation strategy; Calling the real-time decompression interface of the distributed storage node, performing parallel decompression operations on the meteorological file files in the candidate set based on the compression algorithm type, and generating a standardized meteorological element monitoring data stream; The standardized meteorological element monitoring data stream is input into the element correlation scoring model established in the meteorological storage optimization model training phase, and the spatiotemporal coverage completeness, element matching degree and historical access popularity weight of each meteorological file and the retrieval condition vector are calculated to generate a final retrieval result sorting list and return it to the user end.

8. The method according to claim 7, characterized in that The calculation process of the factor relevance scoring model includes: Extract the spatiotemporal identification information of the meteorological file, calculate the overlap ratio between its geographic coverage and the geographic boundary coordinate set in the search request, and generate the first dimension coverage score; Analyze the number of data points in the meteorological element monitoring data stream that match the retrieval request element type code sequence, and combine the deviation degree of the data point value fluctuation range and the historical average value to generate the second dimension element matching score; Obtain the dynamic weight factor attenuation curve of the primary index node in the hierarchical index structure, and combine it with the access frequency decrease rate of the file in the historical storage log to generate the third dimension heat attenuation compensation coefficient; The first dimension coverage score, the second dimension element matching score, and the third dimension heat attenuation compensation coefficient are input into the weight distribution function defined in the meteorological storage optimization model training phase. The function dynamically adjusts the weight ratio of each dimension score according to the business priority of each category in the element type code sequence; The total score after weighted summation is normalized to generate a standardized relevance score value, and the candidate meteorological file files are sorted in descending order according to the score value to form the final search result sorting list.

9. A storage system comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images