Storage and retrieval optimization method based on beidou sounding observation data

Abstract

The application relates to the technical field of big data storage, and discloses a storage and retrieval optimization method based on Beidou sounding observation data.The method comprises the following steps: quality control and analysis of original observation data are carried out through data value perception to extract complete observation information; a value evaluation model is established based on meteorological dynamics principles to score the value density of data segments; a dynamic storage routing strategy is designed according to the scoring results to construct a hierarchical storage architecture and realize intelligent scheduling of data among different storage levels; a hybrid index structure is established by fusing time and space, features and values; finally, intelligent retrieval service is used to respond to query requests and return optimized sorting results; and the application significantly improves the storage efficiency and retrieval speed of high-value meteorological data.

Description

Technical Field

[0001] This invention relates to the field of big data storage technology, and in particular to an optimized method for storing and retrieving BeiDou radiosonde observation data. Background Technology

[0002] As one of the core technologies for atmospheric sounding, the BeiDou sounding system, leveraging the wide-area coverage, high-precision positioning, and real-time data transmission capabilities of BeiDou satellites, has been widely applied in fields such as weather forecasting, disaster early warning, and climate research. Through its sounding instruments, it collects multi-dimensional meteorological element data from near the ground to tens of kilometers in the upper atmosphere in real time, including precise timestamps, three-dimensional spatial coordinates, and key information such as temperature, air pressure, humidity, wind direction, and wind speed. This forms a massive, timely, and multi-dimensional observation data stream, providing core data support for the analysis of atmospheric vertical structure and the assessment of weather system evolution.

[0003] With the large-scale application of BeiDou radiosonde technology, the amount of observational data is growing exponentially. A single radiosonde system can generate GB-level raw data per day. Data storage often adopts a single medium or a simple hierarchical architecture, without establishing a mechanism to link data value with storage resources. High-value key layer data is stored together with low-value ordinary observation data, which not only causes access delays for high-value data that cannot meet real-time forecasting needs, but also causes high-performance storage resources to be occupied by low-reusability data, resulting in resource waste. At the same time, existing retrieval relies only on a single index that matches spatiotemporal ranges, which cannot filter data by meteorological characteristics, and the retrieval results are not sorted by value. Users need to manually filter key information, which is time-consuming. Summary of the Invention

[0004] The technical problem to be solved by this invention is that the existing technology for storing BeiDou observation data suffers from the disadvantages of resource waste and slow retrieval efficiency. To address this, we propose an optimized method for the storage and retrieval of BeiDou radiosonde observation data.

[0005] To achieve the above objectives, this application adopts the following technical solution: an optimized method for storing and retrieving BeiDou radiosonde observation data, comprising the following steps:

[0006] Step 1: Data Value Perception: Receive the observation data stream uploaded by the radiosonde in real time through the BeiDou satellite data transmission channel, decode and analyze the raw data, and extract complete observation information including but not limited to precise timestamps, three-dimensional spatial coordinates, and meteorological elements such as temperature, air pressure, humidity, wind direction, and wind speed; use a sliding window quality control algorithm to detect outliers and verify consistency of the data.

[0007] Step 2: Value Quantification Assessment: Based on meteorological dynamics principles and operational application needs, establish a feature extraction and value assessment system; identify key strata and weather phenomena through multi-scale analysis of vertical profiles; and establish a comprehensive value assessment model by combining real-time operational attention and historical access patterns to generate a quantified value density score for each data segment.

[0008] Step 3: Dynamic Storage Routing: Based on the value density score and system operating status, design an adaptive data distribution mechanism; establish a tiered storage architecture including memory cache, high-speed solid-state drives, and large-capacity hard disk drives, and establish a mapping relationship between value density and storage resources; realize the scheduling and migration of data between different storage tiers through multi-objective optimization algorithms;

[0009] Step 4: Multidimensional Index Construction: Based on the value assessment results and meteorological feature parameters, establish a hybrid index structure that supports complex queries; establish a data location mechanism by combining spatiotemporal partition index, meteorological feature inverted index, and value range index; improve index access performance through index compression and caching optimization techniques, and define similarity calculation rules and weight allocation standards for each index dimension;

[0010] Step 5: Intelligent Search Service: Receive user query requests, parse query semantics and business intent; perform rapid data filtering based on a hybrid index structure, and optimize and sort the search results by combining value weights and feature matching degrees.

[0011] Preferably, the quality control algorithm for data value perception in step one specifically includes:

[0012] Cyclic redundancy check and format compliance check are performed on the original BeiDou messages. Hamming code error correction algorithm is used to repair bit errors in the transmission process. Data sources with a check failure rate exceeding the threshold are marked and isolated.

[0013] A sliding window outlier detection algorithm based on statistical principles is used to detect anomalies in temperature, air pressure, and humidity sequences in real time. The interquartile range method is used to identify and remove observations that deviate significantly from the normal range.

[0014] The inherent correlation between data from different sensors is verified through a physical consistency test algorithm. This includes checking physical constraints, including but not limited to temperature-dew point difference and wind pressure relationship, and marking suspicious data points that do not conform to atmospheric physical laws.

[0015] Establish a data quality assessment index system, including completeness, accuracy, consistency, and timeliness, and generate a quality score for each data segment as input parameters for subsequent value assessment.

[0016] Preferably, the feature extraction for value quantification assessment in step two specifically includes:

[0017] Vertical structure feature extraction: Calculate thermodynamic parameters including but not limited to temperature lapse rate, potential temperature gradient, virtual potential temperature, and equivalent potential temperature; identify stable stratifications including but not limited to the top of the troposphere, inversion layer, isotherm, and mixed layer; and analyze the strength, thickness, and vertical distribution characteristics of the stratifications.

[0018] Dynamic feature extraction: Calculates dynamic parameters including but not limited to geostrophic wind, thermal wind, Richardson number, vorticity, divergence, and vertical wind shear; identifies dynamic features including but not limited to jet core region, frontal region, shear line, and vortex system; and analyzes the stability of atmospheric motion and energy conversion processes.

[0019] Water vapor feature extraction: Calculate water vapor parameters including but not limited to specific humidity, mixing ratio, relative humidity, water vapor flux, and precipitable water; identify water vapor features in the dry layer, wet layer, and cloud layer; and analyze the vertical distribution and transport characteristics of water vapor.

[0020] Cloud physics feature extraction: Based on temperature and humidity parameters, calculate cloud base height, cloud top height, and cloud thickness parameters, identify different types of cloud structures, and analyze the relationship between cloud physics processes and weather systems.

[0021] Preferably, the identification and quantification of key layers in step two specifically includes:

[0022] Wavelet transform and multi-scale analysis methods are used to enhance the local features of the vertical profile. Abrupt points and inflection points in the profile are identified by modulus maxima detection, and the boundary positions of key layers are accurately located.

[0023] Based on the sliding correlation algorithm, the coordinated variation characteristics of meteorological elements at different altitudes are analyzed. The stratification interface is identified by the sharp changes in the correlation coefficient, and the vertical range and internal structure of the stratification are determined.

[0024] The vertical structure of the atmosphere is divided into several relatively uniform layers based on the similarity and continuity of meteorological elements, and stratification units with specific meteorological significance are identified.

[0025] For each identified key stratification, its physical intensity index, vertical extension range, duration, and climate anomaly characteristic parameters are calculated to establish a quantitative description system for stratification characteristics.

[0026] Preferably, the hierarchical storage architecture design for dynamic storage routing in step three specifically includes:

[0027] The storage system is structured in three tiers: Tier 1 is a memory-based high-speed cache that stores the most valuable real-time data and frequently accessed hot data, providing nanosecond-level access latency; Tier 2 is a performance storage area based on NVMe solid-state drives that stores medium-value historical data and feature products, providing microsecond-level access latency; and Tier 3 is a capacity storage area based on high-density hard disk drives that stores low-value raw data and archived data, providing millisecond-level access latency.

[0028] Design an intelligent data prefetching mechanism that, based on access pattern prediction and value density assessment, preloads high-value data that is likely to be accessed frequently into the high-speed storage layer, reducing user access waiting time.

[0029] Dynamic replica management automatically adjusts the number of data replicas based on data value scores and access load. Multiple replicas are created for high-value hot data to improve concurrent access performance, while the number of replicas is reduced for low-value cold data to save storage space.

[0030] Establish storage resource monitoring and collect performance metrics for each storage tier in real time. Capacity utilization and load status It provides real-time feedback information for routing decisions, among which These correspond to the first, second, and third levels of storage, respectively. These correspond to the first, second, and third levels of storage, respectively.

[0031] Preferably, the specific implementation of the adaptive data distribution mechanism in step three includes:

[0032] Establish a matching model between value density and storage performance, and establish data value scoring and storage tier performance indicators. The correspondence between them ensures that high-value data receives high-performance storage services;

[0033] Design a load balancing algorithm to monitor the real-time load rate of each storage node. This avoids overloading a single node and affecting the overall system performance, and achieves a reasonable distribution of request load among multiple nodes;

[0034] Data lifecycle management defines the data lifecycle, predicts the data access frequency based on historical access logs using an LSTM model, and automatically adjusts the data storage location according to the time decay law of data value. Newly generated high-value data is prioritized to be stored in the high-speed storage area, while historical data with decaying value is automatically migrated to the low-speed storage area.

[0035] The system is designed with an elastic scaling mechanism to dynamically adjust storage resource configuration based on system load changes. During peak access periods, high-performance storage capacity is automatically expanded, and during off-peak access periods, resources are automatically contracted to reduce operating costs.

[0036] Preferably, for the hybrid index structure constructed in step four using the multidimensional index, multidimensional similarity is defined, specifically including:

[0037] Spatiotemporal index layer: Geohash encoding is used to discretize spatial locations, and a spatiotemporal joint index is established by combining time segmentation strategy. It supports efficient data retrieval by latitude and longitude range and time interval, and defines spatiotemporal dimension similarity based on spatial distance and time difference.

[0038] Feature Index Layer: An inverted index is built based on meteorological feature labels. An inverted list containing the identifiers of related data segments is maintained for each feature label. Feature retrieval by meteorological elements and weather phenomena is supported. Feature dimension similarity is calculated based on the number of feature matches.

[0039] Value Index Layer: Based on value density scores, an interval index is built, dividing the data into multiple value intervals according to value density. An independent index partition is built for each interval, supporting data filtering by value level. Value dimension similarity is defined based on the value density difference.

[0040] Topology Index Layer: Based on the spatiotemporal and physical relationships between data, a topology index is built to record the adjacency relationships of data segments in the spatiotemporal dimension and the causal relationships in the physical process, supporting complex data association queries.

[0041] Preferably, the meteorological significance intensity of the key strata is quantified using the following mathematical expression:

[0042] ;

[0043] in: The meteorological significance intensity of the key strata; The potential temperature gradient of the k-th layer reflects the stability of the stratification. This is a highly weighted coefficient; Let be the relative humidity of the k-th layer; The average relative humidity; The horizontal wind speed reflects the dynamic effect; For reference wind speed, 1.5 times the average wind speed in recent years is used; and The parameters were determined by fitting historical sounding data to adjust them; It is a hyperbolic tangent function to avoid excessive interference with intensity assessment when the wind speed is too high.

[0044] Preferably, the utility function used in the dynamic routing strategy to evaluate the storage level of data segments is defined by the following mathematical expression:

[0045] ;

[0046] in: Calculate the utility score for storing the data segment in storage level i; Meteorological significance intensity; This refers to the performance metrics of storage level i in a three-tier storage system. is the unit capacity cost coefficient for storage level i; The real-time load rate of storage tier i; This represents the access frequency predicted based on the LSTM model. For the data lifecycle; This refers to the data's lifespan. To adjust the parameters, they are determined by fitting recent storage resource monitoring data, which is used to balance the weight of data value and access frequency in scheduling decisions.

[0047] Preferably, in a multi-objective ranking function, the final ranking score of the candidate data segment is determined by the following mathematical expression:

[0048] ;

[0049] in The final sorting score for data segment d; Let be the similarity in the j-th dimension; The weights set for the similarity; The meteorological significance intensity of data segment d; The maximum meteorological significance in the current query result set; This represents the average meteorological significance level in the current query result set. The standard deviation of the significance level; To adjust the parameters, calibration was performed using recent user search feedback data.

[0050] The technical effects and advantages of this invention are as follows:

[0051] In this invention, the quality control and feature extraction of BeiDou radiosonde observation data are carried out through data value perception and quantitative evaluation. Key meteorological layers are accurately identified and a value density score is generated. Based on this score, a dynamic hierarchical storage architecture is constructed to realize intelligent scheduling and optimized allocation of storage resources for high-value data. At the same time, by establishing a hybrid indexing mechanism that integrates spatiotemporal, feature and value, the retrieval efficiency of complex meteorological queries is significantly improved. Attached Figure Description

[0052] The disclosure of this invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings, the same reference numerals are used to refer to the same parts:

[0053] Fig. 1 This is a schematic diagram of the method of the present invention;

[0054] Fig. 2 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0055] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0056] This invention provides a technical solution: an optimized method for storing and retrieving BeiDou radiosonde observation data.

[0057] Reference Figs. 1-2 The overall process of the method of the present invention includes the following steps:

[0058] Step 1: Data Value Perception

[0059] First, the observation data stream uploaded by the radiosonde is received in real time through the BeiDou satellite data transmission channel. These raw data streams are usually in encoded format and need to be decoded and parsed to extract complete observation information. This information includes, but is not limited to, precise timestamps, three-dimensional spatial coordinates, and core meteorological elements such as temperature, air pressure, humidity, wind direction, and wind speed.

[0060] To ensure data reliability, a sliding window quality control algorithm is used to detect outliers and verify consistency. Specifically, this quality control algorithm includes:

[0061] Data transmission quality check and error correction: Cyclic redundancy check and format compliance check are performed on the original BeiDou messages to ensure that no errors or damage occur during data transmission. If a check failure is detected, the Hamming code error correction algorithm is used to try to repair bit errors in the transmission process. For data sources with a check failure rate exceeding a preset threshold, the system will mark them as unreliable data sources and isolate them to avoid contaminating subsequent data processing procedures.

[0062] Real-time outlier detection: A sliding window outlier detection algorithm based on statistical principles is used to detect anomalies in meteorological element sequences such as temperature, air pressure, and humidity in real time. For example, the interquartile range method is used to identify and remove observations that deviate significantly from the normal range. This method calculates the upper and lower quartiles of the data sequence and defines an outlier judgment range. Data points that exceed this range are considered outliers. The sliding window mechanism ensures the real-time and local adaptability of outlier detection.

[0063] Physical consistency verification: Verify the inherent correlation between data from different sensors using physical consistency verification algorithms; this includes checking, but is not limited to, whether the temperature-dew point difference conforms to physical laws, for example, the dew point temperature cannot be higher than the ambient temperature, and whether the wind pressure relationship is reasonable; mark suspicious data points that do not conform to atmospheric physical laws, which may be due to sensor failure or environmental interference.

[0064] Data quality assessment: Establish a data quality assessment indicator system, including completeness, accuracy, consistency, and timeliness; generate a comprehensive quality score for each data segment, which will serve as an input parameter for subsequent value assessment, ensuring that only high-quality data can enter the subsequent processing flow.

[0065] Step Two: Value Quantification Assessment

[0066] First, based on the principles of meteorological dynamics and the needs of operational applications, a feature extraction and value assessment system is established; through multi-scale analysis of vertical profiles, key strata and weather phenomena are identified.

[0067] Feature extraction for value quantification assessment specifically includes:

[0068] Vertical structure feature extraction: Calculate thermodynamic parameters including but not limited to temperature lapse rate, potential temperature gradient, virtual potential temperature, and equivalent potential temperature; based on these parameters, identify stable strata including but not limited to the tropopause, inversion layer, isotherm, and mixing layer top; analyze the intensity, thickness, and vertical distribution characteristics of these strata, which are crucial for understanding atmospheric vertical structure and predicting convective activity.

[0069] Dynamic feature extraction: Calculate dynamic parameters including but not limited to geostrophic wind, thermal wind, Richardson number, vorticity, divergence, and vertical wind shear; based on these parameters, identify dynamic features including but not limited to jet core region, frontal region, shear line, and vortex system; analyze the stability of atmospheric motion and energy conversion process.

[0070] Water vapor feature extraction: Calculate water vapor parameters including but not limited to specific humidity, mixing ratio, relative humidity, water vapor flux, and precipitable water; based on these parameters, identify water vapor features such as dry layer, wet layer, and cloud layer; and analyze the vertical distribution and transport characteristics of water vapor.

[0071] Cloud physics feature extraction: Calculate parameters such as cloud base height, cloud top height, and cloud thickness based on temperature and humidity parameters; identify different types of cloud structures; and analyze the relationship between cloud physics processes and weather systems.

[0072] The identification and quantification of key layers in step two specifically includes:

[0073] Local feature enhancement and abrupt change detection: Wavelet transform and multi-scale analysis methods are used to enhance the local features of the vertical profile to highlight subtle changes in the profile; abrupt changes and turning points in the profile are identified by modulus maxima detection, thereby accurately locating the boundary positions of key layers.

[0074] Synergistic Variation Analysis and Interface Identification: Based on the sliding correlation algorithm, the synergistic variation characteristics of meteorological elements at different altitudes are analyzed; the stratification interface is identified by the sharp changes in the correlation coefficient, and the vertical range and internal structure of the stratification are determined; for example, at the top and bottom of the inversion layer, the temperature gradient will change significantly, and the correlation coefficient will also change accordingly.

[0075] Fuzzy clustering and stratification unit division: The fuzzy clustering algorithm is used to intelligently divide the vertical layers; based on the similarity and continuity of meteorological elements, the vertical structure of the atmosphere is divided into several relatively uniform layers, thereby identifying stratification units with specific meteorological significance.

[0076] Stratification of features: For each identified key strata, calculate its physical strength index, vertical extension range, duration and climate anomaly, and establish a quantitative description system for stratification features.

[0077] The meteorological significance intensity of the key strata is quantified using the following mathematical expression:

[0078] ;

[0079] in: The meteorological significance intensity of the key strata; The potential temperature gradient of the k-th layer reflects the stability of the stratification. This is a highly weighted coefficient; Let be the relative humidity of the k-th layer; The average relative humidity; The horizontal wind speed reflects the dynamic effect; For reference wind speed, 1.5 times the average wind speed in recent years is used; and The parameters were determined by fitting historical sounding data to adjust them; It is a hyperbolic tangent function to avoid excessive interference with the intensity assessment when the wind speed is too high; Used to measure the thermodynamic core strength of critical strata; Essentially, it represents the standard deviation of relative humidity, reflecting the dispersion of water vapor distribution in the vertical direction. If the denominator is too small, indicating extremely uniform water vapor distribution, it suggests that there is no significant water vapor gradient in the region. Even if there are fluctuations in thermodynamic parameters, it may be due to observational errors rather than the true critical strata. Enlarging the denominator can reduce the score of such data. Conversely, if the denominator is large, indicating uneven water vapor distribution, such as the boundary between dry and wet layers, the weight of thermodynamic parameters is strengthened, highlighting the value of the true critical strata. This is used to eliminate invalid data with uniform water vapor distribution but abnormal thermodynamic parameters, thereby improving the accuracy of value quantification. By incorporating atmospheric dynamic characteristics, the quantitative bias of focusing solely on thermodynamics while neglecting dynamics is corrected, making the value assessment more comprehensive.

[0080] Finally, by combining real-time business attention and historical access patterns, a comprehensive value assessment model is established. For example, is there a major weather event that needs special attention? Which data types or regions are frequently accessed? A quantitative value density score is generated for each data segment.

[0081] Step 3: Dynamically store routes

[0082] This step designs an adaptive data distribution mechanism based on data value density scores and system operating status, enabling intelligent scheduling and migration of data between different storage tiers.

[0083] First, establish a tiered storage architecture that includes memory cache, high-speed solid-state drives, and large-capacity hard disk drives; define the performance indicators, cost coefficients, and load rate monitoring standards for each storage tier, and establish a mapping relationship between value density and storage resources.

[0084] The hierarchical storage architecture design for dynamic storage routing specifically includes:

[0085] Build a three-tier storage system:

[0086] The first level is a memory-based cache: storing the most valuable real-time data and frequently accessed hot data, providing nanosecond-level access latency, defined as a performance metric. Its unit capacity cost coefficient is The load rate is collected in real time through the load monitoring module. .

[0087] The second tier is a performance storage area based on NVMe solid-state drives: storing medium-value historical data and feature products, providing microsecond-level access latency, defined as a performance metric. Its unit capacity cost coefficient is Real-time data collection load rate .

[0088] The third level is a capacity storage area based on high-density hard disk drives: storing low-value raw data and archived data, providing millisecond-level access latency, defined as a performance metric. Its unit capacity cost coefficient is Real-time data collection load rate .

[0089] Intelligent data prefetching mechanism: Based on access pattern prediction and value density assessment, an intelligent data prefetching mechanism is designed to preload high-value data that is likely to be accessed frequently into the high-speed storage layer to reduce user access waiting time.

[0090] Dynamic replica management: Automatically adjust the number of data replicas based on data value scores and access load; create multiple replicas for high-value, frequently accessed data to improve concurrent access performance and data reliability; reduce the number of replicas for low-value, cold data to save storage space.

[0091] Storage resource monitoring: Establish a comprehensive storage resource monitoring system to collect performance metrics of each storage tier in real time. Capacity utilization and load factor This provides real-time feedback for routing decisions, ensuring the accuracy and timeliness of those decisions.

[0092] The specific implementation of the adaptive data distribution mechanism in step three includes:

[0093] A model for matching value density with storage performance: establishing data value scoring and storage tier performance indicators. The correspondence between these ensures that high-value data receives high-performance storage services; that is, data with higher value density (D) is more likely to be allocated to storage tiers with higher performance metrics (P).

[0094] Load balancing algorithm: based on storage resource monitoring load rate We designed a load balancing algorithm to avoid overloading a single storage node and affecting the overall system performance, and to achieve a reasonable distribution of request load among multiple nodes.

[0095] Data lifecycle management: defining the data lifecycle Based on historical access logs, the data access frequency is predicted using a Long Short-Term Memory (LSTM) network model. Based on the time decay of data value, the storage location of the data is automatically adjusted: newly generated high-value data is preferentially stored in the high-speed storage area, while historical data with decaying value is automatically migrated to the low-speed storage area.

[0096] Elastic scaling mechanism: Design an elastic scaling mechanism to dynamically adjust storage resource configuration according to changes in system load; automatically expand high-performance storage capacity during peak access periods and automatically shrink resources during off-peak access periods to reduce operating costs.

[0097] Adjust parameters Define adjustment parameters The value was determined by fitting nearly three months of storage resource monitoring data, and it is used to balance the weight of data value and access frequency in scheduling decisions.

[0098] Data scheduling and migration between different storage tiers are achieved through a multi-objective optimization algorithm. This algorithm aims to optimize data value, storage cost, and access efficiency, and calculates scheduling priorities by considering data lifecycle and predicted access frequency.

[0099] The utility function used in the dynamic routing strategy to evaluate the storage level of data segments is defined by the following mathematical expression:

[0100] ;

[0101] in: Calculate the utility score for storing the data segment in storage level i; Meteorological significance intensity; This refers to the performance metrics of storage level i in a three-tier storage system. is the unit capacity cost coefficient for storage level i; The real-time load rate of storage tier i; This represents the access frequency predicted based on the LSTM model. For the data lifecycle; This refers to the data's lifespan. To adjust the parameters, they were determined by fitting recent storage resource monitoring data, which is used to balance the weight of data value and access frequency in scheduling decisions; This is used to balance data value with storage resource costs, performance, and load, ensuring that high-value data uses high-performance, low-cost, and low-load storage resources. The numerator... High-value data should be matched with high-performance storage, denominator The higher the cost and the fuller the load, the lower the priority of resource adaptation, in order to avoid wasting high-cost resources or causing node overload. The larger the overall ratio, the higher the compatibility between the data and the storage level, and the better the overall benefits of storing data in that level. By integrating business scenario characteristics, we can correct decision-making biases that focus solely on value and storage while ignoring business needs. middle This predicts access frequency; the more frequent the access, the larger the value and the higher the priority. The logarithmic function ensures that the impact of access frequency converges smoothly, avoiding excessive resource consumption by extremely high-frequency data. Smaller data values ​​are closer to expiration, and larger overall values ​​are used to avoid high-frequency data that is about to expire consuming high resources.

[0102] Step 4: Multidimensional Index Construction

[0103] This step establishes a hybrid index structure that supports complex queries based on value assessment results and meteorological characteristic parameters, in order to achieve efficient data location.

[0104] Hybrid index structures built for multidimensional indexes specifically include:

[0105] Spatiotemporal index layer: Geohash encoding is used to discretize spatial locations, converting latitude and longitude coordinates into short strings to enable approximate queries of spatial data; a spatiotemporal joint index is established by combining time segmentation strategy to support efficient data retrieval by latitude and longitude range and time interval; spatiotemporal dimension similarity is calculated and defined based on spatial distance and time difference to measure the degree of spatiotemporal matching between query conditions and data segments.

[0106] Feature Index Layer: An inverted index is built based on meteorological feature tags, such as "inversion layer", "jet stream", and "strong convection". An inverted list containing the identifiers of related data segments is maintained for each feature tag, supporting feature retrieval by meteorological elements and weather phenomena. Feature dimension similarity is calculated based on the number of feature matches, which is used to measure the degree of matching between query conditions and data segments in terms of meteorological features.

[0107] Value Index Layer: Establishes interval indexes based on value density scores; divides data into multiple value intervals according to value density, such as high value, medium value, and low value, and establishes independent index partitions for each interval. It supports data filtering by value level and calculates and defines value dimension similarity based on value density difference to measure the degree of matching between query conditions and data segments in terms of value.

[0108] Topology Index Layer: Builds a topology index based on the spatiotemporal and physical relationships between data; records the adjacency relationships of data segments in the spatiotemporal dimension, such as the next time step or adjacent spatial location of a certain sounding data segment, and the causal relationships in physical processes, such as the relationship between the formation of a certain stratum and the upstream air mass. It supports complex data association queries, such as querying the evolution process of a certain frontal system in different time periods.

[0109] The similarity weights of spatiotemporal similarity, feature similarity, and value similarity are calibrated through meteorological business query scenarios; index access performance is improved through index compression and caching optimization techniques, such as differential encoding or variable-length encoding of inverted lists, and caching frequently accessed index blocks in memory.

[0110] Step 5: Intelligent Search Service

[0111] This step receives user query requests, parses the query semantics and business intent, performs rapid data filtering based on a hybrid index structure, and optimizes and sorts the search results by combining value weights and feature matching degrees.

[0112] When a user submits a query request, the intelligent retrieval service module first parses the query semantics and business intent, converting them into executable query conditions; then, based on the hybrid index structure built in step four, it performs rapid data filtering to locate candidate data segments that meet the query conditions.

[0113] To provide smarter results that better meet user needs, the search results are optimized and ranked by combining value weights and feature matching degrees.

[0114] In the multi-objective ranking function, the final ranking score of the candidate data segment is determined by the following mathematical expression:

[0115] ;

[0116] in The final sorting score for data segment d; Let be the similarity in the j-th dimension; The weights set for the similarity; The meteorological significance intensity of data segment d; The maximum meteorological significance in the current query result set; This represents the average meteorological significance level in the current query result set. The standard deviation of the significance level; To adjust the parameters, calibration was performed using recent user search feedback data; To ensure that the sorting results first meet the user's query requirements, the matching degree is calculated based on the multi-dimensional similarity of the hybrid index. Once the matching accuracy meets the standards, prioritize high-value data that aligns with core business needs. This sorting function is used to correct the impact of extreme data on sorting, making the sorting results more balanced and more in line with actual business scenarios. It comprehensively considers the multi-dimensional similarity between the data segment and the query conditions, as well as the business value of the data segment itself, so that high-value and highly matched data can be presented to users first.

[0117] The technical scope of this invention is not limited to the content described above. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the protection scope of this invention.

Claims

1. An optimized method for storing and retrieving BeiDou radiosonde observation data, characterized in that, Includes the following steps: Step 1: Data Value Perception: Receive the observation data stream uploaded by the radiosonde in real time through the BeiDou satellite data transmission channel, decode and analyze the raw data, and extract complete observation information including but not limited to precise timestamps, three-dimensional spatial coordinates, and meteorological elements such as temperature, air pressure, humidity, wind direction, and wind speed; use a sliding window quality control algorithm to detect outliers and verify consistency of the data. Step 2: Value Quantification Assessment: Based on meteorological dynamics principles and operational application needs, establish a feature extraction and value assessment system; identify key strata and weather phenomena through multi-scale analysis of vertical profiles; and establish a comprehensive value assessment model by combining real-time operational attention and historical access patterns to generate a quantified value density score for each data segment. Step 3: Dynamic Storage Routing: Based on the value density score and system operating status, design an adaptive data distribution mechanism; establish a tiered storage architecture including memory cache, high-speed solid-state drives, and large-capacity hard disk drives, and establish a mapping relationship between value density and storage resources; realize the scheduling and migration of data between different storage tiers through multi-objective optimization algorithms; Step 4: Multidimensional Index Construction: Based on the value assessment results and meteorological feature parameters, establish a hybrid index structure that supports complex queries; establish a data location mechanism by combining spatiotemporal partition index, meteorological feature inverted index, and value range index; improve index access performance through index compression and caching optimization techniques, and define similarity calculation rules and weight allocation standards for each index dimension; Step 5: Intelligent Search Service: Receive user query requests and parse the query semantics and business intent; Rapid data filtering is achieved based on a hybrid index structure, and the search results are optimized and sorted by combining value weights and feature matching degrees.

2. The method for optimizing the storage and retrieval of BeiDou radiosonde observation data according to claim 1, characterized in that: The quality control algorithm for data value perception in step one specifically includes: Cyclic redundancy check and format compliance check are performed on the original BeiDou messages. Hamming code error correction algorithm is used to repair bit errors in the transmission process. Data sources with a check failure rate exceeding the threshold are marked and isolated. A sliding window outlier detection algorithm based on statistical principles is used to detect anomalies in temperature, air pressure, and humidity sequences in real time. The interquartile range method is used to identify and remove observations that deviate significantly from the normal range. The inherent correlation between data from different sensors is verified through a physical consistency test algorithm. This includes checking physical constraints, including but not limited to temperature-dew point difference and wind pressure relationship, and marking suspicious data points that do not conform to atmospheric physical laws. Establish a data quality assessment index system, including completeness, accuracy, consistency, and timeliness, and generate a quality score for each data segment as input parameters for subsequent value assessment.

3. The method for optimizing the storage and retrieval of BeiDou radiosonde observation data according to claim 1, characterized in that: The feature extraction for value quantification assessment in step two specifically includes: Vertical structure feature extraction: Calculate thermodynamic parameters including but not limited to temperature lapse rate, potential temperature gradient, virtual potential temperature, and equivalent potential temperature; identify stable stratifications including but not limited to the top of the troposphere, inversion layer, isotherm, and mixed layer; and analyze the strength, thickness, and vertical distribution characteristics of the stratifications. Dynamic feature extraction: Calculates dynamic parameters including but not limited to geostrophic wind, thermal wind, Richardson number, vorticity, divergence, and vertical wind shear; identifies dynamic features including but not limited to jet core region, frontal region, shear line, and vortex system; and analyzes the stability of atmospheric motion and energy conversion processes. Water vapor feature extraction: Calculate water vapor parameters including but not limited to specific humidity, mixing ratio, relative humidity, water vapor flux, and precipitable water; identify water vapor features in the dry layer, wet layer, and cloud layer; and analyze the vertical distribution and transport characteristics of water vapor. Cloud physics feature extraction: Based on temperature and humidity parameters, calculate cloud base height, cloud top height, and cloud thickness parameters, identify different types of cloud structures, and analyze the relationship between cloud physics processes and weather systems.

4. The method for optimizing the storage and retrieval of BeiDou radiosonde observation data according to claim 3, characterized in that: The identification and quantification of key layers in step two specifically includes: Wavelet transform and multi-scale analysis methods are used to enhance the local features of the vertical profile. Abrupt points and inflection points in the profile are identified by modulus maxima detection, and the boundary positions of key layers are accurately located. Based on the sliding correlation algorithm, the coordinated variation characteristics of meteorological elements at different altitudes are analyzed. The stratification interface is identified by the sharp changes in the correlation coefficient, and the vertical range and internal structure of the stratification are determined. The vertical structure of the atmosphere is divided into several relatively uniform layers based on the similarity and continuity of meteorological elements, and stratification units with specific meteorological significance are identified. For each identified key stratification, its physical intensity index, vertical extension range, duration, and climate anomaly characteristic parameters are calculated to establish a quantitative description system for stratification characteristics.

5. The method for optimizing the storage and retrieval of BeiDou radiosonde observation data according to claim 1, characterized in that: The hierarchical storage architecture design for dynamic storage routing in step three specifically includes: The storage system is structured in three tiers: Tier 1 is a memory-based high-speed cache that stores the most valuable real-time data and frequently accessed hot data, providing nanosecond-level access latency; Tier 2 is a performance storage area based on NVMe solid-state drives that stores medium-value historical data and feature products, providing microsecond-level access latency; and Tier 3 is a capacity storage area based on high-density hard disk drives that stores low-value raw data and archived data, providing millisecond-level access latency. Design an intelligent data prefetching mechanism that, based on access pattern prediction and value density assessment, preloads high-value data that is likely to be accessed frequently into the high-speed storage layer, reducing user access waiting time. Dynamic replica management automatically adjusts the number of data replicas based on data value scores and access load. Multiple replicas are created for high-value hot data to improve concurrent access performance, while the number of replicas is reduced for low-value cold data to save storage space. Establish storage resource monitoring and collect performance metrics for each storage tier in real time. Capacity utilization and load status It provides real-time feedback information for routing decisions, among which These correspond to the first, second, and third levels of storage, respectively. These correspond to the first, second, and third levels of storage, respectively.

6. The method for optimizing the storage and retrieval of BeiDou radiosonde observation data according to claim 5, characterized in that: The specific implementation of the adaptive data distribution mechanism in step three includes: Establish a matching model between value density and storage performance, and establish data value scoring and storage tier performance indicators. The correspondence between them ensures that high-value data receives high-performance storage services; Design a load balancing algorithm to monitor the real-time load rate of each storage node. This avoids overloading a single node and affecting the overall system performance, and achieves a reasonable distribution of request load among multiple nodes; Data lifecycle management defines the data lifecycle, predicts the data access frequency based on historical access logs using an LSTM model, and automatically adjusts the data storage location according to the time decay law of data value. Newly generated high-value data is prioritized to be stored in the high-speed storage area, while historical data with decaying value is automatically migrated to the low-speed storage area. The system is designed with an elastic scaling mechanism to dynamically adjust storage resource configuration based on system load changes. During peak access periods, high-performance storage capacity is automatically expanded, and during off-peak access periods, resources are automatically contracted to reduce operating costs.

7. The method for optimizing the storage and retrieval of BeiDou radiosonde observation data according to claim 6, characterized in that: For the hybrid index structure constructed in step four, multi-dimensional similarity is defined, specifically including: Spatiotemporal index layer: Geohash encoding is used to discretize spatial locations, and a spatiotemporal joint index is established by combining time segmentation strategy. It supports efficient data retrieval by latitude and longitude range and time interval, and defines spatiotemporal dimension similarity based on spatial distance and time difference. Feature Index Layer: An inverted index is built based on meteorological feature labels. An inverted list containing the identifiers of related data segments is maintained for each feature label. Feature retrieval by meteorological elements and weather phenomena is supported. Feature dimension similarity is calculated based on the number of feature matches. Value Index Layer: Based on value density scores, an interval index is built, dividing the data into multiple value intervals according to value density. An independent index partition is built for each interval, supporting data filtering by value level. Value dimension similarity is defined based on the value density difference. Topology Index Layer: Based on the spatiotemporal and physical relationships between data, a topology index is built to record the adjacency relationships of data segments in the spatiotemporal dimension and the causal relationships in the physical process, supporting complex data association queries.

8. The method for optimizing the storage and retrieval of BeiDou radiosonde observation data according to claim 7, characterized in that: The meteorological significance intensity of the key strata is quantified using the following mathematical expression: ； in: The meteorological significance intensity of the key strata; The potential temperature gradient of the k-th layer reflects the stability of the stratification. This is a highly weighted coefficient; Let be the relative humidity of the k-th layer; The average relative humidity; The horizontal wind speed reflects the dynamic effect; For reference wind speed, 1.5 times the average wind speed in recent years is used; and The parameters were determined by fitting historical sounding data to adjust them; It is a hyperbolic tangent function to avoid excessive interference with intensity assessment when the wind speed is too high.

9. The method for optimizing the storage and retrieval of BeiDou radiosonde observation data according to claim 8, characterized in that: The utility function used in dynamic routing strategies to evaluate the storage level of data segments is defined by the following mathematical expression: ； in: Calculate the utility score for storing the data segment in storage level i; Meteorological significance intensity; This refers to the performance metrics of storage level i in a three-tier storage system. is the unit capacity cost coefficient for storage level i; The real-time load rate of storage tier i; This represents the access frequency predicted based on the LSTM model. For the data lifecycle; This refers to the data's lifespan. To adjust the parameters, they are determined by fitting recent storage resource monitoring data, which is used to balance the weight of data value and access frequency in scheduling decisions.

10. The method for optimizing the storage and retrieval of BeiDou radiosonde observation data according to claim 8, characterized in that: In the multi-objective ranking function, the final ranking score of the candidate data segment is determined by the following mathematical expression: ； in The final sorting score for data segment d; Let be the similarity in the j-th dimension; The weights set for the similarity; The meteorological significance intensity of data segment d; The maximum meteorological significance in the current query result set; This represents the average meteorological significance level in the current query result set. The standard deviation of the significance level; To adjust the parameters, calibration was performed using recent user search feedback data.

Images