Multidimensional data index optimization method for hydropower big data storage

Abstract

The invention relates to a multidimensional data index optimization method for hydropower big data storage. The method comprises the following steps: data access and feature extraction: extracting physical dimension and service dimension attributes of time sequence data; dynamic weight calculation: calculating the comprehensive weight of the dimension combination in real time based on the query frequency, the data selectivity and the service priority; constructing and updating a hierarchical fusion index, constructing an adaptive grid partition R-tree main index and an inverted auxiliary index according to the weight, and establishing bidirectional pointer association; routing and execution are intelligently queried, an optimal index path is selected according to the weight, and data are quickly positioned and merged through a pointer; online monitoring and self-adaptive adjustment are carried out, data distribution and query performance are monitored in real time, and weight recalculation and index increment recombination are triggered; the method further comprises storage management of life cycle perception, and differential index strategies are implemented on hot, warm and cold data; the method can dynamically adapt to query load changes, improve the multi-dimensional query efficiency and reduce the storage cost.

Description

Technical Field

[0001] This invention belongs to the field of database indexing technology, specifically relating to a multi-dimensional data indexing optimization method for hydropower big data storage, which is particularly suitable for dynamic index construction, query optimization and storage management of multi-source heterogeneous time-series data in smart hydropower scenarios. Background Technology

[0002] With the continuous deepening of smart hydropower and watershed digitalization construction, massive amounts of data are continuously generated and accumulated during hydropower production and operation. These data have distinct characteristics of multi-source heterogeneity, originating from monitoring systems, hydrological forecasting systems, equipment status monitoring systems, and information management systems. Their typical feature is strict temporal sequence, and they naturally contain multiple dimensions, including timestamps, spatial locations, equipment identifiers, operating conditions, and various physical parameters.

[0003] Efficient storage and rapid retrieval of this multidimensional data are the core technological foundation for real-time equipment status assessment, accurate fault early warning, optimized power generation scheduling, and intelligent management decision-making. Faced with such needs, traditional single-dimensional index structures such as B-trees are no longer sufficient to handle complex multi-condition combination queries. Existing technologies typically turn to multi-dimensional spatial indexes, such as R-trees and their variants, or design hybrid index strategies to improve query performance.

[0004] However, when these general solutions are applied to the specific business area of ​​hydropower, several limitations have gradually been exposed. First, the various dimensions of hydropower data are not independent, but are closely related by business logic. For example, the vibration data of the generating unit is only meaningful when analyzed in conjunction with specific operating conditions, time periods, and head parameters. General indexes treat these as unrelated dimensions, which leads to multiple index jumps and complex data merging calculations when executing queries involving multi-dimensional relationships, resulting in low efficiency.

[0005] Secondly, hydropower data streams exhibit significant periodicity, trends, and suddenness. For example, the data generation rate and query patterns differ greatly between the flood season and the non-flood season, and between peak shaving and stable operation periods. Most existing index structures are statically built or only support limited offline reorganization, making it impossible to make online and adaptive adjustments based on the dynamic changes in data distribution and query hotspots, resulting in significant fluctuations in index performance with the business cycle.

[0006] In addition, to accelerate multidimensional queries, a common technique is to build multiple independent indexes or composite indexes containing a large amount of redundant data. While this can improve query speed to some extent, it leads to a sharp increase in storage costs and introduces huge maintenance overhead when updating data, which does not conform to the economic principle of big data storage.

[0007] Finally, existing methods generally lack consideration for the differences in value throughout the entire data lifecycle; using the same high-cost indexing strategy for frequently accessed real-time monitoring data and rarely queried historical archive data results in a waste of valuable storage and computing resources; therefore, there is an urgent need in this field for a multi-dimensional data indexing method that can deeply integrate the characteristics of hydropower business, has dynamic adaptive capabilities, and achieve an optimized balance between storage costs and query performance. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a multi-dimensional data indexing optimization method for hydropower big data storage, comprising the following steps:

[0009] S1: Data Access and Feature Extraction: Real-time access to multi-source heterogeneous time-series data streams in the hydropower field. For each data object, extract its physical dimension attributes and business dimension attributes. The physical dimension attributes include timestamps and spatial location information. The business dimension attributes include equipment identifiers, operating status codes, and operating parameters. S2: Dynamic Weight Calculation: Continuously collect historical query logs, analyze query load patterns, and dynamically calculate and update the comprehensive weights of each dimension and dimension combination based on query frequency, data selectivity, and predefined business priorities. S3: Hierarchical Fusion Index Construction and Update: Based on the dynamic weights at the current moment, construct and incrementally update the hierarchical fusion index structure.

[0010] The main index is constructed using a grid partitioning R-tree structure based on dynamic weights and adaptive values. The node partitioning strategy of the R-tree is dynamically adjusted according to the single spatiotemporal dimension with the highest weight.

[0011] S4: Intelligent query routing and execution: Receive user query requests and parse the dimensions in the query conditions; select the query path that prioritizes traversing the main index and corresponding auxiliary indexes based on the real-time weight of the dimensions; quickly locate related data through the bidirectional pointers, and merge and filter query results; S5: Online monitoring and adaptive adjustment: Monitor the offset of data distribution and query performance indicators in real time; when the change exceeds a preset threshold, trigger root cause diagnosis and perform the adjustment of the dynamic weights and the online, incremental reorganization of the hierarchical fusion index structure based on the diagnosis results.

[0012] Preferably, in step S1, for each data object, the extracted physical dimension attributes include a timestamp accurate to milliseconds and three-dimensional spatial coordinates composed of longitude, latitude, and elevation; the extracted business dimension attributes include a device code used to uniquely identify the source of the device, a working condition category code divided according to preset rules, and the values ​​of voltage, current, flow rate, and pressure parameters obtained from real-time monitoring.

[0013] Preferably, in S2, the dimension query frequency is obtained by statistically analyzing the number of conditional constraint operations on a single dimension and dimension combinations within a specified time window; the data selectivity is evaluated based on the distribution dispersion of dimension values ​​within their value range; the predefined business priority is associated with the equipment importance level and the operational security level; based on the aforementioned factors, a comprehensive weight value reflecting query popularity and filtering effectiveness is obtained in real time through a weighted calculation model.

[0014] Furthermore, the process of establishing the weighted calculation model is as follows:

[0015] The system takes continuously collected historical query logs, dimensional value range distribution statistics, and preset business priority rules as inputs. The processing first normalizes the query frequency value, data selectivity evaluation value, and business priority value of each dimension and dimension combination. Then, it performs linear weighted summation based on the coefficients dynamically set by domain expert experience and reinforcement learning mechanism. The output is a real-time, quantitative comprehensive weight score for each dimension and dimension combination. The comprehensive weight score is directly used to guide the selection of splitting dimensions for nodes within the index structure and the arrangement order of dimensions in the composite index key.

[0016] Preferably, in S3, the construction and updating of the hierarchical fusion index structure are based on the comprehensive weight score obtained in S2;

[0017] The main index adopts a grid-partitioned R-tree structure. When dividing the tree nodes in space, the granularity and level of the grid division are adaptively determined according to the weight score corresponding to the current high-weight spatiotemporal dimension. The auxiliary index is an inverted index built for high-weight business dimensions. Each inverted list is associated with a bidirectional pointer to the leaf node data block in the main index R-tree that stores the corresponding dimension value data object. The leaf node data block records the set of data objects and their corresponding time range and physical storage location.

[0018] Preferably, in S4, each dimension and combination of dimensions in the query conditions will be matched with the real-time comprehensive weight updated in S2; the query route will first traverse the primary and secondary indexes corresponding to the dimension with the highest weight, and quickly associate and reference related data sets in other indexes through the bidirectional pointers established in S3; subsequently, the data sets obtained from different index paths will be subjected to union or intersection operations based on their timestamps and spatial locations to complete the merging and final filtering of the results.

[0019] Preferably, in S5, the offset of the data distribution is quantified by statistically analyzing the rate of change of the value distribution of newly added data in key time and business dimensions. The query performance indicators include average query latency and index hit rate. During monitoring, the above-mentioned rate of change and performance indicators are compared with their respective preset static and dynamic thresholds. When any indicator continuously exceeds the threshold, a trigger signal is generated. The trigger signal then initiates root cause diagnosis. The root cause diagnosis performs correlation analysis on the anomaly to locate the core dimension causing the anomaly. Subsequently, based on the diagnosis results, directed weight adjustment and index reorganization are triggered: if the diagnosis points to a specific dimension, the weights of the relevant dimension are adjusted in a targeted manner and a local index reorganization involving the data of that dimension is triggered; if the diagnosis indicates widespread changes, dynamic weight recalculation as described in S2 and index reorganization within the corresponding range are triggered. The reorganization is performed online at the data block level.

[0020] As a preferred option, the multi-dimensional data index optimization method for hydropower big data storage also includes: S6: Lifecycle-aware index storage management, which marks data objects as hot data, warm data, or cold data based on the data access frequency, generation time, and business criticality; for the hot data, maintain its complete hierarchical fusion index in high-speed storage media; for the warm data, maintain its coarse-grained structure of the main index and some key auxiliary indexes in disk storage; for the cold data, archive it to low-cost storage, and extract its core dimension values ​​to form a lightweight metadata index in columnar storage format, and reconstruct part of the index as needed based on the metadata index when a query is triggered.

[0021] Preferably, in S6, the labeling of data objects is determined based on their access frequency within the sliding time window, the time elapsed since the current time, and a predefined business criticality level. For data objects labeled as hot data, their complete R-tree primary index structure and inverted secondary index reside on high-speed storage media. For warm data, the primary index retains upper-level coarse-grained nodes on disk to reduce space usage, while the secondary index retains the portion for high-frequency query dimensions. For cold data, while archiving to low-cost storage, their core dimension values ​​are extracted and stored column-wise to form a lightweight metadata index. When subsequent queries hit this part of the data, the local index structure of the query-required dimensions is temporarily reconstructed in memory as needed based on the metadata index.

[0022] The beneficial effects of this invention are as follows:

[0023] The invention continuously collects historical query logs and dynamically calculates the comprehensive weights of each dimension and dimension combination, enabling the core index structure to perceive and adapt to changes in business query patterns in real time. This method abandons the fixed index design, solves the problem of general indexes being insensitive to the dynamics of water and electricity business, and ensures the continuous efficiency and stability of system performance in various scenarios, from daily monitoring to special scheduling.

[0024] A hierarchical and integrated index architecture is adopted, which uses a grid partition R-tree based on dynamic weights as the main index and combines it with an inverted auxiliary index built for high-weight business dimensions, and achieves efficient association through bidirectional pointers. This structure, together with an intelligent query routing mechanism based on real-time weights, can automatically select the optimal retrieval path for different multi-dimensional combination queries, which significantly reduces unnecessary disk input / output operations and complex data merging calculations, thereby greatly improving the response speed of complex queries.

[0025] A lifecycle-aware index storage management mechanism was introduced, which classifies data into different levels such as hot, warm, and cold based on data access frequency, generation time, and business criticality, and applies differentiated indexing strategies accordingly. In particular, for cold data, only a lightweight metadata index based on columnar storage is retained, which greatly saves storage space while ensuring its queryability. This enables high-performance index resources to be accurately allocated to high-value data, effectively reducing the total cost of ownership for long-term storage of massive amounts of data.

[0026] By employing online monitoring and adaptive adjustment steps, real-time tracking of data distribution shifts and query performance metrics is achieved. A progressive root cause diagnosis mechanism and differentiated reorganization strategies are also introduced. This addresses the resource waste and inaccurate adjustments caused by the direct and rigid binding of monitoring signals and adjustment actions in traditional adaptive indexing. By embedding intelligent diagnostic steps before triggering reorganization, anomalies are analyzed in detail to accurately pinpoint the core dimensions causing the changes. Based on the diagnostic conclusions, a gradient-based precise response is implemented: for localized problems, temporary weight shifts and highly targeted local index subtree reorganization are used for rapid intervention; global adjustments are only initiated when the diagnosis indicates widespread changes. This method upgrades the index's adaptive process from a phenomenon-based, coarse-grained passive response to a root cause-based, precise, proactive, and targeted optimization. Ultimately, while ensuring the index structure maintains optimal fit, the system overhead of the reorganization operation itself is reduced. Furthermore, through multi-version control technology at the data block level, the efficiency, stability, and continuity of online services throughout the self-optimization process are ensured, achieving the best balance between optimization costs and benefits.

[0027] This invention incorporates business priorities defined by domain knowledge into the dynamic weight calculation model, enabling query requests for important equipment or safety-critical data to receive priority processing resources. At the same time, by finely extracting and utilizing multi-dimensional attributes such as millisecond-level timestamps, three-dimensional spatial coordinates, precise equipment codes, and operating status codes, it improves the accuracy of data organization and the targeting of queries, reflecting the optimization design driven by domain knowledge. Attached Figure Description

[0028] To better understand and implement this application, the technical solution is described in detail below with reference to the accompanying drawings.

[0029] Figure 1 This is a schematic diagram of the main process of a multidimensional data indexing optimization method for hydropower big data storage provided in Embodiment 1 of this application;

[0030] Figure 2 This is a flowchart illustrating step six, lifecycle-aware index storage management, of a multidimensional data index optimization method for hydropower big data storage provided in Embodiment 1 of this application. Detailed Implementation

[0031] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, exemplary embodiments will be described in detail below, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0032] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0033] The following detailed description of the specific implementation methods, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided in detail.

[0034] Example 1

[0035] Please see Figures 1-2 This embodiment provides a multi-dimensional data indexing optimization method for hydropower big data storage, including the following steps:

[0036] S1: Data access and feature extraction: Real-time access to multi-source heterogeneous time-series data streams in the hydropower field; for each data object, extract its physical dimension attributes and business dimension attributes; the physical dimension attributes include timestamp and spatial location information; the business dimension attributes include equipment identifier, operating status code and operating parameters.

[0037] Furthermore, the data access and feature extraction steps are executed through a stream processing service deployed on the data aggregation layer. The accessed multi-source heterogeneous time-series data streams originate from multiple business areas such as production monitoring, water situation forecasting, equipment status monitoring, and management information. During the access process, the data is parsed and formatted in real time. For each data object, the extraction of physical dimension attributes includes: parsing the original acquisition time accurate to milliseconds from the data packet and converting it into a long integer timestamp based on standard time; simultaneously, based on the physical installation location of the measuring point or equipment, querying and extracting its corresponding geographical latitude and longitude coordinates and altitude information from a pre-set asset ledger to jointly constitute a spatial location description; the extraction of business dimension attributes includes: parsing or generating a globally unique equipment identifier from the original data according to predefined asset coding rules; mapping the original state signal to a standardized working condition enumeration code by matching it with a pre-set working condition judgment rule base; in addition, directly extracting or simply converting the original values ​​of physical parameters characterizing the equipment's operating status, such as voltage, current, active power, flow rate, pressure, temperature, and vibration amplitude.

[0038] Specifically, the data access and feature extraction steps address the challenges of real-time integration of multi-source hydropower data and the low standardization of feature extraction. Through stream processing services, unified parsing and format standardization of data streams from multiple business domains are achieved, transforming previously scattered and heterogeneous raw data into standardized data objects with a unified spatiotemporal benchmark and standardized business semantics. In the physical dimension, a high-precision time benchmark and quantifiable three-dimensional spatial coordinates are provided, laying the foundation for accurate spatiotemporal retrieval. In the business dimension, a globally unique equipment identification system, a refined chemical condition classification based on multi-rule judgment, and a complete parameter set covering core operational health states are established. This enables comprehensive, refined, and structured description of data features, providing an accurate, consistent, and information-rich dimensional data foundation for the subsequent construction of high-performance multidimensional indexes.

[0039] S2: Dynamic weight calculation, continuously collects historical query logs, analyzes query load patterns, and dynamically calculates and updates the comprehensive weight of each dimension and dimension combination based on the query frequency, data selectivity, and predefined business priority of each dimension and dimension combination.

[0040] The dynamic weight calculation step is executed by the background analysis service, which subscribes to and processes the structured logs generated by the query engine in real time; it analyzes the query load pattern by parsing the logs and identifies the dimension filtering conditions, range conditions and correlations contained therein; the dynamic calculation is executed in a configurable period or event-triggered manner, and the calculation results are persistently stored in a distributed cache for real-time access by the index building and query routing components;

[0041] Preferably, in step S2, the dimension query frequency is obtained by statistically analyzing the number of conditional constraint operations on a single dimension and dimension combinations within a specified time window; the data selectivity is evaluated based on the distribution dispersion of dimension values ​​within their value range; the predefined business priority is associated with the equipment importance level and the operational security level; based on the aforementioned factors, a comprehensive weight value reflecting query popularity and filtering effectiveness is obtained in real time through a weighted calculation model.

[0042] The specified time window is typically set to the most recent 24 hours or one week, and is updated using a sliding window mechanism; the statistics of the number of condition constraint operations distinguish between different operators such as equal to, greater than, less than, and between; the evaluation of data selectivity is specifically achieved by calculating the Gini coefficient or entropy value of the probability distribution of all different values ​​in that dimension. The more concentrated the value distribution, the lower the selectivity; the equipment importance level is predefined based on the criticality of the equipment in power generation, while the operation safety level is defined based on the severity of the consequences that equipment failure may cause; in each calculation cycle, the weighted calculation model first retrieves the latest query frequency statistical histogram, the latest value distribution statistical summary of each dimension and dimension combination, and the business priority mapping table read from the configuration center from the cache as the input source for this calculation;

[0043] Furthermore, the process of establishing the weighted calculation model is as follows:

[0044] The system takes continuously collected historical query logs, dimensional value range distribution statistics, and preset business priority rules as input. The processing first normalizes the query frequency value, data selectivity evaluation value, and business priority value of each dimension and dimension combination. Then, it performs linear weighted summation based on domain expert experience and coefficients dynamically set by reinforcement learning mechanism. The output is a real-time, quantitative comprehensive weight score for each dimension and dimension combination. The comprehensive weight score is directly used to guide the selection of splitting dimensions for nodes within the index structure and the arrangement order of dimensions in the composite index key.

[0045] Furthermore, the normalization process employs a min-max scaling method, mapping the original frequency value, Gini coefficient value, and priority level value of each dimension to a range of zero to one. Domain expert experience is embodied in an initial set of static coefficient configurations, while the reinforcement learning mechanism is implemented through an online agent. This agent uses the change in query latency before and after index reorganization as a reward signal, continuously adjusting the aforementioned linear weighting coefficients to pursue long-term query performance gains. The formula for calculating the linear weighted sum is: the weight score equals the frequency normalized value multiplied by coefficient A, plus one minus the selective normalized value multiplied by coefficient B, plus the priority normalized value multiplied by coefficient C, where the sum of coefficients A, B, and C is one and is dynamically adjusted by the reinforcement learning agent. The final output comprehensive weight score is a floating-point number, with higher-scoring dimension combinations assigned a higher priority in the index structure.

[0046] Specifically, the dynamic weight calculation step solves the problem that traditional indexes, due to their fixed weights, cannot adapt to dynamically changing query loads and business needs. By analyzing query logs in real time and quantitatively evaluating dimension query frequency, data filtering efficiency, and business importance, a comprehensive weight that truly reflects the current query pattern and business focus is dynamically calculated. This process transforms the index structure's organizational principle from static preset to continuous self-adjustment, achieving the effect of guiding index resources to accurately tilt towards high-value query dimensions. It provides a real-time, quantitative decision-making basis for building high-performance indexes with adaptive capabilities, thereby optimizing overall query performance and the rationality of resource utilization.

[0047] S3: Hierarchical fusion index construction and update, based on the dynamic weight at the current moment, construct and incrementally update the hierarchical fusion index structure; the hierarchical fusion index construction and update steps are executed by the index management service, which triggers the construction of new indexes or incremental adjustment of existing indexes according to the latest comprehensive weight score stored in the distributed cache; the whole process is designed to be carried out online, and avoids blocking concurrent read and write operations through copy-on-write technology;

[0048] The main index is constructed using a dynamically weighted, adaptive grid partitioning R-tree structure. The node partitioning strategy of the R-tree is dynamically adjusted based on the single spatiotemporal dimension with the highest current weight. During implementation, the average weight scores of the time and spatial dimension sets are compared first, and the dimension with the higher score is selected as the priority partitioning criterion. If the time dimension has a higher weight, a first-level partition is performed along the time axis, for example, segmenting the data by hour, day, or month. If the spatial dimension has a higher weight, a first-level grid partition is performed based on the geographic coordinate range. After the selected priority dimension partitioning is completed, the data within each partition is further partitioned based on the weight of another dimension, thus forming a multi-level grid. Each grid cell corresponds to a leaf node in the R-tree, which contains the set of identifiers for all data objects falling within that spatiotemporal range. The intermediate nodes of the R-tree are used to aggregate the spatiotemporal ranges of their child nodes.

[0049] The specific data structure definition of a grid partitioning R-tree node is as follows: Node type identifies whether it is an intermediate node or a leaf node; node level indicates its depth in the tree, with the root node having a level of zero; the current partition dimension records the dimension used for subspace partitioning, such as time or spatial coordinate axes; the current partition threshold records the specific value or value range of the partition point; the node data range is a multi-dimensional boundary description, including the minimum and maximum timestamp values, the boundary rectangle of the spatial coordinates, and a summary of the key values ​​of the covered business dimensions; intermediate nodes store the union of the data ranges of all their child nodes, as well as a list of pointers to these child nodes; leaf nodes store a list of pointers to physical data blocks, which contain the actual collection of data objects.

[0050] One or more auxiliary indexes are constructed for a single business dimension with high weight, using an inverted index structure, and bidirectional pointer associations are established with the leaf node data blocks of the main index. The construction of auxiliary indexes is for business dimensions with weight scores exceeding a preset threshold, such as specific working condition status codes or key equipment identifiers. The key of the inverted index is the specific value of the dimension, and the specific value is a list of identifiers of the leaf node data blocks of the main index containing the data object with that value. The bidirectional pointer association is implemented by recording the inverted keys involved in the metadata of the leaf node data blocks, and storing the physical address and internal offset of the leaf node data blocks in the entries of the inverted list.

[0051] The above process describes the basic mechanism by which the index grows dynamically as data is inserted; to further optimize the index structure to adapt to changes in query load and data distribution, the following incremental reorganization algorithm is introduced;

[0052] The specific algorithm flow for building and updating the main index is as follows: Based on the latest dynamic weight scores, the index management service first determines the priority partitioning dimensions. For the time dimension, the algorithm finds a split point within the time value domain to ensure that the number of data objects in the two sub-partitions is as balanced as possible, while also considering the distribution density of timestamps. For the spatial dimension, the algorithm performs similar operations on the selected coordinate axes or adopts a recursive two-dimensional grid subdivision strategy. The granularity of the partitioning is determined by the weight scores and the total amount of data in the current partition, calculated through a predefined heuristic function to ensure that high-weight dimensions receive finer partitioning. The tree structure grows dynamically based on data insertion, initially as a single node, and splits layer by layer to form a multi-level grid as data grows.

[0053] The incremental reorganization algorithm for grid partitioned R-trees is activated under the following triggering conditions: when the monitoring module detects that the query latency of a certain subtree region continuously exceeds the threshold or the index hit rate drops significantly; when new data causes a significant change in the distribution of values ​​in a certain dimension, such as the KL divergence exceeding the threshold or the increase in data volume causing uneven node load; when the dynamic weight calculation results show that the change in the weight of a certain dimension exceeds the preset threshold and the partitioning strategy needs to be adjusted; and during regular maintenance triggered during periods of low system load.

[0054] Data migration and node adjustment include splitting and merging operations: Splitting is triggered when the size of a leaf node's data block reaches its upper limit or the number of child nodes in an intermediate node exceeds a threshold. The algorithm finds the optimal splitting point based on the current data distribution of the high-weight dimension, divides the data into two sets, creates new nodes, and updates the parent node pointer. Merging is triggered when the total data volume of adjacent nodes is below a threshold and the data ranges overlap significantly. After merging, the node data range is recalculated and the parent node is updated recursively upwards. Data migration is performed using copy-on-write, and the old version is retained until it is no longer referenced and then asynchronously recycled, ensuring that concurrent read and write operations are not affected.

[0055] Pointer updates and associated maintenance cover internal pointer updates, bidirectional pointer synchronization, and metadata updates: after a node split or merge, the list of pointers from the parent node to the child node is updated and the node data range summary is recalculated; if the reorganization involves leaf node data blocks, the bidirectional pointers in the associated inverted secondary index are updated synchronously to ensure cross-index reference consistency; at the same time, the metadata of the leaf node data blocks is updated, including time range, spatial boundaries, and reverse links to the inverted list.

[0056] Tree balancing ensures height and load balance: Tree height is maintained through splitting and merging operations to prevent degradation into a chain-like structure; data distribution across nodes is optimized to prevent hotspot nodes; a multi-version concurrency control mechanism is employed to allow concurrent read and write operations during reorganization, ensuring service continuity; the algorithm executes online at the data block level, achieving smooth reorganization through copy-on-write and multi-version control, and automatically checks pointer and range consistency before and after reorganization, ensuring the index structure remains efficient and stable during dynamic adjustments.

[0057] Preferably, in S3, the construction and updating of the hierarchical fusion index structure are based on the comprehensive weight score obtained in S2;

[0058] The main index adopts a grid-partitioned R-tree structure. When the tree nodes are spatially partitioned, the granularity and level of the grid partitioning are adaptively determined based on the weight score corresponding to the current high-weight spatiotemporal dimension. The adaptiveness is reflected in the fact that the partitioning granularity is positively correlated with the weight score. The higher the weight score, the finer the partitioning, that is, the smaller the spatiotemporal range of the grid cell, and the more likely the tree level will increase. The partitioning threshold is dynamically calculated through a heuristic function based on data density and weight score.

[0059] Furthermore, the auxiliary index is an inverted index built for high-weight business dimensions. Each inverted list is linked to a leaf node data block in the main index R-tree that stores the corresponding dimension value data object via a bidirectional pointer. This bidirectional pointer association allows for quick location of the main index storage unit containing the target data from the auxiliary index. Conversely, it also allows for quick identification of which business dimension indexes reference the data from the leaf nodes of the main index. The leaf node data block records the set of data objects and their corresponding time range and physical storage location. The leaf node data block is the basic unit of persistent storage, storing the unique identifier, precise timestamp, spatial coordinates, and associated business dimension value of the data object in a column-compressed format. The metadata in the data block header records the minimum and maximum timestamp values, spatial coordinate bounding boxes, and a list of reverse link pointers pointing to the associated inverted lists.

[0060] The core data operation process is as follows: Data insertion, based on the partitioning dimension selected by the current weighting strategy, starts from the root node and recursively compares the dimension value of the data object with the node's partitioning threshold, routes to the appropriate leaf node, and writes the data identifier into the data block associated with that leaf node. After writing, the data range summary of all intermediate nodes on the path is updated from bottom to top. When the size of the data block associated with a leaf node reaches its upper limit, node splitting is triggered. The splitting process, based on the distribution of data within the node on the current high-weight dimension, finds the optimal split point, divides the data into two sets, creates two new leaf nodes, and updates or splits the parent node upwards. When the system detects that the total data volume of adjacent nodes is below a certain threshold and their data ranges overlap significantly, a node merging operation is triggered, merging the data of two nodes into a new node and adjusting the tree structure. Incremental reorganization is triggered by the monitoring module. By analyzing query load and data distribution changes, hotspots or subtree regions that need adjustment are identified. The weights of nodes in this region are recalculated, and a splitting, merging, or data migration plan is formulated. The structural adjustment is executed online at the data block level.

[0061] Specifically, the hierarchical fusion index construction and update steps address the problem that traditional single or static multidimensional index structures cannot efficiently adapt to dynamically changing query patterns and data distributions. By constructing a fusion primary and secondary structure index based on real-time weights, differentiated and collaborative organization of spatiotemporal dimensions and high-value business dimensions is achieved. This method enables the physical structure of the index to adaptively fit current data hotspots and query preferences, and the grid division granularity of the primary index is dynamically adjusted with weights, ensuring the efficiency of basic spatiotemporal queries. Simultaneously, through the association of auxiliary inverted indexes and bidirectional pointers, a fast channel is provided for high-frequency business dimension queries, and real-time data location between different indexes is achieved. Ultimately, this achieves the effect of simultaneously optimizing the performance of multiple query patterns, reducing redundant storage, and supporting complex multidimensional combined queries within a single architecture, providing a flexible and efficient index organization solution for dynamic big data environments.

[0062] S4: Intelligent query routing and execution, receiving user query requests and parsing the dimensions in the query conditions; selecting the query path that prioritizes traversing the main index or the corresponding auxiliary index based on the real-time weight of the dimensions; quickly locating related data through the bidirectional pointers, and merging and filtering query results;

[0063] The specific process is executed by the query engine. This engine receives requests from Structured Query Language (SCL) or specific application programming interfaces (APIs), first performing syntax parsing and semantic analysis to extract explicitly specified or implicit dimensional constraints. Then, the query engine accesses the distributed cache to obtain the real-time comprehensive weight score calculated and stored by S2, related to the current query dimension. Based on this score, all dimensions involved in the query are sorted in descending order. The query path selection logic prioritizes the dimension with the highest weight based on the sorting result: if the dimension belongs to the spatiotemporal dimension, the query is routed to the primary index based on a grid partitioned R-tree; if the dimension belongs to the business dimension, it is routed to the corresponding inverted secondary index. After obtaining an initial set of leaf node data blocks or data object identifiers through the initial retrieval, the bidirectional pointers established in S3 are used to directly locate and retrieve the index data of other dimensions associated with this set, thereby quickly narrowing the overall search scope. Finally, the intermediate result sets returned by different index paths are merged based on the global identifier of the data object, and a final filtering is performed strictly according to the time and spatial ranges in the query conditions to output an accurate result set.

[0064] Preferably, in S4, the combination of dimensions in the query conditions will be matched with the real-time comprehensive weight updated in S2; the query route will first traverse the primary and secondary indexes corresponding to the dimension with the highest weight, and quickly associate and reference related data sets in other indexes through the bidirectional pointers established in S3; subsequently, the data sets obtained from different index paths will be subjected to union or intersection operations based on their timestamps and spatial locations to complete the merging and final filtering of the results;

[0065] Furthermore, the matching process occurs during the query optimization phase, which generates one or more potential execution plans and estimates costs. The main basis for cost estimation is the real-time weight score of the dimension. The higher the weight, the higher the expected efficiency of filtering data using that dimension, so it is preferred. When traversing the index corresponding to the high-weight dimension, if there is an auxiliary index for that dimension, the inverted index is directly queried to obtain the list of target data blocks. If it is the primary index, a range search is performed in the R-tree according to the range in the query conditions. The fast association reference implemented through bidirectional pointers essentially loads the associated leaf node data blocks directly into memory based on the pointer information, avoiding a full index scan. The merging operation of the result set first takes the intersection of sets based on the same data block identifier, and then for sets from different data blocks that meet some conditions, the timestamps and spatial coordinate metadata of their records are precisely compared with the query conditions to complete the final filtering and remove data objects that do not meet all constraints.

[0066] Specifically, the intelligent query routing and execution steps address the inefficient path selection problem caused by the lack of real-time quantitative evaluation of dimension filtering performance in traditional query optimizers when facing complex queries with multi-dimensional constraints. By prioritizing query dimensions based on real-time weight scores and selecting the optimal initial retrieval entry point accordingly, this method guides query execution to the most filtering index path. It utilizes bidirectional pointers between indexes to achieve fast data association across indexes, effectively avoiding redundant full scans and random input / output access. Finally, through precise set operations and conditional filtering, it achieves efficient response to complex multi-dimensional queries, significantly reducing query latency and improving processing throughput.

[0067] S5: Online monitoring and adaptive adjustment, real-time monitoring of data distribution offset and query performance indicators, when the change exceeds the preset threshold, triggering root cause diagnosis and performing the adjustment of the dynamic weight and the online, incremental reorganization of the hierarchical fusion index structure based on the diagnosis results;

[0068] The online monitoring and adaptive adjustment steps are executed by the background daemon service, which continuously collects and analyzes metrics in two dimensions: first, data distribution offset, which is measured by comparing the statistical characteristics of recently added data with historical baseline data in key dimensions, such as the proportional change of data volume in different time periods in the time dimension, the change of the clustering degree of newly added data coordinates in the spatial dimension, and the numerical distribution change of key business parameters such as vibration amplitude; second, query performance metrics, which are collected through instrumentation to determine the actual execution latency of each query, the number of data blocks scanned, and the size of the returned result set. The monitoring service periodically aggregates these raw metrics, such as calculating the average query latency and 95th percentile latency every minute, as well as the index hit rate.

[0069] Preferably, in step S5, the data distribution offset is quantified by statistically analyzing the rate of change of the value distribution of newly added data in key time and business dimensions. The query performance indicators include average query latency and index hit rate. During monitoring, the above-mentioned rate of change and performance indicators are compared with their respective preset static and dynamic thresholds. When any indicator continuously exceeds the threshold, a root cause diagnosis is initiated. The key time usually refers to a period with business cycle, such as the daily peak load period or flood season. The quantification of the rate of change of value distribution can be achieved by using methods such as chi-square test or KL divergence to calculate the difference between the recent data distribution and the reference historical distribution. The static threshold is preset by the administrator based on experience, while the dynamic threshold is automatically calculated based on the historical rolling average and standard deviation of the indicator. For example, the threshold can be set as the average latency plus twice the standard deviation. The generation of the trigger signal follows the anti-jitter logic, which requires the indicator to exceed the threshold within three consecutive monitoring periods to prevent invalid reorganization caused by instantaneous fluctuations.

[0070] The root cause diagnosis performs correlation analysis on the anomalies to locate the core dimensions causing the anomalies; subsequently, based on the diagnosis results, it triggers directed weight adjustment and index reorganization: if the diagnosis points to a specific dimension, it adjusts the weights of the relevant dimension and triggers local index reorganization involving the data of that dimension; if the diagnosis indicates widespread changes, it triggers dynamic weight recalculation as described in S2 and index reorganization within the corresponding range; the reorganization is performed online at the data block level.

[0071] Furthermore, the correlation analysis is performed through a diagnostic engine, which acquires the abnormal time period identifiers and aggregated indicators output by the monitoring service. The diagnostic engine first queries fine-grained statistics on the query frequency of each dimension and dimension combination within the same time period, calculates the rate of change in the number of queries for each dimension and dimension combination, compares it with the historical baseline, and filters out dimensions with a rate of change exceeding a preset multiple as a candidate set. Simultaneously, the engine analyzes frequently occurring query condition patterns within the abnormal time period and extracts co-occurring dimension combinations. Next, the engine verifies the data selectivity changes of candidate dimensions within the abnormal time period by calculating the sliding window difference of their value distribution entropy. Finally, the engine identifies dimensions with a surge in query frequency, high-frequency co-occurrence, and significant fluctuations in data selectivity as core dimensions. For targeted adjustments, after receiving diagnostic results containing clear core dimension identifiers, the index management service sets temporary weight boost offsets for these dimensions in the distributed cache. These offsets are superimposed with the original weight scores to form an immediately effective adjusted weight. Simultaneously, based on this weight, the location logic quickly identifies all leaf nodes by querying the mapping relationship between the metadata of the main index and the auxiliary indexes. The data block stores the subtree range or inverted list fragments of data related to the core dimensions. Local reorganization operations are then planned, targeting only these specific subtrees or index fragments. The reorganization tasks are placed in a high-priority background queue and retrieved and executed by independent reorganization worker threads in the order of the data blocks. The reorganization worker threads split or merge the target data blocks based on the adjusted weights and update the relevant pointers. Row-level locking and multi-version control are used during the process to ensure that concurrent read and write operations are not affected. For cases where changes are deemed widespread, the diagnostic conclusion does not output a specific dimension identifier but indicates that the indicators of multiple dimensions have changed to a moderate degree or above and there is no single prominent factor. At this time, a complete weight recalculation process and a reorganization plan covering a wider range of index partitions are triggered. Online reorganization is performed at the data block level by designing each data block as an independent storage and version management unit. When the reorganization operation is executed, a new version of the data block is created and the structure is adjusted within it. Then, the block pointers in the index are atomically updated to point to the new version. The old version is asynchronously recycled after it is no longer referenced. This mechanism ensures the isolation and rollback of the reorganization operation and also ensures service continuity.

[0072] Specifically, the online monitoring and adaptive adjustment steps address the problems of inaccurate index adaptive optimization and excessive resource overhead caused by the disconnect between monitoring mechanisms and adjustment actions in existing technologies. Traditional methods, when detecting performance or data distribution anomalies, typically trigger global weight recalculation and index reconstruction directly. This coarse-grained adjustment ignores the local root causes of the problem, introduces high computational and input / output costs, and may affect the stability of online services if not handled properly. This invention introduces a progressive root cause diagnosis and differentiated reorganization mechanism, first performing refined correlation analysis on anomalies exceeding thresholds to accurately locate the core dimensions or factors causing performance changes. Dimensional combination; then, based on the directionality of the diagnostic conclusions, a gradient response is implemented: for problems caused by a few dimensions, temporary weight shifts and local index subtree reorganization are used for rapid and accurate intervention; for widespread changes, a global adjustment is initiated; the online monitoring and adaptive adjustment steps upgrade the adaptive process from a general response based on phenomena to a targeted treatment based on root causes. While ensuring the effectiveness of optimization, it significantly reduces the computational and storage overhead of the reorganization operation itself, and ensures business continuity and data consistency during the reorganization period through multi-version control of data block granularity. Ultimately, it achieves efficient, stable, and economically sustainable self-optimization of the index structure.

[0073] As a preferred option, a multi-dimensional data index optimization method for hydropower big data storage further includes: S6: Lifecycle-aware index storage management, which marks data objects as hot data, warm data, or cold data based on data access frequency, generation time, and business criticality; during implementation, an independent data lifecycle management service periodically runs a marking job; this job scans the metadata repository and calculates a comprehensive heat score for each data object; the heat score is obtained by weighted summation of access frequency score, time decay score, and business criticality bonus; wherein the access frequency score is calculated logarithmically normalized based on the number of times the object has been accessed by query or analysis tasks within the most recent sliding time window; the time decay score is calculated using an exponential decay function based on the time elapsed between the data object's generation time and the current time; the business criticality bonus is directly mapped from a predefined equipment and measurement point criticality level comparison table; finally, based on the preset interval in which the heat score falls, the data object is determined to be hot data, warm data, or cold data;

[0074] The data lifecycle management service initiates marking jobs at fixed scheduling intervals. These jobs connect to a database that records detailed access logs and a repository that stores metadata. The scanning process is executed in parallel by data partition, with each compute node processing an independent data shard. For access frequency scores, the sliding time window length is configured to seven or thirty days. The access count is first incremented by one to avoid taking the logarithm of zero, and then normalized using the formula: Score = log10(access count + 1) / log10(maximum global access count within the time window + 1). The time decay score uses a Newton's law of cooling model, i.e., score = e^(-λ*duration), where the decay coefficient λ is set according to the validity period of the business data, and the duration unit can be day or month; the business criticality bonus is a static weight coefficient, for example, 1.0 for ordinary test points, 1.5 for key equipment core test points, and 2.0 for security-related test points; when weighted summing, the weights of access frequency, time decay, and business criticality can be configured as 0.5, 0.3, and 0.2, respectively; the calculated heat score and the corresponding hot, warm, and cold status labels will be updated to the record of each data object in the metadata warehouse and synchronized to the distributed cache for real-time querying by the index management service; for the hot data, its complete hierarchical fusion index is maintained in a high-speed storage medium; the high-speed storage medium refers to persistent memory or solid-state drive; the complete index includes the entire tree node structure of the main index and all constructed auxiliary inverted indexes, all of which reside in a memory-friendly data structure to ensure millisecond-level response;

[0075] All index structures corresponding to hot data are managed and serviced by a resident index loader. Upon startup, this loader loads the index files corresponding to all partitions marked as hot data from high-speed storage media into off-heap memory or a persistent memory pool, based on the hotness tags in the metadata. The R-tree node structure of the primary index is stored in memory as a contiguous byte array. Internal node fields, such as range boundaries and sub-pointers, are accessed with fixed offsets, and memory pools are used for allocation and reclamation to reduce fragmentation. The inverted lists of all auxiliary inverted indexes are also stored in memory, employing a hybrid data structure combining skip lists and compressed bitmaps to balance query speed and memory usage. To maintain the index's up-to-date state, any insertion, update, or deletion operation on hot data generates a redo log. While modifying the in-memory index structure, a background thread asynchronously flushes the logs to a persistent index copy on high-speed storage media. Furthermore, a separate index synchronization thread periodically dumps the index status checkpoints from memory to high-speed storage media to ensure rapid reconstruction during fault recovery. For the warm data, a coarse-grained structure of the primary index and some key auxiliary indexes are maintained in disk storage. The coarse-grained structure of the primary index refers to retaining only the root node of the R-tree and a few high-level intermediate nodes, which contain a spatiotemporal summary of their subtrees, sufficient for query routing and preliminary filtering. Some key auxiliary indexes refer to retaining only the inverted lists corresponding to business dimensions whose query frequency exceeds a threshold, based on historical query statistics. The complete data content and fine-grained indexes are stored on a large-capacity mechanical hard drive.

[0076] The coarse-grained structure of WenData's main index is generated using an index pruning algorithm. This algorithm takes a complete R-tree as input, traverses upwards from the leaf nodes, merges child nodes whose sibling nodes' data ranges are less than a preset merging threshold, and replaces all child nodes with the range summary of that parent node. This process is iterated until a predetermined tree height or node count target is reached. The final retained high-level nodes are serialized and stored in a specific area of ​​the solid-state drive (SSD). The filtering of some key auxiliary indexes relies on a historical query statistics table, which records the number of times query conditions for all business dimensions have been hit over a period of time. Only when the query frequency of a dimension value exceeds N times the dynamically calculated average frequency will its corresponding inverted list be retained on the SSD; the remaining inverted lists are transferred to the hard disk drive (HDD). When a query is routed to WenData, the query engine first accesses the coarse-grained main index on the SSD for initial positioning. If further filtering is needed, relevant key auxiliary indexes are loaded. If the query involves auxiliary dimensions that are not retained... If a query is triggered, a random read of the complete fine-grained index file on the hard disk is initiated. The complete data blocks of warm data are stored in a column-compressed format within a contiguous large file on the hard disk, and the starting offset and length of each data block are recorded in an independent block location table. For cold data, it is archived to low-cost storage, and its core dimension values ​​are extracted and stored in a column-oriented format to form a lightweight metadata index. When a query is triggered, a portion of the index is reconstructed as needed based on this metadata index. Low-cost storage refers to object storage or a tape library. Core dimension values ​​include at least timestamps, device identifiers, key operating condition codes, and a few summary statistical indicators. These values ​​are stored in column-compressed format to form an independent metadata file. When a query needs to access cold data, the query engine first loads this metadata index into memory, quickly filters out potentially matching data archive blocks, and then reads only the relevant columns from these archive blocks as needed. A temporary hash index or bitmap index containing only the dimensions required for this query is dynamically built in memory to accelerate the filtering process within the archived data. This temporary index is released after the query is completed.

[0077] Furthermore, building a lightweight metadata index is a necessary step in the archiving process. Before data is migrated to object storage or tape library, a dedicated archiving processor scans the data blocks to be archived, extracting core dimension values ​​such as timestamps, device identifiers, and key condition codes row by row, as well as pre-calculated statistical indicators such as maximum, minimum, and average values. The extracted data is organized by column, with each column independently compressed using frame compression or dictionary encoding, and a metadata header for that column is appended. Finally, it is packaged into an independent metadata index file, which is uploaded to low-cost storage along with the archived data blocks, but its storage path is independently registered in the metadata repository. When a query needs to access cold data, the query engine downloads the corresponding metadata index file from object storage based on the time range, device set, etc., in the query conditions. Local memory; In memory, the engine performs vectorized scanning and filtering on columnar data to quickly determine a list of archived data blocks that may contain the target data; then, based on the dimensions involved in the query, the engine reads only the necessary columnar data blocks from the archived blocks into memory, and on this basis, immediately builds a local index structure in memory for this query. This structure may be an open-addressable hash table for device identifiers or a bitmap index for discrete condition codes; the lifecycle of this temporary index is strictly bound to the execution context of this query, and the memory and index structure it occupies are immediately reclaimed after the query results are returned; to accelerate frequently queried cold data, a metadata index cache can be configured to keep recently used metadata index files in the local solid-state drive cache for a period of time;

[0078] Specifically, the data lifecycle management service steps address the challenge of efficient index resource allocation in massive, multi-dimensional time-series data storage within hydropower big data scenarios, caused by uneven access patterns and storage cost pressures. Traditional methods struggle to dynamically adapt query performance to storage economy, often resulting in slow response times for frequently accessed data due to insufficient index resources, or wasteful maintenance of expensive indexes for low-frequency historical data. By introducing lifecycle-aware index storage management, data is dynamically categorized into hot, warm, and cold levels based on a quantified comprehensive popularity score. Differentiated index completeness, storage media, and access strategies are matched to each level, achieving precise adaptation between storage and computing resources. Ultimately, within a single architecture, it ensures millisecond-level real-time query performance for hot data while reducing overall storage costs and maintenance overhead for warm and cold data, especially massive historical archives. Simultaneously, through metadata indexing and on-demand reconstruction mechanisms, it retains the ability to effectively query and analyze all historical data, thus achieving a dynamically adjustable balance between performance, cost, and data value.

[0079] This embodiment provides a multi-dimensional data index optimization method for hydropower big data storage. The method first uses a stream processing service to access and standardize multi-source hydropower time-series data in real time, accurately extracting multi-dimensional features including millisecond-level timestamps, three-dimensional spatial coordinates, unique equipment codes, detailed operational status, and various operating parameters, laying a high-quality data foundation for index construction. Subsequently, based on continuous analysis of historical query logs, a dynamic weighted model integrating query frequency, data selectivity, and business priority is adopted to calculate and update the comprehensive weights of each dimension and dimension combinations in real time. Based on these weights, a hierarchical fusion index structure is constructed: the main index uses a weight-adaptive grid partitioned R-tree, with its partitioning granularity and hierarchy dynamically adjusted according to high-weight spatiotemporal dimensions; simultaneously, an inverted auxiliary index is constructed for high-weight business dimensions and associated with the leaf nodes of the main index through bidirectional pointers. During queries, the optimal index path is intelligently selected based on real-time weights, and bidirectional pointers are used to quickly locate and merge data, achieving efficient response to complex multi-dimensional queries. By continuously monitoring data distribution offset and query performance metrics through a background guardian service, and triggering weight recalculation and incremental online index reorganization, the index structure can be continuously self-optimized. In addition, a lifecycle-aware management mechanism is introduced to classify data according to access frequency, time and business criticality, and implement differentiated index storage strategies to optimize overall storage costs while ensuring query performance. The method forms a complete technical closed loop from data access, dynamic index construction and adjustment, intelligent query to storage resource optimization and configuration, effectively improving the efficiency and adaptability of multi-dimensional queries in the hydropower big data environment.

[0080] Example 2

[0081] Focusing on the dynamic weight calculation step in Example 1, namely the adaptive weight calculation model based on reinforcement learning and multi-factor fusion, this paper elaborates on how it, as an independent intelligent indexing strategy decision-making component, can collaborate decoupled with downstream index building services. This method upgrades the core logic of index optimization from fixed rules or empirical formulas to an intelligent agent that can learn online and iterate on its own, thereby achieving continuous autonomous evolution of the indexing strategy.

[0082] The dynamic weight calculation model is a closed-loop learning system consisting of environment perception, a policy network, and a reward mechanism. The environment perception continuously obtains three types of real-time state information from the distributed cache, including historical query log streams, statistical summaries of data distribution for each dimension and dimension combinations, and business rules and priorities synchronized from the configuration center. This information is transformed into a state vector observable by the model. The core of the policy network is a parameterized weight coefficient generator, which takes the environment state vector as input and outputs a set of dynamic coefficients for linear weighted summation. These coefficients correspond to three influencing factors: query frequency, data selectivity, and business priority. The policy network is initially initialized with initial coefficients preset by domain experts, but its parameters are continuously adjusted through subsequent reinforcement learning. The reward mechanism is the key to driving model learning. This model defines a composite reward function: the main reward signal comes from the average latency change of the same type of query pattern after the index reorganization operation is completed; if the latency decreases, a positive reward is given; if the latency increases or remains unchanged, a negative reward or zero reward is given; the auxiliary reward signal considers the computation and input / output overhead of the index reorganization operation itself, encouraging the model to seek a balance between performance improvement and reorganization cost.

[0083] The model's operation is divided into two phases: offline initialization and online learning and decision-making. In the offline initialization phase, historical data is used for pre-training. By simulating historical query loads and index changes, reinforcement learning algorithms such as policy gradients are used to initially train the policy network, enabling it to learn basic historical pattern associations. The online learning and decision-making loop is executed periodically: within each decision cycle, the environmental sensor first collects the latest state information; the policy network generates a set of dynamic coefficients based on the current state. For example, during peak flood season monitoring, the model may automatically increase the weight coefficients of the time dimension and specific water condition parameters; during equipment maintenance season, it may increase the weight coefficients of equipment identification and maintenance status. Next, the comprehensive weight score calculated by the model is published to the distributed cache; the index management service subscribes to this result and triggers subsequent index building or incremental reorganization accordingly. After an evaluation period, the model collects data such as changes in query latency caused by index changes, calculates reward values, and uses the obtained rewards to fine-tune the parameters of the policy network through online reinforcement learning algorithms, thereby optimizing the next decision.

[0084] Furthermore, the dynamic weight calculation model, designed as an independent intelligent component, is decoupled from the specific index structure. It collaborates solely by publishing weight scores through a clearly defined interface. The core output of the model is a dimension weight mapping table, independent of the downstream index structure. This provides architectural flexibility, allowing downstream index services to freely choose the most suitable physical index implementation based on the weight scores. The model, as a weight calculation service, is triggered by timers or monitoring events. The index management service, as another independent service, listens for weight score update events. When a new weight score is published, the index service determines, based on its own logic, whether and how to adjust the index. This loosely coupled design allows for independent deployment, upgrades, and expansion of both parts. The model can maintain multiple policy networks or automatically switch policies through context recognition. For example, for two different query scenarios—real-time fault diagnosis and long-term energy efficiency analysis—the model can learn and apply two different weight preference policies and adapt to the scenario through a meta-learning mechanism.

[0085] Furthermore, the detailed method enables autonomous and continuous optimization of indexing strategies, transforming the core of index optimization from manual design or static configuration to a data-driven, goal-oriented reinforcement learning process. This allows indexes to automatically adapt to unpredictable business changes. Simultaneously, it constructs a general indexing strategy decision layer, acting as an independent intelligent middleware. This abstracts the strategic question of when and on what basis to optimize indexes from specific index data structures, providing general intelligent optimization decision support for different big data storage environments. The method deeply integrates business feedback and domain knowledge. The reward mechanism directly uses business performance indicators as optimization targets, while the initial state of the model and business priority inputs incorporate domain knowledge, ensuring consistency between the optimization direction and business objectives. Moreover, the method is interpretable and operable. Although the model learns automatically, its output weight scores and coefficient changes are observable and analyzable. Administrators can intervene in the model's decision direction by adjusting business priority configurations or setting strategy constraints, achieving human-machine collaborative optimization.

[0086] Through the implementation of this embodiment, the dynamic weight calculation model becomes the intelligent hub of the entire multidimensional data indexing optimization method. It not only solves the problem of the inapplicability of static weights, but also provides an advanced paradigm that enables the big data indexing environment to have self-awareness, decision-making and evolution capabilities, significantly improving the long-term adaptability and overall efficiency in complex dynamic scenarios.

[0087] Example 3

[0088] This embodiment further illustrates the specific application of a multi-dimensional data indexing optimization method for hydropower big data storage in the monitoring and analysis scenario of a large-scale cascade hydropower station group in a river basin;

[0089] In this scenario, the data originates from multiple cascade hydropower stations within the river basin, encompassing tens of thousands of measuring points and generating hundreds of thousands of time-series data records per second. Data dimensions include time, station and unit location, equipment type, operating conditions, and hundreds of parameters such as head, flow rate, power, vibration, and temperature. Historical data has accumulated over decades, totaling petabytes in size, and query requirements are complex and varied, including real-time monitoring, historical backtracking, cross-station correlation analysis, and equipment fault diagnosis.

[0090] This method is implemented in this scenario as follows;

[0091] First, in the data access and feature extraction steps, edge stream processing services deployed at the data aggregation nodes of each power station access and preprocess real-time data streams locally. For each data point, the collection timestamp is precisely extracted to the millisecond, and its corresponding three-dimensional geographic coordinates are mapped from the unified asset model based on the location of the dam section, unit, and specific component where the measuring point is installed. In terms of business dimensions, equipment identification adopts the nationally unified power equipment code. The operating status is calculated in real time by a rule engine deployed at the edge, which not only identifies basic states such as start-up, power generation, and shutdown, but also pinpoints specific abnormal sub-states such as load shedding, overspeed, and bearing overheating. Operating parameters are extracted in their original values ​​after quality code verification. All standardized data objects are sent to the central data platform.

[0092] Secondly, the dynamic weight calculation process continuously analyzes query logs from production monitoring dashboards, mobile inspection applications, and backend analysis models. For example, during the flood season, the query load is clearly biased towards combinations of time and space dimensions, such as frequent queries for the vibration status of all power plant units within a specific time period. During equipment maintenance seasons, queries are highly concentrated on specific equipment identifiers and historical operating condition sequences. The backend analysis service calculates weights on an hourly basis: it counts the number of times each dimension and dimension combination has been used for query filtering in the past 24 hours; it calculates the information entropy of each dimension and dimension combination value based on the latest data distribution to assess its filtering capability; simultaneously, business priority configuration assigns higher weights to critical main equipment and safety-related parameters. A weighted model that integrates initial expert settings and can adjust coefficients online based on query latency feedback calculates real-time comprehensive weights. For example, the calculation shows that the spatiotemporal combination has the highest weight for the current time period, followed by specific vibration parameters, while some auxiliary equipment parameters have lower weights.

[0093] Next, based on the current weights, the hierarchical fusion index structure is built and updated. Since the spatiotemporal dimension has the highest weight, the grid partitioning R-tree of the main index is first coarsely divided along the time axis, for example, by day. Within each day, secondary spatial grids are then divided according to the geographical location of the power plant. For business dimensions such as vibration parameters with higher weights, an inverted index is built, with each vibration value associated with a data block in the leaf node of the main index containing that data. The data blocks use columnar compression to store the specific values. When the data write pressure is low during the off-peak hours at night, the index management service will incrementally merge or split some data blocks according to the new weight scores, for example, merging sparsely accessed cold data blocks and further subdividing data blocks from hot periods.

[0094] When an analyst submits a complex query, such as retrieving information on other key parameters of all units in a geographic area that exceeded vibration limits within the past week, the intelligent query routing and execution steps are initiated. After parsing, the query engine identifies spatiotemporal and vibration as high-weight dimensions. Therefore, it prioritizes using the inverted index of vibration parameters to quickly find all records that exceeded the limits and their corresponding data blocks. Then, through the time and spatial range information in the metadata of these data blocks, it quickly correlates them with the data in the main index that corresponds to the spatiotemporal range, directly loads the relevant data blocks to complete the filtering of other parameters, and avoids a full table scan.

[0095] Online monitoring and adaptive adjustment steps ensure the continued efficiency of the above process. The background guardian service monitoring found that as maintenance work began, the query frequency for a specific unit code increased by 300% within an hour, and the related query latency showed an upward trend, triggering an emergency recalculation of dynamic weights. After the weight update, the weight of the unit identifier was greatly increased, and the index management service adjusted the index structure online, splitting the main index data block where the unit-related data was located to improve query granularity and strengthen the organization of the unit identifier in the inverted index.

[0096] Furthermore, the lifecycle-aware index storage management steps run automatically according to preset strategies. For example, real-time monitoring data that has been frequently accessed in the last three days is marked as hot data, and its complete index is stored in an all-flash storage array. Data from last month is marked as warm data, and only the top-level node of its main index is retained to save disk space. Historical data from a year ago is marked as cold data, archived to object storage, and only a lightweight metadata index stored in columnar format by time, power plant, equipment, and key indicators is retained. When it is necessary to analyze the historical performance trend of a unit, the query engine can first quickly locate the archived data package of the required year and month through the lightweight metadata index, decompress and load the relevant column data, and build a temporary index in memory to complete the query.

[0097] Through the application of this embodiment, when facing the challenges of massive, multi-dimensional, and dynamic data from cascade hydropower station groups, the method can significantly improve the response speed of complex queries. Actual verification shows that the latency of typical cross-station multi-dimensional association queries has been reduced from minutes to seconds. At the same time, through differentiated lifecycle index management, the overall storage cost has been reduced by about 40%, effectively supporting the intensive and intelligent operation of hydropower in the basin.

[0098] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A multi-dimensional data indexing optimization method for hydropower big data storage, characterized in that, Includes the following steps: S1: Data access and feature extraction: Real-time access to multi-source heterogeneous time-series data streams in the hydropower field; for each data object, extract its physical dimension attributes and business dimension attributes; the physical dimension attributes include timestamp and spatial location information; the business dimension attributes include equipment identifier, operating status code and operating parameters. S2: Dynamic weight calculation, continuously collects historical query logs, analyzes query load patterns, and dynamically calculates and updates the comprehensive weight of each dimension and dimension combination based on the query frequency, data selectivity, and predefined business priority of each dimension and dimension combination. S3: Hierarchical fusion index construction and update, based on the dynamic weights at the current moment, construct and incrementally update the hierarchical fusion index structure; The main index is constructed using a grid partitioning R-tree structure based on dynamic weights and adaptive values. The node partitioning strategy of the R-tree is dynamically adjusted according to the single spatiotemporal dimension with the highest weight. One or more auxiliary indexes are built for a single business dimension with high weight, using an inverted index structure, and bidirectional pointer associations are established with the leaf node data blocks of the main index; S4: Intelligent query routing and execution, receiving user query requests and parsing the dimensions in the query conditions; selecting the query path that prioritizes traversing the main index and corresponding auxiliary indexes based on the real-time weight of the dimensions; quickly locating related data through the bidirectional pointers, and merging and filtering query results; S5: Online monitoring and adaptive adjustment, real-time monitoring of data distribution offset and query performance indicators. When the change exceeds the preset threshold, root cause diagnosis is triggered and the dynamic weight adjustment and online, incremental reorganization of the hierarchical fusion index structure are performed based on the diagnosis results.

2. The method according to claim 1, characterized in that, In step S1, for each data object, the extracted physical dimension attributes include a timestamp accurate to milliseconds and three-dimensional spatial coordinates composed of longitude, latitude, and elevation; the extracted business dimension attributes include a device code used to uniquely identify the source of the device, a working condition category code divided according to preset rules, and the values ​​of voltage, current, flow rate, and pressure parameters obtained from real-time monitoring.

3. The method according to claim 1, characterized in that, In S2, the dimension query frequency is obtained by counting the number of conditional constraint operations on a single dimension and a combination of dimensions within a specified time window; the data selectivity is evaluated based on the dispersion of dimension values ​​within their value range; the predefined business priority is associated with the equipment importance level and the operational security level. Based on the aforementioned factors, a weighted calculation model is used to derive a comprehensive weight value that reflects query popularity and filtering effectiveness in real time.

4. The method according to claim 3, characterized in that, The specific process of establishing the weighted calculation model is as follows: The system takes continuously collected historical query logs, dimensional value range distribution statistics, and preset business priority rules as inputs. The processing first normalizes the query frequency value, data selectivity evaluation value, and business priority value of each dimension and dimension combination. Then, it performs linear weighted summation based on the coefficients dynamically set by domain expert experience and reinforcement learning mechanism. The output is a real-time, quantitative comprehensive weight score for each dimension and dimension combination. The comprehensive weight score is directly used to guide the selection of splitting dimensions for nodes within the index structure and the arrangement order of dimensions in the composite index key.

5. The method according to claim 1, characterized in that, In S3, the construction and updating of the hierarchical fusion index structure are based on the comprehensive weight score obtained in S2; The main index adopts a grid-partitioned R-tree structure. When dividing the tree nodes in space, the granularity and level of the grid division are adaptively determined according to the weight score corresponding to the current high-weight spatiotemporal dimension. The auxiliary index is an inverted index built for high-weight business dimensions. Each inverted list is associated with a bidirectional pointer to the leaf node data block in the main index R-tree that stores the corresponding dimension value data object. The leaf node data block records the set of data objects and their corresponding time range and physical storage location.

6. The method according to claim 1, characterized in that, In S4, each dimension and combination of dimensions in the query conditions will be matched with the real-time comprehensive weight updated in S2; the query route will first traverse the primary and secondary indexes corresponding to the dimension with the highest weight, and quickly associate and reference related data sets in other indexes through the bidirectional pointers established in S3; subsequently, the data sets obtained from different index paths will be subjected to union and intersection operations based on their timestamps and spatial locations to complete the merging and final filtering of the results.

7. The method according to claim 1, characterized in that, In S5, the offset of the data distribution is quantified by statistically analyzing the rate of change of the value distribution of newly added data in key time and business dimensions, and the query performance indicators include average query latency and index hit rate. During monitoring, the aforementioned rate of change and performance indicators are compared with their respective preset static and dynamic thresholds. When any indicator continuously exceeds the threshold, root cause diagnosis is initiated. The root cause diagnosis performs correlation analysis on the anomaly to locate the core dimension causing the anomaly. Subsequently, based on the diagnosis results, directed weight adjustment and index reorganization are triggered: if the diagnosis points to a specific dimension, the weights of the relevant dimension are adjusted in a targeted manner and a local index reorganization involving the data of that dimension is triggered; if the diagnosis indicates widespread changes, dynamic weight recalculation as described in S2 and index reorganization within the corresponding range are triggered; the reorganization is performed online at the data block level.

8. The method according to any one of claims 1-7, characterized in that, Also includes: S6: Lifecycle-aware index storage management marks data objects as hot data, warm data, and cold data based on data access frequency, generation time, and business criticality; For the hot data, its complete hierarchical fusion index is maintained in a high-speed storage medium; for the warm data, the coarse-grained structure of its main index and some key auxiliary indexes are maintained in disk storage. The cold data is archived to low-cost storage, and its core dimension values ​​are extracted to form a lightweight metadata index in columnar storage format. When a query is triggered, a portion of the index is reconstructed as needed based on the metadata index.

9. The method according to claim 8, characterized in that, In S6, the labeling of data objects is determined based on their access frequency within the sliding time window, the time elapsed since the current time, and a predefined business criticality level. For data objects labeled as hot data, their complete R-tree primary index structure and inverted secondary index reside on high-speed storage media. For warm data, the primary index retains upper-level coarse-grained nodes on disk to reduce space usage, while the secondary index retains the portion for frequently queried dimensions. For cold data, while archiving to low-cost storage, their core dimension values ​​are extracted and stored column-wise to form a lightweight metadata index. When subsequent queries hit this part of the data, the local index structure of the dimensions required for the query is temporarily reconstructed in memory as needed based on the metadata index.

Images