Space-time big data dynamic index construction method and device and storage medium

Abstract

The invention relates to a space-time big data dynamic index construction method and device and a storage medium. The method comprises the following steps: inputting a spatio-temporal data stream into a distributed system to carry out fragmentation processing and index tree structure construction to obtain an initial index structure, and carrying out parallel feature extraction on the spatio-temporal data stream to obtain a spatio-temporal distribution feature vector; calculating a space-time overlapping degree, a data density similarity and a query frequency correlation degree of adjacent nodes based on the space-time distribution feature vectors to obtain a combined evaluation score of the node pair; performing multi-objective optimization based on the merging evaluation score to generate an optimal merging threshold value; and according to the optimal merging threshold, performing incremental merging operation on the node pairs, and performing data recombination and range boundary updating to obtain an optimized index tree structure. According to the method, smooth adjustment of the index structure is realized, the influence of structure adjustment on online query service is minimized, and the index structure can be continuously optimized and kept in an efficient state.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and particularly to a method, device, and storage medium for constructing a spatio-temporal big data dynamic index. Background Art

[0002] Spatio-temporal big data is increasingly widely used in fields such as smart cities, intelligent transportation, and environmental monitoring. The spatio-temporal data generated by these application scenarios is characterized by high speed, large volume, and persistence, posing severe challenges to the real-time processing and efficient retrieval of data. Most traditional spatio-temporal index methods construct index structures in a batch processing manner, requiring the storage of the complete data set, and this method has obvious limitations when dealing with continuously generated real-time data streams.

[0003] Currently, there are mainly three problems in the spatio-temporal index construction methods: existing methods often focus on optimizing query performance, ignoring the efficiency of index construction and maintenance, resulting in a sharp decline in performance in high-frequency data update scenarios; secondly, most methods adopt static index structures, lacking the adaptive ability to changes in data distribution characteristics and being unable to effectively cope with the dynamic changes of data patterns; existing methods often adopt simple data sharding strategies in a distributed environment, without fully considering the load balancing and resource utilization efficiency among nodes. Summary of the Invention

[0004] The main object of the present invention is to provide a method, device, and storage medium for constructing a spatio-temporal big data dynamic index. The present invention realizes the smooth adjustment of the index structure, minimizes the impact of structure adjustment on online query services, and enables the index structure to be continuously optimized and maintained in an efficient state.

[0005] To achieve the above object, the present invention provides a method for constructing a spatio-temporal big data dynamic index, including the following steps: Input the spatio-temporal data stream into a distributed system for sharding processing and index tree structure construction to obtain an initial index structure, and perform parallel feature extraction on the spatio-temporal data stream to obtain a spatio-temporal distribution feature vector; Based on the spatio-temporal distribution feature vector, calculate the spatio-temporal overlap degree, data density similarity, and query frequency correlation degree of adjacent nodes to obtain a combined evaluation score for node pairs; Perform multi-objective optimization based on the combined evaluation score to generate an optimal merging threshold; Execute an incremental merging operation on node pairs according to the optimal merging threshold, and perform data reorganization and range boundary update to obtain an optimized index tree structure.

[0006] The present invention also provides a device for constructing a spatio-temporal big data dynamic index, including: A construction module for inputting spatio-temporal data streams into a distributed system for sharding processing and index tree structure construction to obtain an initial index structure, and performing parallel feature extraction on the spatio-temporal data streams to obtain spatio-temporal distribution feature vectors; A calculation module for calculating the spatio-temporal overlap degree, data density similarity, and query frequency correlation degree of adjacent nodes based on the spatio-temporal distribution feature vectors to obtain a combined evaluation score of node pairs; A multi-objective optimization module for performing multi-objective optimization based on the combined evaluation score to generate an optimal merging threshold; An update module for performing an incremental merging operation on node pairs according to the optimal merging threshold, and performing data reorganization and range boundary update to obtain an optimized index tree structure.

[0007] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any one of the above are implemented.

[0008] In summary, the technical solution provided by the present invention realizes the balanced distribution of data through dynamic sharding and the consistent hashing algorithm, reduces the performance bottleneck caused by data skew, and improves the overall processing capacity and scalability of the system; adopts a parallel feature extraction mechanism based on time windows, and can quickly construct and update the index structure without storing all historical data, greatly reducing the consumption of memory resources; introduces a recursive node merging evaluation model, and accurately grasps the node merging opportunity through comprehensive evaluation of multi-dimensional indicators, avoiding performance fluctuations caused by blind merging; designs a multi-objective optimization algorithm based on dynamic programming, achieving a good balance among system load, query performance, and storage efficiency, and ensuring the stability of the index structure; realizes the smooth adjustment of the index structure through a concurrent lock mechanism and incremental merging operations, minimizing the impact of structure adjustment on online query services; establishes a complete distributed performance evaluation system, and enables the index structure to be continuously optimized and maintained in an efficient state through real-time monitoring and feedback adjustment of multi-dimensional indicators. Description of the Drawings

[0009] Figure 1 is a schematic diagram of the steps of a spatio-temporal big data dynamic index construction method in an embodiment of the present invention; Figure 2 is a block diagram of the structure of a spatio-temporal big data dynamic index construction device in an embodiment of the present invention.

[0010] The realization, functional characteristics, and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. Detailed Embodiments

[0011] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0012] Referring to Figure 1 , this embodiment provides a method for constructing a spatio-temporal big data dynamic index, including the following steps: S1. Input the spatio-temporal data stream into a distributed system for sharding processing and index tree structure construction to obtain an initial index structure, and perform parallel feature extraction on the spatio-temporal data stream to obtain a spatio-temporal distribution feature vector; Among them, the spatio-temporal data stream is input into the distributed system for data range analysis and boundary calculation to determine the value range of the time dimension and the distribution range of the space dimension, and generate a data shard vector. Statistical calculations are performed on the distribution characteristics of the spatio-temporal data to extract the maximum value, minimum value of its spatio-temporal range, and the density characteristics of the data distribution, so as to divide reasonable spatial units and time segments. Based on the data shard vector, the hash value distribution of virtual nodes is calculated to ensure the balance of data storage. Since the distributed system uses the hash consistent mapping method for data management, the hash value is calculated according to the spatial range and time span of the data, and combined with the query frequency and update frequency of the data, an appropriate load weight coefficient is assigned to each virtual node, thus avoiding the data skew problem. According to the relationship between the load weight and the hash distribution, a data shard mapping table is generated to ensure that different data segments are reasonably mapped to multiple computing nodes, realizing load-balanced data storage and access. The spatio-temporal data stream is recursively divided at multiple levels according to the data shard mapping table to construct a distributed index tree. The recursive division method uses data structures such as quadtrees to ensure that the hierarchical organizational structure of the index tree adapts to different-scale data storage requirements. During the construction of the index tree, the data range of the root node is determined, and the data is gradually subdivided into child nodes according to the data shard strategy until the storage capacity of the leaf nodes reaches the set threshold, thus forming a distributed index tree. Function module parameters are configured for each leaf node in the distributed index tree to support the dynamic management of data. A data write buffer is deployed at each leaf node to improve the data write efficiency and reduce disk I / O operations. At the same time, the data in the buffer is used for batch processing to optimize the update performance of the data index. A split trigger is configured at the leaf node. The role of this trigger is to automatically trigger the split operation of the index node when the data volume of a certain leaf node exceeds the set threshold, ensuring that the structure of the index tree remains balanced and avoiding the data storage hot spot problem. At the same time, a merge processor is configured. The function of this processor is to perform a node merge operation if the data volume of some leaf nodes is lower than the set threshold after data deletion or update operations, so as to reduce the index level and optimize the query efficiency. Through the collaborative work of these three function modules, the dynamic adaptability of the index structure is ensured, supporting the efficient management of spatio-temporal data streams. The data receiving unit, node splitting unit, and node merging unit are registered with the distributed coordination center to realize the management of the global index. The distributed coordination center manages the communication between different computing nodes through the message queue method to ensure that the update operations of the index tree can be consistently propagated throughout the distributed system. In this process, a communication channel between nodes based on the message queue is established, enabling the change information of the index tree to be quickly propagated to other computing nodes and updated in the global index directory table. The global index directory table records the location information, data storage range, and load conditions of each index node, thus supporting the fast routing of query requests and improving the query efficiency.Set distributed transaction control parameters according to the global index directory table to ensure that data access operations between index nodes comply with consistency constraints. Since a distributed system involves multiple computing nodes accessing the same index data concurrently, a distributed lock mechanism is adopted to control data read and write operations to avoid data competition and inconsistency problems. By using a distributed lock (such as Zookeeper, Redis distributed lock), read and write lock identifiers are assigned to index nodes, enabling concurrent query operations to be executed correctly and ensuring the transaction consistency of write operations. Through these control mechanisms, an efficient and scalable initial index structure is constructed. After the index structure is completed, parallel feature extraction is performed on spatio-temporal data streams according to a preset time window to optimize index query performance. According to the set time window size, the data is segmented in the time series, and based on the data sharding rules in the initial index structure, the spatio-temporal distribution feature vectors within each time window are calculated. This includes calculating indicators such as data density, spatial coverage, and data change trend within each time window, thereby extracting high-dimensional spatio-temporal feature vectors. Through the spatio-temporal feature extraction method, the distribution law of spatio-temporal data is identified, and the index structure is dynamically optimized to improve the management efficiency and query performance of large-scale spatio-temporal data.

[0013] Segment the spatio-temporal data stream according to a preset time window, and perform efficient parallel computing on the spatio-temporal characteristics of the data. Window the data stream according to the set time interval. Each window corresponds to a data set within a time period, and independently extract and calculate the features of the data in different time segments. Within each time window, calculate the time aggregation degree and spatial distribution density of the data samples to reflect the concentration degree of the data in different time periods and the coverage characteristics in the spatial dimension. The time aggregation degree is measured by calculating the change rate of the number of samples per unit time, while the spatial distribution density is based on the spatial distribution of data points, and methods such as kernel density estimation or grid division are used to calculate the data density value within each spatial unit. Through these calculation processes, a window-level feature matrix is obtained, which contains the time distribution characteristics and spatial density characteristics of the data within each time window. Standardize the feature matrix according to the data sharding rules in the initial index structure, using methods such as mean normalization or Z-score normalization to convert the time aggregation degree and spatial density characteristics within different time windows into numerical values within the same range, eliminating the influence brought by different data scales, and obtaining a standardized feature sequence. Based on the standardized feature sequence, calculate the covariance of the feature distributions in the time dimension and spatial dimension to measure the correlation of the data in different dimensions, and construct a dimension correlation matrix. Covariance calculation analyzes the change trends between different feature variables to identify the linear relationship between them, while the dimension correlation matrix is used to describe the correlation degree between the time characteristics and spatial characteristics. Based on the dimension correlation matrix, calculate the main feature direction to find the main change pattern of the data distribution. The calculation of the main feature direction uses methods such as principal component analysis or singular value decomposition to extract the main change trend of the data in the spatio-temporal dimension and perform feature dimensionality reduction, thereby reducing redundant information and improving calculation efficiency. In the process of feature dimensionality reduction, select the principal components that can explain the largest variance, so that the dimensionality-reduced features can still well describe the distribution law of the data, and obtain a compressed feature set. Weightedly fuse the time aggregation features and spatial distribution features of the compressed feature set to generate a feature vector that can comprehensively reflect the data distribution law. The method of weighted fusion dynamically adjusts the weights based on historical query patterns, data change trends, and index optimization objectives. In the fusion process, methods such as linear weighting or non-linear transformation are used to combine the relative importance of the time characteristics and spatial characteristics to calculate the final comprehensive feature vector. Binary encode the comprehensive feature vector to reduce the storage space and speed up the feature matching speed. In the binary encoding process, according to the numerical range and distribution law of the feature vector, use hash mapping or quantization methods to convert the continuous feature values into binary feature codes of a fixed length, and obtain the spatio-temporal distribution feature vector.

[0014] S2. Based on the spatio-temporal distribution feature vectors, calculate the spatio-temporal overlap degree, data density similarity degree, and query frequency correlation degree of adjacent nodes, and obtain the combined evaluation score of the node pair; Specifically, inverse quantization decoding is performed on the spatio-temporal distribution feature vector to restore the time and space distribution features of the nodes. Through inverse quantization decoding, the data coverage range of each node is restored, including the time span, the space range, and the specific distribution pattern of the data density, and a corresponding minimum bounding rectangle is constructed for each node. The minimum bounding rectangle is constructed by calculating the maximum and minimum coordinate ranges of the node data points to determine the upper and lower bounds of the rectangle, forming the basic unit for spatial indexing. The R-tree index construction is performed based on the minimum bounding rectangle. The R-tree is a tree-like index structure for storing and retrieving spatial data. Through recursive spatial partitioning, spatio-temporal data is organized into a hierarchical index system, thereby improving the query efficiency. During the R-tree index construction process, it is organized in layers according to the minimum bounding rectangles of the nodes, so that adjacent nodes are merged into the same parent node, and at the same time, the compactness of the spatial region is maintained as much as possible to reduce the data access range. After the R-tree is constructed, its index characteristics are used for K-nearest neighbor search, aiming to find pairs of data nodes with similar spatial positions and close time ranges in the index tree structure. The K-nearest neighbor search method recursively queries the child nodes of the R-tree, finds a group of adjacent nodes according to the approximate degree of the spatial distance, and generates a list of node pairs for subsequent similarity calculation. The overlap degree of the data boundaries of each node pair in the node pair list is calculated to measure the intersection degree of the two nodes in time and space. The calculation method of the spatio-temporal overlap degree is based on the ratio of the intersection area of the minimum bounding rectangles, that is, if the minimum bounding rectangles of two nodes overlap in the spatial or time dimension, the ratio of the intersection area to the area of the smaller rectangle is calculated to obtain a normalized overlap degree score. This calculation method reflects the spatial compactness of the node data and is used to guide the optimization of the index structure. While calculating the spatio-temporal overlap degree, the density similarity of the node pairs is calculated. The density similarity mainly measures whether the spatio-temporal data distributions of two nodes are similar. The calculation method is to calculate the data point density in the unit space region and compare the density differences of the two nodes. The similarity of the density distribution is measured by means such as mean square error or cosine similarity, and finally normalized to a density similarity score. The frequency analysis of the query access records of the node pairs is performed to obtain the query frequency correlation. The calculation of the query frequency correlation is based on the historical query logs. The specific method is to count the query times of adjacent nodes within a certain time range and calculate their correlation. The query frequency of each node is calculated by the sliding average of the time window, and the similarity of the access patterns of the node pairs is calculated. The query frequency correlation reflects whether the query patterns of two nodes are close. If there is a high correlation in the query requests of two nodes, it means that they can be merged to optimize the query efficiency.Normalize the spatio-temporal overlap degree, density similarity, and query frequency correlation, and obtain the combined evaluation score of the node pair through weighted calculation. This score is used for the dynamic optimization of the index tree structure, enabling the combination of nodes with high similarity and strong query correlation, thereby optimizing the efficiency of index storage and improving query performance.

[0015] S3. Perform multi-objective optimization based on the combined evaluation score to generate the optimal combination threshold; It should be noted that the system load level, query response latency, and storage space occupancy rate are monitored in real time to obtain a set of system performance parameters, so as to ensure that the merge operation will not affect the overall performance of the system. The system load level is measured by calculating the CPU utilization rate, memory occupancy rate, and I / O throughput of the server, while the query response latency is calculated based on the average response time of user requests to reflect the impact of index optimization on query efficiency. At the same time, the storage space occupancy rate is determined by monitoring the number of nodes and data storage volume of the index tree to evaluate the optimization effect of index merging on storage resources. According to the set of system performance parameters, weight coefficients are set for each performance indicator to construct multi-objective constraint conditions. The setting of the weight coefficients is adjusted according to the actual application scenario. For example, in an environment with high-concurrency queries, the weight of query response latency needs to be increased, while in a system with limited storage resources, the weight of storage space occupancy rate should be greater. A method based on historical data analysis is used to calculate the influence degree of different indicators on the overall performance, and an optimal weight allocation scheme is set based on this. The merge evaluation score is segmented based on the multi-objective constraint conditions to construct a state transition equation and form a dynamic programming solution model. The process of segmentation is to classify the merge evaluation scores in different ranges into different optimization intervals. When constructing the state transition equation, the conversion relationship between the current state and the subsequent state is considered. For example, if a pair of nodes is merged at the current moment, it will affect the subsequent query performance or storage space. Therefore, the state transition equation needs to reflect this temporal dependence and find the optimal merge strategy through dynamic programming. The dynamic programming solution model is recursively calculated, and the solutions of sub-problems are merged in a bottom-up manner to optimize the calculation efficiency. During the recursive calculation process, the optimal merge scheme for the smallest granularity data blocks is calculated and gradually extended upwards to a larger set of nodes to ensure the optimality of the overall optimization result. The bottom-up calculation method can effectively reduce duplicate calculations and accelerate the solution process by caching intermediate results, enabling efficient calculation of the optimal merge threshold even for large-scale data sets. After obtaining a set of candidate threshold sequences by solving, these thresholds are evaluated and calculated to determine the optimal merge threshold. The evaluation method is to perform backtesting on the performance of each candidate threshold in different time periods based on the data of historical merge operations, including the impact on query response time, system load, and storage occupancy after merging, and calculate the performance evaluation value corresponding to each candidate threshold. The candidate thresholds are sorted according to these evaluation values, and the threshold with the highest performance evaluation value is selected as the final optimal merge threshold.

[0016] S4. Perform an incremental merge operation on the pair of nodes according to the optimal merge threshold, and perform data reorganization and range boundary update to obtain an optimized index tree structure.

[0017] Specifically, a threshold comparison is performed on the combined evaluation scores of all node pairs to determine which node pairs need to perform the merging operation. The optimal merging threshold calculated through the previous steps is used as the judgment criterion, and all node pairs in the index structure are traversed one by one. Node pairs with a combined evaluation score higher than this threshold are added to the merging operation queue to obtain the queue of nodes to be merged. The read-write lock flag is set according to the data access status of the nodes in the queue of nodes to be merged to ensure data consistency and isolation. In a distributed environment, multiple query or index update operations may access the same data nodes simultaneously. To avoid data competition, the nodes to be merged are locked, so that during the execution of the merging operation, other query or update operations cannot modify the data of these nodes simultaneously, obtaining a data isolation state. After locking the data access, the boundary calculation of the spatio-temporal data of the nodes to be merged is performed based on the data isolation state to ensure that the spatial coverage range after data integration is reasonable. The key to the merging operation lies in how to handle the data overlapping part of the two nodes. When performing the merging, the minimum bounding rectangle of the two nodes is calculated, and data integration is performed based on their overlapping area. If the spatio-temporal data of the two nodes is continuous in the time dimension and there is an overlapping area in the spatial dimension, the data in the overlapping area is merged, thereby reducing the redundant data points in the index tree and improving the storage efficiency. After completing the data reorganization and calculating the new data boundary, the hierarchical structure of the index tree is recalculated to ensure that the hierarchical relationship of the index remains reasonable. Since the merging operation changes the original node structure, the parent-child node relationship is re-established so that all child nodes can still be correctly attached to the parent node, and it is ensured that the index query path is not affected. While updating the parent-child node relationship, the index path is reconstructed based on the new node links so that the newly merged nodes can be correctly accessed by index. The reconstruction of the index path involves the adjustment of index structures such as the R-tree or KD-tree. For example, in the R-tree index, if two leaf nodes are merged, the minimum bounding rectangle of their parent node is modified, and the boundary range of the ancestor nodes is adjusted recursively upward to ensure the consistency of the index structure. After completing the reconstruction of the index path, the data of the newly merged nodes is written into the storage system, and the index table is updated synchronously to ensure that subsequent query requests can correctly access the merged data distribution. After the data writing is completed, the integrity check of the updated data distribution is performed to check whether the index structure still remains correct, including checking whether the depth of the index tree remains balanced, whether all data can be correctly retrieved through the index path, and whether the newly generated nodes conform to the spatial coverage rules. After completing the integrity check, the read-write lock of the merged nodes is released, so that these nodes can provide query access externally again, obtaining an optimized index tree structure.

[0018] Perform query tests on the optimized index tree structure to evaluate the query performance and system load of the index in actual applications. During the query tests, simulate different types of query requests, including range queries, K-nearest neighbor queries, and spatio-temporal aggregation queries, and record the number of query requests completed and the response time of each query request within a unit time, so as to calculate the query throughput data. The measurement criteria for query throughput include the number of queries processed per second and the average latency of query requests. These data reflect the impact of index optimization on query performance and ensure that index adjustment can improve query efficiency. Calculate the load balance based on the data volume distribution of each computing node to measure whether the storage and computing loads of the data are evenly distributed after index optimization. Since in a distributed system, the balance of storage load and computing tasks on different computing nodes directly affects the overall performance of the system, calculate the degree of dispersion of the data distribution and quantify the load distribution through the method of information entropy to obtain the load distribution entropy value. The calculation of the load distribution entropy value is based on the probability density of the data distribution. If the data volumes of each computing node are relatively balanced, the entropy value is higher, while if the data volume of some nodes far exceeds that of other nodes, the entropy value is lower. Therefore, optimize the data sharding and migration strategies to improve the load balance of the system and reduce the impact of query hotspots. At the same time, sample and statistically analyze the CPU usage rate, memory occupancy rate, and disk I / O of each computing node to calculate the utilization level of computing node resources. The CPU usage rate reflects the computing load of the computing node during query and index update operations, the memory occupancy rate determines the storage efficiency of index data and cache data, and the disk I / O performance affects the throughput capacity of data query and index update. By regularly sampling these resource utilization data, obtain the occupancy of the current index structure on system resources, so as to evaluate whether index adjustment has brought optimization of computing and storage resources. The calculation of resource utilization data is through the weighted average method, which integrates the utilization rates of CPU, memory, and disk I / O to form a comprehensive resource utilization evaluation index. Normalize the query throughput data, load distribution entropy value, and resource utilization data numerically, using the Min-Max normalization or Z-score standardization method to convert the data of each index to a unified numerical range, so as to eliminate the impact of different data scales on the optimization calculation. After normalization, assign weight coefficients to each index according to the performance optimization goal of the system to form a performance evaluation weight. The setting of the weight is dynamically adjusted based on different optimization requirements of the system. Perform weighted summation on each performance index based on the performance evaluation weight to calculate the comprehensive performance score of the current index structure. Construct a performance optimization objective function, and the value of this objective function is used to measure the optimization degree of the current index structure and guide the generation of the next round of index optimization strategies.The calculation method of the objective function adopts the linear weighting method, which weights and sums the normalized values of query throughput, load balancing, and resource utilization according to their respective weights to obtain a quantified comprehensive performance score. Based on the performance optimization objective function, the next round of index optimization strategy is generated. The index optimization strategy includes three core aspects, namely, the node splitting threshold, the merging condition, and the data migration strategy. The adjustment of the node splitting threshold is to ensure the stability of query efficiency during the data growth process of the index tree. If the query throughput is low and the load balancing is poor, the splitting threshold is reduced, causing the index tree to perform splitting operations earlier to avoid excessive data volume growth in some nodes and affecting query performance. The adjustment of the merging condition is to optimize the storage space occupancy rate. If it is found that the current index structure leads to too high a storage space utilization rate or the data distribution in the query hot area is too scattered, the merging threshold is appropriately reduced, enabling data nodes to merge more frequently, thereby reducing storage redundancy and improving query efficiency. The adjustment of the data migration strategy is to optimize the load balancing of computing nodes. If the load distribution entropy value is low, it indicates that some computing nodes consume too much storage or computing resources. Therefore, the data sharding rules are readjusted to migrate some data on high-load nodes to low-load nodes to ensure that the computing power of the entire distributed system can be evenly utilized. Through the optimization process, based on the comprehensive analysis of multiple performance indicators such as query throughput, load balancing, and resource utilization, the index structure is continuously adjusted to ensure that the index can adapt to data growth and changes in query patterns, thereby improving the overall performance and scalability of the system.

[0019] In an example, the spatio-temporal data stream is input into the distributed system for sharding processing and index tree structure construction to obtain the initial index structure, and parallel feature extraction is performed on the spatio-temporal data stream to obtain spatio-temporal distribution feature vectors, including: The spatio-temporal data stream is input into the distributed system for data range analysis and boundary calculation to obtain a data sharding vector with the value range of the time dimension and the distribution range of the space dimension; The hash value distribution of virtual nodes is calculated according to the data sharding vector, and a load weight coefficient is assigned to each virtual node to obtain a data sharding mapping table; According to the data sharding mapping table, the spatio-temporal data stream is recursively divided at multiple levels to obtain a distributed index tree; Function module parameters are configured for each leaf node in the distributed index tree, and a data write buffer, a split trigger, and a merge processor are respectively deployed to obtain a data receiving unit, a node splitting unit, and a node merging unit; The data receiving unit, the node splitting unit, and the node merging unit are registered in the distributed coordination center, and an inter-node communication channel based on the message queue is established to obtain a global index directory table; Set distributed transaction control parameters according to the global index directory table, and assign read-write lock identifiers to data access operations between nodes to obtain an initial index structure; Perform parallel feature extraction on the spatio-temporal data stream according to a preset time window, and calculate the spatio-temporal distribution feature vectors within each time window based on the data sharding rules in the initial index structure.

[0020] In this example, the spatio-temporal data stream is input into a distributed computing environment and range analysis and boundary calculation are performed on the data to determine the value range of the time dimension and the distribution range of the space dimension. Assume the input data stream is , where represents the timestamp, and represent the spatial coordinates. Then, the value range is determined by calculating the maximum and minimum values of the data stream in time and space, that is: Among them, and represent the start and end times of the time dimension respectively, while represent the boundary range of the spatial coordinates. Based on these values, construct the value range of the time dimension and the distribution range of the space dimension to obtain the data sharding vector : Calculate the hash value distribution of virtual nodes according to the data sharding vector, and assign a load weight coefficient to each virtual node to generate a data sharding mapping table. Assume the system uses consistent hashing for data storage management, then use the hash function to map the data: Among them, represents the number of virtual nodes, and the hash function maps the data point to a specific virtual node. To ensure load balancing, calculate the load weight of each virtual node: Among them, represents the data volume allocated to the th virtual node, while Represents the total amount of data on all nodes. In this way, a data shard mapping table is generated to ensure the balanced distribution of data in a distributed environment. Based on the data shard mapping table, the spatio-temporal data stream is recursively divided at multiple levels to construct a distributed index tree. For example, an R-tree or a quadtree is used for index construction, where the R-tree organizes data through recursive partitioning. For each node , its minimum bounding rectangle is calculated as: And when inserting new data, if the amount of data in a certain leaf node exceeds the set threshold , then node splitting is triggered, and it splits into child nodes and , and its boundary range is recalculated: Thus, a multi-level index tree structure is formed to optimize the query efficiency. After the index tree is constructed, function module parameters are configured for each leaf node in the distributed index tree, including a data write buffer, a split trigger, and a merge processor, to form a data receiving unit, a node splitting unit, and a node merging unit. Among them, the data write buffer is used to store newly written data and batch-update the index when a certain amount is accumulated, while the split trigger is used to detect the data growth situation and automatically perform node splitting operations when the data volume exceeds the threshold. The merge processor is used to monitor low-density areas and perform node merging when the data is sparse to reduce storage overhead. To ensure the efficient operation of the distributed index, the function modules are registered with the distributed coordination center, and a node-to-node communication channel based on a message queue is established to maintain the global index directory table. The global index directory table stores information about all index nodes and is used for the distribution of query requests. For example, with the support of Kafka or RabbitMQ message queues, index updates are synchronized to multiple computing nodes through a publish-subscribe mechanism to ensure that the latest data can be accessed during queries. On this basis, distributed transaction control parameters are set according to the global index directory table, and read-write lock identifiers are assigned to data access operations between nodes to ensure transaction consistency during index updates. Assuming a two-phase locking strategy, during a query operation, the node reading the index will acquire a shared lock , while during index updates, an exclusive lock is required to avoid concurrent modification conflicts. Through these transaction control strategies, data consistency and query efficiency are ensured. After the index structure is constructed, parallel feature extraction is performed on the spatio-temporal data stream according to a preset time window, and the spatio-temporal distribution feature vectors within each time window are calculated based on the data shard rules in the initial index structure. For example, a time window is defined as a sliding window, and the data density is calculated within each window : wherein, represents the number of data within the window, and is the time span of the window. Meanwhile, calculate the spatial distribution uniformity of the data to measure the spatial expansion of the data: wherein, represents the data density of a certain grid cell, is the average density of all cells, and is the total number of cells. Through calculation, obtain the spatio-temporal distribution feature vector of each time window for optimizing the adaptive adjustment of the index.

[0021] In an example, perform parallel feature extraction on the spatio-temporal data stream according to a preset time window, and calculate the spatio-temporal distribution feature vectors within each time window based on the data sharding rule in the initial index structure, including: Segment and slice the spatio-temporal data stream according to the preset time window, and calculate the time aggregation degree and spatial distribution density of the data samples within each time window to obtain a window-level feature matrix; Standardize the window-level feature matrix according to the data sharding rule of the initial index structure to obtain a standardized feature sequence; Based on the standardized feature sequence, calculate the covariance of the feature distributions in the time dimension and the spatial dimension to obtain a dimension correlation matrix, and calculate the main feature direction according to the dimension correlation matrix to perform feature dimensionality reduction on the distribution law of the data samples in different spatio-temporal dimensions to obtain a compressed feature set; Perform weighted fusion on the time aggregation feature and the spatial distribution feature of the compressed feature set to obtain a comprehensive feature vector, and perform binary feature coding on the comprehensive feature vector to obtain a spatio-temporal distribution feature vector.

[0022] In this example, segment and slice the spatio-temporal data stream according to the preset time window for feature analysis within different time ranges. Assume the spatio-temporal data stream is , wherein represents the timestamp, and represent the spatial coordinates, then define the time window as a sliding window: wherein, is the start time of the current window, and is the time span of the window, ensuring that the data can be segmented at fixed time intervals. For the data samples within each time window, calculate the time aggregation degree and the spatial distribution density to obtain the window-level feature matrix. The time aggregation degree is used to measure the concentration degree of data points in the time dimension, and its calculation formula is: where represents the number of data points within the time window and is the duration of the time window. The higher the time aggregation degree, the higher the concentration degree of the data within that time period, indicating a hot spot area where events occur. At the same time, calculate the spatial distribution density , which is used to measure the distribution of data in the spatial dimension. The spatial distribution density is calculated using a grid-based statistical method. Assume that the entire space is divided into grid cells, and the density of each cell is calculated as follows: where is the number of data points falling into the th grid cell within the window and is the area of the grid cell. Construct the window-level feature matrix : This matrix contains the spatial distribution of the data within the time window and is used for subsequent feature extraction and dimensionality reduction. After obtaining the window-level feature matrix, standardize it according to the data sharding rules of the initial index structure to eliminate the imbalance in data distribution between different time windows. The standardization method uses mean normalization or Z-score normalization. The formula for mean normalization is as follows: where and are the minimum and maximum values of the entire matrix respectively. For Z-score standardization, its calculation formula is: where is the average density of all grid cells, and is the standard deviation of density. Through standardization, the feature matrix is converted into a standardized feature sequence, making the data in different time windows comparable. Based on the standardized feature sequence, the feature distribution covariance in the time dimension and the space dimension is calculated to analyze the correlation between data distributions, and principal component analysis is used for dimensionality reduction. The covariance matrix is calculated as follows: where represents the feature vector within the window and is the mean of all window feature vectors. The eigenvectors of the covariance matrix are used to calculate the principal feature direction, that is, to find the dimension with the most significant change in data distribution. Assuming the feature vector matrix is , the principal feature direction is solved by singular value decomposition: where are the eigenvalues and are the corresponding eigenvectors. The eigenvector corresponding to the largest eigenvalue is selected as the principal direction, and the data is projected onto this direction to obtain the compressed feature set after dimensionality reduction. After obtaining the compressed feature set, the time aggregation feature and the space distribution feature are weighted and fused to form a comprehensive feature vector. The formula for weighted fusion is: where and are the weight coefficients of the time and space features respectively, which are dynamically adjusted according to the query requirements. To optimize the storage efficiency and computational performance, the comprehensive feature vector is binary-coded to form a spatio-temporal distribution feature vector. The generation method of the binary feature code adopts hash coding or quantization methods, such as binary coding based on uniform quantization: where and are the minimum and maximum values of all comprehensive feature vectors respectively, is the number of coding bits, and bin represents the binary conversion function. By this method, the high-dimensional spatio-temporal feature data is compressed into a binary code with a fixed length, thereby improving the index query efficiency.

[0023] In an example, based on the spatio-temporal distribution feature vector, the spatio-temporal overlap degree, data density similarity, and query frequency correlation degree of adjacent nodes are calculated to obtain the combined evaluation score of node pairs, including: Inverse quantization decoding is performed on the spatio-temporal distribution feature vector to obtain the time and space distribution features of the node, and the minimum bounding rectangle is constructed according to the time and space distribution features of the node; Construct an R-tree index according to the minimum bounding rectangle, generate a hierarchical spatial index structure through recursive spatial partitioning, perform a K-nearest neighbor search on the hierarchical spatial index structure to obtain a list of node pairs; Calculate the overlap degree of the data boundaries of each node pair in the node pair list to obtain the node spatio-temporal overlap degree, and calculate the data density of the node pairs in the node pair list to obtain the data density similarity; Perform frequency analysis on the query access records of the node pairs to obtain the query frequency correlation, and normalize the spatio-temporal overlap degree, density similarity score, and query frequency correlation to obtain the combined evaluation score of the node pairs.

[0024] In this example, inverse quantization decoding is performed on the spatio-temporal distribution feature vector to restore the time and space distribution features of the nodes. Assume that the binary encoding form of the feature vector is , where each represents the binary encoding value of a feature dimension, and the decoding process restores the original data through inverse mapping. For the time feature and the spatial feature , their decoding formulas are as follows: where are the quantized binary values of the time and space axes and axis respectively, is the minimum and maximum values of the time range, are the boundaries of the spatial range respectively, and and are the corresponding number of binary encoding bits. Through inverse quantization calculation, the original time distribution and spatial coordinates are restored to obtain the time and space distribution features of the nodes. Construct a minimum bounding rectangle based on the spatio-temporal distribution of the nodes for the construction of the spatial index. Assume that the data coverage range of a certain node is , then its minimum bounding rectangle is defined as follows: where is the minimum coordinate value among all data points, and is the maximum coordinate value. For the time dimension, define the time span: The MBRs of all nodes are organized using an R-tree index for subsequent efficient queries. The construction of the R-tree is a recursive process. During index construction, the MBRs of each node are stored hierarchically according to the spatial hierarchy. Assume the hierarchical index of the R-tree is (where represents the level). For each level of nodes , if the number of its child nodes exceeds the set threshold , then node splitting is performed to form multiple new child nodes, and the MBR of its parent node is updated to cover the ranges of all child nodes: where, is the parent node, is the child node, is the number of child nodes. In this way, a hierarchical spatial index structure is generated, enabling query operations to be efficiently executed through hierarchical traversal. After the index construction is completed, a K-nearest neighbor search is performed on the index structure to find a list of adjacent node pairs. K-nearest neighbor search is a method for querying the nearest neighbor data points in space. Given a query point , its K-nearest neighbor set consists of nodes that satisfy the following conditions: where, is the Euclidean distance from node to the query point , and the calculation formula is: After obtaining the list of adjacent node pairs, calculate the spatio-temporal overlap degree of each node pair to measure their merging potential. The spatio-temporal overlap degree is defined as the ratio of the intersection area of the MBRs of two nodes to the area of the smallest MBR: where Area(·) represents the area calculation of a rectangle. The higher the spatio-temporal overlap degree, the greater the overlap of the spatial regions of the two nodes, and they are more suitable for merging. Calculate the data density similarity , which measures whether the data distributions of two nodes are consistent. The density similarity is calculated through the density difference of data points: where, Area(MBR and Area(MBR respectively represent the data densities of two nodes. The higher the density similarity, the closer the data distributions are, and the smaller the impact on the index after merging. Analyze the query frequency correlation , to evaluate whether the query patterns of the two nodes are similar. The query frequency correlation is statistically calculated from the query logs, and the query count ratio is calculated as follows: where and are the query counts of the two nodes. The higher the frequency correlation, the more similar their access patterns are in historical queries and they are optimized for merging. For the spatio-temporal overlap , density similarity and query frequency correlation , normalization processing is performed to ensure weighted calculation of metrics with different measurement units. The normalization uses the Min-Max method: where represents , or 's original value, and are its minimum and maximum values. Calculate the merging evaluation score for the node pair: where are the weight coefficients of different features. The higher the merging evaluation score of the node pair, the greater their merging potential, and they are preferentially optimized for index merging.

[0025] In an example, multi-objective optimization is performed based on the merging evaluation score to generate an optimal merging threshold, including: Real-time monitor the system load level, query response latency, and storage space occupancy rate to obtain a set of system performance parameters; According to the set of system performance parameters, set weight coefficients for each performance metric to obtain multi-objective constraint conditions; Based on the multi-objective constraint conditions, segment the merging evaluation score, construct a state transition equation, and obtain a dynamic programming solution model; Perform recursive calculation on the dynamic programming solution model, merge the solutions of sub-problems in a bottom-up manner to obtain a candidate threshold sequence; Evaluate and calculate the historical merging operations according to the candidate threshold sequence, obtain the performance evaluation values corresponding to each candidate threshold, and sort the candidate thresholds based on the performance evaluation values, and select the threshold with the highest performance evaluation as the optimal merging threshold.

[0026] In this example, the key performance indicators of the system are monitored in real time to construct a set of system performance parameters. The key performance parameters include the system load level ( ), the query response latency ( ), and the storage space occupancy rate ( ). The system load level is determined by calculating the comprehensive weighted average of the CPU usage rate, memory occupancy, and I / O load: where are the weight parameters of the CPU, memory, and I / O load respectively, which are adjusted according to specific business requirements. The query response latency reflects the query performance of the system, and the calculation method is: where represents the response time of the th query request, and is the number of queries completed within the monitoring time window. The storage space occupancy rate is used to measure the storage overhead of the index structure and is expressed as: Weight coefficients are set for each performance indicator according to the optimization requirements of the system to form multi-objective constraint conditions. These weight coefficients determine the importance of different performance objectives in the index optimization process. The set optimization objective function is expressed as: where are the weight factors of the system for the load level, query latency, and storage occupancy rate. For services with high query throughput, is higher, while for scenarios with storage constraints, is larger. After constructing the multi-objective constraint conditions, the combined evaluation scores are segmented to form a state transition equation suitable for dynamic programming solution. Let be the combined evaluation score of the th layer and the th node in the index tree. Then the segmentation is performed by setting a threshold for partitioning: where and are empirical thresholds calculated based on the system historical data. A dynamic programming solution model is constructed, and the optimal merging path is described through the state transition equation. Let represent the optimal merging strategy for the first nodes, then its recurrence equation is: Among them, represents the merging cost of a single node, represents the merging cost of two adjacent nodes, and the calculation method is as follows: Among them, is the spatio-temporal overlap degree of nodes and , is the data density similarity, is the query frequency correlation, and is the adjustment weight. This recursive solution calculates the optimal solution through dynamic programming, that is, it is optimized in a bottom-up manner. First, it solves sub-problems of smaller scales and gradually merges them to obtain the complete optimal solution. After solving the dynamic programming model, multiple candidate threshold sequences are obtained, that is, different index structures corresponding to different merging strategies. Let the candidate threshold sequence be , and evaluate its historical merging operations to find the optimal threshold. The performance evaluation value corresponding to each threshold is calculated as follows: Among them, is the system load level when using the threshold , is the query latency, is the storage occupancy rate. These values are calculated through backtesting of historical data, that is, simulating the execution results of different merging strategies in the historical query log and calculating their corresponding performance changes. Sort the evaluation values of all candidate thresholds to select the optimal merging threshold : That is, select the threshold with the optimal comprehensive performance as the final merging threshold.

[0027] In an example, perform an incremental merging operation on node pairs according to the optimal merging threshold, and perform data reorganization and range boundary update to obtain an optimized index tree structure, including: Compare the merging evaluation scores of node pairs with the threshold, add the node pairs higher than the optimal merging threshold to the merging operation queue to obtain a queue of nodes to be merged; Set read-write lock identifiers according to the data access status of the nodes in the queue of nodes to be merged, and lock the data access operations of the merged node pairs to obtain a data isolation state; Based on the data isolation state, calculate the boundaries of the spatio-temporal data in the merged nodes, and merge the overlapping regions of the data ranges to obtain the data boundaries of the new nodes; Recalculate the hierarchical structure of the index tree according to the data boundaries of the new nodes, update the parent-child node relationships, and obtain the modified node links; Reconstruct the index path based on the modified node links, write the merged new node data into the storage system, obtain the updated data distribution, and perform integrity verification on the updated data distribution. Release the read-write locks of the merged nodes to obtain the optimized index tree structure.

[0028] In this example, compare the merging evaluation scores of all node pairs in the index tree with a threshold to determine which nodes need to be merged. Assume that each node pair has a merging evaluation score and the system has calculated the optimal merging threshold , then the condition for a node pair to enter the merging operation queue is: For all node pairs that meet this condition, add them to the merging operation queue to obtain the queue of nodes to be merged : After generating the queue of nodes to be merged, ensure that the query operation is not affected during the merging operation. Therefore, set appropriate read-write lock identifiers for the nodes in the queue of nodes to be merged to achieve data access isolation. Set the data access status of each node . If a node is being accessed by a query operation, its status is "read" (R). If a node is being accessed by an index update or deletion operation, its status is "write" (W). Before the merging operation starts, obtain the exclusive lock of the merged nodes to ensure that there is no data competition during the merging process: When all the merged nodes are correctly locked, the system enters the data isolation state to ensure the atomicity of the merging operation. In the data isolation state, calculate the boundaries of the spatio-temporal data in the merged nodes to ensure that the merged new nodes can correctly represent the coverage range of the original data. Each node and both have the minimum bounding rectangle MBR and MBR . The boundary MBR of the merged new node is calculated by the following formula: where The boundary calculation in the time dimension is similar. Assume that represents the time range of a node. Then the time range of the merged node is: Through these calculations, determine the boundary range of the new node so that it can cover all regions of the original data. After calculating the boundaries of the new node, readjust the hierarchical structure of the index tree to ensure that the index path still remains correct. Delete the original node from the index tree and 's parent-child relationship, and insert the new node as a child of its parent node. If the parent node of the original node has only these two child nodes, then the list of its child nodes needs to be updated: If the number of child nodes of the original parent node exceeds two, recalculate the splitting strategy of the index tree to decide whether the index structure needs to be adjusted. After completing the adjustment of the index tree structure, reconstruct the index path based on the modified node links to ensure that the query operation can correctly locate the data. The way of reconstructing the index path depends on the type of the index tree. For example, in an R-tree, starting from the bottom leaf nodes, update the minimum bounding rectangle MBR of all parent nodes upward until the root node. The index path update formula is as follows: where is the parent node at a certain index level, are all its child nodes, is the number of child nodes. After completing the reconstruction of the index path, write the data of the merged new node into the storage system to form a new data distribution. Before writing to the storage, clean up the data of the original node and ensure that the data of the new node is complete. For example, in a distributed storage system, calculate the data migration volume through the following formula: where represents the incremental storage requirement after data merging. If , it means that the new node requires additional storage space, while if , it means that the merging reduces the storage occupancy. After the data writing is completed, perform integrity verification to ensure the correctness of the index structure. The content of the integrity verification includes: Query path correctness check: Randomly sample query data to ensure that the query can correctly return results; Data consistency check: Verify that the number of data points of the new node is equal to the total data volume before merging, that is: Index tree balance check: Calculate the maximum hierarchical depth of the index tree Whether it is maintained within a reasonable range, that is: where is the branching factor of the index tree, is the total number of nodes in the index tree. After completing the integrity check, release the read-write lock of the merged nodes to restore query access rights, obtain the optimized index tree structure, enable it to adapt to the dynamic changes of data distribution, and improve query efficiency.

[0029] In one example, the method for dynamically constructing spatio-temporal big data indexes further includes: Conduct query tests on the optimized index tree structure, record the number of query requests completed per unit time and the query response time, and obtain query throughput data; Calculate the load balance based on the data volume distribution of each computing node, quantify the degree of dispersion of the data distribution, and obtain the load distribution entropy value; Sample and statistically analyze the CPU usage rate, memory occupancy rate, and disk I / O of each computing node, calculate the resource utilization level of the computing node, and obtain resource utilization data; Normalize the query throughput data, load distribution entropy value, and resource utilization data numerically, and assign weight coefficients to each index according to the performance optimization goal to obtain the performance evaluation weights; Based on the performance evaluation weights, perform weighted summation on each performance index, calculate the comprehensive performance score of the current index structure, and obtain the performance optimization objective function; Generate the next round of index optimization strategies according to the performance optimization objective function. The next round of index optimization strategies includes node splitting thresholds, merging conditions, and data migration strategies.

[0030] In this example, query tests are conducted on the optimized index tree structure to evaluate the query performance of the index and provide a basis for subsequent optimization. The main goal of the query test is to record the number of query requests completed per unit time and the query response time, so as to calculate the query throughput data. It is set that within the time window the total number of query requests received by the system is , then the query throughput is expressed as: where represents the number of query requests completed per unit time, that is, the query throughput. At the same time, in order to measure the latency performance of the query, record the response time of all query requests and calculate the average query response time : where represents the The response time of a query request. The higher the query throughput and the lower the query response time, the better the index optimization effect. After obtaining the query throughput data, analyze the data distribution on each computing node to calculate the load balance and quantify the dispersion degree of the data distribution. Assume the system contains computing nodes, and the data volumes stored on each computing node are respectively. Then the total data volume of the computing nodes is: To measure the load balance, calculate the information entropy of the data on each computing node, that is, the load distribution entropy value : Among them, represents the data proportion occupied by the th computing node: When the data distribution of all computing nodes tends to be balanced, the entropy value is close to the maximum value, indicating good load balance; when the data proportion of some nodes is too large, the entropy value decreases, indicating that there is a problem of load skew in the system and optimization is needed. After the load balance analysis is completed, sample and count the CPU usage rate, memory occupancy rate, and disk I / O of each computing node to calculate the resource utilization level. Let be the CPU usage rate of the th computing node, be the memory occupancy rate, be the disk I / O load, then the comprehensive resource utilization rate is calculated as: Among them, are the weight coefficients of CPU, memory, and disk I / O, adjusting the relative importance of different computing resources. The resource utilization rate reflects the overall load situation of the system after index optimization. If the resource utilization rate is too high, the query performance will decline, and if the utilization rate is too low, it means that the computing resources are not fully utilized. Normalize the query throughput data, load distribution entropy value, and resource utilization rate data for comparison between different metrics. The normalization uses the Min-Max normalization method to map all metrics to the range of [0,1]: Among them, is the original value, and are the minimum and maximum values of this metric respectively. For the query throughput and the load distribution entropy value and resource utilization , and the normalized values are respectively expressed as . To form the final comprehensive performance evaluation, weight coefficients are assigned to each indicator according to the performance optimization goal to obtain the performance evaluation weights , and the comprehensive performance score of the current index structure is calculated : Among them, is used to measure the query throughput, focuses on load balancing, focuses on system resource utilization. The final comprehensive performance score represents the overall optimization effect of the current index structure. The higher the score, the better the performance of the index structure in all aspects. After calculating the comprehensive performance score of the current index structure, the next round of index optimization strategies are generated based on the performance optimization objective function, including the node splitting threshold, merging conditions, and data migration strategy. The node splitting threshold is dynamically adjusted to make the index tree split faster in high-load areas and improve the query efficiency: Among them, is the initial splitting threshold, is the adjustment factor. When the load is unbalanced ( is low), the splitting threshold is reduced to accelerate the index adjustment. The merging conditions are determined by the query throughput and storage utilization. If the query throughput is low and the storage space occupancy is too high, the merging operation is accelerated: Among them, is the default merging threshold, controls the merging speed, reducing the merging when the query load is high and increasing the merging when the storage load is high. The data migration strategy determines the redistribution of data by calculating the load and resource utilization of each computing node. When the load of a certain node is much higher than the average value, some data needs to be migrated: When ( is the load balancing threshold), part of the data is migrated to the computing node with a lower load to improve the balance. Through the above optimization strategies, the index tree is dynamically adjusted according to the system load, query performance, and storage requirements.

[0031] Optionally, multi-objective optimization is performed based on the combined evaluation score to generate an optimal merging threshold, including: converting the physical connection relationship of computing nodes into an adjacency matrix, constructing a weighted connected graph according to the network latency and bandwidth between nodes, and determining the node communication path based on the minimum spanning tree algorithm. Assign a local objective function including query response time, resource utilization rate, and data distribution balance degree to each node to obtain a distributed optimization constraint model; construct a two-layer optimization problem according to the overall system performance index and node local resource constraints, where the upper-layer constraints include global query throughput and load balance degree, and the lower-layer constraints include the upper limit of node CPU utilization rate, memory capacity limit, and network bandwidth threshold. Convert the constraints into penalty terms through the Lagrange multiplier method to obtain an augmented objective function; calculate the first-order gradient of the augmented objective function. Each node constructs a momentum term based on the current state and historical gradient information, and adjusts the parameter update step size through an adaptive learning rate. The node only exchanges local gradient information with directly adjacent nodes and uses an asynchronous communication mechanism for parameter update to obtain a first-order optimization result; construct an approximation of the Hessian matrix based on the first-order optimization result, decompose the global Hessian matrix into local sub-matrices through the block diagonalization method, and each node only calculates and stores the Hessian matrix block related to itself. Use the trust region method to determine the optimization step size to obtain a local second-order correction; perform parallel calculation on the local second-order correction, adopt a pipelined communication strategy between nodes, decompose the large-scale data exchange into multiple small batches, update the local parameters through an asynchronous iteration method, and dynamically adjust the iteration step size based on the feedback information of neighbor nodes to obtain a local optimal solution; input the local optimal solution into the convergence analysis module, calculate the parameter change rate and the improvement degree of the objective function value between adjacent iterations, judge the convergence state of the optimization process according to the preset convergence threshold, and dynamically adjust the node communication frequency and optimization step size to obtain a global solution that meets the convergence conditions; perform normalization processing on the global solution, calculate the normalization coefficient based on the current load level and performance requirements of the system, and unify the performance indicators in different dimensions into the same measurement space to obtain a normalized performance metric value; construct a non-linear mapping function according to the normalized performance metric value, define the threshold mapping rule in the form of a piecewise function, use a smaller mapping coefficient in the performance sensitive interval to improve the sensitivity of the threshold, and use a larger mapping coefficient in the performance stable interval to enhance the stability of the threshold to obtain the optimal merging threshold.

[0032] Referring to Figure 2 , this embodiment provides a spatio-temporal big data dynamic index construction device, including: A construction module 1, configured to input spatio-temporal data streams into a distributed system for sharding processing and index tree structure construction to obtain an initial index structure, and perform parallel feature extraction on the spatio-temporal data streams to obtain spatio-temporal distribution feature vectors; A calculation module 2, configured to calculate the spatio-temporal overlap degree, data density similarity, and query frequency correlation degree of adjacent nodes based on the spatio-temporal distribution feature vector, and obtain the combined evaluation score of the node pair; A multi-objective optimization module 3, configured to perform multi-objective optimization based on the combined evaluation score to generate an optimal merging threshold; An update module 4, configured to perform an incremental merging operation on the node pair according to the optimal merging threshold, and perform data reorganization and range boundary update to obtain an optimized index tree structure.

[0033] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the description in the above method embodiment, and details are not described herein again.

[0034] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0035] Those of ordinary skill in the art can understand that all or part of the processes of implementing the above method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to a memory, storage, database, or other medium provided by the present invention and used in the embodiments can include non-volatile and / or volatile memories. The non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. The volatile memory can include random access memory (RAM) or an external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM, etc.

[0036] It should be noted that in this text, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article or method comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, apparatus, article or method. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, apparatus, article or method comprising such element.

[0037] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of the present invention.

Claims

1. A method for constructing a dynamic index of spatio-temporal big data, characterized in that, Including the following steps: Input the spatio-temporal data stream into a distributed system for sharding processing and index tree structure construction to obtain an initial index structure, and perform parallel feature extraction on the spatio-temporal data stream to obtain spatio-temporal distribution feature vectors; Based on the spatio-temporal distribution feature vectors, calculate the spatio-temporal overlap degree, data density similarity, and query frequency correlation of adjacent nodes to obtain the combined evaluation scores of node pairs; Perform multi-objective optimization based on the combined evaluation scores to generate an optimal merging threshold; Execute incremental merging operations on node pairs according to the optimal merging threshold, and perform data reorganization and range boundary updates to obtain an optimized index tree structure.

2. The method for constructing a spatio-temporal big data dynamic index according to claim 1, wherein, The step of inputting the spatio-temporal data stream into a distributed system for sharding processing and index tree structure construction to obtain an initial index structure, and performing parallel feature extraction on the spatio-temporal data stream to obtain spatio-temporal distribution feature vectors includes: Input the spatio-temporal data stream into a distributed system for data range analysis and boundary calculation to obtain a data sharding vector with a time dimension value range and a spatial dimension distribution range; Calculate the hash value distribution of virtual nodes according to the data sharding vector, and assign load weight coefficients to each virtual node to obtain a data sharding mapping table; According to the data sharding mapping table, perform multi-level recursive partitioning on the spatio-temporal data stream to obtain a distributed index tree; Configure function module parameters for each leaf node in the distributed index tree, and deploy a data write buffer, a split trigger, and a merge processor respectively to obtain a data receiving unit, a node splitting unit, and a node merging unit; Register the data receiving unit, the node splitting unit, and the node merging unit to a distributed coordination center, establish an inter-node communication channel based on a message queue, and obtain a global index directory table; Set distributed transaction control parameters according to the global index directory table, and assign read-write lock identifiers to data access operations between nodes to obtain an initial index structure; Perform parallel feature extraction on the spatio-temporal data stream according to a preset time window, and calculate spatio-temporal distribution feature vectors within each time window based on the data sharding rules in the initial index structure.

3. The method for constructing a spatio-temporal big data dynamic index according to claim 2, wherein The step of performing parallel feature extraction on the spatio-temporal data stream according to a preset time window, and calculating spatio-temporal distribution feature vectors within each time window based on the data sharding rules in the initial index structure includes: Segment and split the spatio-temporal data stream according to a preset time window, and calculate the time aggregation degree and spatial distribution density of data samples within each time window to obtain a window-level feature matrix; Perform normalization processing on the window-level feature matrix according to the data sharding rules of the initial index structure to obtain a normalized feature sequence; Based on the normalized feature sequence, calculate the covariance of the feature distributions in the time dimension and the spatial dimension to obtain a dimension correlation matrix, and calculate the main feature direction according to the dimension correlation matrix, and perform feature dimensionality reduction on the distribution rules of data samples in different spatio-temporal dimensions to obtain a compressed feature set; Perform weighted fusion on the temporal aggregation features and spatial distribution features of the compressed feature set to obtain a comprehensive feature vector, and perform binary feature coding on the comprehensive feature vector to obtain a spatio-temporal distribution feature vector.

4. The method for constructing a spatio-temporal big data dynamic index according to claim 1, wherein Based on the spatio-temporal distribution feature vector, calculate the spatio-temporal overlap degree, data density similarity, and query frequency correlation degree of adjacent nodes to obtain a combined evaluation score for node pairs, including: Perform inverse quantization decoding on the spatio-temporal distribution feature vector to obtain the temporal and spatial distribution features of the nodes, and construct a minimum bounding rectangle according to the temporal and spatial distribution features of the nodes; Perform R-tree index construction according to the minimum bounding rectangle, generate a hierarchical spatial index structure through recursive spatial partitioning, and perform K-nearest neighbor search on the hierarchical spatial index structure to obtain a list of node pairs; Calculate the overlap degree of the data boundaries of each node pair in the node pair list to obtain the spatio-temporal overlap degree of the nodes, and calculate the data density of the node pairs in the node pair list to obtain the data density similarity; Perform frequency analysis on the query access records of the node pairs to obtain the query frequency correlation degree, and perform normalization processing on the spatio-temporal overlap degree, the density similarity score, and the query frequency correlation degree to obtain a combined evaluation score for node pairs.

5. The method for constructing a spatio-temporal big data dynamic index according to claim 1, wherein Based on the combined evaluation score, perform multi-objective optimization to generate an optimal merging threshold, including: Monitor the system load level, query response latency, and storage space occupancy rate in real time to obtain a set of system performance parameters; According to the set of system performance parameters, set weight coefficients for each performance index to obtain multi-objective constraint conditions; Based on the multi-objective constraint conditions, perform segmented partitioning on the combined evaluation score, construct a state transition equation, and obtain a dynamic programming solution model; Perform recursive calculation on the dynamic programming solution model, merge the solutions of sub-problems in a bottom-up manner to obtain a candidate threshold sequence; Evaluate and calculate the historical merging operations according to the candidate threshold sequence to obtain the performance evaluation values corresponding to each candidate threshold, and sort the candidate thresholds based on the performance evaluation values, and select the threshold with the highest performance evaluation as the optimal merging threshold.

6. The method for constructing a spatio-temporal big data dynamic index according to claim 1, characterized in that According to the optimal merging threshold, perform incremental merging operations on node pairs, and perform data reorganization and range boundary update to obtain an optimized index tree structure, including: Compare the combined evaluation scores of node pairs with the threshold, and add the node pairs higher than the optimal merging threshold to the merging operation queue to obtain a queue of nodes to be merged; Set read-write lock flags according to the data access status of the nodes in the queue of nodes to be merged, and lock the data access operations of the merged node pairs to obtain a data isolation state; Based on the data isolation state, calculate the boundaries of the spatio-temporal data in the merged nodes, and merge the overlapping regions of the data ranges to obtain the data boundaries of the new nodes; Recalculate the hierarchical structure of the index tree according to the data boundaries of the new nodes, and update the parent-child node relationship to obtain the modified node links. Reconstruct the index path based on the modified node links, write the merged new node data into the storage system to obtain the updated data distribution, and perform integrity verification on the updated data distribution. Release the read-write locks of the merged nodes to obtain the optimized index tree structure.

7. The method for constructing a spatio-temporal big data dynamic index according to claim 1, wherein The method for constructing a spatio-temporal big data dynamic index further includes: Perform query tests on the optimized index tree structure, record the number of query requests completed and the query response time per unit time to obtain query throughput data; Calculate the load balance based on the data volume distribution of each computing node, quantify the dispersion degree of the data distribution to obtain the load distribution entropy value; Sample and count the CPU usage rate, memory occupancy rate, and disk I / O of each computing node, calculate the resource utilization level of the computing node to obtain resource utilization data; Normalize the query throughput data, the load distribution entropy value, and the resource utilization data numerically, and assign weight coefficients to each index according to the performance optimization goal to obtain the performance evaluation weights; Perform weighted summation on each performance index based on the performance evaluation weights, calculate the comprehensive performance score of the current index structure to obtain the performance optimization objective function; Generate the next round of index optimization strategies according to the performance optimization objective function. The next round of index optimization strategies includes node splitting thresholds, merging conditions, and data migration strategies.

8. A device for constructing a dynamic index of spatio-temporal big data, characterized in that, For implementing the steps of the method according to any one of claims 1 to 7, the device includes: A construction module for inputting spatio-temporal data streams into a distributed system for sharding processing and index tree structure construction to obtain an initial index structure, and performing parallel feature extraction on the spatio-temporal data streams to obtain spatio-temporal distribution feature vectors; A calculation module for calculating the spatio-temporal overlap degree, data density similarity, and query frequency correlation degree of adjacent nodes based on the spatio-temporal distribution feature vectors to obtain the merging evaluation scores of node pairs; A multi-objective optimization module for performing multi-objective optimization based on the merging evaluation scores to generate an optimal merging threshold; An update module for performing incremental merging operations on node pairs according to the optimal merging threshold, and performing data reorganization and range boundary update to obtain an optimized index tree structure.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images