Storage method, device and storage medium of spatiotemporal big data

Abstract

The embodiment of the application provides a kind of storage method, equipment and storage medium of space-time big data, obtain candidate storage engine and space-time big data;Determine the storage cost of space-time big data on candidate storage engine;According to storage cost, determine first target engine from candidate storage engine, and store space-time big data into first target engine;Determine the storage free energy of space-time big data on candidate storage engine;According to storage free energy, determine second target engine from candidate storage engine;If first target engine is different from second target engine, and the storage free energy of first target engine and the storage free energy of second target engine meet the preset migration condition, then space-time big data is migrated from first target engine to second target engine storage.The embodiment of the application introduces thermodynamic phase change theory into the data migration decision between storage engines, establishes an interpretable data migration physical model, and further improves data storage efficiency.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a method for storing spatiotemporal big data, an electronic device, and a storage medium. Background Technology

[0002] Spatiotemporal big data refers to large-scale datasets that simultaneously possess spatial location and temporal attributes. Data such as moving object trajectories, remote sensing image sequences, time-series sensor observations, and changes in the state of geographic entities all fall under this category. Spatiotemporal big data is characterized by its large total volume, numerous data dimensions, diverse data formats, and dynamically changing access patterns that adapt to changes in the spatiotemporal focus. Furthermore, different types of spatiotemporal data exhibit vastly different query patterns: real-time sensor readings require sub-millisecond point queries, historical trajectory analysis requires gigabyte-level sequential scanning, and remote sensing images require large-block continuous readings. Current data storage solutions, once written to a storage engine, store spatiotemporal big data permanently within that engine, failing to adapt to the aforementioned characteristics of spatiotemporal big data and resulting in low data storage efficiency.

[0003] Therefore, how to store spatiotemporal big data to improve data storage efficiency is an urgent problem to be solved in this field. Summary of the Invention

[0004] This application provides a method for storing spatiotemporal big data to address the problem of how to store spatiotemporal big data in order to improve data storage efficiency.

[0005] Accordingly, embodiments of this application also provide a spatiotemporal big data storage device, an electronic device, and a storage medium to ensure the implementation and application of the above methods.

[0006] To address the aforementioned problems, this application discloses a method for storing spatiotemporal big data, the method comprising: Obtain candidate storage engines and the spatiotemporal big data; Determine the storage cost of the spatiotemporal big data on the candidate storage engine, where the storage cost represents the cost of storing the spatiotemporal big data in the candidate storage engine; Based on the storage cost, a first target engine is determined from the candidate storage engines, and the spatiotemporal big data is stored in the first target engine; Determine the storage free energy of the spatiotemporal big data on the candidate storage engine, where the storage free energy represents the stability of the spatiotemporal big data storage in the candidate storage engine; A second target engine is determined from the candidate storage engines based on the storage free energy; If the first target engine is different from the second target engine, and the storage free energy of the first target engine and the storage free energy of the second target engine meet the preset migration conditions, then the spatiotemporal big data will be migrated from the first target engine to the second target engine for storage.

[0007] Optionally, determining the storage cost of the spatiotemporal big data on the candidate storage engine includes: The data features of the spatiotemporal big data and the engine features of the candidate storage engines are obtained. The data features include at least data type, expected query frequency, data volume and expected query pattern. The engine features include at least baseline query latency, unit storage cost and feature query pattern. Based on the data type and the baseline query latency, determine the query latency of the candidate storage engine for the data type; The query latency cost is obtained by multiplying the expected query frequency by the query latency of the candidate storage engine for the data type. Calculate the product of the data volume and the unit storage cost to obtain the economic cost of storage; Determine the degree of matching between the expected query pattern and the feature query pattern to obtain the pattern adaptation penalty; The storage cost of the spatiotemporal big data on the candidate storage engine is determined based on the query latency cost, the storage economic cost, and the pattern adaptation penalty. The step of determining the first target engine from the candidate storage engines based on the storage cost includes: The candidate storage engine with the lowest storage cost is selected from the candidate storage engines and designated as the first target engine.

[0008] Optionally, determining the storage free energy of the spatiotemporal big data on the candidate storage engine includes: Determine the state vector of the spatiotemporal big data at a preset time, wherein the state vector includes access frequency, data density, data volume, and query pattern entropy; The storage energy of the spatiotemporal big data in the candidate storage engine is obtained by multiplying the data volume by the unit storage cost of the candidate storage engine. The storage energy represents the basic storage cost of the spatiotemporal big data in the candidate storage engine. Based on the storage internal energy and the state vector of the spatiotemporal big data at a preset time, the storage free energy of the spatiotemporal big data on the candidate storage engine is determined; The step of determining the second target engine from the candidate storage engines based on the storage free energy includes: The candidate storage engine with the minimum storage free energy is determined from the candidate storage engines and selected as the second target engine.

[0009] Optionally, after determining the second target engine from the candidate storage engines based on the storage free energy, the method further includes: If the first target engine is different from the second target engine, then obtain the engine distance between the first target engine and the second target engine; Based on the engine distance and the data volume of the spatiotemporal big data, the migration cost between the first target engine and the second target engine is determined; Obtain the free energy difference threshold; If the difference between the storage free energy of the first target engine and the storage free energy of the second target engine is greater than the migration cost, and the difference between the storage free energy of the first target engine and the storage free energy of the second target engine is greater than the free energy difference threshold, then the spatiotemporal big data will be migrated from the first target engine to the second target engine for storage.

[0010] Optionally, after storing the spatiotemporal big data into the first target engine, the method further includes: Determine the spatiotemporal popularity of the aforementioned spatiotemporal big data; The heat level corresponding to the spatiotemporal big data is determined based on the spatiotemporal heat, and the heat level has corresponding storage rules; Adjust the storage engine to which the spatiotemporal big data belongs according to the storage rules.

[0011] Optionally, determining the spatiotemporal heat of the spatiotemporal big data includes: Obtain the number of times the spatiotemporal big data is queried within a preset time window, the set of hotspot spatial regions, and the spatial range of the spatiotemporal big data; The temporal popularity of the spatiotemporal big data is determined based on the number of times it is queried. The spatial heat of the spatiotemporal big data is determined based on the set of hotspot spatial regions and the spatial range of the spatiotemporal big data. The spatiotemporal heat of the spatiotemporal big data is obtained by weighted fusion of the temporal heat and the spatial heat.

[0012] Optionally, the heat level includes a hot layer, a warm layer, and a cold layer; the candidate storage engines include a key-value storage engine, a row-based storage engine, and a column-based storage engine; and adjusting the storage engine to which the spatiotemporal big data belongs according to the storage rules includes: If the spatiotemporal big data belongs to the hot layer, then the spatiotemporal big data will be moved from the first target engine to the key-value storage engine and the row storage engine for storage respectively; If the spatiotemporal big data belongs to the thermosphere, then the spatiotemporal big data data is stored in the first target engine; If the spatiotemporal big data belongs to the cold layer, then the spatiotemporal big data will be moved from the first target engine to the columnar storage engine for storage.

[0013] Optionally, the candidate storage engine has a corresponding local spatial index subtree, the local spatial index subtree including multiple spatial sharding information, the multiple spatial sharding information being used to indicate the spatial range of the spatiotemporal big data stored by the candidate storage engine, and the method further includes: Obtain the time partition directory, which is used to indicate the storage engine to which spatiotemporal big data of different time ranges belong; Receive a data query request, the data query request including a query space range and a query time range; Filter the target space shard information that intersects with the query space range from the space shard information of each candidate storage engine, and determine the candidate storage engine corresponding to the target space shard information as the third target engine; Determine the target time range that intersects with the query time range from the time partition directory, and determine the storage engine to which the spatiotemporal big data of the target time range belongs as the fourth target engine; Determine the intersection of the third target engine and the fourth target engine to obtain the storage engine to be queried; The query is performed on the storage engine to be queried according to the data query request, and the query result is obtained.

[0014] This application also discloses a storage device for spatiotemporal big data, the device comprising: The data acquisition module is used to acquire candidate storage engines and the spatiotemporal big data; The cost determination module is used to determine the storage cost of the spatiotemporal big data on the candidate storage engine, wherein the storage cost represents the cost required to store the spatiotemporal big data in the candidate storage engine; The first storage module is used to determine a first target engine from the candidate storage engines according to the storage cost, and store the spatiotemporal big data into the first target engine. The free energy determination module is used to determine the storage free energy of the spatiotemporal big data on the candidate storage engine, wherein the storage free energy represents the stability of the spatiotemporal big data storage in the candidate storage engine. The migration determination module is used to determine a second target engine from the candidate storage engines based on the storage free energy; The second storage module is used to migrate the spatiotemporal big data from the first target engine to the second target engine for storage if the first target engine is different from the second target engine and the storage free energy of the first target engine and the storage free energy of the second target engine meet preset migration conditions.

[0015] This application also discloses an electronic device, including: a processor; and a memory storing executable code thereon, wherein when the executable code is executed, the processor performs one or more spatiotemporal big data storage methods as described in the embodiments of this application.

[0016] This application also discloses a machine-readable medium storing executable code thereon, which, when executed, causes a processor to perform one or more spatiotemporal big data storage methods as described in this application.

[0017] Compared with the prior art, the embodiments of this application have the following advantages: In this embodiment, candidate storage engines and spatiotemporal big data are obtained; the storage cost of the spatiotemporal big data on the candidate storage engines is determined, where the storage cost represents the cost required to store the spatiotemporal big data in the candidate storage engines; a first target engine is determined from the candidate storage engines based on the storage cost, and the spatiotemporal big data is stored in the first target engine; the storage free energy of the spatiotemporal big data on the candidate storage engines is determined, where the storage free energy represents the stability of the spatiotemporal big data stored in the candidate storage engines; a second target engine is determined from the candidate storage engines based on the storage free energy; if the first target engine and the second target engine are different, and the storage free energy of the first target engine and the storage free energy of the second target engine satisfy a preset migration condition, then the spatiotemporal big data is migrated from the first target engine to the second target engine for storage.

[0018] This application embodiment considers the storage cost of storing spatiotemporal big data in candidate storage engines and automatically selects the first target engine for initial writing based on the storage cost of spatiotemporal big data in candidate storage engines. This avoids situations where the storage cost is too high due to incompatibility between the total amount of spatiotemporal big data, data dimensions, data format, query mode, and other characteristics of the spatiotemporal big data and the candidate storage engines. Simultaneously, this application embodiment introduces thermodynamic phase transition theory into the data migration decision between storage engines. Storage free energy is used to characterize the stability of spatiotemporal big data stored in candidate storage engines, and data migration is triggered based on storage free energy, thereby establishing an interpretable physical model for data migration and improving data storage efficiency. In this application embodiment, after spatiotemporal big data is written to the first target engine according to the storage cost, it is not fixed to that first target engine. Instead, it can select a suitable second target engine again based on the dynamic changes in storage free energy. When the first target engine and the second target engine are different, and the storage free energy of the first target engine and the storage free energy of the second target engine meet preset migration conditions, the spatiotemporal big data is migrated from the first target engine to the second target engine for storage. This transforms the selection of storage engines from a static manual decision to a dynamic adaptive decision, requiring no manual intervention throughout the process. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the steps of an embodiment of a spatiotemporal big data storage method according to this application; Figure 2 This is a three-phase diagram of an embodiment of a spatiotemporal big data storage method according to this application; Figure 3 This is a migration lag diagram of an embodiment of a spatiotemporal big data storage method of this application; Figure 4 This is a phase transition flowchart of an embodiment of a spatiotemporal big data storage method according to this application; Figure 5 This is a schematic diagram of a federated R-tree index, representing an embodiment of a spatiotemporal big data storage method according to this application. Figure 6 This is a system architecture diagram of an embodiment of a spatiotemporal big data storage method according to this application; Figure 7 This is a structural block diagram of an embodiment of a spatiotemporal big data storage device according to this application; Figure 8 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0020] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Spatiotemporal big data refers to large-scale datasets that simultaneously possess spatial location and temporal attributes. Data such as moving object trajectories, remote sensing image sequences, sensor time-series observations, and changes in the state of geographic entities all fall under this category. Spatiotemporal big data accumulates at the petabyte (PB) level and possesses multi-dimensional data including spatial x-axis, spatial y-axis, spatial z-axis, time t, and attributes. It also includes data in various formats such as vectors, raster, point clouds, and trajectories. Furthermore, the access patterns for spatiotemporal big data dynamically shift with changes in the spatial area of ​​interest and the temporal window of interest. Therefore, spatiotemporal big data is characterized by its large total data volume, multiple data dimensions, diverse data formats, and dynamically changing access patterns that change with the spatiotemporal focus. Different types of spatiotemporal data exhibit vastly different query patterns; real-time sensor readings require sub-millisecond point queries, historical trajectory analysis requires gigabyte-level sequential scanning, and remote sensing images require large-block continuous readings. How to store spatiotemporal big data is a challenging research direction in this field.

[0022] For example, in some related technologies, the technical solutions for spatiotemporal big data storage and querying are as follows: The first example is a single storage backend solution, which encodes all spatiotemporal big data uniformly and writes it into the same storage system. However, this solution has insufficient throughput for analytical query patterns and is difficult to efficiently compress large blocks of binary data such as raster and point cloud. The second example is a multi-backend static partitioning solution, which statically routes different data types to a fixed storage sub-engine during writing. It cannot migrate across engines afterward and cannot adaptively adjust when the access patterns of spatiotemporal big data change. The third example is a federated query solution, which performs federated routing at the query layer. However, the underlying storage engines run independently and lack a unified index federation layer and cross-engine data migration mechanism, resulting in poor performance of cross-engine joint queries and the inability to automatically transfer hot and cold data.

[0023] In addition to the examples mentioned above, there are many other technical solutions for spatiotemporal big data storage and querying. However, these solutions typically suffer from drawbacks such as static binding between the storage engine and data types, cold and hot tiering based solely on time or access frequency, strong coupling between the index structure and the storage engine, data being permanently stored in the engine once written and lacking dynamic migration capabilities, and migration decisions relying on heuristic rules and lacking interpretable and adjustable physical model guidance.

[0024] This application addresses the shortcomings of the aforementioned related technologies by proposing a method for storing spatiotemporal big data. This method transforms the selection of storage engines from static manual decision-making to dynamic adaptive decision-making. Furthermore, the basis for hot / cold data stratification is upgraded from a single-dimensional stratification basis to a spatiotemporal dual-dimensional stratification basis. Thermodynamic phase transition theory is introduced into the data migration decision-making between storage engines, establishing an interpretable, adjustable, and oscillation-resistant physical model. Simultaneously, the index layer and storage layer are decoupled into a federated architecture, thereby resolving the deficiencies of related technologies in spatiotemporal big data storage and querying.

[0025] Reference Figure 1 This is a flowchart illustrating the steps of an embodiment of a spatiotemporal big data storage method according to this application, including the following steps: Step 101: Obtain candidate storage engines and the spatiotemporal big data.

[0026] In step 101, spatiotemporal big data and candidate storage engines are obtained. Candidate storage engines are storage engines that can be used to store spatiotemporal big data.

[0027] In one embodiment, the candidate storage engines of this application can be three: a key-value storage engine, a row-based storage engine, and a column-based storage engine. The key-value storage engine (KV storage engine) uses memory and solid-state drives (SSDs) as storage media; the row-based storage engine uses a B+ tree as its data index structure; and the column-based storage engine uses columnar compression technology for data storage. These three engines cover three commonly used spatiotemporal big data query modes: point lookup, range lookup, and scan analysis.

[0028] Step 102: Determine the storage cost of the spatiotemporal big data on the candidate storage engine. The storage cost represents the cost required to store the spatiotemporal big data in the candidate storage engine.

[0029] When the data characteristics of spatiotemporal big data are mismatched with the engine characteristics of candidate storage engines, it will lead to a large query delay. Therefore, in step 102, the storage cost of spatiotemporal big data on candidate storage engines is determined. The storage cost represents the cost that spatiotemporal big data needs to pay to store in candidate storage engines, so as to quantify the degree of matching between the data characteristics of spatiotemporal big data and the engine characteristics of candidate storage engines.

[0030] Step 103: Determine the first target engine from the candidate storage engines based on the storage cost, and store the spatiotemporal big data into the first target engine.

[0031] In step 103, a first target engine is determined from the candidate storage engines based on the storage cost, and the first target engine is used as the storage engine for the initial writing of spatiotemporal big data, and the spatiotemporal big data is stored in the first target engine.

[0032] Step 104: Determine the storage free energy of the spatiotemporal big data on the candidate storage engine. The storage free energy represents the stability of the spatiotemporal big data storage on the candidate storage engine.

[0033] This application's embodiments introduce thermodynamic phase transition theory into data migration decisions between storage engines, establish an interpretable physical model, and compare data migration between storage engines to phase transitions of matter in the gas, liquid, and solid phases. Storage free energy is used to determine which candidate storage engine holds the stable phase for spatiotemporal big data.

[0034] In step 104, after the spatiotemporal big data is stored in the first target engine, the storage free energy of the spatiotemporal big data is continuously calculated on each candidate storage engine. The storage free energy represents the stability of the spatiotemporal big data stored in the candidate storage engine.

[0035] Step 105: Determine the second target engine from the candidate storage engines based on the storage free energy.

[0036] In step 105, the storage free energy is used to determine which candidate storage engine is the most stable for storing spatiotemporal big data, and the second target engine is selected from the candidate storage engines accordingly. The spatiotemporal big data is most stable when stored in the second target engine.

[0037] Step 106: If the first target engine and the second target engine are different, and the storage free energy of the first target engine and the storage free energy of the second target engine meet the preset migration conditions, then the spatiotemporal big data is migrated from the first target engine to the second target engine for storage.

[0038] In step 106, if the first target engine for storing the spatiotemporal big data is different from the second target engine determined based on the storage free energy, it may be necessary to migrate the spatiotemporal big data to the second target engine for storage. When the storage free energy of the first target engine and the storage free energy of the second target engine meet the preset migration conditions, the spatiotemporal big data is migrated from the first target engine to the second target engine for storage.

[0039] This application embodiment considers the storage cost of storing spatiotemporal big data in candidate storage engines and automatically selects the first target engine for initial writing based on the storage cost of spatiotemporal big data in candidate storage engines. This avoids situations where the storage cost is too high due to incompatibility between the total amount of spatiotemporal big data, data dimensions, data format, query mode, and other characteristics of the spatiotemporal big data and the candidate storage engines. Simultaneously, this application embodiment introduces thermodynamic phase transition theory into the data migration decision between storage engines. Storage free energy is used to characterize the stability of spatiotemporal big data stored in candidate storage engines, and data migration is triggered based on storage free energy, thereby establishing an interpretable physical model for data migration and improving data storage efficiency. In this application embodiment, after spatiotemporal big data is written to the first target engine according to the storage cost, it is not fixed to that first target engine. Instead, it can select a suitable second target engine again based on the dynamic changes in storage free energy. When the first target engine and the second target engine are different, and the storage free energy of the first target engine and the storage free energy of the second target engine meet preset migration conditions, the spatiotemporal big data is migrated from the first target engine to the second target engine for storage. This transforms the selection of storage engines from a static manual decision to a dynamic adaptive decision, requiring no manual intervention throughout the process.

[0040] Optionally, determining the storage cost of the spatiotemporal big data on the candidate storage engine includes: The data features of the spatiotemporal big data and the engine features of the candidate storage engines are obtained. The data features include at least data type, expected query frequency, data volume and expected query pattern. The engine features include at least baseline query latency, unit storage cost and feature query pattern. Based on the data type and the baseline query latency, determine the query latency of the candidate storage engine for the data type; The query latency cost is obtained by multiplying the expected query frequency by the query latency of the candidate storage engine for the data type. Calculate the product of the data volume and the unit storage cost to obtain the economic cost of storage; Determine the degree of matching between the expected query pattern and the feature query pattern to obtain the pattern adaptation penalty; The storage cost of the spatiotemporal big data on the candidate storage engine is determined based on the query latency cost, the storage economic cost, and the pattern adaptation penalty. The step of determining the first target engine from the candidate storage engines based on the storage cost includes: The candidate storage engine with the lowest storage cost is selected from the candidate storage engines and designated as the first target engine.

[0041] Because different types of spatiotemporal data differ greatly in their query patterns—real-time sensor readings require sub-millisecond point queries, historical trajectory analysis requires gigabyte-level sequential scanning, and remote sensing imagery requires large-block continuous readings—relevant technical solutions using a single type of storage engine to store all types of spatiotemporal big data can lead to a sharp degradation in the query performance of the storage engine under certain query patterns. However, blindly increasing the number of engines will increase system complexity and cross-engine coordination overhead.

[0042] To address this, this application constructs three candidate storage engines, corresponding to the thermodynamic gas, liquid, and solid phases respectively, in order to achieve the maximum workload diversity with the minimum number of engines.

[0043] Specifically, the candidate storage engine pool can be denoted as E = {e1, e2, e3}, where each candidate storage engine e i The engine features are described by the quadruple, as shown in Formula (1) and Table 1. The engine features of each candidate storage engine may include unit storage cost, baseline query latency, index amplification factor, and compression ratio: e i = <C i storage , L i , η i index , r i compress (1) Table 1 Candidate Storage Engines e i Engine features

[0044] In addition to the engine features shown in Formula (1) and Table 1, engine features also include feature query patterns, which refer to the query patterns that best leverage the advantages of the candidate storage engine.

[0045] In this embodiment, the specific configurations of the three candidate storage engines are shown in Table 2 below, where C_storage is the unit storage cost, L_base is the baseline query latency, η_index is the index amplification factor, and r_compress is the compression ratio. Table 2. Specific configurations of the three candidate storage engines

[0046] Because related technical solutions specify the storage engine manually or through static rules during data writing, they cannot perceive the multidimensional data characteristics of spatiotemporal big data. When the data characteristics of spatiotemporal big data mismatch with the engine characteristics of the storage engine storing the spatiotemporal big data, query latency may increase. Therefore, embodiments of this application also extract data characteristics for spatiotemporal big data.

[0047] Specifically, for each spatiotemporal big data d, its five-dimensional data features F(d) are extracted, as shown in Formula (2) and Table 3 below. The data features of each spatiotemporal big data d may include the data type code corresponding to the data type, spatial scale, temporal sampling density, query pattern code corresponding to the expected query pattern, data volume, and R. 5 Representing five dimensions: F(d) = (τ d , σ d , ρ d , ω d , ν d ) ∈ R 5 (2) Table 3 Data characteristics of spatiotemporal big data d

[0048] Among them, A d The area covered by data d is expressed in meters. 2 For example, A of a vehicle trajectory d Let A be the area of ​​the bounding rectangle of the latitude and longitude range of the trajectory. d σ represents the ground area covered by the image. d In the calculation formula, A min The area of ​​the smallest spatial unit of the system (e.g., 1m²) 2 In one example, σ is used, which includes log-normalization. d σ is calculated using the formula d Typically in the range of [0, 6], it can cover data space scales from square meters to square kilometers.

[0049] N sample The number of samples per unit time, i.e., the number of times data d is sampled within the reference time window Δ. tref The number of records written or sampled within the range. For example, if the sampling frequency of a vehicle's GPS trajectory is 1 time / second, then N... sample = 1×Δ tref The sensor time-series data is sampled once per minute, then N sample = Δ tref / 60. ρ d This represents the normalized time sampling density.

[0050] |d| represents the size of data d in bytes (|·| indicates the absolute value of the data size), in bytes. For example, a vehicle trajectory record is approximately 1KB, then ν d = log 10 (1KB / 1KB) = 0; a remote sensing image is approximately 100MB, then ν d = log 10 (10 8 / 10 3 = 5. Using ν including logarithmic normalization. d The ν calculated by the formula d Typically in the range of [0, 6], it can cover data volumes from KB to GB.

[0051] In addition to the data features shown in formula (2) and Table 3 above, the data features also include the expected query frequency of spatiotemporal big data.

[0052] This application embodiment defines spatiotemporal big data d in candidate storage engine e. i Storage cost function Φ(e) on i ,d), consists of three components: query latency cost, storage cost, and schema adaptation penalty, used to calculate the spatiotemporal big data d in candidate storage engines e. i The storage cost is shown in the following formula (3): Φ(e i , d) = α·Q(d)·L i (τ d ) + β·ν d ·C i storage + γ·M i (ω d (3) The query latency cost is the product of the expected query frequency and the query latency of the candidate storage engine for the data type, i.e., Q(d)·L. i (τ d ), Q(d) represents the expected query frequency of spatiotemporal big data, and L i (τ d ) as a candidate storage engine e i For data type τ d The query latency of spatiotemporal big data depends on the data type τ of the spatiotemporal big data. d and candidate storage engines e i Baseline query latency L i base The calculation yields the following formula (4): L i (τ d )= L ibase × μ i (τ d (4) Where, μ i (τ d ) as a candidate storage engine e i For data type τ d The type adaptation factor represents the scaling factor (or scaling factor) of the query latency of a candidate storage engine for that data type relative to the baseline query latency.

[0053] In one embodiment, the value of the type adaptation factor is obtained by benchmarking each engine. Specifically, for each data type τ d In candidate storage engines e i If the standard query load is executed, and the average latency L is measured, then μ i (τ d ) = L / L i base In one example, μ i (τ d The values ​​for ) can be set as shown in Table 4 below: Table 4 μ i (τ d Examples of possible values ​​for )

[0054] In one example, μ i (τ d )<1 indicates that the candidate storage engine outperforms the benchmark for this data type, meaning the candidate storage engine is well-suited for this data type; μ i (τ d A type fit factor greater than 1 indicates that the candidate storage engine's query performance for this data type is inferior to the benchmark, meaning the candidate storage engine is not well-suited for this data type. For example, the key-value storage engine is best suited for point lookups on vector data, with a calculated type fit factor of 0.8; the columnar storage engine is best suited for scanning raster data, also with a calculated type fit factor of 0.8.

[0055] In one embodiment, when adding a new candidate storage engine, it is only necessary to run benchmark tests to calibrate its type adaptation factor value table, and the type adaptation factor is automatically determined based on the table when calculating storage costs.

[0056] The economic cost of storage is the product of the data volume of spatiotemporal big data and the unit storage cost of candidate storage engines, i.e., ν. d ·C i storage .

[0057] Pattern fit penalty represents the degree of matching between the expected query pattern of spatiotemporal big data and the feature query pattern of candidate storage engines, i.e., M. i (ω d When the query patterns required by spatiotemporal big data do not match the feature query patterns of candidate storage engines, a performance penalty is applied in formula (3) through pattern adaptation penalty.

[0058] In one example, the pattern adaptation penalty can take the value 0 or μ. penalty μ penalty 2 , representing perfect match, partial match, and severe mismatch, respectively. Where μ penalty and μ penalty 2 These are two independent system parameters, namely μ penalty 2 Not μ penalty Instead of the square of the value, it is another pattern adaptation penalty value, which ranges from [0,1] and is set according to the business's sensitivity to latency.

[0059] Specifically, the query pattern of spatiotemporal big data ω d When the feature query pattern does not match the candidate storage engine, M i (ω d Take the corresponding value and add it to the storage cost function Φ(e). i In d), M i (ω d When ) = 0, it indicates a perfect match with no penalty, such as when performing a point lookup on a key-value store engine; M i (ω d )= μ penalty `M` indicates a partial match with a moderate penalty, such as when performing a range lookup on a row-oriented storage engine; i (ω d )= μ penalty 2 This indicates a severe mismatch and incurs a heavy penalty, such as when performing high-frequency lookups on a columnar storage engine.

[0060] In one example, μ penalty It can take a value of 0.3, indicating that a 30% increase in storage cost is required when there is a partial mismatch; μ penalty 2 A value of 0.6 is acceptable, indicating a 60% increase in storage cost for severe mismatches. μ must be satisfied. penalty 2 >μ penalty >0, ensuring that the severity of punishment is proportional to the degree of mismatch.

[0061] α, β, and γ are the weight coefficients corresponding to the query latency cost, storage economic cost, and pattern adaptation penalty, respectively, satisfying α + β + γ = 1. The specific values ​​are adjusted according to the business situation.

[0062] After calculating the storage cost, the storage cost Φ(e) is determined from the candidate storage engines. i d) Minimum candidate storage engine As the primary target engine, it is used for the initial storage of spatiotemporal big data, as shown in the following formula (5): (5) This application's embodiments employ query latency cost, storage economic cost, and pattern adaptation penalty to calculate the storage cost of spatiotemporal big data on candidate storage engines. This makes the storage cost calculation more accurately reflect the cost of storing spatiotemporal big data in candidate storage engines, improving the storage efficiency of spatiotemporal big data storage. The first target engine is automatically selected based on the storage cost, transforming the selection of storage engines from a static, manual decision to a dynamic, adaptive decision, requiring no manual intervention throughout the process. Furthermore, the weighting coefficients α, β, and γ allow for flexible adjustment of the storage engine selection between performance-first and cost-first priorities, enhancing the flexibility of storage engine selection.

[0063] Optionally, determining the storage free energy of the spatiotemporal big data on the candidate storage engine includes: Determine the state vector of the spatiotemporal big data at a preset time, wherein the state vector includes access frequency, data density, data volume, and query pattern entropy; The storage energy of the spatiotemporal big data in the candidate storage engine is obtained by multiplying the data volume by the unit storage cost of the candidate storage engine. The storage energy represents the basic storage cost of the spatiotemporal big data in the candidate storage engine. Based on the storage internal energy and the state vector of the spatiotemporal big data at a preset time, the storage free energy of the spatiotemporal big data on the candidate storage engine is determined; The step of determining the second target engine from the candidate storage engines based on the storage free energy includes: The candidate storage engine with the minimum storage free energy is determined from the candidate storage engines and selected as the second target engine.

[0064] In the relevant solutions, once the data is written to the storage engine, it is permanently stored in that storage engine, lacking an automatic migration mechanism. Furthermore, the few solutions that support data migration use heuristic rules to trigger data migration, which have problems such as the heuristic rules being uninterpretable and failing to comprehensively consider the multi-dimensional data characteristics of spatiotemporal big data.

[0065] To this end, this application introduces thermodynamic phase transition theory into the data migration decision between storage engines, establishing an interpretable physical model. Specifically, the data migration between storage engines is analogized to the phase transition of matter between gas, liquid, and solid phases. The thermodynamic potential function (i.e., Gibbs free energy function) is used to determine which candidate storage engine will be in a stable phase for spatiotemporal big data, that is, which candidate storage engine has the highest stability when storing spatiotemporal big data.

[0066] In this embodiment of the application, a four-dimensional thermodynamic state vector Θ(d, t) is defined for each spatiotemporal big data data block d at time t, that is, the state vector of spatiotemporal big data at a preset time, including access frequency, data density, data volume and query pattern entropy, as shown in the following formula (6) and Table 5: Θ(d, t) = (T d , P d V d , S d (6) Table 5. State vector of data block d at time t

[0067] Where, N q The number of times data d is queried within the time window [t-Δt, t] is the read-side indicator, reflecting the access frequency of the data; and the above formula (2) and N in Table 3 sample This is a write-side metric, reflecting the frequency of data generation; the two have different physical meanings. For example, a piece of real-time vehicle location data that is written infrequently but queried frequently, N... sample Very small but N q Very large; a raw sensor reading written frequently but rarely queried, N sample It's very big, but N q It's very small.

[0068] Query pattern entropy S d The meaning is: Suppose that the query patterns of data block d can be divided into K types (point query, range query, aggregation query, etc.), and the proportion of the k-th type of query is p. k (satisfies Σp) k =1), then S d = -Σp k ·log(p k S d =0 indicates that all queries targeting this data block follow the same query pattern, meaning the query pattern entropy is low; S d =log KThis indicates that the various query patterns for this data block are evenly distributed, meaning the query pattern entropy is high. In one embodiment, high-entropy data is suitable for flexible storage engines, such as key-value storage engines or row-based storage engines; low-entropy data is suitable for dedicated storage engines, such as column-based storage engines.

[0069] This application embodiment stores spatiotemporal big data d in candidate storage engine e. i The basic storage cost is defined as the storage internal energy of spatiotemporal big data in candidate storage engines. Through the spatiotemporal big data d data volume and candidate storage engines Unit storage cost The product is calculated as shown in formula (7) below: (7) Furthermore, drawing an analogy to the thermodynamic Gibbs free energy calculation formula G = U + PV - TS, this application embodiment defines data d in candidate storage engine e. i Gibbs free energy stored on In other words, the storage free energy of spatiotemporal big data on candidate storage engines, through storage internal energy The state vector of spatiotemporal big data at a preset time is calculated as shown in the following formula (8): (8) in, For data density, For spatiotemporal big data storage, e is a candidate storage engine. i The actual data volume occupied at that time For access frequency, This is the query pattern entropy. The meanings of each term in formula (8) are shown in Table 6 below: Table 6. Meaning of Gibbs Free Energy Term

[0070] In one embodiment, The calculation needs to consider index overhead and compression ratio, and can be calculated using the following formula (9): (9) in, For the spatiotemporal big data d, For candidate storage engines e i Index magnification factor, For candidate storage engines e i Compression ratio.

[0071] According to formula (8), when (Access frequency) high, When the query pattern entropy is large, As the term increases, the storage free energy decreases, and spatiotemporal big data tends to be stored in gaseous storage engines, such as key-value storage engines, just as high-temperature substances tend to be in a gaseous state; when Low, When low, As the term decreases, the stored free energy is mainly due to The decision is that spatiotemporal big data tends to be stored in solid-state storage engines, such as columnar storage engines, just as low-temperature materials tend to be solid-state; when When (data density) is high, As the number of items increases, spatiotemporal big data tends to occupy small storage engines, and the high compression ratio of columnar storage engines makes... Because of their smaller size, high-density data tends to be stored in solid-state storage engines, such as columnar storage engines.

[0072] In this embodiment, the phase balance condition is defined as follows: spatiotemporal big data d in candidate storage engine e i The storage engine is in a stable phase if and only if its storage free energy is the smallest among all candidate engines. That is, the candidate storage engine with the smallest storage free energy is determined from the candidate storage engines and used as the second target engine, as shown in the following formula (10): (10) in, , For candidate storage engines e i e j Storage free energy. When spatiotemporal big data d is stored in candidate storage engines e i If a candidate storage engine e exists j ≠e i， Make < Then the spatiotemporal big data d is in a metastable state, and there is a possibility of it starting from e. i Migrate to e j Thermodynamic driving force.

[0073] This application's embodiments introduce thermodynamic phase transition theory into data migration decisions between storage engines. Based on thermodynamic principles, the state vector of spatiotemporal big data at a preset time is determined, and storage free energy is used to characterize the stability of spatiotemporal big data stored in candidate storage engines. This makes the data migration decision based on the Gibbs free energy minimization principle in thermodynamic phase transition theory, that is, selecting the candidate storage engine with the minimum storage free energy as the second target engine. Each data migration decision can be traced back to its physical cause, thus possessing physical interpretability. When the data access mode changes, the state vector is automatically updated, and the data migration corresponding to the phase transition can occur naturally without manual intervention throughout the process.

[0074] Optionally, after determining the second target engine from the candidate storage engines based on the storage free energy, the method further includes: If the first target engine is different from the second target engine, then obtain the engine distance between the first target engine and the second target engine; Based on the engine distance and the data volume of the spatiotemporal big data, the migration cost between the first target engine and the second target engine is determined; Obtain the free energy difference threshold; If the difference between the storage free energy of the first target engine and the storage free energy of the second target engine is greater than the migration cost, and the difference between the storage free energy of the first target engine and the storage free energy of the second target engine is greater than the free energy difference threshold, then the spatiotemporal big data will be migrated from the first target engine to the second target engine for storage.

[0075] This application embodiment defines phase boundaries between candidate storage engines corresponding to different phase states based on thermodynamic phase transition theory, for use in data migration decisions. Specifically, candidate storage engine e i and e j The phase boundary between them is the state space (access frequency T, data density P) that satisfies the candidate storage engine e i and e j The storage free energy is equal to G. ei (d)=G ej The point set of (d). Expand on the phase boundary and eliminate identical points. After that, the PT relationship on the phase boundary is obtained by simplification. Taking the derivative with respect to T, the stored Clausius-Clapeyron equation is obtained, as shown in the following formula (11): (dP / dT)|boundary = ΔU / (ΔV · T) (11) Where (dP / dT)|boundary is the slope of the phase boundary, and the storage internal energy difference ΔU = U ej - U ei U ej U ei Spatiotemporal big data in candidate storage engines e i and e j Storage internal energy; data volume difference ΔV = V ej (d) - V ei (d), V ej (d) 、 V ei (d) represents the spatiotemporal big data d stored in candidate storage engines e. i and e jThe actual data volume occupied at that time. The Clausius-Clapeyron equation describes the slope of the phase boundary in the (T,P) plane, and relates the geometry of the phase boundary to the engine parameters (ΔU, ΔV) of the candidate memory engine, so that the phase diagram can be analytically calculated through the engine parameters of the candidate memory engine without empirical fitting.

[0076] In one embodiment, the phase boundary can serve the following functions in the migration decision of this application embodiment: First, the phase boundary constitutes the geometric criterion for migration decision, used for migration trigger determination, when the thermodynamic state (T) of the large spatiotemporal data d is... d ,P d When the current phase region crosses a phase boundary into another phase region, it indicates that the storage free energy of the candidate storage engine for the target migration is lower than that of the current storage engine, thus generating a migration driving force. Secondly, the three phase boundaries will (T d , P d The state space is divided into three phase regions: gas, liquid, and solid. Each phase region corresponds to a candidate storage engine. The three-phase diagram, which includes the phase boundaries, can intuitively show which spatiotemporal big data in which state should be placed in which candidate storage engine, providing a visual basis for migration decisions. Thirdly, according to the Clausius-Clapeyron equation in formula (11), the slope (dP / dT)|boundar of the phase boundary can be calculated analytically by the engine parameters of the candidate storage engine without relying on empirical fitting. Therefore, when adding a candidate storage engine or adjusting the engine parameters, the three-phase diagram can be automatically updated, realizing analytical computability and ensuring the scalability and adjustability of the candidate storage engine.

[0077] Among them, reference Figure 2 This is a three-phase phase diagram of an embodiment of a spatiotemporal big data storage method according to this application. The three-phase phase diagram is based on T... d (Access frequency, i.e., access temperature) is the vertical axis, P d The phase diagram, plotted with data density on the horizontal axis, includes three phase boundaries. These three boundaries divide the phase diagram into three phase regions: gas, liquid, and solid, as shown in Table 7 below. Table 7. Three phase regions corresponding to the three-phase phase diagram.

[0078] The intersection of the three phase boundaries is the three-phase point, as shown in the following formula (12): G e1 (d) = G e2 (d) = G e3 (d)(12) That is, at the triple point (T tp , P tpAt point (), the storage free energy of spatiotemporal big data is completely equal in the three engines, reaching a critical equilibrium state. The location of the triple point is determined by the engine parameters of the candidate storage engines and is an inherent property of the phase diagram.

[0079] Specifically, the triple point (T tp , P tp As a critical state marker, the triple point indicates that the storage free energy of the three engines is completely equal, and spatiotemporal big data is in a critical equilibrium state. No candidate storage engine possesses a thermodynamic advantage. This means that near the triple point, spatiotemporal big data is extremely sensitive to state changes; even small fluctuations in access temperature or data density can cause data to favor different candidate storage engines. Therefore, the area near the triple point is a high-risk region for migration oscillations, where data may repeatedly migrate between the three candidate storage engines. Secondly, the triple point is the intersection of three phase boundaries and the geometric anchor point constituting the three-phase phase diagram. Its position is uniquely determined by the engine parameters of the candidate storage engines and is an inherent property of the three-phase phase diagram.

[0080] In related technical solutions, data migration between storage engines lacks an anti-oscillation mechanism. The migration trigger condition is based solely on a single threshold, and the migration operation is executed as soon as the threshold is met. This causes data to migrate back and forth frequently near the threshold boundary, leading to system performance oscillations. To solve this problem, the embodiments of this application are based on thermodynamic phase transition theory, using phase boundary equations to describe the data migration conditions, and using supercooling / superheating effects to prevent data from repeatedly migrating and oscillating back and forth.

[0081] Specifically, if the first target engine and the second target engine are different, it indicates that the spatiotemporal big data may need to be migrated from the currently stored first target engine to the second target engine. The engine distance D(e) between the first and second target engines is then calculated. i , e j ) is used to evaluate two candidate storage engines. i e j The degree of difference in physical properties is shown in the following formula (13): D(e i , e j ) = |C i -C j | / max(C i C j ) + |L i -L j | / max(L i ,L j (13) Among them, C i C j For the unit storage cost of the two candidate storage engines, L i L jThe baseline query latency for the two candidate storage engines.

[0082] In one example, the engine is located at distance D(e). i , e j The range of D is [0,2]. The larger the value of D, the greater the difference between the two candidate storage engines, and the higher the migration cost across engines. For example, the engine distance between a key-value storage engine and a columnar storage engine is much greater than the engine distance between a key-value storage engine and a row-based storage engine.

[0083] In this application embodiment, migration latent heat is defined as the migration cost between the first target engine and the second target engine. Based on the engine distance and the data volume of the spatiotemporal big data, the first target engine e is determined. i With the second target engine e j The migration cost L(e) between i →e j As shown in the following formula (14): L(e i →e j ) = κ · V d · (1 + D(e i ,e j ) / D max (14) Where κ is the unit data migration cost coefficient, V d For the data volume of spatiotemporal big data, D(e i ,e j D represents the distance between engines. max This represents the maximum distance between engines.

[0084] In this embodiment, the migration cost is analogous to the latent heat of migration that a material phase transition requires to absorb or release latent heat. Data migration requires the consumption of system resources. Only when the storage free energy gain exceeds the migration cost can performing data migration yield a net benefit.

[0085] Among them, due to the distance between engines D(e) i ,e j The larger the value, the greater the difference in storage economy and query latency between the first and second target engines, and the higher the cost of cross-engine migration. This is consistent with the physical law in thermodynamics that gas-solid phase transitions (such as sublimation) require more latent heat than liquid-solid phase transitions (such as solidification).

[0086] To address the issue that frequent migrations of data states near phase boundaries can lead to system thrashing and severely impact performance, this application introduces a thermodynamic supercooling / superheating mechanism, referring to... Figure 3 This is a migration lag diagram of an embodiment of a spatiotemporal big data storage method of this application.

[0087] The hysteresis window is defined by the hysteresis barrier δ, which is the free energy difference threshold in this embodiment. If the free energy difference threshold δ is set to > 0, then when G... ej (d) <G ei (d) - Trigger source engine e when δ (ΔG > δ) i To target engine e j Forward migration of data, when G ei (d) <G ej (d) - Trigger the target engine e when δ (ΔG > δ) j Xiangyuan engine e i Reverse reflux data. In other words, there is a 2δ loop width gap between the forward migration threshold and the reverse reflux threshold, forming a hysteresis loop. Data migration is not triggered within the hysteresis window, avoiding repeated oscillations near the phase boundary. This can be compared to the supercooling / superheating effect in thermodynamics. Supercooled water does not necessarily freeze immediately below 0°C; it needs to be below the freezing point to crystallize.

[0088] In one example, the hysteresis window is represented by the access temperature T (i.e., access frequency). When T = 6.0, ΔG > δ, triggering the source engine e. i To target engine e j Forward data migration; when T=4.0, ΔG>δ, triggering the target engine e. j Xiangyuan engine e i Reverse flow data. The loop width is 2δ = 2.0.

[0089] If the difference between the storage free energy of the first target engine and the storage free energy of the second target engine is greater than the migration cost, and the difference between the storage free energy of the first target engine and the storage free energy of the second target engine is greater than the free energy difference threshold, then the spatiotemporal big data will be migrated from the first target engine to the second target engine for storage.

[0090] In this embodiment of the application, the spatiotemporal big data d is obtained from the currently stored first target engine e. i Migration to the second target engine e j Two conditions must be met simultaneously: Condition 1 is a thermodynamic threshold, i.e., the hysteresis barrier condition, which requires G... ei (d) - G ej (d)>δ; Condition 2 is the economic threshold, i.e., the latent heat compensation condition, requiring G ei (d) - G ej (d)>L(e i →e j The larger of the two values ​​is taken as the final migration threshold when making the final decision; that is, data migration occurs if and only if G... ei (d) - G ej (d)>max(δ, L(ei →e j This occurs when )).

[0091] The embodiments of this application employ a dual-threshold triggering data migration mechanism to ensure that migration triggering requires both crossing the thermodynamic hysteresis energy barrier and compensating for the latent heat cost of migration, effectively preventing frequent invalid data migration caused by small fluctuations in storage free energy.

[0092] Reference Figure 4 This is a phase transition flow diagram of an embodiment of a spatiotemporal big data storage method of this application.

[0093] Step 401: The spatiotemporal big data d is currently stored in the first target engine e. i In the process, periodic checks or detections of access pattern changes are triggered to determine whether to initiate subsequent migration decision-making processes.

[0094] Step 402: Calculate the thermodynamic state vector Θ(d, t) = (T) of data d at the current moment. d , P d V d ,S d ).

[0095] Step 403: For each non-first target engine e i Candidate storage engines e j The storage Gibbs free energy of spatiotemporal big data d in this candidate storage engine is calculated. .

[0096] Step 404: Find the second objective engine that minimizes the storage free energy among all candidate storage engines, e* = argmin G. ej (d) and calculate the free energy gain ΔG = G between the first target engine and the second target engine. ei (d) -G e* (d)

[0097] Step 405: Determine whether ΔG is less than or equal to the lag barrier δ. If ΔG does not exceed δ, it is considered to be within the lag window and no migration is performed.

[0098] Step 406: If ΔG exceeds δ, then calculate the latent heat of migration L(e) i →e j ) = κ · V d · (1 + D(e i ,e j ) / D max If ΔG is not greater than the latent heat of migration, then migration will not occur.

[0099] Step 407: If ΔG is greater than the migration latent heat, then perform the migration, update the corresponding spatial sharding information MBR in the federated index, and record the migration log.

[0100] Optionally, after storing the spatiotemporal big data into the first target engine, the method further includes: Determine the spatiotemporal popularity of the aforementioned spatiotemporal big data; The heat level corresponding to the spatiotemporal big data is determined based on the spatiotemporal heat, and the heat level has corresponding storage rules; Adjust the storage engine to which the spatiotemporal big data belongs according to the storage rules.

[0101] While related technologies often employ hot / cold data stratification based solely on access time or frequency, the application scenarios of spatiotemporal big data frequently suffer from spatial shifts in the area of ​​interest. For instance, in urban traffic management, the area of ​​interest during the morning rush hour is the direction from residential areas to the CBD, while during the evening rush hour it shifts to the direction from the CBD to residential areas. In the event of an emergency, the area of ​​interest can also switch instantaneously. Hot / cold data stratification based solely on time cannot capture these spatial changes, resulting in cold spatial data occupying hot storage resources.

[0102] Therefore, in addition to performing migration of spatiotemporal big data according to thermodynamic principles, the embodiments of this application also perform layering and migration of spatiotemporal big data according to the spatiotemporal heat of spatiotemporal big data.

[0103] Specifically, the spatiotemporal heat of spatiotemporal big data is determined, and the corresponding heat level of spatiotemporal big data is determined based on the spatiotemporal heat. The heat level has corresponding storage rules, and the storage engine to which the spatiotemporal big data belongs is adjusted according to the storage rules corresponding to the heat level.

[0104] This application's embodiment adjusts the storage engine to which spatiotemporal big data belongs based on spatiotemporal heat. It not only considers the heat of spatiotemporal big data in the time dimension, but also in the spatial dimension. Compared with the single time dimension layering in related solutions, it can prevent spatially cold data from occupying hot storage resources, thereby improving the utilization rate of hot storage space.

[0105] Optionally, determining the spatiotemporal heat of the spatiotemporal big data includes: Obtain the number of times the spatiotemporal big data is queried within a preset time window, the set of hotspot spatial regions, and the spatial range of the spatiotemporal big data; The temporal popularity of the spatiotemporal big data is determined based on the number of times it is queried. The spatial heat of the spatiotemporal big data is determined based on the set of hotspot spatial regions and the spatial range of the spatiotemporal big data. The spatiotemporal heat of the spatiotemporal big data is obtained by weighted fusion of the temporal heat and the spatial heat.

[0106] In this application embodiment, the spatiotemporal heat of spatiotemporal big data d at time t is defined as a weighted combination of temporal heat and spatial heat.

[0107] Specifically, obtain the number N of spatiotemporal big data queries within a preset time window Δt. q Based on the number of times spatiotemporal big data is queried, the temporal popularity H of the spatiotemporal big data is determined. t (d, t), as shown in the following formula (15): H t (d, t) = N q (d, [t-Δt, t]) / N max (15) Where, N max This is the normalization constant.

[0108] Obtain the set of hotspot spatial regions of spatiotemporal big data d at time t. Based on the spatial range of the spatiotemporal big data d, the spatial heat of the spatiotemporal big data is determined according to the hotspot spatial region set and the spatial range of the spatiotemporal big data. As shown in the following formula (16): (16) in, Let t be the set of spatially relevant regions at time t, i.e., the set of hotspot spatial regions, which can be obtained by business rules or automatic detection; I(·) is an indicator function used to determine whether the spatial range of spatiotemporal big data d falls into the set of spatially relevant regions. For spatial weighting, the closer the spatiotemporal big data is to the center of the spatial focus area, the higher the spatial weight.

[0109] By weighting and fusing temporal and spatial heat, the spatiotemporal heat H(d, t) of spatiotemporal big data is obtained, as shown in the following formula (17): H(d, t) = λ t ·H t (d, t) + λ s ·H s (d, t)(17) Where, λ t and λ s For weight fusion, λ needs to be satisfied. t + λ s = 1, through λ t and λ s Weighting adjusts the relative importance of the time and space dimensions.

[0110] This application embodiment not only considers the popularity of spatiotemporal big data in the time dimension, but also in the spatial dimension. When the hot spot spatial region set drifts, for example, when a sudden event occurs, the spatial popularity of the relevant spatiotemporal big data automatically increases, thereby increasing the spatiotemporal popularity and triggering the adjustment of the storage engine to which the spatiotemporal big data belongs, thus improving the utilization rate of hot storage space.

[0111] Optionally, the heat level includes a hot layer, a warm layer, and a cold layer; the candidate storage engines include a key-value storage engine, a row-based storage engine, and a column-based storage engine; and adjusting the storage engine to which the spatiotemporal big data belongs according to the storage rules includes: If the spatiotemporal big data belongs to the hot layer, then the spatiotemporal big data will be moved from the first target engine to the key-value storage engine and the row storage engine for storage respectively; If the spatiotemporal big data belongs to the thermosphere, then the spatiotemporal big data data is stored in the first target engine; If the spatiotemporal big data belongs to the cold layer, then the spatiotemporal big data will be moved from the first target engine to the columnar storage engine for storage.

[0112] This application embodiment divides the heat level into a hot layer, a warm layer, and a cold layer. Specifically, two thresholds H are set. hot and H cold H hot Greater than H cold As shown in Table 8 below.

[0113] Table 8. Classification of Heat Levels

[0114] When the spatiotemporal heat H of spatiotemporal big data is greater than H hot The time-division is assigned to a hot layer, and the spatiotemporal big data is redundantly stored in both the key-value storage engine and the row-based storage engine, instead of the primary target engine. If the primary target engine is either a key-value storage engine or a row-based storage engine, then the spatiotemporal big data only needs to be redundantly stored in the other engine between the row-based and key-value storage engines.

[0115] The storage rules of the hot layer are actually a thermal expansion mechanism. Since the hot layer data is redundantly stored in multiple storage engines, the storage cost router can select the fastest storage engine to respond when querying. Although this increases storage overhead, it eliminates the latency of cross-engine queries, which is analogous to molecules occupying more space in thermal expansion to obtain higher degrees of freedom of movement.

[0116] When the spatiotemporal heat H of spatiotemporal big data is greater than H cold And less than or equal to H hot The time division is set to the temperature layer, while the spatiotemporal big data is stored in the first target engine.

[0117] When the spatiotemporal heat H of spatiotemporal big data is less than or equal to H cold The spatiotemporal big data was moved from the primary target engine to the columnar storage engine for compressed archiving and storage.

[0118] This application embodiment eliminates the latency of cross-engine querying of hot layer data through the mechanism of thermal expansion, thereby improving the query efficiency of hot layer data and archiving cold layer data to avoid cold layer data consuming too much system resources.

[0119] Optionally, the candidate storage engine has a corresponding local spatial index subtree, the local spatial index subtree including multiple spatial sharding information, the multiple spatial sharding information being used to indicate the spatial range of the spatiotemporal big data stored by the candidate storage engine, and the method further includes: Obtain the time partition directory, which is used to indicate the storage engine to which spatiotemporal big data of different time ranges belong; Receive a data query request, the data query request including a query space range and a query time range; Filter the target space shard information that intersects with the query space range from the space shard information of each candidate storage engine, and determine the candidate storage engine corresponding to the target space shard information as the third target engine; Determine the target time range that intersects with the query time range from the time partition directory, and determine the storage engine to which the spatiotemporal big data of the target time range belongs as the fourth target engine; Determine the intersection of the third target engine and the fourth target engine to obtain the storage engine to be queried; The query is performed on the storage engine to be queried according to the data query request, and the query result is obtained.

[0120] In related technologies, each candidate storage engine typically maintains its own local index. For example, key-value storage engines use hash indexes, row-oriented storage engines use B+ tree indexes, and column-oriented storage engines use sparse indexes. The index structures are incompatible with each other, and when querying across engine spaces, a unified index cannot be used for pruning. Instead, the query is broadcast to all N engines and then merged. The query latency degrades linearly with the number of engines N.

[0121] Therefore, in this embodiment of the application, each candidate storage engine is configured to have a corresponding local spatial index subtree. The local spatial index subtree includes multiple spatial sharding information, which is used to indicate the spatial range of the spatiotemporal big data stored by the candidate storage engine.

[0122] In one embodiment, all candidate storage engines jointly maintain a logically unified federated R-tree T. fedThe global root node of this federated R-tree maintains the spatial range of global spatiotemporal big data, while the leaf nodes point to the local spatial index shards of each candidate storage engine, as shown in the following formula (18): (18) in, Candidate storage engine e i The local spatial index subtree, whose root node's MBR is the candidate storage engine e i The MBR (Minimum Bounding Rectangle) of spatial partitioning refers to the spatial partitioning information. Specifically, It is a candidate storage engine e i An internal, complete native R tree serves as its spatial index structure, and the root node of this native R tree has an MBR (i.e., The space-fragmented MBR used for federal layer routing is the one that is used.

[0123] Formula (18) means a federated R-tree. It consists of a global root node (Root) and local spatial index subtrees for each candidate storage engine. The Root maintains a global spatial range (the union of the MBRs of all candidate storage engine spatial shards). The Root is the entry point for spatial query routing. Queries start from the Root and are routed downwards to the target engine based on the intersection relationship between the query spatial range and the MBRs of each child node.

[0124] The above federated R-tree has the following properties: the MBR of the root node is the union of the spatial sharding MBRs of all candidate storage engines, and the spatial sharding MBRs of each candidate storage engine do not overlap, so that spatial queries can be precisely routed to a single engine, as shown in the following formula (19), candidate storage engine e i and candidate storage engines e j Spatial MBRs do not overlap. For candidate storage engines e i Spatial fragmented MBR, For candidate storage engines e j Spatial fragmented MBR: (19) When data migration causes changes in the spatial sharding MBR, only the root node level of the federated R tree is updated, without affecting the local index structure of each candidate storage engine.

[0125] Due to the Federal R-tree T fedLogically, it unifies the spatial index structure of all candidate storage engines. Its root node maintains the global MBR, which is the union of the spatial shard MBRs of all candidate storage engine shards. The leaf nodes point to the local spatial shard MBR of each candidate storage engine, ensuring that the spatial shard MBRs of each candidate storage engine shard do not overlap, thus supporting precise routing for spatial queries. This federated R tree serves the spatial dimension pruning for pure spatial queries and spatiotemporal joint queries.

[0126] In one example, when performing a pure spatial query, given a query space range q® (R is the query rectangle), the routing process starts from the root node of the federated R tree and searches downwards. If the query rectangle R only matches a candidate storage engine e i If the spatial sharded MBRs intersect, then the query will be precisely routed to the candidate storage engine. i Only for this candidate storage engine e i Subqueries are distributed, and other candidate storage engines do not participate. If the query rectangle R spans the spatial sharding MBR of multiple candidate storage engines, the query is decomposed into several subqueries, which are only distributed to the relevant 2 to 3 candidate storage engines for parallel execution. Compared with related technologies that need to broadcast the query to all N engines, the federated R-tree in this application embodiment only needs to distribute the query to the candidate storage engine that intersects with R (usually 1), reducing the query latency from O(N) to O(1) (when a single engine hits) or O(k) (where k is the number of intersecting engines, and k is much smaller than N), thus improving the efficiency of spatial queries.

[0127] In one example, refer to Figure 5 This is a schematic diagram of a federated R-tree index, which is an embodiment of a spatiotemporal big data storage method of this application.

[0128] like Figure 5 As shown, the root node of the federated R tree maintains the spatial range MBR(Root) of the global spatiotemporal big data, that is, MBR(Root) = MBR(e1) ∪ MBR(e2) ∪ MBR(e3), where MBR(e1), MBR(e2), and MBR(e3) are the MBRs of the spatial partitions of the three candidate storage engines, respectively.

[0129] Below the root node of the federated R tree are the local spatial index subtrees for each candidate storage engine, and each local spatial index subtree includes the MBR of the spatial shards for each candidate storage engine.

[0130] Specifically, the e1 spatial index shard belongs to the KV engine's local R tree and can manage hotspot spatial key data. Its MBR(e1) range is within the local high-frequency attention spatial region. The e2 spatial index shard belongs to the row storage engine's local R tree and can manage vector data. Its MBR(e2) range is within the structured layer data of the region. The e3 spatial index shard belongs to the column storage engine's local R tree and can manage trajectory and raster data. Its MBR(e3) range is the global historical archive data. The spatial shard MBRs of each candidate storage engine do not overlap, allowing spatial queries to be precisely routed to a single engine, as shown in the above formula (19).

[0131] Meanwhile, in this embodiment, the federated layer maintains a time-partition directory, which indicates the storage engine to which spatiotemporal big data of different time ranges belongs. That is, the federated time index I. time The union of local time indexes for each candidate storage engine is partitioned by time interval. Each time interval records the storage engine to which its corresponding spatiotemporal big data belongs in the time partition directory. This time partition directory serves the time dimension pruning for pure time queries and spatiotemporal joint queries.

[0132] In one example, when performing a pure time query, the time partition directory is first consulted to determine which candidate storage engines the corresponding data is distributed across based on the query time range. Then, a precise time lookup is performed within the corresponding target engine. For instance, when querying trajectory data from "January 1, 2024 to January 31, 2024," the time partition directory shows that within this range, data before January 15th is located in row-oriented storage engine e2, and archived data after January 15th is located in column-oriented storage engine e3. Therefore, the query is only sent to engines e2 and e3 for execution, avoiding broadcasting to all engines and thus reducing query overhead.

[0133] Specifically, in this embodiment, when a data query request q arrives at the federated index layer, the data query request q is received. The data query request q includes a query space range and a query time range. The data query request q is decomposed into a set of subqueries {q1,q2,...,qk} according to the query space range and query time range. Each subquery involves only a single candidate storage engine, a subquery space range, and a subquery time range. The subqueries are executed in parallel.

[0134] If a subquery involves both spatial and temporal queries (i.e., a spatiotemporal joint query), then when performing a spatial query, target spatial shard information that intersects with the query spatial range is selected from the spatial shard information of each candidate storage engine, and the candidate storage engine corresponding to the target spatial shard information is determined as the third target engine. When performing a temporal query, the target time range that intersects with the query time range is determined from the time partition directory, and the storage engine to which the spatiotemporal big data of the target time range belongs is determined as the fourth target engine. The intersection of the third and fourth target engines is determined to obtain the storage engine to be queried. The query is then performed on the storage engine to be queried according to the data query request to obtain the query results, which are sorted and merged by spatial and temporal order.

[0135] In one example, refer to Figure 5 The execution process of the cost routing query in the code is as follows: Step 501: Data query request q carries spatial range R q and time range T q The cost route lookup maker uses the global root node Root as the entry point for spatial route lookup.

[0136] Step 502: Decompose the data query request q into a set of subqueries {q1, q2, ..., qk} based on the query space range and query time range. Each subquery involves only a single candidate storage engine, a subquery space range, and a subquery time range. The cost routing queryer queries the federated R-tree T. fed Search downwards from the root node to find the node that is related to R. q The third target engine set {e} corresponding to the intersecting engine fragment MBR i Since the spatial partitions do not overlap, typically only one third-target engine will hit. Simultaneously, the federated time index I is queried. time Find the partition directory and locate the one related to T. q The fourth target engine set corresponding to the intersecting time intervals {e j The intersection of the third and fourth target engine sets is used to obtain the target engine set E. target = {e i} ∩ {e j}

[0137] Step 503: The cost routing queryer uses E... target Baseline query latency L for each target engine i The query space range and the degree of intersection with the engine's sharded MBR are considered, along with the query time range. The relative cost of each subquery is estimated, and the subquery with the lowest cost is prioritized. For example, for E... target Each engine in the middle i Estimate the cost of the subquery, Cost(q). i, e i ) = L i (τ d ) · |R q ∩ MBR_e i | · |T q | Select the engine with the lowest cost and execute the corresponding subquery first. Where L i (τ d ) represents the benchmark query latency normalized by the engine, |R q ∩ MBR_e i |To query the degree of intersection between the spatial range and the engine fragment MBR,|T q | represents the time range for the query.

[0138] Step 504: Send the k subqueries {q1,…,qk} to each target engine in parallel for execution, and sort and merge the results by space and time before returning them.

[0139] The federated index layer in this embodiment decouples the index layer from the storage engine. When adding a new engine, only its shard MBR needs to be registered, and the federated index can be automatically expanded without adjusting the existing structure, thus exhibiting good scalability. Furthermore, the shard MBRs of each engine do not overlap, ensuring that spatial queries can be accurately routed to a single engine, avoiding broadcasting queries to all engines, thereby significantly reducing the latency of cross-engine join queries. When data migration occurs, only the root node of the federated R tree needs to be updated; the local index structure of each engine remains unaffected, resulting in low maintenance overhead and ensuring the high efficiency of the migration operation.

[0140] To enable those skilled in the art to more clearly understand an embodiment of a spatiotemporal big data storage method shown in this application, the overall system architecture of an embodiment of a spatiotemporal big data storage method shown in this application is described below.

[0141] Reference Figure 6 This is a system architecture diagram of an embodiment of a spatiotemporal big data storage method according to this application.

[0142] The system includes a query interface layer, which provides a unified spatiotemporal query interface, including a unified SQL (Structured Query Language) and a spatial API (Application Programming Interface) for receiving spatiotemporal big data query requests.

[0143] The system also includes a cross-modal federated index layer, consisting of a federated spatial index (federated R-tree), a federated temporal index, and a cost-route queryer, used to perform data queries based on data query requests.

[0144] The system further includes an adaptive storage engine selector and a phase-change migration controller. The adaptive storage engine selector, based on the characteristic profiles of the data and the engine (i.e., data characteristics and engine characteristics), employs a cost model Φ(e...). i d) Calculate the storage cost of spatiotemporal big data on each candidate storage engine, and determine the first target engine for initial writing based on the storage cost. The phase transition controller performs phase transition decisions based on the comparison results of the storage Gibbs free energy G.

[0145] The system also includes a multimodal storage engine pool containing three heterogeneous candidate storage engines, each corresponding to one of the three thermodynamic phases. e1 is a KV engine, corresponding to the gas phase, using a hybrid storage medium of memory and SSD, and performing data queries through an O(1) lookup query mode. It is used to store real-time hot data, high-frequency key-value data, etc., and its MBR(e1) range is within the local high-frequency attention space area. e2 is a row storage engine, corresponding to the liquid phase, using B+ trees for indexing, and performing data queries through a low-latency point lookup query mode. It is used to store the current state vector quantity, active layer data, etc., and its MBR(e2) range is within the structured layer data of the region. e3 is a column storage engine, corresponding to the solid phase, using columnar compression to store data, and performing data queries through a high-throughput scanning query mode. It is used to store historical trajectory data, raster archive data, etc., and its MBR(e3) range is the global historical archive data.

[0146] This application embodiment considers the storage cost of storing spatiotemporal big data in candidate storage engines and automatically selects the first target engine for initial writing based on the storage cost of spatiotemporal big data in candidate storage engines. This avoids situations where the storage cost is too high due to incompatibility between the total amount of spatiotemporal big data, data dimensions, data format, query mode, and other characteristics of the spatiotemporal big data and the candidate storage engines. Simultaneously, this application embodiment introduces thermodynamic phase transition theory into the data migration decision between storage engines. Storage free energy is used to characterize the stability of spatiotemporal big data stored in candidate storage engines, and data migration is triggered based on storage free energy, thereby establishing an interpretable physical model for data migration and improving data storage efficiency. In this application embodiment, after spatiotemporal big data is written to the first target engine according to the storage cost, it is not fixed to that first target engine. Instead, it can select a suitable second target engine again based on the dynamic changes in storage free energy. When the first target engine and the second target engine are different, and the storage free energy of the first target engine and the storage free energy of the second target engine meet preset migration conditions, the spatiotemporal big data is migrated from the first target engine to the second target engine for storage. This transforms the selection of storage engines from a static manual decision to a dynamic adaptive decision, requiring no manual intervention throughout the process.

[0147] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0148] Based on the above embodiments, this embodiment also provides a spatiotemporal big data storage device, which can be applied to electronic devices such as terminal devices and servers.

[0149] Reference Figure 7 The diagram illustrates a structural block diagram of an embodiment of a spatiotemporal big data storage device according to this application, which may specifically include the following modules: Data acquisition module 701 is used to acquire candidate storage engines and the spatiotemporal big data; The cost determination module 702 is used to determine the storage cost of the spatiotemporal big data on the candidate storage engine, wherein the storage cost represents the cost required to store the spatiotemporal big data in the candidate storage engine; The first storage module 703 is used to determine a first target engine from the candidate storage engines according to the storage cost, and store the spatiotemporal big data into the first target engine. Free energy determination module 704 is used to determine the storage free energy of the spatiotemporal big data on the candidate storage engine, wherein the storage free energy represents the stability of the spatiotemporal big data storage in the candidate storage engine; Migration determination module 705 is used to determine a second target engine from the candidate storage engines based on the storage free energy; The second storage module 706 is used to migrate the spatiotemporal big data from the first target engine to the second target engine for storage if the first target engine is different from the second target engine and the storage free energy of the first target engine and the storage free energy of the second target engine meet a preset migration condition.

[0150] Optionally, the cost determination module 702 includes: The feature acquisition submodule is used to acquire the data features of the spatiotemporal big data and the engine features of the candidate storage engines. The data features include at least data type, expected query frequency, data volume and expected query pattern. The engine features include at least baseline query latency, unit storage cost and feature query pattern. The first latency calculation submodule is used to determine the query latency of the candidate storage engine for the data type based on the data type and the baseline query latency. The second latency calculation submodule is used to calculate the product of the expected query frequency and the query latency of the candidate storage engine for the data type to obtain the query latency cost; The storage economic cost calculation submodule is used to calculate the product of the data volume and the unit storage cost to obtain the storage economic cost; The pattern adaptation penalty determination submodule is used to determine the degree of matching between the expected query pattern and the feature query pattern, and to obtain the pattern adaptation penalty; The storage cost determination submodule is used to determine the storage cost of the spatiotemporal big data on the candidate storage engine based on the query latency cost, the storage economic cost, and the pattern adaptation penalty. The first storage module 703 includes: The first engine determination submodule is used to determine the candidate storage engine with the lowest storage cost from the candidate storage engines, and use it as the first target engine.

[0151] Optionally, the free energy determination module 704 includes: The state vector acquisition submodule is used to determine the state vector of the spatiotemporal big data at a preset time. The state vector includes access frequency, data density, data volume and query pattern entropy. The storage internal energy determination submodule is used to calculate the product of the data volume and the unit storage cost of the candidate storage engine to obtain the storage internal energy of the spatiotemporal big data in the candidate storage engine. The storage internal energy represents the basic storage cost of the spatiotemporal big data in the candidate storage engine. The storage free energy determination submodule is used to determine the storage free energy of the spatiotemporal big data on the candidate storage engine based on the storage internal energy and the state vector of the spatiotemporal big data at a preset time. The migration judgment module 705 is specifically used for: The candidate storage engine with the minimum storage free energy is determined from the candidate storage engines and selected as the second target engine.

[0152] Optionally, the device further includes: An engine distance determination module is used to obtain the engine distance between the first target engine and the second target engine if the first target engine is different from the second target engine; The migration cost determination module is used to determine the migration cost between the first target engine and the second target engine based on the engine distance and the data volume of the spatiotemporal big data. The free energy difference threshold acquisition submodule is used to acquire the free energy difference threshold. The migration condition judgment module is used to migrate the spatiotemporal big data from the first target engine to the second target engine for storage if the difference between the storage free energy of the first target engine and the storage free energy of the second target engine is greater than the migration cost, and the difference between the storage free energy of the first target engine and the storage free energy of the second target engine is greater than the free energy difference threshold.

[0153] Optionally, the device further includes: The spatiotemporal heat determination module is used to determine the spatiotemporal heat of the spatiotemporal big data; A heat level determination module is used to determine the heat level corresponding to the spatiotemporal big data based on the spatiotemporal heat, and the heat level has corresponding storage rules; The storage engine adjustment module is used to adjust the storage engine to which the spatiotemporal big data belongs according to the storage rules.

[0154] Optionally, the spatiotemporal heat determination module includes: The heat data acquisition submodule is used to acquire the number of times the spatiotemporal big data is queried, the set of hot spot spatial regions, and the spatial range of the spatiotemporal big data within a preset time window; The time popularity determination submodule is used to determine the time popularity of the spatiotemporal big data based on the number of times the spatiotemporal big data is queried. The spatial hotspot determination submodule is used to determine the spatial heat of the spatiotemporal big data based on the set of hotspot spatial regions and the spatial range of the spatiotemporal big data. The weighted fusion submodule is used to weightedly fuse the temporal heat and the spatial heat to obtain the spatiotemporal heat of the spatiotemporal big data.

[0155] Optionally, the heat level includes a hot layer, a warm layer, and a cold layer; the candidate storage engines include a key-value storage engine, a row-based storage engine, and a column-based storage engine; and the storage engine adjustment module includes: The hot layer adjustment submodule is used to adjust the spatiotemporal big data from the first target engine to the key-value storage engine and the row storage engine respectively if the spatiotemporal big data belongs to the hot layer. The thermosphere adjustment submodule is used to store the spatiotemporal big data in the first target engine if the spatiotemporal big data belongs to the thermosphere. The cold layer adjustment submodule is used to adjust the spatiotemporal big data from the first target engine to the columnar storage engine if the spatiotemporal big data belongs to the cold layer.

[0156] Optionally, the candidate storage engine has a corresponding local spatial index subtree, the local spatial index subtree including multiple spatial sharding information, the multiple spatial sharding information being used to indicate the spatial range of the spatiotemporal big data stored by the candidate storage engine, and the device further includes: Obtain the time partition directory, which is used to indicate the storage engine to which spatiotemporal big data of different time ranges belong; A query request receiving module is used to receive data query requests, which include a query space range and a query time range. The third target engine determination module is used to filter target space fragment information that intersects with the query space range from the space fragment information of each candidate storage engine, and determine the candidate storage engine corresponding to the target space information as the third target engine. The fourth target engine determination module is used to determine the target time range that intersects with the query time range from the time partition directory, and to determine the storage engine to which the spatiotemporal big data of the target time range belongs as the fourth target engine. The query engine determination module is used to determine the intersection of the third target engine and the fourth target engine to obtain the storage engine to be queried; The data query module is used to perform a query on the storage engine to be queried according to the data query request and obtain the query results.

[0157] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0158] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0159] The embodiments of this disclosure can be implemented as an apparatus configured to use any suitable hardware, firmware, software, or any combination thereof, including electronic devices such as terminal devices, servers (clusters), etc. Figure 8 An exemplary apparatus 800 is schematically shown that can be used to implement the various embodiments described in this application.

[0160] In one embodiment, Figure 8An exemplary device 800 is shown, which includes one or more processors 802, a control module (chipset) 804 coupled to at least one of the processors 802, a memory 806 coupled to the control module 804, a non-volatile memory (NVM) / storage device 808 coupled to the control module 804, one or more input / output devices 810 coupled to the control module 804, and a network interface 812 coupled to the control module 804.

[0161] Processor 802 may include one or more single-core or multi-core processors, and processor 802 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 800 can serve as a terminal device, server (cluster), or other device as described in the embodiments of this application.

[0162] In some embodiments, apparatus 800 may include one or more computer-readable media (e.g., memory 806 or NVM / storage device 808) having instructions 814 and one or more processors 802 that are combined with the one or more computer-readable media and configured to execute the instructions 814 to implement the module and thus perform the actions described in this disclosure.

[0163] In one embodiment, the control module 804 may include any suitable interface controller to provide any suitable interface to at least one of the processors 802 and / or any suitable device or component communicating with the control module 804.

[0164] The control module 804 may include a memory controller module to provide an interface to the memory 806. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0165] Memory 806 may be used, for example, to load and store data and / or instructions 814 for device 800. In one embodiment, memory 806 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 806 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0166] In one embodiment, the control module 804 may include one or more input / output controllers to provide an interface to the NVM / storage device 808 and (one or more) input / output devices 810.

[0167] For example, NVM / storage device 808 may be used to store data and / or instructions 814. NVM / storage device 808 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drive (HDD), one or more optical disc (CD) drives, and / or one or more digital universal optical disc (DVD) drives).

[0168] NVM / storage device 808 may include storage resources that are physically part of a device on which device 800 is mounted, or that are accessible to the device but do not necessarily have to be part of the device. For example, NVM / storage device 808 may be accessed via a network via one or more input / output devices 810.

[0169] One or more input / output devices 810 may provide an interface for device 800 to communicate with any other suitable device. Input / output devices 810 may include communication components, audio components, sensor components, etc. A network interface 812 may provide an interface for device 800 to communicate via one or more networks. Device 800 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof.

[0170] In one embodiment, at least one of the processors 802 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 804. In one embodiment, at least one of the processors 802 may be logically packaged with one or more controllers of the control module 804 to form a system-in-package (SiP). In one embodiment, at least one of the processors 802 may be integrated with the logic of one or more controllers of the control module 804 on the same die. In one embodiment, at least one of the processors 802 may be integrated with the logic of one or more controllers of the control module 804 on the same die to form a system-on-a-chip (SoC).

[0171] In various embodiments, device 800 may be, but is not limited to, a terminal device such as a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, device 800 may have more or fewer components and / or different architectures. For example, in some embodiments, device 800 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0172] The detection device can use a main control chip as a processor or control module, and sensor data, position information, etc. can be stored in a memory or NVM / storage device. The sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0173] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0174] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0175] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable spatiotemporal big data storage terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable spatiotemporal big data storage terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0176] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable spatiotemporal big data storage terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable spatiotemporal big data storage terminal device, causing a series of operational steps to be executed on the computer or other programmable terminal device to produce computer-implemented processing, thereby providing instructions that execute on the computer or other programmable terminal device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0178] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0179] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0180] The present application provides a detailed description of a spatiotemporal big data storage method and apparatus, an electronic device, and a storage medium. Specific examples have been used to illustrate the principles and implementation methods of the present application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present application. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present application. Therefore, the content of this specification should not be construed as a limitation of the present application.

Claims

1. A method for storing spatiotemporal big data, characterized in that, The method includes: Obtain candidate storage engines and the spatiotemporal big data; Determine the storage cost of the spatiotemporal big data on the candidate storage engine, where the storage cost represents the cost of storing the spatiotemporal big data in the candidate storage engine; Based on the storage cost, a first target engine is determined from the candidate storage engines, and the spatiotemporal big data is stored in the first target engine; Determine the storage free energy of the spatiotemporal big data on the candidate storage engine, where the storage free energy represents the stability of the spatiotemporal big data storage in the candidate storage engine; A second target engine is determined from the candidate storage engines based on the storage free energy; If the first target engine is different from the second target engine, and the storage free energy of the first target engine and the storage free energy of the second target engine meet the preset migration conditions, then the spatiotemporal big data will be migrated from the first target engine to the second target engine for storage.

2. The method for storing spatiotemporal big data according to claim 1, characterized in that, Determining the storage cost of the spatiotemporal big data on the candidate storage engine includes: The data features of the spatiotemporal big data and the engine features of the candidate storage engines are obtained. The data features include at least data type, expected query frequency, data volume and expected query pattern. The engine features include at least baseline query latency, unit storage cost and feature query pattern. Based on the data type and the baseline query latency, determine the query latency of the candidate storage engine for the data type; The query latency cost is obtained by multiplying the expected query frequency by the query latency of the candidate storage engine for the data type. Calculate the product of the data volume and the unit storage cost to obtain the economic cost of storage; Determine the degree of matching between the expected query pattern and the feature query pattern to obtain the pattern adaptation penalty; The storage cost of the spatiotemporal big data on the candidate storage engine is determined based on the query latency cost, the storage economic cost, and the pattern adaptation penalty. The step of determining the first target engine from the candidate storage engines based on the storage cost includes: The candidate storage engine with the lowest storage cost is selected from the candidate storage engines and designated as the first target engine.

3. The method for storing spatiotemporal big data according to claim 2, characterized in that, Determining the storage free energy of the spatiotemporal big data on the candidate storage engine includes: Determine the state vector of the spatiotemporal big data at a preset time, wherein the state vector includes access frequency, data density, data volume, and query pattern entropy; The storage energy of the spatiotemporal big data in the candidate storage engine is obtained by multiplying the data volume by the unit storage cost of the candidate storage engine. The storage energy represents the basic storage cost of the spatiotemporal big data in the candidate storage engine. Based on the storage internal energy and the state vector of the spatiotemporal big data at a preset time, the storage free energy of the spatiotemporal big data on the candidate storage engine is determined; The step of determining the second target engine from the candidate storage engines based on the storage free energy includes: The candidate storage engine with the minimum storage free energy is determined from the candidate storage engines and selected as the second target engine.

4. The method for storing spatiotemporal big data according to claim 3, characterized in that, After determining the second target engine from the candidate storage engines based on the storage free energy, the method further includes: If the first target engine is different from the second target engine, then obtain the engine distance between the first target engine and the second target engine; Based on the engine distance and the data volume of the spatiotemporal big data, the migration cost between the first target engine and the second target engine is determined; Obtain the free energy difference threshold; If the difference between the storage free energy of the first target engine and the storage free energy of the second target engine is greater than the migration cost, and the difference between the storage free energy of the first target engine and the storage free energy of the second target engine is greater than the free energy difference threshold, then the spatiotemporal big data will be migrated from the first target engine to the second target engine for storage.

5. The method for storing spatiotemporal big data according to claim 1, characterized in that, After storing the spatiotemporal big data into the first target engine, the method further includes: Determine the spatiotemporal popularity of the aforementioned spatiotemporal big data; The heat level corresponding to the spatiotemporal big data is determined based on the spatiotemporal heat, and the heat level has corresponding storage rules; Adjust the storage engine to which the spatiotemporal big data belongs according to the storage rules.

6. The method for storing spatiotemporal big data according to claim 5, characterized in that, Determining the spatiotemporal heat of the spatiotemporal big data includes: Obtain the number of times the spatiotemporal big data is queried within a preset time window, the set of hotspot spatial regions, and the spatial range of the spatiotemporal big data; The temporal popularity of the spatiotemporal big data is determined based on the number of times it is queried. The spatial heat of the spatiotemporal big data is determined based on the set of hotspot spatial regions and the spatial range of the spatiotemporal big data. The spatiotemporal heat of the spatiotemporal big data is obtained by weighted fusion of the temporal heat and the spatial heat.

7. The method for storing spatiotemporal big data according to claim 5, characterized in that, The heat hierarchy includes a hot layer, a warm layer, and a cold layer; the candidate storage engines include a key-value storage engine, a row-based storage engine, and a column-based storage engine; adjusting the storage engine to which the spatiotemporal big data belongs according to the storage rules includes: If the spatiotemporal big data belongs to the hot layer, then the spatiotemporal big data will be moved from the first target engine to the key-value storage engine and the row storage engine for storage respectively; If the spatiotemporal big data belongs to the thermosphere, then the spatiotemporal big data data is stored in the first target engine; If the spatiotemporal big data belongs to the cold layer, then the spatiotemporal big data will be moved from the first target engine to the columnar storage engine for storage.

8. The method for storing spatiotemporal big data according to claim 1, characterized in that, The candidate storage engine has a corresponding local spatial index subtree, which includes multiple spatial sharding information. These multiple spatial sharding information are used to indicate the spatial range of the spatiotemporal big data stored by the candidate storage engine. The method further includes: Obtain the time partition directory, which is used to indicate the storage engine to which spatiotemporal big data of different time ranges belong; Receive a data query request, the data query request including a query space range and a query time range; Filter the target space shard information that intersects with the query space range from the space shard information of each candidate storage engine, and determine the candidate storage engine corresponding to the target space shard information as the third target engine; Determine the target time range that intersects with the query time range from the time partition directory, and determine the storage engine to which the spatiotemporal big data of the target time range belongs as the fourth target engine; Determine the intersection of the third target engine and the fourth target engine to obtain the storage engine to be queried; The query is performed on the storage engine to be queried according to the data query request, and the query result is obtained.

9. An electronic device, characterized in that, include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the spatiotemporal big data storage method as described in any one of claims 1-8.

10. A machine-readable medium having executable code stored thereon, which, when executed, causes a processor to perform a method for storing spatiotemporal big data as claimed in any one of claims 1-8.

Images