Hierarchical data storage method based on data format analysis

Abstract

The invention relates to the technical field of data storage, and discloses a hierarchical data storage method based on data format analysis, which comprises the following steps of: a, automatically identifying and analyzing the format of input data, classifying different types of original data by adopting a dynamic format identification algorithm, and automatically selecting a proper storage mode according to data format characteristics; b, converting the analyzed data into a unified standardized format, standardizing the data format by adopting a self-adaptive conversion module, and performing feature extraction on the data, including but not limited to improving the storage efficiency of the data through data segmentation, compression, aggregation and the like; and c, based on the data access frequency, the data type and the timeliness. By analyzing and standardizing data in different formats, heterogeneous data is unified into a standard format to be stored. In this way, on one hand, chaos on a storage structure is avoided, and on the other hand, the data can be more efficiently indexed and managed.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a hierarchical data storage method based on data format analysis. Background Art

[0002] With the continuous development of information technology and the advent of the big data era, all walks of life are accumulating massive, heterogeneous and dynamically changing data. Especially in the fields of intelligent transportation, finance, e-commerce, and medical care, the rapid growth of data requires related systems to be able to efficiently store, process and query these large-scale data. However, due to the diversity of data sources and the heterogeneity of data formats, how to effectively store, manage and optimize queries has become a technical problem that needs to be solved urgently.

[0003] At present, although many traditional data storage systems can handle massive amounts of data, their storage efficiency and query performance are often greatly limited when faced with data of different types and formats. Most existing technologies store all data in a unified structure and cannot effectively distinguish between hot and cold data, resulting in a large difference in storage and query efficiency between high-frequency data and low-frequency data, and an inability to flexibly adjust the allocation of storage resources. In addition, existing query mechanisms usually rely on single-dimensional indexes and cannot meet multi-dimensional query requirements, making query performance difficult to guarantee in large-scale data scenarios.

[0004] In the prior art, most data storage systems lack effective tiered storage strategies and dynamic indexing mechanisms, and are unable to adapt to the rapid growth of data and the ever-changing query requirements. Specifically, existing storage solutions are usually unable to dynamically adjust the storage hierarchy according to the access frequency of data, resulting in a waste of storage resources and bottlenecks in query performance. In addition, the traditional index structure has low processing efficiency for complex queries and cannot meet the needs of multi-dimensional, high-concurrency query scenarios. These problems seriously affect the performance and scalability of the system and cannot meet the needs of modern big data application scenarios.

[0005] Therefore, how to improve storage efficiency, query response speed, and reduce system operation and maintenance costs through efficient data format analysis and standardization, reasonable hierarchical storage strategy, dynamic multidimensional index structure, and intelligent query optimization mechanism in the data storage system has become a technical problem that needs to be solved urgently. In response to this problem, the present invention proposes a hierarchical data storage method based on data format analysis, which aims to optimize the data storage structure and improve the query efficiency by performing standardized analysis and hierarchical storage strategy on different data formats, thereby effectively coping with the storage and query challenges in big data scenarios. Summary of the invention

[0006] In view of the deficiencies of the prior art, the present invention provides a hierarchical data storage method based on data format analysis, which solves the problems of the inability to dynamically adjust the storage hierarchy according to the data access frequency and the low query efficiency in the traditional data storage system.

[0007] The following steps are involved: a. Automatically identify and parse the format of input data, use dynamic format recognition algorithm to classify different types of raw data, and automatically select storage method according to data format characteristics; b. Convert the parsed data into a unified standardized format, use an adaptive conversion module to standardize the data format, and extract features from the data, including but not limited to improving data storage efficiency through data segmentation, compression and aggregation; c. Multi-dimensional data stratification is performed based on data access frequency, data type, and timeliness, combined with data content characteristics. A three-tier storage strategy for hot data, cold data, and extremely cold data is adopted. Hot data is stored on high-performance storage media, cold data is stored on low-cost storage media, and extremely cold data is migrated to low-speed long-term archival storage media; d. Build a multi-dimensional dynamic index structure to support efficient data query and fast positioning. By adaptively adjusting the dimension and distribution of the index structure according to the data access mode, the query efficiency is improved. It supports multiple query methods such as complex data retrieval, range query and high-concurrency query. e. Introduce a data migration mechanism during the data storage process. When data access frequency, data life cycle or other external factors change, based on real-time monitoring system feedback, dynamically adjust the data storage level and automatically migrate hot and cold data to ensure optimal use of storage resources; f. Provide an efficient query mechanism based on query caching, parallel computing, and query path optimization, by caching popular data, processing multi-user query requests in parallel, dynamically optimizing query paths to minimize response time, and dynamically adjusting cache strategies based on query history data.

[0008] Preferably, the data format identification in step a includes automatic identification of the data format through metadata, file header information, extension or content features, and parsing using regular expressions or rule matching algorithms.

[0009] Preferably, the feature extraction in step b uses convolutional neural networks for feature extraction of image data, Fourier transform for periodic feature extraction of time series data, and sensor data feature extraction based on data patterns.

[0010] Preferably, the data migration mechanism in step c includes dynamic migration rules based on data access frequency. When the data access frequency is lower than a predetermined threshold, the data is migrated from a high-performance storage medium to a low-cost storage medium, and data integrity is ensured during the migration process through data consistency verification.

[0011] Preferably, the multidimensional index structure in step d is a multidimensional B+ tree, R tree or KD tree, and the index structure supports range query, exact query and spatial query.

[0012] Preferably, the data storage hierarchy management module in step e adopts an LRU algorithm to dynamically adjust the data storage location according to the data access history, and adjusts its storage medium according to the access characteristics of hot and cold data.

[0013] Preferably, the query response mechanism in step f includes a query caching mechanism, a parallel query mechanism and a dynamic query path optimization mechanism. The query caching mechanism reduces repeated queries by caching frequently accessed data; the parallel query mechanism processes query tasks in parallel through a distributed computing platform; and the dynamic query path optimization mechanism adjusts the query execution order according to the query mode and data distribution.

[0014] Preferably, the query cache mechanism manages cache data through the LRU cache strategy, caches hot data preferentially, and updates cache content regularly; the parallel query mechanism decomposes and schedules query tasks through a distributed computing platform to achieve fast query of large-scale data.

[0015] Preferably, the data formats include time series data, image data, video data and sensor data, and the method can adapt to storage and query requirements of different data types.

[0016] The present invention provides a hierarchical data storage method based on data format analysis. It has the following beneficial effects: 1. The present invention parses and standardizes data in different formats to unify heterogeneous data into a standard format for storage. This avoids confusion in the storage structure on the one hand, and enables data to be indexed and managed more efficiently on the other. Through a tiered storage strategy, the system stores hot data, warm data, and cold data on different storage media according to the access frequency and life cycle of the data, thereby minimizing storage costs and ensuring that high-frequency data can obtain efficient storage and access performance.

[0017] 2. The present invention uses a multi-dimensional dynamic index structure to establish efficient indexes for data in multiple dimensions. When querying, the required data can be quickly located by dynamically selecting the optimal index path. In particular, for complex queries, the present invention supports parallel queries and joint queries, which can significantly improve the query response speed. Combined with the query cache mechanism, the system can cache hot data in advance to avoid repeated queries, thereby further optimizing query performance and shortening query latency.

[0018] 3. The hierarchical storage strategy of the present invention not only effectively improves data access efficiency, but also enables the system to dynamically adjust the data storage location according to the different usage frequencies and life cycles of the data. Hot data is stored on high-performance storage media to ensure real-time performance and low latency; cold data is migrated to lower-cost storage media, thereby effectively reducing storage costs. This strategy improves the system's ability to manage different types of data and ensures the rational use of data resources.

[0019] 4. The query optimization mechanism can cope with the complexity of large-scale data queries by intelligently adjusting the priority of query dimensions, selecting the best query path, and supporting parallel queries. In high-concurrency situations, query tasks can be executed in parallel, effectively sharing the query load and reducing query time. At the same time, the system can dynamically adjust the query strategy based on the popularity and access pattern of the data to ensure that queries in different scenarios can achieve optimal performance.

[0020] 5. The present invention greatly improves data analysis and mining capabilities through unified standardization of data formats and a query mechanism based on multi-dimensional indexes. Data can be comprehensively analyzed by multiple dimensions such as time, location, and user. The system can quickly locate relevant data according to user needs, thereby supporting real-time data analysis, trend prediction, and intelligent decision-making. This enables the present invention to have powerful data processing capabilities in various application scenarios, especially in the fields of intelligent transportation, financial risk control, and big data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] 1. Data format analysis and standardization This embodiment provides a hierarchical data storage method based on data format analysis. In this method, data of different formats are first automatically identified and standardized to ensure that all input data can enter the system uniformly and be stored and processed in a predetermined format. Standardization of data formats can not only improve the efficiency of data storage, but also simplify subsequent data operations and analysis processes. For different data types (such as structured data, semi-structured data, and unstructured data), this embodiment adopts dynamic format recognition and data standardization processing mechanisms to ensure that data can be seamlessly migrated between storage media at different levels.

[0024] 1.1 Automatic identification of data format In this embodiment, in order to efficiently process data from various sources and types, it is first necessary to automatically identify the format of the input data. The system automatically identifies different data formats and determines the appropriate parsing method in the following ways: File metadata reading: The system will first read the file's metadata (such as file extension, file header information, byte stream, etc.) to preliminarily determine the data type. Common data formats, such as CSV, JSON, XML, etc., can be identified by simple file extensions or file header information. For example, .csv files will be automatically identified as table data by the system, .json files will be identified as JSON format data, and .xml files will be identified as XML format data.

[0025] Regular expressions and pattern matching: For file types that are not clearly identified (such as some custom log files or sensor data), the system uses regular expressions and pattern matching algorithms to analyze the structural characteristics of the file content. This approach can not only identify text data, but also process some files in custom formats. For example, the system can identify data in a specific format by a specific timestamp or data delimiter.

[0026] Dedicated parsers handle complex data formats: For complex or nested data formats (such as JSON and XML), the system uses dedicated parsers to parse the data. The parser converts the file content into a standard format based on predefined rules. For example, when parsing JSON data, the system will parse the nested data through a tree structure, extract useful information, and store it in a standardized manner.

[0027] Custom format recognition: This embodiment also supports parsing data in custom formats. For example, when encountering a log file generated by a specific sensor device, the system can identify its data structure through preset rules or machine learning algorithms and convert it into a unified format for subsequent processing.

[0028] 1.2 Data Standardization After the data format is parsed, the parsed data needs to be standardized. The purpose of data standardization is to eliminate the differences between various data formats so that all data can be uniformly managed and accessed in the same storage system. In this embodiment, the data standardization process includes steps such as data segmentation, data compression, and data aggregation to improve storage efficiency, reduce redundancy, and improve subsequent processing efficiency.

[0029] Data segmentation: In this embodiment, for large file data (such as video, image, audio, etc.), a fixed-size block strategy is used to divide the large file into several data blocks. Each data block has a certain size (for example, 5MB or 10MB) to facilitate subsequent access operations in the storage system. This block mechanism can effectively reduce the storage burden of large files and improve reading efficiency. Each data block is also attached with metadata to mark its position in the entire file.

[0030] For time series data (such as temperature and humidity data collected by sensors), the system will segment the data according to time periods. For example, the data can be divided into multiple time periods by hour or day for storage, so that the storage system can query and retrieve data based on time.

[0031] Data compression: For some data with high redundancy (such as log data, historical sensor data, etc.), this embodiment uses a compression algorithm to compress the data to save storage space. The compression process selects different compression algorithms according to the data type. Common lossless compression algorithms such as ZIP and GZIP are used to process text data or structured data, while lossy compression algorithms such as JPEG and MP4 are used to process image and video data. Through compression, storage costs can be effectively reduced, especially when processing large-scale data.

[0032] For example, for temperature sensor data, the system uses a difference-based compression algorithm to store only the difference between two adjacent readings instead of storing all the original data, thereby significantly reducing the amount of data.

[0033] Data aggregation: For similar types of repeated data, the system reduces redundancy through data aggregation. For example, when processing sensor data, the system aggregates multiple data in the same time period (such as temperature, humidity, air pressure, etc.) into a unified record. For sensor data at different times, the system aggregates them according to time period or other related dimensions (such as device number).

[0034] For example, multiple sensor data such as temperature, humidity, and light intensity can be aggregated according to timestamps to form a composite data record containing multiple sensor values. In this way, the data is not only more compact, but also provides more contextual information in the subsequent analysis process.

[0035] Data format conversion: During the standardization process, the system converts data according to the target storage format. For text data, it is converted to JSON or CSV format for storage; for binary data (such as images and audio files), the system converts it to a unified binary encoding format for storage. The data conversion process supports two-way conversion. For example, the system can convert JSON format data to CSV format, or convert XML format data to structured database records.

[0036] 1.3 Data verification and consistency check In order to ensure that data is not lost or damaged during the parsing and standardization process, a data verification and consistency check mechanism is added in this embodiment. The specific process is as follows: Data verification: At each step of parsing and standardization, the system verifies the data to ensure its integrity. For example, for sensor data, the system verifies the rationality of timestamps and data values, and detects whether there are duplicate, missing, or abnormal timestamps. Once data errors or inconsistencies are found, the system will automatically repair them or notify the administrator.

[0037] Consistency check: For data transmission across systems, the system also performs consistency checks. When the data is parsed and stored in the distributed storage system, the system ensures the consistency of the data on all nodes to avoid data loss or duplication. The system uses a distributed consistency protocol (such as Paxos or Raft) to ensure the consistency of multi-node data.

[0038] Error reporting and recovery mechanism: When a data parsing error occurs, the system can automatically generate an error report and perform data recovery. The recovery mechanism uses incremental backup and transaction log technology to restore to the most recent valid state in the event of a failure.

[0039] 1.4 Optimization strategies in data standardization In this embodiment, the data standardization process is not limited to format conversion and compression processing, but also introduces the following optimization strategies: Adaptive standardization strategy: For different types of data, the system adopts an adaptive standardization strategy. For example, for structured data (such as table data), more stringent format verification and cleaning steps are adopted; for text data, natural language processing technology is used to perform semantic analysis on the data, extract key information, and reduce the storage of irrelevant data.

[0040] Batch processing and streaming processing: Data standardization can be optimized through batch processing and streaming processing. For historical data, batch processing is used to complete data standardization at one time; for real-time data, streaming processing is used to perform standardization while data is generated to ensure real-time performance.

[0041] Through the above technical implementation methods, the present invention realizes an efficient data format parsing and standardization process. This process can automatically identify and standardize data from different sources and in different formats to ensure the availability and consistency of the data. Storage space is optimized by means of segmentation, compression, aggregation, etc., while adaptive standardization strategies, data verification and consistency checking technologies are used to ensure data quality and reliability. The whole process not only reduces the cost of data storage, but also improves the efficiency of subsequent data processing and analysis, and is suitable for data management needs in the fields of big data, cloud computing, and the Internet of Things.

[0042] 2. Tiered Storage Strategy This embodiment provides a hierarchical data storage method based on data format analysis, and designs a hierarchical storage strategy by combining different types of data, access frequency and life cycle characteristics. This strategy allocates data to different storage media according to different storage requirements, thereby optimizing storage efficiency and access performance. Through intelligent data hierarchical storage, this embodiment can effectively reduce storage costs, improve data access speed, and ensure that data is effectively managed throughout its life cycle. The hierarchical storage strategy mainly includes dynamic adjustments based on data access frequency, data life cycle and storage medium performance differences, thereby achieving efficient storage and fast retrieval of data.

[0043] 2.1 Storage Tier Definition In this embodiment, the tiered storage strategy divides the storage medium into multiple tiers according to the importance, access frequency and storage requirements of the data. Each storage tier has different performance, capacity and access speed characteristics, including: Hot data tier: The hot data tier stores the most frequently accessed data. This type of data is usually generated in real time, has high user access requirements, and is frequently updated. At this level, the system gives priority to storage media with higher performance, such as solid-state drives (SSDs) or memory (RAM). By storing frequently accessed data in fast-responding storage media, the performance of data reading and writing can be significantly improved.

[0044] Warm data layer: The warm data layer stores data that is accessed less frequently but still needs to be accessed regularly. This type of data is usually historical data or periodically generated data, and the system can choose to use hard disk drives (HDDs) or other high-capacity but slower storage media. The storage cost of warm data is relatively low, but because it is not accessed frequently, slower storage media will not have much impact on system performance.

[0045] Cold data layer: The cold data layer stores data that has not been accessed for a long time or is about to expire. This data is not important for the direct use of the system, but may still need to be retained for future queries or audits. Cold data is generally stored on high-capacity storage media, such as tape storage, cloud storage, etc. Although the access speed is slow, the storage cost is low. Data at this level is mainly stored through archiving strategies, which reduces the occupation of storage resources.

[0046] Archive data layer: The archive data layer is used to store data that no longer needs to be frequently accessed. The system will select long-term low-cost storage media for storage, such as cold storage, optical disk storage, etc. Archive data is mainly used for compliance storage or historical data backup. It has the lowest storage cost but a large access delay. Data at this level is not frequently read in normal operations and is usually retrieved only when a specific audit or recovery is required.

[0047] 2.2 Dynamic Adjustment of Storage Tiers In order to ensure the efficiency and flexibility of tiered storage, this embodiment introduces a dynamic adjustment mechanism. This mechanism evaluates the storage requirements of data in real time and adjusts its storage tier based on factors such as data access mode, life cycle, and storage cost. The specific implementation is as follows: Data access frequency monitoring: In this embodiment, by real-time monitoring of data access frequency, the system can determine whether the data is hot data, warm data, or cold data. Whenever the access frequency of data changes, the system will automatically adjust the data storage location according to the preset threshold. For example, if some cold data is suddenly frequently accessed, the system will automatically migrate it from the cold data layer to the warm data layer or hot data layer to ensure its access performance.

[0048] Data lifecycle management: The system automatically migrates data from one storage layer to another based on the data lifecycle. After data is generated, it is usually first stored in the hot data layer. As time goes by, the access frequency gradually decreases, and the system will migrate it to the warm data layer or cold data layer. When the data reaches the end of its lifecycle (for example, when the data no longer needs to be retained), it will be archived to the archive data layer for storage, or deleted according to the policy.

[0049] Storage cost optimization: The system optimizes data storage strategies by evaluating the costs of each storage tier. For example, when storage space is limited or storage costs need to be saved, the system can prioritize the migration of infrequently accessed cold data to lower-cost storage media, while selecting high-performance storage media for hot data that requires high performance. The system continuously adjusts the storage location of data to ensure that storage costs and performance between different tiers are optimally balanced.

[0050] 2.3 Data Migration Strategy between Storage Tiers To ensure seamless data migration between different storage layers, this embodiment designs an intelligent data migration strategy. Data migration is not only about moving data from one storage layer to another layer according to predetermined rules, but also takes into account the performance impact, cost changes, and actual data usage requirements during migration. Specific strategies include: Time-based migration strategy: Based on the timestamp and generation time of the data, the system will set a scheduled migration plan. For example, the system can automatically check the access frequency of data every month or quarter, identify data that is no longer frequently accessed, and migrate it to a lower storage tier.

[0051] Migration strategy based on access patterns: The system analyzes the access patterns of data through machine learning algorithms and dynamically adjusts the storage tier based on the analysis results. For example, for cold data that is frequently accessed suddenly, the system can migrate it to the warm data tier; and for data that is no longer accessed in a short period of time, it can be quickly transferred to the archive tier.

[0052] Migration strategy based on storage pressure: When the storage medium space reaches the preset threshold, the system will automatically trigger the data migration strategy. Hot data with greater storage pressure will be migrated to the warm data layer or cold data layer first to free up storage space; at the same time, for cold data or archived data, the system will also decide whether to delete it or migrate it to a cheaper storage medium based on demand and cost.

[0053] 2.4 Data Redundancy and Fault Tolerance In order to ensure data reliability in the multi-layer storage system, this embodiment also introduces data redundancy and fault tolerance mechanisms. In tiered storage, important data is not only stored on high-performance storage media, but also redundant backups are created on low-cost storage media. Specifically, it includes: Redundant backup strategy: For important data, the system will perform redundant backup between the hot data layer and the warm data layer. For example, the system will store data in multiple locations and arrange different redundant backup strategies according to the different access frequencies. For important data that needs to be accessed temporarily, redundant storage is performed in high-performance media; for cold data, less frequent redundant backup can be performed.

[0054] Fault-tolerance mechanism: The storage system uses a fault-tolerance mechanism to ensure that data can be fully recovered even if part of the storage media fails. For example, the system uses RAID technology (Redundant Array of Independent Disks) to ensure that data has multiple copies in a multi-hard disk system. Even if a hard disk fails, the data can still be recovered through other copies.

[0055] 3. Multidimensional dynamic index structure In order to efficiently support fast search and access of multi-layer storage data, this embodiment designs a multi-dimensional dynamic index structure for fast indexing and retrieval between different data levels in the data storage system. The multi-dimensional dynamic index structure constructs a dynamically adjustable index mechanism based on the characteristics of data in different dimensions (such as time, space, access frequency, etc.) and the relationship between data, so that efficient query, update and data migration operations can be achieved during large-scale data storage and access. The structure adopts flexible multi-dimensional data indexing technology to make the data query process more efficient, especially suitable for storing multi-dimensional, high-frequency and large-capacity data sets.

[0056] 3.1 Design Concept of Multidimensional Index The multidimensional index structure in this embodiment is designed based on multiple characteristic dimensions of the data, and the index is updated and adjusted in real time according to the dynamic changes of the data. The index of each dimension can be operated independently of other dimensions, and during the data storage process, its storage location is dynamically adjusted according to the access mode and life cycle changes. The multidimensional dynamic index structure has the following characteristics: Multidimensional data support: In traditional single-dimensional indexes, data retrieval mainly depends on a fixed condition, such as time or space coordinates. Multidimensional index structures can support queries in multiple dimensions, such as time, space, data type, access frequency, etc., which allows flexible queries based on different needs. In particular, when the query conditions of data involve multiple feature dimensions, multidimensional indexes can accelerate the query process through multi-level matching.

[0057] Dynamic adjustment mechanism: The multi-dimensional dynamic index in this embodiment is not statically constructed, but dynamically adjusts the index level and structure according to changes in data, changes in query patterns, changes in storage levels, etc. As the data storage environment changes, the index structure can be continuously optimized, avoiding the efficiency bottleneck caused by the static index structure in large-scale data storage.

[0058] Support for large-scale data query: The multi-dimensional dynamic index structure is particularly suitable for query and retrieval of large-scale data sets, especially when multiple query conditions (such as time and space coordinates, access frequency, etc.) need to be considered at the same time, and efficient query response can be achieved. By dynamically allocating index resources and automatically adjusting the index granularity, the system can ensure efficient query performance when processing massive data.

[0059] 3.2 Implementation of multi-dimensional dynamic index structure In this embodiment, the index structure based on multidimensional data adopts a scheme combining a tree structure and a hash table, which specifically includes the following steps: Dimension selection and modeling: When performing multidimensional indexing, you first need to select the index dimension based on the characteristics of the data. Common dimensions include time dimension, space dimension, data access frequency dimension, data type dimension, etc. In this embodiment, the following dimensions are used for data modeling: Time dimension: Data is layered according to the time when it was generated. The data life cycle is divided into different stages such as hot, warm, and cold. Data at different stages will be allocated to different storage levels based on time characteristics.

[0060] Spatial dimension: The spatial location of the data (e.g., geographic location, device location, etc.) is indexed as another dimension. Based on the spatial distribution of the data, the system can quickly determine where the data is stored.

[0061] Access frequency dimension: Data is classified according to the frequency of data access. Frequently accessed data is stored in higher-performance storage media, while less frequently accessed data is stored in lower-cost storage media.

[0062] Index structure construction: In this embodiment, a multi-dimensional index structure based on a combination of B+ tree and hash table is adopted: B+ tree: B+ tree is used to process multi-dimensional range queries in large-scale data storage, and can efficiently process interval queries and adjacent data searches. Each node contains index items for multiple data dimensions, and the leaf node points to the location where the data is actually stored.

[0063] Hash table: Hash table is used to handle high-frequency query scenarios and can directly locate the target data through hash mapping. In the multidimensional index structure, hash table is mainly used to quickly locate the target data according to dimensions such as time or space to improve the search speed.

[0064] Data index creation and update: During the data storage process, indexes are added to the corresponding locations according to its multidimensional characteristics. During the life cycle of the data, as the data access pattern changes, the system will regularly update the index structure according to the preset dynamic update mechanism. For example, if some data is suddenly frequently accessed, the system will automatically update its index information, move it to a higher performance storage tier, and readjust the corresponding index. Dynamic update methods include: Incremental update: When new data is added to the system, the system will automatically perform incremental updates to the index based on the multi-dimensional attributes of the new data to ensure the real-time nature of the index.

[0065] Batch update: Batch update of index structure is triggered regularly in the system, especially when the data storage level changes, the system will recalculate the index structure to ensure fast access to data.

[0066] Index granularity and optimization: The multi-dimensional dynamic index structure of this embodiment has an adjustable granularity. According to the frequency of data access and changes in the storage hierarchy, the system can adaptively adjust the index granularity. For example, when data changes frequently, the index structure will adopt a finer granularity to speed up data access; and for infrequently accessed data, the index granularity can be appropriately reduced to reduce storage and computing overhead.

[0067] 3.3 Multi-dimensional dynamic index query strategy In a multi-dimensional dynamic index structure, data query is achieved through multi-dimensional matching to achieve efficient search. Specific query strategies include: Multi-dimensional query optimization: When a query involves multiple dimensions, the system will prioritize index matching based on the most frequently queried dimension. For example, if the time dimension is frequently queried, the system will first search in the B+ tree or hash table of the time dimension, and then filter through other dimensions. This strategy can reduce invalid index queries and improve query efficiency.

[0068] Condition combination query: supports the combination of multiple query conditions, and quickly obtains matching results by processing indexes of different dimensions in parallel. For combinations of multiple dimensions such as time, space, and access frequency, the system can quickly perform multi-dimensional data retrieval based on the characteristics of the index structure.

[0069] Query result optimization: This embodiment supports optimized sorting of query results. According to the access frequency and storage level of the data, the system will prioritize the query results and give priority to returning data that is frequently accessed or stored in high-performance storage media to improve the response speed of data access.

[0070] 3.4 Maintenance and Adjustment of Dynamic Indexes As data access patterns and storage structures change, the maintenance and adjustment of index structures are crucial. This embodiment ensures the efficiency and accuracy of the index by: Automatic optimization: The system automatically optimizes the index structure based on changes in data and query patterns. For example, the system will reduce the index granularity of data that is no longer frequently accessed, while increasing the index granularity of frequently accessed data and ensuring that the data storage location is optimized.

[0071] Periodic reconstruction: The system will regularly rebuild multi-dimensional dynamic indexes, especially when the amount of data increases sharply or the storage level changes significantly. Index reconstruction can further improve query performance.

[0072] The multi-dimensional dynamic index structure designed in this embodiment provides an efficient data retrieval and query method for large-scale data storage systems by supporting indexing and querying of multi-dimensional data, dynamic adjustment mechanism, and intelligent optimization strategy. This structure can ensure the efficiency and scalability of data retrieval under the background of dynamic adjustment of storage levels, changes in data access frequency, and diversified query conditions, and is particularly suitable for data-intensive application fields such as cloud computing, big data analysis, and the Internet of Things.

[0073] 4. Query Optimization Mechanism In data storage systems, query efficiency is one of the key factors that determine system performance. Especially in the context of multi-layer data storage and multi-dimensional dynamic indexing, how to improve query response speed, reduce unnecessary computing overhead, and optimize data access paths has become an important goal of technical implementation. In this embodiment, a mechanism based on query optimization is designed. By combining multi-dimensional index structure, dynamic data update strategy and intelligent query scheduling, efficient processing of complex query requirements in a multi-layer data storage environment is achieved. Through reasonable query optimization, the present invention can significantly improve data retrieval performance in large-scale data storage scenarios and ensure the rapid response of the system in a high-frequency access environment for multi-dimensional data.

[0074] 4.1 Design Idea of ​​Query Optimization Mechanism The query optimization mechanism in this embodiment is mainly based on the following design ideas: Query priority management based on data dimensions: This embodiment analyzes multiple dimensions in the query conditions, intelligently determines the query priority, and dynamically adjusts the query priority order based on the access frequency, storage level, and update status of each dimension. For example, in the multidimensional data query process, first start with the frequently accessed dimensions to quickly locate relevant data, and then further refine the query based on other dimensional conditions. In this way, invalid data access can be reduced during querying and retrieval efficiency can be improved.

[0075] Dynamic query path selection and optimization: As the data storage environment changes (for example, changes in data popularity, changes in access frequency, etc.), the query optimization mechanism can dynamically select the optimal query path based on the current storage structure and index status. Different query paths are used for hot data and cold data to avoid frequent access to cold data, thereby reducing storage burden and computing overhead.

[0076] Parallel query processing: In order to cope with high-concurrency query scenarios, this embodiment supports parallel query processing. The query optimization mechanism can divide the query request into multiple subtasks and assign the query tasks to appropriate storage nodes for parallel execution according to the data storage structure. This method can effectively reduce the query response time and improve the system's processing capacity.

[0077] Caching mechanism and result pre-fetching: Combined with query history and data access patterns, the query optimization mechanism can cache frequently used queries to reduce the overhead of repeated queries. In addition, through the pre-fetching mechanism, the system can load the data that may be needed into the cache in advance before the user initiates a query request, thereby improving the query response speed.

[0078] 4.2 Technical Implementation of Query Optimization Mechanism According to the above design ideas, the query optimization mechanism in this embodiment is mainly supported by the following technical implementation methods: Dimension priority adjustment: During the query optimization process, the query conditions of multidimensional data are first analyzed, and the system determines which dimensions dominate the current query. For example, for some data, the time dimension may be the most prioritized query condition, while for other data, the space or access frequency dimension may be more important. By prioritizing the dominant dimension, the query system can quickly lock in the target data range and reduce redundant retrieval operations.

[0079] In this embodiment, the following algorithm strategy is used for the analysis of query conditions: Frequency analysis: Based on historical query records and data access frequency, analyze which dimensions are the key conditions for the current query. For example, if the time dimension changes frequently and the query demand is high, the system will give priority to the index of the time dimension for query.

[0080] Heat determination: The query path is selected based on the heat and life cycle of the data (such as hot data, cold data, etc.). Hot data is preferentially queried using high-performance storage media and query paths, while cold data is queried through low-cost storage paths.

[0081] Dynamic query path selection and adjustment: The query path selection in this embodiment is dynamically adjusted based on changes in the storage level and data access frequency. For example, if the data is in a high-frequency access state and is stored on a slower storage medium, the query optimization mechanism will automatically choose to query from a high-frequency storage medium (such as SSD or memory). Conversely, for low-frequency data, query requests will preferentially point to low-cost storage levels (such as disk arrays or cloud storage). During the process of data migration or updating, the query path will be dynamically adjusted to ensure that the system can always provide efficient query performance.

[0082] The query path selection algorithm is based on the following logic: Data heat analysis: Monitor the heat of data in real time, evaluate the access frequency of data in the current time window, and select the access path accordingly.

[0083] Multi-layer storage strategy: In the system, data is stored in layers according to its popularity, importance, and access frequency. Query requests dynamically adjust access paths based on the query dimensions and the level of data storage.

[0084] Parallel query optimization: For large-scale data storage and high-concurrency query requests, this embodiment further improves the query efficiency of the system through the optimization method of parallel query. During the query request processing, the system will decompose the query request into multiple subtasks and execute them in parallel through multi-threading or distributed processing. This method is particularly suitable for multidimensional data query scenarios and can significantly reduce query time.

[0085] The implementation steps of parallel query are as follows: Task decomposition: The system divides the query request into multiple subtasks and performs queries on different data dimensions.

[0086] Task scheduling: According to the distribution of data storage and the characteristics of query requests, subtasks are scheduled to be executed on appropriate storage nodes to ensure the optimal access path to data.

[0087] Query merging and result return: After each subtask is executed, the system will merge the query results and return them to the user. Query merging adopts an on-demand return strategy to ensure the accuracy of the query results.

[0088] Cache mechanism and result pre-fetching: To further optimize the query speed, this embodiment designs a cache mechanism and result pre-fetching strategy. Before the user queries, the system predicts the data that may be queried by analyzing the user's historical query records and data access patterns, and loads it into the cache in advance. In this way, during the actual query, the system can return the results directly from the cache, avoiding repeated access to the storage layer.

[0089] The key technologies for result prefetching are as follows: Historical query analysis: By analyzing past query logs, the system can identify the data that users frequently query, and will preload this data into the cache first.

[0090] Data access pattern recognition: By analyzing the access pattern of data (such as sequential access, random access, etc.), the system can dynamically predict which data will be accessed in the near future and load it in advance through the prefetch mechanism.

[0091] 4.3 Query Optimization Effect By adopting the above query optimization mechanism, this embodiment can achieve the following technical effects: Improve query response speed: The query optimization mechanism can effectively reduce invalid operations during the query process, prioritize the retrieval of the most relevant data, and select the best query path, thereby greatly reducing the query response time.

[0092] Optimize storage resource usage: By intelligently adjusting query paths and data access strategies, the utilization of storage resources is optimized, especially in multi-tier storage environments, which enables dynamic selection of storage media to reduce costs.

[0093] Support high-concurrency query: The parallel query and caching mechanism can maintain good query performance in high-concurrency scenarios and effectively support real-time query requirements for large-scale data.

[0094] The query optimization mechanism of this embodiment effectively improves query performance in large-scale data storage and multi-level data access environments by means of dimensional analysis of query conditions, dynamic adjustment of query paths, parallel processing of query requests, and intelligent caching and pre-fetching strategies. This mechanism can not only significantly improve data retrieval speed, but also intelligently allocate storage and computing resources, optimize the overall performance of the storage system, and provide an efficient solution for complex multi-dimensional data queries. Example

[0095] In the big data analysis platform, data is of various types and comes from a wide range of sources, and needs to be stored, queried and analyzed in real time and efficiently. These data include data from different sensors, user behavior logs, social media data and other data in various formats and dimensions. Due to the huge amount of data and rapid changes, traditional data storage methods often face problems such as high storage cost, low query efficiency, and inflexible data access. To solve these problems, this embodiment adopts a hierarchical data storage method based on data format analysis, and combines a multi-dimensional dynamic index structure and a query optimization mechanism to provide the platform with efficient and flexible data storage and query capabilities.

[0096] Implementation steps 1. Data format analysis and standardization In the big data analysis platform, data comes from different sources and formats, including structured data (such as SQL database tables), semi-structured data (such as JSON log files), and unstructured data (such as pictures, videos and other multimedia files). In order to ensure efficient storage and retrieval of these heterogeneous data, the data format must first be parsed and standardized.

[0097] In this embodiment, the platform performs data format parsing through the following steps: Data format identification: Perform preliminary analysis on all received raw data to identify the data type and format. For example, for sensor data, identify it as time series data; for user logs, identify it as text data or JSON format data; for images, identify it as binary data.

[0098] Data standardization: For each type of data, specific parsing rules are used to convert it into a unified standard format. For time series data, the data is uniformly converted into a standard form of timestamp-value pairs; for logs in JSON format, they are parsed into key-value pairs and uniformly encoded; for image data, they are converted into a storage method based on file path and metadata.

[0099] Data storage specifications: The parsed data is stored in a unified format and a corresponding identifier is assigned to each data element for subsequent indexing and querying.

[0100] Through the above standardization process, the platform ensures that data in different formats can be stored and retrieved according to consistent rules, providing a good data foundation for subsequent hierarchical storage and multidimensional indexing.

[0101] 2. Tiered storage strategy In the big data platform, due to the differences in data access frequency, a tiered storage strategy is adopted to divide the data into hot data, warm data and cold data, and automatically select the most appropriate storage medium and storage level based on the data's popularity and access pattern.

[0102] Hot data: refers to frequently accessed data, which is usually stored in high-performance storage media (such as memory or SSD). For example, real-time sensor data, user active behavior data, etc.

[0103] Warm data: refers to data that is accessed with moderate frequency and is stored in more efficient but lower-cost storage media (such as disk arrays). For example, recent user behavior data and platform analysis report data.

[0104] Cold data: refers to data that is rarely accessed and is usually stored in low-cost storage media (such as cloud storage or large-capacity tapes). For example, historical log files, expired user data, etc.

[0105] Data migration between storage tiers is automated. When the access frequency of data changes, the platform automatically adjusts its storage location based on the data usage pattern. For example, if certain data is frequently accessed over a period of time, it will automatically migrate to the hot data tier; conversely, if certain data becomes cold after a period of time, the system will migrate it to the cold data tier. This strategy effectively reduces storage costs and improves data access efficiency by combining tiered storage management with a dynamic migration mechanism.

[0106] 3. Multidimensional dynamic index structure In the big data analysis platform, the conditions for data query usually involve multiple dimensions, such as time dimension, space dimension, user dimension, etc. In order to improve query efficiency, the platform adopts a multi-dimensional dynamic index structure to support the rapid retrieval of multi-dimensional data.

[0107] Multidimensional index modeling: The platform creates multidimensional indexes by dimensional modeling of data. Specifically, it includes: Time dimension: Use timestamp as the primary key to index data and support queries by time range.

[0108] Spatial dimension: For data related to geographic location, longitude and latitude coordinates are used for indexing, supporting geographic location range queries.

[0109] User dimension: Indexed by user ID, supporting quick query of related data based on user attributes.

[0110] Dynamic index update: Changes in data popularity and access patterns will affect the dynamic update of multidimensional indexes. For example, the platform will regularly analyze the frequency of data access and adopt a more fine-grained update strategy for the dimension index of hot data, while reducing the index granularity for cold data to save computing resources.

[0111] Efficient query: Through the multi-dimensional index structure, users can quickly query on the platform. For example, which sensor data a user accessed within a certain period of time, or which key events occurred near a certain geographical location, can all be quickly retrieved through multi-dimensional indexes. The platform also supports joint query, which reduces the scope of data search through the combination of multiple dimensions, thereby improving query speed.

[0112] 4. Query optimization mechanism In the platform, the efficiency of data query directly affects the response time of the system, especially in high-concurrency and large-data query scenarios. In order to further optimize query efficiency, the platform has designed a query optimization mechanism that combines dimension priority adjustment, query path selection, parallel query and other technologies.

[0113] Dimension priority optimization: Based on historical query logs and data access patterns, the platform intelligently adjusts the query dimension priority. For example, when a user queries activity data within a certain time period, the time dimension has a higher priority, so the system will first query the index of the time dimension.

[0114] Dynamic query path selection: In different query scenarios, the platform can automatically select the optimal query path. For example, when frequently accessed hot data, the system will first query from the memory or SSD storage layer; for cold data, the query will first be performed from the cloud storage layer to reduce the burden of high-cost storage.

[0115] Parallel query and cache: When the platform receives multiple concurrent query requests, the system speeds up data access by parallelizing query processing. For frequently queried data, the system will cache it in advance to avoid accessing the original storage for each query, further improving query response speed.

[0116] Through these query optimization mechanisms, the platform can ensure that results are returned in a shorter time when faced with large-scale data retrieval, greatly improving user experience and system response speed.

[0117] Application Effect The application of this embodiment in the big data analysis platform significantly improves the efficiency of data storage and query, which is specifically reflected in the following aspects: Storage efficiency optimization: Through the tiered storage strategy, the platform effectively reduces storage costs, especially the low-cost storage method for cold data, avoiding excessive resource consumption for low-frequency data.

[0118] Improved query response speed: Combining the multi-dimensional dynamic index structure with the query optimization mechanism, the platform can flexibly select the query path according to the query conditions and significantly shorten the query response time through parallel query.

[0119] Large-scale data processing capabilities: Even in high-concurrency, large-data volume scenarios, the platform can still maintain good query performance, support large-scale user query needs, and be able to make adaptive adjustments as data changes dynamically.

[0120] Through this embodiment, the advantages of the hierarchical data storage method based on data format analysis are fully demonstrated. While improving query efficiency, the system can also effectively manage storage resources, providing strong technical support for the big data platform. Example

[0121] In the intelligent transportation system, various types of traffic data (such as traffic flow, vehicle speed, vehicle location, traffic light status, etc.) are collected in real time in multiple formats and large amounts of data. These data come from different sensors (such as traffic monitoring cameras, vehicle-mounted GPS devices, roadside radars, etc.) and data sources, and the amount of data is exploding over time. How to efficiently store, manage, query and analyze these massive, heterogeneous and dynamically changing data has become a technical challenge in the intelligent transportation system. In order to meet the real-time, accuracy and scalability requirements of the system, this embodiment adopts a hierarchical data storage method based on data format analysis, combined with a multi-dimensional dynamic index structure and a query optimization mechanism, to optimize the data storage structure and improve the query efficiency and response speed of the system.

[0122] Implementation steps 1. Data format analysis and standardization The data sources in the intelligent transportation system are extremely diverse, including real-time traffic flow data, vehicle GPS location data, road condition video monitoring data, traffic light status, etc. Since the formats of these data are different, in order to ensure unified storage and improve the subsequent data processing efficiency, this embodiment first performs data format analysis and standardization.

[0123] Data format identification: The system first identifies the formats of different data sources. For example, video data from traffic cameras is usually in image stream format, vehicle GPS location data is usually in JSON format, and traffic flow data is in CSV format. Based on the data type, the system can automatically identify the data source and mark it in the corresponding format.

[0124] Data standardization: Convert data in different formats into a unified storage format to ensure efficient storage and indexing. For example, traffic flow data will be uniformly converted into the form of timestamp-traffic flow pairs; GPS data will be converted into a structured format containing latitude and longitude coordinates, timestamps, speed and other information; video data will extract key frame information and annotate it according to time sequence and event characteristics.

[0125] Metadata and label processing: In addition to the original data, the system will also annotate the data with metadata. For example, for video data, the video shooting time, location, event type and other information will be annotated; for GPS data, the vehicle ID, current speed and other information will be annotated. These metadata will be used for subsequent query and analysis.

[0126] 2. Tiered storage strategy There are many types of traffic data, and the access frequencies vary greatly. Therefore, this embodiment adopts a tiered storage strategy to divide the data into hot data, warm data, and cold data, and stores them in different storage media according to the access frequency and the life cycle of the data to improve storage efficiency and reduce storage costs.

[0127] Hot data: real-time collected traffic flow, vehicle speed, and traffic light data, which usually require low latency and fast response and are stored on high-speed storage devices such as SSDs and memory. For example, traffic flow data and vehicle location data in the current period.

[0128] Warm data: historical traffic data, stored in low-cost but relatively fast storage media, such as disk arrays or HDDs. For example, traffic flow data and vehicle speed data from the past hour to a week, which are accessed less frequently but need to be analyzed and processed.

[0129] Cold data: data stored for a long time, with very low access frequency, usually stored in low-cost, low-speed storage devices such as cloud storage or tape storage. For example, traffic surveillance video data from a few months or years ago, old accident record data, etc.

[0130] Through tiered storage, the system can store high-frequency data on high-speed storage media, reducing the access pressure on low-frequency data and effectively saving storage costs.

[0131] 3. Multidimensional dynamic index structure In an intelligent transportation system, data query usually involves multiple dimensions, such as time dimension, geographic dimension, vehicle dimension, etc. In order to improve query efficiency, this embodiment adopts a multi-dimensional dynamic index structure to support fast query based on multiple dimensions.

[0132] Time dimension index: Traffic data queries usually involve data such as traffic flow and vehicle speed within a specific time period. For example, the system needs to query the traffic flow of a certain road section within a certain period of time. The system supports queries by time range through timestamp-based indexes.

[0133] Geographic location dimension index: Traffic data queries are often related to geographic location. For example, to query the traffic conditions of a certain area or road section, the system uses longitude and latitude coordinates to establish a spatial index and supports queries by geographic location range. Using spatial index structures such as R-tree, efficient geographic location queries can be achieved.

[0134] Vehicle dimension index: Vehicle data in the system (such as GPS location, vehicle speed, etc.) usually needs to be queried by vehicle. For example, query the location change of a vehicle within a specific time, or query the real-time status of a group of specific vehicles. The platform creates an index by vehicle ID and supports fast query by vehicle dimension.

[0135] Dynamic index update: As data continues to change, query priorities and data popularity will also change. For example, real-time data queries need to prioritize time and geographic location dimensions, while historical data queries can be optimized based on vehicle dimensions. Therefore, the system will dynamically update the index structure based on data access frequency and query priority to ensure continuous optimization of query performance.

[0136] 4. Query optimization mechanism In intelligent transportation systems, query optimization is one of the key technologies, especially when facing a large number of concurrent queries. In order to improve the query response speed, the platform has designed a comprehensive query optimization mechanism.

[0137] Query path selection: The system dynamically selects the query path based on the query type and data distribution. For example, when a user queries the traffic flow in a certain area, the system first uses the geographic location dimension index, and then combines the time dimension and vehicle dimension for further query optimization.

[0138] Parallel query processing: For complex queries (such as querying joint information of multiple dimensions), the system will split the query task into multiple subtasks and execute them in parallel. This can effectively reduce query time and improve query throughput. For example, when querying the traffic flow of multiple sections in a certain time period, the system will execute the query task in parallel on multiple storage nodes, and finally merge the results and return them to the user.

[0139] Query caching and result pre-fetching: To further optimize query speed, the system stores frequently queried data through a caching mechanism. For example, real-time traffic flow data on certain roads, or traffic accident data in a certain area, the system will cache these hot data to avoid accessing the storage medium for each query. In addition, the system will predict the user's possible query requests based on query history and data access patterns, and load relevant data into the cache in advance.

[0140] Application Effect By implementing the above technical solutions, the intelligent transportation system has achieved significant optimization effects in many aspects: Improved query efficiency: Due to the use of multi-dimensional index structure and dynamic query path selection, query response time is greatly shortened. For example, users can obtain traffic flow data in a certain area within a certain time period within a few seconds, and the system can support high-concurrency query requests.

[0141] Reduced storage costs: Through tiered storage, the system can store high-frequency data in high-performance devices and low-frequency data in low-cost media, effectively reducing storage costs while ensuring fast access to hot data.

[0142] Enhanced system scalability: Query optimization mechanism and dynamic index update enable the system to flexibly expand as the amount of data grows, meeting the processing needs of large-scale traffic data in the future.

[0143] Real-time guarantee: The system can process data from various sensors in real time and respond quickly to user query requests, playing an important role in traffic management, accident warning and traffic planning.

[0144] Through the application of this embodiment, the intelligent transportation system can ensure efficient data storage, fast data query and flexible system expansion in a massive data environment, providing strong support for smart cities and modern transportation management.

[0145] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A hierarchical data storage method based on data format analysis, characterized in that: The following steps are involved: a. Automatically identify and parse the format of input data, use dynamic format recognition algorithm to classify different types of raw data, and automatically select storage method according to data format characteristics; b. Convert the parsed data into a unified standardized format, use an adaptive conversion module to standardize the data format, and extract features from the data, including but not limited to improving data storage efficiency through data segmentation, compression and aggregation; c. Multi-dimensional data stratification is performed based on data access frequency, data type, and timeliness, combined with data content characteristics. A three-tier storage strategy for hot data, cold data, and extremely cold data is adopted. Hot data is stored on high-performance storage media, cold data is stored on low-cost storage media, and extremely cold data is migrated to low-speed long-term archival storage media; d. Build a multi-dimensional dynamic index structure to support efficient data query and fast positioning. By adaptively adjusting the dimension and distribution of the index structure according to the data access mode, the query efficiency is improved. It supports complex data retrieval, range query and high-concurrency query. e. Introduce a data migration mechanism during the data storage process. When data access frequency, data life cycle or other external factors change, based on real-time monitoring system feedback, dynamically adjust the data storage level and automatically migrate hot and cold data to ensure optimal use of storage resources; f. Provide an efficient query mechanism based on query caching, parallel computing, and query path optimization, by caching popular data, processing multi-user query requests in parallel, dynamically optimizing query paths to minimize response time, and dynamically adjusting cache strategies based on query history data.

2. The hierarchical data storage method based on data format analysis according to claim 1 is characterized in that: The data format recognition in step a includes automatic recognition of the data format through metadata, file header information, extension or content features, and parsing using regular expressions or rule matching algorithms.

3. The hierarchical data storage method based on data format analysis according to claim 1, characterized in that: The feature extraction in step b uses convolutional neural networks for feature extraction of image data, Fourier transform for periodic feature extraction of time series data, and sensor data feature extraction based on data patterns.

4. The hierarchical data storage method based on data format analysis according to claim 1, characterized in that: The data migration mechanism in step c includes dynamic migration rules based on data access frequency. When the data access frequency is lower than a predetermined threshold, the data is migrated from a high-performance storage medium to a low-cost storage medium, and data integrity is ensured during the migration process through data consistency verification.

5. The hierarchical data storage method based on data format analysis according to claim 1, characterized in that: The multidimensional index structure in step d is a multidimensional B+ tree, R tree or KD tree, and the index structure supports range query, exact query and spatial query.

6. The hierarchical data storage method based on data format analysis according to claim 1, characterized in that: The data storage hierarchy management module in step e uses an LRU algorithm to dynamically adjust the data storage location according to the data access history, and adjusts its storage medium according to the access characteristics of hot and cold data.

7. The hierarchical data storage method based on data format analysis according to claim 1, characterized in that: The query response mechanism in step f includes a query cache mechanism, a parallel query mechanism and a dynamic query path optimization mechanism. The query cache mechanism reduces repeated queries by caching frequently accessed data; the parallel query mechanism processes query tasks in parallel through a distributed computing platform; and the dynamic query path optimization mechanism adjusts the query execution order according to the query mode and data distribution.

8. The hierarchical data storage method based on data format analysis according to claim 1, characterized in that: The query cache mechanism manages cache data through the LRU cache strategy, prioritizes caching of hot data, and regularly updates cache content; the parallel query mechanism decomposes and schedules query tasks through a distributed computing platform.

9. The hierarchical data storage method based on data format analysis according to claim 1, characterized in that: The data formats include time series data, image data, video data and sensor data, and the method can adapt to the storage and query requirements of different data types.

Images