ES cold and hot data read-write separation-based metadata management method

Abstract

The invention discloses a metadata management method based on ES cold and hot data read-write separation, and relates to the technical field of metadata management, comprising: marking an SSD disk as a hot node, and marking an SATA disk as a cold node; assigning label parameters in a node starting command to enable the nodes to join according to roles; creating an index template to divide a hot index and a cold index, writing a default route to a hot node, and reading an automatic route of a coordination node; setting an automatic index rolling strategy, writing an active index into a hot node, automatically establishing a new index when a threshold value is reached, and preferentially allocating resources to the hot index; an ILM migration rule is configured, and the overdue index is automatically migrated to a cold node; writing real-time metadata into a hot node, migrating overdue metadata to a cold node regularly or manually, and deleting cold node data according to a strategy; real-time data are read from the hot nodes, historical data are read from the cold nodes, and on-demand access is achieved. The performance, reliability and resource utilization rate of the ES cluster are remarkably improved on the premise that the cost is controllable.

Description

Technical Field

[0001] This invention relates to the field of metadata management technology, specifically a metadata management method based on the separation of hot and cold data read and write in Elasticsearch. Background Technology

[0002] Currently, metadata is mainly stored in Elasticsearch (ES) and used by various systems. During peak hours, the frequency of writes and queries is high, putting significant pressure on the cluster and often resulting in slow queries. Using replicas will halve the index write speed and double the disk usage; without replicas, if one node goes down, the entire cluster will be unable to write, with serious consequences.

[0003] To ensure ES's read and write performance, the official recommendation is to use SSDs for disks. However, ES aims to solve the problem of storing and retrieving massive amounts of data, which means a large amount of storage space is required. If SSDs are used for all data, cost will become a significant issue.

[0004] In a traditional Elasticsearch cluster, all nodes use the same configuration. However, Elasticsearch does not require consistency in node specifications, so each node can be of any specification, resulting in inconsistent performance among nodes in the cluster.

[0005] Furthermore, the index write and query speeds of an Elasticsearch cluster primarily depend on disk I / O speeds. The key to separating hot and cold data lies in matching appropriate storage media to different data types—hot data requires the high I / O performance of SSDs, while cold data can be stored on ordinary SATA disks to control costs. Using only SSDs, while ensuring performance, would be too expensive, and storing infrequently accessed cold data would be a waste of resources. Therefore, a hybrid storage solution combining ordinary SATA disks and SSDs achieves efficient resource utilization and significantly improves overall performance.

[0006] In practice, real-time metadata (within 7 days) can be stored on hot nodes (equipped with SSD disks), while historical metadata (7 days ago) can be stored on cold nodes (equipped with ordinary SATA disks). By leveraging Elasticsearch's own index lifecycle management features, the metadata of hot nodes can be automatically migrated to cold nodes over time, thereby achieving dynamic optimization of data storage.

[0007] In comparison, the existing storage method that only uses ordinary SATA disks has the lowest cost, but also the lowest read and write efficiency, making it difficult to meet the high-frequency access requirements of hot data; while storing all the indexes and copies of the day on SSD disks can achieve the highest read and write efficiency, but it will lead to excessive SSD disk usage, which may cause excessive costs. Summary of the Invention

[0008] This invention addresses the needs and shortcomings of current technological development by providing a metadata management method based on the separation of hot and cold data reads and writes in Elasticsearch.

[0009] The present invention provides a metadata management method based on Elasticsearch's hot and cold data read / write separation, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0010] A metadata management method based on Elasticsearch's hot and cold data read / write separation includes the following steps:

[0011] S1. Set node labels, marking SSD disk nodes as hot nodes and SATA disk nodes as cold nodes;

[0012] S2. Specify the label parameter in the node startup command to start the ES cluster, so that the node automatically joins the ES cluster according to the label role;

[0013] S3. Create an index template, divide the index into hot index and cold index, configure the default route to the hot node when writing, and the coordinating node to automatically route to the corresponding node when reading;

[0014] S4. Set an automatic index rolling strategy to ensure that the currently active index is always written to hot nodes. When the preset time, size or document number threshold is reached, a new index is automatically created, and priority settings are used to ensure that hot indexes are given priority in allocating computing resources.

[0015] S5. Set ILM migration rules to automatically migrate to cold nodes after the index age exceeds a preset number of days;

[0016] S6. Write real-time metadata to hot nodes and migrate metadata that has exceeded a specified number of days to cold nodes by timed or manual means.

[0017] S7. Based on actual needs, delete cold index data in cold nodes using a preset deletion strategy;

[0018] S8. Read real-time metadata from hot nodes and historical metadata from cold nodes to enable on-demand access.

[0019] Optionally, step S1 is performed to mark the SSD disk node as a hot node for storing real-time data that exceeds the access frequency; and to mark the SATA disk node as a cold node for storing historical data that is less frequent than the access frequency.

[0020] Node labels are defined through node configuration files or startup parameters, ensuring that the ES cluster can identify the storage media characteristics and functional positioning of nodes, providing a foundation for subsequent data routing and resource allocation.

[0021] Optionally, perform step S2 to start the ES cluster. After the nodes automatically join the ES cluster according to their label roles, the nodes register their own label information with the ES cluster. This enables the cluster management layer to identify the node roles based on the labels, providing a basis for index allocation and data migration.

[0022] Optionally, perform step S3 to create an index template, which specifically includes:

[0023] Define the mapping rules for hot and cold indexes;

[0024] Parameters that set the default route for write operations to hot nodes;

[0025] Configure the logic for coordinating nodes to automatically route to the corresponding nodes based on the index type during read operations.

[0026] Optionally, step S4 is executed, and the specific index auto-scrolling strategy set includes:

[0027] Set a time threshold, size threshold, or document count threshold;

[0028] When any of the time threshold, size threshold, or document number threshold is triggered, a new index is automatically created and the write target is switched.

[0029] By configuring the cluster to prioritize hot indexes over cold indexes, computing resources are prioritized for hot indexes that are accessed more frequently.

[0030] Optionally, step S5 is executed, and the ILM migration rules set include: when the index age exceeds a preset number of days, the index is automatically migrated from the hot node to the cold node;

[0031] After index migration, perform optimization operations: merge index segment files to reduce disk usage and improve query efficiency; reduce the number of shards to optimize cold data query performance; increase the number of replicas to balance the storage efficiency and availability of cold data.

[0032] Optionally, step S6 specifically includes:

[0033] Metadata generated in real time within a specified number of days is directly stored in the hot index of the hot node through the write interface;

[0034] For historical metadata that exceeds a specified number of days, the historical metadata can be transferred from the hot node to the cold index of the cold node through the ILM migration rules set by the index lifecycle management function or by manually triggering the migration script.

[0035] Optionally, step S7 is executed, and the preset deletion strategy includes:

[0036] Write a scheduled script based on business needs, and use the scheduled script to check the creation time of cold indexes in cold nodes.

[0037] Automatically delete cold index data in cold nodes that is older than one month to free up storage space.

[0038] Optionally, step S8 specifically includes:

[0039] Query requests for real-time metadata within a specified number of days are automatically routed to hot nodes;

[0040] Query requests for historical metadata that exceed a specified number of days are automatically routed to cold nodes.

[0041] The metadata management method based on Elasticsearch's hot and cold data read / write separation, as proposed in this invention, has the following advantages compared to existing technologies:

[0042] 1. This invention stores real-time metadata in hot nodes and historical metadata in cold nodes. It also utilizes the characteristics of Elasticsearch to migrate metadata from hot nodes to cold nodes based on time. This approach leverages the advantages of SSDs in write performance while reducing overall storage costs using SATA disks, achieving a trade-off between efficiency and cost optimization.

[0043] 2. The hot nodes of this invention use SSD disks to ensure high IO performance for real-time metadata accessed at high frequencies, solving the problem of slow read and write speeds of all-SATA disks; the cold nodes use ordinary SATA disks to store historical data accessed at low frequencies, avoiding the high cost of investing in all-SSD disks.

[0044] 3. This invention ensures that computing resources are prioritized for frequently accessed hot data by setting hot index priority, reducing resource contention during peak hours; automatic index rolling and ILM migration rules realize automated management of the data lifecycle, avoiding performance degradation caused by an excessively large single index and improving the overall cluster response speed.

[0045] 4. By increasing the number of replicas, the cold nodes of this invention ensure redundant backup of historical metadata while controlling costs, thereby reducing the risk of data loss. At the same time, the cold data structure is optimized through operations such as sharding shrinkage and segment merging, balancing reliability and storage efficiency.

[0046] 5. Based on the differentiated configuration of node tags, this invention enables SSD disks to focus on high-frequency read and write scenarios, and SATA disks to focus on large-capacity storage, thus avoiding resource waste; the automatic migration and deletion strategy reduces manual intervention and realizes the full lifecycle automation of data from "hot → cold → deletion", improving operation and maintenance efficiency. Attached Figure Description

[0047] Appendix Figure 1 This is a schematic diagram illustrating the implementation principle of the method of this invention. Detailed Implementation

[0048] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.

[0049] The terms used in the embodiments are explained as follows:

[0050] Metadata refers to management and control data that defines and describes information such as the structure, content, quality, source, and management method of data, and is used to support the search, understanding, use, and management of data.

[0051] ES, short for Elasticsearch, is a distributed, open-source real-time search and analytics engine built on Apache Lucene and designed to provide fast, scalable, and high-performance search solutions.

[0052] A SATA hard disk (HDD) is a storage device that combines a traditional mechanical hard disk with the SATA (Serial ATA) interface standard. It consists of rotating magnetic platters, a moving read / write head assembly, and a precision mechanical structure. It connects to a computer system through the SATA interface and features stable interface transmission speed and strong compatibility.

[0053] An SSD (Solid State Drive) is a type of hard drive made using an array of solid-state electronic storage chips. It uses NAND flash memory as its core storage medium and reads and writes data via electronic signals rather than mechanical movement, offering advantages such as fast read / write speeds, good shock resistance, and low power consumption.

[0054] I / O speed, short for Input / Output Speed, refers to the rate at which data is transferred between a computer system and storage devices, and is a key indicator of storage performance. In computer architecture, I / O (input / output) broadly refers to all data transfer processes between internal computer components (such as the CPU and memory) or between the computer and external devices (such as disks and networks). The speed of I / O directly affects system response time and data processing efficiency, and is one of the major bottlenecks in modern computing performance.

[0055] Hot data refers to the latest and most frequently accessed data, which usually requires fast read and write responses and is commonly found in scenarios such as real-time transactions and high-frequency queries.

[0056] A hot node is a physical or logical node specifically designed to carry hot data. It needs to have high-performance storage and processing capabilities to meet the high-frequency access requirements of hot data.

[0057] Hot indexes refer to index units assigned to hot data nodes. They are characterized by high query frequency and high write pressure, and usually need to be optimized to improve real-time response speed.

[0058] Cold data refers to historical data that is accessed infrequently. It has lower requirements for real-time read / write performance but needs to be stored reliably for a long time to meet the needs of archiving, auditing, etc.

[0059] Cold nodes refer to physical or logical nodes specifically designed to carry cold data. They prioritize storage capacity and cost-effectiveness and have lower requirements for real-time processing performance.

[0060] A cold index refers to an index unit allocated to a cold data node. It is characterized by low query frequency and virtually no write operations, and is usually stored in compressed form to save space.

[0061] Example:

[0062] Reference Appendix Figure 1 This embodiment proposes a metadata management method based on the separation of hot and cold data read and write in Elasticsearch, which includes the following steps:

[0063] S1. Set node labels to mark SSD disk nodes as hot nodes and SATA disk nodes as cold nodes.

[0064] SSD disk nodes are marked as hot nodes to store real-time data that is accessed more frequently than the specified frequency; SATA disk nodes are marked as cold nodes to store historical data that is accessed less frequently than the specified frequency.

[0065] Node labels are defined through node configuration files or startup parameters, ensuring that the ES cluster can identify the storage media characteristics and functional positioning of nodes, providing a foundation for subsequent data routing and resource allocation.

[0066] S2. Specify the label parameter in the node startup command to start the ES cluster, so that the node will automatically join the ES cluster according to the label role.

[0067] After nodes are automatically added to the Elasticsearch cluster by their tagged roles, the nodes register their own tag information with the Elasticsearch cluster, enabling the cluster management layer to identify node roles based on tags, providing a basis for index allocation and data migration.

[0068] S3. Create an index template, divide the index into hot index and cold index, configure the default route to the hot node when writing, and the coordinating node to automatically route to the corresponding node when reading.

[0069] This step of creating the index template specifically includes:

[0070] Define the mapping rules for hot and cold indexes;

[0071] Parameters that set the default route for write operations to hot nodes;

[0072] Configure the logic for coordinating nodes to automatically route to the corresponding nodes based on the index type during read operations.

[0073] S4. Set an automatic index rolling strategy to ensure that the currently active index is always written to hot nodes. When a preset time, size, or document number threshold is reached, a new index is automatically created, and priority settings are used to ensure that hot indexes are given priority in allocating computing resources.

[0074] The specific index auto-scrolling strategy settings include:

[0075] Set a time threshold, size threshold, or document count threshold;

[0076] When any of the time threshold, size threshold, or document number threshold is triggered, a new index is automatically created and the write target is switched.

[0077] By configuring the cluster to prioritize hot indexes over cold indexes, computing resources are prioritized for hot indexes that are accessed more frequently.

[0078] S5. Set ILM migration rules to automatically migrate to cold nodes after the index age exceeds a preset number of days.

[0079] The configured ILM migration rules include: automatically migrating the index from the hot node to the cold node when the index age exceeds a preset number of days.

[0080] Perform optimization operations after index migration:

[0081] Merging index segment files reduces disk usage and improves query efficiency;

[0082] Reduce the number of shards to optimize cold data query performance;

[0083] Increase the number of replicas to balance the storage efficiency and availability of cold data.

[0084] S6. Write real-time metadata to hot nodes and migrate metadata that has exceeded a specified number of days to cold nodes via scheduled or manual methods; specifically including:

[0085] Metadata generated in real time within a specified number of days is directly stored in the hot index of the hot node through the write interface;

[0086] For historical metadata that exceeds a specified number of days, the historical metadata can be transferred from the hot node to the cold index of the cold node through the ILM migration rules set by the index lifecycle management function or by manually triggering the migration script.

[0087] S7. Based on actual needs, delete cold index data in cold nodes using a preset deletion strategy.

[0088] The preset deletion strategies include:

[0089] Write a scheduled script based on business needs, and use the scheduled script to check the creation time of cold indexes in cold nodes.

[0090] Automatically delete cold index data in cold nodes that is older than one month to free up storage space.

[0091] S8. Read real-time metadata from hot nodes and historical metadata from cold nodes to achieve on-demand access, specifically including:

[0092] Query requests for real-time metadata within a specified number of days are automatically routed to hot nodes;

[0093] Query requests for historical metadata that exceed a specified number of days are automatically routed to cold nodes.

[0094] In summary, the metadata management method based on hot and cold data read / write separation of ES, as proposed in this invention, leverages the characteristic that ES nodes do not have specification consistency requirements. It allows for the simultaneous use of SSD disks and traditional SATA disks to run ES processes. Through configuration and indexing, real-time metadata is stored in hot nodes, while historical metadata is stored in cold nodes. Furthermore, by utilizing the inherent characteristics of ES, metadata from hot nodes is migrated to cold nodes based on time, thus achieving full utilization of resources and significantly improving performance.

[0095] The above specific examples illustrate the principles and implementation methods of the present invention in detail. These embodiments are merely for the purpose of helping to understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made to the present invention by those skilled in the art without departing from the principles of the present invention should fall within the patent protection scope of the present invention.

Claims

1. A metadata management method based on Elasticsearch's cold and hot data read / write separation, characterized in that, Includes the following steps: S1. Set node labels, marking SSD disk nodes as hot nodes and SATA disk nodes as cold nodes; S2. Specify the label parameter in the node startup command to start the ES cluster, so that the node automatically joins the ES cluster according to the label role; S3. Create an index template, divide the index into hot index and cold index, configure the default route to the hot node when writing, and the coordinating node to automatically route to the corresponding node when reading; S4. Set an automatic index rolling strategy to ensure that the currently active index is always written to hot nodes. When the preset time, size or document number threshold is reached, a new index is automatically created, and priority settings are used to ensure that hot indexes are given priority in allocating computing resources. S5. Set ILM migration rules to automatically migrate to cold nodes after the index age exceeds a preset number of days; S6. Write real-time metadata to hot nodes and migrate metadata that has exceeded a specified number of days to cold nodes by timed or manual means. S7. Based on actual needs, delete cold index data in cold nodes using a preset deletion strategy; S8. Read real-time metadata from hot nodes and historical metadata from cold nodes to enable on-demand access.

2. The metadata management method based on ES cold and hot data read / write separation according to claim 1, characterized in that, Perform step S1 to mark the SSD disk node as a hot node for storing real-time data that exceeds the access frequency; mark the SATA disk node as a cold node for storing historical data that is less frequent than the access frequency. Node labels are defined through node configuration files or startup parameters, ensuring that the ES cluster can identify the storage media characteristics and functional positioning of nodes, providing a foundation for subsequent data routing and resource allocation.

3. The metadata management method based on ES cold and hot data read / write separation according to claim 1, characterized in that, Execute step S2 to start the ES cluster. After the nodes automatically join the ES cluster according to their label roles, the nodes register their own label information with the ES cluster. This enables the cluster management layer to identify the node roles based on the labels, providing a basis for index allocation and data migration.

4. The metadata management method based on ES cold and hot data read / write separation according to claim 1, characterized in that, Execute step S3 to create an index template, which specifically includes: Define the mapping rules for hot and cold indexes; Parameters that set the default route for write operations to hot nodes; Configure the logic for coordinating nodes to automatically route to the corresponding nodes based on the index type during read operations.

5. A metadata management method based on ES cold and hot data read / write separation according to claim 1, characterized in that, The specific steps for setting the index auto-scrolling strategy in step S4 include: Set a time threshold, size threshold, or document count threshold; When any of the time threshold, size threshold, or document number threshold is triggered, a new index is automatically created and the write target is switched. By configuring the cluster to prioritize hot indexes over cold indexes, computing resources are prioritized for hot indexes that are accessed more frequently.

6. The metadata management method based on ES cold and hot data read / write separation according to claim 1, characterized in that, Execute step S5, and set the ILM migration rules including: when the index age exceeds the preset number of days, automatically migrate the index from the hot node to the cold node; After index migration, perform optimization operations: merge index segment files to reduce disk usage and improve query efficiency; reduce the number of shards to optimize cold data query performance; increase the number of replicas to balance the storage efficiency and availability of cold data.

7. A metadata management method based on ES cold and hot data read / write separation according to claim 1, characterized in that, Step S6 specifically includes: Metadata generated in real time within a specified number of days is directly stored in the hot index of the hot node through the write interface; For historical metadata that exceeds a specified number of days, the historical metadata can be transferred from the hot node to the cold index of the cold node through the ILM migration rules set by the index lifecycle management function or by manually triggering the migration script.

8. A metadata management method based on ES cold and hot data read / write separation according to claim 1, characterized in that, Execute step S7, the preset deletion strategy includes: Write a scheduled script based on business needs, and use the scheduled script to check the creation time of cold indexes in cold nodes. Automatically delete cold index data in cold nodes that is older than one month to free up storage space.

9. A metadata management method based on ES cold and hot data read / write separation according to claim 1, characterized in that, Step S8 specifically includes: Query requests for real-time metadata within a specified number of days are automatically routed to hot nodes; Query requests for historical metadata that exceed a specified number of days are automatically routed to cold nodes.

Images