A data hot and cold separation storage method and device and related equipment

Abstract

This invention discloses a method, apparatus, and related equipment for separating hot and cold data storage. The method includes: storing collected log data into an index corresponding to Elasticsearch; the index identifier corresponding to the log data includes a date and type; determining whether the log data is within a preset validity period based on the index identifier; if not, obtaining Elasticsearch parameter information and determining whether the Elasticsearch parameter information satisfies at least one first business condition; if yes, migrating at least a portion of the log data not within the preset validity period to ClickHouse for storage based on the highest priority first business condition satisfied. This method can significantly improve storage efficiency and query performance, enhance log data processing efficiency and storage economy, and provide more robust data support for network security protection.

Description

Technical Field

[0001] This invention relates to the field of secure information technology, and in particular to a method, apparatus, and related equipment for separating cold and hot data storage. Background Technology

[0002] With the rapid development of information technology, cybersecurity threats are becoming increasingly complex and diverse, with attack methods constantly evolving. To effectively address these threats, enterprises and organizations need to collect, store, and analyze massive amounts of attack log data. This log data not only records detailed information about network activity but also contains potential attack behaviors and anomaly patterns, making it crucial for the detection, response, and attribution of security incidents. Summary of the Invention

[0003] To increase the options for cold and hot data separation storage methods and improve data storage efficiency, embodiments of the present invention provide a data cold and hot separation storage method, apparatus, and related equipment.

[0004] In a first aspect, embodiments of the present invention provide a method for separating hot and cold data storage, comprising:

[0005] The collected log data is stored in the corresponding index in Elasticsearch; the index identifier for the log data includes the date and type.

[0006] Based on the index identifier corresponding to the log data, determine whether the log data is within the preset time period;

[0007] If not, obtain the parameter information of ElasticSearch, and based on the first business rule base, determine whether the parameter information of ElasticSearch meets at least one first business condition; the first business rule base includes multiple first business rules sorted by priority; the parameter information of ElasticSearch includes log data access frequency, system load, storage capacity and data importance;

[0008] If so, based on the highest priority first business condition, at least some of the log data that is not within the preset time limit will be migrated to ClickHouse for storage.

[0009] In one or more optional embodiments, the data cold and hot separation storage method further includes:

[0010] Obtain the parameter information of ClickHouse, and based on the second business rule base, determine whether the parameter information of ClickHouse satisfies at least one second business rule; the second business rule base includes multiple second business rules sorted by priority; the parameter information of ClickHouse includes query response time and cold data access frequency;

[0011] If so, migrate the log data that meets the second business rule with the highest priority to ElasticSearch for storage.

[0012] In one or more optional embodiments, the step of migrating at least a portion of log data that is not within a preset time limit to ClickHouse for storage based on the highest priority first business condition includes:

[0013] When the access frequency of log data in Elasticsearch is lower than a preset frequency threshold, the log data is identified as cold data and migrated to ClickHouse for storage.

[0014] or,

[0015] If the log data in Elasticsearch does not contain the preset keyword, the log data is identified as cold data and migrated to ClickHouse for storage.

[0016] or,

[0017] When the system load of ElasticSearch exceeds a preset load threshold, the preset number of log data collected earliest in ElasticSearch will be identified as cold data and migrated to ClickHouse for storage.

[0018] or,

[0019] When the storage capacity of ElasticSearch exceeds a preset capacity threshold, the preset number of log data collected earliest in ElasticSearch will be identified as cold data and migrated to ClickHouse for storage.

[0020] In one or more optional embodiments, migrating log data from ElasticSearch to ClickHouse for storage includes:

[0021] A real-time synchronization strategy was adopted to migrate log data from Elasticsearch to ClickHouse for storage;

[0022] or,

[0023] A delayed synchronization strategy was adopted to migrate log data from ElasticSearch to ClickHouse for storage.

[0024] In one or more optional embodiments, when multiple first service conditions have the same priority, migration is performed randomly based on any one of the first service conditions.

[0025] In one or more optional embodiments, migrating log data that meets the highest priority second business condition to Elasticsearch for storage includes:

[0026] When the query response time for cold data in ClickHouse exceeds a preset time threshold, the cold data is reclassified as hot data and migrated to the corresponding index in ElasticSearch.

[0027] or,

[0028] When the access frequency of cold data in ClickHouse exceeds a preset frequency threshold, the cold data is reclassified as hot data and migrated to the corresponding index in ElasticSearch.

[0029] Secondly, embodiments of the present invention provide a data cold and hot separation storage device, comprising:

[0030] The storage module is used to store the collected log data into the corresponding index in ElasticSearch;

[0031] The first judgment module is used to determine whether the log data is within a preset time period based on the index where the log data is located;

[0032] The second judgment module is used to obtain the parameter information of the ElasticSearch and, based on the first business rule base, determine whether the parameter information of the ElasticSearch meets at least one first business condition.

[0033] The migration module is used to migrate at least a portion of log data that is not within the preset time limit to ClickHouse storage based on the highest priority first business condition.

[0034] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data cold and hot separation storage method as described in the first aspect.

[0035] Fourthly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the data cold and hot separation storage method as described in the first aspect.

[0036] Fifthly, embodiments of the present invention provide a computer program product containing instructions that, when the computer program product is run on a computer device, cause the computer device to execute the data cold and hot separation storage method as described in the first aspect.

[0037] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:

[0038] The data cold / hot separation storage method provided in this invention first filters out log data that is not within its expiration period based on the log data's expiration date. Then, based on the acquired parameter information and a first business rule base, it filters out log data that is not within its expiration date but meets the highest priority first business condition. This portion of data is then migrated to ClickHouse for storage, reducing the load on Elasticsearch. This ensures that more valuable log data is stored in Elasticsearch, guaranteeing that all data retained in Elasticsearch is high-value. Combining the advantages of both Elasticsearch and ClickHouse, by storing high-value log data in Elasticsearch and lower-value log data in ClickHouse, storage efficiency and query performance can be significantly improved, enhancing log data processing efficiency and storage economy, and providing stronger data support for network security protection.

[0039] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0040] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0041] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0042] Figure 1 This is a flowchart illustrating the data cold and hot separation storage method provided in an embodiment of the present invention;

[0043] Figure 2 This is a structural block diagram of the data cold and hot separation storage device provided in the embodiments of the present invention;

[0044] Figure 3 This is a structural block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0045] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0046] The inventors discovered that with the expansion of network scale and the frequency of attacks, the size of attack log data is also growing explosively. Traditional log management systems often employ single storage and query solutions, such as relational databases or file storage systems. These solutions face numerous challenges when handling large-scale data. First, storage costs are high because sufficient storage space needs to be provided for all log data, and the persistence and availability of the data must be ensured. Second, query efficiency is low, especially when faced with complex queries and real-time analysis requirements, traditional systems often struggle to respond quickly.

[0047] Based on this, embodiments of the present invention provide a method, apparatus, and related equipment for separating hot and cold data storage, which will be described in detail below through specific embodiments.

[0048] Example 1

[0049] This invention provides a method for separating hot and cold data storage, referring to... Figure 1 As shown, it includes:

[0050] S101: Store the collected log data into the corresponding index in Elasticsearch; the index identifier corresponding to the log data includes the date and type;

[0051] S102: Based on the index identifier corresponding to the log data, determine whether the log data is within the preset time period. If yes, determine the corresponding log data as hot data and retain it in the corresponding ElasticSearch index. If no, proceed to step S103.

[0052] S103: Obtain ElasticSearch parameter information. Based on the first business rule base, determine whether the ElasticSearch parameter information meets at least one first business condition. If not, determine the corresponding log data as hot data and retain it in the corresponding ElasticSearch index. If yes, execute step S104. The first business rule base includes multiple first business conditions sorted by priority. The ElasticSearch parameter information includes log data access frequency, system load, storage capacity, and data importance.

[0053] S104: Based on the highest priority first business condition, migrate at least some log data that is not within the preset time limit to ClickHouse for storage.

[0054] In this embodiment of the invention, log data is collected from different security devices, and different security devices correspond to different log types. Therefore, before step S101, a corresponding index can be created in Elasticsearch based on the log data type and generation date. For example, the index prefix for log data collected by WAF is waf, and the index prefix for log data collected by IPS is ips. The date of log generation should also be added after the index prefix. For example, the complete index identifier for WAF log data generated on August 23, 2024 is waf-2024.08.23. The index identifier can be used for subsequent timeliness determination of log data. When determining timeliness, the date after the hyphen in the index name, combined with the expiration time range of the timeliness restriction, can be used to determine whether this part of the log data is invalid.

[0055] In an optional implementation, before storing log data in Elasticsearch, the corresponding log ID needs to be recorded in Redis. When storing log data in Elasticsearch, the corresponding log data first needs to be retrieved from Elasticsearch based on the log ID stored in Redis, and then stored in the index corresponding to the hot data in Elasticsearch.

[0056] In this embodiment of the invention, step S101: storing the collected log data into the corresponding ElasticSearch index. Specifically, this may include: collecting the log data into text files via the Syslog protocol and storing them in a specified location, transmitting them using different Filebeats, and storing them in the corresponding ElasticSearch index. Before transmitting the log data to ElasticSearch, all the raw log data needs to be parsed into JSON format using the Filebeat filter plugin Grok regular expressions. The JSON format also includes a message field to store the original log data. Optionally, the specified location may be a caching module, etc.

[0057] The inventors discovered that ElasticSearch supports complex query syntax, enabling it to quickly retrieve relevant information from massive amounts of data, making it ideal for handling hot data. ClickHouse, on the other hand, employs columnar storage and vectorized query technology, efficiently handling aggregation, sorting, and filtering operations on large-scale data, making it ideal for handling cold data.

[0058] In this embodiment of the invention, the preset validity period can be reasonably set as needed, for example, it can be set to 3 days. Step S102: Based on the index identifier corresponding to the log data, determine whether the log data is within the preset validity period, which may specifically include:

[0059] Based on the log generation time information in the index identifier corresponding to the log data, determine whether the log data is within the preset validity period.

[0060] In this embodiment of the invention, the parameter information of ElasticSearch includes:

[0061] (1) Data Access Frequency: This refers to the number of times log data in Elasticsearch is accessed within a certain time window, usually measured by the number of accesses per hour or per day. If the access frequency of log data in Elasticsearch exceeds a preset frequency threshold, it indicates that this part of the log data has a high access frequency and can be identified as hot data, suitable for storage in the Elasticsearch index; if the access frequency of log data in Elasticsearch does not exceed the preset frequency threshold, it indicates that the access frequency is low and can be identified as cold data, suitable for migration to ClickHouse for storage to reduce the load on Elasticsearch. It should be noted that in Elasticsearch, directly determining the number of times a log data is queried does not have a ready-made counter field like in relational databases. Elasticsearch is a Lucene-based search engine that focuses primarily on indexing and search efficiency, rather than the number of queries for each piece of data. Therefore, we need to record the query frequency of logs at the application layer and use a unique identifier in Elasticsearch. When a log is queried, the access frequency is recorded by combining the unique identifier with the number of accesses.

[0062] (2) System load: This reflects the current processing capacity utilization of the Elasticsearch system, including CPU utilization, I / O and other resource usage. When the system load is high, Elasticsearch may not be able to respond to new requests quickly, so some data needs to be migrated to ClickHouse to reduce the pressure; while when the system load is low, more hot data can be kept in Elasticsearch.

[0063] (3) Storage capacity: The current storage space usage of ElasticSearch. Storage capacity affects whether the system can continue to store new data. If ElasticSearch storage is close to full capacity, you can choose to migrate some hot data to ClickHouse to free up space.

[0064] (4) Data Importance: Some data can be marked as high value (high importance) or low value (low importance) based on business rules. When a log contains a preset keyword (e.g., the keyword "rm"), it indicates that the log is a high-risk log with high importance. Even if the access frequency is low, it can be judged as hot data and is suitable for storage in Elasticsearch. If the log does not contain a keyword, it indicates that the log is of low importance and is suitable for migration to ClickHouse for storage.

[0065] In this embodiment of the invention, the first business rule base includes multiple first business rules ordered by priority, each first business rule corresponding to parameter information from Elasticsearch. Each first business rule includes a first business condition and a first operation. If the first business condition in a given first business rule is met, the corresponding first operation in that first business rule can be executed. Specifically, the multiple first business rules include, but are not limited to:

[0066] (1) First business condition: The access frequency of log data in ElasticSearch is lower than the preset frequency threshold;

[0067] First step: Determine this portion of log data as cold data and migrate it to ClickHouse for storage.

[0068] (2) First business condition: The log data in ElasticSearch does not contain the preset keyword (indicating that the log data is of low importance);

[0069] First step: Determine this portion of log data as cold data and migrate it to ClickHouse for storage.

[0070] (3) First business condition: The system load of ElasticSearch is higher than the preset load threshold;

[0071] The first step is to identify the earliest collected, preset number of log data in Elasticsearch as cold data and migrate them to ClickHouse for storage.

[0072] (4) First business condition: The storage capacity of ElasticSearch is higher than the preset capacity threshold;

[0073] The first step is to identify the earliest collected, preset number of log data in Elasticsearch as cold data and migrate them to ClickHouse for storage.

[0074] It should be noted that both hot data and cold data are log data, used only to distinguish between log data stored in Elasticsearch or ClickHouse. That is, log data stored in Elasticsearch is hot data, and log data stored in ClickHouse is cold data.

[0075] In an optional implementation, step S103: Obtain the parameter information of the ElasticSearch, and based on the first business rule base, determine whether the parameter information of the ElasticSearch satisfies at least one first business condition, specifically:

[0076] Determine whether the parameter information of the ElasticSearch satisfies at least one of the following first business conditions:

[0077] (1) The access frequency of log data in ElasticSearch is lower than the preset frequency threshold;

[0078] (2) The log data in Elasticsearch does not contain preset keywords;

[0079] (3) The system load of ElasticSearch is higher than the preset load threshold;

[0080] (4) The storage capacity of ElasticSearch is higher than the preset capacity threshold.

[0081] In an optional implementation, step S104: Based on the highest priority first business condition, at least a portion of the log data that is not within the preset time limit is migrated to ClickHouse for storage, which may specifically include:

[0082] When the access frequency of log data in ElasticSearch is lower than the preset frequency threshold, this part of the log data that is not within the preset time period is identified as cold data and migrated to ClickHouse for storage.

[0083] or,

[0084] When the log data in Elasticsearch does not contain the preset keywords, this part of the log data that is not within the preset time period is judged as cold data and migrated to ClickHouse for storage;

[0085] or,

[0086] When the ElasticSearch system load exceeds the preset load threshold, log data that is not within the preset time period is identified as cold data and migrated to ClickHouse for storage.

[0087] or,

[0088] When the storage capacity of ElasticSearch exceeds the preset capacity threshold, log data that is not within the preset time period is identified as cold data and migrated to ClickHouse for storage.

[0089] It should be noted that the priority order of the first business condition can be reasonably set according to actual needs. Furthermore, if the parameter information of ElasticSearch satisfies two or more first business conditions with the same priority, any first business condition can be randomly selected, and the corresponding first operation can be executed according to the corresponding first business rule.

[0090] The data hot / cold separation storage method provided in this invention, based on the different characteristics of log data such as timeliness, access frequency, and importance, and combined with parameters such as system load and storage capacity, divides log data into two parts: hot data and cold data. By storing hot data and cold data in storage systems suitable for their characteristics respectively, storage efficiency and query performance can be significantly improved. Storing high-value log data with high freshness and importance in Elasticsearch can effectively solve the problem of traditional systems' inability to respond quickly to complex queries and real-time analysis needs, greatly improving query efficiency. Migrating a large amount of log data with low freshness and low importance from Elasticsearch can effectively solve the problems of high storage costs and insufficient storage space, ensuring the persistence and availability of log data.

[0091] In optional implementations, log data from Elasticsearch can be migrated to ClickHouse using either a real-time synchronization strategy or a delayed synchronization strategy. Specifically, a real-time synchronization strategy synchronizes log data stored in Elasticsearch to ClickHouse immediately to ensure data consistency. This strategy is suitable for data requiring high real-time performance, such as real-time monitoring data, to prevent inconsistencies between hot and cold data. A delayed synchronization strategy synchronizes log data from Elasticsearch to ClickHouse after a certain period, typically through batch tasks executed during off-peak hours. This strategy is suitable for situations where business requirements for log data consistency are lower, such as migrating expired log data or migrating data when the system load is too high.

[0092] In optional implementations, security information management platforms typically have functional modules related to log analysis and alerts, such as an alert module. Based on the alert module, when an alert is generated, the security event corresponding to the alert, as well as the logs associated with the alert, can also be identified as hot data and stored in Elasticsearch.

[0093] In an optional implementation, the data cold and hot separation storage method provided in this embodiment of the invention further includes:

[0094] S201: Obtain the parameter information of ClickHouse, and based on the second business rule base, determine whether the parameter information of ClickHouse satisfies at least one second business rule; the second business rule base includes multiple second business rules sorted by priority; the parameter information of ClickHouse includes query response time and cold data access frequency. If yes, proceed to step S202.

[0095] S202: Migrate the log data that meets the second business rule with the highest priority to Elasticsearch for storage.

[0096] In this embodiment of the invention, the parameter information of ClickHouse includes:

[0097] (1) Query response time: The response time for users to query log data. When the query response time exceeds the preset time threshold, this part of the data can be migrated to Elasticsearch to optimize query speed.

[0098] (2) Access frequency of cold data: This refers to the number of times the log data in ClickHouse is accessed within a certain time window, usually measured by the number of accesses per hour or per day. When cold data in ClickHouse is accessed frequently, it indicates that this part of the cold data has high value. In this case, the cold data can be migrated back to ElasticSearch.

[0099] In this embodiment of the invention, the second business rule base includes multiple second business rules ordered by priority, and each second business rule corresponds to the parameter information of a ClickHouse. Each second business rule includes a second business condition and a second operation. If the second business condition in a certain second business rule is met, the corresponding second operation in that second business rule can be executed. Specifically, the multiple second business rules include, but are not limited to:

[0100] (1) Second business condition: The query response time for cold data in ClickHouse exceeds the preset time threshold;

[0101] The second step is to reclassify this cold data as hot data and migrate it to the corresponding index in Elasticsearch.

[0102] (2) Second business condition: The access frequency of cold data in ClickHouse is higher than the preset frequency threshold;

[0103] The second step is to reclassify this cold data as hot data and migrate it to the corresponding index in Elasticsearch.

[0104] In an optional implementation, step 202: migrating log data that meets the highest priority second business condition to Elasticsearch for storage, specifically includes:

[0105] When the query response time for cold data in ClickHouse exceeds a preset time threshold, the cold data is reclassified as hot data and migrated to the corresponding index in ElasticSearch.

[0106] or,

[0107] When the access frequency of cold data in ClickHouse exceeds a preset frequency threshold, the cold data is reclassified as hot data and migrated to the corresponding index in ElasticSearch.

[0108] In an optional embodiment, when migrating log data from ElasticSearch to Clickhouse, ElasticSearch log indexes with the corresponding date suffix can be deleted as a cleanup of log data that has been converted from hot to cold.

[0109] The data cold and hot separation storage method provided in this embodiment of the invention migrates log data in ClickHouse back to ElasticSearch under specific circumstances (e.g., when the access frequency of cold data is too high), further ensuring that high-value data can be stored in ElasticSearch, thereby guaranteeing high query efficiency.

[0110] The data cold / hot separation storage method provided in this invention first filters out log data that is not within its expiration period based on the log data's expiration date. Then, based on the acquired parameter information and a first business rule base, it filters out log data that is not within its expiration date but meets the highest priority first business condition. This portion of data is then migrated to ClickHouse for storage, reducing the load on Elasticsearch. This ensures that more valuable log data is stored in Elasticsearch, guaranteeing that all data retained in Elasticsearch is high-value. Combining the advantages of both Elasticsearch and ClickHouse, by storing high-value log data in Elasticsearch and lower-value log data in ClickHouse, storage efficiency and query performance can be significantly improved, enhancing log data processing efficiency and storage economy, and providing stronger data support for network security protection.

[0111] Example 2

[0112] Based on the same inventive concept, embodiments of the present invention also provide a data cold and hot separation storage device, referring to... Figure 2 As shown, it includes:

[0113] The storage module 101 is used to store the collected log data into the corresponding index in ElasticSearch;

[0114] The first judgment module 102 is used to determine whether the log data is within a preset time period based on the index where the log data is located;

[0115] The second judgment module 103 is used to obtain the parameter information of the ElasticSearch and, based on the first business rule base, determine whether the parameter information of the ElasticSearch meets at least one first business condition.

[0116] The migration module 104 is used to migrate at least a portion of the log data that is not within the preset time limit to ClickHouse for storage, based on the first business condition with the highest priority.

[0117] Example 3

[0118] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data cold and hot separation storage method as described in Embodiment 1.

[0119] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires; a portable computer disk drive; a hard disk drive; a random access memory (RAM); a read-only memory (ROM); an erasable programmable read-only memory (EPROM); a register; a hard disk drive; an optical fiber; a portable compact disc read-only memory (CD-ROM); an optical storage device; a magnetic storage device; or any suitable combination thereof; or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In the embodiments of this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0120] Example 4

[0121] Based on the same inventive concept, this embodiment of the invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the data cold and hot separation storage method as described in Embodiment 1.

[0122] Figure 3 A possible structural diagram of the computer device involved in the above embodiments is shown. The computer device includes a processor 1002 and a communication interface 1003. The processor 1002 is used to control and manage the operation of the computer device, for example, executing the alarm association method described above, and / or other processes of the technology described herein. The communication interface 1003 is used to support communication between the computer device and other network entities, for example, executing the steps performed by the communication unit 902 described above. The computer device may also include a memory 1001 and a bus 1004, the memory 1001 being used to store the program code and data of the computer device.

[0123] The memory 1001 may be a memory in a computer device, and the memory may include: volatile memory, such as random access memory; the memory may also include: non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; the memory may also include: a combination of the above types of memory.

[0124] The processor 1002 described above can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0125] Bus 1004 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 1004 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0126] Example 5

[0127] Based on the same inventive concept, this embodiment of the invention also provides a computer program product containing instructions, characterized in that, when the computer program product is run on a computer device, it causes the computer device to execute the data cold and hot separation storage method as described in Embodiment 1.

[0128] The principles by which the above-described apparatus, client, medium, related equipment, and system in this embodiment solve the problem are similar to those of the aforementioned method. Therefore, their implementation can refer to the implementation of the aforementioned method, and repeated details will not be repeated.

[0129] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0130] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0133] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for separating hot and cold data storage, characterized in that, include: The collected log data is stored in the corresponding index in Elasticsearch; The index identifier corresponding to the log data includes the date and type; Based on the index identifier corresponding to the log data, determine whether the log data is within the preset time period; If not, obtain the parameter information of ElasticSearch, and based on the first business rule base, determine whether the parameter information of ElasticSearch meets at least one first business condition; the first business rule base includes multiple first business rules sorted by priority; the parameter information of ElasticSearch includes log data access frequency, system load, storage capacity and data importance; If so, based on the highest priority first business condition, at least some of the log data that is not within the preset time limit will be migrated to ClickHouse for storage.

2. The data cold and hot separation storage method according to claim 1, characterized in that, Also includes: Obtain the parameter information of ClickHouse, and based on the second business rule base, determine whether the parameter information of ClickHouse satisfies at least one second business rule; the second business rule base includes multiple second business rules sorted by priority; the parameter information of ClickHouse includes query response time and cold data access frequency; If so, migrate the log data that meets the second business rule with the highest priority to ElasticSearch for storage.

3. The data cold and hot separation storage method according to claim 1 or 2, characterized in that, The step of migrating at least a portion of log data that is not within a preset time limit to ClickHouse storage based on the highest priority first business condition includes: When the access frequency of log data in Elasticsearch is lower than a preset frequency threshold, the log data is identified as cold data and migrated to ClickHouse for storage. or, If the log data in Elasticsearch does not contain the preset keyword, the log data is identified as cold data and migrated to ClickHouse for storage. or, When the system load of ElasticSearch exceeds a preset load threshold, the preset number of log data collected earliest in ElasticSearch will be identified as cold data and migrated to ClickHouse for storage. or, When the storage capacity of ElasticSearch exceeds a preset capacity threshold, the preset number of log data collected earliest in ElasticSearch will be identified as cold data and migrated to ClickHouse for storage.

4. The data cold and hot separation storage method according to claim 3, characterized in that, Migrate log data from Elasticsearch to ClickHouse for storage, including: A real-time synchronization strategy was adopted to migrate log data from Elasticsearch to ClickHouse for storage; or, A delayed synchronization strategy was adopted to migrate log data from ElasticSearch to ClickHouse for storage.

5. The data cold and hot separation storage method according to claim 2, characterized in that, When multiple first business conditions have the same priority, migration is performed randomly based on any one of the first business conditions.

6. The data cold and hot separation storage method according to claim 2, characterized in that, The process of migrating log data that meets the highest priority second business condition to Elasticsearch for storage includes: When the query response time for cold data in ClickHouse exceeds a preset time threshold, the cold data is reclassified as hot data and migrated to the corresponding index in ElasticSearch. or, When the access frequency of cold data in ClickHouse exceeds a preset frequency threshold, the cold data is reclassified as hot data and migrated to the corresponding index in ElasticSearch.

7. A data cold and hot separation storage device, characterized in that, include: The storage module is used to store the collected log data into the corresponding index in ElasticSearch; The first judgment module is used to determine whether the log data is within a preset time period based on the index where the log data is located; The second judgment module is used to obtain the parameter information of the ElasticSearch and, based on the first business rule base, determine whether the parameter information of the ElasticSearch meets at least one first business condition. The migration module is used to migrate at least a portion of log data that is not within the preset time limit to ClickHouse storage based on the highest priority first business condition.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data cold and hot separation storage method as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data cold and hot separation storage method as described in any one of claims 1-6.

10. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer device, it causes the computer device to perform the data cold and hot separation storage method as described in any one of claims 1-6.

Images