Batch log storage method
Through the batch log storage method, the data bucket is used to receive logs and memory storage cache files concurrently, and combined with the dual threshold mechanism, the problems of low log data cache efficiency and low database import efficiency are solved, and efficient and real-time log data storage is achieved. and high concurrency.
Patent Information
- Application Number
- CN202411660624.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-20
- Publication Date
- 2025-02-21
AI Technical Summary
In the prior art, log data caching efficiency is low, cache file management is difficult, and database data import efficiency is low, resulting in log data entry efficiency and insufficient reception concurrency.
The batch log library method is adopted, including writing the normalized logs to the data bucket, flushing the log data in the data bucket to the cache file, outputting the cache file to the database import file queue, and executing the database import by the data import thread. This method improves the efficiency and concurrency of log database entry through the data bucket concurrent reception of logs, memory storage of cached files and dual threshold mechanisms.
It improves the efficiency of log data storage, improves the concurrency of log data reception, ensures real-time log data storage, and optimizes resource utilization during database import.
Smart Images
Figure CN119493832A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of network security technology, and in particular to a method for storing batch logs. Background Art
[0002] In the cybersecurity industry, as cyber attacks become increasingly complex and frequent, massive amounts of security logs and operation logs are generated every day. Faced with such a huge amount of data, how to store these logs quickly and efficiently has become a key challenge.
[0003] Since writing a single piece of data into a data warehouse occupies too many data warehouse connections, and batch SQL writing into the database is limited by the length of SQL statements and the number of log data, importing data through files is currently a more efficient, easy-to-use and reliable processing method in the data warehouse; using files as a carrier to execute log data import can not only improve the efficiency and quality of data processing, but also support batch processing of large-scale data.
[0004] In the prior art, the generation of database import files mostly relies on the local file system. The files are stored on the local disk, and their read and write performance is limited by the disk IO performance.
[0005] In addition, since each data import will occupy the number of connections, CPU central processing unit and other resources of the data warehouse, the data warehouse is more suitable for importing a small number of large files (above GB level) rather than a large number of small files; this also requires that log data be aggregated in large batches and large cache files be generated during the log data processing process. Furthermore, the security analysis business also has high requirements for the real-time storage of log data, which means that the cache file must be imported into the database once within a certain period of time after the log data is written to the cache file. Summary of the invention
[0006] The present invention provides a method for batch log storage, which aims to solve the problems of low log data cache efficiency, difficult cache file management, and low database import data efficiency, improve log data storage efficiency, and improve log data receiving concurrency.
[0007] The batch log storage method provided by the present disclosure includes the following steps:
[0008] S1, write the normalized logs into the data bucket;
[0009] S2, refresh the log data in the data bucket to the cache file;
[0010] S3, cache files are output to the database import file queue;
[0011] S4, take out the data file from the queue and execute data import.
[0012] Furthermore, the step S1 specifically includes:
[0013] M data buckets are started at the same time, and N*M normalized logs are received concurrently. A hash algorithm is used to make each data bucket store N logs.
[0014] Furthermore, in step S1, the specific method of writing the log into the data bucket includes:
[0015] If the logs are the same business log data and are eventually written to the same table in the database, only one processing flow module group is required;
[0016] If it is a different log type, multiple process module groups are started, and different types of process module groups are automatically diverted according to different log types; and according to the weight of the log, different parameters are set for each module group to control resource allocation, and the parameters include: one or more of the number of threads, memory size, and cache duration.
[0017] Furthermore, all log cache files, including free cache files, free cache files being imported, and cache files to be imported, are generated in the tmpfs directory, and operations on the cache files in this directory are all performed in memory.
[0018] Furthermore, the step S2 comprises:
[0019] Determine whether the total size of log data in the data bucket reaches the set data size threshold, or whether to execute the next refresh cycle;
[0020] When the log data in the data bucket triggers any of the two thresholds, a refresh operation is performed on the log data, that is, the data in the data bucket is written to the cache file and the data bucket is emptied;
[0021] The log data of M data buckets are transferred to one or more cache files. Since cache files can be reused for additional writing, no more than M cache files are written at the same time.
[0022] Furthermore, in step S2, when multiple data buckets reach the data refresh threshold at the same time, the cache file is written in an "exclusive + recycling" manner, wherein a separate thread is responsible for executing the data refresh operation of each data bucket;
[0023] Specifically, "exclusive" means:
[0024] The thread that refreshes the log data in the data bucket seizes a cache file at the head of the cache file queue and writes the log data. When the thread successfully seizes the cache file, it removes the cache file from the cache file queue. The above steps are executed as an indivisible operation to ensure that no two threads seize the cache file at the same time.
[0025] If the thread does not obtain the cache file, that is, the cache file queue is empty, a new cache file is created;
[0026] “Recycling” means:
[0027] When the data refresh thread finishes writing log data, it puts the cache file back into the cache file queue for other data refresh threads to preempt writing. That is, multiple data refresh threads perform batch append writing on the same cache file in different occupation cycles.
[0028] Furthermore, the step S3 of outputting the cache file to the database import file queue includes:
[0029] The following thresholds are used to determine the cache files in the cache file queue:
[0030] After a cache file is taken out from the cache file queue and the log data is written, it is determined whether the cache file size reaches the set "file size" threshold;
[0031] For cache files in the cache file queue waiting to be taken out for data writing, as well as cache files being written, determine whether the log cache duration reaches the set "data cache duration" threshold;
[0032] If any one of the above two judgments reaches the threshold, the cache file is removed from the cache file queue and put into the database import file queue.
[0033] Furthermore, in step S3, the step of determining whether the cache file size reaches the set “file size” threshold specifically includes:
[0034] When taking out a cache file from the cache file queue, the file size s1 is obtained from the file metadata information, and the current size of the log data to be written s2 is added, which is the total size of the cache file after writing is completed s = s1 + s2;
[0035] Determine the size of s and the cache file size threshold T1,
[0036] If s is less than T1, the cache file is put back into the cache file queue and waits to be taken out next time;
[0037] If s is greater than or equal to T1, the cache file is removed from the cache file queue and placed into the database import file queue.
[0038] Furthermore, in step S3, the step of determining whether the log cache duration reaches the set "data cache duration" threshold specifically includes:
[0039] Convert the cache file name into a timestamp, calculate the time difference with the current time, and obtain the creation time of the cache file; when the cache file is created, the timestamp of its creation time is used as the file name of the cache file;
[0040] Assume that the creation time of the cache file is t and the current time is t', then the cache duration is: Δt = t'-t;
[0041] Determine the size of Δt and cache file cache duration threshold T2;
[0042] If Δt is greater than or equal to T2, the cache file is removed from the cache file queue and placed into the database import file queue.
[0043] Furthermore, the step S4 specifically includes:
[0044] Start a separate data import thread and monitor the database import file queue.
[0045] Take out the cache file from the queue and perform data import, including: the client program of the data warehouse establishes a network connection with the data warehouse server, sends the cache data file to the data warehouse server in the form of a stream, and performs data parsing, verification, writing, and persistence;
[0046] After the import is complete, delete the cache file.
[0047] Compared with the prior art, the beneficial effects of the present invention are:
[0048] (1) Modularization of the execution process from receiving to storing log data;
[0049] (2) If the business has a requirement to store different log data in separate tables in the data warehouse, different types of log data can have their own module pipelines enabled. Different types of log data will not cross each other during the module pipeline processing. Different resources can also be allocated to different modules based on the importance of different log types through flexible parameter configuration. The efficiency of storing important data is prioritized within limited resources, so as to achieve isolation and differentiation of log data.
[0050] (3) The "double threshold" mechanism for data bucket and cache file settings ensures that when the log volume is large, the single database import file will not be too large, resulting in a long data import time; at the same time, when the log volume is small, scattered log data can also be stored in time, so that the log data can be stored in real time;
[0051] (4) Using the memory file system mechanism, the creation, reading, writing, and deletion of cache files are fast, effectively supporting high-concurrency scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.
[0053] Figure 1 This is a flow chart of log data storage according to an exemplary embodiment of the present disclosure;
[0054] Figure 2 is a structural diagram of a log data processing module according to the present disclosure;
[0055] Figure 3 This is the flow chart for batch log storage;
[0056] Figure 4 This is a relationship diagram between physical files and cache queues. DETAILED DESCRIPTION
[0057] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.
[0058] The present disclosure provides a method for batch log storage. The processing method includes: writing a large number of normalized logs into a data bucket; refreshing the log data in the data bucket to a cache file; and a data import thread executing database import.
[0059] In an exemplary embodiment, the overall flow chart is as shown in the attached Figure 1 As shown, it mainly includes the following steps:
[0060] 1. Step 103: Write logs to the data bucket
[0061] In this embodiment, if the logs are the same business log data and are eventually written into the same table of the database, only one processing flow module group is needed, such as Figure 2As shown in the structural diagram of the modules in ; if it is a different log type, multiple process module groups are started, and different types of process module groups can be automatically diverted according to different log types; and different parameters (number of threads, memory size, cache duration, etc.) can be set for each module group according to the weight of the log to control resource allocation.
[0062] 2. Step 104: refresh the data bucket to the cache file
[0063] Writing logs in a data bucket to a cache file includes: determining a timing for writing log data in the data bucket to the cache file, and a method for obtaining the cache file.
[0064] (1) Timing
[0065] A data bucket is a memory block in a program and is a container for temporarily collecting log data. In step 103, M data buckets are started at the same time to concurrently receive a large number of normalized N*M logs. At this time, a hash algorithm can be used to store N logs in each data bucket, and then write these batch logs into the cache file at the appropriate time.
[0066] The purpose of using data buckets is to avoid file lock contention caused by frequent writing of massive single logs to files and system overhead caused by frequent opening and closing of files; and because multiple data buckets process log data concurrently / in parallel, the concurrency performance of the service is improved from the entire process, and the efficiency of log writing is improved. The principle of data buckets is that two thresholds can be set - "data size" (ChunkBytes) and "refresh cycle" (FlushInterval), which are used to determine the timing of writing log data in the data bucket to the cache file, that is, it is necessary to determine:
[0067] Whether the total size of log data reaches the set data size threshold, or whether to execute the next refresh cycle; when the log data in the data bucket triggers any one of the two thresholds, a refresh operation is performed on the log data, that is, the data in the data bucket is written to the cache file, and the data bucket is emptied; at this time, the log data of M data buckets is transferred to one or more cache files. Since cache files can be reused for append writing, no more than M cache files are written at the same time.
[0068] (2) How to obtain cache files
[0069] To perform a log refresh operation, you must first obtain a cache file. Each data bucket data refresh operation is executed by a separate thread.
[0070] Multiple data buckets may reach the data refresh threshold at the same time. In this case, if they all write to the same cache file, the data written to the file by multiple data refresh threads will inevitably conflict because the file writing process is in append form. If each data bucket refresh thread creates a cache file separately, a large number of small cache files will be generated, and these cache files cannot be appended to and written to a second time.
[0071] In view of the above problems of log data writing conflicts and file writing dispersion, this embodiment adopts the form of "exclusive use + recycling" to solve them. Specifically:
[0072] 1) Exclusive
[0073] "Exclusive" means that the thread that refreshes the log data in the data bucket seizes a cache file at the head of the cache file queue and executes the writing of log data; when the thread successfully seizes the cache file, it removes the cache file from the cache file queue. The "atomic operation" in software engineering is used here, which refers to an indivisible operation, that is, multiple actions are either completed or not performed at all; it also means that no two threads will seize the cache file at the same time; if the thread does not obtain the cache file (the cache file queue is empty), a new cache file is created; this exclusive method can solve the problem of file conflicts when writing log data.
[0074] 2) Recycling
[0075] "Recycling" means that after the data refresh thread finishes writing log data, the cache file is put back into the cache file queue so that other data refresh threads can preempt writing. From the perspective of the cache file, multiple data refresh threads perform batch append writes on the same cache file in different occupation cycles, realizing the aggregation of log data in a single cache file, thereby solving the problem of scattered file writing.
[0076] Preferably, in order to further aggregate data, the cache file can be put back to the head of the cache file queue (the head of the queue is also the position occupied by the data refresh thread) so that other data refresh threads can obtain the cache file more times and perform more additional writes to the cache file before the cache file data ages.
[0077] (3) Storage management method of cache files
[0078] In the prior art, the generation of database import files mostly relies on the local file system. The files are stored on the local disk, and their read and write performance is limited by the disk IO performance. Since the database import file has high requirements for read and write performance, it is only a "temporary" feature of log data carrier in the business, which coincides with the characteristics of the tmpfs memory file system - it is completely located in the memory, so the access speed is extremely fast; it is mainly used to store data that does not need to be permanently saved. Therefore, in this embodiment, tmpfs is used as the basic storage for temporary cache files to overcome the disk IO limitation.
[0079] The cache file adopts tmpfs (a memory-based file system), which completely resides in the system's RAM (random access memory), does not occupy disk space, and is almost not limited by the disk I / O speed; tmpfs is a directory mounted on the host, and all log cache files (including idle cache files, free cache files, and cache files to be imported) are generated under this directory. The operations (creation, opening, reading, writing, closing, and deletion) of the cache files under this directory are all performed in the memory, so the access speed is extremely fast; this is very useful for the application scenario of frequent reading and writing of massive logs in this invention, and can greatly improve the log receiving performance and the database import file generation rate, thereby improving the log data storage rate.
[0080] 3. Step 105: Output the cached file to the database import file queue
[0081] This step requires judging two thresholds:
[0082] Whether the cache file size reaches the set "file size" threshold;
[0083] Whether the log cache duration reaches the set "data cache duration" threshold;
[0084] If either the cache file size or the data cache duration reaches a threshold, the cache file is removed from the cache file queue (which is used to store all free cache file paths) and placed into the database import file queue.
[0085] (1) Cache file size threshold
[0086] One of the purposes of setting this is to prevent the database import file from being too large, which would cause the data import to take too long. At the same time, because the database needs to prepare and clean up for each import, there will be a certain amount of overhead when importing log data, so the second purpose is to prevent a large number of small cache files from performing database import operations, which consumes database server resources and network IO resources.
[0087] Furthermore, the cache file size threshold is judged after a cache file is taken out from the cache file queue and the log data is written. When taking out a cache file from the cache file queue, the file size s1 can be obtained from the file metadata information, plus the current log data size s2 to be written, which is the total size of the cache file after writing is completed s=s1+s2; judge the size of s and the cache file size threshold T1; if s is less than T1, put the cache file back into the cache file queue and wait for the next time to be taken out; if s is greater than or equal to T1, remove the cache file from the cache file queue and put the cache file into the database import file queue;
[0088] (2) Log cache duration threshold
[0089] Target file to be judged: Since all cache files are in the same directory on the server host, these cache files can be logically divided into three categories, as shown in the following Figure 4 As shown: one type is the cache file in the cache file queue, waiting to be taken out to execute data writing; the second type is the cache file in the database import file queue, waiting to be taken out to execute database import operation; the third type is the cache file that is being written. At this time, the cache file is neither in the cache file queue nor in the database import file queue, and is called a "free" cache file.
[0090] Since the files in the database import file queue are already in the process of waiting for import, there is no need to determine the cache duration; in contrast, the remaining two types of files, cache files in the cache file queue and free cache files, need to determine the log data cache duration.
[0091] Log data cache duration judgment logic: The purpose of setting the cache duration threshold is to prevent the log data written to the cache file from being cached for too long, which prevents the log from being stored in real time. The first log written to a cache file must have the longest cache duration, which is approximately equal to the cache file creation duration; the data cache duration judgment problem can be converted into a cache file creation duration problem.
[0092] First of all, we can get the creation time of a file through its metadata. Some file systems (such as NTFS on Windows) do store the creation time of a file, but other file systems (such as traditional UNIX file systems, such as ext3 and ext4) may not store the creation time directly, or store it as an optional, non-standard attribute. Due to differences in file systems and operating systems, the standard libraries of some programming languages (such as golang) cannot universally provide methods for certain file metadata attributes, such as the attribute "file creation time" that needs to be obtained here.
[0093] In this embodiment, the common file metadata attribute - "file name" is cleverly used to carry the uncommon file metadata attribute - "file creation time"; that is, when a cache file is created, the timestamp of the cache file creation time is used as the cache file file name; when it is necessary to judge the cache file creation time, the file name can be converted into a timestamp to calculate the time difference with the current time, thereby obtaining the cache file creation time.
[0094] Specifically, assuming that the creation time of the cache file is t and the current time is t', the cache duration is: Δt = t'-t. Determine the size of Δt and the cache file cache duration threshold T2; if Δt is greater than or equal to T2, some data in this cache file has "aged" (exceeded the set cache duration threshold), and log data import needs to be performed immediately, that is, the cache file is removed from the cache file queue and placed in the database import file queue.
[0095] 4. Step 106, take out the data file from the queue and execute data import
[0096] Start a separate data import thread and listen to the database import file queue (a list data structure in a memory cache database with a first-in-first-out feature, which is used here to store the database cache file path to be imported);
[0097] Take out the cache file from the queue and perform data import;
[0098] The client program that comes with the data warehouse establishes a network connection with the data warehouse server, sends the cached data files to the data warehouse server in the form of a stream, and performs data parsing, verification, writing, and persistence;
[0099] After the import is complete, delete the cache file.
[0100] The specific execution judgment logic flow chart of the above process is as follows Figure 3 shown.
[0101] It can be seen that this embodiment realizes the modularization of the execution process from receiving to storing log data, such as Figure 2 As shown. For different types of log data, you can enable their own module pipelines and allocate resources according to their importance to improve storage efficiency;
[0102] The "double threshold" mechanism for data bucket and cache file settings ensures that when the log volume is large, the single database import file will not be too large, resulting in a long data import time; at the same time, when the log volume is small, scattered log data can also be stored in time, so that the log data can be stored in real time;
[0103] The traditional caching method of log cache files is changed to realize the rapid generation of database import files from batch log data: the traditional method is to use ordinary files of the local file system as cache files, and each reading and writing consumes disk IO. The performance of this method of generating database import files is positively correlated with the disk IO performance; in this embodiment, the memory file system tmpfs is used, which is completely based on memory storage, with almost no disk IO consumption, no network IO consumption, and extremely fast reading and writing speed; tmpfs itself is a file system, and the final storage form of log data is a file, which is convenient for the database to directly execute the import operation, and the cache file is in the memory at this time. When the database imports data from the cache file, it also reduces the CPU consumption and disk IO consumption of reading files from the disk and then sending them to the network. Therefore, this embodiment is based on the tmpfs memory file system to realize the rapid generation of massive log data carrier files, breaking through the performance bottleneck limitation of the entire intermediate flow process of massive log data from receiving to warehousing, and greatly improving the efficiency of log data warehousing.
[0104] The above technical scheme is only an exemplary embodiment of the present invention. For those skilled in the art, it is easy to make various types of improvements or modifications based on the application methods and principles disclosed in the present invention, and it is not limited to the method described in the above specific embodiment of the present invention. Therefore, the method described above is only preferred and does not have a restrictive meaning.
Claims
1. A method for storing batch logs, comprising the following steps: S1, write the normalized logs into the data bucket; S2, refresh the log data in the data bucket to the cache file; S3, cache files are output to the database import file queue; S4, take out the data file from the queue and execute data import.
2. The method according to claim 1, characterized in that: The step S1 specifically includes: M data buckets are started at the same time, and N*M normalized logs are received concurrently. A hash algorithm is used to make each data bucket store N logs.
3. The method according to claim 1 or 2, characterized in that: In step S1, the specific method of writing logs into the data bucket includes: If the logs are the same business log data and are eventually written to the same table in the database, only one processing flow module group is required; If it is a different log type, multiple process module groups are started, and different types of process module groups are automatically diverted according to different log types; and according to the weight of the log, different parameters are set for each module group to control resource allocation, and the parameters include: one or more of the number of threads, memory size, and cache duration.
4. The method according to claim 1, characterized in that: All log cache files, including free cache files, free cache files in the process of importing, and cache files to be imported, are generated in the tmpfs directory. All operations on the cache files in this directory are performed in memory.
5. The method according to claim 2, characterized in that: The step S2 comprises: Determine whether the total size of log data in the data bucket reaches the set data size threshold, or whether to execute the next refresh cycle; When the log data in the data bucket triggers any of the two thresholds, a refresh operation is performed on the log data, that is, the data in the data bucket is written to the cache file and the data bucket is emptied; The log data of M data buckets are transferred to one or more cache files. Since cache files can be reused for additional writing, no more than M cache files are written at the same time.
6. The method according to claim 5, characterized in that In step S2, when multiple data buckets reach the data refresh threshold at the same time, the cache file is written in an "exclusive + recycling" manner, wherein a separate thread is responsible for executing the data refresh operation of each data bucket; Specifically, "exclusive" means: The thread that refreshes the log data in the data bucket seizes a cache file at the head of the cache file queue and writes the log data. When the thread successfully seizes the cache file, it removes the cache file from the cache file queue. The above steps are executed as an indivisible operation to ensure that no two threads seize the cache file at the same time. If the thread does not obtain the cache file, that is, the cache file queue is empty, a new cache file is created; "Recycling" means: When the data refresh thread finishes writing log data, it puts the cache file back into the cache file queue for other data refresh threads to preempt writing. That is, multiple data refresh threads perform batch append writing on the same cache file in different occupation cycles.
7. The method according to claim 1, characterized in that The step S3 of outputting the cache file to the database import file queue includes: The following thresholds are used to determine the cache files in the cache file queue: When a cache file is taken out from the cache file queue and the log data is written, it is determined whether the cache file size reaches the set "file size" threshold; For cache files in the cache file queue waiting to be taken out for data writing, as well as cache files being written, determine whether the log cache duration reaches the set "data cache duration" threshold; If any one of the above two judgments reaches the threshold, the cache file is removed from the cache file queue and put into the database import file queue.
8. The method according to claim 7, characterized in that In step S3, the step of determining whether the cache file size reaches the set "file size" threshold specifically includes: When taking out a cache file from the cache file queue, the file size s1 is obtained from the file metadata information, and the current size of the log data to be written s2 is added, which is the total size of the cache file after writing is completed s = s1 + s2; Determine the size of s and the cache file size threshold T1, If s is less than T1, the cache file is put back into the cache file queue and waits to be taken out next time; If s is greater than or equal to T1, the cache file is removed from the cache file queue and placed into the database import file queue.
9. The method according to claim 7, characterized in that: In step S3, the step of determining whether the log cache duration reaches the set "data cache duration" threshold specifically includes: Convert the cache file name into a timestamp, calculate the time difference with the current time, and obtain the creation time of the cache file; when the cache file is created, the timestamp of its creation time is used as the file name of the cache file; Assume that the creation time of the cache file is t and the current time is t', then the cache duration is: Δt = t'-t; Determine the size of Δt and cache file cache duration threshold T2; If Δt is greater than or equal to T2, the cache file is removed from the cache file queue and placed into the database import file queue.
10. The method according to claim 1, characterized in that The step S4 specifically includes: Start a separate data import thread and monitor the database import file queue. Take out the cache file from the queue and perform data import, including: the client program of the data warehouse establishes a network connection with the data warehouse server, sends the cache data file to the data warehouse server in the form of a stream, and performs data parsing, verification, writing, and persistence; After the import is complete, delete the cache file.