Log management methods, devices, electronic equipment and storage media
By constructing a circular queue of activity and backup log table names and performing rollback operations in batches, the problem of speed degradation in massive log management was solved, and efficient log insertion, querying, and deletion were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING TOPSEC NETWORK SECURITY TECH
- Filing Date
- 2023-11-13
- Publication Date
- 2026-07-17
AI Technical Summary
Existing technologies severely reduce the speed of inserting, deleting, and querying logs in the management of massive log volumes, causing the front-end query page to time out or fail to query normally.
A circular queue of activity log table names and a circular queue of backup log table names are constructed using database table names. The log monitoring thread periodically checks the activity logs and backup logs in batches and performs rollback operations in batches to improve log management speed.
By storing logs in a partitioned table structure and performing periodic checks, the speed of inserting, querying, and deleting massive amounts of logs is improved, avoiding the problem of slow log management.
Smart Images
Figure CN117421293B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of network security technology, and in particular to a log management method, apparatus, electronic device, and storage medium. Background Technology
[0002] Database sharding and table partitioning are used to address the performance degradation caused by excessive data volume. By reducing the data size of a single database or table, database performance is improved. Embedded devices, such as network security equipment, need to record large amounts of logs. Providing log management functionality allows for log rollback based on the number of log entries, log size, or percentage of disk space used by the logs. This involves overwriting or deleting ineligible log records in the database table before adding new ones.
[0003] The related technology employs large-capacity hard drives, establishes date fields for all logs stored in database tables, creates indexes for querying log fields, deletes some log records that do not meet the criteria, and inserts new logs into the database tables. When querying on the management page, all data in the tables needs to be retrieved, and the front-end management page needs to display the data in batches. Therefore, when there are massive amounts of log records in the database tables, the speed of log management, including inserting, deleting, and querying logs, will be severely reduced. If the front-end query page times out or even fails to query logs normally, this technology may cause issues. Summary of the Invention
[0004] In view of this, embodiments of this disclosure provide a log management method, apparatus, electronic device, and storage medium. By constructing a circular queue of active log table names and a circular queue of backup log table names using database table names, storing active log data through the circular queue of active log table names and storing backup log data through the circular queue of backup log table names, and finally using a log monitoring thread to periodically batch detect active logs and backup logs and batch execute rollback operations on active logs and backup logs, the speed of log management such as insertion, query, and deletion of massive logs is improved.
[0005] In a first aspect, embodiments of this disclosure provide a log management method, employing the following technical solution:
[0006] A circular queue of activity log table names is constructed based on the first database table name, wherein the circular queue of activity log table names contains several activity tables for storing activity logs using a partitioned table structure.
[0007] When the activity log table name circular queue meets the backup conditions, the first database table name of the activity table is modified to the second database table name of the corresponding backup table, and a backup log table name circular queue is constructed based on the second database table name. The backup log table name circular queue contains several backup tables obtained after modifying the table name of the activity table. The backup tables are used to store backup logs using a table partitioning structure.
[0008] The activity logs and backup logs are periodically and batch-checked using a log monitoring thread.
[0009] When the activity log and the backup log meet the preset log rollback conditions, a rollback operation is performed on the activity log and the backup log.
[0010] In some embodiments, the method further includes:
[0011] Call the logging interface to encapsulate the activity logs into a data structure;
[0012] The encapsulated activity log is sent to the log storage thread through the circular queue of the activity log table name;
[0013] The log storage thread periodically polls the circular queue of the activity log table name to obtain the writable queue nodes in the circular queue of the activity log table name.
[0014] The encapsulated activity log is stored in the writable queue node.
[0015] In some embodiments, storing the encapsulated activity log into the writable queue node includes:
[0016] Obtain the storage table header identifier, storage table tail identifier, and storage table maximum identifier of the activity log table name circular queue, wherein the storage table header identifier represents the ID value of the queue node that can be written, the storage table tail identifier represents the ID value of the queue node that has been written, and the storage table maximum identifier represents a preset queue node number threshold.
[0017] Obtain the writable queue node corresponding to the storage header identifier;
[0018] The encapsulated activity log is stored in the writable activity table corresponding to the writable queue node using a partitioned table structure.
[0019] When the number of active logs in the writable activity table is greater than or equal to the preset active log threshold, obtain the next writable queue node in the circular queue of the activity log table name.
[0020] The encapsulated activity log is stored in the writable activity table corresponding to the next writable queue node.
[0021] In some embodiments, the encapsulated activity log is stored in a writable activity table corresponding to the writable queue node using a partitioned table structure, including:
[0022] Generate SQL statements based on the encapsulated activity logs;
[0023] The SQL statement is stored in the writable active table corresponding to the writable queue node.
[0024] In some embodiments, when the active log table name circular queue meets the backup conditions, the first database table name of the active table is modified to the second database table name of the corresponding backup table, and a backup log table name circular queue is constructed based on the second database table name, including:
[0025] When the sum of the number of writable queue nodes and the number of written queue nodes in the circular queue of the activity log table name is greater than or equal to a preset queue node number threshold, an activity table is randomly selected from the circular queue of the activity log table name.
[0026] Call the log backup interface to change the first database table name of the active table to the second database table name of the corresponding backup table;
[0027] A circular queue of backup log table names is constructed based on the second database table name;
[0028] Add the backup table obtained after renaming the active table to the circular queue of the backup log table name;
[0029] Obtain the backup table header identifier and backup table tail identifier of the backup log table name circular queue, wherein the backup table header identifier represents the ID value of the backup node that can be written to the queue, and the backup table tail identifier represents the ID value of the backup node that has been written to the queue.
[0030] The backup logs are stored in a partitioned table structure in the writable backup table of the writable queue backup node corresponding to the backup table header identifier.
[0031] In some embodiments, when the activity log and the backup log meet preset log rollback conditions, a rollback operation is performed on the activity log and the backup log, including:
[0032] The log monitoring thread periodically performs batch checks on at least one of the following: the total number of log records in the activity log and the backup log; the database storage space; and the hard disk usage ratio.
[0033] When the total number of log records is greater than or equal to a preset log record threshold, or the database storage space is greater than or equal to a preset storage space threshold, or the hard disk occupancy ratio is greater than or equal to a preset hard disk occupancy ratio threshold, it is determined that the activity log and the backup log meet the preset log rollback conditions.
[0034] Obtain the tail identifier of the backup log table name circular queue;
[0035] Call the SQL statement to delete the backup table whose name corresponds to the tail identifier of the backup table;
[0036] Delete the record information for the backup table name in the backup information table, and update the number of backup table records in the log information table to the number of backup tables deleted.
[0037] In some embodiments, the method further includes:
[0038] During the rollback operation, the percentage of log entries deleted from the activity log and the backup log, the percentage of space released from the database storage space, and the percentage of hard disk released from the hard disk usage ratio are calculated.
[0039] When the percentage of log entries deleted, the percentage of space released, and the percentage of hard disk released are all greater than or equal to the preset rollback percentage threshold, the rollback operation is stopped.
[0040] When at least one of the log entry deletion percentage, the space release percentage, and the hard disk release percentage is less than a preset rollback percentage threshold, the rollback operation continues.
[0041] Secondly, this disclosure also provides a log management device, which adopts the following technical solution:
[0042] The building unit is configured to build a circular queue of activity log table names based on the first database table name, wherein the circular queue of activity log table names contains several activity tables for storing activity logs using a partitioned table structure.
[0043] The table name modification unit is configured to modify the first database table name of the active table to the second database table name of the corresponding backup table when the active log table name circular queue meets the backup conditions, and construct a backup log table name circular queue based on the second database table name. The backup log table name circular queue contains several backup tables obtained after modifying the table name of the active table. The backup tables are used to store backup logs using a table partitioning structure.
[0044] The log monitoring unit is configured to periodically batch check the activity logs and the backup logs via a log monitoring thread.
[0045] The rollback operation unit is configured to perform a rollback operation on the activity log and the backup log when the activity log and the backup log meet preset log rollback conditions.
[0046] Thirdly, this disclosure also provides an electronic device that adopts the following technical solution:
[0047] The electronic device includes:
[0048] At least one processor; and,
[0049] A memory communicatively connected to the at least one processor; wherein,
[0050] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform any of the log management methods described above.
[0051] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions for causing a computer to perform any of the log management methods described above.
[0052] This disclosure provides a log management method that improves the speed of log management, including inserting, querying, and deleting massive amounts of logs, by constructing a circular queue of activity log names and a circular queue of backup log names using database table names, storing activity log data in the circular queue of activity log names and backup log data in the circular queue of backup log names, storing activity logs and backup logs in a partitioned table structure, and finally using a log monitoring thread to periodically batch detect activity logs and backup logs and batch execute rollback operations on activity logs and backup logs.
[0053] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0054] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0055] Figure 1A flowchart illustrating a log management method provided in an embodiment of this disclosure;
[0056] Figure 2 A schematic diagram of the structure of the log information table and the circular queue of the activity log table provided in this embodiment of the disclosure;
[0057] Figure 3 A schematic diagram of the structure of the circular queue of the backup information table and backup log table provided in the embodiments of this disclosure;
[0058] Figure 4 This is a schematic diagram of the structure of a log management device provided in an embodiment of the present disclosure;
[0059] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0060] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0061] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0062] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.
[0063] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0064] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.
[0065] Figure 1 This is a flowchart illustrating a log management method provided in an embodiment of the present disclosure. The log management method includes the following steps:
[0066] S101. Construct a circular queue of activity log table names based on the first database table name, wherein the circular queue of activity log table names contains several activity tables used to store activity logs in a partitioned table structure.
[0067] Figure 2 This is a schematic diagram of the structure of the log information table and the circular queue of the activity log table provided in this embodiment. The log information table stores basic information about the log system. The fields in the log information table include: head_id, representing the header identifier (i.e., the header ID value, also representing the ID value of the queue node that can be written to); tail_id, representing the tail identifier (i.e., the tail ID value, also representing the ID value of the queue node that has been written to); max_id, representing the maximum identifier of the table (i.e., the maximum ID value, also representing a preset threshold for the number of queue nodes); max_num, representing the maximum number of records in the table (i.e., a preset activity log threshold); bu_h `ead_id` represents the backup table header identifier, i.e., the backup table header ID value, and also the ID value of the backup node that can be written to the queue; `bu_tail_id` represents the backup table tail identifier, i.e., the backup table tail ID value, and also the ID value of the backup node that has been written to the queue; `bu_rec_num` represents the number of backup table records; `rec_max_num` represents the maximum number of log records, i.e., the preset log record threshold; `db_max_size` represents the data storage size, i.e., the preset storage space threshold; `disk_max_vp` represents the capacity utilization percentage, i.e., the preset hard disk utilization percentage threshold; `rollback_np` represents the preset rollback percentage threshold.
[0068] The activity log table name is the same as the first database table name. Building a circular queue of activity log table names based on this first database table name involves: constructing a circular queue with the creation table name = "tbl_" + string (id) based on head_id, tail_id, and max_id. head_id is the ID value of the writable queue node, max_id is the maximum number of nodes in the queue (i.e., the preset queue node number threshold), and tail_id is the ID value of the already written queue node. This table constitutes the activity table. Newly added activity logs are recorded in the activity table corresponding to head_id. When recording activity logs, logs are recorded in the activity table corresponding to head_id in the circular queue of activity log table names. When the number of activity logs is greater than or equal to the preset activity log threshold max_num, a new writable queue node is obtained from the circular queue of activity log table names, and the activity log data is written to the activity table corresponding to the next writable queue node (head_id++).
[0069] Optionally, ① when head_id > max_id and head_id = 1, perform a rollback operation on the activity log. ② when head_id == tail_head, call the log backup interface to perform a backup, and increment tail_id after the backup. ③ Update the head_id and tail_id values in the log information table. The default values for the parameters in the log information table are set to: head_id = 1, tail_id = 1, max_num = 10K, max_id = 10.
[0070] S102. When the active log table name circular queue meets the backup conditions, the first database table name of the active table is changed to the second database table name of the corresponding backup table, and a backup log table name circular queue is constructed based on the second database table name. The backup log table name circular queue contains several backup tables obtained after the active table name is changed. The backup tables are used to store backup logs using a partitioned table structure.
[0071] S103. Periodically batch check activity logs and backup logs through the log monitoring thread.
[0072] S104. When the activity log and backup log meet the preset log rollback conditions, perform a rollback operation on the activity log and backup log.
[0073] Figure 3This diagram illustrates the structure of the backup information table and the circular queue for backup log table names provided in this embodiment. The backup table name is the same as the second database table name. The backup information table and the circular queue for backup log table names are managed through the log backup interface based on `bu_head_id`, `bu_tail_id`, and `bu_rec_num`, and update information for the backup table is recorded in the backup information table. The circular queue for backup log table names is constructed using `bu_head_id` and `bu_tail_id`, with initial values set to: `bu_head_id = 1`, `bu_tail_id = 1`, and `bu_rec_num = 0`. Backups are performed by calling the log backup interface, whose parameters include `tail_id` and the number of log records. Internally, the log backup interface changes the table named "tbl_" + string (tail_id) to "b_tbl_" + string (bu_head_id) based on the tail_id value, creates a new table named "tbl_" + string (tail_id), records a record in the backup information table with bu_id = bu_head_id, date, and record count, and finally increments bu_head_id and increments bu_rec_num for the record count, and updates the bu_head_id, bu_tail_id, and bu_rec_num values in the log information table.
[0074] This embodiment of the disclosure improves the speed of log management, including inserting, querying, and deleting massive amounts of logs, by constructing a circular queue of activity log table names and a circular queue of backup log table names using database table names, storing activity log data through the circular queue of activity log table names and the circular queue of backup log table names, storing activity logs and backup logs through a partitioned table structure, and finally improving the speed of log management, including inserting, querying, and deleting massive amounts of logs, by using a log monitoring thread to periodically batch detect activity logs and backup logs and batch execute rollback operations on activity logs and backup logs.
[0075] In some embodiments, the method further includes:
[0076] Call the logging interface to encapsulate the activity logs into a data structure;
[0077] The encapsulated activity log is sent to the log storage thread through a circular queue of activity log table names;
[0078] The log storage thread periodically polls the circular queue of the activity log table name to obtain the writable queue nodes in the circular queue of the activity log table name.
[0079] The encapsulated activity logs are stored in a writable queue node.
[0080] In some embodiments, storing the encapsulated activity log into a writable queue node includes:
[0081] Get the storage header identifier, storage tail identifier, and storage maximum identifier of the circular queue of the activity log table name. The storage header identifier represents the ID value of the queue node that can be written, the storage tail identifier represents the ID value of the queue node that has been written, and the storage maximum identifier represents the preset threshold for the number of queue nodes.
[0082] Retrieve the writable queue node corresponding to the storage header identifier;
[0083] The encapsulated activity logs are stored in the writable activity table corresponding to the writable queue node using a table partitioning structure.
[0084] When the number of active logs that can be written to the active table is greater than or equal to the preset active log threshold, retrieve the next writable queue node in the circular queue of the active log table name.
[0085] The encapsulated activity log is stored in the writable activity table corresponding to the next writable queue node.
[0086] In some embodiments, the encapsulated activity log is stored in a writable activity table corresponding to the writable queue node using a partitioned table structure, including:
[0087] Generate SQL statements based on the encapsulated activity logs;
[0088] Store the SQL statement in the writable active table corresponding to the writable queue node.
[0089] In some embodiments, when the active log table name circular queue meets the backup conditions, the first database table name of the active table is modified to the second database table name of the corresponding backup table, and a backup log table name circular queue is constructed based on the second database table name, including:
[0090] When the sum of the number of writable queue nodes and the number of already written queue nodes in the circular queue of the activity log table name is greater than or equal to the preset queue node number threshold, an activity table is randomly selected from the circular queue of the activity log table name.
[0091] Call the log backup interface to change the first database table name of the active table to the second database table name of the corresponding backup table;
[0092] A circular queue of backup log table names is constructed based on the second database table name;
[0093] Add the backup table obtained after renaming the active table to the backup log table name circular queue.
[0094] Retrieve the backup table header and tail identifier of the backup log table name circular queue. The backup table header identifier represents the ID value of the backup node that can be written to the queue, and the backup table tail identifier represents the ID value of the backup node that has been written to the queue.
[0095] The backup logs are stored in a partitioned table structure in the writable backup table of the writable queue backup node corresponding to the backup table header identifier.
[0096] In some embodiments, when the activity log and backup log meet preset log rollback conditions, a rollback operation is performed on the activity log and backup log, including:
[0097] The log monitoring thread periodically and in batches checks at least one of the following: the total number of log records in the activity log and backup log, database storage space, and hard disk usage ratio.
[0098] When the total number of log records is greater than or equal to the preset log record threshold, or the database storage space is greater than or equal to the preset storage space threshold, or the hard disk usage ratio is greater than or equal to the preset hard disk usage ratio threshold, the active log and backup log are determined to meet the preset log rollback conditions.
[0099] Get the tail identifier of the backup log table name circular queue;
[0100] Use an SQL statement to delete the backup table whose name corresponds to the identifier at the end of the backup table;
[0101] Delete the record information for the backup table name in the backup information table, and update the number of backup table records in the log information table to the number of backup tables deleted.
[0102] In some embodiments, the method further includes:
[0103] During the rollback operation, the percentage of log entries deleted from the activity log and backup log, the percentage of database storage space released, and the percentage of hard disk space released are calculated.
[0104] When the percentage of log entries deleted, the percentage of space released, and the percentage of hard disk released are all greater than or equal to the preset rollback percentage threshold, the rollback operation will be stopped.
[0105] If at least one of the following percentages—log entry deletion percentage, space release percentage, and hard disk release percentage—is less than the preset rollback percentage threshold, the rollback operation will continue.
[0106] Optionally, during the rollback operation, the `bu_tail_id` value in the backup information table is retrieved, and the SQL statement (`truncate`) is used to delete tables whose table name = "b_tbl_" + string (`bu_tail_id`). Records in the backup information table where `bu_id` = `bu_tail_id` are queried and deleted. The value of `bu_rec_num` in the log information table is updated to `bu_rec_num -= the number of tables deleted`. The percentage of log entries deleted from the active log and backup log, the percentage of database storage space released, or the percentage of disk space released are calculated to determine if they are greater than or equal to the preset rollback percentage threshold `rollback_np`. If they are less than the preset rollback percentage threshold `rollback_np`, the rollback operation continues until the percentage of log entries deleted, the percentage of space released, and the percentage of disk space released are all greater than or equal to the preset rollback percentage threshold.
[0107] The system periodically reads and calculates the values of fields such as head_id, tail_id, max_id, and bu_rec_num from the log information table. When rec_max_num = -1 (a value set by the user through the management page, indicating that log record count checks are not performed), the log record count is not checked. When rec_max_num > 0, the total number of log records, v_rec_max_num, equals the number of active log records plus the number of backup log records. The number of active log records represents the estimated number of data tables that have been logged based on the head_id, tail_id, and max_id values. The number of backup log records is bu_rec_num. When v_rec_max_num ≥ rec_max_num, a rollback operation is initiated.
[0108] The database storage space is checked periodically. When db_max_size = -1, the database file storage space size is not checked. When the database storage space is greater than or equal to a preset storage space threshold, a rollback operation is performed.
[0109] The system periodically checks the disk usage ratio. When disk_max_v = -1, the system does not check the disk usage ratio. When disk_max_vp > 0, the system calls a system function to obtain disk data and calculates the disk usage ratio. When the calculated disk usage ratio is greater than or equal to the preset disk usage ratio threshold of disk_max_vp, the system initiates a rollback operation.
[0110] For example, the preset disk usage percentage threshold `disk_max_vp` can be set to 50%. Specifically, the preset rollback percentage threshold can be set during the initialization phase. Additionally, during initialization, the parameters for the active log table name circular queue can be set as follows: `head_id = 10`, `tail_id = 1`, `max_id = 10`, `max_num = 10K`; and the parameters for the backup log table name circular queue can be set as follows: `bu_head_id = 101`, `bu_tail_head = 1`. The log monitoring thread can be configured to check the disk usage percentage every 5 seconds.
[0111] It should be noted that users can set preset rollback percentage thresholds, disk usage ratio thresholds, and log monitoring thread timing periods according to actual business needs, and this disclosure does not limit these settings.
[0112] Optionally, preset log recording thresholds, preset storage space thresholds, or preset hard disk usage percentage thresholds can be set through the management page, while preset rollback percentage thresholds can also be set, and the relevant information can be stored in the log information table.
[0113] Optionally, this embodiment provides a query interface for querying activity logs and backup logs. Querying activity logs includes: organizing SQL statements according to parameters to query data from all active tables in the circular queue of activity log table names, as well as the activity logs within those tables. Querying backup logs includes: querying backup log records in backup tables that match the date range in the query request; using SQL statements to count the number of records (v_num) that meet the conditions across all backup tables; and returning 1.2 times the total number of records in the active tables, i.e., vq_num = max_num * max_id * 1.2. v_num is compared with vq_num; if v_num > vq_num, backup log records in backup tables matching the vq_num count are queried, and the backup log records and information about the remaining unqueried backup tables are returned, prompting the user to query again. Upon re-querying, backup tables matching the conditions are selected and queried again according to the parameters until all matching backup tables are queried. If v_num ≤ vq_num, backup log records in backup tables matching the vq_num count are queried, and the results are returned.
[0114] This embodiment of the disclosure utilizes circular queues for activity logs and backup logs, built based on database table names. These queues allow for rapid table partitioning and storage of activity logs or backup logs, preventing a slowdown in log management speed due to a significant increase in log volume. The queues, built from database table names, allow renaming activity log tables to backup table names and adding them to the queue for management, avoiding a slowdown in data backup speed due to increased log volume. A log monitoring thread periodically checks the total number of log entries, storage space, or disk occupancy. When conditions are met, a backup table is truncated from the backup log queue, preventing a slowdown in data deletion speed due to increased log volume.
[0115] Figure 4 This is a schematic diagram of a log management device provided in an embodiment of the present disclosure. The log management device provided in an embodiment of the present disclosure includes:
[0116] Construction unit 41 is configured to construct a circular queue of activity log table names based on the first database table name, wherein the circular queue of activity log table names contains several activity tables for storing activity logs using a partitioned table structure.
[0117] The table name modification unit 42 is configured to modify the first database table name of the active table to the second database table name of the corresponding backup table when the active log table name circular queue meets the backup conditions, and construct the backup log table name circular queue based on the second database table name. The backup log table name circular queue contains several backup tables obtained after modifying the table names of the active tables. The backup tables are used to store backup logs using a table partitioning structure.
[0118] Log monitoring unit 43 is configured to periodically batch detect activity logs and backup logs through the log monitoring thread;
[0119] The rollback operation unit 44 is configured to perform a rollback operation on the active log and backup log when the active log and backup log meet the preset log rollback conditions.
[0120] An electronic device according to embodiments of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, flash memory, etc.
[0121] The processor may be a central processing unit (CPU) or other processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the electronic device to perform all or part of the steps of a log management method according to the foregoing embodiments of this disclosure.
[0122] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.
[0123] like Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the electronic device in the embodiment of the present disclosure. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0124] like Figure 5 As shown, an electronic device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processor, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.
[0125] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow electronic devices to communicate wirelessly or wiredly with other devices (such as edge computing devices) to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.
[0126] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from ROM. When the computer program is executed by a processor, all or part of the steps of a log management method according to embodiments of this disclosure are performed.
[0127] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0128] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions. When these non-transitory computer-readable instructions are executed by a processor, all or part of the steps of a log management method according to the foregoing embodiments of the present disclosure are performed.
[0129] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).
[0130] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.
[0131] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0132] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.
[0133] Additionally, as used herein, the “or” used in a list of items beginning with “at least one” indicates a separate list, such that a list of, for example, “at least one of A, B, or C” means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word “exemplary” does not imply that the described example is preferred or better than other examples.
[0134] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.
[0135] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.
[0136] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0137] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
Claims
1. A log management method, characterized in that, include: A circular queue of activity log table names is constructed based on the first database table name, wherein the circular queue of activity log table names contains several activity tables for storing activity logs using a partitioned table structure. When the activity log table name circular queue meets the backup conditions, the first database table name of the activity table is modified to the second database table name of the corresponding backup table, and a backup log table name circular queue is constructed based on the second database table name. The backup log table name circular queue contains several backup tables obtained after modifying the table name of the activity table. The backup tables are used to store backup logs using a table partitioning structure. The activity logs and backup logs are periodically and batch-checked using a log monitoring thread. When the activity log and the backup log meet the preset log rollback conditions, a rollback operation is performed on the activity log and the backup log, including: Obtain the backup table tail identifier of the circular queue of the backup log table name; the backup table tail identifier represents the ID value of the backup node that has been written into the queue. Delete the backup table whose name corresponds to the tail identifier of the backup table; Delete the record information of the backup table name of the backup table mentioned in the backup information table, and update the number of backup table records in the log information table to the number of backup tables deleted. The fields in the log information table include: storage table header identifier, storage table footer identifier, storage table maximum identifier, backup table header identifier, and backup table footer identifier.
2. The log management method according to claim 1, characterized in that, The method further includes: Call the logging interface to encapsulate the activity logs into a data structure; The encapsulated activity log is sent to the log storage thread through the circular queue of the activity log table name; The log storage thread periodically polls the circular queue of the activity log table name to obtain the writable queue nodes in the circular queue of the activity log table name. The encapsulated activity log is stored in the writable queue node.
3. The log management method according to claim 2, characterized in that, Storing the encapsulated activity log into the writable queue node includes: Obtain the storage table header identifier, storage table tail identifier, and storage table maximum identifier of the activity log table name circular queue, wherein the storage table header identifier represents the ID value of the queue node that can be written, the storage table tail identifier represents the ID value of the queue node that has been written, and the storage table maximum identifier represents a preset queue node number threshold. Obtain the writable queue node corresponding to the storage header identifier; The encapsulated activity log is stored in the writable activity table corresponding to the writable queue node using a partitioned table structure. When the number of active logs in the writable activity table is greater than or equal to the preset active log threshold, obtain the next writable queue node in the circular queue of the activity log table name. The encapsulated activity log is stored in the writable activity table corresponding to the next writable queue node.
4. The log management method according to claim 3, characterized in that, The encapsulated activity logs are stored in a partitioned table structure in the writable activity table corresponding to the writable queue node, including: Generate SQL statements based on the encapsulated activity logs; The SQL statement is stored in the writable active table corresponding to the writable queue node.
5. The log management method according to claim 3, characterized in that, When the active log table name circular queue meets the backup conditions, the first database table name of the active table is modified to the second database table name of the corresponding backup table, and a backup log table name circular queue is constructed based on the second database table name, including: When the sum of the number of writable queue nodes and the number of written queue nodes in the circular queue of the activity log table name is greater than or equal to a preset queue node number threshold, an activity table is randomly selected from the circular queue of the activity log table name. Call the log backup interface to change the first database table name of the active table to the second database table name of the corresponding backup table; A circular queue of backup log table names is constructed based on the second database table name; Add the backup table obtained after renaming the active table to the circular queue of the backup log table name; Obtain the backup table header identifier and backup table tail identifier of the backup log table name circular queue, wherein the backup table header identifier represents the ID value of the backup node that can be written to the queue, and the backup table tail identifier represents the ID value of the backup node that has been written to the queue. The backup logs are stored in a partitioned table structure in the writable backup table of the writable queue backup node corresponding to the backup table header identifier.
6. The log management method according to claim 5, characterized in that, When the activity log and the backup log meet the preset log rollback conditions, a rollback operation is performed on the activity log and the backup log, including: The log monitoring thread periodically performs batch checks on at least one of the following: the total number of log records in the activity log and the backup log; the database storage space; and the hard disk usage ratio. When the total number of log records is greater than or equal to a preset log record threshold, or the database storage space is greater than or equal to a preset storage space threshold, or the hard disk occupancy ratio is greater than or equal to a preset hard disk occupancy ratio threshold, it is determined that the activity log and the backup log meet the preset log rollback conditions. Obtain the tail identifier of the backup log table name circular queue; Call the SQL statement to delete the backup table whose name corresponds to the tail identifier of the backup table; Delete the record information for the backup table name in the backup information table, and update the number of backup table records in the log information table to the number of backup tables deleted.
7. The log management method according to claim 6, characterized in that, The method further includes: During the rollback operation, the percentage of log entries deleted from the activity log and the backup log, the percentage of space released from the database storage space, and the percentage of hard disk released from the hard disk usage ratio are calculated. When the percentage of log entries deleted, the percentage of space released, and the percentage of hard disk released are all greater than or equal to the preset rollback percentage threshold, the rollback operation is stopped. When at least one of the log entry deletion percentage, the space release percentage, and the hard disk release percentage is less than a preset rollback percentage threshold, the rollback operation continues.
8. A log management device, characterized in that, include: The building unit is configured to build a circular queue of activity log table names based on the first database table name, wherein the circular queue of activity log table names contains several activity tables for storing activity logs using a partitioned table structure. The table name modification unit is configured to modify the first database table name of the active table to the second database table name of the corresponding backup table when the active log table name circular queue meets the backup conditions, and construct a backup log table name circular queue based on the second database table name. The backup log table name circular queue contains several backup tables obtained after modifying the table name of the active table. The backup tables are used to store backup logs using a table partitioning structure. The log monitoring unit is configured to periodically batch check the activity logs and the backup logs via a log monitoring thread. The rollback operation unit is configured to perform a rollback operation on the activity log and the backup log when the activity log and the backup log meet preset log rollback conditions, including: Obtain the backup table tail identifier of the circular queue of the backup log table name; the backup table tail identifier represents the ID value of the backup node that has been written into the queue. Delete the backup table whose name corresponds to the tail identifier of the backup table; Delete the record information of the backup table name of the backup table mentioned in the backup information table, and update the number of backup table records in the log information table to the number of backup tables deleted. The fields in the log information table include: storage table header identifier, storage table footer identifier, storage table maximum identifier, backup table header identifier, and backup table footer identifier.
9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the log management method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the log management method as described in any one of claims 1 to 7.