A log data management method, device, equipment and medium

Abstract

The application discloses a log data management method and device, equipment and medium, and relates to the technical field of log management. The method comprises the following steps: acquiring log data, and storing target metadata corresponding to each log data and the log data into a ring buffer; the target metadata comprises an event type, a log source, a keyword and a timestamp; determining a priority score of each log data based on the target metadata, and storing target pointers corresponding to the log data into corresponding priority queues based on the priority score; the target pointer is a pointer or an index pointing to the log data stored in the ring buffer; if the remaining space of the ring buffer does not meet a preset condition, determining a to-be-evicted queue in the priority queues based on the current occupancy rate of the ring buffer, and evicting the corresponding log data in the ring buffer based on the target pointers in the to-be-evicted queue, so as to complete the management of the log data. The application can preferentially retain key information according to the importance of the log.

Description

Technical Field

[0001] This invention relates to the field of log management technology, and in particular to a log data management method, apparatus, device, and medium. Background Technology

[0002] In modern information systems, logs are crucial data generated during system operation, commonly used for troubleshooting, performance monitoring, and security auditing. However, traditional log collection systems typically employ a uniform strategy to save all logs, resulting in a large amount of irrelevant log data stored in the system. This wastes storage space and negatively impacts system performance. Especially in resource-constrained environments such as embedded devices and IoT terminals, traditional log collection methods not only consume excessive computing resources and storage space but also affect the stability of core business operations.

[0003] Existing log collection systems, such as those based on FIFO (First Input First Output) or LRU (Least Recently Used) strategies, employ FIFO, which manages logs based on the first-in, first-out principle, and LRU, which evicts logs based on the least recently used logs. While these methods are suitable for some scenarios, they often fail to guarantee the priority preservation of critical information in high-concurrency or resource-constrained environments. Especially when logs overflow or buffer space is insufficient, they may discard log data crucial for system fault diagnosis and performance analysis. For example, in high-concurrency scenarios, FIFO and LRU strategies discard logs in chronological or least-used order, failing to prioritize logs based on their actual importance. Furthermore, as log data volume increases, traditional methods like FIFO and LRU can easily lead to indiscriminate discarding of log data, failing to guarantee system stability and performance under resource constraints. Therefore, existing log management solutions suffer from the deficiency of not intelligently distinguishing log importance. Thus, how to intelligently prioritize and evict logs based on their actual importance to ensure the effective retention of critical logs is a pressing issue that needs to be addressed. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a log data management method, apparatus, device, and medium. It enables intelligent priority evaluation and eviction based on the actual importance of logs, ensuring that critical logs are effectively retained, thereby improving system stability and performance. The specific solution is as follows:

[0005] Firstly, this application discloses a log data management method, including:

[0006] Acquire log data and store the target metadata corresponding to each log data item and the log data in a circular buffer; the target metadata includes event type, log source, keyword, and timestamp;

[0007] Based on the target metadata, a priority score is determined for each log data. Based on the priority score, the target pointers corresponding to the log data are stored in the corresponding priority queues. The target pointer is a pointer or index pointing to the log data stored in the circular buffer.

[0008] If the remaining space of the circular buffer does not meet the preset conditions, the priority queue to be eliminated is determined based on the current occupancy rate of the circular buffer, and the corresponding log data in the circular buffer is eliminated based on the target pointer in the queue to complete the management of log data.

[0009] Optionally, storing the target metadata corresponding to each log data and the log data in a circular buffer includes:

[0010] Merge several related log lines into a single log line.

[0011] The log data is stored in the circular buffer, and the target metadata corresponding to the log data is stored in the circular buffer using a numerical tagging method.

[0012] Optionally, determining the priority score of each log data based on the target metadata includes:

[0013] The initial priority is determined based on the event type, log source, keyword, and timestamp of the log data; where error-type log data has a higher priority than correct-type log data.

[0014] The priority score is obtained by weighting and summing the weights and initial priorities of the event type, log source, keyword, and timestamp of the log data.

[0015] Optionally, the priority queue includes a first priority queue, a second priority queue, and a third priority queue;

[0016] Accordingly, storing the target pointers corresponding to the log data into the corresponding priority queues based on the priority scores includes:

[0017] If the priority score meets the first preset range, then the target pointer corresponding to the log data is stored in the first priority queue;

[0018] If the priority score meets the second preset range, then the target pointer corresponding to the log data is stored in the second priority queue;

[0019] If the priority score meets the third preset range, then the target pointer corresponding to the log data is stored in the third priority queue.

[0020] Optionally, the method further includes:

[0021] The storage period of the log data corresponding to the target pointer in the third priority queue in the circular buffer is set to the target duration, so as to control the log data corresponding to the target pointer in the third priority queue from being evicted based on the target duration.

[0022] Optionally, determining the queue of items to be evicted in the priority queue based on the current occupancy rate of the circular buffer includes:

[0023] If the current occupancy rate of the circular buffer is greater than the first value and less than the second value, then the first priority queue is determined as the queue to be eliminated.

[0024] If the current occupancy rate of the circular buffer is greater than the second value, then the second priority queue is determined as the queue to be eliminated.

[0025] Optionally, the step of evicting corresponding log data in the circular buffer based on the target pointer in the queue to be evicted includes:

[0026] If the current occupancy rate of the circular buffer is greater than the first value and less than the second value, then the corresponding log data in the circular buffer is evicted based on the target pointer in the first priority queue until the current occupancy rate of the circular buffer is less than the first value or all the log data corresponding to the target pointer in the first priority queue has been evicted.

[0027] If the current occupancy rate of the circular buffer is greater than the second value, the corresponding log data in the circular buffer is evicted based on the target pointer in the second priority queue until the current occupancy rate of the circular buffer is less than the second value or all the log data corresponding to the target pointer in the second priority queue has been evicted.

[0028] Secondly, this application discloses a log data management device, comprising:

[0029] The log data storage module is used to acquire log data and store the target metadata corresponding to each log data and the log data in a circular buffer; the target metadata includes event type, log source, keyword and timestamp;

[0030] The pointer storage module is used to determine the priority score of each log data based on the target metadata, and to store the target pointers corresponding to the log data into the corresponding priority queues based on the priority scores; the target pointer is a pointer or index pointing to the log data stored in the circular buffer;

[0031] The log data eviction module is used to determine the queue of log data to be evicted in the priority queue based on the current occupancy rate of the ring buffer if the remaining space of the ring buffer does not meet the preset conditions, and to evict the corresponding log data in the ring buffer based on the target pointer in the queue of log data to be evicted, so as to complete the management of log data.

[0032] Thirdly, this application discloses an electronic device, comprising:

[0033] Memory, used to store computer programs;

[0034] A processor is used to execute computer programs to implement log data management methods as described above.

[0035] Fourthly, this application discloses a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the log data management method as described above.

[0036] This application first acquires log data and stores the target metadata corresponding to each log data point and the log data itself in a circular buffer. The target metadata includes event type, log source, keyword, and timestamp. Based on the target metadata, a priority score is determined for each log data point. Based on the priority score, target pointers corresponding to each log data point are stored in corresponding priority queues. Each target pointer is a pointer or index pointing to the log data stored in the circular buffer. If the remaining space in the circular buffer does not meet preset conditions, a queue for elimination is determined based on the current occupancy rate of the circular buffer. Based on the target pointers in the queue for elimination, the corresponding log data in the circular buffer is eliminated to complete the management of the log data. Therefore, this application dynamically calculates log priorities by comprehensively evaluating multiple dimensions of logs (such as event type, log source, keyword, and timestamp), intelligently allocating resources and avoiding the problem of indiscriminately discarding logs in the traditional approach. Layered management based on log priorities ensures that high-priority log information is retained when resources are scarce, effectively improving the intelligence and efficiency of log management. Simultaneously employing techniques such as circular buffers and pointer queues, the system can operate efficiently in resource-constrained environments while minimizing the consumption of memory and computing resources. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 This is a flowchart of a log data management method disclosed in this application;

[0039] Figure 2 This is a schematic diagram of the overall structure of a log data management system disclosed in this application;

[0040] Figure 3 This is a schematic diagram of a ring-shaped buffer zone disclosed in this application;

[0041] Figure 4 This is a schematic diagram of a priority queue disclosed in this application;

[0042] Figure 5 This is a schematic diagram of the structure of a log data management device disclosed in this application;

[0043] Figure 6 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] While existing FIFO and LRU log management strategies provide basic log management functionality, they lack the ability to intelligently assess the actual importance of logs. In high-concurrency or resource-constrained scenarios, these strategies often discard critical information, impacting system fault localization, performance optimization, and security auditing. For example, when log buffer space is insufficient, the FIFO strategy discards logs in chronological order of their creation, while the LRU strategy discards logs based on their usage frequency. This can lead to the discarding of important error or anomaly logs, preventing the system from detecting problems in a timely manner. To address these technical problems, this invention discloses a log data management method, apparatus, device, and medium. It enables intelligent priority assessment and eviction based on the actual importance of logs, ensuring that critical logs are effectively retained, thereby improving system stability and performance.

[0046] See Figure 1 As shown in the figure, an embodiment of the present invention discloses a log data management method, including:

[0047] Step S11: Obtain log data and store the target metadata corresponding to each log data and the log data in a circular buffer; the target metadata includes event type, log source, keyword and timestamp.

[0048] In this embodiment, the overall system structure of this application is as follows: Figure 2 As shown, it mainly includes four core modules:

[0049] Log collection module: responsible for obtaining raw logs from the system and storing them in a circular buffer;

[0050] Log evaluation module: Evaluates logs based on a multidimensional dynamic priority scoring model and calculates the priority score for each log entry;

[0051] Log eviction module: Deletes low-priority logs according to a tiered eviction policy to ensure efficient use of buffer resources;

[0052] Log transmission module: periodically sends log data in the buffer to remote storage or file system.

[0053] Specifically, the process begins by acquiring log data and merging several related log lines into a single log entry. This log data is then stored in the circular buffer, and the target metadata corresponding to the log data is stored in the circular buffer using numerical tags. The log acquisition module is responsible for acquiring raw logs from the system or device and merging multiple related log lines into a single log entry (e.g., Java's stack trace). Each log entry will have its metadata, including timestamp, event type, source, keyword matching, and time sensitivity, extracted and stored along with the log entry in a buffer such as... Figure 3 The logs are stored in a circular buffer, as shown. To save memory, this metadata is stored using numerical tags, similar to Google's Protocol Buffers. The use of a circular buffer not only reduces memory usage but also ensures efficient storage and access to logs. Log collection is done line by line, writing to the buffer instead of loading large amounts of log data at once, effectively reducing memory consumption and preventing system crashes. Furthermore, by dynamically calculating log priorities based on a comprehensive evaluation of event type, log source, keywords, and timestamps, resources can be intelligently allocated, avoiding the problem of indiscriminately discarding logs in the past.

[0054] Step S12: Determine the priority score of each log data based on the target metadata, and store the target pointers corresponding to the log data into the corresponding priority queues based on the priority scores; the target pointer is a pointer or index pointing to the log data stored in the circular buffer.

[0055] In this embodiment, after the log collection module obtains the event type, log source, keyword, and timestamp, it determines the initial priority based on the event type, log source, keyword, and timestamp of the log data. Error-type log data has a higher priority than correct-type log data. The priority score is obtained by weighted summation of the weights corresponding to the event type, log source, keyword, and timestamp of the log data, along with the initial priority. Specifically, the core of the log evaluation module is a multi-dimensional dynamic priority scoring model, which calculates the log priority score based on the following dimensions: Event Type: Priority is determined based on the log type (e.g., error, warning, message, etc.), with error logs typically having a higher priority. Log Source: Logs are categorized based on their source, such as system kernel logs, service component logs, and application logs, with kernel logs and critical service component logs having higher priority. Keyword: Priority is calculated by matching specific keywords (e.g., error codes, exception identifiers, etc.) in the logs; important error or exception information is assigned a higher priority. Time sensitivity (Recency): Priority is adjusted based on the timestamp of the logs. Recently generated log information is usually more important than old logs, so logs with high time sensitivity will be retained first.

[0056] Then, each factor is assigned a certain weight (w1, w2, w3, w4), and the final priority score for each log entry is obtained by weighted summation. This scoring process is dynamic and can be adjusted according to actual business needs. In a specific implementation, in a security audit scenario, the weight of the log source can be increased; in a real-time monitoring scenario, the weight of time sensitivity can be increased.

[0057] The calculation formula is as follows:

[0058] .

[0059] After obtaining the priority score, the target pointers corresponding to the log data can be stored in the corresponding priority queues based on the priority score. The priority queues include a first priority queue, a second priority queue, and a third priority queue. Therefore, if the priority score meets a first preset range, the target pointer corresponding to the log data is stored in the first priority queue; if the priority score meets a second preset range, the target pointer is stored in the second priority queue; and if the priority score meets a third preset range, the target pointer is stored in the third priority queue. Specifically, based on the priority score, such as... Figure 4 As shown, a pointer to each log entry is stored in one of three priority queues: high, medium, and low. These three queues only store pointers or indices to the original log entries in the circular buffer, without copying the complete log content. Therefore, each log entry is stored in memory only once. In other words, the priority queues in this application can be high, medium, and low priority queues.

[0060] It should be noted that in this application, the priority queue only stores pointers or indexes to the original logs in the circular buffer, without copying the complete log content. In other words, the data entity is stored in the circular buffer, and the index pointer is stored in the corresponding priority queue. This "index-based queuing" strategy significantly reduces redundant storage and memory consumption. Through this multi-dimensional dynamic priority scoring model, the log priority is not a static label, but a precise priority score calculated in real time based on the actual content and context of the log. This innovative scoring mechanism enables the system to determine which logs need to be retained with priority based on their actual importance, avoiding the shortcomings of traditional indiscriminate log discarding. Hierarchical management based on log priority ensures that high-priority log information is retained when resources are scarce, effectively improving the intelligence and efficiency of log management.

[0061] Step S13: If the remaining space of the circular buffer does not meet the preset conditions, then the priority queue to be eliminated is determined based on the current occupancy rate of the circular buffer, and the corresponding log data in the circular buffer is eliminated based on the target pointer in the queue to complete the management of log data.

[0062] In this embodiment, during system operation, the buffer usage is monitored in real time. When the buffer space is insufficient (e.g., reaching a set memory limit or log quantity limit), logs are evicted based on priority scores. The specific eviction mechanism is as follows: if the current occupancy rate of the circular buffer is greater than a first value and less than a second value, the first priority queue is designated as the queue to be evicted; if the current occupancy rate of the circular buffer is greater than the second value, the second priority queue is designated as the queue to be evicted. The storage period for the log data corresponding to the target pointer in the third priority queue within the circular buffer is set to a target duration, so that the log data corresponding to the target pointer in the third priority queue is not evicted based on the target duration. If the current occupancy rate of the circular buffer is greater than a first value and less than a second value, then the corresponding log data in the circular buffer is evicted based on the target pointer in the first priority queue, until the current occupancy rate of the circular buffer is less than the first value or all the log data corresponding to the target pointer in the first priority queue has been evicted. If the current occupancy rate of the circular buffer is greater than the second value, then the corresponding log data in the circular buffer is evicted based on the target pointer in the second priority queue, until the current occupancy rate of the circular buffer is less than the second value or all the log data corresponding to the target pointer in the second priority queue has been evicted. In a specific embodiment, when the buffer occupancy rate exceeds 80%, logs in the low-priority queue are evicted first, until the buffer occupancy rate drops below 80%, or all logs in the low-priority queue are evicted. When the buffer occupancy rate exceeds 90%, the system further evicts logs in the medium-priority queue, until the buffer occupancy rate drops below 90%, or all logs in the medium-priority queue are evicted. Logs in the high-priority queue are never evicted, ensuring that critical system information is not lost. This tiered eviction strategy ensures that the system always retains the most important data under resource constraints, thereby guaranteeing the reliability of critical log information. This is a significant innovation of the present invention, which makes log collection and eviction no longer indiscriminate, but an intelligent decision based on log content and actual importance.

[0063] Additionally, it should be noted that to ensure efficient system operation and reduce resource consumption, this invention designs a low-resource-consumption mechanism to minimize the impact of the log module on the main business thread. Firstly, the buffer adopts a circular structure, occupying only a fixed size of contiguous memory. The buffer size can be configured according to the total system memory or business needs, for example, set to 5%~10% of the system's available memory, or limiting the maximum number of log entries. Once the circular buffer usage reaches a set threshold, the system releases space according to an eviction policy. Secondly, the priority queue only stores pointers to log nodes in the circular buffer, saving memory and facilitating quick location and deletion of specific logs. By employing technologies such as circular buffers and pointer queues, the system can operate efficiently in resource-constrained environments, minimizing memory and computational resource consumption. It is applicable to resource-constrained environments such as embedded devices and IoT (Internet of Things) terminals, avoiding the high computational overhead of traditional machine learning-based methods.

[0064] In summary, this application first acquires log data and stores the target metadata corresponding to each log data and the log data itself in a circular buffer. The target metadata includes event type, log source, keyword, and timestamp. Based on the target metadata, a priority score is determined for each log data. Based on the priority score, target pointers corresponding to the log data are stored in corresponding priority queues. The target pointer is a pointer or index pointing to the log data stored in the circular buffer. If the remaining space in the circular buffer does not meet preset conditions, a queue for elimination is determined in the priority queue based on the current occupancy rate of the circular buffer. Based on the target pointers in the queue for elimination, the corresponding log data in the circular buffer is eliminated to complete the management of log data. It is evident that this application dynamically calculates log priority by comprehensively evaluating multiple dimensions of logs (such as event type, log source, keyword, and timestamp), intelligently allocating resources and avoiding the problem of indiscriminately discarding logs in the traditional approach. Layered management based on log priority ensures that high-priority log information is retained when resources are scarce, effectively improving the intelligence and efficiency of log management. Simultaneously employing techniques such as circular buffers and pointer queues, the system can operate efficiently in resource-constrained environments while minimizing the consumption of memory and computing resources.

[0065] See Figure 5 As shown, an embodiment of the present invention discloses a log data management device, comprising:

[0066] The log data storage module 11 is used to acquire log data and store the target metadata corresponding to each log data and the log data in a circular buffer; the target metadata includes event type, log source, keyword and timestamp;

[0067] Pointer storage module 12 is used to determine the priority score of each log data based on the target metadata, and to store the target pointers corresponding to the log data into the corresponding priority queues based on the priority scores; the target pointer is a pointer or index pointing to the log data stored in the circular buffer;

[0068] The log data eviction module 13 is used to determine the queue to be evicted in the priority queue based on the current occupancy rate of the ring buffer if the remaining space of the ring buffer does not meet the preset conditions, and to evict the corresponding log data in the ring buffer based on the target pointer in the queue to be evicted, so as to complete the management of log data.

[0069] In summary, this application first acquires log data and stores the target metadata corresponding to each log data and the log data itself in a circular buffer. The target metadata includes event type, log source, keyword, and timestamp. Based on the target metadata, a priority score is determined for each log data. Based on the priority score, target pointers corresponding to the log data are stored in corresponding priority queues. The target pointer is a pointer or index pointing to the log data stored in the circular buffer. If the remaining space in the circular buffer does not meet preset conditions, a queue for elimination is determined in the priority queue based on the current occupancy rate of the circular buffer. Based on the target pointers in the queue for elimination, the corresponding log data in the circular buffer is eliminated to complete the management of log data. It is evident that this application dynamically calculates log priority by comprehensively evaluating multiple dimensions of logs (such as event type, log source, keyword, and timestamp), intelligently allocating resources and avoiding the problem of indiscriminately discarding logs in the traditional approach. Layered management based on log priority ensures that high-priority log information is retained when resources are scarce, effectively improving the intelligence and efficiency of log management. Simultaneously employing techniques such as circular buffers and pointer queues, the system can operate efficiently in resource-constrained environments while minimizing the consumption of memory and computing resources.

[0070] In some specific embodiments, the log data storage module 11 may specifically include:

[0071] The merging unit is used to merge several related lines of log data into a single log data.

[0072] A storage unit is used to store the log data into the circular buffer and to store the target metadata corresponding to the log data into the circular buffer using a digital tagging method.

[0073] In some specific embodiments, the pointer storage module 12 may specifically include:

[0074] An initial priority determination unit is used to determine the initial priority based on the event type, log source, keyword, and timestamp of the log data; wherein log data of error type has a higher priority than log data of correct type.

[0075] The weighted summation unit is used to perform a weighted summation of the event type, log source, keyword, and timestamp of the log data, as well as the initial priority, to obtain the priority score.

[0076] In some specific embodiments, the pointer storage module 12 may specifically include:

[0077] The first pointer storage unit is used to store the target pointer corresponding to the log data into the first priority queue if the priority score meets the first preset range.

[0078] The second pointer storage unit is used to store the target pointer corresponding to the log data into the second priority queue if the priority score meets the second preset range.

[0079] The third pointer storage unit is used to store the target pointer corresponding to the log data into the third priority queue if the priority score meets the third preset range.

[0080] In some specific embodiments, the apparatus may further include:

[0081] The target duration setting module is used to set the storage period of the log data corresponding to the target pointer in the third priority queue in the circular buffer to the target duration, so as to control the log data corresponding to the target pointer in the third priority queue from being evicted based on the target duration.

[0082] In some specific embodiments, the log data eviction module 13 may specifically include:

[0083] The first queue to be eliminated determination unit is used to determine the first priority queue as the queue to be eliminated if the current occupancy rate of the circular buffer is greater than a first value and less than a second value.

[0084] The second queue to be eliminated determination unit is used to determine the second priority queue as the queue to be eliminated if the current occupancy rate of the circular buffer is greater than the second value.

[0085] In some specific embodiments, the log data eviction module 13 may specifically include:

[0086] The first elimination unit is used to eliminate the corresponding log data in the circular buffer based on the target pointer in the first priority queue if the current occupancy rate of the circular buffer is greater than a first value and less than a second value, until the current occupancy rate of the circular buffer is less than the first value or all the log data corresponding to the target pointer in the first priority queue has been eliminated.

[0087] The second elimination unit is used to eliminate the corresponding log data in the circular buffer based on the target pointer in the second priority queue if the current occupancy rate of the circular buffer is greater than the second value, until the current occupancy rate of the circular buffer is less than the second value or all the log data corresponding to the target pointer in the second priority queue has been eliminated.

[0088] Furthermore, embodiments of this application also disclose an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0089] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the log data management method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0090] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0091] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0092] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the log data management method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0093] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed log data management method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0094] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0095] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0096] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0097] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0098] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A log data management method, characterized in that, include: Acquire log data and store the target metadata corresponding to each log data and the log data in a circular buffer; The target metadata includes event type, log source, keywords, and timestamp; Based on the target metadata, a priority score is determined for each log data. Based on the priority score, the target pointers corresponding to the log data are stored in the corresponding priority queues. The target pointer is a pointer or index pointing to the log data stored in the circular buffer. The priority queues include a first priority queue, a second priority queue, and a third priority queue. If the remaining space of the circular buffer does not meet the preset conditions, the priority queue to be eliminated is determined based on the current occupancy rate of the circular buffer, and the corresponding log data in the circular buffer is eliminated based on the target pointer in the queue to complete the management of log data. The determination of priority scores for each log data based on the target metadata includes: The initial priority is determined based on the event type, log source, keyword, and timestamp of the log data; where error-type log data has a higher priority than correct-type log data. The priority score is obtained by weighting and summing the weights and initial priorities of the event type, log source, keyword, and timestamp of the log data. The step of determining the queue of items to be evicted in the priority queue based on the current occupancy rate of the circular buffer includes: If the current occupancy rate of the circular buffer is greater than the first value and less than the second value, then the first priority queue is determined as the queue to be eliminated. If the current occupancy rate of the circular buffer is greater than the second value, then the second priority queue is determined as the queue to be eliminated. The step of eliminating corresponding log data in the circular buffer based on the target pointer in the queue to be eliminated includes: If the current occupancy rate of the circular buffer is greater than the first value and less than the second value, then the corresponding log data in the circular buffer is evicted based on the target pointer in the first priority queue until the current occupancy rate of the circular buffer is less than the first value or all the log data corresponding to the target pointer in the first priority queue has been evicted. If the current occupancy rate of the circular buffer is greater than the second value, then the corresponding log data in the circular buffer is evicted based on the target pointer in the second priority queue, until the current occupancy rate of the circular buffer is less than the second value or all the log data corresponding to the target pointer in the second priority queue has been evicted.

2. The log data management method according to claim 1, characterized in that, The step of storing the target metadata corresponding to each log data and the log data into a circular buffer includes: Merge several related log lines into a single log line. The log data is stored in the circular buffer, and the target metadata corresponding to the log data is stored in the circular buffer using a numerical tagging method.

3. The log data management method according to claim 1 or 2, characterized in that, The step of storing the target pointers corresponding to the log data into the corresponding priority queues based on the priority scores includes: If the priority score meets the first preset range, then the target pointer corresponding to the log data is stored in the first priority queue; If the priority score meets the second preset range, then the target pointer corresponding to the log data is stored in the second priority queue; If the priority score meets the third preset range, then the target pointer corresponding to the log data is stored in the third priority queue.

4. The log data management method according to claim 3, characterized in that, Also includes: The storage period of the log data corresponding to the target pointer in the third priority queue in the circular buffer is set to the target duration, so as to control the log data corresponding to the target pointer in the third priority queue from being evicted based on the target duration.

5. A log data management device, characterized in that, include: A log data storage module is used to acquire log data and store the target metadata corresponding to each log data and the log data into a circular buffer. The target metadata includes event type, log source, keywords, and timestamp; A pointer storage module is used to determine the priority score of each log data based on the target metadata, and to store the target pointers corresponding to the log data into the corresponding priority queues based on the priority scores; the target pointer is a pointer or index pointing to the log data stored in the circular buffer; wherein, the priority queues include a first priority queue, a second priority queue, and a third priority queue; The log data eviction module is used to determine the queue to be evicted in the priority queue based on the current occupancy rate of the ring buffer if the remaining space of the ring buffer does not meet the preset conditions, and to evict the corresponding log data in the ring buffer based on the target pointer in the queue to be evicted, so as to complete the management of log data. The pointer storage module is used to determine the initial priority based on the event type, log source, keyword, and timestamp of the log data; wherein the priority of error type log data is higher than the priority of correct type log data; and to obtain the priority score by weighting and summing the weights corresponding to the event type, log source, keyword, and timestamp of the log data and the initial priority. The log data eviction module is configured to determine the first priority queue as the queue to be evicted if the current occupancy rate of the circular buffer is greater than a first value and less than a second value; and to determine the second priority queue as the queue to be evicted if the current occupancy rate of the circular buffer is greater than the second value. The log data eviction module is configured to, if the current occupancy rate of the circular buffer is greater than a first value and less than a second value, evict the corresponding log data in the circular buffer based on the target pointer in the first priority queue until the current occupancy rate of the circular buffer is less than the first value or all the log data corresponding to the target pointer in the first priority queue has been evictened; if the current occupancy rate of the circular buffer is greater than the second value, evict the corresponding log data in the circular buffer based on the target pointer in the second priority queue until the current occupancy rate of the circular buffer is less than the second value or all the log data corresponding to the target pointer in the second priority queue has been evictened.

6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program to implement the log data management method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, implements the log data management method as described in any one of claims 1 to 4.

Images