Log processing method and device, equipment and medium

Abstract

The invention discloses a log processing method and device, equipment and a medium, and relates to the technical field of log processing.The log processing method includes the steps that after an original log event is obtained, a target log event is screened out through a preset filtering mask, redundant storage of invalid logs can be avoided, and the log processing efficiency can be improved; then, the logs are written into a multi-level storage medium in a grading manner, and a reliable disaster recovery backup system is constructed by utilizing storage characteristics of different levels, so that the logs can be reliably retained in various scenes, and multi-level disaster recovery backup is realized; meanwhile, the storage state is monitored in real time, the storage hierarchy can be dynamically switched when the state exceeds a set range, continuous recording of logs is ensured, the mask filtering mode can be adjusted, and the scene change can be flexibly adapted. In this way, through combination of multi-level storage and dynamic adjustment, the reliability and flexibility of log storage can be improved, a log system can efficiently and stably operate in a complex service and variable environment, and the adaptability is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of log processing technology, and in particular to a log processing method, apparatus, device and medium. Background Art

[0002] In remote device management, the management controller, as a core component, must continuously monitor and record log events. These log events are crucial for maintaining healthy and secure system operations. However, traditional log processing methods often have significant issues: limited disaster recovery capabilities and reliance on fixed storage media or locations. Failure of this media or overflow of storage capacity can result in permanent loss of log data. Furthermore, logging policies are often fixed during the initial design or configuration phase, making them inflexible in complex and changing operational environments and difficult to meet log management requirements. Summary of the Invention

[0003] The present invention provides a log processing method, apparatus, device and medium to at least solve the problem in related technologies that log storage disaster recovery capability is limited and difficult to adapt to dynamic needs.

[0004] The present invention provides a log processing method, comprising: Get the original log event; Filtering the original log events according to a preset filter mask to filter out target log events; Classifying the target log events and writing the classified log events into a storage medium of a corresponding level; the storage medium is divided into at least two levels of storage tiers; The status of each level of storage in the storage medium is monitored, and when the status of any storage is monitored to be out of a set range, the switching of the storage level or the adjustment of the filtering mask is triggered.

[0005] The present invention also provides a log processing device, comprising: Log acquisition module, used to obtain original log events; A log filtering module is used to filter the original log events according to a preset filter mask to filter out target log events; A log grading storage module is used to grade the target log events and write the graded log events into a storage medium of a corresponding level; the storage medium is divided into at least two levels of storage tiers; The risk processing module is used to monitor the status of storage at all levels in the storage medium, and when the status of any storage is monitored to be out of a set range, trigger the switching of the storage level or the adjustment of the filtering mask.

[0006] The present invention also provides an electronic device, comprising: a log source, a filtering engine provided with a filtering mask, a processor, and a storage medium divided into at least two levels of storage layers; The log source is used to generate original log events; The filtering engine is configured to obtain the original log event, filter the original log event using the filtering mask, and filter out the target log event; The processor is used to classify the target log events and write the classified log events to the storage medium of the corresponding level; it is also used to monitor the status of each level of storage in the storage medium, and when the status of any storage is monitored to exceed the set range, it triggers the switching of the storage level or the adjustment of the filtering mask.

[0007] The present invention also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned log processing methods are implemented.

[0008] Through the present invention, after obtaining the original log event, the target log event is accurately filtered out with a preset filter mask, which can avoid redundant storage of invalid logs and improve log processing efficiency; then the log is written into a multi-level storage medium in a hierarchical manner, and the storage characteristics of different levels are used to build a reliable disaster recovery backup system, which can ensure that the log is reliably retained in various scenarios and realize multi-level disaster recovery backup; at the same time, the storage status is monitored in real time, and the storage level can be dynamically switched when the status exceeds the set range to ensure continuous recording of the log, and the filter mask mode can be adjusted to flexibly adapt to scene changes. In this way, through the combination of multi-level storage and dynamic adjustment, the reliability foundation of log storage is strengthened through layered storage, and disaster recovery is achieved in response to risks such as hardware failures and network fluctuations. The log recording strategy can also be flexibly changed with the scene, so that the log system can operate efficiently and stably in complex business and changing environments, significantly improving adaptability and providing more solid and intelligent support for operation and maintenance work such as log management and fault tracing.

[0009] In addition, the present invention also provides a corresponding log processing device, electronic device and computer-readable storage medium for the log processing method, which have the same or corresponding technical features as the above-mentioned log processing method and have the same effects as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1A flow chart of a log processing method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the architecture corresponding to the log processing method provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a log processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0013] It should be noted that, in the description of the present invention, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. The terms "first," "second," etc., in the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence.

[0014] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0015] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the log processing method depends, the specific application environment architecture or specific hardware architecture is described here.

[0016] An embodiment of the present invention provides a log processing method, which is described in detail in conjunction with the execution flow of the log processing method. Figure 1 A flow chart of a log processing method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes: S101: Obtain original log events.

[0017] It should be noted that the above-mentioned raw log events can be generated by log sources. Log sources can include firmware functional modules, hardware sensor monitoring processes, user operation interfaces, etc. All sources that generate log records are collectively referred to as log sources.

[0018] The aforementioned raw log events can include software and hardware event logs and security audit logs. These logs can contain important information such as power status changes, hardware failure alarms, sensor threshold alarms, network status changes, and remote command execution results. Security audit logs can include all security-related events such as access to the server (including the management controller itself), operational activities, and permission changes, for security review and exception tracing.

[0019] S102: Filter the original log events according to a preset filter mask to filter out target log events.

[0020] In practice, the present invention can utilize a filtering engine to perform step S102. This filtering engine can be a dynamic log filtering engine, located at the heart of the log processing pipeline, and used to filter raw log events according to a preset filter mask. The filtering engine maintains a filter mask internally, matching and judging the attributes of log events to determine which events to record and which to ignore. The filter mask is dynamically updateable; the filter mask mode is also dynamically adjustable, enabling real-time adjustment of the log recording range during operation.

[0021] S103: Classify the target log event and write the classified log event into a storage medium of a corresponding level; the storage medium is divided into at least two levels of storage tiers.

[0022] In practice, the present invention can hierarchically store the log events filtered in step S102, storing them in storage media of corresponding levels. That is, the storage media utilizes at least a two-tier storage architecture. By combining at least two tiers of storage, the present invention can balance different performance and persistence media.

[0023] S104: Monitor the status of storage at all levels in the storage medium. When the status of any storage is monitored to be out of a set range, trigger the switching of the storage level or the adjustment of the filter mask.

[0024] It's important to note that disaster recovery ensures that log data is not lost or can be recovered from backups when a system failure occurs (such as storage media damage, power outages, or software crashes). The log processing method of the present invention provides multi-level backup or redundancy, allowing log data to be retained even if the state of any storage device exceeds a set range. This is the so-called disaster recovery requirement.

[0025] The log processing method provided by the embodiment of the present application can avoid the redundant storage of invalid logs, improve the log processing efficiency, and then write the logs into the multi-level storage medium, use the storage characteristics of different levels, build a reliable disaster recovery backup system, ensure the reliable storage of logs in various scenarios, realize multi-level disaster recovery backup, monitor the storage state in real time, dynamically switch the storage level when the state exceeds the set range, ensure continuous log recording, and adjust the mode of the filter mask to flexibly adapt to changes in the scene. In this way, through the combination of multi-level storage and dynamic adjustment, the layered storage can build a reliable foundation for log storage, cope with risks such as hardware failure and network fluctuation to realize disaster recovery, and make the log recording strategy flexible according to the scene, so that the log system can efficiently and stably operate in complex business and variable environment, significantly improve the adaptability, and provide more solid and intelligent support for log management, fault tracing and other operation and maintenance work.

[0026] It should be noted that the above-mentioned log processing method of the present application can be applied to a management controller. The management controller can monitor and control the server hardware state, running health status, power management and the like. The management controller can be a baseboard management controller (BMC). The BMC can be an embedded BMC.

[0027] Further, in the specific implementation, in the above-mentioned log processing method provided by the embodiment of the present application, step S102 filters the original log event according to the preset filter mask to filter out the target log event, which can specifically include: converting the attribute of the original log event into a code, and performing matching calculation with the preset filter mask, and filtering out the target log event according to the calculation result.

[0028] In the implementation, the present application can greatly improve the efficiency and accuracy of log screening by converting the attribute of the original log event into a code form and accurately matching and calculating with the preset filter mask. It not only avoids the tediousness and omissions of manual screening, but also can quickly locate the target event meeting the conditions from the massive logs by means of the logic of code processing and mask matching, effectively reduces the information redundancy interference, and provides accurate and efficient data support for subsequent log analysis, problem troubleshooting and the like.

[0029] Furthermore, in the above steps, the attributes of the original log event are converted into codes, and a matching calculation is performed with a preset filter mask, and the target log event is filtered out according to the calculation result. Specifically, the following steps may be performed: mapping the attributes of the original log event into a bit field for matching; at the same time, defining a filter mask of the same length as the bit field; the value in the filter mask determines whether the corresponding category of logs is allowed to be recorded; performing a bitwise AND operation on the bit field and the filter mask; if the bitwise AND result is non-zero, it indicates that at least one of the attributes contained in the original log event is within the allowable range of the filter mask, and the original log event is determined to have passed the filtering and is used as the target log event; if the bitwise AND result is zero, it indicates that the original log event does not belong to any category allowed by the filter mask, and the original log event is filtered out.

[0030] In practice, the dynamic filtering engine introduced by the present invention realizes fine-grained screening of log events through a configurable mask. The filtering engine pre-maps the attributes of various log events into a bit field (Event Bits) for matching. For example, several bits can be used to represent the level of the log (error, warning, information, debugging, etc.), and other bits can represent the module or category to which the log belongs (power management, network, sensor, security, etc.). At the same time, a binary filtering mask of equal length is defined, in which the value of each bit determines whether the log of the corresponding category is allowed to be recorded. When a log event arrives, the filtering engine performs a bitwise AND operation on the event bit field and the current filtering mask. If the result is not zero, it means that at least one of the attributes contained in the event is within the allowed range of the mask, then it is determined to pass the filtering and the log is submitted for storage; if the bitwise AND result is zero, it means that the event does not belong to any currently allowed category, and it is filtered out and not recorded. This judgment process requires minimal computation (only one bitwise operation and comparison), and can run efficiently in scenarios such as resource-constrained management controllers, avoiding additional burden on system performance. At the same time, flexible attribute matching logic uses different bits of the bit field to correspond to different log attributes (such as level, module, etc.), which can accurately cover multi-dimensional filtering needs, ensuring that only target logs that meet the mask's allowable range are retained, effectively reducing the storage occupancy and subsequent processing costs of invalid logs, and ultimately achieving efficient, accurate and lightweight log management, providing a reliable data foundation for system monitoring and problem troubleshooting, and is very suitable for performance-demanding environments such as management controllers.

[0031] In addition to using bitmasks for log filtering, the present invention can also utilize a rule engine or script-based log filtering framework, allowing the use of Boolean expressions, regular matching, and even contextual association to determine whether to retain logs. Alternatively, a multidimensional filtering matrix based on permissions and sensitivity can be introduced, and the dynamic mask can be expanded into multiple tables or policy sets. For example, after filtering the original log events according to a preset filter mask to filter out the target log events, the target log events can be filtered using one or more of the following methods: Boolean expressions, regular matching, contextual association, and a multidimensional filtering matrix. The filtered log events are then hierarchically stored. This ensures efficient preliminary screening based on bitmask filtering, and then performs refined processing through scalable methods such as Boolean expressions, regular matching, contextual association, and a multidimensional filtering matrix. This not only retains the efficiency of bitmask filtering, but also utilizes diversified filtering logic to meet complex filtering requirements in different scenarios, achieving a progression from rapid preliminary screening to precise in-depth filtering. At the same time, the dynamically expanded policy set allows filtering rules to flexibly adapt to multidimensional scenarios such as permission management and sensitivity classification, effectively improving the flexibility, accuracy, and adaptability of log filtering and reducing invalid data interference.

[0032] It should be noted that the present invention can control the working strategies between the aforementioned filtering engine and various levels of storage, including log writing process control, storage status monitoring, failover processing, and multi-copy synchronization. When the log system receives a new event, the control mechanism drives the filtering engine to make a judgment and distribute the event to one or more storage levels based on the filtering results. It also monitors the health status and space usage of each level of storage and triggers storage switching or data transfer when necessary. Furthermore, the control mechanism is responsible for responding to external commands (such as changing filtering rules, exporting logs, clearing logs, etc.), thereby achieving dynamic controllability of the entire system.

[0033] Furthermore, in a specific implementation, in the above-mentioned log processing method provided in an embodiment of the present invention, before executing step S103 to perform hierarchical processing on the target log event, it may also include: dividing the storage medium into a primary storage for temporarily storing logs within a set time period, a secondary storage for persistently storing logs, and a tertiary storage for providing off-site redundant backup according to volatility, storage location and purpose.

[0034] Figure 2 Schematic diagram of the architecture corresponding to the log processing method provided by the embodiment of the present invention. Figure 2As shown, the present invention can adopt a three-tier storage architecture: the primary storage is high-speed volatile storage (such as the log ring buffer in the BMC memory), which is used to temporarily store recent logs. In other words, the primary storage can be used for in-memory log caching; the secondary storage is local non-volatile storage (such as the BMC's serial peripheral interface flash memory or non-volatile random access memory), which is used to persistently store key logs. In other words, the secondary storage can be used for flash log files; the tertiary storage is remote backup storage (such as the management server that receives and stores logs, or the persistent log area in the host system), which is used to provide off-site redundant backup. In other words, the tertiary storage can be used for remote log backup. In the present invention, the primary storage has the fastest read and write speed but is easily volatile during power outages. The secondary storage has limited capacity but is preserved during power outages. The tertiary storage is not limited to the local storage, which can achieve larger capacity and further disaster recovery backup.

[0035] Correspondingly, step S103 classifies the target log event and writes the classified log event into the storage medium of the corresponding level, which may specifically include: if the target log event is classified into a log to be temporarily stored, the log to be temporarily stored is written into the first-level storage; if the target log event is classified into a log to be persistently stored, the log to be persistently stored is written into the second-level storage; if the target log event is classified into a log to be redundantly backed up in a different location, the log to be redundantly backed up in a different location is written into the third-level storage.

[0036] In implementation, Figure 2 As shown, multiple log source events first enter the dynamic filtering engine and are written to the primary, secondary, and tertiary storage after being filtered by mask rules. Using primary storage for logs that need to be temporarily stored enables fast reading and writing and temporary caching to meet immediate access needs; storing logs that need to be persisted in secondary storage ensures long-term and stable data retention, balancing storage efficiency and reliability; using tertiary storage for logs that require off-site redundant backup strengthens data security through off-site storage, effectively preventing the risk of loss caused by local failures. This hierarchical storage strategy not only achieves precise management of different types of logs and optimizes the rational allocation of storage resources, but also improves the availability, persistence, and security of log data through a layered protection mechanism, providing efficient and reliable support for scenarios such as real-time processing, long-term archiving, and disaster recovery of logs.

[0037] Furthermore, in specific implementation, in the above-mentioned log processing method provided in an embodiment of the present invention, step S104 triggers the switching of the storage level when the status of any storage is monitored to be beyond the set range, which may specifically include: when the status of any storage is monitored to be beyond the set range, obtaining the new log event generated; writing the new log event to the first-level storage; if the writing to the first-level storage is successful, the recording is completed; if the writing to the first-level storage fails, switching to the second-level storage; if the writing to the second-level storage is successful, the recording is completed; if the writing to the second-level storage fails, switching to the third-level storage; if the writing to the third-level storage is successful, the recording is completed; if the writing to the third-level storage fails, it is determined that the recording has failed, and an alarm prompt is triggered.

[0038] In practice, when the health status and / or space usage of any monitored storage device exceeds a set range, the system first captures new log events and attempts to write to the primary storage device. If this fails, it switches to the next level, reporting a failure only when writing to all levels is unsuccessful. This multi-level storage switching logging mechanism maximizes the success rate of recording new log events by progressively attempting to write to primary, secondary, and tertiary storage when a storage anomaly is detected. This progressive redundancy significantly reduces the risk of log loss and avoids a single point of failure for log storage. If writing to tertiary storage fails, an alarm is triggered, alerting operations personnel to intervene and address potential storage system issues. Even if one level of storage media fails, log data is retained at other levels, significantly reducing the risk of critical log loss and ensuring the integrity of the information required for fault analysis and security audits. This overall mechanism ensures the continuity and reliability of logging through the orderly switching of tiered storage and enables rapid response to storage anomalies through alarms. This provides complete and stable log data for system status monitoring and fault tracing, while effectively enhancing the system's fault tolerance for storage failures.

[0039] Furthermore, in a specific implementation, in the log processing method provided in an embodiment of the present invention, when the status of any storage device is monitored to be outside a set range, an adjustment of the filter mask is triggered. Specifically, this may include: when the status of the secondary storage device is monitored to be outside a set range, adjusting the filter mask mode to filter out only logs with a target priority and write them to the secondary storage device, while writing the filtered logs to the primary storage device or the tertiary storage device. When the status of the primary storage device and the tertiary storage device is monitored to be outside a set range, the filter mask mode may also be adjusted to adaptively filter logs, without limitation herein.

[0040] In practice, the synergy between the filtering engine and storage is reflected in the adjustment of the filtering mask mode based on the status of the storage medium. For example, when it detects that the secondary flash storage space is approaching the full threshold, the system can temporarily adjust the filtering mask mode, raising the filtering threshold, retaining only higher-priority logs in the flash memory, and directing more common logs to the primary memory or the tertiary remote, thereby alleviating the pressure on the flash memory. Conversely, when space is abundant or during critical periods, the threshold is lowered to capture as much information as possible. In addition, when the primary storage periodically dumps logs in batches to the secondary storage, the filtering engine can also participate in determining which cached logs need to be persisted. This makes the primary and secondary storage not just a simple master-slave relationship, but a tiered storage strategy that not only ensures the long-term preservation of critical logs, but also reduces overall storage and input and output overhead, ensuring efficient and stable system operation.

[0041] Furthermore, in specific implementation, the above-mentioned log processing method provided in the embodiment of the present invention may also include: when receiving a filter mask update instruction, updating the filter mask in response to the filter mask update instruction; when a sensor alarm or hardware failure is monitored, adjusting the filter mask mode so that the generated log passes the filter.

[0042] In practice, because the filter mask can be dynamically modified as needed, the present invention can adjust the logging policy at any time during operation: for example, temporarily opening the mask bits corresponding to certain modules to record debugging information, or lowering the level threshold to capture more log details when faults are frequent, and then raising the threshold to restore normal operation after the faults are resolved. In practice, updates to the filter mask can be triggered by remote management instructions (for example, an administrator issuing new rules through a graphical user interface or application programming interface) or automatically adjusted by local policies (for example, relaxing the filter upon detecting an increase in error rate). This dynamic mask matching mechanism can suppress the generation of massive amounts of irrelevant logs in daily situations while ensuring complete and detailed logging at critical moments, taking into account both system overhead and troubleshooting needs.

[0043] When the logging system is operating healthily and stably, the present invention reduces the level of log detail by setting the filter mask to a strict mode, allowing only critical events at the warning level and above to pass through. This allows most general information logs to be stored in the primary storage's memory buffer rather than written to flash memory. This reduces the frequency of writes to flash memory, extending its lifespan and saving space, while maintaining low overhead and streamlined logging. When the system exhibits abnormal signs (e.g., multiple consecutive sensor alarms or signs of hardware failure) or enters debug mode, the filter mask can be automatically or by management instruction adjusted to a looser mode (including informational and even debug levels), allowing more detailed logs to pass through the filter and be written to secondary storage, ensuring a complete record of the abnormal process. When a critical fault occurs and causes the management controller to restart, tertiary storage (e.g., a remote server) will also receive a copy of the critical logs in a timely manner. Even if the subsequent restart of the management controller causes the memory logs to be lost, problem diagnosis can still be carried out based on the logs stored in flash memory and remote storage. This multi-level collaborative mechanism ensures a combination of high performance in normal conditions and high security in abnormal conditions, better adapting to the logging needs of different operating scenarios and improving the logging system's responsiveness to environmental changes.

[0044] It should be added that the present invention supports remote centralized management of log policies. The remote operation and maintenance end can obtain log statistics and storage health information in real time through the interface provided by the management controller (such as intelligent platform management interface commands or other), and issue adjustment policies as needed, such as changing the filter mask settings, triggering log archiving or backup, etc. This remote collaboration makes the log system policy from passive and static to active and controllable: when the administrator finds abnormal signs, he can increase the log detail level in advance to capture details, or uniformly issue policy templates during batch deployment to keep the log behavior of all devices consistent. This centralized management improves the efficiency of log operation and maintenance in large-scale deployment environments and reduces the workload of manual debugging of each device.

[0045] The present invention defines a base class, BaseLogStorage, to represent a universal log storage medium interface, providing basic methods such as storage (events). Different levels of storage (memory, flash memory, and remote) inherit from this base class and implement their own storage details. Furthermore, the present invention defines a base class, LogFilterEngine, to encapsulate common log filtering behaviors. The DynamicFilterEngine inherits from it, adding methods such as dynamic mask updates. A LogManager class is responsible for managing and coordinating filter engine instances and multiple base log storage instances, providing a unified log processing interface. The present invention allows for the addition of new storage levels (for example, introducing four-level storage) or new filtering algorithms (for example, employing artificial intelligence algorithms for intelligent filtering) by extending and inheriting the corresponding base classes and integrating them into the LogManager, seamlessly integrating them into the system. This abstract base class and interface design enable pluggable extensions of storage media and filtering algorithms, ensuring the versatility and scalability of log processing methods, and enabling the logging system to be easily ported to different platforms and upgraded as needed.

[0046] It should be noted that the log processing method of the present invention can be used for edge gateways, industrial controllers, and other applications. By utilizing a hierarchical model of local storage and cloud-based log servers, logs can be retrieved in the cloud even when edge devices experience failures. Furthermore, dynamic filtering can adjust the frequency and level of log synchronization based on the edge device's network bandwidth or operating mode (idle / busy), enabling efficient management of cloud-edge collaboration. As data centers scale, the demand for centralized management increases. The present invention can be integrated with higher-level centralized management software to enable remote distribution and unified orchestration of log policies. Administrators can pre-define different log filtering templates for different server batches or node roles and distribute them through the management controller's interface for immediate policy implementation. This centralized distribution ensures consistent log records across large-scale systems and facilitates batch adjustments to log policies for all devices based on new threat intelligence or failure patterns, improving security response speed. The dynamic filtering engine can integrate with machine learning models to automatically identify abnormal patterns and adjust log policies. For example, detailed logging can be automatically enabled when potential failure signs are detected, capturing more data for AI analysis before a failure actually occurs. Conversely, the rich data provided by the log system will improve the accuracy of predictive models, enabling a positive feedback-based intelligent operation and maintenance system.

[0047] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0048] An embodiment of the present invention also provides a log processing device. Figure 3 This is a structural diagram of the log processing device provided by an embodiment of the present invention. This embodiment is based on the perspective of functional modules, such as Figure 3 As shown, the device includes: Log acquisition module 10, used to obtain original log events; The log filtering module 11 is used to filter the original log events according to the preset filter mask to filter out the target log events; The log classification storage module 12 is used to classify the target log events and write the classified log events into the storage medium of the corresponding level; the storage medium is divided into at least two levels of storage tiers; The risk processing module 13 is used to monitor the status of storage at all levels in the storage medium. When the status of any storage is monitored to be out of a set range, it triggers the switching of storage levels or the adjustment of filtering masks.

[0049] In the log processing device provided by the embodiment of the present invention, the interaction of the four modules can first obtain the original log events, and then accurately filter out the target log events with a preset filter mask to avoid redundant storage of invalid logs and improve log processing efficiency; then write the logs into multi-level storage media in a hierarchical manner, and use the storage characteristics of different levels to build a reliable disaster recovery backup system to ensure that the logs are reliably retained in various scenarios and realize multi-level disaster recovery backup; at the same time, the storage status is monitored in real time, and when the status exceeds the set range, the storage level can be dynamically switched to ensure continuous recording of logs, and the filter mask mode can be adjusted to flexibly adapt to changes in the scenario. In this way, through the combination of multi-level storage and dynamic adjustment, the reliability foundation of log storage is strengthened through tiered storage, and disaster recovery is achieved in response to risks such as hardware failures and network fluctuations. The log recording strategy can also be flexibly changed according to the scenario, so that the log system can operate efficiently and stably in complex business and changing environments, significantly improving adaptability and providing more solid and intelligent support for operation and maintenance work such as log management and fault tracing.

[0050] Since the embodiments of the log processing device correspond to the embodiments of the log processing method, the description of the features in the corresponding embodiments of the log processing device can be found in the relevant description of the corresponding embodiments of the log processing method, and will not be repeated here. The same beneficial effects as the aforementioned log processing method are achieved.

[0051] Furthermore, in a specific implementation, in the above-mentioned log processing device provided by an embodiment of the present invention, the log filtering module 11 can be specifically used to convert the attributes of the original log event into a code, and perform a matching calculation with a preset filter mask, and filter out the target log event according to the calculation result. The log filtering module 11 can specifically map the attributes of the original log event to a bit field for matching; at the same time, define a filter mask of the same length as the bit field; the value in the filter mask determines whether the log of the corresponding category is allowed to be recorded; perform a bitwise AND operation on the bit field and the filter mask; if the bitwise AND result is non-zero, it indicates that at least one of the attributes contained in the original log event is within the range allowed by the filter mask, and the original log event is determined to have passed the filter and is used as the target log event; if the bitwise AND result is zero, it indicates that the original log event does not belong to any category allowed by the filter mask, and the original log event is filtered out.

[0052] Furthermore, in a specific implementation, the above-mentioned log processing device provided in an embodiment of the present invention may also include: a storage partitioning module for dividing the storage medium into a primary storage for temporarily storing logs within a set time period, a secondary storage for persistently storing logs, and a tertiary storage for providing off-site redundant backup based on volatility, storage location, and purpose.

[0053] Correspondingly, the log hierarchical storage module 12 can be specifically used to write the log to be temporarily stored into the first-level storage if the target log event is graded into a log to be temporarily stored; if the target log event is graded into a log to be persistently stored, the log to be persistently stored is written into the second-level storage; if the target log event is graded into a log to be redundantly backed up in a different location, the log to be redundantly backed up in a different location is written into the third-level storage.

[0054] Furthermore, in specific implementation, in the above-mentioned log processing device provided in the embodiment of the present invention, the risk processing module 13 can be specifically used to obtain new log events generated when the status of any storage is monitored to be beyond the set range; write the new log event to the primary storage; if the writing to the primary storage is successful, the recording is completed; if the writing to the primary storage fails, switch to the secondary storage; if the writing to the secondary storage is successful, the recording is completed; if the writing to the secondary storage fails, switch to the tertiary storage; if the writing to the tertiary storage is successful, the recording is completed; if the writing to the tertiary storage fails, it is determined that the recording has failed, and an alarm prompt is triggered.

[0055] Furthermore, in specific implementation, in the above-mentioned log processing device provided in an embodiment of the present invention, the risk processing module 13 can also be used to adjust the filtering mask mode when the status of the monitored secondary storage exceeds the set range, so as to filter out only the logs with the target priority and write them to the secondary storage, and write the filtered logs to the primary storage or the tertiary storage.

[0056] Furthermore, in a specific implementation, the log processing device provided in the embodiment of the present invention may further include: a mask updating module, configured to update the filtering mask in response to the filtering mask updating instruction when receiving the filtering mask updating instruction.

[0057] Correspondingly, the risk processing module 13 may also be specifically configured to adjust the filter mask mode when a sensor alarm or hardware failure is detected, so that the generated logs pass the filter.

[0058] An embodiment of the present invention also provides an electronic device, comprising: a log source, a filtering engine provided with a filtering mask, a processor, and a storage medium divided into at least two levels of storage layers; the log source is used to generate original log events; the filtering engine is used to obtain original log events, filter the original log events through the filtering mask, and filter out target log events; the processor is used to classify the target log events and write the classified log events to the storage medium of the corresponding level; and is also used to monitor the status of each level of storage in the storage medium, and when the status of any storage is monitored to exceed the set range, trigger the switching of the storage layer or the adjustment of the filtering mask.

[0059] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned log processing method embodiments when running.

[0060] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0061] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned log processing method embodiments are implemented.

[0062] An embodiment of the present invention further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned log processing method embodiments are implemented.

[0063] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0064] The above is a detailed introduction to the log processing method, device, equipment, and medium provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only intended to help understand the method and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A log processing method, characterized in that: include: Get the original log event; Filtering the original log events according to a preset filter mask to filter out target log events; Classifying the target log events and writing the classified log events into a storage medium of a corresponding level; The storage medium is divided into at least two levels of storage tiers; The status of each level of storage in the storage medium is monitored, and when the status of any storage is monitored to be out of a set range, the switching of the storage level or the adjustment of the filtering mask is triggered.

2. The log processing method according to claim 1, characterized in that: Filter the original log events according to a preset filter mask to filter out target log events, including: The attributes of the original log event are converted into codes, and a matching calculation is performed with a preset filter mask, and the target log event is filtered out according to the calculation result.

3. The log processing method according to claim 2, characterized in that: The attributes of the original log event are converted into codes, and matched with the preset filter mask. The target log event is filtered out according to the calculation result, including: Mapping attributes of the original log event to bit fields for matching; At the same time, a filter mask of the same length as the bit field is defined; the value in the filter mask determines whether the log of the corresponding category is allowed to be recorded; performing a bitwise AND operation on the bit field and the filter mask; If the bitwise AND result is non-zero, it indicates that at least one attribute of the original log event is within the allowable range of the filter mask, and the original log event is determined to pass the filtering and is used as the target log event; If the bitwise AND result is zero, it indicates that the original log event does not belong to any category allowed by the filter mask, and the original log event is filtered out.

4. The log processing method according to claim 1, wherein: Before performing graded processing on the target log event, the method further includes: Based on volatility, storage location, and purpose, storage media are divided into primary storage for temporarily storing logs within a set time period, secondary storage for persistent log storage, and tertiary storage for providing off-site redundant backup. Classifying the target log event and writing the classified log event to a storage medium of a corresponding level, including: If the target log event is classified into a log to be temporarily stored, writing the log to be temporarily stored into the primary storage; If the target log event is classified into a log to be persisted, writing the log to be persisted into the secondary storage; If the target log event is classified into a log to be redundantly backed up in a different location, the log to be redundantly backed up in a different location is written into the tertiary storage.

5. The log processing method according to claim 4, characterized in that: When the monitored status of any storage exceeds the set range, the storage tier switch is triggered, including: When the status of any storage is monitored to be out of the set range, new log events are generated; Write the new log event to the primary storage; if writing to the primary storage succeeds, complete the recording; if writing to the primary storage fails, switch to the secondary storage; If writing into the secondary storage is successful, the recording is completed; if writing into the secondary storage fails, switching to the tertiary storage; If writing into the tertiary storage is successful, the recording is completed; if writing into the tertiary storage fails, the recording is determined to have failed and an alarm prompt is triggered.

6. The log processing method according to claim 4, characterized in that: When the monitored state of any storage exceeds the set range, the filter mask adjustment is triggered, including: When the status of the secondary storage is monitored to be out of a set range, the mode of the filter mask is adjusted to filter out only logs with a target priority and write them to the secondary storage, and write the filtered logs to the primary storage or the tertiary storage.

7. The log processing method according to claim 1, wherein: Also includes: When receiving a filter mask update instruction, updating the filter mask in response to the filter mask update instruction; When a sensor alarm or hardware failure is detected, the filter mask mode is adjusted to allow the generated logs to pass the filter.

8. A log processing device, characterized in that: include: Log acquisition module, used to obtain original log events; A log filtering module is used to filter the original log events according to a preset filter mask to filter out target log events; A log classification storage module is used to classify the target log events and write the classified log events into a storage medium of a corresponding level; The storage medium is divided into at least two levels of storage tiers; The risk processing module is used to monitor the status of storage at all levels in the storage medium, and when the status of any storage is monitored to be out of a set range, trigger the switching of the storage level or the adjustment of the filtering mask.

9. An electronic device, characterized in that: include: A log source, a filtering engine provided with a filtering mask, a processor, and a storage medium divided into at least two levels of storage tiers; The log source is used to generate original log events; The filtering engine is configured to obtain the original log event, filter the original log event using the filtering mask, and filter out the target log event; The processor is configured to classify the target log event and write the classified log event into a storage medium of a corresponding level; It is also used to monitor the status of storage at all levels in the storage medium, and when the status of any storage is monitored to be out of a set range, it triggers the switching of the storage level or the adjustment of the filtering mask.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the log processing method according to any one of claims 1 to 7 are implemented.

Images