A log inspection method, apparatus, device, and storage medium

Abstract

This application relates to the field of data processing technology, specifically to a log inspection method, apparatus, device, and storage medium, aiming to uniformly manage and monitor log data of a distributed storage system. The method includes: reading a log inspection task model, which includes task parameters and corresponding monitoring parameters for log inspection tasks targeting at least one distributed storage component; interfacing with a distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model, the distributed storage underlying interface being used for log data transmission; and executing the log inspection task corresponding to the distributed storage component according to the task parameters and the monitoring parameters, wherein the log data corresponding to the log inspection task is obtained through the distributed storage underlying interface.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a log inspection method, apparatus, device, and storage medium. Background Technology

[0002] A distributed storage system is a system that provides distributed storage services to external cloud platforms. A typical distributed storage system comprises multiple distributed storage components, each generating a large number of service logs daily, requiring unified management. Related technologies address the recording and packaging of service logs in distributed storage systems.

[0003] In related technologies, logs of distributed storage systems are simply recorded and packaged, which cannot effectively manage or monitor logs systematically, and thus cannot guarantee the stable operation of distributed storage services. Summary of the Invention

[0004] This application provides a log inspection method, apparatus, device, and storage medium, which aims to uniformly manage and monitor log data of a distributed storage system.

[0005] A first aspect of this application provides a log inspection method, the method comprising:

[0006] A log reading and inspection task model is provided, which includes task parameters and corresponding monitoring parameters for a log inspection task targeting at least one distributed storage component.

[0007] Connect to the distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model, wherein the distributed storage underlying interface is used for log data transmission;

[0008] Based on the task parameters and the monitoring parameters, the log check task corresponding to the distributed storage component is executed, and the log data corresponding to the log check task is obtained through the underlying interface of the distributed storage.

[0009] Optionally, the steps for creating the log inspection task model include:

[0010] Set the target log for the log inspection task;

[0011] Configure the monitoring items and monitoring parameters corresponding to the log inspection task;

[0012] Configure the task execution strategy corresponding to the log inspection task, the execution strategy including one-time inspection and scheduled inspection;

[0013] Once the target log, the monitoring items, the monitoring parameters, and the task execution strategy are set, the creation of the log inspection task model is completed.

[0014] Optionally, setting the target log corresponding to the log inspection task includes:

[0015] Identify the target component corresponding to the log inspection task;

[0016] Set the log corresponding to the target component as the target log.

[0017] Optionally, setting the monitoring items and monitoring parameters corresponding to the log inspection task includes: setting a log size monitoring item, which is used to monitor the storage space occupied by the log;

[0018] Set up a log packaging monitoring item, which is used to determine whether the logs of the distributed storage component are packaged in accordance with the log packaging strategy corresponding to the distributed storage component.

[0019] Set up a log expiration check monitoring item, which is used to determine whether the distributed storage component cleans up the logs in accordance with the expiration policy corresponding to the distributed storage component.

[0020] Set a log location monitoring item, which is used to monitor whether the logs corresponding to the distributed storage component are in the expected location;

[0021] Set a log format monitoring item, which is used to verify whether the log format of the log corresponding to the distributed storage component is the same as the expected format;

[0022] Set up a log anomaly monitoring item, which is used to monitor whether the logs corresponding to the distributed storage component are running normally;

[0023] Set the corresponding monitoring parameters for each of the log monitoring items.

[0024] Optionally, the underlying distributed storage interface corresponding to at least one log inspection task in the log inspection task model includes:

[0025] Determine the underlying distributed storage interface corresponding to the log inspection task;

[0026] Establish a data connection with the underlying interface of the distributed storage.

[0027] Optionally, executing the log inspection task corresponding to the distributed storage component based on the task parameters and the monitoring parameters includes:

[0028] Based on the task parameters, determine the execution strategy corresponding to the log checking task;

[0029] When the execution strategy is a one-time check, the logs corresponding to the distributed storage component are obtained from the underlying interface of the distributed storage component.

[0030] Based on the task parameters, determine the monitoring items corresponding to the log inspection task;

[0031] Call the log processing function corresponding to the monitored item from the preset log processing function library;

[0032] The monitoring parameters corresponding to the log inspection task and the log parameters of the log are read through the log processing function.

[0033] The log processing function compares the log parameters and the monitoring parameters to obtain the comparison result.

[0034] If the comparison results indicate that the log parameters and the monitoring parameters are inconsistent, the corresponding abnormal alarm information will be output.

[0035] Optionally, the method further includes:

[0036] When the execution strategy is a timed check, a preset time interval is determined for the log check task;

[0037] The log check task is executed periodically according to the preset time interval;

[0038] The check results are reported after each log check task is completed.

[0039] A second aspect of this application provides a log inspection apparatus, the apparatus comprising:

[0040] The model reading module is used to read the log inspection task model, which includes task parameters and corresponding monitoring parameters for log inspection tasks targeting at least one distributed storage component.

[0041] The interface interface module is used to interface with the distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model.

[0042] The task execution module is used to execute the log inspection task corresponding to the distributed storage component according to the task parameters and the monitoring parameters. The log data corresponding to the log inspection task is obtained through the underlying interface of the distributed storage.

[0043] Optionally, the steps for creating the log inspection task model include:

[0044] Set the target log for the log inspection task;

[0045] Configure the monitoring items and monitoring parameters corresponding to the log inspection task;

[0046] Configure the task execution strategy corresponding to the log inspection task, the execution strategy including one-time inspection and scheduled inspection;

[0047] Once the target log, the monitoring items, the monitoring parameters, and the task execution strategy are set, the creation of the log inspection task model is completed.

[0048] Optionally, setting the target log corresponding to the log inspection task includes:

[0049] Identify the target component corresponding to the log inspection task;

[0050] Set the log corresponding to the target component as the target log.

[0051] Optionally, setting the monitoring items and monitoring parameters corresponding to the log inspection task includes: setting a log size monitoring item, which is used to monitor the storage space occupied by the log;

[0052] Set up a log packaging monitoring item, which is used to determine whether the logs of the distributed storage component are packaged in accordance with the log packaging strategy corresponding to the distributed storage component.

[0053] Set up a log expiration check monitoring item, which is used to determine whether the distributed storage component cleans up the logs in accordance with the expiration policy corresponding to the distributed storage component.

[0054] Set a log location monitoring item, which is used to monitor whether the logs corresponding to the distributed storage component are in the expected location;

[0055] Set a log format monitoring item, which is used to verify whether the log format of the log corresponding to the distributed storage component is the same as the expected format;

[0056] Set up a log anomaly monitoring item, which is used to monitor whether the logs corresponding to the distributed storage component are running normally;

[0057] Set the corresponding monitoring parameters for each of the log monitoring items.

[0058] Optionally, the underlying distributed storage interface corresponding to at least one log inspection task in the log inspection task model includes:

[0059] Determine the underlying distributed storage interface corresponding to the log inspection task;

[0060] Establish a data connection with the underlying interface of the distributed storage.

[0061] Optionally, the task execution module includes:

[0062] The execution strategy determination submodule is used to determine the execution strategy corresponding to the log inspection task based on the task parameters.

[0063] The log acquisition submodule is used to acquire the logs corresponding to the distributed storage component from the underlying interface of the distributed storage component when the execution strategy is a one-time check.

[0064] The monitoring item determination submodule is used to determine the monitoring items corresponding to the log inspection task based on the task parameters.

[0065] The function call submodule is used to call the log processing function corresponding to the monitored item from the preset log processing function library;

[0066] The parameter reading submodule is used to read the monitoring parameters corresponding to the log inspection task and the log parameters of the log through the log processing function;

[0067] The comparison result acquisition submodule is used to compare the log parameters and the monitoring parameters through the log processing function to obtain the comparison result;

[0068] The abnormal alarm information output submodule is used to output corresponding abnormal alarm information when the comparison result indicates that the log parameters and the monitoring parameters are inconsistent.

[0069] Optionally, the device further includes:

[0070] The time interval determination submodule is used to determine the preset time interval corresponding to the log check task when the execution strategy is a timed check;

[0071] The task execution submodule is used to periodically execute the log check task according to the preset time interval;

[0072] The inspection result reporting submodule is used to report the inspection results after each log inspection task is completed.

[0073] A third aspect of this application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in the first aspect of this application.

[0074] A fourth aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in the first aspect of this application.

[0075] The log inspection method provided in this application reads a log inspection task model, which includes task parameters and corresponding monitoring parameters for log inspection tasks targeting at least one distributed storage component; it then interfaces with the underlying distributed storage interface corresponding to at least one log inspection task in the log inspection task model, the underlying distributed storage interface being used for log data transmission; and finally, it executes the log inspection task corresponding to the distributed storage component based on the task parameters and the monitoring parameters, wherein the log data corresponding to the log inspection task is obtained through the underlying distributed storage interface.

[0076] In this method, a log inspection task model is set up, which includes log inspection tasks and monitoring parameters for each distributed storage component. By connecting to the underlying interface of the distributed storage, the log data of each distributed storage component is obtained. According to the task parameters and monitoring parameters in the task model, the corresponding log inspection task is executed, so that the distributed storage system can coordinate the logs generated by all services and ensure the normal operation of user services. Attached Figure Description

[0077] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 This is a flowchart of a log checking method proposed in an embodiment of this application;

[0079] Figure 2 This is a schematic diagram of a log inspection process proposed in an embodiment of this application;

[0080] Figure 3This is a schematic diagram of a log checking device according to an embodiment of this application;

[0081] Figure 4 This is a schematic diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0082] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0083] refer to Figure 1 , Figure 1 This is a flowchart of a log checking method proposed in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0084] S11: Read log inspection task model, which includes task parameters and corresponding monitoring parameters for log inspection tasks targeting at least one distributed storage component.

[0085] In this embodiment, the log inspection task model is established in the settings page of the distributed storage system server. It includes log inspection tasks and corresponding monitoring parameters for each component in the distributed storage system. A distributed storage system is a data storage technology that distributes data across multiple independent devices, using a network to create a virtual storage device from these distributed resources. This system utilizes the disk space on each machine within an enterprise, allowing data to be stored throughout the organization, thereby improving data reliability and availability. The task parameters include specific information about the distributed storage components to be monitored, specific monitoring strategies, etc. The monitoring parameters are the normal parameters of the logs corresponding to each distributed storage component. Distributed storage components are the components in a distributed storage system that implement various functions.

[0086] In this embodiment, the server first reads the log inspection task model, which includes task parameters and corresponding monitoring parameters for log inspection tasks targeting at least one distributed storage component.

[0087] S12: Connect to the distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model, wherein the distributed storage underlying interface is used for the transmission of log data.

[0088] In this embodiment, the distributed storage underlying interface is used for the transmission of log data.

[0089] In this embodiment, after receiving the log inspection task model, the server determines the distributed storage underlying interface corresponding to each log inspection task in the log inspection task model, and then interfaces with each distributed storage underlying interface to ensure data transmission between the server and the corresponding components.

[0090] In this embodiment, the specific steps for interfacing with the distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model include:

[0091] S12-1: Determine the underlying distributed storage interface corresponding to the log inspection task.

[0092] In this embodiment, each log inspection task corresponds to a distributed storage component. The server calls the log data corresponding to the distributed storage component through the underlying interface of the distributed storage, and calls the corresponding log processing tool to process the log data.

[0093] In this embodiment, the server first determines the distributed storage component that the log inspection task needs to inspect, and then determines the underlying distributed storage interface corresponding to the distributed storage component.

[0094] S12-2: Establish a data connection with the underlying interface of the distributed storage.

[0095] In this embodiment, after determining the distributed storage component and the underlying interface of the distributed storage corresponding to the log inspection task, the server establishes a data connection with the underlying interface of the distributed storage.

[0096] S13: Execute the log check task corresponding to the distributed storage component according to the task parameters and the monitoring parameters. The log data corresponding to the log check task is obtained through the underlying interface of the distributed storage.

[0097] In this embodiment, after determining the task parameters and monitoring parameters, a log check task corresponding to each distributed storage component is executed. When executing the log check task, the log data of the distributed storage component needs to be obtained through the underlying interface of the distributed storage, the corresponding log processing tool is called to analyze and process the log data, the processing result is obtained in combination with the monitoring parameters, and then it is decided whether to perform alarm processing, thereby completing the execution of the log check task corresponding to the distributed storage component.

[0098] In this embodiment, the specific steps for executing the log inspection task corresponding to the distributed storage component based on the task parameters and the monitoring parameters include:

[0099] S13-1: Determine the execution strategy corresponding to the log checking task based on the task parameters.

[0100] In this embodiment, the task parameters include the execution strategy of the log checking task. The execution strategy includes one-time checks and timed checks. One-time checks means that the log checking task is executed only once at the current moment and the task result is returned. Timed checks mean that a check is performed every preset time period and the check result is returned.

[0101] In this embodiment, the server first determines the execution strategy corresponding to each log inspection task based on the task parameters in the log inspection task model.

[0102] S13-2: When the execution strategy is a one-time check, obtain the logs corresponding to the distributed storage component from the underlying interface of the distributed storage component.

[0103] In this embodiment, when the execution strategy of the log inspection task is a one-time inspection, the server obtains the log data corresponding to the distributed storage component from the underlying interface of the distributed storage component. The log data is stored in the underlying storage device of the distributed storage system, and the log data of each component is obtained from the underlying storage device through the underlying storage interface of the distributed system.

[0104] S13-3: Determine the monitoring items corresponding to the log inspection task based on the task parameters.

[0105] In this embodiment, the task parameters include monitoring items corresponding to each log inspection task. When the task parameters of any log inspection task are obtained from the log inspection task model, the monitoring items corresponding to that log inspection task are read from the task parameters.

[0106] For example, if the log inspection task is to check whether the log data of a certain component is packaged correctly, then the monitoring item is the packaging strategy monitoring.

[0107] S13-4: Call the log processing function corresponding to the monitored item in the preset log processing function library.

[0108] In this embodiment, the log processing function is a function used to process log data. It is a function that is pre-written according to the content that needs to be compared for each monitoring item. These functions are stored in the log processing function library and can be called at any time.

[0109] In this embodiment, after determining the monitoring item corresponding to the log inspection task, the corresponding log processing function is called in the preset log processing function library.

[0110] For example, if the monitored item is log packaging monitoring, then the log packaging monitoring function is called.

[0111] S13-5: Read the monitoring parameters corresponding to the log inspection task and the log parameters of the log through the log processing function.

[0112] In this embodiment, log parameters are the specific values ​​of various types of data contained in the log, such as the size of each log data packet, the total size of the log data, and the time of log data packaging.

[0113] In this embodiment, after calling the log processing function, the monitoring parameters corresponding to the log inspection task and the log parameters of the log are read through the log processing function.

[0114] S13-6: The log parameters and the monitoring parameters are compared using the log processing function to obtain the comparison result.

[0115] In this embodiment, corresponding log checking tasks and monitoring parameters are pre-set for each component. After the server obtains the log data from the distributed storage component, it determines whether there are differences between the log parameters and monitoring parameters in the log data. This comparison is typically performed using log processing functions to obtain the comparison results.

[0116] For example, a log expiration detection task can compare whether logs are deleted according to a preset time.

[0117] S13-7: If the comparison result indicates that the log parameters and the monitoring parameters are inconsistent, output the corresponding abnormal alarm information.

[0118] In this embodiment, the abnormal alarm information is used to inform the user that the log data is abnormal.

[0119] In this embodiment, if the comparison results indicate that the log parameters and monitoring parameters are inconsistent, the corresponding abnormal alarm information is output.

[0120] For example, for a log expiration detection task, if the comparison result indicates that the logs have not been deleted according to the preset time, an abnormal alarm message is issued to notify the user that the logs have not been deleted according to the preset time.

[0121] For example, the log monitoring task is performed using the logcheck service (a tool for analyzing system log files, primarily used to detect potential security risks and other abnormal activities, and to notify specified users of the detection results via email. Logcheck identifies potential security issues and abnormal behaviors by filtering keywords in log files, thereby helping system administrators to discover and handle these issues in a timely manner). The core functions are log policy judgment and log alerts.

[0122] For example, taking log packaging monitoring strategies as an example, log packaging strategies are generally divided into daily packaging or packaging exceeding a fixed size. `logcheckTar` (a processing function in the log detection tool) inherits from and implements the `check` method provided by the abstract class `logcheckBase` (another processing function in the log detection tool). After the logcheck logging framework generates a `LogCheckTarEvent`, it obtains the log packaging strategy and submits it to `LogcheckTarTask` (the processing function in the log detection tool) for task processing. In `LogcheckTarTask`, the log packaging strategy is processed, and then compared with the current number or size of log compressed packages. The final result is output to determine whether it meets the pre-set packaging strategy. If it does not meet the pre-set packaging strategy, the corresponding alarm information is output to inform the user that log packaging has encountered an error. The specific code is as follows: `public class LogcheckTar` <e>extends LogcheckBase <e>{

[0123] @Override

[0124] protected void check(E logPolicy) {

[0125] LogCheckTarEvent event = filter(logPolicy);

[0126] LogCheck.getExecutor().submit(new LogcheckTarTask(event,LogNodeFactory.compareTarLog()));

[0127] }

[0128] }

[0129] By running the above code, a log packaging monitoring strategy can be implemented.

[0130] When the log policy comparison results do not meet expectations, the LogcheckTarTask result will trigger the log_alarm (an event monitor used to trigger alarms when a specific regular expression is matched in the log) service. The log_alarm service will format the exception information and send it to the interface to display the exception information.

[0131] In this embodiment, by setting up a log inspection task model, the log data of each distributed storage component in the distributed storage system is monitored, which enhances the scalability and maintainability of the logs, realizes unified management of the logs, and ensures the normal operation of the business.

[0132] In this embodiment, the method further includes:

[0133] S13-8: When the execution strategy is a timed check, determine the preset time interval corresponding to the log check task.

[0134] In this embodiment, when the execution strategy is timed checking, the preset time interval corresponding to the log checking task is determined.

[0135] S13-9: Execute the log check task periodically according to the preset time interval.

[0136] In this embodiment, after determining the preset time interval, the log data of the corresponding distributed storage component is obtained periodically according to the preset time interval, the log parameters are compared with the monitoring parameters to obtain the comparison results, and the log check task is executed periodically.

[0137] S13-10: Report the inspection results after each log check task is completed.

[0138] In this embodiment, when checking the logs periodically, the check result is reported after each log check task is completed. If a fault is found, an alarm message is issued; if there is no fault, a log normal message is sent.

[0139] In another embodiment of this application, the steps for creating the log inspection task model include:

[0140] S21: Set the target log for the log checking task.

[0141] In this embodiment, the target log is the log that needs to be obtained.

[0142] In this embodiment, when constructing the log inspection task model, the target log is first selected. Generally, the target log is all the logs supported in the log service list of the distributed storage system.

[0143] For example, select the distributed storage components that need log checking, including: OSD (Object Storage Device node, playing a key role in the distributed storage system, mainly responsible for storing and managing data. Each OSD node manages a portion of the data and uses the CRUSH algorithm to achieve data redundancy backup and load balancing), service logs; MON (monitoring nodes in the distributed storage system, mainly responsible for cluster status monitoring, management, and maintenance. They ensure the health of the cluster and handle various cluster events, such as adding or deleting storage nodes), service logs; MDS (Metadata Server node, responsible for managing metadata in the cluster, including the directory structure of the file system, file permissions, and the mapping relationship between files and directories. The health of the MDS node is crucial to the stability and reliability of the entire distributed storage cluster), service logs; CTDB service logs (containing the services of each child process); Lib (server library, a software resource library stored on a server. The server library contains a variety of software components, tools, and functional modules to support the operation and management of the server), logs; and protocol logs (file, block, object, and HDFS (Distributed File System designed to run on commodity hardware) logs. HDFS (System) shares many similarities with existing distributed file systems. However, its differences from other distributed file systems are also significant. HDFS is a highly fault-tolerant system, suitable for deployment on inexpensive machines. HDFS provides high-throughput data access, making it ideal for applications with large-scale datasets. Select the server components that need log inspection, such as message logs. You can also customize the logs of other services on the monitoring server.

[0144] In this embodiment, the specific steps for setting the target log corresponding to the log inspection task include:

[0145] S21-1: Determine the target component corresponding to the log inspection task.

[0146] In this embodiment, the server first determines the target component corresponding to the log inspection task. The task parameters of the log inspection task include the distributed storage component to be inspected, i.e. the target component. The corresponding component information can be obtained by reading the task parameters.

[0147] S21-2: Set the log corresponding to the target component as the target log.

[0148] In this embodiment, after the target component is determined, the log corresponding to the target component is set as the target log.

[0149] S22: Set the monitoring items and monitoring parameters corresponding to the log inspection task.

[0150] In this embodiment, the monitoring item corresponding to the log inspection task is the item that needs to be monitored.

[0151] In this embodiment, after selecting the target log, the monitoring items and monitoring parameters corresponding to the log inspection task are set. The specific steps include:

[0152] S22-1: Set a log size monitoring item, which is used to monitor the storage space occupied by the log.

[0153] In this embodiment, the log size monitoring item is one of the monitoring items. When setting the log size monitoring item, the monitoring value of the log size can be entered. The server supports importing and exporting the expected log size value of each server. When the actual log size is greater than the monitoring value, an alarm message is sent to inform the user that the log capacity is abnormal, thereby realizing the monitoring of the storage space occupied by the log.

[0154] S22-2: Set a log packaging monitoring item, which is used to determine whether the logs of the distributed storage component are packaged according to the log packaging strategy corresponding to the distributed storage component.

[0155] In this embodiment, log packaging monitoring is one type of monitoring item. Different services have different packaging strategies. The server supports importing and exporting the packaging strategies of the services corresponding to each distributed storage component in the distributed storage system (e.g., packaging log data exceeding the preset space capacity, packaging log data for each preset time period, etc.). According to the packaging strategy, the server monitors whether the distributed storage system packages the log data according to the corresponding packaging strategy, whether the packaging is successful, and whether the space occupied by the packaged logs meets expectations. If there is an anomaly, an anomaly alarm message is sent to inform the user of the log packaging anomaly.

[0156] S22-3: Set up a log expiration check monitoring item. The log expiration check monitoring item is used to determine whether the distributed storage component cleans up the logs in accordance with the expiration policy corresponding to the distributed storage component.

[0157] In this embodiment, log expiration check monitoring is one of the monitoring items. Some log data is set with expiration deletion or periodic cleanup policies. The server supports importing and exporting the expiration policies of the services corresponding to each distributed storage component (such as deleting log data that exceeds the preset capacity, deleting log data at preset time intervals, etc.). Based on the uploaded expiration policy, the server monitors whether the log is cleaned up in accordance with the expiration policy. If the log data is not cleaned up in accordance with the expiration policy, the server sends the corresponding abnormal alarm information to inform the user that the log has not been cleaned up in accordance with the expiration policy.

[0158] S22-4: Set a log location monitoring item, which is used to monitor whether the logs corresponding to the distributed storage component are in the expected location.

[0159] In this embodiment, the log location monitoring item is one of the monitoring items in the distributed storage system. Each distributed storage component corresponds to a different service with a different storage location. The server supports importing and exporting the log location information of each service. Based on the log location information, the server monitors whether the log data corresponding to the distributed storage component is in the expected location (mainly to verify the log location when the storage upgrade function changes the log storage structure, to prevent the log from not being recorded normally or being recorded in the wrong location). If the log data is not in the expected storage location, the server sends the corresponding alarm information to inform the user that the log storage location is incorrect.

[0160] S22-5: Set a log format monitoring item, which is used to verify whether the log format of the log corresponding to the distributed storage component is the same as the expected format.

[0161] In this embodiment, the log format monitoring item is one of the monitoring items in the distributed storage system. The log format of the service corresponding to each distributed storage component is different. The server supports importing and exporting the expected log format of each service. The server monitors the log format by verifying whether the log format of the log corresponding to the distributed storage component is the same as the expected format. If the log format is different from the expected format, an alarm message is sent to inform the user that the log format is incorrect.

[0162] S22-6: Set up a log anomaly monitoring item, which is used to monitor whether the logs corresponding to the distributed storage component are running normally.

[0163] In this embodiment, there are other log services in the distributed storage system. Corresponding log anomaly monitoring items can be set for these services to monitor whether the logs of the distributed storage components are running normally, whether they are continuously recording, and whether the changes in log levels are correct. If any log service is abnormal, corresponding alarm information will be generated to inform the user of the log anomaly.

[0164] S22-7: Set the corresponding monitoring parameters for each of the log monitoring items.

[0165] In this embodiment, after setting up the log monitoring items, corresponding monitoring parameters are set for each log monitoring item.

[0166] For example, the time interval in the expiration policy corresponding to log expiration detection is set to 10 minutes, and the data size in the log packaging policy is set to 100MB.

[0167] S23: Set the task execution strategy corresponding to the log inspection task. The execution strategy includes one-time inspection and timed inspection.

[0168] In this embodiment, after setting the monitoring items and monitoring parameters, the corresponding task execution strategy is set, including one-time checks and timed checks. For each monitoring item, a corresponding execution strategy can be set. The strategy can be set according to the characteristics of the monitoring item. For example, the log anomaly monitoring item can be set to timed checks to ensure that log anomalies can be detected in a timely manner.

[0169] S24: Once the target log, the monitoring items, the monitoring parameters, and the task execution strategy are set, the creation of the log inspection task model is completed.

[0170] In this embodiment, a log inspection task model is set up on the server side to manage and configure the log inspection tasks in a unified manner. By loading the log inspection task model, log monitoring can be performed on various components of the distributed storage system, thus realizing unified monitoring of logs of various components of the distributed storage system.

[0171] refer to Figure 2 , Figure 2 This is a schematic diagram of a log inspection process proposed in an embodiment of this application, as shown below. Figure 2 As shown, users can select services on the server, select all logs supported by the distributed storage system, then select the corresponding system service item, select the corresponding component service item, and then configure custom services. Custom monitoring items include log size monitoring, log packaging monitoring, log expiration detection, log location monitoring, log format monitoring, and log anomaly detection. The system also supports importing and exporting monitoring policies, and setting each monitoring task to be a real-time task or a scheduled task. Finally, it generates anomaly alerts based on the monitored log data.

[0172] In another embodiment of this application, when the server detects that a component of the distributed storage system has been expanded or changed, it obtains the component information of the newly added or changed component, sets a corresponding log check task for the distributed storage component according to the service corresponding to the distributed storage component, and adds the log check task and the corresponding monitoring parameters into the log check task model, and dynamically updates the model to ensure that each distributed storage component in the distributed storage system can be monitored at all times, thereby ensuring the normal operation of the distributed storage system's business.

[0173] In another embodiment of this application, after each log check task is completed, the server records the corresponding log check result data and adjusts the log monitoring strategy based on the log check result data. For example, if the check result data of a certain log check task indicates that its log data is normal over a long period of time, the time interval of the log check task can be lengthened to save server resources. If the check result data of a certain log check task has many errors, the time interval of the log check task can be shortened to strengthen the monitoring of the log data of the distributed storage component and ensure timely alarms for log data errors. When a log check task is a one-time check task and an error is detected in the log, an alarm is issued and the log check task is changed to a scheduled check until the corresponding log data has not shown any errors for a long time. Then, the log check task can be changed back to a one-time check. By dynamically adjusting the execution strategy of the log check task based on the log data check results, comprehensive monitoring of the log data of the distributed storage component is ensured, and the stable operation of the system is guaranteed while saving server resources.

[0174] In the above embodiments of this application, by constructing a log inspection task model for a distributed storage system, unified monitoring of the log data of the services of each distributed storage component in the distributed storage system is achieved. Anomalies in the logs are promptly alerted, and the log inspection task model is dynamically updated according to the update status of the distributed storage system. This enhances the scalability and maintainability of the logs, prevents other chain system problems, and ensures the normal operation of the distributed storage system's business.

[0175] Based on the same inventive concept, one embodiment of this application provides a log checking device. (Reference) Figure 3 , Figure 3 This is a schematic diagram of a log checking device 300 according to an embodiment of this application. Figure 3 As shown, the device includes:

[0176] The model reading module 301 is used to read the log inspection task model, which includes task parameters and corresponding monitoring parameters for log inspection tasks targeting at least one distributed storage component.

[0177] The interface interface module 302 is used to interface with the distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model. The distributed storage underlying interface is used for the transmission of log data, and the log data corresponding to the log inspection task is obtained through the distributed storage underlying interface.

[0178] The task execution module 303 is used to execute the log inspection task corresponding to the distributed storage component according to the task parameters and the monitoring parameters. The log data corresponding to the log inspection task is obtained through the underlying interface of the distributed storage.

[0179] Optionally, the steps for creating the log inspection task model include:

[0180] Set the target log for the log inspection task;

[0181] Configure the monitoring items and monitoring parameters corresponding to the log inspection task;

[0182] Configure the task execution strategy corresponding to the log inspection task, the execution strategy including one-time inspection and scheduled inspection;

[0183] Once the target log, the monitoring items, the monitoring parameters, and the task execution strategy are set, the creation of the log inspection task model is completed.

[0184] Optionally, setting the target log corresponding to the log inspection task includes:

[0185] Identify the target component corresponding to the log inspection task;

[0186] Set the log corresponding to the target component as the target log.

[0187] Optionally, setting the monitoring items and monitoring parameters corresponding to the log inspection task includes: setting a log size monitoring item, which is used to monitor the storage space occupied by the log;

[0188] Set up a log packaging monitoring item, which is used to determine whether the logs of the distributed storage component are packaged in accordance with the log packaging strategy corresponding to the distributed storage component.

[0189] Set up a log expiration check monitoring item, which is used to determine whether the distributed storage component cleans up the logs in accordance with the expiration policy corresponding to the distributed storage component.

[0190] Set a log location monitoring item, which is used to monitor whether the logs corresponding to the distributed storage component are in the expected location;

[0191] Set a log format monitoring item, which is used to verify whether the log format of the log corresponding to the distributed storage component is the same as the expected format;

[0192] Set up a log anomaly monitoring item, which is used to monitor whether the logs corresponding to the distributed storage component are running normally;

[0193] Set the corresponding monitoring parameters for each of the log monitoring items.

[0194] Optionally, the underlying distributed storage interface corresponding to at least one log inspection task in the log inspection task model includes:

[0195] Determine the underlying distributed storage interface corresponding to the log inspection task;

[0196] Establish a data connection with the underlying interface of the distributed storage.

[0197] Optionally, the task execution module includes:

[0198] The execution strategy determination submodule is used to determine the execution strategy corresponding to the log inspection task based on the task parameters.

[0199] The log acquisition submodule is used to acquire the logs corresponding to the distributed storage component from the underlying interface of the distributed storage component when the execution strategy is a one-time check.

[0200] The monitoring item determination submodule is used to determine the monitoring items corresponding to the log inspection task based on the task parameters.

[0201] The function call submodule is used to call the log processing function corresponding to the monitored item from the preset log processing function library;

[0202] The parameter reading submodule is used to read the monitoring parameters corresponding to the log inspection task and the log parameters of the log through the log processing function;

[0203] The comparison result acquisition submodule is used to compare the log parameters and the monitoring parameters through the log processing function to obtain the comparison result;

[0204] The abnormal alarm information output submodule is used to output corresponding abnormal alarm information when the comparison result indicates that the log parameters and the monitoring parameters are inconsistent.

[0205] Optionally, the device further includes:

[0206] The time interval determination submodule is used to determine the preset time interval corresponding to the log check task when the execution strategy is a timed check;

[0207] The task execution submodule is used to periodically execute the log check task according to the preset time interval;

[0208] The inspection result reporting submodule is used to report the inspection results after each log inspection task is completed.

[0209] Based on the same inventive concept, another embodiment of this application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the log checking method as described in any of the above embodiments of this application.

[0210] Based on the same inventive concept, another embodiment of this application provides an electronic device. Figure 4 This is a schematic diagram of an electronic device 400 according to an embodiment of this application, including a memory 402, a processor 401, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps in the log checking method described in any of the above embodiments of this application.

[0211] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0212] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0213] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0214] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0215] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0216] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0217] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0218] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0219] The log inspection method, apparatus, device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.< / e> < / e>

Claims

1. A log inspection method, characterized in that, The method includes: A log inspection task model is provided, which includes task parameters and corresponding monitoring parameters for a log inspection task targeting at least one distributed storage component. Connect to the distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model, wherein the distributed storage underlying interface is used for log data transmission; Execute the log check task corresponding to the distributed storage component based on the task parameters and the monitoring parameters; The steps for creating the log inspection task model include: Set the target log for the log inspection task; Setting the monitoring items and monitoring parameters corresponding to the log inspection task includes: setting a log size monitoring item, which is used to monitor the storage space capacity occupied by the log; Set up a log packaging monitoring item, which is used to determine whether the logs of the distributed storage component are packaged in accordance with the log packaging strategy corresponding to the distributed storage component. Set up a log expiration check monitoring item, which is used to determine whether the distributed storage component cleans up the logs in accordance with the expiration policy corresponding to the distributed storage component. Set a log location monitoring item, which is used to monitor whether the logs corresponding to the distributed storage component are in the expected location; Set a log format monitoring item, which is used to verify whether the log format of the log corresponding to the distributed storage component is the same as the expected format; Set up a log anomaly monitoring item, which is used to monitor whether the logs corresponding to the distributed storage component are running normally; Set the corresponding monitoring parameters for each log monitoring item; Configure the task execution strategy corresponding to the log inspection task, the execution strategy including one-time inspection and scheduled inspection; Once the target log, the monitoring items, the monitoring parameters, and the task execution strategy are set, the creation of the log inspection task model is completed.

2. The log inspection method according to claim 1, characterized in that, Setting the target log corresponding to the log inspection task includes: Identify the target component corresponding to the log inspection task; Set the log corresponding to the target component as the target log.

3. The log inspection method according to claim 1, characterized in that, The underlying distributed storage interface corresponding to at least one log inspection task in the log inspection task model includes: Determine the underlying distributed storage interface corresponding to the log inspection task; Establish a data connection with the underlying interface of the distributed storage.

4. The log inspection method according to claim 1, characterized in that, The step of executing the log inspection task corresponding to the distributed storage component based on the task parameters and the monitoring parameters includes: Based on the task parameters, determine the execution strategy corresponding to the log checking task; When the execution strategy is a one-time check, the logs corresponding to the distributed storage component are obtained from the underlying interface of the distributed storage component. The log parameters of the log are compared with the monitoring parameters to obtain the comparison results; If the comparison results indicate that the log parameters and the monitoring parameters are inconsistent, the corresponding abnormal alarm information will be output.

5. The log inspection method according to claim 4, characterized in that, The method further includes: When the execution strategy is a timed check, a preset time interval is determined for the log check task; The log check task is executed periodically according to the preset time interval; The check results are reported after each log check task is completed.

6. A log checking device, characterized in that, The device includes: The model reading module is used to read the log inspection task model, which includes task parameters and corresponding monitoring parameters for log inspection tasks targeting at least one distributed storage component. The interface interface module is used to interface with the distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model. The distributed storage underlying interface is used for the transmission of log data. The task execution module is used to execute the log checking task corresponding to the distributed storage component according to the task parameters and the monitoring parameters. The steps for creating the log inspection task model include: Set the target log for the log inspection task; Setting the monitoring items and monitoring parameters corresponding to the log inspection task includes: setting a log size monitoring item, which is used to monitor the storage space capacity occupied by the log; Set up a log packaging monitoring item, which is used to determine whether the logs of the distributed storage component are packaged in accordance with the log packaging strategy corresponding to the distributed storage component. Set up a log expiration check monitoring item, which is used to determine whether the distributed storage component cleans up the logs in accordance with the expiration policy corresponding to the distributed storage component. Set a log location monitoring item, which is used to monitor whether the logs corresponding to the distributed storage component are in the expected location; Set a log format monitoring item, which is used to verify whether the log format of the log corresponding to the distributed storage component is the same as the expected format; Set up a log anomaly monitoring item, which is used to monitor whether the logs corresponding to the distributed storage component are running normally; Set the corresponding monitoring parameters for each log monitoring item; Configure the task execution strategy corresponding to the log inspection task, the execution strategy including one-time inspection and scheduled inspection; Once the target log, the monitoring items, the monitoring parameters, and the task execution strategy are set, the creation of the log inspection task model is completed.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.

Images