Log checking method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of data processing, in particular to a log checking method, device and equipment and a storage medium, and aims to uniformly manage and monitor log data of a distributed storage system. The method comprises the steps that a log checking task model is read, wherein the log checking task model comprises task parameters of log checking tasks for at least one distributed storage assembly and corresponding monitoring parameters; docking a distributed storage bottom layer interface corresponding to at least one log checking task in the log checking task model, wherein the distributed storage bottom layer interface is used for transmitting log data; according to the task parameters and the monitoring parameters, the log checking task corresponding to the distributed storage component is executed, and the log data corresponding to the log checking task are obtained through the distributed storage bottom layer interface.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a log checking method, device, equipment and storage medium. Background Art

[0002] A distributed storage system is a system that provides distributed storage services to external cloud platforms. A distributed storage system includes multiple distributed system storage components. Each component generates a large number of service logs every day, and the service logs need to be managed uniformly. In related technologies, the service logs of the distributed storage system are recorded and packaged.

[0003] In the related art, the logs of the distributed storage system are simply recorded and packaged, which cannot effectively manage the logs in a systematic manner, nor can the logs be effectively monitored, and the stable operation of the distributed storage business cannot be guaranteed. Summary of the invention

[0004] The embodiments of the present application provide a log checking method, apparatus, device and storage medium, which are intended to uniformly manage and monitor log data of a distributed storage system.

[0005] A first aspect of an embodiment of the present application provides a log checking method, the method comprising: Reading a log check task model, wherein the log check task model includes task parameters of a log check task for at least one distributed storage component and corresponding monitoring parameters; Connecting to a distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model, wherein the distributed storage underlying interface is used for transmitting log data; The log checking task corresponding to the distributed storage component is executed according to the task parameters and the monitoring parameters, and the log data corresponding to the log checking task is obtained through the distributed storage underlying interface.

[0006] Optionally, the step of creating the log checking task model includes: Set the target log corresponding to the log checking task; Set the monitoring items and monitoring parameters corresponding to the log checking task; Setting a task execution strategy corresponding to the log check task, wherein the execution strategy includes a one-time check and a scheduled check; When the target log, the monitoring items, the monitoring parameters and the task execution strategy are set, the creation of the log inspection task model is completed.

[0007] Optionally, setting a target log corresponding to the log checking task includes: Determine the target component corresponding to the log checking task; The log corresponding to the target component is set as the target log.

[0008] Optionally, the setting of the monitoring item and the monitoring parameter corresponding to the log checking task includes: setting a log size monitoring item, the log size monitoring item being used to monitor the capacity of the storage space occupied by the log; Setting a log packaging monitoring item, wherein the log packaging monitoring item is used to determine whether the log of the distributed storage component is packaged according to the log packaging policy corresponding to the distributed storage component; Setting a log expiration check monitoring item, wherein the log expiration check monitoring item is used to determine whether the distributed storage component cleans up the log according to the expiration policy corresponding to the distributed storage component; Setting a log location monitoring item, wherein the log location monitoring item is used to monitor whether the log corresponding to the distributed storage component is in the expected location; Setting a log format monitoring item, wherein the log format monitoring item is used to verify whether the log format of the log corresponding to the distributed storage component is the same as the expected format; Setting a log exception monitoring item, wherein the log exception monitoring item is used to monitor whether the log corresponding to the distributed storage component is running normally; The corresponding monitoring parameters are set for each of the log monitoring items.

[0009] Optionally, the interconnection with the distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model includes: Determine the distributed storage underlying interface corresponding to the log checking task; Establish a data connection with the distributed storage underlying interface.

[0010] Optionally, executing the log checking task corresponding to the distributed storage component according to the task parameter and the monitoring parameter includes: Determine the execution strategy corresponding to the log checking task according to the task parameters; When the execution strategy is a one-time check, obtaining a log corresponding to the distributed storage component from the distributed storage underlying interface corresponding to the distributed storage component; Determine the monitoring items corresponding to the log checking task according to the task parameters; Call the log processing function corresponding to the monitoring item in the preset log processing function library; Reading the monitoring parameters corresponding to the log checking task and the log parameters of the log through the log processing function; Comparing the log parameter and the monitoring parameter by the log processing function to obtain a comparison result; When the comparison result indicates that the log parameter is inconsistent with the monitoring parameter, corresponding abnormal alarm information is output.

[0011] Optionally, the method further comprises: In the case where the execution strategy is a scheduled check, determining a preset time interval corresponding to the log check task; According to the preset time interval, regularly execute the log checking task; Each time the log checking task is completed, the checking result is reported.

[0012] A second aspect of an embodiment of the present application provides a log checking device, the device comprising: A model reading module, used to read a log checking task model, wherein the log checking task model includes task parameters of a log checking task for at least one distributed storage component and corresponding monitoring parameters; An interface docking module, used for docking with a distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model; The task execution module is used to execute the log checking task corresponding to the distributed storage component according to the task parameters and the monitoring parameters, and the log data corresponding to the log checking task is obtained through the distributed storage underlying interface.

[0013] Optionally, the step of creating the log checking task model includes: Set the target log corresponding to the log checking task; Set the monitoring items and monitoring parameters corresponding to the log checking task; Setting a task execution strategy corresponding to the log check task, wherein the execution strategy includes a one-time check and a scheduled check; When the target log, the monitoring items, the monitoring parameters and the task execution strategy are set, the creation of the log inspection task model is completed.

[0014] Optionally, setting a target log corresponding to the log checking task includes: Determine the target component corresponding to the log checking task; The log corresponding to the target component is set as the target log.

[0015] Optionally, the setting of the monitoring item and the monitoring parameter corresponding to the log checking task includes: setting a log size monitoring item, the log size monitoring item being used to monitor the capacity of the storage space occupied by the log; Setting a log packaging monitoring item, wherein the log packaging monitoring item is used to determine whether the log of the distributed storage component is packaged according to the log packaging policy corresponding to the distributed storage component; Setting a log expiration check monitoring item, wherein the log expiration check monitoring item is used to determine whether the distributed storage component cleans up the log according to the expiration policy corresponding to the distributed storage component; Setting a log location monitoring item, wherein the log location monitoring item is used to monitor whether the log corresponding to the distributed storage component is in the expected location; Setting a log format monitoring item, wherein the log format monitoring item is used to verify whether the log format of the log corresponding to the distributed storage component is the same as the expected format; Setting a log exception monitoring item, wherein the log exception monitoring item is used to monitor whether the log corresponding to the distributed storage component is running normally; The corresponding monitoring parameters are set for each of the log monitoring items.

[0016] Optionally, the interconnection with the distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model includes: Determine the distributed storage underlying interface corresponding to the log checking task; Establish a data connection with the distributed storage underlying interface.

[0017] Optionally, the task execution module includes: An execution strategy determination submodule, used to determine the execution strategy corresponding to the log inspection task according to the task parameters; A log acquisition submodule, used for acquiring the log corresponding to the distributed storage component from the distributed storage underlying interface corresponding to the distributed storage component when the execution strategy is a one-time check; A monitoring item determination submodule, used to determine the monitoring item corresponding to the log inspection task according to the task parameters; The function calling submodule is used to call the log processing function corresponding to the monitoring item in the preset log processing function library; A parameter reading submodule, used for reading the monitoring parameters corresponding to the log checking task and the log parameters of the log through the log processing function; A comparison result acquisition submodule is used to compare the log parameter and the monitoring parameter through the log processing function to obtain a comparison result; The abnormal alarm information output submodule is used to output corresponding abnormal alarm information when the comparison result indicates that the log parameter is inconsistent with the monitoring parameter.

[0018] Optionally, the device further comprises: A time interval determination submodule, used to determine a preset time interval corresponding to the log inspection task when the execution strategy is a scheduled inspection; The inspection task execution submodule is used to regularly execute the log inspection task according to the preset time interval; The inspection result reporting submodule is used to report the inspection result each time the log inspection task is completed.

[0019] A third aspect of an embodiment of the present application provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the method described in the first aspect of the present application are implemented.

[0020] A fourth aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect of the present application are implemented.

[0021] Using the log checking method provided in the present application, a log checking task model is read, wherein the log checking task model includes task parameters and corresponding monitoring parameters of a log checking task for at least one distributed storage component; a distributed storage underlying interface corresponding to at least one log checking task in the log checking task model is connected, and the distributed storage underlying interface is used for transmitting log data; according to the task parameters and the monitoring parameters, the log checking task corresponding to the distributed storage component is executed, and the log data corresponding to the log checking task is obtained through the distributed storage underlying interface.

[0022] In this method, a daily inspection task model is set up, which includes log inspection tasks and monitoring parameters for each distributed storage component. By connecting to the underlying interface of distributed storage, the log data of each distributed storage component is obtained. According to the task parameters and monitoring parameters in the task model, the corresponding log inspection task is executed, so that the distributed storage system can coordinate the logs generated by all services and ensure the normal operation of user business. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 is a flow chart of a log checking method proposed in an embodiment of the present application; Figure 2 This is a log checking process diagram proposed in an embodiment of the present application; Figure 3 is a schematic diagram of a log checking device proposed in an embodiment of the present application; Figure 4 It is a schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] refer to Figure 1 , Figure 1 1 is a flow chart of a log checking method proposed in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps: S11: Reading a log check task model, wherein the log check task model includes task parameters of a log check task for at least one distributed storage component and corresponding monitoring parameters.

[0027] In this embodiment, the log check task model is a model established in the settings page of the server of the distributed storage system, which includes the log check tasks and corresponding monitoring parameters for each component in the distributed storage system. The distributed storage system is a data storage technology that stores data in a distributed manner on multiple independent devices and forms a virtual storage device with these distributed storage resources through the network. This system utilizes the disk space on each machine in the enterprise, so that data can be stored in every corner of the enterprise, thereby improving the reliability and availability of the data. The task parameters include the specific information of the distributed storage components that need to be monitored, the specific monitoring strategies, etc. The monitoring parameters are the normal parameters of the logs corresponding to each distributed storage component. Distributed storage components are components that implement various functions in the distributed storage system.

[0028] In this embodiment, the server first reads the log checking task model, which includes task parameters of the log checking task for at least one distributed storage component and corresponding monitoring parameters.

[0029] S12: Connecting to a distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model, wherein the distributed storage underlying interface is used for transmission of log data.

[0030] In this embodiment, the distributed storage underlying interface is used for the transmission of log data.

[0031] In this embodiment, after receiving the log checking task model, the server determines the distributed storage underlying interface corresponding to each log checking task in the log checking task model, and then connects with each distributed storage underlying interface to ensure data transmission between the server and the corresponding components.

[0032] In this embodiment, the specific steps of connecting to the distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model include: S12-1: Determine the distributed storage underlying interface corresponding to the log checking task.

[0033] In this embodiment, each log checking task corresponds to a distributed storage component. The server calls the log data corresponding to the distributed storage component through the distributed storage underlying interface, and calls the corresponding log processing tool to process the log data.

[0034] In this embodiment, the server first determines the distributed storage component that needs to be checked by the log checking task, and then determines the distributed storage underlying interface corresponding to the distributed storage component.

[0035] S12-2: Establish a data connection with the distributed storage underlying interface.

[0036] In this embodiment, after determining the distributed storage component and the distributed storage underlying interface corresponding to the log checking task, the server establishes a data connection with the distributed storage underlying interface.

[0037] S13: Execute the log checking task corresponding to the distributed storage component according to the task parameters and the monitoring parameters, and the log data corresponding to the log checking task is obtained through the distributed storage underlying interface.

[0038] In this embodiment, after determining the task parameters and monitoring parameters, the log checking task corresponding to the distributed storage component is executed for each distributed storage component. When executing the log checking task, it is necessary to obtain the log data of the distributed storage component through the distributed storage underlying interface, call the corresponding log processing tool to analyze and process the log data, and obtain the processing results in combination with the monitoring parameters, and then decide whether to perform alarm processing, thereby completing the execution of the log checking task corresponding to the distributed storage component.

[0039] In this embodiment, according to the task parameters and the monitoring parameters, the specific steps of executing the log checking task corresponding to the distributed storage component include: S13-1: Determine the execution strategy corresponding to the log checking task according to the task parameters.

[0040] In this embodiment, the task parameters include the execution strategy of the log check task, and the execution strategy includes a one-time check and a scheduled check. The one-time check is to execute the log check task only once at the current moment and return the task result. The scheduled check is to perform a check once every preset time period and return the inspection result.

[0041] In this embodiment, the server first determines the execution strategy corresponding to each log checking task according to the task parameters in the log checking task model.

[0042] S13-2: When the execution strategy is a one-time check, the log corresponding to the distributed storage component is obtained from the distributed storage underlying interface corresponding to the distributed storage component.

[0043] In this embodiment, when the execution strategy of the log check task is a one-time check, the server obtains the log data corresponding to the distributed storage component from the distributed storage underlying interface corresponding to the distributed storage component. The log data is stored in the underlying storage device of the distributed storage system. The log data of each component is obtained from the underlying storage device through the underlying storage interface of the distributed system.

[0044] S13-3: Determine the monitoring items corresponding to the log checking task according to the task parameters.

[0045] In this embodiment, the task parameters include monitoring items corresponding to each log checking task. When the task parameters of any log checking task are obtained from the log checking task model, the monitoring items corresponding to the log checking task are read from the task parameters.

[0046] For example, if the log check task is to check whether the log data of a component is packaged normally, the monitoring item is the packaging strategy monitoring.

[0047] S13-4: Calling the log processing function corresponding to the monitoring item in the preset log processing function library.

[0048] In this embodiment, the log processing function is a function used to process log data, and is a function pre-written according to the content that needs to be compared for each monitoring item. These functions are stored in a log processing function library and can be called at any time.

[0049] In this embodiment, after the monitoring item corresponding to the log checking task is determined, the corresponding log processing function is called in the preset log processing function library.

[0050] For example, if the monitoring item is log packaging monitoring, the log packaging monitoring function is called.

[0051] S13-5: Read the monitoring parameters corresponding to the log checking task and the log parameters of the log through the log processing function.

[0052] In this embodiment, the log parameters are specific values ​​of various types of data contained in the log, such as the size of each log data packet, the total size of the log data, the packaging time of the log data, etc.

[0053] In this embodiment, after the log processing function is called, the monitoring parameters corresponding to the log checking task and the log parameters of the log are read through the log processing function.

[0054] S13-6: Compare the log parameter and the monitoring parameter through the log processing function to obtain a comparison result.

[0055] In this embodiment, a corresponding log check task and corresponding monitoring parameters are pre-set for each component. After obtaining the log data of the distributed storage component, the server determines whether there is a difference between the log parameters in the log data and the monitoring parameters. When comparing, a log processing function is generally used for comparison to obtain a comparison result.

[0056] For example, for a log expiration detection task, it is possible to compare whether the log is deleted according to a preset time.

[0057] S13-7: When the comparison result indicates that the log parameter is inconsistent with the monitoring parameter, output corresponding abnormal alarm information.

[0058] In this embodiment, the abnormal alarm information is used to inform the user that the log data is abnormal.

[0059] In this embodiment, when the comparison result indicates that the log parameter and the monitoring parameter are inconsistent, corresponding abnormal alarm information is output.

[0060] For example, for a log expiration detection task, if the comparison result indicates that the log is not deleted according to the preset time, an abnormal alarm message is issued to inform the user that the log is not deleted according to the preset time.

[0061] For example, the log monitoring task uses the logcheck service to perform log monitoring. The service is a tool for analyzing system log files. It is mainly used to detect potential security risks and other abnormal activities, and notify the specified user of the detection results via email. Logcheck identifies potential security issues and abnormal behaviors by filtering keywords in log files, thereby helping system administrators to discover and handle these problems in a timely manner. The core is divided into log policy judgment and log alarm.

[0062] For example, taking the log packaging monitoring strategy as an example, the log packaging strategy is generally divided into daily packaging or packaging exceeding a fixed size. LogcheckTar (a processing function in the log detection tool) inherits and implements the check method provided by the logcheckBase (a processing function in the log detection tool) abstract class. After the logcheck log framework generates LogCheckTarEvent, it obtains the log packaging strategy mode, and then submits it to LogcheckTarTask (the processing function of the log detection tool) for task processing. The log packaging strategy is processed in LogcheckTarTask, and then compared with the number or size of the current log compression package. The final result is output to see whether it meets the preset packaging strategy. If it does not meet the preset packaging strategy, the corresponding alarm information is output to inform the user that the log packaging is abnormal. The specific code is as follows: public class LogcheckTar <e>extends LogcheckBase <e>{ @Override protected void check(E logPolicy) { LogCheckTarEvent event = filter(logPolicy); LogCheck.getExecutor().submit(new LogcheckTarTask(event,LogNodeFactory.compareTarLog())); } } By running the above code, you can implement the log packaging monitoring strategy.

[0063] When the log strategy comparison result does not meet expectations, the LogcheckTarTask result will trigger the log_alarm (event monitor, used to trigger an alarm when a specific regular expression is matched in the log) service. The log_alarm service formats the exception information and transmits it to the interface to display the exception information.

[0064] In this embodiment, by setting a log check task model, the log data of each distributed storage component in the distributed storage system is monitored, the scalability and maintainability of the log are enhanced, the unified management of the log is achieved, and the normal operation of the business is ensured.

[0065] In this embodiment, the method further includes: S13-8: When the execution strategy is scheduled inspection, determine the preset time interval corresponding to the log inspection task.

[0066] In this embodiment, when the execution strategy is a scheduled check, a preset time interval corresponding to the log check task is determined.

[0067] S13-9: Execute the log checking task regularly according to the preset time interval.

[0068] In this embodiment, after determining the preset time interval, the log data of the corresponding distributed storage component is obtained regularly according to the preset time interval, the log parameters are compared with the monitoring parameters to obtain the comparison results, and the log checking task is executed regularly.

[0069] S13-10: reporting the inspection result each time the log inspection task is completed.

[0070] In this embodiment, when checking the logs at regular intervals, each time the log checking task is completed, the checking result is reported once, if a fault occurs, an alarm message is sent, if there is no fault, a normal log message is sent.

[0071] In another embodiment of the present application, the step of creating the log checking task model includes: S21: Setting the target log corresponding to the log checking task.

[0072] In this embodiment, the target log is the log that needs to be obtained.

[0073] In this embodiment, when constructing a log checking task model, a target log is first selected. Generally, the target log is all logs supported by the log service list of the distributed storage system.

[0074] For example, select the distributed storage components that need to be checked for logs, including osd (Object StorageDevice node, which plays a key role in the distributed storage system and is mainly responsible for storing and managing data. Each osd node manages a part of the data and implements data redundancy and load balancing through the CRUSH algorithm.) service log, mon (monitoring node in the distributed storage system, mainly responsible for cluster status monitoring, management and maintenance. They ensure the health of the cluster and handle various events of the cluster, such as adding or deleting storage nodes, etc.) service log, mds (Metadata Server node is responsible for managing the metadata in the cluster, including the directory structure of the file system, file permissions, and the mapping relationship between files and directories. The health of the MDS node is crucial to the stability and reliability of the entire distributed storage cluster) service log, ctdb service log (including each child process service), lib (server library, which is a software resource library stored on the server. The server library contains a variety of software components, tools, and functional modules to support the operation and management of the server.) log, protocol log (file, block, object and HDFS (Distributed File System designed to run on commodity hardware) System). It has a lot in common with existing distributed file systems. But at the same time, it is also very different from other distributed file systems. HDFS is a highly fault-tolerant system that is suitable for deployment on cheap machines. HDFS can provide high-throughput data access and is very suitable for applications on large-scale data sets). Select the server component that needs to be checked for logs, message logs. You can also customize the logs of other service items of the monitoring server.

[0075] In this embodiment, the specific steps of setting the target log corresponding to the log checking task include: S21-1: Determine the target component corresponding to the log checking task.

[0076] In this embodiment, the server first determines the target component corresponding to the log checking task. The task parameters of the log checking task include the distributed storage component to be checked, that is, the target component. The corresponding component information can be obtained by reading the task parameters.

[0077] S21 - 2: Setting the log corresponding to the target component as the target log.

[0078] In this embodiment, after the target component is determined, the log corresponding to the target component is set as the target log.

[0079] S22: Setting the monitoring items and monitoring parameters corresponding to the log checking task.

[0080] In this embodiment, the monitoring item corresponding to the log checking task is the item that needs to be monitored.

[0081] In this embodiment, after selecting the target log, the monitoring items and monitoring parameters corresponding to the log inspection task are set. The specific steps include: S22-1: Setting a log size monitoring item, wherein the log size monitoring item is used to monitor the capacity of the storage space occupied by the log.

[0082] In this embodiment, the log size monitoring item is one of the monitoring items. When setting the log size monitoring item, the monitoring value of the log size can be entered, and the server supports importing and exporting the expected log size value of each server. When the actual size of the generated log is larger than the monitoring value, an alarm message is sent to inform the user that the log capacity is abnormal, thereby realizing the monitoring of the capacity of the storage space occupied by the log.

[0083] S22-2: Setting a log packaging monitoring item, wherein the log packaging monitoring item is used to determine whether the logs of the distributed storage component are packaged according to the log packaging policy corresponding to the distributed storage component.

[0084] In this embodiment, the log packaging monitoring item is one of the monitoring items. Different services have different packaging strategies. The server supports importing and exporting the packaging strategy of the service corresponding to each distributed storage component in the distributed storage system (for example, packaging log data that exceeds the preset space capacity, packaging log data in each preset time period, etc.). According to the packaging strategy, the distributed storage system is monitored to see whether it packages the log data according to the corresponding packaging strategy, whether the packaging is successful, and whether the space occupied by the packaged log is in line with expectations. If there is an abnormality, an abnormal alarm message is sent to inform the user of the log packaging abnormality.

[0085] S22-3: Setting a log expiration check monitoring item, wherein the log expiration check monitoring item is used to determine whether the distributed storage component cleans up the log according to the expiration policy corresponding to the distributed storage component.

[0086] In this embodiment, the log expiration check monitoring item is one of the monitoring items. Some log data are set with expiration deletion or regular cleaning policies. The server supports importing and exporting the expiration policy of the service corresponding to each distributed storage component (for example, deleting log data that exceeds a preset capacity, deleting log data every preset time period, etc.). According to the uploaded expiration policy, it is monitored whether the log is cleaned up according to the expiration policy. If the log data is not cleaned up according to the expiration policy, the corresponding abnormal alarm information is sent to inform the user that the log is not cleaned up according to the expiration policy.

[0087] S22-4: Setting a log location monitoring item, wherein the log location monitoring item is used to monitor whether the log corresponding to the distributed storage component is in the expected location.

[0088] In this embodiment, the log location monitoring item is one of the monitoring items of the distributed storage system. The storage location of the service corresponding to each distributed storage component is different. The server supports importing and exporting log location information of each service. According to the log location information, it monitors whether the log data corresponding to the distributed storage component is in the expected location (mainly to verify the log location when the storage upgrade function changes the log storage structure to prevent the log from being recorded normally or the recording location is wrong). If the log data is not in the expected storage location, a corresponding alarm message is sent to inform the user that the storage location of the log is wrong.

[0089] S22-5: Setting a log format monitoring item, wherein the log format monitoring item is used to verify whether the log format of the log corresponding to the distributed storage component is the same as the expected format.

[0090] In this embodiment, the log format monitoring item is one of the distributed storage system monitoring items. The log format of the service corresponding to each distributed storage component is different. The server supports importing and exporting the expected log format of each service. The server monitors the log format by verifying whether the log format of the log corresponding to the distributed storage component is the same as the expected format. If the log format is different from the expected format, an alarm message is sent to inform the user that the log format is wrong.

[0091] S22-6: Setting a log exception monitoring item, wherein the log exception monitoring item is used to monitor whether the log corresponding to the distributed storage component is running normally.

[0092] In this embodiment, there are other log services in the distributed storage system. Corresponding log exception monitoring items can be set for these services to monitor whether the logs corresponding to the distributed storage components are running normally, whether they are continuously recorded, whether the changes in log levels are correct, etc. If any log service has an exception, a corresponding alarm message is generated to inform the user of the log exception.

[0093] S22-7: Set the corresponding monitoring parameters for each of the log monitoring items.

[0094] In this embodiment, after the log monitoring items are set, corresponding monitoring parameters are set for each log monitoring item.

[0095] For example, the time interval in the expiration policy corresponding to the log expiration detection is set to 10 minutes, and the data size in the log packaging policy is set to 100 MB.

[0096] S23: Setting a task execution strategy corresponding to the log check task, wherein the execution strategy includes a one-time check and a scheduled check.

[0097] In this embodiment, after setting the monitoring items and monitoring parameters, the corresponding task execution strategy is set, including one-time inspection and scheduled inspection. For each monitoring item, a corresponding execution strategy can be set, which can be set according to the characteristics of the monitoring item. For example, a log exception monitoring item can be set to a scheduled inspection to ensure that the log anomaly can be detected in time.

[0098] S24: When the target log, the monitoring items, the monitoring parameters and the task execution strategy are set, the log checking task model is created.

[0099] In this embodiment, a log checking task model is set up on the server side, and a unified management setting is performed on the log checking tasks. By loading the log checking task model, log monitoring can be performed on each component of the distributed storage system, thereby realizing unified monitoring of the logs of each component of the distributed storage system.

[0100] refer to Figure 2 , Figure 2 : is a log checking flow chart of an embodiment of the present application. Figure 2 As shown, users can select services in the server, select all logs supported by the distributed storage system, select the corresponding system service items, select the corresponding component service items, and then configure custom services. Custom monitoring items include log size monitoring, log packaging monitoring, log expiration detection, log location monitoring, log format monitoring, and log anomaly detection. The system also supports the import and export of monitoring policies, and sets each monitoring task as a real-time task or a scheduled task. Finally, abnormal alarms are issued based on the monitored log data.

[0101] In another embodiment of the present application, when the server detects that a component of the distributed storage system has been expanded or changed, it obtains component information of the newly added or changed component, sets a corresponding log check task for the distributed storage component based on the service corresponding to the distributed storage component, and adds the log check task and the corresponding monitoring parameters to the log check task model, and dynamically updates the model to ensure that each distributed storage component in the distributed storage system can be monitored at all times, thereby ensuring the normal operation of the distributed storage system's services.

[0102] In another embodiment of the present application, after each log check task is completed, the server records the corresponding log check result data, and adjusts the log monitoring strategy according to the log check result data. For example, if the check result data of a certain log check task in a longer period of time all indicate that its log data is normal, the time interval of the log check task can be lengthened, thereby saving server resources. If the check result data of a certain log check task has many errors, the time interval of the log check task is shortened, and the monitoring of the log data of the distributed storage component is strengthened to ensure that the errors in the log data can be promptly alarmed. When a certain log check task is a one-time inspection task, and an error is found in the log, the log check task is changed to a scheduled inspection while the alarm is issued, until the corresponding log data does not have an error for a long time, the log check task can be changed to a one-time inspection. Through the inspection results of the log data, the strategy for the execution of the log check task is dynamically adjusted, ensuring comprehensive monitoring of the log data of the distributed storage component, and ensuring the stable operation of the system while saving server resources.

[0103] In the above embodiments of the present application, by constructing a log inspection task model of a distributed storage system, unified monitoring of the log data of the services of each distributed storage component in the distributed storage system is achieved, timely alarms are given for abnormalities in the logs, and the log inspection task model is dynamically updated according to the update status of the distributed storage system, thereby enhancing the scalability and maintainability of the logs, preventing the occurrence of other chain system problems, and ensuring the normal operation of the distributed storage system's services.

[0104] Based on the same inventive concept, an embodiment of the present application provides a log checking device. Figure 3 , Figure 3 Schematic diagram of a log checking device 300 proposed in one embodiment of the present application. Figure 3 As shown, the device comprises: A model reading module 301 is used to read a log checking task model, wherein the log checking task model includes task parameters of a log checking task for at least one distributed storage component and corresponding monitoring parameters; An interface docking module 302 is used to dock with a distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model, wherein the distributed storage underlying interface is used for the transmission of log data, and the log data corresponding to the log inspection task is obtained through the distributed storage underlying interface; The task execution module 303 is used to execute the log checking task corresponding to the distributed storage component according to the task parameters and the monitoring parameters. The log data corresponding to the log checking task is obtained through the distributed storage underlying interface.

[0105] Optionally, the step of creating the log checking task model includes: Set the target log corresponding to the log checking task; Set the monitoring items and monitoring parameters corresponding to the log checking task; Setting a task execution strategy corresponding to the log check task, wherein the execution strategy includes a one-time check and a scheduled check; When the target log, the monitoring items, the monitoring parameters and the task execution strategy are set, the creation of the log inspection task model is completed.

[0106] Optionally, setting a target log corresponding to the log checking task includes: Determine the target component corresponding to the log checking task; The log corresponding to the target component is set as the target log.

[0107] Optionally, the setting of the monitoring item and the monitoring parameter corresponding to the log checking task includes: setting a log size monitoring item, the log size monitoring item being used to monitor the capacity of the storage space occupied by the log; Setting a log packaging monitoring item, wherein the log packaging monitoring item is used to determine whether the log of the distributed storage component is packaged according to the log packaging policy corresponding to the distributed storage component; Setting a log expiration check monitoring item, wherein the log expiration check monitoring item is used to determine whether the distributed storage component cleans up the log according to the expiration policy corresponding to the distributed storage component; Setting a log location monitoring item, wherein the log location monitoring item is used to monitor whether the log corresponding to the distributed storage component is in the expected location; Setting a log format monitoring item, wherein the log format monitoring item is used to verify whether the log format of the log corresponding to the distributed storage component is the same as the expected format; Setting a log exception monitoring item, wherein the log exception monitoring item is used to monitor whether the log corresponding to the distributed storage component is running normally; The corresponding monitoring parameters are set for each of the log monitoring items.

[0108] Optionally, the interconnection with the distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model includes: Determine the distributed storage underlying interface corresponding to the log checking task; Establish a data connection with the distributed storage underlying interface.

[0109] Optionally, the task execution module includes: An execution strategy determination submodule, used to determine the execution strategy corresponding to the log inspection task according to the task parameters; A log acquisition submodule, used for acquiring the log corresponding to the distributed storage component from the distributed storage underlying interface corresponding to the distributed storage component when the execution strategy is a one-time check; A monitoring item determination submodule, used to determine the monitoring item corresponding to the log inspection task according to the task parameters; The function calling submodule is used to call the log processing function corresponding to the monitoring item in the preset log processing function library; A parameter reading submodule, used for reading the monitoring parameters corresponding to the log checking task and the log parameters of the log through the log processing function; A comparison result acquisition submodule is used to compare the log parameter and the monitoring parameter through the log processing function to obtain a comparison result; The abnormal alarm information output submodule is used to output corresponding abnormal alarm information when the comparison result indicates that the log parameter is inconsistent with the monitoring parameter.

[0110] Optionally, the device further comprises: A time interval determination submodule, used to determine a preset time interval corresponding to the log inspection task when the execution strategy is a scheduled inspection; The inspection task execution submodule is used to regularly execute the log inspection task according to the preset time interval; The inspection result reporting submodule is used to report the inspection result each time the log inspection task is completed.

[0111] Based on the same inventive concept, another embodiment of the present application provides a readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the log checking method described in any of the above embodiments of the present application are implemented.

[0112] Based on the same inventive concept, another embodiment of the present application provides an electronic device, Figure 4 It is a schematic diagram of an electronic device 400 proposed in one embodiment of the present application, including a memory 402, a processor 401 and a computer program stored in the memory and executable on the processor, and when the processor is executed, the steps in the log checking method described in any of the above embodiments of the present application are implemented.

[0113] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0114] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0115] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0116] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0119] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0120] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0121] The log checking method, device, equipment and storage medium provided by the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.< / e> < / e>

Claims

1. A log checking method, characterized in that: The method comprises: Reading a log check task model, wherein the log check task model includes task parameters of a log check task for at least one distributed storage component and corresponding monitoring parameters; Connecting to a distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model, wherein the distributed storage underlying interface is used for transmitting log data; The log checking task corresponding to the distributed storage component is executed according to the task parameters and the monitoring parameters, and the log data corresponding to the log checking task is obtained through the distributed storage underlying interface.

2. The log checking method according to claim 1, characterized in that: The steps of creating the log checking task model include: Set the target log corresponding to the log checking task; Set the monitoring items and monitoring parameters corresponding to the log checking task; Setting a task execution strategy corresponding to the log check task, wherein the execution strategy includes a one-time check and a scheduled check; When the target log, the monitoring items, the monitoring parameters and the task execution strategy are set, the creation of the log inspection task model is completed.

3. The log checking method according to claim 2, characterized in that: The setting of the target log corresponding to the log checking task includes: Determine the target component corresponding to the log checking task; The log corresponding to the target component is set as the target log.

4. The log checking method according to claim 2, characterized in that: The setting of the monitoring items and the monitoring parameters corresponding to the log checking task includes: setting a log size monitoring item, wherein the log size monitoring item is used to monitor the capacity of the storage space occupied by the log; Setting a log packaging monitoring item, wherein the log packaging monitoring item is used to determine whether the log of the distributed storage component is packaged according to the log packaging policy corresponding to the distributed storage component; Setting a log expiration check monitoring item, wherein the log expiration check monitoring item is used to determine whether the distributed storage component cleans up the log according to the expiration policy corresponding to the distributed storage component; Setting a log location monitoring item, wherein the log location monitoring item is used to monitor whether the log corresponding to the distributed storage component is in the expected location; Setting a log format monitoring item, wherein the log format monitoring item is used to verify whether the log format of the log corresponding to the distributed storage component is the same as the expected format; Setting a log exception monitoring item, wherein the log exception monitoring item is used to monitor whether the log corresponding to the distributed storage component is running normally; The corresponding monitoring parameters are set for each of the log monitoring items.

5. The log checking method according to claim 1, characterized in that: The distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model is connected, including: Determine the distributed storage underlying interface corresponding to the log checking task; Establish a data connection with the distributed storage underlying interface.

6. The log checking method according to claim 1, characterized in that: The executing the log checking task corresponding to the distributed storage component according to the task parameter and the monitoring parameter includes: Determine the execution strategy corresponding to the log checking task according to the task parameters; When the execution strategy is a one-time check, obtaining a log corresponding to the distributed storage component from the distributed storage underlying interface corresponding to the distributed storage component; Determine the monitoring items corresponding to the log checking task according to the task parameters; Call the log processing function corresponding to the monitoring item in the preset log processing function library; Reading the monitoring parameters corresponding to the log checking task and the log parameters of the log through the log processing function; Comparing the log parameter and the monitoring parameter by the log processing function to obtain a comparison result; When the comparison result indicates that the log parameter is inconsistent with the monitoring parameter, corresponding abnormal alarm information is output.

7. The log checking method according to claim 6, characterized in that: The method further comprises: In the case where the execution strategy is a scheduled check, determining a preset time interval corresponding to the log check task; According to the preset time interval, regularly execute the log checking task; Each time the log checking task is completed, the checking result is reported.

8. A log checking device, characterized in that: The device comprises: A model reading module, used to read a log checking task model, wherein the log checking task model includes task parameters of a log checking task for at least one distributed storage component and corresponding monitoring parameters; An interface docking module, used to dock with a distributed storage underlying interface corresponding to at least one log inspection task in the log inspection task model, wherein the distributed storage underlying interface is used for the transmission of log data; The task execution module is used to execute the log checking task corresponding to the distributed storage component according to the task parameters and the monitoring parameters, and the log data corresponding to the log checking task is obtained through the distributed storage underlying interface.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images