A raid log detection method, electronic equipment and storage medium

Abstract

The present application relates to a kind of RAID log detection method, electronic equipment and storage medium, in server, each key in first dictionary is traversed, the weight corresponding to key in second dictionary is found, the number of times of key appearance is multiplied by the weight corresponding to key, and weighted score is accumulated to obtain, initialize alarm list, if the key in first dictionary satisfies the trigger condition of composite rule, the name of composite rule is added to alarm list, first dictionary, weighted score and alarm list are stored in result dictionary.The RAID log of the present application is analyzed in server, instead of downloading RAID log to local, analysis is carried out in local, improve the security of RAID log data, and multiple servers can be executed RAID log analysis task in parallel.The RAID log processing skill of the present application has the efficient process such as local downloading, intelligent analysis, batch processing etc., greatly accelerates the detection efficiency of RAID log.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a RAID log detection method, electronic equipment and storage medium. BACKGROUND

[0002] In the related art, the detection and analysis of RAID logs still generally follow the traditional mode of "downloading each device locally and analyzing locally": the operation and maintenance personnel need to copy the log files generated by the RAID controller to the personal workstation completely, and then use scripts or special tools to perform offline analysis. However, as the scale of the data center expands, the log capacity of a single device often exceeds 10 GB, and the cross-network transmission of massive data not only occupies valuable bandwidth, but also makes the analysis time-consuming exponentially, directly reducing the fault locating efficiency. More seriously, the traditional solution uses single-node serial operation, which must perform the "collection-transmission-analysis" closed loop on hundreds of servers one by one, lacks parallel processing capability, and leads to a long troubleshooting period, which is difficult to meet the stringent requirements of business continuity. In addition, RAID logs contain highly sensitive information such as disk serial numbers, volume group topologies, and user data block addresses, and once stored on public networks or external terminals, there is a risk of leakage, tampering, and failure of compliance audits; offline analysis also forces enterprises to migrate core data out of controlled computer rooms, which obviously violates the regulatory requirements of important industries such as finance and telecommunications for data sovereignty and security isolation, and cannot adapt to customers' demand for "zero export and zero landing" security. SUMMARY

[0003] The present application provides a RAID log detection method, electronic equipment and storage medium, aiming to at least solve one of the technical problems existing in the prior art.

[0004] The technical solution of the present application is a RAID log detection method, comprising the following steps:

[0005] S100, obtaining a first dictionary, a second dictionary and a composite rule on a server, wherein the first dictionary includes a keyword and the number of occurrences of the keyword, and the second dictionary includes the keyword and the weight corresponding to the keyword;

[0006] S200, traversing each keyword in the first dictionary, finding the weight corresponding to the keyword in the second dictionary, multiplying the number of occurrences of the keyword by the weight corresponding to the keyword, and accumulating to obtain a weighted score;

[0007] S300, initializing an alarm list, and judging whether the keyword in the first dictionary meets the trigger condition of the composite rule; if the keyword in the first dictionary meets the trigger condition of the composite rule, the name of the composite rule is added to the alarm list;

[0008] S400, store the first dictionary, the weighted score and the alert list into a result dictionary, wherein an IP address of the server is a key and the first dictionary, the weighted score and the alert list are values.

[0009] According to some embodiments of the present application, the step S100 comprises:

[0010] S101, obtain a JSON configuration file, wherein the JSON configuration file comprises remote node information, the keyword, a weight corresponding to the keyword and the composite rule;

[0011] S102, connect the server using SSH according to the remote node information;

[0012] S103, execute a RAID configuration command, count the number of occurrences of the keyword in a log file on the server to obtain the first dictionary;

[0013] S104, obtain the second dictionary according to the keyword and the weight corresponding to the keyword.

[0014] According to some embodiments of the present application, the remote node information comprises an IP address, a username and a password of the server; and the step S102 comprises:

[0015] create an SSHClient instance;

[0016] set a host key policy to automatically accept unknown host keys;

[0017] connect the server using SSH by using the IP address, the username and the password of the server and set a timeout time;

[0018] capture and record exceptions occurring in the connection process and rethrow the exceptions for processing by a calling party.

[0019] According to some embodiments of the present application, in the step S103, counting the number of occurrences of the keyword in a log file on the server to obtain the first dictionary comprises:

[0020] construct a grep command pipeline to count the number of occurrences of the keyword in the log file on the server, the grep command pipeline involving case-insensitive search, uniform conversion to lowercase, sorting, duplicate count and formatted output;

[0021] execute the grep command pipeline;

[0022] Check whether the grep command pipeline is executed successfully by reading stderr, if error information is detected, throw an exception and record the error information; if the grep command pipeline is executed successfully, parse the content of stdout, split the "keyword: number" format data of each line into key-value pairs, and store them in the first dictionary.

[0023] According to some embodiments of the application, in the step S103, the RAID configuration command is executed, including:

[0024] Through the storcli64 tool, the alarm log information of the RAID controller is obtained, and the alarm log information is output to the log file.

[0025] According to some embodiments of the application, the composite rule includes a key error rule, a warning detection rule, and a multiple error rule; and the step S300 includes:

[0026] Initializing the alarm list;

[0027] By traversing the composite rule, the key in the first dictionary is checked,

[0028] When "error" and "fail" appear at the same time, the trigger condition of the key error rule is met, and the name of the key error rule is recorded in the alarm list;

[0029] When "warning" appears, the trigger condition of the warning detection rule is met, and the name of the warning detection rule is recorded in the alarm list;

[0030] When the number of occurrences of "error" reaches a first threshold, the trigger condition of the multiple error rule is met, and the name of the multiple error rule is recorded in the alarm list.

[0031] According to some embodiments of the application, in the step S400,

[0032] If the SSH connection fails or the log analysis is wrong, an exception is captured and the error information is stored in the result dictionary, wherein the IP address of the server is the key and the error information is the value.

[0033] According to some embodiments of the application, before the step S100, the RAID log detection method further includes the following steps:

[0034] Through the import configuration button of the visual interface, the import of the JSON configuration file is controlled to obtain the JSON configuration file.

[0035] The RAID configuration command is inputted by the RAID configuration command input box and the RAID configuration button of the visual interface, and the RAID configuration command is controlled to be inputted to obtain the RAID configuration command.

[0036] The analysis button of the visual interface is used to control triggering of the analysis task of the RAID log.

[0037] The technical scheme of the present application also relates to an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method as described above when executing the computer program.

[0038] The technical scheme of the present application also relates to a computer readable storage medium, which stores program instructions, and the program instructions are executed by a processor to implement the method as described above.

[0039] The beneficial effects of the present application include: obtaining a first dictionary, a second dictionary and a composite rule on a server, wherein the first dictionary includes keywords and the number of occurrences of the keywords, the second dictionary includes the keywords and the weights corresponding to the keywords, each keyword in the first dictionary is traversed, the weight corresponding to the keyword in the second dictionary is found, the number of occurrences of the keyword is multiplied by the weight corresponding to the keyword, and a weighted score is accumulated, an alarm list is initialized, and it is determined whether the keyword in the first dictionary meets the triggering condition of the composite rule; if the keyword in the first dictionary meets the triggering condition of the composite rule, the name of the composite rule is added to the alarm list, and the first dictionary, the weighted score and the alarm list are stored in a result dictionary, wherein the IP address of the server is the key, and the first dictionary, the weighted score and the alarm list are the values.

[0040] The RAID log is analyzed on the server in the present application, instead of being downloaded to the local and analyzed on the local, so that the security of the RAID log data is improved, and the analysis task of the RAID log can be executed in parallel by multiple servers. The RAID log processing skill in the present application has the efficient process of no local download, intelligent analysis and batch processing, and greatly accelerates the detection efficiency of the RAID log.

[0041] In addition, additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter in the description. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is an optional flowchart of a RAID log detection method in an embodiment of the present application.

[0043] Figure 2 is an optional flowchart of a RAID log detection method in an embodiment of the present application.

[0044] Figure 3 is a schematic diagram of a visualization interface in embodiments of the present invention.

[0045] Figure 4 is a schematic diagram of an analysis result of a RAID log in embodiments of the present invention. DETAILED DESCRIPTION

[0046] The concept, specific structure and generated technical effects of the present application will be described clearly and completely in combination with embodiments and drawings, so as to fully understand the purpose, scheme and effects of the present application. It should be noted that the embodiments and features in the embodiments in the present application can be combined with each other without conflict.

[0047] It should be noted that, unless otherwise specified, when a certain feature is referred to as being "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. In addition, the up, down, left, right, top, bottom and other descriptions used in the present application are only relative to the relative positions of the components of the present application in the drawings.

[0048] In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art. The terms used in the specification herein are only for describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any combination of one or more related listed items.

[0049] It should be understood that although the terms first, second, third, etc. are used to describe various elements in the present application, these elements should not be limited by these terms. These terms are only used to distinguish elements of the same type from each other. For example, without departing from the scope of the present application, the first element can also be referred to as the second element, and similarly, the second element can also be referred to as the first element.

[0050] Reference Figures 1 to 4 In some embodiments, a RAID log detection method of the technical scheme of the present application comprises at least the following steps:

[0051] S100, obtaining a first dictionary, a second dictionary and a composite rule on a server, wherein the first dictionary comprises a keyword and the number of occurrences of the keyword, and the second dictionary comprises a keyword and the weight corresponding to the keyword;

[0052] S200, traversing each keyword in the first dictionary, finding the weight corresponding to the keyword in the second dictionary, multiplying the number of occurrences of the keyword by the weight corresponding to the keyword, and accumulating to obtain a weighted score;

[0053] S300, initialize the alarm list, judge whether the keyword in the first dictionary meets the trigger condition of the composite rule; if the keyword in the first dictionary meets the trigger condition of the composite rule, the name of the composite rule is added to the alarm list.

[0054] S400, store the first dictionary, the weighted score and the alarm list into the result dictionary, wherein the IP address of the server is the key, and the first dictionary, the weighted score and the alarm list are the values.

[0055] Specifically, the RAID log (Redundant Array of Independent Disks Log) is an event record file automatically generated by the RAID controller during operation, which is used to record all key operations, state changes and abnormal information of the array and its member hard disks in detail. The RAID log is the core basis for diagnosing faults, predicting failures, performance tuning and compliance auditing.

[0056] It can be seen that the present application is a kind of efficient and safe RAID log collection and detection method, the RAID log of the present application is analyzed in the server, instead of downloading the RAID log to the local and analyzing in the local, which improves the security of RAID log data, and can make multiple servers execute RAID log analysis tasks in parallel. The RAID log processing skill of the present application has the efficient process of no local download, intelligent analysis, batch processing, etc., which greatly speeds up the detection efficiency of the RAID log.

[0057] Referring to Figure 2 In a possible implementation, a RAID log detection method includes at least the following steps:

[0058] Develop an interactive interface; import a JSON configuration file; use SSH to establish a connection with the system; remotely collect RAID logs; calculate weighted scores; detect composite rules; execute analysis in parallel; and display analysis results.

[0059] Specifically, the JSON configuration file is a pure text file written in JSON (JavaScript Object Notation) syntax, which is used to centrally store the parameters and settings of programs, systems or services, and can be read by software once and mapped to a memory object, so that the running behavior can be changed without recompilation.

[0060] The step S100 is specifically implemented as follows:

[0061] The method of the embodiment of the present application obtains a first dictionary, a second dictionary and a composite rule on a server, wherein the first dictionary includes a keyword and the number of occurrences of the keyword, and the second dictionary includes a keyword and the weight corresponding to the keyword.

[0062] In some embodiments, step S100 comprises:

[0063] S101, acquiring a JSON configuration file, wherein the JSON configuration file comprises remote node information, a keyword, a weight corresponding to the keyword, and a composite rule;

[0064] S102, connecting to a server using SSH according to the remote node information;

[0065] S103, executing a RAID configuration command to count the number of occurrences of the keyword in a log file on the server to obtain a first dictionary;

[0066] S104, obtaining a second dictionary according to the keyword and the weight corresponding to the keyword.

[0067] In some embodiments, the remote node information comprises an IP address, a username, and a password of the server; and step S102 comprises:

[0068] creating an SSHClient instance;

[0069] setting a host key policy to automatically accept unknown host keys;

[0070] connecting to the server using the IP address, the username, and the password of the server through SSH, and setting a timeout time;

[0071] capturing and recording exceptions occurring in the connection process, and re-throwing the exceptions for processing by a calling party.

[0072] In some embodiments, in step S103, counting the number of occurrences of the keyword in the log file on the server to obtain the first dictionary comprises:

[0073] building a grep command pipeline to count the number of occurrences of the keyword in the log file on the server, the grep command pipeline involving case-insensitive search, uniform conversion to lowercase, sorting, duplicate counting, and formatted output;

[0074] executing the grep command pipeline;

[0075] checking whether the grep command pipeline is executed successfully by reading stderr, throwing an exception and recording error information if error information is detected, and parsing the content of stdout if the grep command pipeline is executed successfully, splitting the “keyword: number” format data of each line into key-value pairs, and storing them into the first dictionary.

[0076] Specifically, the grep command pipeline uses a Shell pipe symbol to let grep act as a text filter to real-time intercept lines matching the pattern in the output of the previous command.

[0077] In some embodiments, in step S103, the RAID configuration command is executed, including:

[0078] Through the storcli64 tool, the alarm log information of the RAID controller is obtained, and the alarm log information is output to a log file.

[0079] Specifically, the storcli64 tool is a StorCLI64 storage command line tool, that is, a 64-bit command line storage management tool provided by Broadcom (originally LSI / Avago), which is used for all operations such as configuration, monitoring, maintenance and log collection of MegaRAID / Broadcom RAID controllers in the operating system, without entering the BIOS interface.

[0080] Referring to Figure 3 In some embodiments, before step S100, the RAID log detection method further includes the following steps:

[0081] Through the import configuration button of the visual interface, the import of the JSON configuration file is controlled to obtain the JSON configuration file.

[0082] Through the RAID configuration command input box and the execute RAID configuration button of the visual interface, the input of the RAID configuration command is controlled to obtain the RAID configuration command.

[0083] Through the analysis button of the visual interface, the analysis task of the RAID log is triggered.

[0084] Specifically, the target object can select a local JSON configuration file by clicking the "Import Configuration (JSON)" button, and the JSON configuration file contains remote node information, keywords, weights and composite rules, etc. After successful import, the program will prompt "Configuration file has been imported", and enable the "Start Analysis" and "Execute RAID Configuration" buttons. The target object is the user.

[0085] The target object can input the RAID configuration command in the visual interface and click the "Execute RAID Configuration" button. The program will connect to the first node specified in the JSON configuration file and execute the input RAID configuration command. The execution result (success or failure information) will be displayed in the text box. The node represents a server operating system, that is, a server.

[0086] Start analysis function: After clicking the "Start Analysis" button, the program will analyze the remote nodes' logs based on the imported RAID configuration file content. All operation results (analysis results, RAID configuration results) will be displayed in real-time in the text box of the interface, allowing the target object to easily view the analysis results of each node and the execution status of RAID configuration.

[0087] Specifically, remote node information: Set the remote server information that needs to be connected. Each node contains an IP address, username, and password for connecting to the remote server via SSH. In actual applications, the program will establish a connection based on this information and perform subsequent log analysis and RAID configuration operations. Example code is as follows:

[0088] "nodes": [

[0089] {"ip": "192.168.1.1", "username": "user", "password": "password"},

[0090] {"ip": "192.168.1.2", "username": "user", "password": "password"} ]

[0092] It should be understood that the keyword and weight description: Set the keywords and their weights that need to be focused on in log analysis. Keywords are used to search for specific error or warning information in log files, while weights are used to calculate the importance of each keyword. For example, the weight of "fail" is the highest (5.0), indicating that its appearance in the log may indicate a more serious problem. Example code is as follows:

[0093] "keywords": ["error", "warning", "fail"],

[0094] "keywordWeights": {"error": 3.0, "warning": 2.0, "fail": 5.0}

[0095] It can be understood that the composite rule description: Composite rules are used to detect more complex log patterns. For example, the Critical Error rule will only trigger when "error" and "fail" appear simultaneously; the Warning Detected rule will trigger when "warning" appears; and the Multiple Errors rule will trigger when the number of "error" occurrences reaches 5 or more. These rules help users identify specific failure patterns or problems. Example code is as follows:

[0096] "compositeRules": [

[0097] {

[0098] "name": "Critical Error",

[0099] "condition": "all",

[0100] "keywords": ["error", "fail"]

[0101] },

[0102] {

[0103] "name": "Warning Detected",

[0104] "condition": "any",

[0105] "keywords": ["warning"]

[0106] },

[0107] {

[0108] "name": "Multiple Errors",

[0109] "condition": "count",

[0110] "keywords": ["error"],

[0111] "threshold": 5

[0112] }

[0114] Specifically, the log path is as follows: "logPath": " / var / log / raid.log"

[0115] Specifically, the input RAID configuration command for the collection of RAID logs is: storcli64 / c0 show alilog> alilog.log

[0116] Among them, through the storcli64 tool, the alarm log information of the device with controller number 0 is obtained, and these information is output to the log file alilog.log.

[0117] ​In a specific embodiment, "storcli64 / c0 show alilog > alilog.log" is a shell command to collect RAID logs. After connecting to the operating system using SSH, this command is executed in the operating system to obtain the RAID logs.

[0118] It is important to note that the connection with the system is established using SSH: the main purpose of this function is to establish a secure remote connection channel through SSH. It does this by following these steps:

[0119] Creating an instance of SSHClient;

[0120] Setting the host key policy to automatically accept unknown host keys;

[0121] Establishing a connection using node configuration (IP address, port, username, password) and setting a timeout;

[0122] Catching and recording exceptions that occur during the connection process, and re-throwing the exception for the caller to handle.

[0123] Specifically, in this way, the program can efficiently establish a connection with the remote server and provide clear error information when problems are encountered. The example code is as follows:

[0124] # Establishing an SSH connection

[0125] def create_ssh_connection(node: Dict[str, Any]) ->paramiko.SSHClient:

[0126] ssh = paramiko.SSHClient()

[0127] ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

[0128] try:

[0129] ssh.connect(

[0130] hostname=node["ip"],

[0131] port=node.get("port", 22),

[0132] username=node["username"],

[0133] password=node["password"],

[0134] timeout=30, # Connection timeout setting )

[0136] return ssh

[0137] except Exception as err:

[0138] print(f"SSH connection failed {node['ip']}: {err}")

[0139] raise

[0140] Understandably, the core purpose of the remote log collection function `get_remote_keyword_stats` is to efficiently count the occurrences of specified keywords in log files on a remote server and return the results as a dictionary. It achieves this by building an efficient grep command pipeline, which includes case-insensitive searching, conversion to lowercase, sorting, deduplication, and formatted output. After command execution, the function checks for error output; if any is found, it throws an exception; otherwise, it parses the standard output, converting it to a dictionary format where keywords are the keys and the occurrence counts are the values.

[0141] During execution, the function uses paramiko's SSHClient to execute the constructed command on the remote server. It checks the command execution success by reading stderr; if an error occurs, an exception is thrown and logged. If the command executes successfully, the function parses the stdout content, splitting each line's "keyword:count" formatted data into key-value pairs and storing them in a first dictionary. Finally, the function returns the first dictionary, providing a clear and concise data structure for subsequent log analysis and processing. Example code is as follows:

[0142] # Remote Log Statistics

[0143] def get_remote_keyword_stats(ssh: paramiko.SSHClient, log_path: str,keywords: List[str]) -> Dict[str, int]:

[0144] keyword_pattern = "|".join(re.escape(kw) for kw in keywords)

[0145] cmd = f"""

[0146] grep -i -o -E '{keyword_pattern}' {log_path} | \

[0147] tr '[:upper:]' '[:lower:]' | \

[0148] sort | uniq -c | \

[0149] awk '{{print $2":"$1}}'

[0150] """

[0151] stdin, stdout, stderr = ssh.exec_command(cmd)

[0152] stderr_output = stderr.read().decode("utf-8")

[0153] if stderr_output:

[0154] raise Exception(f"Remote statistics failed: {stderr_output}")

[0155] stats = {}

[0156] for line in stdout.read().decode("utf-8").strip().split("\n"):

[0157] keyword, count = line.split(":")

[0158] stats[keyword] = int(count)

[0159] return stats

[0160] In a specific embodiment, the first step is to start batch log collection by clicking the "Execute RAID Configuration" button to obtain the latest RAID logs. Then, the "Start Analysis" button is clicked to analyze the RAID log content in batches.

[0161] Note that stderr and stdout are two standard I / O streams (file descriptors) automatically opened by Unix / Linux / Windows and other operating systems for each process to "output" information from the program internally to the terminal or file. Among them, stderr is used to output error / diagnosis information, and stdout is used to output normal results.

[0162] Step S200 specific implementation:

[0163] The method of the embodiment of the application traverses each keyword in the first dictionary, finds the corresponding weight of the keyword in the second dictionary, multiplies the number of occurrences of the keyword by the weight corresponding to the keyword, and accumulates to obtain a weighted score.

[0164] Specifically, the weighted score is calculated: the function calculate_weighted_score receives two parameters: stats (a dictionary containing keywords and their occurrence times, that is, the first dictionary described above) and keyword_weights (a dictionary containing keywords and their corresponding weights, that is, the second dictionary described above). It traverses each keyword in stats, finds the corresponding weight of the keyword in keyword_weights (if the keyword has no corresponding weight, the default value 1.0 is used), then multiplies the number of occurrences of the keyword by its weight, and accumulates to obtain the final weighted score. This weighted score reflects the severity of the problem in the log, and the higher the weight or the more the number of occurrences of the keyword, the greater the impact on the total score.

[0165] The specific execution process is as follows:

[0166] Initialize a variable score to 0, which is used to store the final weighted score;

[0167] Traverse each keyword and its occurrence times in the stats dictionary;

[0168] For each keyword, get its weight from the keyword_weights dictionary; if the keyword has no corresponding weight, use the default value 1.0;

[0169] Multiply the number of occurrences of the keyword by its weight, and accumulate the result to score;

[0170] After traversal, return the final weighted score score.

[0171] The example code is as follows:

[0172] # Calculate the weighted score

[0173] def calculate_weighted_score(stats: Dict[str, int], keyword_weights:Dict[str, float]) -> float:

[0174] score = 0

[0175] for kw, count in stats.items():

[0176] weight = keyword_weights.get(kw, 1.0)

[0177] score += count * weight

[0178] return score

[0179] Step S300 specific implementation:

[0180] The method of the embodiment of the application initializes an alarm list, and judges whether the keywords in the first dictionary meet the triggering condition of the composite rule; if the keywords in the first dictionary meet the triggering condition of the composite rule, the name of the composite rule is added to the alarm list.

[0181] In some embodiments, the composite rule includes a critical error rule, a warning detection rule and a multiple error rule; step S300 includes:

[0182] initializing the alarm list;

[0183] by traversing the composite rule, checking the keywords in the first dictionary,

[0184] when "error" and "fail" appear at the same time, the triggering condition of the critical error rule is met, and the name of the critical error rule is recorded to the alarm list;

[0185] when "warning" appears, the triggering condition of the warning detection rule is met, and the name of the warning detection rule is recorded to the alarm list;

[0186] when the number of occurrences of "error" reaches the first threshold, the triggering condition of the multiple error rule is met, and the name of the multiple error rule is recorded to the alarm list.

[0187] Specifically, the composite rule detection: the function receives two parameters: stats (a dictionary containing keywords and their occurrence counts, i.e., the first dictionary mentioned above) and composite_rules (a list containing multiple composite rules). Each composite rule defines a specific keyword combination pattern and its triggering condition. The function checks whether the keywords in stats meet the conditions of the composite rules by iterating through these composite rules. If the conditions are met, the name of the composite rule is added to the alert list. Finally, the list of all triggered composite rule names is returned.

[0188] Initialize the alert list: create an empty list alerts (i.e., the alert list mentioned above) to store the names of triggered composite rules, and iterate through each composite rule in the composite_rules list. Each composite rule is a dictionary containing the following fields:

[0189] name: the name of the composite rule.

[0190] condition: the condition type of the composite rule, which can be "all", "any", or "count".

[0191] keywords: the list of keywords to be detected.

[0192] threshold (only needed when condition is "count"): the minimum number threshold of keyword occurrences (i.e., the first threshold mentioned above).

[0193] Check according to the condition type:

[0194] "all": all keywords must appear. Use the all() function to check whether stats contains all the keywords defined in the rule. If all appear, add the rule name to the alerts list.

[0195] "any": any keyword appearance triggers. Use the any() function to check whether stats contains any of the keywords defined in the rule. If any appear, add the rule name to the alerts list.

[0196] "count": reach the specified number of keyword occurrences. Calculate the number of keywords that appear in stats, and if the number is greater than or equal to the first threshold defined in the rule, add the rule name to the alerts list.

[0197] Return the list of triggered composite rule names: after the iteration is complete, return the list alerts containing all the triggered rule names.

[0198] Example: stats is:

[0199] stats = {"error": 2, "warning": 1, "fail": 1}

[0200] composite_rules is:

[0201] composite_rules = [

[0202] {"name": "Critical Error", "condition": "all", "keywords": ["error", "fail"]},

[0203] {"name": "Warning Detected", "condition": "any", "keywords": ["warning"]},

[0204] {"name": "Multiple Errors", "condition": "count", "keywords": ["error"], "threshold": 3}

[0206] The function execution process is as follows:

[0207] Check the "Critical Error" rule (i.e., the aforementioned critical error rule): the condition is "all", and "error" and "fail" are included in stats, so this rule is triggered.

[0208] Check the "Warning Detected" rule (i.e., the aforementioned warning detected rule): the condition is "any", and "warning" is included in stats, so this rule is triggered.

[0209] Check the "Multiple Errors" rule (i.e., the aforementioned multiple errors rule): the condition is "count", and "error" appears twice in stats, which is less than the first threshold of 3, so this rule is not triggered.

[0210] The final returned alerts list is:

[0211] ["Critical Error", "Warning Detected"]

[0212] ​In this way, the function can efficiently detect complex error patterns in the log and trigger corresponding alerts according to predefined rules.

[0213] The example code for composite rule detection is as follows:

[0214] # Composite rule detection

[0215] def check_composite_patterns(stats: Dict[str, int], composite_rules:List[Dict[str, Any]]) -> List[str]:

[0216] alerts = []

[0217] for rule in composite_rules:

[0218] if rule["condition"] == "all" and all(kw in stats for kw inrule["keywords"]):

[0219] alerts.append(rule["name"])

[0220] elif rule["condition"] == "any" and any(kw in stats for kw inrule["keywords"]):

[0221] alerts.append(rule["name"])

[0222] elif rule["condition"] == "count":

[0223] matched = sum(1 for kw in rule["keywords"] if kw instats)

[0224] if matched >= rule["threshold"]:

[0225] alerts.append(rule["name"])

[0226] return alerts

[0227] Step S400 specific implementation:

[0228] The method of the embodiment of the application stores the first dictionary, the weighted score and the alarm list into a result dictionary, wherein the IP address of the server is the key and the first dictionary, the weighted score and the alarm list are the values.

[0229] In some embodiments, in step S400, if the SSH connection fails or the log analysis is wrong, an exception is captured and error information is stored into the result dictionary, wherein the IP address of the server is the key and the error information is the value.

[0230] Specifically, the first dictionary, the weighted score and the alarm list are stored into the result dictionary, wherein the IP address of the server is the key and the first dictionary, the weighted score and the alarm list are the values in the form of a dictionary; if the SSH connection fails or the log analysis is wrong, an exception is captured and error information is stored into the result dictionary, wherein the IP address of the server is the key and the error information is the value. The result dictionary is the result of the RAID log detection.

[0231] It can be understood that the analysis is performed in parallel.

[0232] The results are collected asynchronously: the concurrent.futures.as_completed method is used to asynchronously wait for the completion of the tasks. When a task is completed, the program obtains the result from the Future object.

[0233] The success and failure cases are handled: if a task is successfully completed, the result is stored into the results dictionary (i.e. the aforementioned result dictionary) with the IP address of the node as the key. If a task fails (for example, the SSH connection fails or the log analysis is wrong), an exception is captured and error information is recorded, and a dictionary containing the error information is stored into the results dictionary for the node.

[0234] The results in a unified format are returned: finally, the function returns a dictionary, wherein each key is the IP address of a node and the value is a dictionary containing the analysis result. The result dictionary contains the keyword statistics, the weighted score and the triggered composite rules, etc.

[0235] Specifically, an example is as follows:

[0236] Suppose that the nodes contain two nodes:

[0237] nodes = [

[0238] {"ip": "192.168.1.1", "username": "user", "password": "password"},

[0239] {"ip": "192.168.1.2", "username": "user", "password": "password"} ]

[0241] config contains configuration information such as keywords, weights, and composite rules.

[0242] The execution process is as follows:

[0243] First, create a thread pool with a maximum concurrency of 10 and submit the analyze_node task (i.e., the aforementioned RAID log analysis task) for each node:

[0244] For 192.168.1.1, the analyze_node function connects to the node, obtains log statistics, calculates the weighted score, and detects composite rules.

[0245] For 192.168.1.2, the same operations are performed.

[0246] Second, when the analysis task for a node is completed, the program retrieves the result from the Future object and stores it in the results dictionary; if the analysis of a node fails, the program catches the exception and records the error information.

[0247] Third, the final returned results dictionary is as follows:

[0248] {

[0249] "192.168.1.1": {

[0250] "stats": {"error": 2, "warning": 1},

[0251] "score": 7.0,

[0252] "alerts": ["Warning Detected"]

[0253] },

[0254] "192.168.1.2": {

[0255] "stats": {"error": 3, "fail": 1},

[0256] "score": 14.0,

[0257] "alerts": ["Critical Error"]

[0258] }

[0259] }

[0260] In this way, the program can efficiently process analysis tasks on multiple nodes in parallel, significantly improving overall efficiency.

[0261] The complete code example is as follows:

[0262] Step 6: Parallel Execution Analysis

[0263] def analyze_all_nodes(nodes: List[Dict[str, Any]], config: Dict[str,Any]) -> Dict[str, Dict[str, Any]]:

[0264] results = {}

[0265] with concurrent.futures.ThreadPoolExecutor(max_workers=10) asexecutor:

[0266] futures = {

[0267] executor.submit(analyze_node, node, config): node["ip"]for node in nodes

[0268] }

[0269] for future in concurrent.futures.as_completed(futures):

[0270] ip = futures[future]

[0271] try:

[0272] results[ip] = future.result()

[0273] except Exception as err:

[0274] print(f"Analysis of node {ip} failed: {err}")

[0275] results[ip] = {"status": f"Analysis failed: {err}", "error":True}

[0276] return results

[0277] def analyze_node(node: Dict[str, Any], config: Dict[str, Any]) ->Dict[str, Any]:

[0278] analyzer = RemoteLogAnalyzer(config)

[0279] ssh = create_ssh_connection(node)

[0280] stats = get_remote_keyword_stats(ssh, analyzer.log_path,analyzer.keywords)

[0281] score = calculate_weighted_score(stats, analyzer.keyword_weights)

[0282] alerts = check_composite_patterns(stats, analyzer.composite_rules)

[0283] ssh.close()

[0284] return {"stats": stats, "score": score, "alerts": alerts}

[0285] Specifically, the display of the analysis result of the RAID log is shown in Figure 4 .

[0286] In specific embodiments, the RAID log detection method comprises:

[0287] Importing a JSON configuration file to obtain the JSON configuration file;

[0288] SSH connecting a server according to the JSON configuration file;

[0289] Executing a RAID configuration command to collect RAID logs to a log file;

[0290] According to the log file, performing an analysis task of the RAID log.

[0291] Specifically, the RAID configuration command is executed: collect the RAID log into a log file. SSH connection (Secure Shell Connection) is an encrypted network session used to securely remotely log in, execute commands, transfer files, or manage servers in an insecure network.

[0292] In specific embodiments, the analysis task of the RAID log is performed, including:

[0293] Count the number of occurrences of the keywords in the log file on the server to obtain a first dictionary;

[0294] Traverse each keyword in the first dictionary, find the corresponding weight of the keyword in the second dictionary, multiply the number of occurrences of the keyword by the corresponding weight of the keyword, and accumulate to obtain a weighted score;

[0295] Initialize the alarm list, and determine whether the keyword in the first dictionary meets the triggering condition of the composite rule; if the keyword in the first dictionary meets the triggering condition of the composite rule, the name of the composite rule is added to the alarm list;

[0296] Store the first dictionary, the weighted score, and the alarm list in the result dictionary, wherein the IP address of the server is the key, and the first dictionary, the weighted score, and the alarm list are the values.

[0297] In one possible implementation, after detecting the composite rule (i.e., the aforementioned step S300), the analysis is performed in parallel. The parallel analysis includes the following steps:

[0298] Create a thread pool and submit the analysis task of the RAID log of each node;

[0299] Asynchronously wait for the completion of the analysis task of the RAID log;

[0300] When the analysis task of the RAID log of a certain node is completed, the analysis result is obtained from the Future object, and the analysis result is stored in the result dictionary;

[0301] If the analysis task of the RAID log of a certain node fails, the exception is captured and the error information is recorded, and the error information is stored in the result dictionary;

[0302] Return the result dictionary and display the content of the result dictionary on the visual interface.

[0303] It can be seen that the present application places the collection and analysis of logs on the client by means of SSH remote connection and sending shell commands, and does not need to return, which greatly simplifies the process and breaks through the problem of traditional analysis of large logs. The present application forms a solution for cross-system management, network security and data security by means of a log analysis algorithm (keyword weighting + composite rule) and parallel computing (TreadPool). The present application provides a visual interface with interactivity and convenience. The present application is very light, has few dependencies, and has the advantages of strong practicality and convenient deployment.

[0304] The embodiment of the present application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method described above when executing the computer program. The electronic device can be any intelligent terminal such as a computer.

[0305] The embodiment of the present application also provides a computer readable storage medium, which stores program instructions, and the program instructions are executed by a processor to implement the method described above.

[0306] It should be appreciated that the method steps in the embodiment of the present application can be realized or implemented by computer hardware, a combination of hardware and software, or through computer instructions stored in a non-transitory computer readable storage medium. The method can use standard programming techniques. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can run on a programmed special-purpose integrated circuit for this purpose.

[0307] In addition, the operations of the processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) can be performed under the control of one or more computer systems configured with executable instructions (e.g., executable instructions, one or more computer programs or one or more applications), by hardware or a combination thereof. The computer programs include a plurality of instructions executable by one or more processors.

[0308] Further, the methods can be implemented in any type of computing platform operably connected to a suitable computing platform, including but not limited to a personal computer, mini-computer, mainframe, workstation, network or distributed computing environment, separate or integrated computer platforms, or in communication with charged particle tools or other imaging devices, and the like. Aspects of the present application can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated to the computing platform, such as a hard disk, optical read and / or write storage media, RAM, ROM, and the like, such that it can be read by a programmable computer to configure and operate the computer to perform the processes described herein when the storage medium or device is read by the computer. In addition, the machine-readable code, or portions thereof, can be transmitted over wired or wireless networks. The present application described herein includes these and other different types of non-transitory computer readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The present application can also include the computer itself when programmed in accordance with the methods and techniques described herein.

[0309] The computer program can be applied to input data to perform the functions described herein to transform the input data to generate output data that is stored to non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present application, the transformed data represents a physical and tangible object, including a particular visual depiction of the physical and tangible object produced on a display.

[0310] The above description is only preferred embodiments of the present application, and the present application is not limited to the above-described embodiments, but any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the present application. The technical solutions and / or embodiments of the present application can have various modifications and changes within the scope of the present application.

Claims

1. A method of RAID log detection, the method comprising: The method comprises the following steps: S100, obtaining a first dictionary, a second dictionary and a composite rule on a server, wherein the first dictionary comprises keywords and the number of occurrences of the keywords, and the second dictionary comprises the keywords and the weights corresponding to the keywords; S200, searching for the weight corresponding to each keyword in the first dictionary in the second dictionary, multiplying the number of occurrences of the keyword by the weight corresponding to the keyword, and accumulating to obtain a weighted score; S300, initializing an alarm list, and determining whether the keyword in the first dictionary meets the triggering condition of the composite rule; if the keyword in the first dictionary meets the triggering condition of the composite rule, the name of the composite rule is added to the alarm list; S400, storing the first dictionary, the weighted score and the alarm list into a result dictionary, wherein the IP address of the server is the key, and the first dictionary, the weighted score and the alarm list are the values.

2. The RAID log detection method of claim 1, wherein, The step S100 comprises: S101, obtaining a JSON configuration file, wherein the JSON configuration file comprises remote node information, the keywords, the weights corresponding to the keywords and the composite rule; S102, connecting the server by using SSH according to the remote node information; S103, executing a RAID configuration command to count the number of occurrences of the keywords in a log file on the server to obtain the first dictionary; S104, obtaining the second dictionary according to the keywords and the weights corresponding to the keywords.

3. The RAID log detection method of claim 2, wherein, The remote node information comprises the IP address, the username and the password of the server; the step S102 comprises: creating an SSHClient instance; setting a host key policy to automatically accept unknown host keys; connecting the server by using SSH and setting a timeout time by using the IP address, the username and the password of the server; capturing and recording exceptions occurring in the connection process, and re-throwing the exceptions for processing by a calling party.

4. The RAID log detection method of claim 2, wherein, In the step S103, counting the number of occurrences of the keywords in the log file on the server to obtain the first dictionary comprises: constructing a grep command pipeline to count the number of occurrences of the keywords in the log file on the server, the grep command pipeline involving case-insensitive search, uniform conversion to lowercase, sorting, duplicate count and formatted output; executing the grep command pipeline; checking whether the grep command pipeline is executed successfully by reading stderr, throwing an exception and recording error information if error information is detected, and parsing the content of stdout if the grep command pipeline is executed successfully, splitting the "keyword: number” format data of each line into key-value pairs, and storing the key-value pairs into the first dictionary.

5. The method of claim 2, wherein, In the step S103, executing the RAID configuration command comprises: obtaining alarm log information of a RAID controller by using a storcli64 tool, and outputting the alarm log information to the log file.

6. The method of claim 1, wherein, The composite rules include a key error rule, a warning detection rule and a multiple error rule; the step S300 includes: initializing the alarm list; checking the key in the first dictionary by traversing the composite rules, when "error" and "fail" appear simultaneously, the trigger condition of the key error rule is met, the name of the key error rule is recorded to the alarm list; when "warning" appears, the trigger condition of the warning detection rule is met, the name of the warning detection rule is recorded to the alarm list; when the number of "error" reaches a first threshold, the trigger condition of the multiple error rule is met, the name of the multiple error rule is recorded to the alarm list.

7. The method of claim 1, wherein, In the step S400, if the SSH connection fails or the log analysis is wrong, an exception is captured and error information is stored in the result dictionary, wherein the IP address of the server is the key and the error information is the value.

8. The RAID log detection method of claim 1, wherein, Before the step S100, the RAID log detection method further includes the following steps: controlling the import of a JSON configuration file through an import configuration button of a visual interface to obtain the JSON configuration file; controlling the input of a RAID configuration command through a RAID configuration command input box and a RAID configuration execution button of the visual interface to obtain the RAID configuration command; controlling the triggering of an analysis task of the RAID log through an analysis button of the visual interface.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the method in any one of claims 1-8. 10.A computer readable storage medium having program instructions stored thereon, the program instructions being executed by a processor to implement the method in any one of claims 1-8.

Images