Storage system fault root cause positioning method, device and equipment and storage medium

Abstract

The invention discloses a storage system fault root cause positioning method and device, equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: aiming at a fault occurring in the storage system, determining a potential fault time period in which the fault occurs on the basis of at least one of the number of logs generated by the storage system in a target time period and the information amount of the logs, so that the logs generated by the storage system in the time period are preliminarily filtered; according to the method, the matching degree between the fault and the candidate logs generated in the fault time period is combined, the candidate logs generated in the fault time period are further filtered, and the root cause positioning reference information is determined according to the target logs obtained after filtering, so that the number of the logs for positioning the fault root cause is greatly reduced, and the positioning accuracy of the fault root cause is improved. The time consumption for positioning the fault root cause is reduced, and the positioning efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for locating the root cause of a storage system failure. Background Art

[0002] With the continuous development of computer storage technology, the internal structure of storage systems is becoming more and more complex. At present, storage systems often generate massive system operation logs within a certain operation cycle. For example, millions or even tens of millions of lines of logs are generated in a week.

[0003] In the related art, when a storage system fails, the logs generated during the system operation are usually analyzed to locate the root cause of the failure. For example, sample logs with known root causes of failures are regularized to build a rule base of known root causes of failures. When a storage system fails, the logs generated by the storage system are matched line by line with the rules stored in the rule base, and the root cause of the failure is located based on the successfully matched logs.

[0004] However, since the number of logs generated by the storage system in a certain operation cycle is relatively large, the number of logs matched by the above method is also large, resulting in a long time consumption and low efficiency in locating the root cause of the fault. Summary of the invention

[0005] The embodiments of the present application provide a storage system fault root cause location method, device, equipment and storage medium, which can effectively reduce the time consumption of locating the fault root cause and improve the location efficiency. The technical solution is as follows:

[0006] In a first aspect, a method for locating a root cause of a storage system failure is provided, the method comprising:

[0007] In the event of a storage system failure, based on logs generated by the storage system within a target time period, determining at least one failure time period in which the failure occurs from the target time period, wherein the logs generated within the failure time period meet at least one of a quantity condition and an information volume condition;

[0008] Based on the matching degree between the fault and each candidate log generated in the at least one fault time period, filtering the plurality of candidate logs generated in the at least one fault time period to obtain at least one target log;

[0009] Based on at least one of the target logs, reference information for locating the root cause of the fault is determined.

[0010] Through the above method, for a failure occurring in a storage system, based on at least one of the number of logs and the amount of log information generated by the storage system within a target time period, a potential failure time period in which the failure occurs is determined, thereby preliminarily filtering the logs generated by the storage system within this time period, and further, combining the matching degree between the failure itself and the candidate logs generated within the failure time period, further filtering multiple candidate logs generated within the failure time period, and determining the root cause location reference information based on the target log obtained after filtering, thereby greatly reducing the number of logs used to locate the root cause of the failure, reducing the time consumption of locating the root cause of the failure, and improving the location efficiency.

[0011] In some embodiments, the determining, based on the logs generated by the storage system in the target time period, at least one fault time period in which the fault occurs from the target time period includes any of the following:

[0012] Determining at least one fault time period from each sub-time period of the target time period based on the number of logs generated by the storage system in each sub-time period of the target time period;

[0013] Determine the at least one fault time period from each sub-time period of the target time period based on the amount of log information generated by the storage system in each sub-time period of the target time period;

[0014] The at least one fault time period is determined from each sub-time period of the target time period based on the number of logs and the amount of log information generated by the storage system in each sub-time period of the target time period.

[0015] Through the first method mentioned above, based on the distribution of the number of logs generated in the target time period, the potential failure time period is determined from the target time period, which effectively reduces the amount of logs to be processed later and realizes mass log filtering.

[0016] Through the second method described above, based on the distribution of the amount of log information generated in the target time period, the potential failure time period is determined from the target time period, which effectively reduces the amount of logs to be processed subsequently and realizes mass log filtering.

[0017] Through the third method mentioned above, based on the distribution of the number of logs generated in the target time period, some candidate time periods are first determined from the target time period. In other words, by performing anomaly detection on the sequence obtained by performing log number statistics on the time dimension, candidate potential fault time periods can be determined. Then, based on the distribution of the amount of log information generated in these candidate time periods, these candidate time periods are further screened, thereby comprehensively considering the characteristics of the number of logs and the amount of log information when the fault occurs, achieving large-scale streamlining of the number of logs, greatly reducing the amount of logs that need to be processed later, and thereby improving the efficiency of root cause location.

[0018] In some embodiments, determining the at least one fault time period from each sub-time period of the target time period based on the number of logs and the amount of log information generated by the storage system in each sub-time period of the target time period includes:

[0019] Determine at least one candidate time period from each sub-time period of the target time period based on the number of logs generated in each sub-time period of the target time period;

[0020] Based on the logs generated in each of the candidate time periods, determining the information entropy corresponding to each of the candidate time periods, the information entropy indicating the amount of information related to the root cause location of the fault in the logs generated in the candidate time period;

[0021] Based on the information entropy corresponding to each of the candidate time periods, the at least one fault time period is determined from the at least one candidate time period.

[0022] In some embodiments, filtering the multiple candidate logs generated in the at least one fault time period based on the matching degree between the fault and each candidate log generated in the at least one fault time period to obtain at least one target log includes:

[0023] Determine the matching degree between the fault and each of the candidate logs based on the description information of the fault and the log information of each of the candidate logs;

[0024] Based on the matching degree between the fault and each of the candidate logs, the matching degrees corresponding to the plurality of candidate logs are sorted, and the candidate log corresponding to the matching degree ranked in the first digit is determined as the target log.

[0025] In some embodiments, determining the matching degree between the fault and each candidate log based on the description information of the fault and the log information of each candidate log includes:

[0026] The description information and the log information of the candidate log are input into a neural network model, and feature extraction is performed on the description information and the log information respectively through the neural network model to obtain a feature vector of the description information and a feature vector of the log information, and based on the feature vector of the description information and the feature vector of the log information, a matching degree between the fault and the candidate log is output.

[0027] In some embodiments, the log information includes text content of the candidate log; or,

[0028] The log information includes the text content of the candidate log and at least one of the following: the system version number of the storage system, the log category to which the candidate log belongs, the generation time of the candidate log, and the position of the candidate log in the log package to which it belongs.

[0029] Through the above method, combined with the fault phenomenon itself and business knowledge such as log information, multiple candidate logs generated during the fault time period are further filtered, reducing the amount of logs to be processed later, thereby improving the efficiency of root cause location.

[0030] In some embodiments, the root cause location reference information indicates at least one root cause location reference log of the fault;

[0031] The determining, based on at least one of the target logs, the root cause location reference information of the fault includes:

[0032] Based on the matching degree between the fault and each of the target logs, the matching degrees corresponding to the multiple target logs are sorted, and the target log corresponding to the matching degree ranked in the top second digit is determined as the root cause location reference log.

[0033] The above method can be applied to scenarios where the root cause of a fault is unknown. Since a small number of reference logs for root cause location are filtered out from the massive logs generated by the storage system within the target time period, a large-scale reduction in the number of logs is achieved, making it easier for relevant personnel to quickly locate the root cause of the fault, thereby improving the efficiency of root cause location.

[0034] In some embodiments, the root cause location reference information indicates at least one reference root cause of the fault;

[0035] The determining, based on at least one of the target logs, the root cause location reference information of the fault includes:

[0036] Based on a plurality of sample logs marked with fault root causes and at least one target log, determining a hit number of the target logs matched by each fault root cause;

[0037] The hit numbers corresponding to the plurality of root causes of the fault are sorted, and the root cause of the fault corresponding to the hit number ranked in the first third order is determined as the reference root cause.

[0038] The above method can be applied to the scenario where the root cause of the fault is known. It should be understood that since the sample log is marked with the root cause of the fault, by matching the sample log under a certain root cause of the fault with the target log, if the match is successful, it indicates that the target log has a high probability of belonging to this root cause of the fault. For example, a keyword in the sample log is associated with the root cause of the fault. If a target log also includes this keyword, then the target log has a high probability of belonging to this root cause of the fault. In this way, the root cause of the fault is recommended to relevant personnel as a reference root cause, thereby achieving accurate root cause positioning.

[0039] In some embodiments, the method is applied to a fault root cause location system, and the fault root cause location system is configured to:

[0040] a storage device of the storage system;

[0041] a data management engine of the storage system;

[0042] The cloud service platform of the storage system.

[0043] In a second aspect, an embodiment of the present application provides a storage system fault root cause location device, which includes at least one functional module for executing the storage system fault root cause location method provided by the aforementioned first aspect or any possible implementation of the first aspect.

[0044] In a third aspect, a computing device is provided, which includes a processor and a memory, the memory being used to store at least one section of program code, the at least one section of program code being loaded by the processor and executed by a method for locating the root cause of a storage system failure as provided in the first aspect or any possible implementation of the first aspect.

[0045] In a fourth aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the storage system fault root cause locating method provided by the aforementioned first aspect or any possible implementation of the first aspect.

[0046] In a fifth aspect, a computer-readable storage medium is provided, the computer-readable storage medium being used to store at least one section of program code, and when the at least one section of program code is executed by a computing device, the computing device is caused to execute the storage system fault root cause location method provided in the first aspect or any possible implementation of the first aspect. The storage medium includes but is not limited to a volatile memory, such as a random access memory, a non-volatile memory, such as a flash memory, a hard disk drive (HDD), and a solid state drive (SSD).

[0047] In a sixth aspect, a computer program product is provided, which, when executed on a computing device, enables the computing device to execute the method for locating the root cause of a storage system fault as provided in the first aspect or any possible implementation of the first aspect. The computer program product may be a software installation package, and when the functions of the computing device described above need to be implemented, the computer program product may be downloaded and executed on the computing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0049] Figure 2 is a schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application;

[0050] Figure 3 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0051] Figure 4 This is a schematic diagram of a connection method of a computing device cluster provided in an embodiment of the present application;

[0052] Figure 5 It is a flow chart of a method for locating the root cause of a storage system failure provided by an embodiment of the present application;

[0053] Figure 6 It is a schematic diagram of determining a fault time period provided by an embodiment of the present application;

[0054] Figure 7 is a schematic diagram of determining a target log provided by an embodiment of the present application;

[0055] Figure 8 It is a schematic diagram of a method for locating the root cause of a storage system failure provided by an embodiment of the present application;

[0056] Fig. 9 is a schematic diagram of a fault root cause location system provided in an embodiment of the present application;

[0057] Fig.10 It is a structural schematic diagram of a storage system fault root cause locating device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the logs generated by the storage system involved in this application, the description information of the fault, etc. are all obtained with full authorization.

[0059] For ease of understanding, the key terms and key concepts involved in this application are explained below.

[0060] Logs are information that records hardware, software, and system problems in a system. Taking a storage system as an example, users can use the logs generated by the storage system during operation to check the cause of a system failure (i.e., locate the root cause of the failure), and can also use the logs generated by the system to query traces left by illegal system operations or attacks, etc., but the present application is not limited to this.

[0061] Artificial intelligence (AI), the basic principle of AI is to combine massive data with super-powerful computing and processing capabilities and intelligent algorithms to build an AI model that solves specific problems, so that the AI ​​model can automatically summarize and learn potential patterns or features from the data, thereby achieving a way of thinking close to that of humans. The AI ​​model is also the AI ​​algorithm (or AI operator), which is a general term for mathematical algorithms built on the principles of AI, and is also the basis for using AI to solve specific problems.

[0062] A neural network model is a type of mathematical algorithm AI model that imitates the structure and function of a biological neural network (the central nervous system of an animal). A neural network model can include a variety of neural network layers with different functions, each of which includes parameters and calculation formulas. Different layers in a neural network model have different names depending on the calculation formula or function. For example, a layer that performs convolution calculations is called a convolution layer, which is often used to extract features from input data. A neural network model can also be composed of a combination of multiple existing neural network models. Neural network models with different structures can be used in different scenarios or provide different effects when used in the same scenario. Different structures of neural network models specifically include one or more of the following: different numbers of network layers in the neural network model, different orders of the network layers, and different weights, parameters or calculation formulas in each network layer.

[0063] The Transformer model is a deep neural network model based on the self-attention mechanism, which is widely used in various tasks in the field of natural language processing, such as text classification, machine translation, and question-answering systems. In the Chinese context, the Chinese name of the Transformer model can be translated as "converter model", which means that the model can convert the input sequence into the output sequence while retaining the important information in the input sequence.

[0064] Term frequency-inverse document frequency (TF-IDF) is a word frequency statistical method used to evaluate the importance of a word to a document set or a document in a corpus. The importance of a word increases in direct proportion to the number of times it appears in a document, but decreases in inverse proportion to the frequency of its appearance in the corpus.

[0065] Information entropy is a basic concept in information theory, which is used to describe the uncertainty of various possible events in an information source. In some embodiments, the average amount of information after excluding redundancy in the information is called information entropy, that is, information entropy can be understood as the probability of occurrence of a certain specific information. In the embodiment of the present application, the "information amount" of the log is quantified by the "information entropy" of the log. The larger the information entropy of the log, the larger the amount of information in the log.

[0066] The following is an introduction to the application scenarios and implementation environment involved in this application.

[0067] The technical solution provided by the embodiment of the present application is applied to the field of computer technology. It can filter out a small number of key logs that can be used to locate the root cause of a fault occurring in a storage system based on the massive logs generated by the storage system during operation, thereby reducing the time spent on locating the root cause of the fault and effectively improving the efficiency of root cause locating.

[0068] Schematically, refer to Figure 1 , Figure 1 Schematic diagram of an implementation environment provided by the embodiment of the present application. Figure 1 As shown, the implementation environment includes a storage system 101 and a fault root cause locating system 102. The storage system 101 is directly or indirectly connected to the fault root cause locating system 102 via a wireless network or a wired network.

[0069] The storage system 101 is used to provide storage services. Schematically, the storage system 101 includes a host, a virtual machine, a network device (such as a switch, a router, etc.), and a storage device, etc., which is not limited in this application. The nodes in the storage system 101 are connected to each other through a wired network or a wireless network. The logs generated by the storage system 101 during operation can be stored in the storage system 101, or sent to the fault root cause location system 102, and stored by the fault root cause location system 102, which is not limited in this application.

[0070] The fault root cause location system 102 is used to provide a fault root cause location service for a fault occurring in the storage system 101 .

[0071] Schematically, the fault root cause location system 102 includes a terminal 1021 and a server 1022, and the terminal 1021 and the server 1022 are connected to each other through a wired network or a wireless network. Among them, the terminal 1021 can be at least one of a smart phone, a desktop computer, an augmented reality terminal, a tablet computer, an e-book reader, and a laptop. The terminal 1021 is installed and runs an application that supports root cause location of the fault occurring in the storage system 101. The application can be a client application or a browser application, etc., which is not limited. For example, the application running on the terminal 1021 is a client application. In the case of a fault in the storage system 101, the user (such as a fault location personnel) can query the logs generated by the storage system 101 during operation through the client application, and locate the root cause of the fault by analyzing the queried logs, so as to facilitate timely strengthening of system maintenance and management. The server 1022 is used to provide background services for the application running on the terminal 1021. The server 1022 is an independent physical server, or a server cluster or distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Taking the server 1022 as a cloud server as an example, in this scenario, the fault root cause location system 102 can also be understood as a cloud service platform of the storage system 101, wherein the cloud server refers to a service based on hardware resources and software resources, providing computing, network and storage capabilities. Through the network "cloud", the huge data is calculated and processed at the remote end for processing and analysis and then returned to the user, which has the characteristics of large-scale, distributed, virtualized, high availability, scalability, on-demand service and security.

[0072] It should be noted that the above deployment method of the fault root cause locating system 102 is only an example, and the present application does not limit the deployment method of the fault root cause locating system 102.

[0073] For example, the fault root cause location system 102 can also be deployed in the data management engine of the storage system 101, where the data management engine is an intelligent storage management platform with the three advantages of integration, intelligence, and openness. It can simplify storage management and operation and maintenance, improve the operating efficiency of data centers, and enable automatic driving of the entire life cycle of storage.

[0074] For another example, the fault root cause location system 102 can also be deployed in a storage device of the storage system 101, so that the storage system 101 can provide not only storage services but also fault root cause location services when a fault occurs.

[0075] For another example, the fault root cause location system 102 can also be deployed synchronously on the storage device and data management engine of the storage system 101, or synchronously deployed on the storage device and cloud service platform of the storage system 101. In this way, the storage system 101 and the fault root cause location system 102 can work together to achieve fault root cause location (this process will be introduced in subsequent method embodiments and will not be repeated here).

[0076] In some embodiments, the wireless network or wired network described above uses standard communication technology and / or protocol. The network includes, but is not limited to, any combination of a data center network, a storage area network (SAN), a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or a virtual private network. In some implementations, the data exchanged through the network is represented by technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML), etc. In addition, conventional encryption technologies such as secure socketslayer (SSL), transport layer security (TLS), virtual private network (VPN), and Internet protocol security (IPsec) can be used to encrypt all or part of the links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0077] The hardware structure of the terminal and server in the above implementation environment is introduced below.

[0078] The present application embodiment provides a computing device that can be configured as any device involved in the above-mentioned fault root cause location system 102. Figure 2 , Figure 2 Schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application. Figure 2 As shown, the computing device 200 includes a memory 201, a processor 202, a communication interface 203 and a bus 204. The memory 201, the processor 202 and the communication interface 203 are connected to each other through the bus 204.

[0079] The memory 201 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. Schematically, taking the computing device that can be configured as the above-mentioned terminal 101 as an example, the memory 201 is used to store at least one section of program code, and when the program code stored in the memory 201 is executed by the processor 202, the processor 202 is used to execute the storage system fault root cause location method provided in the following method embodiment.

[0080] The processor 202 may be a network processor (NP), a central processing unit (CPU), an application-specific integrated circuit (ASIC), or an integrated circuit for controlling the execution of the program of the present application. The processor 202 may be a single-CPU processor or a multi-CPU processor. The number of processors 202 may be one or more. The communication interface 203 uses a transceiver module such as a transceiver to implement communication between the computing device 200 and other devices or communication networks. For example, data may be acquired through the communication interface 203.

[0081] The memory 201 and the processor 202 may be separately provided or integrated together.

[0082] Bus 204 may include a path for transmitting information between various components of computing device 200 (eg, memory 201 , processor 202 , communication interface 203 ).

[0083] The embodiment of the present application also provides a computing device cluster, which includes multiple computing devices. The multiple computing devices include servers and terminals, etc., wherein the server can be a server, such as a central server, an edge server, or a local server in a local data center, and the terminal can be a desktop computer, a laptop computer, or a smart phone. Figure 3 , Figure 3 Schematic diagram of a computing device cluster provided in an embodiment of the present application. Figure 3 As shown, the computing device cluster includes multiple computing devices 200. The memory 201 in the multiple computing devices 200 in the computing device cluster may store the same instructions for executing the storage system fault root cause location method. In some embodiments, the memory 201 of the multiple computing devices 200 in the computing device cluster may also store partial instructions for executing the storage system fault root cause location method. In other words, the combination of the multiple computing devices 200 is used to jointly execute the instructions of the storage system fault root cause location method.

[0084] In some embodiments, multiple computing devices 200 in a computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 4 Schematic diagram of a connection method of a computing device cluster provided in an embodiment of the present application. Figure 4 As shown, two computing devices 200 are connected via a network. Specifically, the network is connected via the communication interface 203 in each computing device 200. It should be understood that Figure 4 The functionality of the computing device 200 shown in FIG. 2 may also be performed by multiple computing devices 200 .

[0085] The following is an introduction to the method for locating the root cause of a storage system failure provided by this application.

[0086] It should be understood that the internal structure of the storage system is often relatively complex, including dozens of physical and logical components such as controllers, front-end ports, back-end ports, various interface cards, hard disks, optical ports, etc. The storage system will generate massive logs during operation. In the event of a storage system failure, the root cause of the failure is often located by analyzing the logs generated by the storage system within a certain operation cycle. For example, after the storage device of the storage system crashes, the root cause of the failure is located to be memory damage by analyzing the logs generated by the storage system. For another example, the service of the storage system is interrupted or, by analyzing the logs generated by the storage system, the root cause of the failure is located to be system downtime maintenance, and so on. Based on this, the present application provides a method for locating the root cause of a storage system failure, which can filter out a small number of key logs that can be used to locate the root cause of the failure from the massive logs generated by the storage system within a certain operation cycle, greatly reducing the time spent on locating the root cause of the failure and improving the efficiency of root cause locating.

[0087] Figure 5 1 is a flow chart of a method for locating the root cause of a storage system failure provided by an embodiment of the present application. Figure 5 As shown, the method is applied to the fault root cause location system 102 in the above implementation environment as an example for introduction. The storage system fault root cause location method is implemented through interaction between the terminal 1021 and the server 1022, including the following steps 501 to 507.

[0088] 501. When a storage system fails, the server obtains logs generated by the storage system within a target time period.

[0089] In an embodiment of the present application, a server is used to provide a root cause location service for a storage system failure. The present application does not limit the type of failure that occurs in the storage system. For example, the failure may be a network communication interruption, a storage device crash, a storage system upgrade interruption, a service interruption, and the like. The storage system is connected to the server in communication. In the event of a storage system failure, the server obtains the logs generated by the storage system during a target time period through the storage system. The target time period refers to a time period related to a storage system failure, and can be set according to actual needs. For example, if a storage system failure is detected at 10 o'clock, 9 to 10 o'clock is determined as the target time period, and so on. It should be understood that this is only an example and does not constitute a limitation of the present application.

[0090] In some embodiments, the storage system sends the logs generated within a preset time period to the server at intervals of a preset time period, and the server stores the logs generated by the storage system in a log database. The preset time period can be set by oneself, for example, the preset time period is 1 day, 1 week, and so on. In this way, the server can obtain the logs generated by the storage system in the target time period from its own running log database in the event of a storage system failure. In other embodiments, the storage system stores the logs generated by the storage system, and in the event of a storage system failure, the server sends a log acquisition request to the storage system to obtain the logs generated by the storage system in the target time period. In this way, the storage space of the server can be saved. The present application does not limit the manner in which the server obtains logs from the storage system.

[0091] 502. The server determines at least one fault time period in which a fault occurs from the target time period based on the logs generated by the storage system in the target time period, and the logs generated in the fault time period meet at least one of a quantity condition and an information volume condition.

[0092] In an embodiment of the present application, the server determines at least one fault time period in which a fault occurs from the target time period based on the number of logs and / or the amount of log information generated by the storage system within the target time period. It should be understood that since the occurrence of a fault is often accompanied by characteristics such as a large amount of log writing and a high amount of information, the potential fault time period can be determined from the target time period based on the number of logs and / or the amount of log information generated within the target time period. In this way, the amount of logs subsequently processed by the server can be reduced, thereby achieving massive log filtering.

[0093] In schematic form, this step includes any of the following implementations:

[0094] First, the server determines at least one failure time period from each sub-time period of the target time period based on the number of logs generated by the storage system in each sub-time period of the target time period.

[0095] Among them, the logs generated in the fault time period meet the quantity condition. Illustratively, the quantity condition is, for example, that the number of logs is greater than or equal to the first threshold (which can be set by itself), or that the number of logs is ranked in the top N (N is a positive integer), etc., and the present application is not limited to this. It should be understood that the target time period can be divided into multiple sub-time periods, and the present application does not limit the granularity of the division of sub-time periods. For example, a sub-time period corresponds to 1 minute, and for another example, a sub-time period corresponds to 1 second, and so on. The server detects the number of logs generated in each sub-time period within the target time period, determines the number of logs generated in each sub-time period within the target time period, and determines at least one fault time period that meets the aforementioned quantity condition from each sub-time period based on the number of logs generated in each sub-time period.

[0096] In some embodiments, the server determines at least one fault moment from each moment of the target time period based on the number of logs generated by the storage system at each moment within the target time period, and determines at least one fault time period based on each fault moment and the time length interval. Among them, the target time period can be divided into multiple moments, and the granularity of the moment division is smaller than the granularity of the aforementioned sub-time period division. The time length interval is used to expand the fault moment to obtain the fault time period. For example, the server determines that a certain fault moment is 10:10:10, and the time length interval is 3 seconds. The fault time period corresponding to the fault moment is 10:10:09 to 10:10:11. It should be understood that this is only an example and does not constitute a limitation of the present application. This process can also be understood as detecting the number of logs at each moment in the target time period, determining the potential fault moment, and taking the specified time length around the fault moment as the fault time period.

[0097] Through the first method mentioned above, the server determines the potential failure time period from the target time period based on the distribution of the number of logs generated in the target time period, effectively reducing the amount of logs that the server needs to process later and realizing massive log filtering.

[0098] Second, the server determines at least one fault time period from each sub-time period of the target time period based on the amount of log information generated by the storage system in each sub-time period of the target time period.

[0099] Among them, the logs generated in the fault time period meet the information volume condition. In the embodiment of the present application, the log information volume is reflected by the information entropy of the log. That is, the server determines the information entropy corresponding to each sub-time period based on the logs generated in each sub-time period in the target time period, and determines at least one fault time period from each sub-time period based on the information entropy corresponding to each sub-time period. Among them, the information entropy indicates the amount of information related to the root cause location of the fault in the log generated in the sub-time period. Schematically, the information volume condition is, for example, that the information entropy is greater than or equal to the second threshold (which can be set by itself), and for example, the information entropy is ranked in the first M places (M is a positive integer), etc., and the present application is not limited thereto. It should be understood that the occurrence of a fault is often accompanied by the characteristics of high information entropy, that is, the amount of log information generated in the fault time period is high. Therefore, by determining the information entropy corresponding to each sub-time period, the potential fault time period can be screened out from the target time period, thereby reducing the amount of logs processed by the server later. It should be noted that the present application does not limit the calculation method of information entropy. For example, it can be obtained by calculating the average value of the amount of information (such as the number of non-repeating characters in multiple logs), or by using different bases to perform logarithmic operations on the amount of information, etc. Of course, in other embodiments, the amount of log information generated in each sub-time period can also be reflected by the number of characters in the multiple logs generated in each sub-time period. For example, the more non-repeating characters in multiple logs, the higher the amount of information in these multiple logs.

[0100] In addition, similar to the first implementation, in some embodiments, the server determines at least one fault moment from each moment of the target time period based on the amount of log information generated by the storage system at each moment in the target time period, and determines at least one fault time period based on each fault moment and time length interval. The principle of this process is not repeated here.

[0101] It should be noted that the introduction of the target time period, sub-time period and time is referred to the first implementation method mentioned above, and will not be repeated hereafter.

[0102] Through the second method mentioned above, the server determines the potential failure time period from the target time period based on the distribution of the log information generated in the target time period, effectively reducing the amount of logs that the server needs to process later and realizing massive log filtering.

[0103] Third, the server determines at least one failure time period from each sub-time period of the target time period based on the number of logs and the amount of log information generated by the storage system in each sub-time period of the target time period.

[0104] The server determines the fault time period by combining the first and second implementation methods, which can further reduce the number of fault time periods and achieve large-scale streamlining of massive logs. The implementation process of this method is introduced below, including the following steps:

[0105] Step 1: Based on the number of logs generated in each sub-time period of the target time period, determine at least one candidate time period from each sub-time period of the target time period. The implementation of this step is the same as the first implementation method mentioned above. That is, based on the distribution of the number of logs generated in the target time period, the server first determines some candidate time periods from the target time period, and then executes the following step 2 to further screen these candidate time periods.

[0106] Step 2: Based on the logs generated in each candidate time period, determine the information entropy corresponding to each candidate time period, where the information entropy indicates the amount of information related to the root cause location of the fault in the logs generated in the candidate time period.

[0107] Step 3: Based on the information entropy corresponding to each candidate time period, determine at least one fault time period from at least one candidate time period.

[0108] Among them, the implementation of the above steps 2 and 3 is the same as the second implementation method. That is, after determining the candidate time periods, the server further screens these candidate time periods based on the distribution of the amount of log information generated in these candidate time periods, thereby comprehensively considering the characteristics of the number of logs and the amount of log information when the fault occurs, achieving a large-scale reduction in the number of logs, greatly reducing the amount of logs that the server needs to process later, and thus improving the efficiency of root cause location.

[0109] In addition, similar to the first implementation method mentioned above, in some embodiments, the server determines at least one candidate moment from each moment of the target time period based on the amount of log information generated by the storage system at each moment in the target time period, and determines at least one candidate time period based on each candidate moment and the time length interval; determines the information entropy corresponding to each candidate time period based on the logs generated in each candidate time period, and determines at least one fault time period from at least one candidate time period based on the information entropy corresponding to each candidate time period. It should be noted that this application does not limit the implementation method for the server to determine the fault time period. The server can first select the candidate moment based on the number of logs, and then select the fault time period based on the amount of information corresponding to the candidate time period where the candidate moment is located. It can also first select the candidate time period based on the number of logs, and then select the fault time period based on the amount of information corresponding to the candidate time period.

[0110] Schematically, the process of the third embodiment described above refers to Figure 6 , Figure 6 FIG. 1 is a schematic diagram of determining a fault time period provided by an embodiment of the present application. Figure 6 As shown, through the third method mentioned above, the server first determines some candidate time periods from the target time period based on the distribution of the number of logs generated in the target time period. In other words, by performing anomaly detection on the sequence obtained by performing log quantity statistics on the time dimension, the candidate potential fault time periods can be determined. Then, based on the distribution of the amount of log information generated in these candidate time periods, these candidate time periods are further screened, thereby comprehensively considering the characteristics of the number of logs and the amount of log information when the fault occurs, achieving large-scale streamlining of the number of logs, greatly reducing the amount of logs that the server needs to process subsequently, and thereby improving the efficiency of root cause location.

[0111] After the above step 502, the server preliminarily filters the logs generated by the storage system in the target time period according to the number of logs and / or the amount of log information generated in the target time period, thereby reducing the amount of logs to be subsequently processed by the server. Next, the server further filters the logs in the fault time period in combination with the fault that occurred in the storage system, and this process is introduced below through steps 503 and 504.

[0112] 503. The server determines a matching degree between the fault and each candidate log based on the description information of the fault and the log information of each candidate log generated within at least one fault time period.

[0113] In the embodiment of the present application, the description information of the fault refers to information used to describe the fault phenomenon. For example, taking the fault as a storage device downtime as an example, the description information of the fault includes: the service of storage device 1 is interrupted, it cannot be turned on, and indicator light A keeps flashing. It should be understood that this is only an example. In actual applications, the description information of the fault is generated according to the external phenomenon of the fault occurring in the storage system, and the present application does not limit this.

[0114] For any candidate log, its log information refers to the information used to describe the candidate log. In some embodiments, the log information includes the text content of the candidate log. That is, the server determines the matching degree between the fault and the candidate log based on the description information of the fault and the text content of the candidate log. In other embodiments, the log information includes the text content of the candidate log and at least one of the following: the system version number of the storage system, the log category to which the candidate log belongs, the generation time of the candidate log, and the position of the candidate log in the log package to which it belongs, etc. Among them, the system version number of the storage system refers to the system version number when the storage system fails. The log category is determined according to the configuration of the storage system. For example, the log category in the storage system includes: memory log, operating system log, hard disk read and write log, etc., and the present application is not limited thereto. The generation time of the candidate log is also the time when the log is written into the log package. The position of the candidate log in the log package to which it belongs is, for example, the XXth line in the log package to which it belongs, etc., and the present application is not limited thereto. That is, the server determines the matching degree between the fault and the candidate log based on the description information of the fault and various relevant information of the candidate log. The present application does not limit the way in which the server determines the matching degree between the fault and the candidate log.

[0115] In some embodiments, the server calls a neural network model to determine the degree of match between the fault and each candidate log. Schematically, taking any candidate log as an example, the server inputs the description information of the fault and the log information of the candidate log into the neural network model, and through the neural network model, the description information and the log information are respectively extracted to obtain a feature vector of the description information and a feature vector of the log information, and based on the feature vector of the description information and the feature vector of the log information, the server outputs the degree of match between the fault and the candidate log. The neural network model is trained based on the description information of historical faults and the log information of historical logs. For example, the neural network model is a convolutional neural network (CNN) model or a recurrent neural network (RNN) model, etc. The embodiments of the present invention do not limit the specific type of the above neural network model.

[0116] Taking a neural network model based on a transformer as an example, the neural network model includes a first transformer layer and a second transformer layer, wherein, taking the log information of a candidate log including the text content of the candidate log as an example, the first transformer layer is used to extract features from the text content of the candidate log to obtain a corresponding feature vector, and the second transformer layer is used to extract features from the description information to obtain a corresponding feature vector, and by fusing these two types of feature vectors, the matching degree between the fault and the candidate log is calculated. It should be understood that if the log information of the candidate log also includes other information, such as the system version number, log category, etc., then a corresponding encoding layer is deployed in the neural network model to extract features from these log information, and this application does not limit the structure and type of the neural network model.

[0117] It should be understood that the matching degree between the fault and the candidate log can reflect the correlation between the fault and the candidate log. The higher the matching degree, the higher the correlation between the fault and the candidate log, and the higher the probability that the candidate log is the root cause log of the fault, that is, the probability of determining the root cause of the fault from the candidate log is higher. In addition, the present application does not limit the form of the matching degree. For example, the matching degree can be a score, such as 80 points, or a percentage, such as 80%, and so on.

[0118] 504. The server filters the multiple candidate logs based on the matching degree between the fault and each candidate log to obtain at least one target log.

[0119] In an embodiment of the present application, the server sorts the matching degrees corresponding to multiple candidate logs based on the matching degree between the fault and each candidate log, and determines the candidate log corresponding to the matching degree ranked in the first number as the target log. The first number can be set according to actual needs, for example, the first number is 1000, and the present application is not limited thereto. The target log is also a key log that can be used to locate the root cause of the fault.

[0120] The above steps 503 and 504 mean that the server filters multiple candidate logs generated in at least one fault time period based on the matching degree between the fault and each candidate log generated in at least one fault time period to obtain at least one target log. Through this process, the fault phenomenon itself and business knowledge such as log information are fully considered, and the multiple candidate logs generated in the fault time period are further filtered, reducing the amount of logs that the server subsequently processes. Schematically, taking the server calling a neural network model to determine the matching degree as an example, this process can also be understood as a key log sorting process based on a neural network model. Reference Figure 7 , Figure 7 Schematic diagram of determining a target log provided by an embodiment of the present application. Figure 7As shown, the neural network model includes a first converter layer, a second converter layer, an encoding layer, a fusion layer, and a fully connected layer. Schematically, taking any candidate log as an example, by performing data preprocessing (such as tokenization, data transformations, categorical encoding, etc.) on the log information of the candidate log, the log features of the candidate log are obtained, such as text content features, system version number features, and log category features. Among them, the text content features are input into the first converter layer, and the semantic feature vector corresponding to the text content features is extracted through the first converter layer. The system version number features and the log category features are respectively input into the corresponding encoding layer to obtain the corresponding encoding vector, and the description information of the fault is preprocessed to obtain the description information features of the fault. The description information features are input into the second converter layer, and the semantic feature vector corresponding to the description information features is extracted through the second converter layer. Then, after the above-mentioned vectors are fused through the fusion layer, the fusion vector is input into the fully connected layer to calculate the matching degree. After obtaining the matching degree between the fault and each candidate log, these matching degrees are sorted to determine the key logs ranked in the first digit. It should be understood that by calling a transformer-based neural network model to determine the degree of matching, the generalization performance of the model is improved by leveraging the semantic extraction capability of the model.

[0121] After the above steps 503 and 504, the server further filters multiple candidate logs generated during the fault time period based on the fault phenomenon itself and business knowledge such as log information, thereby reducing the amount of logs subsequently processed by the server and improving the efficiency of root cause location.

[0122] 505. The server determines the root cause location reference information of the fault based on at least one target log.

[0123] In an embodiment of the present application, the root cause location reference information indicates at least one root cause location reference log of the fault, and / or indicates at least one reference root cause of the fault, which is not limited in the present application. That is, the root cause location reference information can be the log itself, which can reduce the manual analysis time and enhance the auxiliary analysis capability; the root cause location reference information can also be the reference root cause predicted by the server, which can achieve accurate root cause location and improve the root cause location efficiency. The following introduces these two situations respectively:

[0124] Case 1: The root cause location reference information indicates at least one root cause location reference log of the fault.

[0125] Schematically, this situation is applicable to the scenario of unknown root cause of fault, wherein, this step includes: based on the matching degree between the fault and each target log, sorting the matching degrees corresponding to multiple target logs, and determining the target log corresponding to the matching degree ranked in the first second number as the root cause location reference log. Among them, the second number can be set according to actual needs, for example, the second number is 10, and the present application is not limited to this. The root cause location reference log is also a reference log that assists manual root cause location of the fault, and the root cause of the fault can be accurately located from these root cause location reference logs. In combination with the above steps, it can be seen that the server can filter out a small number of root cause location reference logs from the massive logs generated by the storage system within the target time period. In this way, a large-scale reduction in the number of logs is achieved, which facilitates relevant personnel to quickly locate the root cause of the fault and improves the efficiency of root cause location.

[0126] Case 2: The root cause location reference information indicates at least one reference root cause of the fault.

[0127] Schematically, this situation is applicable to the scenario where the root cause of the fault is known. Among them, this step includes: based on multiple sample logs marked with the root cause of the fault and at least one target log, determine the hit number of the target log matched by each root cause of the fault; sort the hit numbers corresponding to multiple root causes of the fault, and determine the root cause of the fault corresponding to the hit number ranked in the first third number as the reference root cause. Among them, the third number can be set according to actual needs, for example, the third number is 3, and the present application is not limited to this. It should be understood that since the sample log is marked with the root cause of the fault, therefore, by matching the sample log under a certain root cause of the fault with the target log, if the match is successful, it indicates that the target log has a high probability of belonging to this root cause of the fault, for example, a keyword in the sample log is associated with the root cause of the fault, if a target log also includes this keyword, then the target log has a high probability of belonging to this root cause of the fault, so that the root cause of the fault is recommended to relevant personnel as a reference root cause, and accurate root cause positioning is achieved.

[0128] In some embodiments, for any root cause of a fault, the server determines the TF-IDF value corresponding to each target log based on the sample log marked with the root cause of the fault and each target log, and determines the target log with the TF-IDF value ranked in the top fourth digit as the target log matched by the root cause of the fault, wherein the fourth digit is also the aforementioned hit number. It should be understood that TF-IDF is a word frequency statistical algorithm. In this application, the higher the TF-IDF value, the more important the target log is in representing the root cause of the fault, wherein TF indicates the frequency of occurrence of the target log under the root cause of the fault, and IDF indicates the distribution of the target log under different root causes of the fault. After obtaining the hit number corresponding to each root cause of the fault, the root cause of the fault with the hit number ranked in the top third digit is determined as the reference root cause. For example, there are 5 known root causes of faults. Based on the sample logs corresponding to each root cause of faults and at least one target log, the hit counts of these 5 root causes of faults are determined to be 10, 15, 5, 20, and 3, respectively. Since the higher the hit count of a root cause of faults, the higher the probability that the root cause of the current fault is this root cause of faults, the top three root causes of faults with the highest hit counts can be taken as reference root causes. In this way, accurate root cause location is achieved and the efficiency of root cause location is improved.

[0129] Of course, the above two situations can also be combined. For example, the root cause positioning reference information includes both the target logs ranked in the top 10 in terms of matching degree and the reference root causes ranked in the top 3 in terms of hit quantity. This application does not limit this.

[0130] 506. The server sends the root cause location reference information of the fault to the terminal.

[0131] 507. The terminal displays reference information for locating the root cause of the fault.

[0132] In an embodiment of the present application, an application supporting root cause location of faults occurring in the storage system is run on the terminal. The terminal can display an application interface of the application, and display reference information of the root cause location of the fault on the application interface for relevant personnel to view.

[0133] Reference below Figure 8 , the method for locating the root cause of the storage system failure shown in the above steps 501 to 507 is illustrated by way of example. Figure 8 FIG. 1 is a schematic diagram of a method for locating the root cause of a storage system failure provided by an embodiment of the present application. Figure 8As shown, in the event of a storage system failure, the logs generated by the storage system within the target time period are obtained, that is, the corresponding log packages (for example, millions of lines) are obtained, and the corresponding logs are obtained by parsing the log packages. Next, the obtained massive logs are preliminarily filtered, that is, according to the number of logs and / or the amount of log information generated within the target time period, the potential failure time period is determined from the target time period (this process can also be understood as a log recall process based on business knowledge), and further filtering is performed based on the multiple candidate logs generated within the failure time period (for example, tens of thousands of lines), that is, combining the failure phenomenon itself with business knowledge such as log information, and further filtering the multiple candidate logs generated within the failure time period to obtain multiple target logs (also called key logs, for example, thousands of lines) for locating the root cause of the failure, and the aforementioned Figure 7 Taking the process shown as an example, this process can also be understood as a model-based log sorting process. After obtaining the target log, the reference information for locating the root cause of the fault is determined, wherein, for the known root cause of the fault, the reference root cause of the fault is determined based on multiple sample logs marked with the root cause of the fault and the target log (for example, the known root cause of the fault with the top 3 hits is taken); for the unknown root cause of the fault, the reference log for locating the root cause of the fault is determined according to the matching degree between the target log and the fault (for example, the target log with the top 10 matching degree is taken). It can be seen that through the above method, for the fault occurring in the storage system, the initial filtering is first performed according to the amount of log information and / or the number of logs, and then further filtered in combination with the matching degree between the fault phenomenon itself and the candidate log. In this way, through two layers of log filtering, the log is reduced from millions of lines to thousands of lines. Further, after obtaining the target log, a small number of reference logs are recommended for the unknown root cause of the fault, which can streamline invalid logs, reduce manual analysis time, and enhance auxiliary analysis capabilities; for the known root cause of the fault, the reference root cause is recommended, which can achieve accurate root cause positioning and improve the efficiency of root cause positioning.

[0134] In addition, based on the above Figure 1 As can be seen from the implementation environment shown, there are many configuration methods for the fault root cause location system provided by this application. Fig. 9 , based on several configuration modes of the fault root cause location system, the above storage system fault root cause location method is illustrated. Fig. 9 It is a schematic diagram of a fault root cause locating system provided in an embodiment of the present application.

[0135] like Fig. 9As shown in Figure (a), the fault root cause location system is configured on the cloud service platform of the storage system. The system is connected to the storage system in communication, and the logs generated by the storage system during operation can be uploaded to the cloud service platform. In the event of a storage system failure, the fault root cause location system can use the methods provided in steps 501 to 507 to perform unknown root cause fault analysis and / or known root cause fault analysis, thereby reducing the time consumption of locating the root cause of the fault and improving the location efficiency.

[0136] like Fig. 9 As shown in Figure (b), the fault root cause location system is synchronously configured in the storage device and data management engine of the storage system, and the storage device and the data management engine work together to achieve fault root cause location. For example, the storage device executes the above steps 501 to 504 to obtain the target log. In the event of a storage system failure, the storage device sends the target log to the data management engine, and the data management engine determines the root cause location reference information based on the target log. In this way, the storage device filters the logs internally and saves the fault-related logs. The log files are small, and the log storage period can be increased from weekly to grade level, for example, which reduces the storage space occupied by the logs themselves and greatly enhances the qualifiability; moreover, since the number of target logs is small, the amount of data transmission between the storage device and the data management engine is also small, thereby saving communication resources.

[0137] It should be understood that Fig. 9 The above is only an example and does not constitute a limitation of the present application. In actual applications, the functions implemented by each component of the fault root cause location system can be configured according to one's own needs.

[0138] In summary, the present application provides a storage system fault locating method, wherein, for a fault occurring in the storage system, based on at least one of the number of logs and the amount of log information generated by the storage system within a target time period, a potential fault time period in which the fault occurs is determined, thereby performing a preliminary filtering of the logs generated by the storage system during this time period, and further, combining the matching degree between the fault itself and the candidate logs generated within the fault time period, further filtering is performed on multiple candidate logs generated within the fault time period, and root cause locating reference information is determined based on the target logs obtained after filtering, thereby greatly reducing the number of logs used to locate the root cause of the fault, reducing the time consumption of locating the root cause of the fault, and improving the locating efficiency.

[0139] Fig.10 1 is a schematic diagram of the structure of a storage system fault location device provided in an embodiment of the present application. The storage system fault location device can be implemented as part or all of the functions of the aforementioned server through software, hardware, or a combination of both. Fig.10As shown, the device includes a first determining module 1001 , a filtering module 1002 and a second determining module 1003 .

[0140] The first determination module 1001 is used to determine, in the case of a storage system failure, at least one failure time period in which the failure occurs from the target time period based on the logs generated by the storage system in the target time period, and the logs generated in the failure time period meet at least one of a quantity condition and an information amount condition;

[0141] A filtering module 1002 is configured to filter multiple candidate logs generated in at least one fault time period based on a matching degree between the fault and each candidate log generated in at least one fault time period to obtain at least one target log;

[0142] The second determination module 1003 is used to determine the root cause location reference information of the fault based on at least one target log.

[0143] In some embodiments, the first determining module 1001 is used for any of the following:

[0144] Determine at least one fault time period from each sub-time period of the target time period based on the number of logs generated by the storage system in each sub-time period of the target time period;

[0145] Determine at least one fault time period from each sub-time period of the target time period based on the amount of log information generated by the storage system in each sub-time period of the target time period;

[0146] Based on the number of logs and the amount of log information generated by the storage system in each sub-time period of the target time period, at least one fault time period is determined from each sub-time period of the target time period.

[0147] In some embodiments, the first determining module 1001 is used to:

[0148] Determine at least one candidate time period from each sub-time period of the target time period based on the number of logs generated in each sub-time period of the target time period;

[0149] Based on the logs generated in each candidate time period, determine the information entropy corresponding to each candidate time period, where the information entropy indicates the amount of information related to the root cause location of the fault in the logs generated in the candidate time period;

[0150] At least one fault time period is determined from at least one candidate time period based on the information entropy corresponding to each candidate time period.

[0151] In some embodiments, the filtering module 1002 includes:

[0152] A first determination unit, configured to determine a matching degree between the fault and each candidate log based on description information of the fault and log information of each candidate log;

[0153] The second determining unit is used to sort the matching degrees corresponding to the multiple candidate logs based on the matching degree between the fault and each candidate log, and determine the candidate log corresponding to the matching degree ranked first as the target log.

[0154] In some embodiments, the first determining unit is configured to:

[0155] The description information and the log information of the candidate log are input into the neural network model. The neural network model is used to extract features of the description information and the log information respectively to obtain a feature vector of the description information and a feature vector of the log information. Based on the feature vector of the description information and the feature vector of the log information, the matching degree between the fault and the candidate log is output.

[0156] In some embodiments, the log information includes text content of the candidate log; or,

[0157] The log information includes the text content of the candidate log and at least one of the following: the system version number of the storage system, the log category to which the candidate log belongs, the generation time of the candidate log, and the position of the candidate log in the log package to which it belongs.

[0158] In some embodiments, the root cause location reference information indicates at least one root cause location reference log of the fault;

[0159] The second determining module 1003 is used to:

[0160] Based on the matching degree between the fault and each target log, the matching degrees corresponding to the multiple target logs are sorted, and the target log corresponding to the matching degree ranked second in the top is determined as the reference log for root cause location.

[0161] In some embodiments, the root cause location reference information indicates at least one reference root cause of the fault;

[0162] The second determining module 1003 is used to:

[0163] Based on multiple sample logs marked with fault root causes and at least one target log, determine the hit number of target logs matched by each fault root cause;

[0164] The hit numbers corresponding to the multiple root causes of faults are sorted, and the root cause of faults corresponding to the hit number ranked in the top third is determined as the reference root cause.

[0165] In some embodiments, the device is applied to a fault root cause location system, and the fault root cause location system is configured to perform at least one of the following:

[0166] Storage devices of storage systems;

[0167] Data management engine for storage systems;

[0168] Cloud service platform for storage systems.

[0169] Through the above-mentioned device, for a failure occurring in a storage system, based on at least one of the number of logs and the amount of log information generated by the storage system within a target time period, a potential failure time period in which the failure occurs is determined, thereby preliminarily filtering the logs generated by the storage system within this time period, and further, combining the matching degree between the failure itself and the candidate logs generated within the failure time period, further filtering the multiple candidate logs generated within the failure time period, and determining the root cause location reference information based on the target log obtained after filtering, thereby greatly reducing the number of logs used to locate the root cause of the failure, reducing the time consumption of locating the root cause of the failure, and improving the location efficiency.

[0170] In addition, in the above device, the first determination module 1001, the filtering module 1002 and the second determination module 1003 can be implemented by software or by hardware. Exemplarily, the implementation of the first determination module 1001 is described below by taking the first determination module 1001 as an example. Similarly, the implementation of the filtering module 1002, the second determination module 1003 and other modules can refer to the implementation of the first determination module 1001.

[0171] As an example of a software functional unit, the first determination module 1001 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the first determination module 1001 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0172] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0173] As an example of a hardware functional unit, the first determination module 1001 may include at least one computing device. Alternatively, the first determination module 1001 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0174] The multiple computing devices included in the first determination module 1001 can be distributed in the same region or in different regions. The multiple computing devices included in the first determination module 1001 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first determination module 1001 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0175] It should be noted that, in other embodiments, the first determination module 1001 can be used to execute any step in the storage system fault root cause location method, that is, the steps that the first determination module 1001, the filtering module 1002, and the second determination module 1003 are responsible for implementing can be specified as needed, and the first determination module 1001, the filtering module 1002, and the second determination module 1003 respectively implement different steps in the storage system fault root cause location method to achieve all functions based on the storage system fault root cause location device. In addition, the storage system fault root cause location device provided in the above embodiment and the storage system fault root cause location method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0176] In this application, the terms "first", "second", etc. are used to distinguish between identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is the quantity and execution order limited. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described examples, the first log can be referred to as the second log, and similarly, the second log can be referred to as the first log. Both the first log and the second log can be logs, and in some cases, can be separate and different logs.

[0177] In the present application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, a plurality of logs means two or more logs.

[0178] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0179] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of program structure information. The program structure information includes one or more program instructions. When the program instructions are loaded and executed on a computing device, all or part of the processes or functions in the embodiments of the present application are generated.

[0180] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0181] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for locating the root cause of a storage system failure, characterized in that: The method comprises: In the event of a storage system failure, based on logs generated by the storage system within a target time period, determining at least one failure time period in which the failure occurs from the target time period, wherein the logs generated within the failure time period meet at least one of a quantity condition and an information volume condition; Based on the matching degree between the fault and each candidate log generated in the at least one fault time period, filtering the plurality of candidate logs generated in the at least one fault time period to obtain at least one target log; Based on at least one of the target logs, reference information for locating the root cause of the fault is determined.

2. The method according to claim 1, characterized in that The determining, based on the logs generated by the storage system in the target time period, at least one fault time period in which the fault occurs from the target time period includes any of the following: Determining at least one fault time period from each sub-time period of the target time period based on the number of logs generated by the storage system in each sub-time period of the target time period; Determine the at least one fault time period from each sub-time period of the target time period based on the amount of log information generated by the storage system in each sub-time period of the target time period; The at least one fault time period is determined from each sub-time period of the target time period based on the number of logs and the amount of log information generated by the storage system in each sub-time period of the target time period.

3. The method according to claim 2, characterized in that The determining the at least one fault time period from each sub-time period of the target time period based on the number of logs and the amount of log information generated by the storage system in each sub-time period of the target time period includes: Determine at least one candidate time period from each sub-time period of the target time period based on the number of logs generated in each sub-time period of the target time period; Based on the logs generated in each of the candidate time periods, determining the information entropy corresponding to each of the candidate time periods, the information entropy indicating the amount of information related to the root cause location of the fault in the logs generated in the candidate time period; Based on the information entropy corresponding to each of the candidate time periods, the at least one fault time period is determined from the at least one candidate time period.

4. The method according to any one of claims 1 to 3, characterized in that The filtering of the plurality of candidate logs generated in the at least one fault time period based on the matching degree between the fault and each candidate log generated in the at least one fault time period to obtain at least one target log includes: Determine the matching degree between the fault and each of the candidate logs based on the description information of the fault and the log information of each of the candidate logs; Based on the matching degree between the fault and each of the candidate logs, the matching degrees corresponding to the plurality of candidate logs are sorted, and the candidate log corresponding to the matching degree ranked in the first digit is determined as the target log.

5. The method according to claim 4, characterized in that The determining, based on the description information of the fault and the log information of each of the candidate logs, a matching degree between the fault and each of the candidate logs includes: The description information and the log information of the candidate log are input into a neural network model, and feature extraction is performed on the description information and the log information respectively through the neural network model to obtain a feature vector of the description information and a feature vector of the log information, and based on the feature vector of the description information and the feature vector of the log information, a matching degree between the fault and the candidate log is output.

6. The method according to claim 4 or 5, characterized in that: The log information includes the text content of the candidate log; or, The log information includes the text content of the candidate log and at least one of the following: the system version number of the storage system, the log category to which the candidate log belongs, the generation time of the candidate log, and the position of the candidate log in the log package to which it belongs.

7. The method according to any one of claims 1 to 6, characterized in that The root cause location reference information indicates at least one root cause location reference log of the fault; The determining, based on at least one of the target logs, reference information for locating the root cause of the fault includes: Based on the matching degree between the fault and each of the target logs, the matching degrees corresponding to the multiple target logs are sorted, and the target log corresponding to the matching degree ranked second in the front is determined as the root cause location reference log.

8. The method according to any one of claims 1 to 6, characterized in that The root cause location reference information indicates at least one reference root cause of the fault; The determining, based on at least one of the target logs, reference information for locating the root cause of the fault includes: Based on a plurality of sample logs marked with fault root causes and at least one target log, determining a hit number of the target logs matched by each fault root cause; The hit numbers corresponding to the plurality of root causes of the fault are sorted, and the root cause of the fault corresponding to the hit number ranked in the first third order is determined as the reference root cause.

9. The method according to any one of claims 1 to 8, characterized in that The method is applied to a fault root cause location system, wherein the fault root cause location system is configured to: a storage device of the storage system; a data management engine of the storage system; The cloud service platform of the storage system.

10. A storage system fault root cause location device, characterized in that: The device comprises: A first determination module is used to determine, in the event of a storage system failure, based on logs generated by the storage system within a target time period, at least one failure time period in which the failure occurs from the target time period, and the logs generated within the failure time period meet at least one of a quantity condition and an information volume condition; A filtering module, configured to filter the plurality of candidate logs generated in the at least one fault time period based on a matching degree between the fault and each candidate log generated in the at least one fault time period, to obtain at least one target log; The second determination module is used to determine the root cause location reference information of the fault based on at least one of the target logs.

11. The device according to claim 10, characterized in that The first determining module is used for any of the following: Determining at least one fault time period from each sub-time period of the target time period based on the number of logs generated by the storage system in each sub-time period of the target time period; Determine the at least one fault time period from each sub-time period of the target time period based on the amount of log information generated by the storage system in each sub-time period of the target time period; The at least one fault time period is determined from each sub-time period of the target time period based on the number of logs and the amount of log information generated by the storage system in each sub-time period of the target time period.

12. The device according to claim 11, characterized in that The first determining module is used to: Determine at least one candidate time period from each sub-time period of the target time period based on the number of logs generated in each sub-time period of the target time period; Based on the logs generated in each of the candidate time periods, determining the information entropy corresponding to each of the candidate time periods, the information entropy indicating the amount of information related to the root cause location of the fault in the logs generated in the candidate time period; Based on the information entropy corresponding to each of the candidate time periods, the at least one fault time period is determined from the at least one candidate time period.

13. The device according to any one of claims 10 to 12, characterized in that The filtering module comprises: A first determining unit, configured to determine a matching degree between the fault and each of the candidate logs based on the description information of the fault and the log information of each of the candidate logs; The second determining unit is configured to sort the matching degrees corresponding to the plurality of candidate logs based on the matching degree between the fault and each of the candidate logs, and determine the candidate log corresponding to the matching degree ranked first as the target log.

14. The device according to claim 13, characterized in that The first determining unit is configured to: The description information and the log information of the candidate log are input into a neural network model, and feature extraction is performed on the description information and the log information respectively through the neural network model to obtain a feature vector of the description information and a feature vector of the log information, and based on the feature vector of the description information and the feature vector of the log information, a matching degree between the fault and the candidate log is output.

15. The device according to claim 13 or 14, characterized in that The log information includes the text content of the candidate log; or, The log information includes the text content of the candidate log and at least one of the following: the system version number of the storage system, the log category to which the candidate log belongs, the generation time of the candidate log, and the position of the candidate log in the log package to which it belongs.

16. The device according to any one of claims 10 to 15, characterized in that The root cause location reference information indicates at least one root cause location reference log of the fault; The second determining module is used to: Based on the matching degree between the fault and each of the target logs, the matching degrees corresponding to the multiple target logs are sorted, and the target log corresponding to the matching degree ranked second in the front is determined as the root cause location reference log.

17. The device according to any one of claims 10 to 15, characterized in that The root cause location reference information indicates at least one reference root cause of the fault; The second determining module is used to: Based on a plurality of sample logs marked with fault root causes and at least one target log, determining a hit number of the target logs matched by each fault root cause; The hit numbers corresponding to the plurality of root causes of the fault are sorted, and the root cause of the fault corresponding to the hit number ranked in the first third order is determined as the reference root cause.

18. The device according to any one of claims 10 to 17, characterized in that The device is applied to a fault root cause location system, and the fault root cause location system is configured in at least one of the following: a storage device of the storage system; a data management engine of the storage system; The cloud service platform of the storage system.

19. A computing device, characterized in that The computing device includes a processor and a memory, wherein the memory is used to store at least one section of program code, and the at least one section of program code is loaded by the processor and executes the storage system fault root cause locating method according to any one of claims 1 to 9.

20. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the storage system fault root cause locating method according to any one of claims 1 to 9.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store at least one section of program code, and the at least one section of program code is used to execute the storage system fault root cause locating method according to any one of claims 1 to 9.

22. A computer program product, characterized in that When the computer program product is executed on a computing device, the computing device is caused to execute the method for locating the root cause of a storage system failure according to any one of claims 1 to 9.

Images