A method and apparatus for locating server faults

Abstract

This invention provides a method and apparatus for locating server faults. The method includes: upon receiving a fault alarm message, determining the category corresponding to the server ID based on the server ID carried in the fault alarm message; determining the log collection protocol corresponding to the category, and obtaining hardware logs from the server corresponding to the server ID according to the log collection protocol; extracting the vector representation corresponding to the hardware logs; inputting the vector representation into a log parsing library model for calculation to obtain fault information; generating fault location information based on the fault information, and outputting the fault location information. Utilizing a log parsing library model to calculate server fault information improves the accuracy of fault location; automatically collecting server logs according to the server's corresponding log collection protocol improves hardware operation and maintenance efficiency and saves significant manpower and material costs.

Description

Technical Field

[0001] This invention relates to the field of server operation and maintenance technology, and in particular to a method and apparatus for locating server faults. Background Technology

[0002] Due to the rapid development of online business, the number of data center servers is constantly increasing. Simultaneously, as the service life of these servers gradually increases, the hardware failure rate also rises. To ensure the continuity of data center server operations, it is crucial to promptly detect and quickly locate the fault when a hardware failure occurs.

[0003] Currently, the main approach to hardware fault maintenance is as follows: when a component of the server fails, the fault log is recorded in the BMC (Brain Management Machine), and a trap message is sent to the monitoring server via the SNMP protocol. Based on the hardware monitoring rules, an alarm is generated and sent to maintenance personnel. The maintenance personnel then log into the server management interface to collect, parse, and locate the fault. However, this method relies heavily on the operations of maintenance personnel, requiring significant manpower and resources, resulting in low efficiency in fault location. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for locating server faults, in order to solve the problem that locating faults currently requires a large amount of manpower and resources.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect of this invention discloses a method for locating server faults, the method comprising:

[0007] When a fault alarm message is received, the category corresponding to the server ID is determined based on the server ID carried in the fault alarm message;

[0008] The log collection protocol is determined according to the category, and logs are obtained from the server corresponding to the server ID according to the log collection protocol, wherein the logs include at least hardware logs;

[0009] Extract the vector representation of the hardware log;

[0010] The vector representation is input into the log parsing library model for calculation to obtain fault information. The log parsing library model is trained based on multiple historical fault information.

[0011] Based on the fault information, fault location information is generated and output.

[0012] Preferably, when a fault alarm message is received, determining the category corresponding to the server ID based on the server ID carried in the fault alarm message includes:

[0013] When a fault alarm message is received, the server ID carried in the fault alarm message is obtained;

[0014] Obtain the topology information of the server corresponding to the server ID;

[0015] The category corresponding to the server ID is determined based on the topology information.

[0016] Preferably, the log parsing library model is trained based on multiple historical fault information, including:

[0017] Obtain information on multiple historical faults;

[0018] The information on the multiple historical faults was compiled into a sample dataset.

[0019] The sample dataset is input into a recurrent neural network model for training until the recurrent neural network model converges, thus obtaining the log parsing library model.

[0020] Preferably, the method further includes:

[0021] Calculate the similarity between the vector representation of the hardware log and the feature vector in the log parsing library model;

[0022] Determine whether the similarity is within a preset threshold range;

[0023] If the similarity is within a preset threshold range, a preset nearest neighbor algorithm is used to classify the hardware logs into the log parsing library model in order to optimize and update the log parsing library model.

[0024] Preferably, the log further includes component status information, and the fault information further includes the target component;

[0025] After calculating the fault information by inputting the vector representation into the log parsing library model, the method further includes:

[0026] Determine whether the target component is in a fault state in the component status information;

[0027] If the target component is in a fault state in the component status information, it is determined that the target component has malfunctioned.

[0028] A second aspect of this invention discloses a server fault location device, the device comprising:

[0029] The first determining unit is used to determine the category corresponding to the server ID based on the server ID carried in the fault alarm information when receiving the fault alarm information.

[0030] The first acquisition unit is used to determine the log collection protocol according to the category, and to acquire logs from the server corresponding to the server ID according to the log collection protocol, wherein the logs include at least hardware logs;

[0031] An extraction unit is used to extract a vector representation of the hardware log;

[0032] The first computing unit is used to input the vector representation into the log parsing library model for calculation to obtain fault information. The log parsing library model is trained based on multiple historical fault information.

[0033] The generation unit is used to generate fault location information based on the fault information and output the fault location information.

[0034] Preferably, the first determining unit includes:

[0035] The first acquisition module is used to acquire the server ID carried in the fault alarm information when a fault alarm information is received.

[0036] The second acquisition module is used to acquire the topology information of the server corresponding to the server ID;

[0037] The determination module is used to determine the category corresponding to the server ID based on the topology information.

[0038] Preferably, the device further includes:

[0039] The second acquisition unit is used to acquire multiple historical fault information;

[0040] The sorting unit is used to sort the multiple historical fault information into a sample dataset;

[0041] The input unit is used to input the sample dataset into the recurrent neural network model for training until the recurrent neural network model converges, thereby obtaining the log parsing library model.

[0042] Preferably, the device further includes:

[0043] The second computing unit is used to calculate the similarity between the vector representation of the hardware log and the feature vector in the log parsing library model;

[0044] The first judgment unit is used to determine whether the similarity is within a preset threshold range;

[0045] An optimization unit is used to classify the hardware logs into the log parsing library model using a preset nearest neighbor algorithm if the similarity is within a preset threshold range, so as to optimize and update the log parsing library model.

[0046] Preferably, the log further includes component status information, and the fault information further includes the target component;

[0047] The device further includes:

[0048] The second judgment unit is used to determine whether the target component is in a fault state in the component status information;

[0049] The second determining unit is used to determine that the target component has malfunctioned if the target component is in a fault state in the component status information.

[0050] Based on the above embodiments of the present invention, a method and apparatus for locating server faults are provided. The method includes: when a fault alarm message is received, determining the category corresponding to the server ID based on the server ID carried in the fault alarm message; determining the log collection protocol corresponding to the category, and obtaining hardware logs from the server corresponding to the server ID according to the log collection protocol; extracting the vector representation corresponding to the hardware logs; inputting the vector representation into a log parsing library model for calculation to obtain fault information; generating fault location information based on the fault information, and outputting the fault location information. Utilizing a log parsing library model to calculate server fault information improves the accuracy of fault location; automatically collecting server logs according to the server's corresponding log collection protocol improves hardware operation and maintenance efficiency and saves significant manpower and material costs. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 A flowchart illustrating a server fault location method provided in an embodiment of the present invention;

[0053] Figure 2 A structural block diagram of a server fault location device provided in an embodiment of the present invention;

[0054] Figure 3 Another structural block diagram of a server fault location device provided in an embodiment of the present invention;

[0055] Figure 4This is another structural block diagram of a server fault location device provided in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0058] As the background technology shows, when a server fails, it generates corresponding alarm information to remind the operation and maintenance personnel of the server failure. The operation and maintenance personnel need to collect the server's fault logs and analyze the logs to locate the fault location. However, this method of hardware failure operation and maintenance relies too much on the operation and maintenance personnel's operation and maintenance, which requires a lot of manpower and resources, resulting in low efficiency in locating faults.

[0059] Therefore, this invention provides a method and apparatus for locating server faults. When a fault alarm message carrying a server ID is received, the method determines the category corresponding to the server ID; retrieves hardware logs from the server corresponding to the server ID according to the log collection protocol corresponding to the category; extracts the vector representation corresponding to the hardware logs; inputs the vector representation into a log parsing library model for similarity calculation to obtain fault information; generates fault location information based on the fault information and outputs the fault location information. Automatically collecting server logs according to the server's corresponding log collection protocol improves hardware operation and maintenance efficiency, and using a log parsing library model to calculate server fault information improves the accuracy of fault location.

[0060] join Figure 1 The flowchart illustrates a server fault location method provided by an embodiment of the present invention, the method comprising:

[0061] It should be noted that this server fault location method is applicable to most types of x86 servers and has good compatibility.

[0062] Step S101: When a fault alarm message is received, determine the category corresponding to the server ID based on the server ID carried in the fault alarm message.

[0063] It should be noted that when a server malfunctions, it generates hardware logs (such as the SEL system event log), which are stored in the Baseboard Manager Controller (BMC). The server sends trap messages to the monitoring server via the SNMP protocol, causing the monitoring server to generate fault alarm information. The trap message is a type of PDU message in the SNMP protocol used to describe the occurrence of a server event; the fault alarm information includes the sequence number, the ID of the faulty server, and alarm abbreviation information, etc.

[0064] In the specific implementation of step S101, when a fault alarm message is received from the monitoring server, the type of the faulty server corresponding to the server ID is determined based on the server ID carried in the fault alarm message.

[0065] It's understandable that the server ID is the ID of the server that failed. Determining the category of the failed server corresponding to the server ID involves: obtaining the server's topology information based on the server ID; and then determining the category corresponding to the server ID, i.e., the category of the failed server, based on the topology information.

[0066] It should be noted that servers are classified into multiple categories according to server manufacturers (such as Huawei, H3C, Inspur, IBM, HP, Sugon, etc.), and each category corresponds to a log collection protocol (such as IPMI, Redfish, SSH, etc.).

[0067] Step S102: Determine the log collection protocol according to the category, and obtain logs from the server corresponding to the server ID according to the log collection protocol.

[0068] In the specific implementation step S102, the log collection protocol corresponding to the server is determined according to the server category corresponding to the server ID, and the server logs are obtained from the server corresponding to the server ID according to the log collection protocol. The logs include at least hardware logs (such as SEL system event logs).

[0069] In some specific embodiments, the logs collected according to the log collection protocol also include component status information, which is the status information of each component in the server recorded by the baseboard management controller. As shown in Table 1, the status information value is OK or Down.

[0070] Table 1

[0071] feature Characteristic significance SERIAL_NUM Server serial number BMC_IP The server corresponds to the BMC's IP address. ALARM_LEVEL Log levels DATETIME Log generation time LOG_DETAIL Log details

[0072] Step S103: Extract the vector representation of the hardware log.

[0073] In the specific implementation step S103, the hardware log is processed into a text vectorization using a preset word vector model (such as the word2vec model) to obtain a vector representation of the hardware log.

[0074] Step S104: Input the vector representation into the log parsing library model for calculation to obtain fault information.

[0075] In the specific implementation step S104, the vector representation of the hardware log is input into the log parsing library model for calculation to obtain the fault information corresponding to the hardware log, wherein the fault information includes the target component and the cause of the fault.

[0076] It is understandable that if the log parsing library model analyzes the hardware logs to obtain fault information, then the target component is the component that has failed as analyzed by the log parsing library model.

[0077] It should be noted that the log parsing library model is pre-trained based on multiple historical fault information, and the specific implementation process is as follows:

[0078] Multiple historical fault information was obtained. These historical fault information consisted of log-fault datasets from servers of multiple vendors, as shown in Table 2. The text content in these historical fault information was vectorized and organized into a sample dataset. The sample indicators were divided into sample features and sample labels.

[0079] Table 2

[0080] Sample Indicators Indicator significance MANUFACTURER Sample characteristics, server manufacturers LOG_DETAIL Sample characteristics, log details ALARM_LEVEL Sample features, log level FAILURE_CAUSE Sample label, cause of failure FAILURE_COMPONENTS Sample label, faulty component COMPONENTS_LOCATION Sample label, location of faulty component

[0081] The sample dataset is input into the Recurrent Neural Network (RNN) model for training until the RNN model converges, resulting in the log parsing library model.

[0082] Understandably, the log parsing library model includes knowledge and experience such as the mapping relationship between log details, log level, fault cause, and fault location.

[0083] In some specific embodiments, after obtaining fault information from the hardware log, the log parsing library model determines whether the target component in the fault information is in a fault state in the component status information; if the target component is in a fault state in the component status information (e.g., the status information value is Down), it is determined that the target component has a fault; if the target component is in a normal state in the component status information (e.g., the status information value is OK), it indicates that the target component may have a hidden fault.

[0084] It should be noted that in the log parsing library model, the weights of hardware logs and component status information can be adjusted according to the actual application situation, thereby obtaining more accurate fault information.

[0085] Step S105: Generate fault location information based on the fault information and output the fault location information.

[0086] In the specific implementation of step S105, fault location information is generated based on the fault information. The fault location information includes at least the cause of the fault and the location of the faulty component, and the fault location information is output.

[0087] In some embodiments, the log parsing library model is optimized and updated through incremental learning. The specific optimization process is as follows: calculate the similarity between the vector representation of the hardware log and the feature vector in the log parsing library model; determine whether the similarity is within a preset threshold range; if the similarity is within the preset threshold range, use a preset nearest neighbor algorithm (such as KNN algorithm) to classify the hardware log into the log parsing library model to optimize the log parsing library model; if the similarity is outside the preset threshold range, feed back the judgment result to the business personnel so that they can provide the sample labels corresponding to the hardware log, thereby optimizing the log parsing library model.

[0088] It is understandable that the similarity between the vector representation of the computational hardware log and the feature vector in the log parsing library model is the cosine distance.

[0089] In this embodiment of the invention, a recurrent neural network model is trained using multiple historical fault information to obtain a log parsing library model, and the log parsing library model is optimized in combination with practical applications to improve the accuracy of fault location. When a fault alarm information is received, the server logs are collected, analyzed, and fault information is calculated, which improves the efficiency of server hardware operation and maintenance, saves manpower and material costs, and provides a more stable operating environment for server applications.

[0090] Corresponding to the server fault location method provided in the above embodiments of the present invention, see also... Figure 2 The diagram shows a structural block diagram of a server fault location device provided in an embodiment of the present invention. The location device includes: a first determining unit 201, a first acquiring unit 202, an extracting unit 203, a first calculating unit 204, and a generating unit 205.

[0091] The first determining unit 201 is used to determine the category corresponding to the server ID based on the server ID carried in the fault alarm information when receiving the fault alarm information.

[0092] The first acquisition unit 202 is used to determine the log collection protocol according to the category, and to obtain logs from the server corresponding to the server ID according to the log collection protocol. The logs include at least hardware logs.

[0093] Extraction unit 203 is used to extract the vector representation of the hardware log.

[0094] The first computing unit 204 is used to input the vector representation into the log parsing library model for calculation to obtain fault information. The log parsing library model is trained based on multiple historical fault information.

[0095] The generation unit 205 is used to generate fault location information based on the fault information and output the fault location information.

[0096] In this embodiment of the invention, when a fault alarm message is received, the log collection protocol is determined by the server category, the server's logs are obtained according to the log collection protocol, the server's fault information is calculated by combining the log parsing library model, fault location information is generated based on the fault information, and the fault location information is output, thereby realizing automated fault location, improving the efficiency of server operation and maintenance, and saving human and material costs.

[0097] Preferred, combined Figure 2 See the content shown. Figure 3 This diagram illustrates another structural block diagram of a server fault location device provided in an embodiment of the present invention. The first determining unit 201 includes: a first acquiring module 2011, a second acquiring module 2012, and a determining module 2013. The execution principle of each module is as follows:

[0098] The first acquisition module 2011 is used to acquire the server ID carried in the fault alarm information when a fault alarm information is received.

[0099] The second acquisition module 2012 is used to acquire the topology information of the server corresponding to the server ID.

[0100] The determination module 2013 is used to determine the category corresponding to the server ID based on the topology information.

[0101] Preferred, combined Figure 2 See the content shown. Figure 4 The diagram shows another structural block diagram of a server fault location device provided in an embodiment of the present invention. The location device further includes: a second acquisition unit 206, a sorting unit 207, and an input unit 208.

[0102] The second acquisition unit 206 is used to acquire multiple historical fault information.

[0103] The sorting unit 207 is used to sort multiple historical fault information into a sample dataset.

[0104] Input unit 208 is used to input the sample dataset into the recurrent neural network model for training until the recurrent neural network model converges, thus obtaining the log parsing library model.

[0105] Preferred, combined Figure 2 As shown, the positioning device also includes: a second calculation unit, a first judgment unit, and an optimization unit.

[0106] The second computational unit is used to calculate the similarity between the vector representation of the hardware log and the feature vector in the log parsing library model.

[0107] The first judgment unit is used to determine whether the similarity is within the preset threshold range.

[0108] The optimization unit is used to classify hardware logs into the log parsing library model using a preset nearest neighbor algorithm if the similarity is within a preset threshold range, so as to optimize and update the log parsing library model.

[0109] Preferred, combined Figure 2 As shown, the positioning device also includes a second judgment unit and a second determination unit.

[0110] The second judgment unit is used to determine whether the target component is in a fault state in the component status information.

[0111] The second determining unit is used to determine that the target component has failed if the target component is in a fault state in the component status information.

[0112] In summary, the embodiments of the present invention provide a method and apparatus for locating server faults. By training a recurrent neural network model with historical fault information, a log parsing library model is obtained. The log parsing library model is used to calculate the vectorized representation of the server's logs to obtain fault information, thereby improving the accuracy of server fault location and providing a more stable operating environment for applications on the server.

[0113] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0114] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0115] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for locating server faults, characterized in that, The method includes: When a fault alarm message is received, the category corresponding to the server ID is determined based on the server ID carried in the fault alarm message; The log collection protocol is determined according to the category, and logs are obtained from the server corresponding to the server ID according to the log collection protocol. The logs include at least hardware logs and component status information. The hardware logs are processed into text vectors using a preset word vector model, and the vector representation of the hardware logs is extracted. The vector representation is input into the log parsing library model for calculation to obtain fault information. The log parsing library model is trained based on multiple historical fault information. The fault information includes the target component and the cause of the fault. Determine whether the target component is in a fault state in the component status information; If the target component is in a fault state in the component status information, it is determined that the target component has a fault; if the target component is in a normal state in the component status information, it is determined that the target component has a latent fault. Based on the fault information, generate fault location information and output the fault location information; When a fault alarm message is received, determining the category corresponding to the server ID based on the server ID carried in the fault alarm message includes: When a fault alarm message is received, the server ID carried in the fault alarm message is obtained; Obtain the topology information of the server corresponding to the server ID; The category corresponding to the server ID is determined based on the topology information.

2. The method according to claim 1, characterized in that, A log parsing library model was trained based on multiple historical fault information, including: Obtain information on multiple historical faults; The information on the multiple historical faults was compiled into a sample dataset. The sample dataset is input into a recurrent neural network model for training until the recurrent neural network model converges, thus obtaining the log parsing library model.

3. The method according to claim 1, characterized in that, The method further includes: Calculate the similarity between the vector representation of the hardware log and the feature vector in the log parsing library model; Determine whether the similarity is within a preset threshold range; If the similarity is within a preset threshold range, a preset nearest neighbor algorithm is used to classify the hardware logs into the log parsing library model to optimize and update the log parsing library model.

4. A server fault location device, characterized in that, The device includes: The first determining unit is used to determine the category corresponding to the server ID based on the server ID carried in the fault alarm information when receiving the fault alarm information. The first acquisition unit is used to determine the log collection protocol according to the category, and to acquire logs from the server corresponding to the server ID according to the log collection protocol. The logs include at least hardware logs and component status information. The extraction unit is used to perform text vectorization processing on the hardware log using a preset word vector model, and extract the vector representation of the hardware log; The first calculation unit is used to input the vector representation into the log parsing library model for calculation to obtain fault information. The log parsing library model is trained based on multiple historical fault information. The fault information includes the target component and the cause of the fault. The second judgment unit is used to determine whether the target component is in a fault state in the component status information; The second determining unit is configured to determine that the target component has a fault if the target component is in a fault state in the component status information; and to determine that the target component has a latent fault if the target component is in a normal state in the component status information. A generation unit is used to generate fault location information based on the fault information and output the fault location information; The first determining unit includes: The first acquisition module is used to acquire the server ID carried in the fault alarm information when a fault alarm information is received. The second acquisition module is used to acquire the topology information of the server corresponding to the server ID; The determination module is used to determine the category corresponding to the server ID based on the topology information.

5. The apparatus according to claim 4, characterized in that, The device further includes: The second acquisition unit is used to acquire multiple historical fault information; The sorting unit is used to sort the multiple historical fault information into a sample dataset; The input unit is used to input the sample dataset into the recurrent neural network model for training until the recurrent neural network model converges, thereby obtaining the log parsing library model.

6. The apparatus according to claim 4, characterized in that, The device further includes: The second computing unit is used to calculate the similarity between the vector representation of the hardware log and the feature vector in the log parsing library model; The first judgment unit is used to determine whether the similarity is within a preset threshold range; An optimization unit is used to classify the hardware logs into the log parsing library model using a preset nearest neighbor algorithm if the similarity is within a preset threshold range, so as to optimize and update the log parsing library model.

Images