Log data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a log data processing method and device, electronic equipment and a storage medium, and the method can comprise the following steps: determining information of a specified category from an obtained equipment log; pre-processing the information of the specified category to obtain a pre-processing result; and converting the preprocessing result into an embedded vector, wherein the embedded vector is used for performing similarity comparison with a predetermined equipment downtime log. According to the embodiment of the invention, the information of the specified category related to fault recognition can be extracted from the log, and interference information can be eliminated by preprocessing the information of the specified category. And finally, converting a preprocessing result into an embedded vector which can be compared. In addition, manual intervention is not needed in the whole process, and the equipment operation and maintenance cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for processing log data. Background Art

[0002] As the scale of cloud computing gradually expands, the failure rate of equipment during operation also increases. Therefore, daily operation and maintenance of equipment is particularly important. In daily operation and maintenance, the analysis of the cause of equipment failure, the location of the failure, and the repair of the failure require professional operation and maintenance personnel to invest a lot of time and energy based on log data to complete. In addition, even if the operation and maintenance personnel are in place, how to make good use of log data to complete daily operation and maintenance such as accurate analysis of the cause of the failure and accurate location of the failure is still very difficult. Summary of the invention

[0003] The embodiment of the present application provides a method, device, electronic device and storage medium for processing log data, which can remove interference information by processing the logs, and finally complete the daily operation and maintenance of the equipment by using the preprocessing results.

[0004] In a first aspect, an embodiment of the present application provides a method for processing log data, which may include:

[0005] Determine information of a specified category from the acquired device logs;

[0006] Preprocess the information of the specified category to obtain a preprocessing result;

[0007] The preprocessing results are converted into embedding vectors, which are used to perform similarity comparison with predetermined equipment downtime logs.

[0008] In a second aspect, an embodiment of the present application provides a log data processing device, which may include:

[0009] An information acquisition module, used to determine information of a specified category from the acquired device logs;

[0010] A preprocessing module is used to preprocess the information of a specified category to obtain a preprocessing result;

[0011] The conversion module is used to convert the preprocessing result into an embedding vector, and the embedding vector is used to compare the similarity with the predetermined device downtime log.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0014] Compared with the prior art, this application has the following advantages:

[0015] According to the embodiments of the present application, information of a specified category related to equipment operation and maintenance such as fault identification can be extracted from the log, and interference information can be eliminated by preprocessing the information of the specified category. The preprocessing results are finally used to complete the daily operation and maintenance of the equipment. The entire process does not require manual intervention, reducing the cost of equipment operation and maintenance. In the fault identification scenario, the efficiency and accuracy of fault identification can be improved.

[0016] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0018] Figure 1 A flowchart of a method for processing log data according to an embodiment of the present application;

[0019] Figure 2 is a structural block diagram of a log data processing device according to an embodiment of the present application; and

[0020] Figure 3 A block diagram of an electronic device used to implement an embodiment of the present application. DETAILED DESCRIPTION

[0021] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0022] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0023] First, the terms involved in one or more embodiments of the present application are explained.

[0024] Downtime: Downtime refers to the phenomenon that the operating system cannot recover from a serious system error, or a serious problem occurs at the system hardware level, causing the operating system to be unresponsive for a long time and the computer has to be restarted. Downtime will cause service interruption.

[0025] Crash call stack: The function call stack at the time of crash, which can be printed through the dmesg command.

[0026] Crash information: Kernel log generated after a crash occurs.

[0027] Taking a downtime failure as an example, this application includes the following methods for fault identification:

[0028] One of them is to obtain logs related to downtime and identify faults based on the vector space model (VSM). This method selects keywords from the logs and recombines the selected keywords using a specified expression. The semantics contained in the combined sentences are highly similar to the original sentences in the logs. The vector space model is a text processing method based on keywords. The recombination is based on word frequency information. The similarity between two sentences (texts) depends on the frequency of occurrence of common words. Using the above method, words are independent of each other, so it is easy to ignore the semantics of the context.

[0029] Another method is to use the term frequency-inverse document frequency (TF-IDF) method for fault identification. This method is a weight distribution technology that reduces the weight of keywords that appear more frequently but have less semantics in different documents. Combining this method with the method of fault identification based on the vector space model can effectively increase the weight of keywords. On the other hand, the above combination only increases the importance of keywords, without considering the impact of the location of keywords on the text, and it is easy to ignore the semantics between different keywords.

[0030] Another method is to extract information of a specified category from the log. The information of the specified category can be information related to possible faults that will be used for fault identification. For example, the information of the specified category can be information corresponding to the exception call stack, information corresponding to the work queue, information corresponding to the loading module, information corresponding to the command, information corresponding to the error content, etc. In order to improve the accuracy, the extracted information of the specified category can be preprocessed. Exemplarily, the preprocessing can include filtering unreliable information during the information extraction process; and after the extraction, deleting or rewriting the address information of the information of the specified category. The purpose of preprocessing is to improve the effectiveness of the information of the specified category. The preprocessed results can be processed using a sentence transformation model (Sentence Transformers). The sentence transformation model can convert a sentence into an embedding vector of a specified dimension.

[0031] The embedding vector is a dense vector of floating point values. The vector length parameter can be set manually and is trainable. The embedding vector can map two incomparable words or two incomparable sentences into a computable vector. The sentence conversion model is obtained based on deep learning training, which enables the sentence conversion model to extract semantic features between words or sentences, thereby generating accurate insights and predictions. Thus, the log processing process is completed. Based on the embedding vector, fault identification and patch selection can be realized. The identification process can include comparing the similarity of the embedding vector with the known fault log vector, and locating the fault according to the comparison result. The above method is used to complete the identification of the fault, and the whole process is automated, which can reduce the operation and maintenance cost of the equipment. In addition, by using the sentence conversion model to convert the information of the specified category, the potential association between the information of the specified category can be identified, making the fault identification result more accurate. For the fault identification result, it can be converted into text to inform the operation and maintenance personnel, or it can be matched with the fault patch to select the patch corresponding to the fault identification result. The selected patch can be used to complete the repair of the fault.

[0032] The present application embodiment provides a method for processing log data, such as Figure 1 The flowchart shown is a process of processing log data according to an embodiment of the present application, which may include:

[0033] Step S101: determining information of a specified category from the acquired device log.

[0034] In the current implementation, the Linux operating system is used as an example for explanation. The system kernel is the most basic part of the operating system. When an error occurs in the operation of the system kernel, it may cause the operating system to crash. Correspondingly, when an error occurs in the operation of the kernel, a corresponding fault report will be generated. The fault report is recorded in the kernel log. For the kernel log, it is collectively referred to as a log in the following text.

[0035] Based on this, the device logs can be processed and used as the basis for fault identification. The device logs can be obtained in real time, periodically, or when a fault occurs.

[0036] The specified category of information may refer to information related to the operating system crash. Exemplarily, the specified category of information may be information corresponding to the exception call stack, information corresponding to the pointer instruction register (RIP), information corresponding to the work queue, information corresponding to the load module, information corresponding to the kernel version, information corresponding to the command, information corresponding to the error content, etc.

[0037] There are many ways to extract information of a specified category. For example, regular expressions can be used for extraction. Regular expressions can include keyword extraction, format extraction, etc. Keyword extraction is used as an example for explanation. The error content in the log usually starts with the field "BUG:". Therefore, keyword matching can be performed in the log, and the content corresponding to the field "BUG:" is extracted as the error content. Similarly, "Call Trace" can also be used as a keyword for the information corresponding to the exception call stack to indicate that the content corresponding to the keyword is extracted as the information corresponding to the exception call stack.

[0038] Let's take the format extraction method as an example. The information corresponding to the kernel version is usually composed of numbers with a specified character length, a specific letter combination (such as "rc", "bm", "smp", etc.), and special characters such as "." and "-". Based on this, this can be used as a regular expression to extract the information corresponding to the kernel version in the log.

[0039] Step S102: preprocessing the information of the specified category to obtain a preprocessing result.

[0040] After extracting information of a specified category from the log, the information of the specified category can be further preprocessed. The purpose of preprocessing is to delete invalid information and improve the effectiveness of information of the specified category. For example, in a patch matching scenario or a fault identification scenario, it is necessary to match information of a specified category with the description information of the patch or fault, so as to determine the fault and select a suitable patch to repair the fault. Generally speaking, the description information of the patch contains the function name corresponding to the call stack function, but does not contain address information. Based on this, address information can be considered invalid information or interference information in the patch matching scenario. Information of a specified category that contains address information can be selected as the preprocessing object.

[0041] Taking the preprocessing as a deletion process as an example, if the information of the specified category contains address information, the address information can be deleted. Here, the address information can be presented as an initial address or an offset. The offset refers to the offset relative to the starting address of the calling function. When the calling function is loaded into a certain place in the memory for execution, the starting address will exist. The code of the calling function usually has multiple lines, and the binary code corresponding to each line occupies memory. When a line of code of the calling function is executed, the address at this time is the starting address + offset.

[0042] In addition, taking the rewriting process as an example of preprocessing, the address can be replaced by the function name. For the patch matching scenario, the description information of the patch contains the function name corresponding to the call stack function, but does not contain the address information. Therefore, the address information can be rewritten by replacing it with the function name.

[0043] That is to say, if the corresponding scenario is to identify the fault only by the call stack information, the function name and offset (offset is the address information) corresponding to the call stack function can be referred to. However, if the corresponding patch matching scenario, the offset will become interference information instead, and the offset needs to be deleted or rewritten.

[0044] Step S103: converting the preprocessing result into an embedding vector, and the embedding vector is used for similarity comparison with a predetermined device downtime log.

[0045] For the preprocessing results, the preprocessing results can be converted using a pre-trained sentence conversion model to obtain an embedding vector of a specified dimension. The sentence conversion model is obtained based on deep learning training, which can enable the sentence conversion model to extract semantic features between keywords or sentences.

[0046] Based on this, the similarity comparison between the embedded vector and the known fault log can be performed to achieve fault identification. The above method can be used to complete the fault identification, which can make the fault identification process faster. In addition, by using the sentence conversion model to convert the information of the specified category, the potential association between the information of the specified category can be identified, making the identification result more accurate. For the fault identification result, it can be converted into text to inform the operation and maintenance personnel, or it can be matched with the fault patch to select the patch corresponding to the fault identification result. The selected patch can be used to repair the fault.

[0047] Through the above process, the specified category of information related to fault identification can be extracted from the log, and the interference information can be removed by preprocessing the specified category of information. Finally, the log is converted into an embedded vector that can be compared for similarity, so as to be used for fault identification.

[0048] In one implementation, determining the information of a specified category from the acquired device log involved in step S101 may include:

[0049] Step S1011: extracting multiple categories of information from the device log using a preset regular expression.

[0050] The multiple categories of information may include information corresponding to the exception call stack, information corresponding to the pointer instruction register, information corresponding to the work queue, information corresponding to the load module, information corresponding to the kernel version, information corresponding to the command, information corresponding to the error content, etc. Thus, corresponding regular expressions can be pre-set according to the characteristics of different types of information.

[0051] Exemplarily, for the information corresponding to the exception call stack, a regular expression can be constructed based on the keyword "Call Trace:". For the information corresponding to the work queue, a regular expression can be constructed based on the keyword "Workqueue:". For the information corresponding to the pointer instruction register, a regular expression can be constructed based on the keyword "RIP:". For the information corresponding to the load module, a regular expression can be constructed based on rule matching, for example, the commonly used module names are stored. Thus, the information of the load module can be determined by searching in the call stack or the work queue according to the pre-stored commonly used module names. For the information corresponding to the kernel version, a regular expression can be constructed according to the format. For example, the format version can be set to ABC. Among them, A can represent a field composed of numbers and separators ".", such as "6.0.0" or "5.1.2". B can represent a field composed of numbers, separators "." and / or letters, such as "47", "rc2", "rc2.bm.1", etc. C can represent a field composed of numbers and / or letters, such as "amd64", "generic", etc. For information corresponding to commands, a regular expression can be constructed based on rule matching, for example, commonly used commands are stored. Thus, command information can be determined by searching the logs. For information corresponding to error content, a regular expression can be constructed based on the keyword "BUG:".

[0052] Step S1012: Reliability identification is performed on each category of information. When the identification result contains unreliable information, the unreliable information is cleaned to obtain information of a specified category; the unreliable information includes information with a specified mark.

[0053] After extracting multiple categories of information, the reliability of each category of information can also be identified. For example, the information of the exception call stack usually includes the function name and offset of the call stack function that begins with a question mark. For example, "?__switch_to+0x106 / 0x420" or "?process_one_work+0x380 / 0x380", etc. That is to say, in the current implementation, the function name and offset of the call stack function that begins with a question mark can be used as unreliable information with a specified mark.

[0054] In this regard, the call function that begins with a question mark can be identified as unreliable information. The unreliable information can be cleaned by deleting it. The information retained after cleaning can be used as information of the specified category.

[0055] In one implementation, when the information of the specified category includes error content, it may also include:

[0056] A query is performed in a pre-built fault classification table to obtain a fault type that matches the error content; the fault classification table at least includes the fault type and description information corresponding to the fault type, and the description information is used to compare with the error content; the fault type is used as information of a specified category.

[0057] The fault classification table can be constructed using historical data. Taking the Linux operating system crash as an example, multiple fault types can be pre-constructed in the fault classification table, such as the fault type of being unable to process kernel page fault requests, the fault type of hardware abnormalities causing crashes, the fault type of CPU abnormalities (such as soft and hard deadlocks), the fault type of accessing unallocated memory, the fault type of kernel null pointer dereference, the fault type of system request triggering an exception causing a crash, the fault type of division by zero error, and other (BUG_ON) fault types.

[0058] Corresponding to each fault type, there may also be description information in the fault classification table. For example, the fault type of being unable to handle kernel paging request may have a description information corresponding to "Unable to handle kernel paging request-at XX", which may be abbreviated as PageErr. XX may be used to represent an offset.

[0059] The fault type caused by hardware abnormality may have a description of "Hardware error fromXXX" or abbreviated as HardErr. XXX may be used to represent the hardware name.

[0060] The fault type of CPU abnormality (such as soft or hard lock) may have a corresponding description information of "Kernel panic not-syncing: Watchdog detected hard LOCKUP on CPUN", which may be abbreviated as LockErr.

[0061] The fault type of accessing unallocated memory may have a corresponding description information of “General protection fault”, which may be abbreviated as ProtErr.

[0062] The fault type of kernel NULL pointer dereference may have a corresponding description of "Unable to handlekernel-NULL pointer dereference at XX", which may be abbreviated as NullErr, where XX may be used to represent an offset.

[0063] The fault type in which a system request triggers an exception and causes a crash. The corresponding description information may be "Kernelpanic not-syncing:sysrq triggered crash", which may be abbreviated as SysrqErr.

[0064] The fault type of division by zero error may have a corresponding description information of “Divide error”, which may be abbreviated as DivErr.

[0065] Other (BUG_ON) fault types, the corresponding description information may be "kernel BUG at XXXX", where XXXX may be used to represent a file name or a line number of code, and may be abbreviated as BugOnErr.

[0066] When searching in the pre-built fault classification table, the keywords of the error content can be used as query parameters, and the fault category can be determined by comparing the similarity between the keywords and the description information. Alternatively, the semantics of the error content can be used as query parameters, and the fault category can be determined by comparing the similarity between the semantics of the error content and the description information. The determined fault category can be represented by an abbreviation, and the fault category is finally used as the information of the derived specified category.

[0067] If the fault category is determined by comparing the similarity between keywords and description information, a pre-trained keyword extraction model can be used to extract keywords. It is not difficult to understand that the keyword extraction model can be trained using error content samples and corresponding keyword samples. If the fault category is determined by comparing the semantics of the error content with the description information, a pre-trained semantic recognition model can be used for semantic recognition. With the same model training principle, the semantic recognition model can be trained using error content samples and corresponding semantic recognition result samples.

[0068] Therefore, the error content can be normalized through classification, and the obtained fault type can be used as new information of a specified category identified from the log, so that the information of the specified category can be enriched.

[0069] In one implementation, when the information of the specified category includes error content, the preprocessing of the information of the specified category involved in step S102 may include:

[0070] Step S1021: When the error content includes address information, obtain the instruction pointer corresponding to the error content stored in the pointer instruction register.

[0071] If the information of the specified category extracted contains error content, it can be detected whether the error content contains address information. If the error content contains address information, it can be used as a query start instruction to query the information of the specified category to query the instruction pointer corresponding to the error content stored in the corresponding pointer instruction register.

[0072] Take the case where the information of the specified category contains two error contents as an example, wherein the first error content is "general protection fault" and the second error content is "kernel NULL pointer dereference at -0000000000000008". Through detection, it can be determined that the second error content contains address information, and the address information is "0000000000000008". Therefore, the information of the specified category can be further queried to obtain the instruction pointer corresponding to the error content stored in the pointer instruction register.

[0073] As mentioned before, the pointer instruction register can be extracted through the keyword "RIP:". Exemplarily, the instruction pointer corresponding to the error content stored in the pointer instruction register can be "shash_ahash_digest". It should be noted that, generally speaking, the instruction pointer in the pointer instruction register also contains address information. For example, in the log, the content corresponding to the pointer instruction register is "RIP: hash_ahash_digest+0x2b / 0x110". The address information is 0x2b, which is the offset relative to the starting address, and 0x110 is the total size of the function compiled into binary code following the address information, that is, the length of the occupied address. Based on this, when determining the instruction pointer, the address information "0x2b / 0x110" can be deleted.

[0074] Step S1022: rewrite the offset in the error content into an instruction pointer.

[0075] The error content can be rewritten for the instruction pointer stored in the pointer instruction register when the fault occurs. That is, the address information in the error content can be rewritten as the instruction pointer. Still taking the second error content as an example, the second error content can be rewritten as "kernel NULL pointer dereference at shash_ahash_digest".

[0076] The significance of rewriting the address information is that in the patch matching scenario, it is necessary to match the information of the specified category with the description information of the patch, so as to select the appropriate patch to repair the fault. The so-called matching can be a similarity comparison. Generally speaking, the description information of the patch contains the function name corresponding to the call stack function, but does not contain the address information. Based on this, when the error content contains the offset, deleting it can improve the accuracy of patch selection in the patch matching scenario.

[0077] In one implementation, the preprocessing of the information of the specified category involved in step S102 may include:

[0078] Step S1023: Detect the target category information, and use the target category information containing the address information as information to be preprocessed; the target category information is selected from the designated category information.

[0079] As mentioned above, the information of the specified category may include information of the exception call stack, information of the pointer instruction register, information of the work queue, information of the loaded module, information of the kernel version, information of the command, error content, etc. The target category information may be selected from the information of the specified category. Exemplarily, the information of the exception call stack and the information of the pointer instruction register may be predetermined as the target category information.

[0080] The predetermined basis may be based on prior experience that the probability of address information appearing in the target category information is usually relatively high. Based on this, the target category information can be selected. For the address information in the information of the exception call stack and the information of the pointer instruction register, the expression is usually an offset. Therefore, the target category information can be detected, and the purpose of the detection is to check whether the target category information contains an offset. If the offset is contained, the target category information containing the offset can be used as information to be preprocessed.

[0081] Step S1024: Delete the address information from the information to be pre-processed.

[0082] As mentioned above, for patch matching scenarios or semantic fault identification scenarios, the address information in the exception call stack or in the pointer instruction register will be considered as invalid information or interference information. Therefore, the address information can be deleted from the information to be preprocessed, thereby improving the effectiveness of the specified category information.

[0083] In one implementation, the step S103 of converting the preprocessing result into an embedding vector may include:

[0084] Step S1031: Process the preprocessing result according to a preset information combination method to obtain an information set in a preset format.

[0085] For information of a specified category, it can be marked as N according to the number of categories. As shown in Table 1, Table 1 shows information of multiple categories. If a category of information contains multiple contents, each content can also be used as a counting object. For example, an exception call stack can be used as a category, but if the exception call stack contains 5 called functions, it can be recorded as 5 when counting. Therefore, Table 1 can be used as a set of information in a preset format. Combined with the example in Table 1, there are 12 rows of information of a specified category, and N can be taken as 12.

[0086]

[0087]

[0088] Table 1

[0089] Step S1042: The information in the information set is converted into an embedding vector of a preset dimension through a pre-trained sentence vector conversion model; the sentence vector conversion model is a model that has the function of identifying the semantics between information of specified categories.

[0090] We can use the pre-trained Sentence Transformers model to transform each row in Table 1 into a 1*M dimensional embedding vector. Thus, corresponding to the N rows in Table 1, we can obtain an N*M dimensional embedding vector. The N*M dimensional embedding vector can be expressed as x=[v1, v2, …, v i , …, v n ]. Among them, v i It can represent the embedding vector of 1*M dimension corresponding to the description of the i-th row in Table 1. M can be flexibly set based on the scene requirements. For example, M can be set to 1024.

[0091] For multi-dimensional embedding vectors, different processing methods can be used. For example, if the model used for patch matching or fault semantics recognition can process long sequences, the N*M-dimensional embedding vector can be directly used as the input of the model. However, when the model used for patch matching does not support long sequence input, the N*M transformation can be spliced ​​into a 1*NM-dimensional vector as input.

[0092] After the embedded vector is obtained, the embedded vector can be used as the input of the fault recognition model to identify the fault. For example, some fault recognition models can directly output the fault recognition result based on the received embedded vector, while other fault recognition models can compare the embedded vector with the embedded vector of the known fault to obtain the fault recognition result.

[0093] In one embodiment, the method may further include the following steps:

[0094] Step S104: Acquire a plurality of pre-recorded operation and maintenance embedding vectors, where the operation and maintenance embedding vectors include embedding vectors related to faults and embedding vectors related to candidate fault patches.

[0095] The operation and maintenance embedding vector may be an embedding vector obtained using information related to device operation and maintenance, such as a fault log recorded when a fault occurred previously, or description information of a candidate patch for resolving the fault.

[0096] It is not difficult to understand that the dimension of the operation and maintenance embedding vector can be the same as the dimension of the embedding vector determined in this application.

[0097] Candidate patches for faults can be provided by the upstream community. The upstream community usually refers to the community maintained by the original initiator or non-profit organization for some open source projects, where others can freely contribute to the original project, such as submitting code, fixing bugs, writing documents, etc. Usually, the upstream community will submit patches to solve various downtime problems, which are called candidate patches for faults. Using the description information of the candidate patches for faults, the embedding vector related to the candidate patches for faults can be obtained.

[0098] Step S105: Compare the embedding vector with multiple operation and maintenance embedding vectors for similarity.

[0099] As mentioned above, the embedding vector is determined based on the aforementioned log data processing method. The log data processing method incorporates key information of specified categories such as exception call stack, RIP, work queue, loading module, kernel version, command, error log, etc. In addition, the semantic information between key information can be obtained based on the sentence vector conversion model. Therefore, by comparing the similarity of the embedding vector with multiple operation and maintenance embedding vectors, the similarity between each operation and maintenance embedding vector and the determined embedding vector can be obtained.

[0100] Step S106: According to the result of the similarity comparison, fault diagnosis is performed on the device.

[0101] According to the similarity comparison results, each operation and maintenance embedding vector will get a similarity score. Finally, by sorting the similarity scores of each operation and maintenance embedding vector, the operation and maintenance detection result can be determined according to the sorting result. The operation and maintenance detection result can include determining the fault type to complete the fault identification, or it can include determining the patch that can repair the fault to complete the fault identification.

[0102] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a log data processing device. Figure 2 The figure shows a structural block diagram of a log data processing device according to an embodiment of the present application. The log data processing device may include:

[0103] The information acquisition module 201 is used to determine information of a specified category from the acquired device logs;

[0104] A preprocessing module 202 is used to preprocess the information of a specified category to obtain a preprocessing result;

[0105] The conversion module 203 is used to convert the preprocessing result into an embedding vector, and the embedding vector is used to perform similarity comparison with a predetermined device downtime log.

[0106] In one implementation, the information acquisition module 201 may include:

[0107] A multi-category information extraction submodule is used to extract multiple categories of information from device logs using a preset regular expression;

[0108] The reliability identification module is used to identify the reliability of each category of information. When the identification result contains unreliable information, the unreliable information is cleaned to obtain information of a specified category; the unreliable information includes information with a specified mark.

[0109] In one implementation, when the information of the specified category includes error content, it may also include:

[0110] The fault type determination module is used to query in a pre-built fault classification table to obtain a fault type that matches the error content; the fault classification table at least contains the fault type and descriptive information corresponding to the fault type, and the descriptive information is used to compare with the error content; the fault type is used as information of the specified category.

[0111] In one implementation, when the information of the specified category includes error content, the pre-processing module 202 may include:

[0112] The instruction pointer acquisition submodule is used to acquire the instruction pointer corresponding to the error content stored in the pointer instruction register when the error content contains address information;

[0113] The rewriting submodule is used to rewrite the address information into an instruction pointer.

[0114] In one implementation, the pre-processing module 202 may include:

[0115] The preprocessing object confirmation submodule is used to detect the target category information and use the target category information containing address information as the information to be preprocessed; the target category information is selected from the designated category information;

[0116] The deletion submodule is used to delete the address information from the information to be preprocessed.

[0117] In one implementation, the conversion module 203 may include:

[0118] The information set construction submodule is used to process the preprocessing results according to a preset information combination method to obtain an information set in a preset format;

[0119] The vector conversion submodule is used to convert the information in the information set into an embedding vector of a preset dimension through a pre-trained sentence vector conversion model; the sentence vector conversion model is a model that has the function of identifying the semantics between information of specified categories.

[0120] In one embodiment, it further includes:

[0121] An operation and maintenance embedding vector acquisition module, used to acquire a plurality of pre-recorded operation and maintenance embedding vectors, the operation and maintenance embedding vectors including an embedding vector related to a fault and an embedding vector related to a candidate fault patch;

[0122] A similarity comparison module, used to compare the similarity between the embedding vector and multiple operation and maintenance embedding vectors;

[0123] The fault identification module is used to identify the fault of the device according to the result of the similarity comparison.

[0124] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0125] Figure 3 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 3 As shown, the electronic device includes: a memory 310 and a processor 320. The memory 310 stores a computer program that can be run on the processor 320. When the processor 320 executes the computer program, the method in the above embodiment is implemented. The number of the memory 310 and the processor 320 can be one or more.

[0126] The electronic device also includes:

[0127] The communication interface 330 is used to communicate with external devices and perform data exchange transmission.

[0128] If the memory 310, the processor 320 and the communication interface 330 are implemented independently, the memory 310, the processor 320 and the communication interface 330 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0129] Optionally, in a specific implementation, if the memory 310, the processor 320 and the communication interface 330 are integrated on a chip, the memory 310, the processor 320 and the communication interface 330 can communicate with each other through an internal interface.

[0130] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0131] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0132] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0133] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0134] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0135] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0136] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0137] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0138] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0139] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0140] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0141] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0142] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for processing log data, characterized in that: include: Determine information of a specified category from the acquired device logs; Preprocessing the information of the specified category to obtain a preprocessing result; The preprocessing result is converted into an embedding vector, and the embedding vector is used for similarity comparison with a predetermined equipment fault log.

2. The method according to claim 1, characterized in that The determining of information of a specified category from the acquired device logs includes: Extracting multiple categories of information from the device log using a preset regular expression; Reliability identification is performed on each of the categories of information. When the identification result contains unreliable information, the unreliable information is cleaned to obtain information of the specified category; the unreliable information includes information with a specified mark.

3. The method according to claim 1, characterized in that In the case where the information of the specified category includes error content, it also includes: Searching in a pre-built fault classification table to obtain a fault type that matches the error content; the fault classification table at least includes a fault type and description information corresponding to the fault type, and the description information is used to compare with the error content; The fault type is used as the designated category of information.

4. The method according to claim 1, characterized in that In the case where the information of the specified category contains error content, the preprocessing of the information of the specified category includes: In the case where the error content includes address information, obtaining an instruction pointer corresponding to the error content stored in a pointer instruction register; The address information is rewritten as the instruction pointer.

5. The method according to claim 1, characterized in that The preprocessing of the information of the specified category includes: Detecting information of a target category, and using the information of the target category containing address information as information to be preprocessed; the information of the target category is selected from the information of the designated category; The address information is deleted from the information to be preprocessed.

6. The method according to claim 1, characterized in that The converting the preprocessing result into an embedding vector comprises: Processing the preprocessing result according to a preset information combination method to obtain an information set in a preset format; The information in the information set is converted into an embedding vector of a preset dimension through a pre-trained sentence vector conversion model; the sentence vector conversion model is a model that has the function of identifying the semantics between information of a specified category.

7. The method according to claim 1 or 6, characterized in that: Also includes: Acquire a plurality of pre-recorded operation and maintenance embedding vectors, wherein the operation and maintenance embedding vectors include an embedding vector related to a fault and an embedding vector related to a candidate fault patch; Comparing the embedding vector with the plurality of operation and maintenance embedding vectors for similarity; According to the result of the similarity comparison, fault identification is performed on the device.

8. A log data processing device, characterized in that: include: An information acquisition module, used to determine information of a specified category from the acquired device logs; A preprocessing module, used for preprocessing the information of the specified category to obtain a preprocessing result; The conversion module is used to convert the preprocessing result into an embedding vector, and the embedding vector is used to perform similarity comparison with a predetermined device downtime log.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images