Log classification analysis method, device, equipment and readable storage medium

Abstract

The present invention discloses a log classification and analysis method, which realizes unified processing of logs with different formats and different data contents by uniformly rearranging the original logs according to data fields, eliminating format differences, and converting the logs into vector form for storage according to the numbers corresponding to their important word segments after obtaining structured logs, extracting the common parts to retain independent information, reducing storage costs, and then calling a pre-trained neural network model to perform log classification processing on the log vectors, realizing classification and labeling of massive logs, calling a neural network model for feature recognition is conducive to discovering regularities and common problems in logs, facilitating troubleshooting and locating faults from massive logs, and laying a foundation for subsequent functions such as log anomaly detection. The present invention also discloses a log classification and analysis device, equipment, and readable storage medium, which have corresponding technical effects.

Description

Technical Field

[0001] The present invention relates to the field of log management technology, and in particular to a log classification and analysis method, device, equipment and readable storage medium. Background Art

[0002] With the rapid development of the Internet, cloud operating systems have become increasingly popular. However, due to their large scale and high complexity, cloud operating systems are more prone to various problems when running. Therefore, how to ensure reliability has become an important issue in system design and management. In the management of modern large-scale distributed systems, system logs have always been the main source of detecting system status. The large amount of log records generated by system operation is usually the main source of information for system administrators to troubleshoot. As the scale of modern systems increases day by day and the complexity of the system continues to increase, various components generate a large amount of log information. For example, there may be potential anomalies in the system, but they are submerged in massive logs; for example, when a new version is launched, the system behavior changes, but it cannot be perceived, etc. This is a huge challenge for system administrators to troubleshoot.

[0003] In summary, how to provide an effective information analysis method for massive logs is a technical problem that those skilled in the art urgently need to solve. Summary of the invention

[0004] The purpose of the present invention is to provide a log classification and analysis method, device, equipment and readable storage medium, which can realize fast and effective classification and marking of massive logs, which is conducive to discovering patterns and common problems in logs, and facilitates troubleshooting and fault location from massive logs and the storage of logs.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A log classification and analysis method, comprising:

[0007] Get the original log to be analyzed;

[0008] Parsing all data fields in the original log, and structurally rearranging the original log according to the data fields to obtain a structured log;

[0009] Extracting all the segmented words in the structured log;

[0010] Screen out important participles from the participles, and determine the participle numbers corresponding to the important participles;

[0011] The word numbers corresponding to the important word are represented in a log according to the arrangement order of the important word in the original log to obtain a log vector;

[0012] The pre-trained neural network model is called to perform log classification processing on the log vector to obtain a classified log.

[0013] Optionally, screening out important participles from the participles and determining participle numbers corresponding to the important participles includes:

[0014] Determine the information statistics type of the original log;

[0015] Determine whether there is a word segmentation library corresponding to the information statistics type;

[0016] If the word segmentation library exists, determine whether the word segmentation has a corresponding item in the word segmentation library;

[0017] If there is a corresponding item, the segmentation with the corresponding item is taken as an important segmentation, and the segmentation number corresponding to the important segmentation is extracted by comparing the segmentation library.

[0018] Optionally, the log classification and analysis method further includes:

[0019] Obtaining the participles for which there are no corresponding items in the participles as differential participles;

[0020] receiving a user's designation of important participles and irrelevant participles in the differential participles;

[0021] Corresponding segmentation numbers are created for the important segmentations in the differential segmentations specified by the user and added to the segmentation library.

[0022] Optionally, if there is no word segmentation library corresponding to the information statistics type, the log classification analysis method includes:

[0023] Extracting the segmented words contained in the structured log, and deleting duplicate items in the segmented words, to generate a full segmented word table;

[0024] receiving a user's instruction to delete irrelevant segmentations in the entire segmentation table, and deleting the irrelevant segmentations from the entire segmentation table as an important segmentation table;

[0025] A corresponding word segmentation number is created for each word segmentation in the important word segmentation table to obtain a word segmentation library, and the step of determining whether the word segmentation has a corresponding item in the word segmentation library is performed.

[0026] Optionally, if there is no word segmentation library corresponding to the information statistics type, the log classification analysis method includes:

[0027] Extracting the segmented words contained in the structured log, and counting the number of occurrences of each segmented word, to generate a segmented word frequency statistics table;

[0028] Delete the words whose occurrence times are lower than the threshold in the word segmentation frequency statistics table to generate a high-frequency word segmentation statistics table;

[0029] A corresponding word segmentation number is created for each word segmentation in the high-frequency word segmentation statistical table to obtain a word segmentation library, and the step of determining whether the word segmentation has a corresponding item in the word segmentation library is performed.

[0030] Optionally, parsing all data fields in the original log includes:

[0031] Determine the data field arrangement type of the original log;

[0032] Obtaining the parsing rules corresponding to the data field arrangement type;

[0033] The parsing rule is called to extract data fields from the original log.

[0034] Optionally, the calling of a pre-trained neural network model to perform log classification processing on the log vector to obtain a classified log includes:

[0035] The pre-trained LVQ neural network model is called to perform log classification processing on the log vector to obtain a classified log.

[0036] A log classification and analysis device, comprising:

[0037] A log acquisition unit, used to acquire the original log to be analyzed;

[0038] A log structuring unit, used for parsing all data fields in the original log, and re-arranging the original log in a structured manner according to the data fields to obtain a structured log;

[0039] A word extraction unit, used to extract all the words in the structured log;

[0040] A number determination unit, used to screen out important participles from the participles and determine the participle numbers corresponding to the important participles;

[0041] A vector representation unit, used for performing log representation on the word numbers corresponding to the important word according to the arrangement order of the important word in the original log to obtain a log vector;

[0042] The classification analysis unit is used to call a pre-trained neural network model to perform log classification processing on the log vector to obtain a classified log.

[0043] A computer device comprising:

[0044] Memory for storing computer programs;

[0045] A processor is used to implement the steps of the above-mentioned log classification and analysis method when executing the computer program.

[0046] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the log classification and analysis method are implemented.

[0047] By applying the method provided by the embodiment of the present invention, the original logs are rearranged in a unified structure according to the data fields, so as to realize the unified processing of logs with different formats and different data contents, eliminate the format differences, obtain the structured logs, and convert the logs into vector form for storage according to the numbers corresponding to the important word segments therein, so as to extract the common parts and retain the independent information, reduce the storage cost, and then call the pre-trained neural network model to perform log classification processing on the log vectors, so as to realize the classification and marking of massive logs. Calling the neural network model for feature recognition is conducive to discovering the rules and common problems in the logs, and is convenient for troubleshooting and locating faults from massive logs. At the same time, it lays the foundation for subsequent functions such as log anomaly detection, thereby ensuring the stable operation of the system and reducing the complexity of system operation and maintenance.

[0048] Correspondingly, the embodiments of the present invention also provide a log classification analysis device, equipment and readable storage medium corresponding to the above-mentioned log classification analysis method, which have the above-mentioned technical effects and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 This is a flow chart of an implementation of a log classification and analysis method in an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of the structure of a log classification and analysis device according to an embodiment of the present invention;

[0052] Figure 3 The figure is a schematic diagram of the structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The core of the present invention is to provide a log classification and analysis method, which can realize effective information analysis for massive logs.

[0054] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0055] The applicant analyzed and found that an important reason why it is difficult to achieve effective troubleshooting based on system logs is that there is too much information and it is difficult to classify it well. At the same time, the logs that record information are often in various formats and difficult to standardize. Therefore, the applicant proposed to collect logs and learn clustering patterns, and divide the massive logs into a few patterns through AI algorithms, so that the original tens of millions of lines of logs can be classified into dozens of lines to speed up the efficiency of log viewing and troubleshooting.

[0056] Based on this, the applicant proposed a log classification analysis method, please refer to Figure 1 , Figure 1 This is a flow chart of a log classification and analysis method according to an embodiment of the present invention, the method comprising the following steps:

[0057] S101, obtaining the original log to be analyzed;

[0058] The original log to be analyzed is a log automatically generated by the system. In this embodiment, there is no limitation on the data objects and log formats recorded in the original log. This embodiment can realize the rapid classification and analysis of massive logs generated for different data objects in different devices and different systems.

[0059] S102, parsing all data fields in the original log, and structurally rearranging the original log according to the data fields to obtain a structured log;

[0060] The original log contains corresponding data recording rules, i.e., the log recording rules of the system that generates the original log. The original log can be obtained by assembling the data fields according to the rules. The data recording rules in different original logs may be different. In order to facilitate the unified analysis of original logs with different data formats and different recording objects, this embodiment summarizes all logs and re-arranges them in a unified structure to eliminate format differences.

[0061] In this step, it is necessary to determine the recording rules in the original log and determine all the data fields contained therein, such as time, special characters, IP, etc. The specific division of data fields can be set according to the needs of actual log analysis.

[0062] The original logs are restructured according to the data fields, and the semi-structured and unstructured original logs are converted into structured logs to facilitate subsequent content analysis.

[0063] Table 1 below shows two syslog original logs:

[0064] Table 1

[0065] Log sequence number log Syslog1 2020-06-30 20:00:00; snmpd

[2247] :connection from UDP:[0.0.0.0]:33929 Syslog2 2020-06-30 20:00:01; snmpd

[2247] : received SNMP packets from UDP:[0.0.0.0]:57638

[0066] The columns can be structured into fields such as timestamp, log source, and log content. After the original log shown in Table 1 is rearranged, a structured log is obtained as shown in Table 2 below:

[0067] Table 2

[0068] Log sequence number Timestamp source log Syslog1 2020-06-30 20:00 snmpd connection from UDP:[0.0.0.0]:33929 Syslog2 2020-06-30 20:00 snmpd received SNMP packets from UDP:[0.0.0.0]:57638

[0069] It can be seen that each data item in the original log record after structural rearrangement is clear and unambiguous, which facilitates the accurate analysis of subsequent logs. It should be noted that the implementation method of parsing the data field in this embodiment is not limited. Since the data field arrangement rules corresponding to different logs may be different, in order to facilitate fast field parsing, a method for parsing all data fields in the original log is as follows:

[0070] (1) Determine the data field arrangement type of the original log;

[0071] The data field arrangement type can be determined according to factors such as the type of the original log or the data object recorded.

[0072] (2) Obtaining parsing rules corresponding to the data field arrangement type;

[0073] (3) Call the parsing rules to extract data fields from the original log.

[0074] In the above data field parsing method, parsing rules corresponding to various data field arrangement types are pre-configured. Rapid matching of parsing rules is achieved through data field arrangement types, which can speed up the parsing process and improve parsing efficiency.

[0075] In this embodiment, only the above-mentioned implementation method is used as an example for detailed description. Other parsing methods, such as determining keywords in the original log, extracting fields based on keywords, or parsing according to pre-set field parsing rules, etc., can all refer to the introduction of this embodiment and will not be repeated here.

[0076] S103, extracting all the segmented words in the structured log;

[0077] Extract all the identifiable word segments in the structured log. For example, if the log information in the structured log is connection from UDP, the extracted word segments may be: connection, from, UDP. The extraction implementation of the word segmentation in the structured log can refer to the extraction implementation of the word segmentation in the related art, which will not be described in detail in this embodiment.

[0078] S104, screening out important participles in the participles, and determining the participle numbers corresponding to the important participles;

[0079] According to the preset rules, important participles in the participles are filtered out, and interfering participles and useless participles are deleted. It should be noted that in this embodiment, important participles refer to participles with high importance that users pay attention to among all participles. In the subsequent classification analysis, in-depth analysis is mainly based on the important participles contained in the log. Important participles include IP addresses, port numbers, hexadecimal numbers, URL links, etc. The specific filtering rules and implementation methods for important participles are not limited in this embodiment. Important participles can be specified by the user, or they can be compared and determined against a preset important participle table. They can be set accordingly according to the needs of actual information analysis.

[0080] After filtering out the important words among all the words, create a unique word number for the important words to facilitate subsequent log management and identification.

[0081] It should be noted that a corresponding relationship data of important segmented words and their numbers is set separately for each original log, so as to facilitate accurate feature analysis of each original log.

[0082] Table 3 below shows an example of important segmentation words and corresponding segmentation word numbers of an original log. Other segmentation words and numbering settings based on this embodiment can refer to the following table:

[0083] Table 3

[0084] serial number Participle 1 connection 2 from 3 packets 4 received 5 SNMP 6 UDP

[0085] S105, the word numbers corresponding to the important word are represented in the log according to the arrangement order of the important word in the original log to obtain a log vector;

[0086] Replace the word segmentation in the log content with the word segmentation number, ignore the word segmentation that is not included in the word segmentation library, keep the original relative order of the word segmentation, and convert the log text into a vector representation.

[0087] The corresponding relationship between the log information corresponding to a log sequence number and the log vector is shown in Table 4 below:

[0088] Table 4

[0089] Log sequence number Log information Log vector Syslog1 connection from UDP (1,2,6) Syslog2 received SNMP packets from UDP (4,5,3,2,6)

[0090] After converting logs into vector form for storage, on the one hand, it is helpful to discover patterns and common problems in the logs, facilitate troubleshooting and locating faults from massive logs, and lay the foundation for subsequent functions such as log anomaly detection. On the other hand, massive logs only need a small amount of log pattern representation, extracting the common parts to retain independent information and reduce storage costs.

[0091] S106: Call the pre-trained neural network model to perform log classification processing on the log vector to obtain a classified log.

[0092] In this embodiment, a neural network model is used to classify massive system logs, which greatly improves the accuracy of classification, lays the foundation for subsequent functions such as log anomaly detection, ensures the stability of system operation, and reduces the complexity of system operation and maintenance.

[0093] The network type of the neural network model is not limited in this embodiment. Nowadays, artificial neural networks have developed more than 100 models and multiple establishment methods, and the model can be selected according to the needs of model training and accuracy requirements. For example, the pre-trained LVQ neural network model can be called to perform log classification processing on the log vector to obtain the classified log.

[0094] LVQ neural network stands for learning vector quantization neural network. It is a learning algorithm that trains the competitive layer in a teacher state. It has a wide range of applications in the field of pattern recognition. The LVQ network has a simple structure and fast training convergence speed, making it very suitable for pattern recognition and classification.

[0095] In addition to the LVQ neural network, other types of network models may also be selected. The settings of other types of network models may refer to the introduction of this embodiment.

[0096] To deepen our understanding, we introduce a training method for the LVQ neural network model, which mainly includes the following steps:

[0097] (1) Setting model parameters, including weight matrix, learning function, and learning rate;

[0098] (2) Initialize the model and give each weight matrix a relatively small random non-zero vector;

[0099] (3) Input the log vector and train the network;

[0100] (4) Based on the comparison between the training results and the target output, the weight vector of the competition layer is modified by the Kohonen rule;

[0101] (5) Determine whether the maximum number of iterations has been reached. If so, go to step 6; if not, go to step 3;

[0102] (6) Determine whether all samples have been learned. If yes, then end; otherwise, go to step 3.

[0103] The technical solution provided by the embodiment of the present invention realizes unified processing of logs with different formats and different data contents by uniformly rearranging the original logs according to the data fields, eliminates format differences, obtains structured logs, and converts the logs into vector form for storage according to the numbers corresponding to the important word segments therein, extracts the common parts and retains the independent information, reduces the storage cost, and then calls the pre-trained neural network model to perform log classification processing on the log vectors, thereby realizing classification and marking of massive logs. Calling the neural network model for feature recognition is conducive to discovering regularities and common problems in the logs, making it convenient to troubleshoot problems and locate faults from massive logs, and at the same time lays the foundation for subsequent functions such as log anomaly detection, thereby ensuring stable system operation and reducing the complexity of system operation and maintenance.

[0104] It should be noted that, based on the above embodiments, the embodiments of the present invention also provide corresponding improved solutions. In the preferred / improved embodiments, the same steps or corresponding steps as those in the above embodiments can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be described one by one in the preferred / improved embodiments of this article.

[0105] The above embodiment does not limit the screening rules for important participles. This embodiment introduces a method for determining important participles, which is as follows:

[0106] (1) Determine the information statistics type of the original log;

[0107] The information statistics type can be divided according to the data objects recorded in the original log, such as all the records are CPU operation information, or all the records are server fan fault information, etc.; or it can be divided according to the device to which the generated data object belongs, such as all the records are server information, or all the records are external device information, etc., which is not limited in this embodiment. For the content of different original logs, the information statistics type is divided according to the information recorded and the similarity of the segmented words. Corresponding segmented word libraries are configured under different information statistics types. The segmented word libraries record common important segmented words, so that it is easy to quickly locate important segmented words according to the segmented word library and realize rapid analysis.

[0108] (2) Determine whether there is a word segmentation library corresponding to the information statistics type;

[0109] (3) If there is a word segmentation library, determine whether the word segmentation has a corresponding item in the word segmentation library;

[0110] (4) If there is a corresponding item, the word with the corresponding item is regarded as an important word, and the word number corresponding to the important word is extracted by comparing it with the word library.

[0111] It should be noted that the above method does not limit the implementation method of the word segmentation for which there is no corresponding item in the word segmentation. One implementation method is as follows:

[0112] (5) obtaining the participles that do not have corresponding items in the participles as differential participles;

[0113] (6) receiving the user's designation of important participles and irrelevant participles in the differential participles;

[0114] (7) Create corresponding segmentation numbers for the important segmentations in the differential segmentations specified by the user and add them to the segmentation library.

[0115] The word segmentation library records a limited number of word segmentations. During the operation of the system, some new but important information may appear at any time and need to be used in subsequent analysis. In order to timely update the important word segmentations in the word segmentation library, the above implementation method outputs the difference word segmentations and accepts the user's designation of importance. For example, if a certain difference analysis is designated as an important word segmentation, it will be added to the word segmentation library to realize the update of the word segmentation library and facilitate the comprehensiveness of subsequent information analysis.

[0116] In addition, the above method does not limit the corresponding processing method when there is no word segmentation library corresponding to the information statistics type. The following introduces two implementation methods:

[0117] (1.1) If there is no word segmentation library corresponding to the information statistics type, extract the word segments contained in the structured log, delete the duplicate items in the word segmentation, and generate a full word segmentation table;

[0118] (1.2) receiving a user's instruction to delete irrelevant segmentations in the entire segmentation table, deleting the irrelevant segmentations from the entire segmentation table as an important segmentation table;

[0119] (1.3) Create a corresponding word segmentation number for each word in the important word segmentation table, obtain a word segmentation library, and execute the step of determining whether the word segmentation has a corresponding item in the word segmentation library.

[0120] In the above method, when encountering a non-existent word segmentation library, all word segmentations are extracted and the user's designation of the importance of all the word segmentations is received, that is, some word segmentations are classified as important word segmentations and other word segmentations are classified as irrelevant word segmentations. A word segmentation library is created according to the user's designation, which can quickly locate the user's focus and facilitate subsequent classification and analysis of logs according to the user's specified focus.

[0121] Another way to handle this is as follows:

[0122] (2.1) If there is no word segmentation library corresponding to the information statistics type, extract the word segments contained in the structured log, count the number of occurrences of each word, and generate a word segmentation frequency statistics table;

[0123] (2.2) Delete the words whose occurrence times are lower than the threshold in the word frequency statistics table to generate a high-frequency word statistics table;

[0124] (2.3) Create a corresponding word segmentation number for each word in the high-frequency word segmentation statistics table, obtain a word segmentation library, and execute the step of determining whether the word segmentation has a corresponding item in the word segmentation library.

[0125] In the above method, when a non-existent word segmentation library is encountered, the frequency of occurrence of word segmentations is automatically counted, and high-frequency word segmentations are automatically used as important word segmentations. This can improve the generation speed of the word segmentation library, quickly locate important word segmentations, improve the overall processing efficiency, and realize the automation of classification analysis.

[0126] In this embodiment, only the above two implementation methods are introduced as examples. Other implementation methods can refer to the above introduction and will not be described in detail here.

[0127] Corresponding to the above method embodiment, the embodiment of the present invention further provides a log classification analysis device. The log classification analysis device described below and the log classification analysis method described above can be referenced to each other.

[0128] See also Figure 2 As shown, the device includes the following modules:

[0129] The log acquisition unit 110 is mainly used to acquire the original log to be analyzed;

[0130] The log structuring unit 120 is mainly used to parse all data fields in the original log, and to rearrange the original log into a structured manner according to the data fields to obtain a structured log;

[0131] The word segmentation extraction unit 130 is mainly used to extract all the word segments in the structured log;

[0132] The number determination unit 140 is mainly used to filter out important participles in the participles and determine the participle numbers corresponding to the important participles;

[0133] The vector representation unit 150 is mainly used to represent the logs according to the arrangement order of the important words in the original logs by the word numbers corresponding to the important words, so as to obtain the log vectors;

[0134] The classification analysis unit 160 is mainly used to call the pre-trained neural network model to perform log classification processing on the log vector to obtain the classified log.

[0135] Corresponding to the above method embodiment, the embodiment of the present invention further provides a computer device. The computer device described below and the log classification and analysis method described above can be referred to each other.

[0136] The computer equipment includes:

[0137] Memory for storing computer programs;

[0138] The processor is used to implement the steps of the log classification and analysis method of the above method embodiment when executing a computer program.

[0139] For details, please refer to Figure 3 , is a specific structural diagram of a computer device provided in this embodiment. The computer device may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 322 (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer applications 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the central processing unit 322 can be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the computer device 301.

[0140] The computer device 301 may also include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0141] The steps in the log classification and analysis method described above can be implemented by the structure of a computer device.

[0142] Corresponding to the above method embodiment, the embodiment of the present invention further provides a readable storage medium. The readable storage medium described below and the log classification and analysis method described above can refer to each other.

[0143] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the log classification and analysis method of the above method embodiment are implemented.

[0144] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0145] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

Claims

1. A log classification and analysis method, characterized in that: include: Get the original log to be analyzed; Parsing all data fields in the original log, and structurally rearranging the original log according to the data fields to obtain a structured log; Extracting all the segmented words in the structured log; Filter out important participles in the participles, and determine the participle numbers corresponding to the important participles; the important participles include IP addresses, port numbers, hexadecimal numbers, and URL links; The word numbers corresponding to the important word are represented in a log according to the arrangement order of the important word in the original log to obtain a log vector; Calling a pre-trained neural network model to perform log classification processing on the log vector to obtain a classified log; The step of filtering out important participles from the participles and determining the participle numbers corresponding to the important participles includes: determining the information statistics type of the original log; judging whether there is a participle library corresponding to the information statistics type; if there is a participle library, judging whether there is a corresponding item in the participle library; if there is a corresponding item, taking the participle with the corresponding item as the important participle, and extracting the participle numbers corresponding to the important participles by comparing the participle library; The method further includes: obtaining a segmentation for which no corresponding item exists in the segmentation as a differential segmentation; receiving a user's designation of important segmentations and irrelevant segmentations in the differential segmentation; creating a corresponding segmentation number for the important segmentations in the differential segmentation designated by the user, and adding the number to the segmentation library; If there is no word segmentation library corresponding to the information statistics type, the method includes: extracting the word segments contained in the structured log, deleting duplicate items in the word segments, and generating a full word segmentation table; receiving a user's deletion instruction for irrelevant word segments in the full word segmentation table, deleting the irrelevant word segmentation from the full word segmentation table as an important word segmentation table; creating a corresponding word segmentation number for each word in the important word segmentation table, obtaining a word segmentation library, and executing the step of determining whether the word segmentation has a corresponding item in the word segmentation library; Or, if there is no word segmentation library corresponding to the information statistics type, the method includes: extracting the word segments contained in the structured log, and counting the number of occurrences of each word segmentation, generating a word segmentation frequency statistics table; deleting the word segmentation whose number of occurrences is lower than a threshold in the word segmentation frequency statistics table, generating a high-frequency word segmentation statistics table; creating a corresponding word segmentation number for each word in the high-frequency word segmentation statistics table, obtaining a word segmentation library, and executing the step of determining whether the word segmentation has a corresponding item in the word segmentation library; The parsing of all data fields in the original log includes: determining keywords in the original log; and extracting data fields from the original log according to the keywords.

2. The log classification and analysis method according to claim 1, characterized in that: The parsing of all data fields in the original log includes: Determine the data field arrangement type of the original log; Obtaining the parsing rules corresponding to the data field arrangement type; The parsing rule is called to extract data fields from the original log.

3. The log classification and analysis method according to claim 1, characterized in that: The calling of the pre-trained neural network model to perform log classification processing on the log vector to obtain the classified log includes: The pre-trained LVQ neural network model is called to perform log classification processing on the log vector to obtain a classified log.

4. A log classification and analysis device, characterized in that: include: A log acquisition unit, used to acquire the original log to be analyzed; A log structuring unit, used for parsing all data fields in the original log, and re-arranging the original log in a structured manner according to the data fields to obtain a structured log; A word extraction unit, used to extract all the words in the structured log; A number determination unit, used to screen out important participles in the participles and determine the participle numbers corresponding to the important participles; the important participles include IP addresses, port numbers, hexadecimal numbers, and URL links; A vector representation unit, used for performing log representation on the word numbers corresponding to the important word according to the arrangement order of the important word in the original log to obtain a log vector; A classification analysis unit, used to call a pre-trained neural network model to perform log classification processing on the log vector to obtain a classified log; The number determination unit is specifically used to determine the information statistics type of the original log; determine whether there is a word segmentation library corresponding to the information statistics type; if there is a word segmentation library, determine whether the word segmentation has a corresponding item in the word segmentation library; if there is a corresponding item, take the word segmentation with the corresponding item as an important word segmentation, and extract the word segmentation number corresponding to the important word segmentation by comparing the word segmentation library; The device is further used to obtain the segmentations that do not have corresponding items in the segmentations as differential segmentations; receive the user's designation of important segmentations and irrelevant segmentations in the differential segmentations; create corresponding segmentation numbers for the important segmentations in the differential segmentations designated by the user, and add them to the segmentation library; If there is no word segmentation library corresponding to the information statistics type, the device is also used to extract the word segments contained in the structured log, delete the duplicate items in the word segmentation, and generate a full word segmentation table; receive a user's deletion instruction for irrelevant word segmentations in the full word segmentation table, delete the irrelevant word segmentations from the full word segmentation table, and use it as an important word segmentation table; create a corresponding word segmentation number for each word in the important word segmentation table to obtain a word segmentation library, and execute the step of determining whether the word segmentation has a corresponding item in the word segmentation library; Or, if there is no word segmentation library corresponding to the information statistics type, the device is also used to extract the word segments contained in the structured log, and count the number of occurrences of each word segmentation to generate a word segmentation frequency statistics table; delete the word segmentation frequency statistics table whose number of occurrences is lower than the threshold to generate a high-frequency word segmentation statistics table; create a corresponding word segmentation number for each word in the high-frequency word segmentation statistics table to obtain a word segmentation library, and perform the step of determining whether the word segmentation has a corresponding item in the word segmentation library; The log structuring unit is specifically used to determine keywords in the original log; and extract data fields from the original log according to the keywords.

5. A computer device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the log classification and analysis method as claimed in any one of claims 1 to 3 when executing the computer program.

6. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the log classification and analysis method according to any one of claims 1 to 3 are implemented.

Images