Log data processing method and device, electronic equipment, storage medium and product

Abstract

The invention discloses a log data processing method and device, electronic equipment, a storage medium and a product which are applied to the technical field of data preprocessing, and the method comprises the steps: determining a variable key value log feature and a fixed key value log feature by analyzing a key value of historical docked system log data; when it is detected that the to-be-docked log data exists, obtaining to-be-docked key value features of the to-be-docked log data; and determining a log analysis mode corresponding to the to-be-docked log system data according to the matching condition of the to-be-docked key value feature, the variable key value log feature and the fixed key value log feature. According to the method and the device, the problem that corresponding log analysis templates cannot be matched for the log data efficiently and precisely in related technologies can be solved, and the log data matching efficiency and matching accuracy are effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of data preprocessing, and in particular to a log data processing method, device, electronic device, non-volatile storage medium and computer program product. Background Art

[0002] With the development of network technology, there are more and more types of network security devices, and the log contents of different network security devices are different. In the scenario where the same user uses multiple network security devices to defend the computer information system, it is necessary to process these different log data.

[0003] At present, when processing multi-source logs, the relevant technology can first classify the log data, and then use the corresponding log parsing template to parse the logs of different categories. However, the relevant technology cannot efficiently and accurately match the corresponding log parsing template for the log data, resulting in the final log data processing efficiency and processing results cannot meet the needs of users.

[0004] In view of this, improving the log data matching efficiency and matching accuracy is a technical problem that technical personnel in this field need to solve.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present application provides a log data processing method, device, electronic device, non-volatile storage medium and computer program product, which can effectively improve the log data matching efficiency and matching accuracy.

[0007] In order to solve the above technical problems, this application provides the following technical solutions:

[0008] On the one hand, the present application provides a log data processing method, including:

[0009] By analyzing the key values ​​of the historically connected system log data, the variable key value log features and the fixed key value log features are determined; when it is detected that there is log data to be connected, the key value features to be connected of the log data to be connected are obtained; based on the matching of the key value features to be connected, the variable key value log features and the fixed key value log features, the log parsing method corresponding to the log system data to be connected is determined.

[0010] Exemplarily, the log parsing method corresponding to the log data to be connected is determined according to the matching of the key-value feature to be connected, the changing key-value log feature and the fixed key-value log feature, including: if the key-value feature to be connected successfully matches the changing key-value log feature, the first type of log parsing template corresponding to the changing key-value log feature is used to perform log docking on the log data to be connected; if the key-value feature to be connected successfully matches the fixed key-value log feature, the second type of log parsing template corresponding to the fixed key-value log feature is used to perform log docking on the log data to be connected; if the key-value feature to be connected fails to match both the changing key-value log feature and the fixed key-value log feature, the third type of log parsing template matching the key-value feature to be connected is used to perform data docking, and the key-value feature to be connected is used as the third type of log feature.

[0011] Exemplarily, the method of determining variable key value log features and fixed key value log features by analyzing key values ​​of historical connected system log data includes: obtaining historical system log data of a target system log type and having a corresponding log parsing template as historical connected system log data; using a preset text string to replace data of a corresponding type in the message body data of the historical system log data; using the target delimiter that appears the most times as a segmentation symbol, performing log data segmentation processing on the replaced historical system log data, using the first data of each segmentation segment as the key value, and removing duplicate key values ​​corresponding to the target type data from each key value to obtain key value information; if at least two of the key values ​​in the key value information are different, then the key value information corresponding to the historical system log data is variable key value data; if all the key values ​​in the key value information are different, then the key value information corresponding to the historical system log data is fixed key value data.

[0012] Exemplarily, the determination of variable key value log features and fixed key value log features includes: calculating hash values ​​of each key value of the fixed key value data, and using the hash value, the total number of key values ​​contained in the fixed key value data and the separator as fixed key value log features; counting the occurrence probability of each key value in the variable key value data, and using each key value and the corresponding occurrence probability as variable key value log features.

[0013] Exemplarily, after determining the variable key value log characteristics and the fixed key value log characteristics, it also includes: if the fixed key value log characteristics of the current historical system log data are different from the fixed key value log characteristics of other historical connected system log data, then storing the fixed key value log characteristics of the current historical system log data; if there is at least one target historical system log data whose fixed key value log characteristics are the same as the fixed key value log characteristics of the current historical system log data, then determining the difference data between the target historical system log data and the current historical system log data, storing the fixed key value log characteristics of the current historical system log data and the difference between the target historical system log data, and adding the difference between the target historical system log data and the current historical system log data to the storage data of the target historical system log data.

[0014] Exemplarily, after determining the variable key value log features and the fixed key value log features, it also includes: for each key value of the variable key value log features of the current historical system log data, counting the number of times it appears in all variable key value log features, and determining the weight value of each key value based on the number of appearances of each key value, the total number of variable key value log features, and the probability of appearance, and storing each key value and the corresponding weight value as the final fixed key value log feature.

[0015] Exemplarily, the key-value features to be connected include a hash value to be connected, a separator to be connected and the total number of key values ​​to be connected, and the matching status according to the key-value features to be connected, the variable key-value log features and the fixed key-value log features includes: determining the log category to which the log data to be connected belongs according to the separator to be connected and the total number of key values ​​to be connected; if there is target historical connected system log data with the same hash value as the hash value to be connected, then the log data to be connected is the system log data with a fixed key value; if the matched target historical connected system log data is unique, then the log data to be connected is log-connected based on the log parsing template of the target historical connected system log data; if the matched target historical connected system log data is not unique, then the difference content between the log data to be connected and each target historical connected system log is compared, and the log parsing template of the first historical system log data that successfully matches the difference content is used to connect the log data to be connected.

[0016] Exemplarily, the matching situation of the key value feature to be connected, the variable key value log feature and the fixed key value log feature includes: if the log data to be connected is system log data with a variable key value, selecting a target historical key value that meets a preset quantity condition from the variable key value log feature; the preset data condition is that the total number of target historical key values ​​is greater than or equal to the total number of key values ​​contained in the log data to be connected; comparing each target historical key value with each key value to be connected in the log data to be connected, and if the log data to be connected contains a key value to be connected that is the same as the current target historical key value, then the current target historical key value is matched with the key value to be connected. The similarity coefficient between the log data to be connected is determined according to the weight value of the current target history key value and the preset adjustment factor; if there is no key value to be connected that is the same as the current target history key value in the log data to be connected, the similarity coefficient between the current target history key value and the log data to be connected is a preset value; the preset value is less than the minimum value determined by the weight value of each target history key value and the preset adjustment factor; if the maximum similarity coefficient of the log data to be connected is greater than the preset similarity threshold, the log parsing template of the target history key value corresponding to the maximum similarity coefficient is used to perform log connection on the log data to be connected.

[0017] On the other hand, the present application provides a log data processing device, including:

[0018] The historical data analysis module is used to determine the changing key value log characteristics and the fixed key value log characteristics by analyzing the key values ​​of the historically connected system log data.

[0019] The key-value feature extraction module is used to obtain the key-value features to be connected of the log data to be connected when detecting the existence of the log data to be connected.

[0020] The log parsing module is used to determine the log parsing method corresponding to the log system data to be connected according to the matching conditions of the key value feature to be connected, the variable key value log feature and the fixed key value log feature.

[0021] The present application also provides an electronic device, comprising a processor, wherein the processor is configured to implement the steps of the log data processing method as described in any of the preceding items when executing a computer program stored in a memory.

[0022] The present application also provides a non-volatile storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the log data processing method as described in any of the preceding items are implemented.

[0023] Finally, the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the log data processing method as described in any of the preceding items.

[0024] The advantage of the technical solution provided by the present application is that the system logs are divided into variable key value logs and fixed key value logs based on whether the key values ​​are variable, and the characteristic data of the two types of log data are extracted respectively to perform log similarity matching, which can not only effectively improve the efficiency of automatic log matching, but also improve the accuracy of automatic docking of data log data, which is conducive to timely discovery of log data that is not within the scope of the existing docking template, effectively improve the processing efficiency of such log data, and reduce data loss during the log data docking process.

[0025] In addition, the present application also provides corresponding implementation devices, electronic devices, non-volatile storage media and computer program products for the log data processing method, which further makes the method more practical, and the devices, electronic devices, non-volatile storage media and computer program products have corresponding advantages.

[0026] The technical features mentioned above, the technical features to be mentioned below, and the technical features shown separately in the drawings can be combined with each other arbitrarily, as long as the combined technical features are not contradictory. All feasible feature combinations are technical contents clearly recorded in this application. Any of the multiple sub-features contained in the same sentence can be applied independently, and does not have to be applied together with other sub-features.

[0027] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 A flow chart of a log data processing method provided by the present application;

[0030] Figure 2 A flow chart of another log data processing method provided by the present application;

[0031] Figure 3 A flowchart of another log data processing method provided in the application;

[0032] Figure 4 A structural diagram of a specific implementation of the log data processing device provided by this application;

[0033] Figure 5 A structural diagram of a specific implementation of the electronic device provided in this application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Among them, the terms "first", "second", "third", "fourth", etc. in the specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations of the two are intended to cover non-exclusive inclusions. The term "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described here as "exemplary" is not necessarily interpreted as being superior or better than other embodiments.

[0035] With the development of artificial intelligence technology, big data, cloud technology and network technology, a large amount of important data is stored in computer systems, and users will use different network security devices to provide all-round defense for their security. However, the log content of different network security devices is different, which requires users to uniformly collect and process these multi-source log data and perform standardized processing as data for security protection analysis.

[0036] At present, the log data processing process is: first classify the log data, and then use different log parsing templates to parse different types of logs to obtain structured log data. When a new log appears, the log data will be matched with the corresponding parsing template for parsing. It can be seen that the fast and accurate matching of log data can improve the efficiency and accuracy of log docking, and thus improve the efficiency and accuracy of log data processing. However, the current related technology cannot efficiently and accurately match the corresponding log parsing template for log data. For example, a related technology uses manual extraction of log feature information such as key values ​​and log length in syslog (system logging protocol, a standard protocol for computer system log recording), classifies log data based on log feature information, and parses different types of logs with different templates. However, with the increase of docking log data, a lot of manpower is needed to extract the corresponding features, and when the docking log data format is updated, it cannot be captured in time, which leads to low efficiency in log data processing and the accuracy cannot be guaranteed. Another related technology is to match the existing log parsing template by polling the original log. If the match is successful, the parsing template is used for parsing. However, as the number of log parsing templates increases, the matching time will increase accordingly, and the data docking efficiency will decrease, which will also lead to low efficiency in log data processing. There is also a related technology that groups logs based on log similarity, that is, log messages are divided into different log groups based on edit distance similarity, and different log groups use different parsing rules. This method has certain requirements on the accuracy of log grouping. When there is an error in the log grouping, the corresponding log cannot be docked, which also ensures efficient and high-precision processing of log data.

[0037] In view of this, in order to improve the processing efficiency and accuracy of log data to meet the actual needs of users, the present invention divides system logs into variable key value logs and fixed key value logs based on whether the key value is variable, and extracts the characteristic data of the two types of log data to perform log similarity matching. After introducing the technical solution of the present application, various non-limiting implementation methods of the present application are described in detail below. In order to better illustrate the present application, numerous specific details are given in the specific implementation methods below. Those skilled in the art should understand that the present application can also be implemented without these specific details. In other examples, methods, means, components and circuits well known to technicians in the relevant field are not described in detail in order to highlight the main purpose of the present application.

[0038] First see Figure 1 , Figure 1 A flow chart of a log data processing method provided by this application, which may include the following contents:

[0039] S101: Determine variable key value log features and fixed key value log features by analyzing key values ​​of historically connected system log data.

[0040] Among them, the historical docked system log data is the syslog log of the historically docked device, and it has a key value and a log parsing template. Exemplarily, the historical log data of a certain type of syslog log of a certain manufacturer or several manufacturers can be collected and summarized. In order to count the common features from the log data, the amount of log data collected is at least 100. Considering that the message content of most manufacturers in the syslog log will report the key and value of the log, and split based on the delimiter, there are also some manufacturers whose message content will not report the key value of the log, but only report the value and split with a delimiter. This part of the log is not considered within the scope of this docking data. When the historical docked system log data that meets the requirements is obtained, the key values ​​in these historical docked system log data are extracted, and the key values ​​are divided into two categories based on their characteristics: fixed key values ​​and variable key values. The feature information of the two types of key values ​​is extracted respectively, and the fixed key value log features and the variable key value log features are obtained.

[0041] S102: When it is detected that there is log data to be connected, a key-value feature to be connected of the log data to be connected is obtained.

[0042] The log data to be connected is the log data that needs to be connected. The previous step divides the logs into variable key logs and fixed key logs. In order to realize automatic log connection, it is necessary to obtain the key value features of the log data to be connected. In order to facilitate the distinction, it is defined as the key value features to be connected. Based on the key value features to be connected, it is determined whether the log data to be connected belongs to the variable key log or the fixed key log.

[0043] S103: Determine a log parsing method corresponding to the log system data to be connected according to a matching condition between the key value feature to be connected, the variable key value log feature and the fixed key value log feature.

[0044] When the previous step obtains new key-containing log data and extracts the key value features to be connected with the log data, they are matched with the fixed key value log features with fixed key values ​​and the variable key value log features with variable key values ​​in turn. If the match is successful, data connection is performed based on the log parsing template of the type that successfully matches. If the match is not successful, the key feature value and the corresponding parsing template of this type of log data are added, and then data connection is performed. In other words, for the sake of ease of description, the log parsing template of the variable key value log feature is defined as the first type of log parsing template, and the log parsing template corresponding to the fixed key value log feature is defined as the second type of log parsing template. If the key-value feature to be connected successfully matches the changing key-value log feature, the first type of log parsing template corresponding to the changing key-value log feature is used to perform log docking on the log data to be connected; if the key-value feature to be connected successfully matches the fixed key-value log feature, the second type of log parsing template corresponding to the fixed key-value log feature is used to perform log docking on the log data to be connected; if the key-value feature to be connected fails to match both the changing key-value log feature and the fixed key-value log feature, the third type of log parsing template matching the key-value feature to be connected is used for data docking, and the key-value feature to be connected is used as the third type of log feature, wherein the third round of log parsing template can be obtained manually, or it can be obtained by matching the existing log parsing module here, which does not affect the implementation of the present invention.

[0045] In the technical solution provided in the present application, the system logs are divided into variable key value logs and fixed key value logs based on whether the key values ​​are variable, and the characteristic data of the two types of log data are extracted respectively to perform log similarity matching. This can not only effectively improve the efficiency of automatic log matching, but also improve the accuracy of automatic docking of data log data, which is conducive to timely discovering log data that is not within the scope of the existing docking template, effectively improving the processing efficiency of such log data, and reducing data loss during the log data docking process.

[0046] In the above embodiment, there is no limitation on analyzing historical data. The present invention also provides an exemplary implementation method, which may include the following contents:

[0047] Obtain historical system log data of the target system log type and with the corresponding log parsing template as historical connected system log data;

[0048] A preset text string is used to replace data of corresponding types in the message body data of the historical system log data; the target separator that appears the most times is used as a segmentation symbol, and the replaced historical system log data is processed for log data segmentation, the first data of each segmentation segment is used as a key value, and repeated key values ​​corresponding to the target type data are removed from each key value to obtain key value information; if at least two key values ​​in each key value in the key value information are different, the key value information corresponding to the historical system log data is variable key value data; if each key value in the key value information is different, the key value information corresponding to the historical system log data is fixed key value data.

[0049] like Figure 2 As shown, the syslog log data set from a single source of the manufacturer is collected. Considering that the Syslog log data consists of two parts: header information and message body. The header information is a fixed format content and is relatively simple. The header data parsing is not considered during data docking, and only the message body is parsed and docked. Therefore, in order to improve the efficiency of feature extraction and reduce the complexity of subsequent log segmentation and analysis, the historical system log data can be pre-processed. The preprocessing may include removing the header structure data and / or text replacement, and the text string may be a pre-set ordinary text string. The data preprocessing process of the historical system log data may include: removing the header structure data of the historical system log data, retaining only the message body, pre-specifying the data type of the body to be replaced, such as time, IP, URL, file path, Mac address, etc., identifying the time, IP address, URL, and Mac address of the message body of the historical system log data based on regular matching and other methods, and replacing them with ordinary text strings. The target delimiter is the delimiter that appears most frequently in the historical system log data, and the delimiter in the historical system log data is extracted, such as: Then, the separator pairs with the highest number of occurrences are counted as the target separators for historical system log data. For example, in the syslog log ( <9> Jul 1723:41:20 blj logType=loginLog : {"id":62,"source_ip":"10.64.0.205","login_datetime":"1721230880","username":"sysadm","realname":"System Administrator","protocol":"https","login_type":"Static Password","result":"Success","details":"Login successful","session_id":"c32075f6-ec0b-40fb-89fb-559391738a34"}), the separators that appear most frequently are ":" and "," , the data is segmented according to: and, in each pair of segmentation results, the first one is the key value and the second one is the value value, thus forming key-value pairs, based on which the key values ​​in the historical system log data can be extracted. Furthermore, in order to improve the efficiency of subsequent data processing, some key values ​​will appear multiple times for json type data. Therefore, if there is json type data in the historical system log data, these repeated key values ​​can be deduplicated. For example, if the key value is (a, b, c, d, e, f, g, e, f, g, h), the deduplicated key is (a, b, c, d, e, f, g, h).

[0050] like Figure 2 As shown, after completing the word segmentation of the historical system log data and extracting the key value of the historical system log data, an exemplary method for identifying whether the key value belongs to a variable key value log feature or a fixed key value log feature is: calculating the hash value of each key value of the fixed key value data, and using the hash value, the total number of key values ​​contained in the fixed key value data and the separator as the fixed key value log feature; counting the occurrence probability of each key value in the variable key value data, and using each key value and the corresponding occurrence probability as the variable key value log feature.

[0051] In this embodiment, the key values ​​of the historical system log data are compared. If all the key values ​​are the same, the key values ​​of the historical system log data are fixed. If there are certain differences between the keys of the logs, the key of the historical system log data is variable. For logs with fixed key values, the number of key values, the log separator, and the hash value calculated in sequence for multiple key values ​​are used as the feature values ​​of the fixed key log, that is, the fixed key value log feature is obtained. For the features of variable key values: the probability of each key value appearing in the log can be calculated, that is, the number of key value appearances / total number of logs is calculated. For example, in 100 logs, the key1 value appears 80 times, then the probability of the key1 value appearing is 80 / 100=0.8, and [key1 (key1 probability value), ..., keyn (keyn probability value)] is used as the variable key value log feature.

[0052] Based on the above embodiment, in order to improve the efficiency and accuracy of log data docking, the present invention further stores the variable key value log features and the fixed key value log features accordingly after obtaining them, which may include the following contents:

[0053] For historical system log data with a fixed key, if the fixed key value log features of the current historical system log data are different from the fixed key value log features of other historical connected system log data, then the fixed key value log features of the current historical system log data are stored; if there is at least one target historical system log data with the same fixed key value log features as the current historical system log data, then the difference data between the target historical system log data and the current historical system log data is determined, the fixed key value log features of the current historical system log data and the difference with the target historical system log data are stored, and the difference between the target historical system log data and the current historical system log data is added to the storage data of the target historical system log data.

[0054] For historical system log data with variable keys, for each key value of the change key value log feature of the current historical system log data, count the number of times it appears in all change key value log features, and determine the weight value of each key value based on the number of appearances of each key value, the total number of change key value log features, and the probability of appearance, and store each key value and the corresponding weight value as the final fixed key value log feature.

[0055] In this embodiment, when storing the fixed key value log features of all historical system log data with fixed keys, different storage methods are used according to whether the fixed key value log features are unique. If the fixed key value log features of the key value of the historical system log data are unique and there is no duplication with the fixed key value log features of the key value of other historical system log data, the fixed key value log features of the key value of the historical system log data and its corresponding log parsing template can be directly stored. If there are two or more historical system log data with the same fixed key value log features, for the convenience of description, the current historical system log data and the target historical system log data are taken as two system log data with the same fixed key value log features, then it is necessary to extract other information of the current historical system log data and the target historical system log data for comparison and distinction, such as whether there are differences in the log header data or whether there are differences in the message body. If additional feature information for log distinction can be found, the number of key values, log separators, hash values, and additional feature information of the current historical system log data and the target historical system log data are stored respectively. If additional feature information that distinguishes the current historical system log data from the target historical system log data cannot be found, the additional feature information can be marked as a default value or a value to be filled in. Similarly, when storing fixed key value log features, the additional feature information after being marked needs to be marked as well. The difference is that since the stored additional feature information is contentless, when performing data docking, if it is a default value, a corresponding different value can be automatically assigned. If it is a value to be filled in, the corresponding data can be filled in with the value through manual intervention. For example, the source manufacturer and source IP of the historical system log data can be used as additional feature information.

[0056] In this embodiment, when storing the changed key value log features of all historical system log data with variable keys, in order to better distinguish each historical system log data, a weight value that can represent the historical system log data can be stored. The weight value of the historical system log data is calculated based on the number of occurrences of each key value, the total number of changed key value log features, and the probability of occurrence. According to the method for determining the weight value, it can be seen that the larger the weight value of the historical system log data with a variable key, the more the key can represent the historical system log data. For example, if it is known that there are 20 groups of historical system log data containing variable keys, and the key change key value log feature of one group of historical system log data, that is, the current historical system log data, is [key1(1), key2(1), key3(0.8), key4(0.6), key5(0.4)], and the number of times key1, ..., key5 appear in the 20 groups of log data is (10, 6, 7, 1, 2), then the weight values ​​of each key value of the current historical system log data are: Qkey1=1*(20 / 10)=2, Qkey2=1*(20 / 6)=3.3, Qkey3=0.8*(20 / 7)=2.3, Qkey4=0.6*(20 / 1)=12, Qkey5=0.4*(20 / 2)=4. After the above-mentioned conversion of the weight values, the change key value log feature of the key value of the current historical system log data is [key1(2), key2(3.3), key3(2.3), key4(12), key5(4)], which is stored as the change key value log feature of the key value of the change key value log feature.

[0057] As can be seen from the above, this embodiment uses different methods to store historical system log data with fixed keys and variable keys, which is beneficial to improving the matching efficiency of log templates, thereby improving the efficiency and accuracy of log data docking.

[0058] The above embodiment does not limit how to match the key value feature of the docking log data with the variable key value log feature and the fixed key value log feature. Based on the above embodiment, the present invention also provides an exemplary matching method, which may include the following content:

[0059] After obtaining the log data to be connected, the features of the key value corresponding to the log data to be connected can be obtained in the same way as the features of the key of the historically connected system log, and defined as the key value features to be connected. Exemplarily, the log data to be connected can be subjected to message text extraction and content replacement of the message text, and then word segmentation processing is performed. After the word segmentation is completed, the separator and each key value are obtained, and the total number of each key value is counted. For the convenience of description, it is defined as the separator to be connected and the total number of key values ​​to be connected. The hash value of each key value is calculated in sequence. For the convenience of description, it is defined as the hash value to be connected. In this way, the key value features to be connected include the hash value to be connected, the separator to be connected, and the total number of key values ​​to be connected. After obtaining the hash value to be connected, the separator to be connected, and the total number of key values ​​to be connected, the key value features to be connected of the log data to be connected can be matched with the variable key value log feature and the fixed key value log feature respectively:

[0060] The matching process of the key-value feature to be connected and the fixed-key-value log feature: determine the log category to which the log data to be connected belongs according to the delimiter to be connected and the total number of key values ​​to be connected; if there is target historical connected system log data with the same hash value as the hash value to be connected, then the log data to be connected is the system log data with a fixed key value; if the matched target historical connected system log data is unique, the log data to be connected is log-connected based on the log parsing template of the target historical connected system log data; if the matched target historical connected system log data is not unique, compare the difference between the log data to be connected and each target historical connected system log, and log-connect the log data to be connected using the log parsing template of the first historical system log data that successfully matches the difference.

[0061] In this embodiment, if Figure 3As shown, the logs can be classified according to the length of the log key and the separator, and then the hash value of the key under the category can be matched. If a log with the same hash value can be matched, then this type of log is a syslog log with a fixed key. If there is only one historical log data with the same hash value to be connected in the key value feature to be connected in each historical log data, it is defined as the target historical connected system log data, and the log parsing template of the target historical connected system log data can be directly used to perform log docking processing on the log data to be connected. If there are multiple historical log data with the same hash value to be connected in the key value feature to be connected in each historical log data, first check whether it has corresponding additional feature information. If there is additional feature information, and the log can match the additional feature information, for the sake of convenience of description, the historical system log data that successfully matches the hash value and the additional feature information at the same time is defined as the first historical system log data, and then matching is performed based on the log parsing template of the first historical system log data. If there is additional feature information, but it cannot be matched with the existing additional feature information from the log data of each target historical system, it is necessary to analyze the log data to be connected separately, extract the feature information and parsing template information of this type of log to match the log data to be connected. When there are multiple logs with the same docking log key value and no additional information, manual intervention can be used to manually match the corresponding log parsing template based on the reporting device type, reporting device IP, and data source registration information, and store the manually reported information as additional information in the fixed key value log feature of the log data to be connected.

[0062] The matching process of the key value feature to be connected and the key value log feature to be changed is as follows: if the log data to be connected is system log data with a changed key value, then select the target historical key value that meets the preset quantity condition from the changed key value log feature; the preset data condition is that the total number of target historical key values ​​is greater than or equal to the total number of key values ​​contained in the log data to be connected; compare each target historical key value with each key value to be connected in the log data to be connected, and if the log data to be connected contains the same key value to be connected as the current target historical key value, then the match between the current target historical key value and the log data to be connected is compared. The similarity coefficient is determined based on the weight value of the current target history key value and a preset adjustment factor; if there is no key value to be connected that is the same as the current target history key value in the log data to be connected, the similarity coefficient between the current target history key value and the log data to be connected is a preset value; the preset value is less than the minimum value determined by the weight value of each target history key value and the preset adjustment factor; if the maximum similarity coefficient of the log data to be connected is greater than a preset similarity threshold, the log parsing template of the target history key value corresponding to the maximum similarity coefficient is used to perform log connection on the log data to be connected.

[0063] In this embodiment, the preset condition factor and the preset value can be flexibly selected according to the actual situation. This embodiment takes the preset condition factor as 1 and the preset value as 0 as an example to illustrate the whole process. First, the log features whose number of key values ​​of the variable key ≥ the number of key values ​​of the log data to be connected are screened and matched with the current log, and the similarity between the log data to be connected and the variable key log is calculated. As a convenient similarity calculation method, under the premise that the preset condition factor is 1 and the preset value is 0, this embodiment can be calculated by the following relationship (1):

[0064] ;

[0065] .

[0066] Among them, equation (2) is an indicator function. When the log data to be connected does not exist When , it returns 0, otherwise it returns 1. The above formula (1) is the similarity value between the key to be connected to the log data to be connected and the change key value log feature counted by S101, where for The weight value of the value is the weight value of the same key between the log data to be connected and the variable key log as the similarity coefficient between the two log data. The similarity coefficient values ​​between the log data to be connected and each variable key log are counted to obtain the maximum similarity coefficient Smax. If the maximum similarity coefficient ,in is the minimum threshold. When it is calculated by the existing docking log data and taken as 0.8, it has a better effect. If docking other log data, this value can also be adjusted in combination with the specific docking data. If there is a difference between the log data to be docked and the existing log data analyzed by S101, it is necessary to re-extract the key value of the log data to be docked and construct its parsing template. If the maximum similarity coefficient , there is a large similarity between the log data to be connected and the existing log data analyzed by S101, and the log parsing template of the existing data analyzed by S101 can be used for connection. The parsing template for connection is the parsing template of the variable key log corresponding to the maximum similarity coefficient.

[0067] As can be seen from the above, this embodiment extracts different features for system log data with fixed keys and variable keys to perform log similarity calculation and docking, so as to improve the efficiency and accuracy of data docking.

[0068] It should be noted that there is no strict order of execution between the steps in this application. As long as they comply with the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 1-Figure 3This is just a schematic and does not mean that this is the only execution order.

[0069] The present application also provides a corresponding device for the log data processing method, which further makes the method more practical. Among them, the device can be described from the perspective of functional modules and hardware. The log data processing device provided by the present application is introduced below. The device is used to implement the log data processing method provided by the present application. In this embodiment, the log data processing device may include or be divided into one or more program modules, and the one or more program modules are stored in a storage medium and executed by one or more processors to complete the log data processing method disclosed in Example 1. The program module referred to in this application refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the execution process of the log data processing device in the storage medium than the program itself. The following description will specifically introduce the functions of each program module of this embodiment. The log data processing device described below and the log data processing method described above can be referenced to each other.

[0070] From the perspective of functional modules, see Figure 4 , Figure 4 This is a structural diagram of a log data processing device provided by the present application in a specific implementation manner. The device may include:

[0071] The historical data analysis module 401 is used to determine the variable key value log features and the fixed key value log features by analyzing the key values ​​of the historically connected system log data.

[0072] The key-value feature extraction module 402 is used to obtain the key-value features of the log data to be connected when detecting that there is log data to be connected.

[0073] The log parsing module 403 is used to determine the log parsing method corresponding to the log system data to be connected according to the matching conditions of the key value feature to be connected, the variable key value log feature and the fixed key value log feature.

[0074] Optionally, in some implementation schemes of the present embodiment, the above-mentioned log parsing module 403 can be further used for: if the key-value feature to be docked matches the changing key-value log feature successfully, then the first type of log parsing template corresponding to the changing key-value log feature is used to perform log docking on the log data to be docked; if the key-value feature to be docked matches the fixed key-value log feature successfully, then the second type of log parsing template corresponding to the fixed key-value log feature is used to perform log docking on the log data to be docked; if the key-value feature to be docked fails to match both the changing key-value log feature and the fixed key-value log feature, then the third type of log parsing template matching the key-value feature to be docked is used to perform data docking, and the key-value feature to be docked is used as the third type of log feature.

[0075] Optionally, in some other implementations of the present embodiment, the historical data analysis module 401 can be further used to: obtain historical system log data of the target system log type and having a corresponding log parsing template as historical connected system log data; use a preset text string to replace data of a corresponding type in the message body data of the historical system log data; use the target separator that appears the most times as a segmentation symbol, perform log data segmentation processing on the replaced historical system log data, use the first data of each segmentation segment as a key value, and remove duplicate key values ​​corresponding to the target type data from each key value to obtain key value information; if at least two of the key values ​​in the key value information are different, then the key value information corresponding to the historical system log data is variable key value data; if all the key values ​​in the key value information are different, then the key value information corresponding to the historical system log data is fixed key value data.

[0076] As an exemplary implementation of the above embodiment, the above historical data analysis module 401 can also be further used to: calculate the hash value of each key value of the fixed key value data, and use the hash value, the total number of key values ​​contained in the fixed key value data and the separator as fixed key value log features; count the occurrence probability of each key value in the variable key value data, and use each key value and the corresponding occurrence probability as variable key value log features.

[0077] Optionally, in some other implementations of the present embodiment, the above-mentioned device may also include a feature storage module, which can be used to: if the fixed key value log feature of the current historical system log data is different from the fixed key value log features of other historical connected system log data, then store the fixed key value log feature of the current historical system log data; if there is at least one target historical system log data whose fixed key value log feature is the same as the fixed key value log feature of the current historical system log data, then determine the difference data between the target historical system log data and the current historical system log data, store the fixed key value log feature of the current historical system log data and the difference between it and the target historical system log data, and add the difference between it and the current historical system log data to the storage data of the target historical system log data.

[0078] Optionally, in some other implementations of the present embodiment, the above-mentioned device may also include a feature storage module, which may be further used to: count the number of times each key value of the change key value log feature of the current historical system log data appears in all change key value log features, and determine the weight value of each key value according to the number of occurrences of each key value, the total number of change key value log features, and the probability of occurrence, and store each key value and the corresponding weight value as the final fixed key value log feature.

[0079] Optionally, in some other implementations of the present embodiment, the above-mentioned log parsing module 403 can be further used to: determine the log category to which the log data to be docked belongs based on the delimiter to be docked and the total number of key values ​​to be docked; if there is target historical docked system log data with the same hash value as the hash value to be docked, then the log data to be docked is system log data with a fixed key value; if the matched target historical docked system log data is unique, then log docking is performed on the log data to be docked based on the log parsing template of the target historical docked system log data; if the matched target historical docked system log data is not unique, then the difference content between the log data to be docked and each target historical docked system log is compared, and the log parsing template of the first historical system log data that successfully matches the difference content is used to log dock the log data to be docked.

[0080] Optionally, in some other implementations of the present embodiment, the above-mentioned log parsing module 403 can be further used for: if the log data to be connected is system log data with a changed key value, then selecting a target historical key value that meets a preset quantity condition from the changed key value log feature; the preset data condition is that the total number of target historical key values ​​is greater than or equal to the total number of key values ​​contained in the log data to be connected; comparing each target historical key value with each key value to be connected in the log data to be connected, and if there is a key value to be connected that is the same as the current target historical key value in the log data to be connected, then the current target historical key value is compared with the key value to be connected. The similarity coefficient between log data is determined based on the weight value of the current target history key value and a preset adjustment factor; if there is no key value to be connected that is the same as the current target history key value in the log data to be connected, the similarity coefficient between the current target history key value and the log data to be connected is a preset value; the preset value is less than the minimum value determined by the weight value of each target history key value and the preset adjustment factor; if the maximum similarity coefficient of the log data to be connected is greater than a preset similarity threshold, the log parsing template of the target history key value corresponding to the maximum similarity coefficient is used to perform log connection on the log data to be connected.

[0081] The log data processing device mentioned above is described from the perspective of functional modules. Furthermore, the present application also provides an electronic device, which is described from the perspective of hardware. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application in one implementation manner. Figure 5 As shown, the electronic device includes a memory 50 for storing computer programs; a processor 51 for implementing the steps of the log data processing method mentioned in any of the above embodiments when executing the computer program.

[0082] Among them, the processor 51 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 51 may also be a controller, a microcontroller, a microprocessor or other data processing chip. The processor 51 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 51 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 51 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 51 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0083] The memory 50 may include one or more computer-readable storage media, which may be non-transitory. The memory 50 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the memory 50 may be an internal storage unit of an electronic device, such as a hard disk of a server. In other embodiments, the memory 50 may also be an external storage device of an electronic device, such as a plug-in hard disk equipped on a server, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Further, the memory 50 may also include both an internal storage unit of an electronic device and an external storage device. The memory 50 may not only be used to store application software and various types of data installed in the electronic device, such as: the code of the program in the process of executing the log data processing method, but also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 50 is at least used to store the following computer program 501, wherein, after the computer program is loaded and executed by the processor 51, the relevant steps of the log data processing method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 50 may also include an operating system 502 and data 503, etc., and the storage method may be temporary storage or permanent storage. The operating system 502 may include Windows, Unix, Linux, etc. The data 503 may include but is not limited to data corresponding to the log data processing result, etc.

[0084] In some embodiments, the electronic device may further include a display screen 52, an input / output interface 53, a communication interface 54 or a network interface, a power supply 55 and a communication bus 56. Among them, the display screen 52 and the input / output interface 53 such as a keyboard belong to the user interface, and the optional user interface may also include a standard wired interface, a wireless interface, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The communication interface 54 may optionally include a wired interface and / or a wireless interface, such as a WI-FI interface, a Bluetooth interface, etc., which is usually used to establish a communication connection between the electronic device and other electronic devices. The communication bus 56 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0085] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than those shown in the figure, for example, it may also include a sensor 57 for realizing various functions.

[0086] It is understandable that if the log data processing method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes but is not limited to: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes. Based on this, the present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of log data processing recorded in any of the above embodiments are performed.

[0087] It is understandable that if the log data processing method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, the computer software product does not need to be stored in a physical storage medium. For example, it can be directly transmitted to a computer or other device with information processing capabilities through a wired network or a wireless network to perform all or part of the steps of the methods of each embodiment of the present invention. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product. Based on this, the present invention also provides a computer program product that stores a computer program. When the computer program is executed by a processor, the steps of log data processing recorded in any of the above embodiments are performed.

[0088] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the hardware disclosed in the embodiments, including devices and electronic devices, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0089] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0090] The above is a detailed introduction to a log data processing method, device, electronic device, non-volatile storage medium and computer program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that based on the embodiments in the present application, for ordinary technicians in this technical field, all other embodiments obtained without creative work are within the scope of protection of the present application. Without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A log data processing method, characterized in that: include: By analyzing the key values ​​of the historically connected system log data, the characteristics of the variable key value logs and the fixed key value logs are determined; When it is detected that there is log data to be connected, the key value feature to be connected of the log data to be connected is obtained; The log parsing method corresponding to the log system data to be connected is determined according to the matching conditions of the key value feature to be connected, the variable key value log feature and the fixed key value log feature.

2. The log data processing method according to claim 1, characterized in that: The determining, according to the matching of the key value feature to be connected, the variable key value log feature and the fixed key value log feature, a log parsing method corresponding to the log data to be connected includes: If the key value feature to be connected successfully matches the changed key value log feature, the first type of log parsing template corresponding to the changed key value log feature is used to perform log connection on the log data to be connected; If the key-value feature to be connected successfully matches the fixed key-value log feature, the second type of log parsing template corresponding to the fixed key-value log feature is used to perform log connection on the log data to be connected; If the key value feature to be connected fails to match the variable key value log feature and the fixed key value log feature, a third type of log parsing template matching the key value feature to be connected is used for data connection, and the key value feature to be connected is used as the third type of log feature.

3. The log data processing method according to claim 1, characterized in that: The step of analyzing the key values ​​of the historically connected system log data to determine the variable key value log features and the fixed key value log features includes: Obtain historical system log data of the target system log type and with the corresponding log parsing template as historical connected system log data; Using a preset text string to replace data of a corresponding type in the message body data of the historical system log data; The target separator with the most occurrences is used as a segmentation symbol, and the replaced historical system log data is segmented, and the first data of each segmentation segment is used as a key value, and the repeated key values ​​corresponding to the target type data are removed from each key value to obtain key value information; If at least two of the key values ​​in the key value information are different, the key value information corresponding to the historical system log data is variable key value data; if all the key values ​​in the key value information are different, the key value information corresponding to the historical system log data is fixed key value data.

4. The log data processing method according to claim 3, characterized in that: The determining of the variable key value log features and the fixed key value log features includes: Calculate the hash value of each key value of the fixed key value data, and use the hash value, the total number of key values ​​included in the fixed key value data and a separator as a fixed key value log feature; The occurrence probability of each key value in the change key value data is counted, and each key value and the corresponding occurrence probability are used as the change key value log feature.

5. The log data processing method according to claim 1, characterized in that: After determining the variable key value log features and the fixed key value log features, the method further includes: If the fixed key value log feature of the current historical system log data is different from the fixed key value log features of other historical connected system log data, then the fixed key value log feature of the current historical system log data is stored; If there is at least one target historical system log data whose fixed key value log feature is the same as the fixed key value log feature of the current historical system log data, the difference data between the target historical system log data and the current historical system log data is determined, the fixed key value log feature of the current historical system log data and the difference with the target historical system log data are stored, and the difference between the target historical system log data and the current historical system log data is added to the storage data of the target historical system log data.

6. The log data processing method according to claim 1, characterized in that: After determining the variable key value log features and the fixed key value log features, the method further includes: For each key value of the change key value log feature of the current historical system log data, count the number of times it appears in all change key value log features, and determine the weight value of each key value based on the number of appearances of each key value, the total number of change key value log features, and the probability of occurrence, and store each key value and the corresponding weight value as the final fixed key value log feature.

7. The log data processing method according to any one of claims 1 to 6, characterized in that: The key value feature to be connected includes a hash value to be connected, a separator to be connected and a total number of key values ​​to be connected. The matching conditions according to the key value feature to be connected, the variable key value log feature and the fixed key value log feature include: Determine the log category to which the log data to be connected belongs according to the delimiter to be connected and the total number of key values ​​to be connected; If there is target historical docked system log data with the same hash value as the hash value to be docked, the log data to be docked is system log data with a fixed key value; If the matched target historical docked system log data is unique, log docking is performed on the log data to be docked based on the log parsing template of the target historical docked system log data; If the matched target historical system log data is not unique, the difference between the log data to be connected and each target historical system log is compared, and the log parsing template of the first historical system log data that successfully matches the difference is used to log connect the log data to be connected.

8. The log data processing method according to any one of claims 1 to 6, characterized in that: The matching conditions according to the key value feature to be connected, the variable key value log feature and the fixed key value log feature include: If the log data to be connected is system log data with a changed key value, select a target historical key value that meets a preset quantity condition from the changed key value log feature; the preset data condition is that the total number of target historical key values ​​is greater than or equal to the total number of key values ​​contained in the log data to be connected; Compare each target history key value with each to-be-connected key value of the to-be-connected log data respectively; if the to-be-connected log data contains a to-be-connected key value identical to the current target history key value, then the similarity coefficient between the current target history key value and the to-be-connected log data is determined according to the weight value of the current target history key value and a preset adjustment factor; if the to-be-connected log data does not contain a to-be-connected key value identical to the current target history key value, then the similarity coefficient between the current target history key value and the to-be-connected log data is a preset value; and the preset value is less than the minimum value determined by the weight value of each target history key value and the preset adjustment factor; If the maximum similarity coefficient of the log data to be connected is greater than a preset similarity threshold, log connection is performed on the log data to be connected using a log parsing template of a target historical key value corresponding to the maximum similarity coefficient.

9. A log data processing device, characterized in that: include: A historical data analysis module is used to determine the characteristics of variable key value logs and fixed key value logs by analyzing the key values ​​of historically connected system log data; A key-value feature extraction module is used to obtain the key-value features of the log data to be connected when detecting the existence of the log data to be connected; The log parsing module is used to determine the log parsing method corresponding to the log system data to be connected according to the matching conditions of the key value feature to be connected, the variable key value log feature and the fixed key value log feature.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is used to implement the steps of the log data processing method according to any one of claims 1 to 8 when executing a computer program stored in the memory.

11. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the log data processing method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the log data processing method according to any one of claims 1 to 8 are implemented.

Images