Log structured processing method and device, equipment and medium

Abstract

The embodiment of the invention relates to a log structured processing method and device, equipment and a medium, and the method comprises the steps: determining a sample log matched with a current log in a plurality of historical logs, and obtaining a sample regular expression corresponding to the sample log; inputting the current log, the sample log, the sample regular expression and the generated prompt word into an expression generation model, and outputting to obtain a current regular expression corresponding to the current log; performing field data extraction on the associated log through the current regular expression to obtain target structured data of the associated log; wherein the associated log is the same as the current log in type. According to the embodiment of the invention, the regular expression of the current log is automatically and accurately generated by referring to the sample log and the sample regular expression through the model, the development efficiency is improved, the log of the same type as the current log is subjected to log analysis processing through the regular expression, and the log analysis and maintenance cost is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a log structured processing method, device, equipment and medium. Background Art

[0002] As the amount of log data increases, the demand for automatic log parsing becomes stronger. In the related art, regular expressions are manually written and used to automatically parse logs. However, the logs generated by different platforms and devices are in different formats, and the required regular expressions are also different, making it difficult to achieve automatic log parsing. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a log structured processing method, device, equipment and medium.

[0004] The present disclosure provides a log structured processing method, including:

[0005] Determine a sample log that matches the current log among multiple historical logs, and obtain a sample regular expression corresponding to the sample log;

[0006] Input the current log, the sample log, the sample regular expression, and the generation prompt word into the expression generation model, and output a current regular expression corresponding to the current log;

[0007] Field data of the associated log is extracted using the current regular expression to obtain target structured data of the associated log; wherein the associated log is a log of the same type as the current log.

[0008] In some embodiments of the present disclosure, the multiple historical logs correspond to multiple historical structure information. Accordingly, determining a sample log matching the current log among the multiple historical logs includes:

[0009] Extracting field data from the current log according to preset fields to obtain current structure information of the current log;

[0010] Among the plurality of historical structure information, the historical log corresponding to the historical structure information matching the current structure information is determined as the sample log.

[0011] In some embodiments of the present disclosure, the historical structure information includes the historical value corresponding to the preset field, and the current structure information includes the current value corresponding to the preset field;

[0012] Accordingly, determining, among the plurality of historical structure information, the historical log corresponding to the historical structure information matching the current structure information as the sample log includes:

[0013] For each piece of the historical structure information, determine the similarity between the historical value corresponding to each of the preset fields in the historical structure information and the current value corresponding to each of the preset fields in the current structure information, and determine the information matching value between the historical structure information and the current structure information according to the similarity;

[0014] Determining sample structure information in the plurality of historical structure information according to the information matching value;

[0015] The historical log corresponding to the sample structure information is determined as the sample log.

[0016] In some embodiments of the present disclosure, the generation prompt word is used to guide the expression generation model to use the sample regular expression as the regular expression corresponding to the current log when the sample regular expression is applicable to the current log, and to generate a regular expression corresponding to the current log with reference to the sample log and the sample regular expression when the sample regular expression is not applicable to the current log.

[0017] In some embodiments of the present disclosure, the step of inputting the current log, the sample log, the sample regular expression, and the generation prompt word into an expression generation model and outputting a current regular expression corresponding to the current log includes:

[0018] Inputting the current log, the sample log, the sample regular expression, and the generation prompt word into an expression generation model to obtain an initial regular expression output by the expression generation model;

[0019] Determine the initial regular expression as the regular expression to be processed, and determine the field ratio; wherein the field ratio is the ratio of the matching field in the log field of the current log, and the matching field is the field where the regular expression to be processed matches the current log;

[0020] If the field proportion is not greater than the proportion threshold, the modified prompt word is input into the expression generation model to obtain the intermediate regular expression output by the expression generation model, and the intermediate regular expression is used as a new regular expression to be processed. The new field proportion corresponding to the new intermediate regular expression is returned to determine until the new field proportion is greater than the proportion threshold, and the new intermediate regular expression is determined as the current regular expression.

[0021] In some embodiments of the present disclosure, determining the field proportion includes:

[0022] The proportion calculation prompt word is input into the expression generation model to obtain the field proportion output by the expression generation model.

[0023] In some embodiments of the present disclosure, the modification prompt word is used to guide the expression generation model to determine unmatched fields and modify the regular expression to be processed according to the unmatched fields.

[0024] The present disclosure also provides a log structured processing device, including:

[0025] A determination module, used to determine a sample log matching the current log among multiple historical logs, and obtain a sample regular expression corresponding to the sample log;

[0026] An input module, used for inputting the current log, the sample log, the sample regular expression, and the generation prompt word into an expression generation model, and outputting a current regular expression corresponding to the current log;

[0027] The extraction module is used to extract field data of the associated log by using the current regular expression to obtain target structured data of the associated log; wherein the associated log is a log of the same type as the current log.

[0028] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the log structured processing method provided by the embodiment of the present disclosure.

[0029] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the log structured processing method provided by the embodiment of the present disclosure.

[0030] The technical solution provided by the embodiments of the present disclosure has the following advantages over the prior art: the log structured processing solution provided in the embodiments of the present disclosure, the method comprising: determining a sample log matching the current log among multiple historical logs, and obtaining a sample regular expression corresponding to the sample log; inputting the current log, the sample log, the sample regular expression, and the generation prompt word into an expression generation model, and outputting a current regular expression corresponding to the current log; extracting field data from the associated log using the current regular expression to obtain target structured data of the associated log; wherein the associated log is a log of the same type as the current log. By adopting the above technical scheme, it is possible to dynamically determine the corresponding sample log and sample regular expression according to different current logs, and generate a current regular expression suitable for the current log by referring to the sample log and sample regular expression through an expression generation model under the guidance of a generation prompt word, and use the current regular expression to perform structured processing on associated logs of the same type as the current log, breaking through the limitations of manually writing regular expressions in related technologies, and realizing automatic and accurate generation of the regular expression of the current log by referring to the sample log and sample regular expression through a model, thereby improving development efficiency, and using the regular expression to perform log parsing processing on logs of the same type as the current log, thereby reducing the cost of log parsing and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0033] Figure 1 A flow chart of a log structured processing method provided by an embodiment of the present disclosure;

[0034] Figure 2 A schematic diagram of a flow chart of another log structured processing method provided in an embodiment of the present invention;

[0035] Figure 3 A flowchart of another log structured processing method provided by an embodiment of the present disclosure;

[0036] Figure 4 A schematic diagram of a log structured processing method provided by an embodiment of the present disclosure;

[0037] Figure 5A schematic diagram of the structure of a log structured processing device provided by an embodiment of the present disclosure;

[0038] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0041] In traditional log parsing technology, great technical progress has been made in many aspects such as automated parsing and large-scale log processing. In related technologies, regular expressions are manually written and used to automatically parse logs. However, as the amount of log data increases, traditional log parsing technology also has some shortcomings. Specifically, the shortcomings include the limitations of regular expressions, the difficulty of parsing unstructured logs, and the limitations of regular expression rules.

[0042] Regarding the limitations of regular expressions: First, the labor cost is high. Specifically, the writing and maintenance of regular expressions requires professional skills and consumes a lot of human time. It is difficult and error-prone to manually write and maintain regular expressions for large and diverse log data. Second, regular expressions have poor scalability. Specifically, regular expressions are effective for logs with fixed formats, but when the log format changes or the log volume surges, regular expressions have poor scalability. In addition, complex regular expressions may cause the performance of parsing logs to degrade when parsing complex logs. Third, maintenance is difficult. Specifically, the log formats of different security devices and applications are different, and manually maintaining multiple regular expression sets is very cumbersome, especially in large environments.

[0043] Regarding the difficulty in parsing unstructured logs: First, logs have a variety of formats. Specifically, in actual applications, logs have various formats, including plain text, semi-structured data, etc. When processing unstructured logs, it is difficult to automatically parse complex contexts and nested information and then generate corresponding regular expressions in related technologies. Second, it is difficult to parse logs across platforms. Specifically, as the complexity of the system increases, the differences between logs generated by different platforms and devices become more and more obvious. In related technologies, it is difficult to parse logs across platforms.

[0044] Regarding the limitations of regular expression rules: First, the rules are not flexible enough. Specifically, the determination of regular expression rules depends on a predefined rule base. This method of determining rules is suitable for processing logs of known forms, but lacks the ability to adapt to newly emerging log forms or unknown attack patterns. The log form may include a log format. Second, the update cycle is long. Specifically, when the log form of the system or software changes, the rules need to be manually updated. The update cycle is long, and it is difficult to respond to dynamic security environments in real time. Third, there is an over-reliance on manual labor. Specifically, the rule base usually needs to be manually updated based on experience, which poses a high risk of human error and subjectivity.

[0045] It can be seen that it is difficult to achieve automatic parsing of logs due to the inconsistency of log formats generated by different platforms and devices.

[0046] In order to solve the above problems, the embodiments of the present disclosure provide a log structured processing method, which is introduced below in conjunction with specific embodiments.

[0047] Figure 1 The present invention provides a flow chart of a log structured processing method provided by an embodiment of the present invention. The log structured processing method can be applied to a log structured processing device. The log structured processing device can be implemented by software and / or hardware, and the log structured processing device can generally be integrated in an electronic device. Figure 1 As shown, the log structured processing method includes:

[0048] Step 101: Determine a sample log that matches the current log among multiple historical logs, and obtain a sample regular expression corresponding to the sample log.

[0049] Among them, the historical log can be an unstructured log corresponding to a historical regular expression. This embodiment does not limit the generation time of the historical log. For example, the historical log can be a log generated before the current log. This embodiment does not limit the source of the historical log. For example, the historical log and the current log can be logs generated by different systems. A regular expression (regex) is a pattern for matching strings. This embodiment does not limit the composition of the regular expression. For example, a regular expression can include ordinary characters and special characters. Ordinary characters can include letters, numbers, etc. Special characters can include metacharacters, etc., and matching rules and conditions can be defined by the metacharacters. The historical regular expression can be a regular expression for structured processing of the historical log. Through structured processing, key information in the log can be extracted to facilitate subsequent data storage, query, analysis, visualization, etc.

[0050] The current log may be an unstructured log that is currently to be structured. The current log may be a log that does not currently have a regular expression or does not match a regular expression currently available. This embodiment does not limit the source of the current log. For example, the current log may be a log generated by a security system such as a firewall, an intrusion detection system (IDS), or an intrusion prevention system (IPS).

[0051] The sample log may be a log having text similar to the current log. For example, the sample log may have a text similarity with the current log that is greater than a similarity threshold. The sample log may be understood as a log referenced by the regular expression for generating the current log. This embodiment does not limit the number of sample logs. For example, the number of sample logs may be one or more. The sample regular expression may be a regular expression for structured processing of the sample log. The sample regular expression may be a historical regular expression corresponding to the sample log. The sample regular expression may be understood as a regular expression referenced by the regular expression for generating the current log.

[0052] In the disclosed embodiment, the log structured processing device can obtain the current log, multiple historical logs, and the historical regular expression corresponding to each historical log. The log structured processing device can match each historical log with the current log based on text similarity, and determine one or more historical logs that best match the current log as sample logs, and determine the historical regular expression corresponding to the sample log as the sample regular expression.

[0053] In some embodiments of the present disclosure, the log structured processing method further includes: collecting logs from a preset system using a log collection tool to obtain a current log.

[0054] Among them, the preset system can be a system that is pre-set to generate logs, and this embodiment does not limit the type of the preset system. For example, the preset system can be a security system. The log collection tool can be a tool for collecting logs generated by the preset system, and this embodiment does not limit the log collection tool. For example, the log collection tool can include tools written using the following protocols or interfaces: system log (syslog) protocol, Kafka (kafka) protocol, Simple Network Management Protocol (SNMP, Simple Network Management Protocol), Open Database Connectivity (Open Database Connectivity, ODBC) interface, Java Database Connectivity (Java Database Connectivity, JDBC) interface, File Transfer Protocol (File Transfer Protocol, ftp).

[0055] Wherein, the system log protocol is a standard protocol for sending system log messages, which is used for logging of network devices, servers and other computer systems. The Kafka protocol is an efficient distributed messaging protocol for the transmission and processing of real-time data streams. The simple network management protocol may include a simple network management protocol trap (SNMPTrap) notification mechanism, which is used to actively send alarm information to the management system when a specific event occurs in a network device or system. The open database connection interface is an application programming interface (API) for accessing a database. Through this interface, different programming languages, applications, etc. can be accessed in a unified manner to various database management systems (Database Management System, DBMS). The Java database connection interface is used to access a database in the Java programming language, and this interface can provide a standard way to enable Java applications to interact with various database management systems. The file transfer protocol can be used to transfer files on the network.

[0056] In this embodiment, the log structured processing device can collect original logs generated by multiple preset systems through a log collection tool, and use the original logs as sample logs.

[0057] Figure 2 FIG. 1 is a flow chart of another log structured processing method provided in the present embodiment. Figure 2As shown, in some embodiments of the present disclosure, multiple historical logs correspond to multiple historical structure information, and the historical structure information can be used to characterize the text structure of the historical log. The historical structure information can be information obtained by performing structured processing on part of the historical log.

[0058] Accordingly, a sample log matching the current log is determined among multiple historical logs, including:

[0059] Step 201 : extract field data of the current log according to preset fields to obtain current structure information of the current log.

[0060] Among them, the preset fields can be pre-set fields based on which field data extraction of the log is performed. The number of the preset fields is less than the number of all fields of the log, and this embodiment does not limit the preset fields. For example, the preset fields may include one or more of the start time, end time, Internet Protocol (IP) address of the attack source, attack source port, attacked asset, file hash value, and file permissions. The current structural information can be used to characterize the text structure of the current log. The current structural information can be information obtained by structured processing of a portion of the current log. The current structural information can be understood as the structured result of a portion of the current log. The field data can be the specific value corresponding to the field in the log.

[0061] In this embodiment, the log structuring device can perform standardized processing on the current log according to the preset field. Specifically, the log structuring device can extract the field data corresponding to the preset field from the current log according to the preset field, and associate the preset field and its corresponding field data to obtain the current structure information of the current log.

[0062] Step 202: among multiple pieces of historical structure information, determine the historical log corresponding to the historical structure information matching the current structure information as the sample log.

[0063] In this embodiment, the log sample library stores multiple historical logs and historical structure information and historical regular expressions corresponding to the historical logs. The log structuring processing device can determine the historical structure information whose text similarity with the current structure information is greater than the similarity threshold among the multiple historical structure information stored in the log sample library, and determine the historical structure information as sample structure information, determine the historical log corresponding to the sample structure information as the sample log, and determine the historical regular expression corresponding to the sample structure information as the sample regular expression.

[0064] In some embodiments of the present disclosure, the historical structure information includes the historical value corresponding to the preset field, and the historical value can be understood as the field data corresponding to the preset field in the historical structure information. The current structure information includes the current value corresponding to the preset field, and the current value can be understood as the field data corresponding to the preset field in the current structure information.

[0065] In some embodiments of the present disclosure, determining, among multiple pieces of historical structure information, a historical log corresponding to historical structure information matching current structure information as a sample log includes:

[0066] For each historical structure information, determine the similarity between the historical value corresponding to each preset field in the historical structure information and the current value corresponding to the current structure information, and determine the information matching value of the historical structure information and the current structure information based on the similarity; determine the sample structure information in multiple historical structure information based on the information matching value; and determine the historical log corresponding to the sample structure information as the sample log.

[0067] The similarity may be the similarity between different vectors converted from different texts. The information matching value may represent the text similarity between the historical structure information and the current structure information.

[0068] In this embodiment, for each historical structure information, the log structure processing device can determine the historical value corresponding to each preset field in the historical structure information, and the current value corresponding to each preset field in the current structure information. Further, the historical value and the current value corresponding to the same preset field are similarity calculated through text vectorization to obtain multiple similarities corresponding to multiple preset fields. Based on the multiple similarities, arithmetic operations such as average value or summation are performed to obtain the information matching value of the historical structure information and the current structure information.

[0069] Further, the sample structure information is determined according to the information matching value. There are multiple methods for determining the sample structure information, which are not limited in this embodiment. For example, the log structure processing device can determine the historical structure information corresponding to the largest first N information matching values ​​among multiple information matching values ​​as the sample structure information, where N is a positive integer. Alternatively, the log structure processing device can determine the historical structure information whose information matching value is not less than the matching value threshold as the sample structure information. Further, the log structure processing device can determine the historical log corresponding to the sample structure information as the sample log.

[0070] In the above scheme, based on the text similarity of structural information, sample logs similar to the current log are retrieved from the log sample library, and the sample regular expression corresponding to the sample log is determined. This retrieval process not only relies on the matching of keywords, but also matches the similarity in the vector space, which can effectively process logs with similar structures but different contents. Through the similarity matching of logs, sample logs and sample regular expressions suitable for the current log are quickly found, which improves the accuracy of the subsequent model generating the regular expression corresponding to the current log and enhances the generalization ability of the log structured processing method. In addition, as different types of samples are expanded in the log sample library, the log structured processing method can adaptively perform structured parsing on the logs of the new expanded type.

[0071] Step 102: input the current log, sample log, sample regular expression, and generation prompt words into an expression generation model, and output a current regular expression corresponding to the current log.

[0072] The prompt word may be a text segment of the input language model, and the prompt word is used to guide the model to generate a specific output. The current regular expression may be a regular expression for structurally processing the current log determined by the expression generation model through neural network processing, and the number of field data extracted by the current regular expression may be greater than or much greater than the number of field data included in the structural information.

[0073] The generation prompt word can be used to guide the expression generation model to refer to the sample log and the sample regular expression to generate the regular expression corresponding to the current log. In some embodiments of the present disclosure, the generation prompt word is used to guide the expression generation model to use the sample regular expression as the regular expression corresponding to the current log when the sample regular expression is applicable to the current log, and to generate the regular expression corresponding to the current log with reference to the sample log and the sample regular expression when the sample regular expression is not applicable to the current log.

[0074] In this embodiment, the generation prompt word can guide the expression generation model to determine the fields that can be extracted by structured processing of the current log based on the sample regular expression, and determine the field proportion of the field in all fields, and judge whether the field proportion is greater than the proportion threshold. If so, the sample regular expression is used as the regular expression corresponding to the current log, otherwise, the regular expression corresponding to the current log is generated based on the current log and with reference to the sample regular expression and the sample log. Thus, under the guidance of the generation prompt word, the expression generation model uses historical logs and historical regular expressions as templates, and based on its own context learning and generation capabilities, automatically generates or fine-tunes regular expressions for the current log, so that the final generated regular expression is adapted to the specific log format and field.

[0075] The expression generation model can be used to generate a regular expression corresponding to a log, and the expression generation model can be a pre-trained language model. The pre-trained language model can refer to a model that is pre-trained on a large amount of text data to learn language structure and semantics. The pre-trained language model can be implemented based on deep learning technology, including a transformer (Transformer) architecture, through which the pre-trained language model can capture contextual information and the complexity of the language. The present embodiment does not limit the scale of the expression generation model, for example, the scale of the expression generation model can be hundreds of millions to hundreds of billions of parameters. The expression generation model can be unsupervised pre-training on text data first, and then supervised fine-tuning on specific tasks such as regular expression generation. The expression generation model can have a certain degree of versatility, and specifically, the expression generation model can be used for natural language processing tasks such as text generation, translation, and question and answer.

[0076] In an embodiment of the present disclosure, after determining a sample log having text similarity with the current log and a sample regular expression corresponding to the sample log, the log structuring processing device can input the current log, the sample log, the sample regular expression, and a generation prompt word into an expression generation model. Under the guidance of the generation prompt word, the expression generation model performs neural network calculations on the current log, the sample log, and the sample regular expression to generate a current regular expression corresponding to the current log.

[0077] Figure 3 A flow chart of another log structured processing method provided by the embodiment of the present disclosure is as follows: Figure 3 As shown, the current log, sample log, sample regular expression, and generation prompt word are input into the expression generation model, and the current regular expression corresponding to the current log is output, including:

[0078] Step 301: input the current log, sample log, sample regular expression, and generation prompt words into an expression generation model to obtain an initial regular expression output by the expression generation model.

[0079] The initial regular expression may be the first regular expression determined by the expression generation model according to the current log, the sample log, the sample regular expression, and the generation prompt word. The initial regular expression may be understood as the regular expression corresponding to the current log determined in the first iteration.

[0080] In this embodiment, the log structure processing device can input the current log, sample log, sample regular expression, and generation prompt words into the expression generation model. Under the guidance of the generation prompt words, the expression generation model generates the regular expression for the first time based on the current log, sample log, and sample regular expression to obtain the initial regular expression.

[0081] Step 302, determine the initial regular expression as the regular expression to be processed, and determine the field ratio; wherein the field ratio is the ratio of the matching field in the log field of the current log, and the matching field is the field that the regular expression to be processed matches the current log.

[0082] Among them, it is to be determined whether the regular expression to be processed is used as the final regular expression of the current log. The matching field can be the field where the regular expression to be processed successfully matches the current log. The matching field can be understood as the field in the current log that the regular expression to be processed can process. The log field can be the key field in the current log. The log field can be all the fields contained in the current log. The field ratio can be the calculation result of the ratio of the matching field to the log field.

[0083] In this embodiment, the log structured processing device may determine the initial regular expression as the regular expression to be processed, and further determine the field ratio based on the regular expression to be processed and the current log.

[0084] There are many methods for determining the field proportion, which are not limited in this embodiment. In an optional implementation, determining the field proportion includes: inputting field matching prompt words into the expression generation model to obtain matching fields, and determining the field proportion based on the matching fields and log fields. Among them, the field matching prompt words are used to guide the expression generation model to determine the matching fields that can be structured by the regular expression to be processed in all fields of the current log. Specifically, the log structured processing device can input the field matching prompt words into the expression generation model to obtain the matching fields output by the expression generation model. The log structured processing device divides the number of matching fields by the number of log fields contained in the current log to obtain the field proportion.

[0085] In another optional implementation, determining the field proportion includes: inputting a proportion calculation prompt word into an expression generation model to obtain the field proportion output by the expression generation model. The proportion calculation prompt word is used to guide the expression generation model to determine the proportion of matching fields that can be structured and processed by the regular expression to be processed in all fields of the current log. Specifically, the log structured processing device can input the proportion calculation prompt word into the expression generation model, so that the expression generation model determines the field proportion according to the current log and the regular expression to be processed under the guidance of the proportion calculation prompt word, and the log structured processing device can obtain the field proportion output by the expression generation model.

[0086] In this embodiment, if the field proportion is greater than the proportion threshold, the regular expression to be processed is determined as the current regular expression.

[0087] Step 303, if the field proportion is not greater than the proportion threshold, the prompt word is modified and input into the expression generation model to obtain the intermediate regular expression output by the expression generation model, and the intermediate regular expression is used as a new regular expression to be processed. Return to determine the new field proportion corresponding to the new intermediate regular expression until the new field proportion is greater than the proportion threshold, and the new intermediate regular expression is determined as the current regular expression.

[0088] The percentage threshold may be a preset minimum field percentage when the regular expression to be processed is determined as the current regular expression. The unmatched field may be a field in the log field of the current log that is not processed by the regular expression to be processed. The unmatched field may be understood as a field outside the scope of the field structured by the regular expression to be processed. The intermediate regular expression may be a regular expression generated by each iteration except the first iteration during the iteration of the regular expression corresponding to the current log.

[0089] In some embodiments of the present disclosure, the modification prompt word is used to guide the expression generation model to determine the unmatched fields and modify the regular expression to be processed according to the unmatched fields. Through the modification prompt word, forward iteration can be performed based on the unmatched fields during the iteration process of the regular expression, thereby improving the adaptability of the regular expression to the current log.

[0090] In this embodiment, the log structured processing device can determine whether the field ratio is greater than the ratio threshold. If not, the log structured processing device can input the modification prompt word into the expression generation model to guide the expression generation model to adjust the regular expression to be processed based on the current log and the regular expression to be processed, and output the intermediate regular expression. The log structured processing device obtains the intermediate regular expression, and uses the intermediate regular expression as a new regular expression to be processed, and returns it for iterative calculation. In the iterative calculation returned, the new field ratio of the new regular expression is determined until the new field ratio is greater than the ratio threshold, and the new regular expression to be processed is determined as the current regular expression.

[0091] In the above scheme, the expression generation model implements multiple rounds of optimization iterations on the regular expression corresponding to the current log through multiple rounds of dialogues based on the input prompt words, so that the final generated current regular expression can be adapted to the current log, and the corresponding regular expressions can be flexibly generated for different logs containing different fields.

[0092] Step 103: extract field data of the associated log using the current regular expression to obtain target structured data of the associated log; wherein the associated log is a log of the same type as the current log.

[0093] Among them, the associated log can be a log that is completely or partially the same as the current log format. The associated log can include the current log and / or other logs of the same type as the current log. Other logs can be logs generated by the same preset system as the current log. This embodiment does not limit the number of associated logs. Structured data can be data with clear meaning, order and data type. The target structured data can be structured data obtained by log parsing the associated log. This embodiment does not limit the fields included in the target structured data. For example, the field can include one or more of an Internet Protocol address, a timestamp, and a protocol. The number of field data contained in the target structured data can be greater than or much greater than the number of field data contained in the structural information, thereby achieving automated parsing and extraction of a large amount of field data based on a small amount of example field data.

[0094] In the disclosed embodiment, after determining the current regular expression corresponding to the current log, the log structured processing device can parse the current log through the current regular expression, extract the corresponding data field, and convert the data field into the target structured data. Alternatively, the log structured processing device can obtain other logs generated by the preset system, parse the other logs through the current regular expression, extract the corresponding data field, and convert the data field into the target structured data.

[0095] Optionally, after determining the target structured data, the log structured processing device may output the target structured data to a log storage platform and / or a log analysis platform to further store and / or analyze the target structured data, and the analysis may include security analysis and / or visualization analysis.

[0096] The log structured processing method provided by the embodiment of the present disclosure includes: determining a sample log matching the current log in multiple historical logs, and obtaining a sample regular expression corresponding to the sample log; inputting the current log, the sample log, the sample regular expression, and the generation prompt word into the expression generation model, and outputting the current regular expression corresponding to the current log; extracting field data from the associated log through the current regular expression to obtain the target structured data of the associated log; wherein the associated log is a log of the same type as the current log. The above technical scheme is adopted to realize the dynamic determination of the corresponding sample log and the sample regular expression according to different current logs, and the current regular expression applicable to the current log is generated under the guidance of the generation prompt word by referring to the sample log and the sample regular expression through the expression generation model, and the current regular expression is used to perform structured processing on the associated log of the same type as the current log, breaking through the limitation of manually writing regular expressions in the related technology, realizing the automatic and accurate generation of the regular expression of the current log by referring to the sample log and the sample regular expression through the model, improving the development efficiency, and performing log parsing processing on the same type of log as the current log through the regular expression, reducing the cost of log parsing and maintenance.

[0097] In addition, the automatic generation of regular expressions avoids the manual writing and dimensioning of regular expressions, reducing labor consumption; the ability to generate regular expressions can increase accordingly based on the growth of the number and types of historical logs, making the generation of regular expressions more scalable. For complex logs and different types of logs, it can also be based on the existing historical logs, historical regular expressions and the model's own generation capabilities to automatically parse context and nested information and generate corresponding regular expressions. It is realized based on historical logs and their corresponding historical regular expressions, but not limited to the specific rules of historical regular expressions, to generate current regular expressions suitable for current logs. This method has good log parsing capabilities and a shorter regular expression generation cycle, and can achieve cross-platform log parsing.

[0098] Since the expression generation model can be a pre-trained large-scale language model, the model has good context understanding capabilities, which enhances the adaptability when generating regular expressions for logs containing complex content.

[0099] Next, the log structured processing method in the embodiment of the present disclosure is further described by using a specific example. Figure 4 A schematic diagram of a log structured processing method provided by an embodiment of the present disclosure is shown as follows: Figure 4As shown, the method can realize security event management and response from log source to visual display based on log receiving module, log preprocessing module, log retrieval module, expression generation module, expression application module and data output module.

[0100] A log receiving module, through which original security logs of different formats can be received, and the original security logs are transmitted as current logs to the log preprocessing module for determining structural information. Specifically, the log receiving module may include one or more of a system log protocol tool, a Kafka protocol tool, a simple network management protocol tool, an open database connection interface tool, a Java database connection interface tool, and a file transfer protocol tool.

[0101] The log preprocessing module, the built-in field set can include multiple preset fields, and the current log as unstructured data is matched, automatically converted, and analyzed through the built-in field set, the corresponding field data is extracted and the log structure is analyzed to form semi-structured or structured current structure information, and the current structure information is sent to the log retrieval module.

[0102] The log retrieval module compares the values ​​corresponding to the same preset fields based on the current structure information generated by the log preprocessing module and the vectorization technology, and then retrieves the sample log with similar structure to the current log from the log sample library, and determines the sample regular expression corresponding to the sample log. The sample log and the sample regular expression are sent to the expression generation module.

[0103] The expression generation module receives the sample logs and their corresponding sample regular expressions sent by the log retrieval module, and uses the generation capability of the expression generation model to analyze the log format based on the sample logs and sample regular expressions, and generates a regular expression that matches the current log. The current regular expression is suitable for the format and field structure of the current log. The expression generation model is based on contextual learning technology and can flexibly generate corresponding regular expressions according to different log formats and fields.

[0104] The expression application module applies the current regular expression to the current log to extract key information. The extracted key information is output in a structured format for subsequent analysis.

[0105] The data output module transmits the parsed target structured data to a storage system such as a database or an analysis platform for further security monitoring and visualization.

[0106] In the above scheme, the expression generation model can automatically generate regular expressions that are suitable for different log formats, avoiding the tedious process of traditional manual writing. By retrieving sample logs similar to the current log and obtaining the sample regular expression corresponding to the sample log, the regular expression generated based on the sample log and the sample regular expression has high accuracy and adaptability, and can efficiently adapt to the processing of logs in a variety of log formats. It reduces the dependence on manual writing and maintenance of regular expressions and improves the efficiency of log parsing. And with the expansion of the log sample library, it can be adaptively expanded to parse more types of logs, with flexibility and scalability.

[0107] Figure 5 An embodiment of the present disclosure provides a schematic diagram of the structure of a log structured processing device, which can be implemented by software and / or hardware, and can also be integrated in an electronic device.

[0108] like Figure 5 As shown, the log structured processing device includes:

[0109] The determination module 501 is used to determine a sample log matching the current log among multiple historical logs, and obtain a sample regular expression corresponding to the sample log;

[0110] An input module 502 is used to input the current log, the sample log, the sample regular expression, and the generation prompt word into an expression generation model, and output a current regular expression corresponding to the current log;

[0111] The extraction module 503 is used to extract field data from the associated log using the current regular expression to obtain target structured data of the associated log; wherein the associated log is a log of the same type as the current log.

[0112] Optionally, the multiple historical logs correspond to multiple historical structure information. Accordingly, determining a sample log matching the current log from the multiple historical logs includes:

[0113] Extracting field data from the current log according to preset fields to obtain current structure information of the current log;

[0114] Among the plurality of historical structure information, the historical log corresponding to the historical structure information matching the current structure information is determined as the sample log.

[0115] Optionally, the historical structure information includes historical values ​​corresponding to the preset fields, and the current structure information includes current values ​​corresponding to the preset fields;

[0116] Accordingly, determining, among the plurality of historical structure information, the historical log corresponding to the historical structure information matching the current structure information as the sample log includes:

[0117] For each piece of the historical structure information, determine the similarity between the historical value corresponding to each of the preset fields in the historical structure information and the current value corresponding to each of the preset fields in the current structure information, and determine the information matching value between the historical structure information and the current structure information according to the similarity;

[0118] Determining sample structure information in the plurality of historical structure information according to the information matching value;

[0119] The historical log corresponding to the sample structure information is determined as the sample log.

[0120] Optionally, the generation prompt word is used to guide the expression generation model to use the sample regular expression as the regular expression corresponding to the current log when the sample regular expression is applicable to the current log, and to generate a regular expression corresponding to the current log with reference to the sample log and the sample regular expression when the sample regular expression is not applicable to the current log.

[0121] Optionally, the input module 502 is used to:

[0122] Inputting the current log, the sample log, the sample regular expression, and the generation prompt word into an expression generation model to obtain an initial regular expression output by the expression generation model;

[0123] Determine the initial regular expression as the regular expression to be processed, and determine the field ratio; wherein the field ratio is the ratio of the matching field in the log field of the current log, and the matching field is the field where the regular expression to be processed matches the current log;

[0124] If the field proportion is not greater than the proportion threshold, the modified prompt word is input into the expression generation model to obtain the intermediate regular expression output by the expression generation model, and the intermediate regular expression is used as a new regular expression to be processed. The new field proportion corresponding to the new intermediate regular expression is returned to determine until the new field proportion is greater than the proportion threshold, and the new intermediate regular expression is determined as the current regular expression.

[0125] Optionally, determining the field proportion includes:

[0126] The proportion calculation prompt word is input into the expression generation model to obtain the field proportion output by the expression generation model.

[0127] Optionally, the modification prompt word is used to guide the expression generation model to determine unmatched fields and modify the regular expression to be processed according to the unmatched fields.

[0128] It should be noted that Figure 5 The log structured processing device 500 shown can execute each step in the above-mentioned log structured processing method embodiment, and realize each process and effect in the above-mentioned log structured processing method embodiment, which will not be described in detail here.

[0129] Figure 6 The structure diagram of an electronic device provided by the embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the electronic device 600 includes one or more processors 601 and a memory 602 .

[0130] The processor 601 may be a central processing unit (CPU) or other forms of processing units having log structured processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 600 to perform desired functions.

[0131] The memory 602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 601 may run the program instructions to implement the log structured processing method of the embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0132] In one example, the electronic device 600 may further include: an input device 603 and an output device 604 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0133] In addition, the input device 603 may also include, for example, a keyboard, a mouse, and the like.

[0134] The output device 604 can output various information to the outside, including the determined distance information, direction information, etc. The output device 604 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0135] Of course, to simplify, Figure 6Only some of the components related to the present disclosure in the electronic device 600 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 600 may also include any other appropriate components.

[0136] In addition to the above-mentioned method and device, the embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the processor executes the log structured processing method provided by the embodiment of the present disclosure.

[0137] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0138] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the log structured processing method provided by the embodiment of the present disclosure.

[0139] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0140] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0141] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A log structured processing method, characterized in that: include: Determine a sample log that matches the current log among multiple historical logs, and obtain a sample regular expression corresponding to the sample log; Input the current log, the sample log, the sample regular expression, and the generation prompt word into the expression generation model, and output a current regular expression corresponding to the current log; Field data of the associated log is extracted using the current regular expression to obtain target structured data of the associated log; wherein the associated log is a log of the same type as the current log.

2. The method according to claim 1, characterized in that The multiple historical logs correspond to multiple historical structure information. Accordingly, determining a sample log matching the current log among the multiple historical logs includes: Extracting field data from the current log according to preset fields to obtain current structure information of the current log; Among the plurality of historical structure information, the historical log corresponding to the historical structure information matching the current structure information is determined as the sample log.

3. The method according to claim 2, characterized in that The historical structure information includes the historical value corresponding to the preset field, and the current structure information includes the current value corresponding to the preset field; Accordingly, determining, among the plurality of historical structure information, the historical log corresponding to the historical structure information matching the current structure information as the sample log includes: For each piece of the historical structure information, determine the similarity between the historical value corresponding to each of the preset fields in the historical structure information and the current value corresponding to each of the preset fields in the current structure information, and determine the information matching value between the historical structure information and the current structure information according to the similarity; Determining sample structure information in the plurality of historical structure information according to the information matching value; The historical log corresponding to the sample structure information is determined as the sample log.

4. The method according to claim 1, characterized in that: The generation prompt word is used to guide the expression generation model to use the sample regular expression as the regular expression corresponding to the current log when the sample regular expression is applicable to the current log, and to generate a regular expression corresponding to the current log with reference to the sample log and the sample regular expression when the sample regular expression is not applicable to the current log.

5. The method according to claim 1, characterized in that The step of inputting the current log, the sample log, the sample regular expression, and the generation prompt word into the expression generation model and outputting the current regular expression corresponding to the current log includes: Inputting the current log, the sample log, the sample regular expression, and the generation prompt word into an expression generation model to obtain an initial regular expression output by the expression generation model; Determine the initial regular expression as the regular expression to be processed, and determine the field ratio; wherein the field ratio is the ratio of the matching field in the log field of the current log, and the matching field is the field where the regular expression to be processed matches the current log; If the field proportion is not greater than the proportion threshold, the modified prompt word is input into the expression generation model to obtain the intermediate regular expression output by the expression generation model, and the intermediate regular expression is used as a new regular expression to be processed. The new field proportion corresponding to the new intermediate regular expression is returned to determine until the new field proportion is greater than the proportion threshold, and the new intermediate regular expression is determined as the current regular expression.

6. The method according to claim 5, characterized in that The determining of the field proportions includes: The proportion calculation prompt word is input into the expression generation model to obtain the field proportion output by the expression generation model.

7. The method according to claim 5, characterized in that The modification prompt word is used to guide the expression generation model to determine the unmatched fields and modify the regular expression to be processed according to the unmatched fields.

8. A log structured processing device, characterized in that: include: A determination module, used to determine a sample log matching the current log among multiple historical logs, and obtain a sample regular expression corresponding to the sample log; An input module, used for inputting the current log, the sample log, the sample regular expression, and the generation prompt word into an expression generation model, and outputting a current regular expression corresponding to the current log; The extraction module is used to extract field data of the associated log by using the current regular expression to obtain target structured data of the associated log; wherein the associated log is a log of the same type as the current log.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the log structured processing method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the log structured processing method described in any one of claims 1 to 7.

Images