Log analysis method using AI large model

Abstract

The invention belongs to the technical field of network security, and particularly relates to a log analysis method using an AI large model, which is applied to a network security monitoring platform and comprises the following steps: acquiring source data of a log file; calling an AI large model in the platform to generate a corresponding regular expression based on the source data; sending the regular expression to the user for verification so as to obtain a final regular expression verified by the user; analyzing the source data according to the final regular expression to obtain an analysis result; and mapping the analysis results of the source data to the corresponding field templates one by one to generate standard log analysis results. According to the method, the regular expressions of the log files are intelligently recognized through the AI large model, efficient and accurate standardized analysis of the log files is achieved, meanwhile, the AI large model is further optimized through the manual feedback result, positive feedback circulation is achieved, and the performance of the AI large model is continuously optimized.

Description

Technical Field

[0001] The present invention belongs to the field of network security technology, and specifically relates to a log parsing method using an AI large model. Background Art

[0002] With the increasing number of network security threats, the log file collection of security products has become an important means for enterprises and organizations to conduct security monitoring, incident response and compliance audits. Faced with the growing number of diverse and heterogeneous security products, processing a large number of security logs in different formats has become a key task for network security operations.

[0003] Standardized parsing of logs requires writing regular expressions based on logs from different device sources to achieve field escape and log standardization. However, with the increase in the number of devices, manual standardized parsing of logs is extremely inefficient. Therefore, for the business scenario of security product log parsing, an efficient parsing method is urgently needed. Summary of the invention

[0004] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to use the AI ​​big model to intelligently identify the regular expressions of log files, so as to achieve efficient and accurate standardized parsing of log files. At the same time, the AI ​​big model is further optimized through manual feedback results, a positive feedback loop is achieved, and the performance of the AI ​​big model is continuously optimized.

[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides a log parsing method using an AI big model, which is applied to a network security monitoring platform, and the method includes: obtaining source data of a log file; calling the AI ​​big model in the platform to generate a corresponding regular expression based on the source data; sending the regular expression to a user for verification to obtain a final regular expression after user verification; parsing the source data according to the final regular expression to obtain a parsing result; mapping the parsing results of the source data one by one to corresponding field templates to generate a standard log parsing result.

[0006] According to a specific embodiment of the present invention, the step of mapping the parsing results of the source data one by one to the corresponding field template to generate a standard log parsing result includes: calling the AI ​​big model to map the parsing results of the source data one by one to the corresponding field template to generate an initial log parsing result; sending the initial log parsing result to the user for verification to obtain the final log parsing result after user verification; optimizing the AI ​​big model based on the source data and its corresponding final log parsing result.

[0007] According to a specific embodiment of the present invention, the step of sending the initial log parsing result to the user for verification to obtain the final log parsing result after user verification includes: sending the initial log parsing result to the user: receiving the final log parsing result after the user performs field processing on the initial log parsing result; wherein, the field processing includes: field format setting, field additional configuration, and field value mapping.

[0008] According to a specific embodiment of the present invention, the step of sending the regular expression to the user for verification to obtain the final regular expression after user verification includes: sending the regular expression to the user: if the user feedbacks that the regular expression is incorrect, calling the AI ​​big model to regenerate a new regular expression until the user verification is passed, and using the finally generated regular expression as the final regular expression.

[0009] According to a specific embodiment of the present invention, after sending the regular expression to the user, the method further includes: when the number of times the regular expression is regenerated exceeds a preset threshold, taking the regular expression input by the user as the final regular expression.

[0010] According to a specific embodiment of the present invention, it also includes: optimizing the AI ​​big model based on the source data and its corresponding final regular expression.

[0011] According to a specific embodiment of the present invention, the method further includes: classifying and aggregating successfully parsed log files, and integrating log files of the same type to generate corresponding event feedback to the user.

[0012] According to a specific embodiment of the present invention, the steps of classifying and aggregating successfully parsed log files, and integrating log files of the same type to generate corresponding event feedback to users include: identifying target fields in log files and dividing them into corresponding alarm types; integrating log files of the same alarm type, generating corresponding risk events and sending them to users for confirmation.

[0013] According to a specific embodiment of the present invention, the steps of classifying and aggregating successfully parsed log files, and integrating log files of the same type to generate corresponding event feedback to the user include: identifying the number of occurrences of log files with the same source IP within a preset time; when the number of occurrences exceeds a preset first threshold, integrating the log files with the same source IP, and generating corresponding risk events to send to the user for confirmation.

[0014] According to a specific embodiment of the present invention, the steps of classifying and aggregating successfully parsed log files, and integrating log files of the same type to generate corresponding event feedback to the user include: identifying the number of occurrences of log files about different source IPs within a preset time; when the number of occurrences exceeds a preset second threshold, integrating the log files of different source IPs, and generating corresponding risk events to send to the user for confirmation.

[0015] The present invention provides an intelligent log parsing and aggregation method, which uses an AI big model to automatically generate regular expressions and perform field mapping, and finally feeds back to the security operation and maintenance personnel for confirmation, so as to improve the processing efficiency of security logs and reduce the workload of security operation and maintenance personnel. It can also achieve efficient and accurate standardized parsing of log data from different security devices and manufacturers in a unified format.

[0016] In addition, the present invention classifies and aggregates log files through correlation analysis, thereby achieving effective deduplication and reducing false positives and repeated alarms. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flow chart of a specific embodiment of a log parsing method using an AI large model provided by the present invention; Figure 2 It is a schematic diagram of a specific embodiment of a log parsing platform; Figure 3 This is a structural schematic diagram of a specific embodiment of a log parsing system using an AI big model provided by the present invention; Figure 4 The present invention provides a structural block diagram of a specific embodiment of an electronic device. DETAILED DESCRIPTION

[0018] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0020] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0021] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0022] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0023] Example 1 The present invention provides a log parsing method using an AI large model, which is applied to a network security monitoring platform. Usually, the network security monitoring platform needs to manage security products of different categories from different manufacturers, and then monitor the operating status of the product objects by collecting and parsing the log files of the product objects.

[0024] With the expansion of network boundaries and the surge in the number of security products, the amount of data generated has also increased dramatically. Security operations personnel need to deal with a large number of log events, many of which are duplicate and invalid. Since different security vendors have their own attack signature libraries and monitoring strategies, in order to improve the overall perception of network security, the alarm threshold has to be lowered, which leads to an increase in false alarms and brings huge work pressure to monitoring personnel. In addition, the same network traffic may be captured repeatedly, which in turn triggers repeated alarms.

[0025] In order to effectively reduce false alarms and duplicate alarms and improve the efficiency of alarm analysis and processing, first of all, it is necessary to achieve efficient, accurate and standardized analysis of multi-source heterogeneous log data and unify the format; secondly, it is necessary to perform correlation analysis on the normalized log data to achieve effective deduplication of events to reduce false alarms and duplicate alarms.

[0026] In this regard, the log parsing method using the AI ​​big model provided in this embodiment is as follows: Figure 1 As shown, including: Step S100, obtaining source data of a log file.

[0027] Step S200, calling the AI ​​big model in the platform to generate a corresponding regular expression based on the source data.

[0028] Step S300, sending the regular expression to the user for verification to obtain a final regular expression after the user's verification.

[0029] Step S400: parsing the source data according to the final regular expression to obtain a parsing result.

[0030] Step S500, mapping the parsing results of the source data to corresponding field templates one by one to generate standard log parsing results.

[0031] Step S600: Classify and aggregate the successfully parsed log files, and integrate the log files of the same type to generate corresponding event feedback to the user.

[0032] First, after collecting the log files of the product objects, they need to be parsed accordingly. It can be understood that since the network security monitoring platform manages different types of security products from different manufacturers, the standardized parsing of the log files of each product object requires first writing corresponding regular expressions based on their different device sources, and then parsing the log files based on the regular expressions to achieve field escape and log standardization. Figure 2 As shown, security operation and maintenance personnel are usually required to input corresponding regular expressions based on log samples, that is, the source data of the log files, so that the network security monitoring platform can parse the log files based on the regular expressions and generate corresponding parsing results. However, when faced with massive log files, it will cause a serious burden on security operation and maintenance personnel.

[0033] In this regard, in this embodiment, intelligent recognition is performed through an AI big model, that is, by configuring the AI ​​big model on the network security monitoring platform, when the source data of the log file is obtained, it is input into the AI ​​big model, and the AI ​​big model's autonomous recognition is used to feedback the relevant results of the regular expression of the log sample. In order to ensure that the log file can be parsed accurately, the results of the AI ​​big model recognition can be fed back to the user, that is, the security operation and maintenance personnel for confirmation.

[0034] Specifically, in actual applications, the regular expressions of log files are first identified and fed back by the AI ​​large model, and then verified by the security operation and maintenance personnel. During the verification process, the security operation and maintenance personnel can directly adjust or modify the results generated by the AI ​​large model, or trigger the AI ​​large model to regenerate a new regular expression. For example, the regular expression generated by the AI ​​large model for the first time may not be very accurate. If the problem is minor, the security operation and maintenance personnel can directly modify and confirm it manually. If the problem is major, the security operation and maintenance personnel can call the AI ​​large model to regenerate the regular expression until the regular expression generated by the AI ​​large model gradually tends to an accurate result. Of course, if the AI ​​large model cannot generate a more accurate regular expression within the preset number of times, the security operation and maintenance personnel can directly enter the corresponding regular expression to save time. Therefore, when the regular expression generated by the AI ​​big model is confirmed to be correct by the security operation and maintenance personnel, the final regular expression will be used as the final regular expression; when the regular expression generated by the big model is confirmed to be incorrect by the security operation and maintenance personnel, the regular expression modified or re-entered by the security operation and maintenance personnel will be used as the final regular expression. This greatly reduces the workload of the security operation and maintenance personnel compared to manually writing the corresponding regular expressions.

[0035] At the same time, the AI ​​big model can also learn and optimize based on the log file and the corresponding regular expression. No matter whether the generated regular expression is correct or wrong, or the regular expression modified by the user or the regular expression re-entered, it can be used as an experience value to optimize its recognition and judgment ability for log files. With the continuous cycle of learning and optimization, the AI ​​big model can eventually deal with different types of log files more comprehensively. This can not only improve the processing efficiency of log files, but also reduce the workload of security operation and maintenance personnel.

[0036] Secondly, based on the above-mentioned final regular expression, the log file can be parsed accordingly. The parsing methods for the log files here include but are not limited to ordinary regular parsing, key-value parsing, JSON parsing, etc., among which key-value parsing and JSON parsing intercept the content to be processed through segmented words: the key-value parsing method requires configuration of field separation values, kv delimiters, and kv special regular expressions. No excessive restrictions are made on this. Without departing from the spirit of the present invention, the modifications and embellishments made to the embodiments of the present invention by those skilled in the art still fall within the scope of the invention patent application of the present invention.

[0037] Based on the parsing of log files, it is necessary to map the parsing results to the corresponding field templates one by one, so as to unify the parsing format of log files and realize standardized parsing. That is, the corresponding field templates are preset in advance for each product object. In the process of parsing the log files of any product object, unified standardized parsing of all log files of the product object can be achieved.

[0038] Specifically, it is necessary to map the fields in the above-mentioned parsing results of the log file to the positions of the corresponding field templates one by one, which correspondingly requires matching the fields in the log file with the preset field templates. Usually, this operation is also manually identified by security operation and maintenance personnel, which will increase the workload of security operation and maintenance personnel. In this regard, in this embodiment, the AI ​​big model is also used to complete it, that is, the field template and the above-mentioned preliminary parsing results (the field index in the log file) are input into it, and the AI ​​big model autonomously identifies and matches it, thereby generating a standard log parsing result, that is, Figure 2 The log parsing results are shown on the right.

[0039] Similarly, in order to ensure the accuracy of the parsing results of the log files, the results generated by the AI ​​big model can be fed back to the security operation and maintenance personnel for further confirmation to obtain the final log parsing results. Accordingly, during the verification process, the security operation and maintenance personnel can adjust or modify the initial log parsing results generated by the AI ​​model. For example, the AI ​​big model mistakenly matches a field to the wrong position in the field template, resulting in the meaning of the position in the field template not matching the field. Finally, after the user confirms that it is correct, the standard log parsing result is obtained, that is, the log file is successfully parsed.

[0040] It should also be added here that the user can perform field processing operations in the process of verifying the log parsing results generated by the AI ​​big model, including but not limited to field format setting, field additional configuration, and field value mapping, etc. Among them, for field format setting, it can support parsing various date formats in the log file into a unified format, for field additional configuration, it supports enrichment of the field, such as adding geographic information including latitude and longitude, region, etc. to the IP field, and for field value mapping, it supports converting various values ​​of the log file into normalized field values, such as: the risk level of the log is 1, 2, 3, etc., and the risk level of the system normalization is high risk, medium risk, low risk, etc. Similarly, the field processing can be preliminarily operated by the AI ​​big model, and then fed back to the user for confirmation. There is no excessive restriction on this. The modifications and embellishments made to the embodiments of the present invention by those skilled in the art without departing from the spirit of the present invention still fall within the scope of the invention patent application of the present invention.

[0041] Based on the above field mapping operation, the corresponding log parsing results can be obtained. Similarly, the log file and the corresponding final log parsing results can be used to optimize the AI ​​big model, continuously enriching the applicable scenarios of the AI ​​big model and the processing capabilities for log files.

[0042] Finally, the successfully parsed log files are normalized, that is, they are deduplicated and aggregated through correlation analysis to reduce false positives and duplicate alarms.

[0043] In this regard, for some simple events, multiple log sources can be supported, that is, multiple product objects can be classified and aggregated. For example, for product 1 and product 2, as long as a certain target data, that is, the target field, can be classified into a certain alarm segment, so that the relevant log files for each segment are integrated, and the corresponding risk events are generated and sent to the security operation and maintenance personnel for confirmation. For some frequency events, the number of occurrences of log files with the same source IP within a preset time can be counted, and when the number of occurrences exceeds the preset first threshold, all log files with the same source IP are integrated to generate corresponding risk events and send them to the security operation and maintenance personnel for confirmation. For some diverse events, the number of occurrences of log files with different source IPs within a preset time can be counted, and when the number of occurrences exceeds the preset second threshold, all log files with different source IPs are integrated to generate corresponding risk events and send them to the security operation and maintenance personnel for confirmation. Based on the above operations, duplicate alarms can be reduced and duplicate removal is effective.

[0044] It should be noted that the step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0045] Example 2 See also Figure 3 As shown, this embodiment also provides a log parsing system using an AI big model, including: The log collection module 10 is used to obtain source data of log files.

[0046] The first log parsing module 20 is used to call the AI ​​big model in the platform to generate a corresponding regular expression based on the source data.

[0047] The second log parsing module 30 is used to send the regular expression to the user for verification to obtain a final regular expression after the user's verification.

[0048] The third log parsing module 40 is used to parse the source data according to the final regular expression to obtain a parsing result.

[0049] The fourth log parsing module 50 is used to map the parsing results of the source data to the corresponding field templates one by one to generate a standard log parsing result.

[0050] The log aggregation module 60 is used to classify and aggregate the successfully parsed log files, and integrate the log files of the same type to generate corresponding event feedback to the user.

[0051] It should be noted that the log parsing system using the AI ​​big model provided in the above embodiment and the log parsing method using the AI ​​big model provided in the above embodiment 1 belong to the same concept, and the specific way in which each module and unit performs the operation has been described in detail in the method embodiment, which will not be repeated here. In actual applications, the log parsing method using the AI ​​big model provided in the above embodiment 1 can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0052] Example 3 See also Figure 4 As shown, an embodiment of the present application further provides an electronic device, comprising a memory 2, a processor 1, and a program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the program.

[0053] Among them, the memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory can also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory can also include both an internal storage unit of the electronic device and an external storage device. The memory can not only be used to store application software and various types of data installed in the electronic device, but also can be used to temporarily store data that has been output or is to be output.

[0054] In some embodiments, the processor may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor is the control core (Control Unit) of the electronic device, which connects the various components of the entire electronic device using various interfaces and lines, and executes various functions of the electronic device and processes data by running or executing programs or modules stored in the memory, and calling data stored in the memory.

[0055] The processor executes the operating system of the electronic device and various installed application programs. The processor executes the application programs to implement the steps in the above method embodiment.

[0056] Exemplarily, the program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of program instruction segments capable of completing specific functions, which are used to describe the execution process of the program in the electronic device.

[0057] The above-mentioned integrated unit implemented in the form of software function modules can be stored in a computer-readable storage medium. The above-mentioned software function modules are stored in a storage medium and include several instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of various embodiments of the present invention.

[0058] In summary, the present invention provides an intelligent log parsing and aggregation method, which uses an AI big model to automatically generate regular expressions and perform field mapping, and finally feeds back to the security operation and maintenance personnel for confirmation, so as to improve the processing efficiency of security logs and reduce the workload of security operation and maintenance personnel. It can also achieve efficient and accurate standardized parsing of log data from different security devices and manufacturers in a unified format.

[0059] In addition, the present invention classifies and aggregates log files through correlation analysis, thereby achieving effective deduplication and reducing false positives and repeated alarms.

[0060] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A log parsing method using an AI big model, characterized in that: Applied to a network security monitoring platform, the method comprises: Get the source data of the log file; Calling the AI ​​big model in the platform to generate a corresponding regular expression based on the source data; Sending the regular expression to the user for verification to obtain a final regular expression verified by the user; Parsing the source data according to the final regular expression to obtain a parsing result; The parsing results of the source data are mapped one by one to the corresponding field templates to generate standard log parsing results.

2. The log parsing method using the AI ​​big model according to claim 1 is characterized in that: The steps of mapping the parsing results of the source data to the corresponding field templates one by one to generate standard log parsing results include: Calling the AI ​​big model to map the parsing results of the source data to the corresponding field templates one by one to generate initial log parsing results; Sending the initial log analysis result to the user for verification to obtain the final log analysis result after the user's verification; The AI ​​big model is optimized based on the source data and its corresponding final log parsing results.

3. The log parsing method using the AI ​​big model according to claim 2 is characterized in that: The step of sending the initial log parsing result to the user for verification to obtain the final log parsing result after the user's verification includes: Send the initial log parsing result to the user: Receive a final log parsing result after the user performs field processing on the initial log parsing result; The field processing includes: field format setting, field additional configuration, and field value mapping.

4. The log parsing method using the AI ​​big model according to claim 1 is characterized in that: The step of sending the regular expression to the user for verification to obtain the final regular expression after the user's verification includes: Send the regular expression to the user: If the user reports that the regular expression is incorrect, the AI ​​big model is called to regenerate a new regular expression until the user verifies it, and the final generated regular expression is used as the final regular expression.

5. The log parsing method using the AI ​​big model according to claim 4 is characterized in that: After sending the regular expression to the user, the following steps are also included: When the number of times the regular expression is regenerated exceeds a preset threshold, the regular expression input by the user is used as the final regular expression.

6. The log parsing method using the AI ​​big model according to claim 1 is characterized in that: Also includes: The AI ​​big model is optimized based on the source data and its corresponding final regular expression.

7. The log parsing method using the AI ​​big model according to claim 1 is characterized in that: Also includes: The successfully parsed log files are classified and aggregated, and the log files of the same type are integrated to generate corresponding event feedback to the user.

8. The log parsing method using the AI ​​big model according to claim 7 is characterized in that: The steps of classifying and aggregating the successfully parsed log files and integrating the log files of the same type to generate corresponding event feedback to the user include: Identify the target fields in the log file and classify them into corresponding alarm types; Integrate log files of the same alarm type, generate corresponding risk events and send them to users for confirmation.

9. The log parsing method using the AI ​​big model according to claim 7 is characterized in that: The steps of classifying and aggregating the successfully parsed log files and integrating the log files of the same type to generate corresponding event feedback to the user include: Identify the number of occurrences of the same source IP in the log file within a preset time period; When the number of occurrences exceeds a preset first threshold, the log files with the same source IP are integrated, and a corresponding risk event is generated and sent to the user for confirmation.

10. The log parsing method using the AI ​​big model according to claim 7 is characterized in that: The steps of classifying and aggregating the successfully parsed log files and integrating the log files of the same type to generate corresponding event feedback to the user include: Identify the number of occurrences of log files related to different source IPs within a preset time; When the number of occurrences exceeds the preset second threshold, the log files of different source IPs are integrated, and corresponding risk events are generated and sent to the user for confirmation.

Images