Intelligent log analysis method, system and equipment and medium
By establishing an intelligent parsing model through the KNN algorithm, the problems of poor adaptability and noise sensitivity of existing log parsing methods are solved, flexible and accurate log parsing and large-scale real-time processing are achieved, and data access efficiency and parsing accuracy are improved.
Patent Information
- Application Number
- CN202510837913.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-23
- Publication Date
- 2025-10-17
AI Technical Summary
Existing log parsing methods rely on manual rules and have poor adaptability. Traditional machine learning methods are sensitive to noise and lack generalization capabilities, making it difficult to adapt to the changes and real-time processing of large-scale multi-source heterogeneous log data.
The KNN algorithm is used to establish an intelligent parsing model. Through real-time stream computing and machine learning, it automatically adapts to changes in log formats, generates new parsing rules, and realizes the conversion of unstructured data into structured data.
It improves data access efficiency and parsing accuracy, reduces maintenance costs, supports intelligent parsing of large-scale real-time log data, and provides reliable data support.
Smart Images
Figure CN120805890A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer information processing, and in particular to a log intelligent analysis method, system, device and medium. BACKGROUND
[0002] In the field of information technology, log data is an important basis for system monitoring, troubleshooting and performance optimization. Traditional log analysis methods mainly rely on manual rules or simple string matching techniques, such as regular expression-based pattern recognition or keyword extraction. However, these methods have obvious limitations when faced with large-scale, multi-source heterogeneous log data:
[0003] Firstly, the analysis method based on manual rules needs to write specific analysis rules for different systems or log formats, which not only has high development cost, but also is difficult to adapt to the dynamic changes of log format. For example, system upgrade or business adjustment may cause changes in log structure, at which time the original rules may be invalid and need to be adjusted, which is low in maintenance efficiency. Secondly, although the traditional string matching method can realize a certain degree of automatic analysis, it relies on fixed keywords or patterns and is difficult to effectively process complex log data, especially logs containing nested structures, dynamic variables or abnormal situations. For example, in a distributed system, logs generated by different modules may have similar semantics but different expressions, and simple pattern matching may easily lead to misanalysis or missed analysis, affecting the accuracy of subsequent analysis.
[0004] In recent years, with the development of big data and machine learning technology, log analysis methods based on intelligent algorithms have gradually become a research hotspot. This kind of method can automatically extract key information and classify by learning the semantic and structural features of the log, thereby reducing the dependence on manual rules. However, existing analysis methods based on machine learning still face challenges, such as insufficient robustness to noisy data and limited generalization ability to rare log patterns. Therefore, there is an urgent need for an efficient and adaptive intelligent log analysis method that can automatically adapt to changes in different log formats, accurately extract key information, and support real-time processing of large-scale log data to meet the higher requirements of modern IT systems for log analysis. SUMMARY
[0005] In view of the above existing problems, the present application is proposed.
[0006] Therefore, the present application aims to solve the problems of existing log analysis methods relying on manual rules, poor adaptability, and traditional machine learning methods being sensitive to noise and having insufficient generalization ability, and to provide an intelligent log analysis method that can automatically adapt to changes in log format, accurately extract key information and support large-scale real-time processing.
[0007] To solve the above technical problems, the present application provides the following technical solutions:
[0008] In a first aspect, the present application provides a log intelligent analysis method, comprising:
[0009] Obtaining security equipment and business system data and performing first processing;
[0010] Summarizing analysis rules according to the first processed data, analyzing corresponding data for each analysis rule through real-time stream computing, and processing data that is not normally analyzed;
[0011] The machine learning program compares the similarity of the unanalyzed data and the analyzed data through the first classification algorithm, and sends the similar unanalyzed data to the warehousing program;
[0012] The warehousing program generates a new analysis rule and inserts it into the database if the field can be automatically decomposed through the data analysis program, otherwise inserts it into the to-be-configured list, so that the user manually configures the analysis rule to realize intelligent data analysis.
[0013] As a preferred scheme of the log intelligent analysis method, wherein: the analysis of corresponding data for each analysis rule through real-time stream computing comprises:
[0014] Real-time analysis of data flowing into the business system to generate real-time results;
[0015] By loading the analysis rules into the memory, data in various different expression forms is converted into a unified description form, and the normalized fields include IP address, port, alarm level and alarm name.
[0016] The beneficial effects of the preferred technical scheme are: realizing the automatic conversion of unstructured data to structured data, greatly improving the data access efficiency and analysis accuracy.
[0017] As a preferred scheme of the log intelligent analysis method, wherein: the processing of data that is not normally analyzed comprises:
[0018] All data that is not normally analyzed is sent to a data analysis engine for machine learning and to a message queue for analysis by the data analysis program and insertion into the database.
[0019] As a preferred scheme of the log intelligent analysis method, wherein: using the already configured analysis rule data in combination with the real-time generated unanalyzed data, the first classification algorithm is used to find similar unanalyzed data, and the similar unanalyzed data is sent to the warehousing program.
[0020] As a preferred scheme of the log intelligent analysis method, the first classification algorithm compares the similarity of the unanalyzed data and the analyzed data, and the comparison includes:
[0021] The real-time unanalyzed data is converted into samples {X1, X2,..., X i ,...X N} after vectorization processing, where X i ={keys, md5(keys)}, each sample contains multidimensional features, and the result obtained after normalization processing will fall within the range of 0 to 1 in each dimension.
[0022] For any sample Y to be classified = [y1, y2,...y n ], the Manhattan distance of each sample in the training set is calculated, and the K nearest neighbors are selected according to the calculated distance.
[0023] The classification labels of the neighbors are counted.
[0024] The class of the sample to be classified is determined by a voting mechanism, that is, the label with the highest frequency is selected as the classification result, and the classification process is completed.
[0025] The beneficial effects of the technical scheme are that the model can automatically learn data features and intelligently match similar data formats, realizing flexible and accurate field parsing.
[0026] As a preferred scheme of the log intelligent analysis method, the operation of the warehousing program includes:
[0027] The warehousing program consumes data in the message queue, and according to the data analysis program, if the field can be automatically decomposed, a new analysis rule is generated and inserted into the database, otherwise, it is inserted into the configuration list, and the user manually configures the analysis rule in the interface.
[0028] The warehousing program obtains the result data of the machine learning program, and generates a new analysis rule according to the processing logic of the data analysis program and inserts it into the database.
[0029] The beneficial effects of the preferred technical scheme are that the new data rules can be dynamically adapted through continuous training, effectively solving the problems of high maintenance cost and poor adaptability of traditional methods.
[0030] As a preferred scheme of the log intelligent analysis method, the first processing includes deduplication processing and denoising processing.
[0031] The deduplication processing is to keep only the last record in the repeated records if the same record appears multiple times in the first time interval, and the only difference is the time, and other attributes are the same.
[0032] The denoising processing is to directly delete the data with inconsistent format, attribute number and requirement.
[0033] In a second aspect, the present application provides a log intelligent analysis system, comprising:
[0034] A data processing module is configured to acquire security and protection equipment data and business system data and perform first processing.
[0035] A real-time calculation module is configured to summarize analysis rules based on the first processed data, analyze corresponding data through real-time stream calculation for each analysis rule, and process data that is not normally analyzed.
[0036] A machine learning module is configured to compare the similarity of unanalyzed data and analyzed data through a first classification algorithm, and send similar unanalyzed data to a warehousing program.
[0037] A warehousing program analysis module is configured to generate a new analysis rule and insert it into a database if a field can be automatically decomposed through a data analysis program, or insert it into a to-be-configured list and let a user manually configure an analysis rule to achieve intelligent data analysis.
[0038] In a third aspect, the present application provides an electronic device comprising a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions to implement the steps of a log intelligent analysis method.
[0039] In a fourth aspect, the present application provides a computer readable storage medium storing computer executable instructions, which are executed by a processor to implement the steps of a log intelligent analysis method.
[0040] Compared with the prior art, the present application has the following beneficial effects: the present application provides a log intelligent analysis method, system, device and medium, and an intelligent analysis model is established by using a KNN algorithm, which has significant advantages compared with a traditional analysis method based on fixed rules, the model can automatically learn data features and intelligently match similar data formats to achieve flexible and accurate field analysis; new data rules can be dynamically adapted through continuous training, effectively solving the problems of high maintenance cost and poor adaptability of traditional methods; at the same time, the automatic conversion of unstructured data to structured data is realized, which greatly improves the data access efficiency and analysis accuracy, and provides more reliable data support for system monitoring, fault troubleshooting and other application scenarios. BRIEF DESCRIPTION OF DRAWINGS
[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.
[0042] Figure 1 The overall flow logical diagram of the log intelligent analysis method according to an embodiment of the present application.
[0043] Figure 2 The overall structure schematic diagram of the log intelligent analysis method according to an embodiment of the present application.
[0044] Figure 3 The algorithm model instance diagram of the log intelligent analysis method according to an embodiment of the present application. DETAILED DESCRIPTION
[0045] In order to make the above objectives, characteristics and advantages of the present application more apparent and comprehensible, the specific embodiments of the present application will be described in detail below with reference to the drawings in the specification. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort should belong to the protection scope of the present application.
[0046] Embodiment 1, refer to Figure 1 For an embodiment of the present application, a log intelligent analysis method is provided, as shown in Figure 1 specifically comprising the following steps:
[0047] S100: acquiring security and protection equipment and business system data and performing first processing;
[0048] S200: summarizing analysis rules according to the data after the first processing, analyzing the corresponding data through real-time stream calculation for each analysis rule, and processing the data that is not normally analyzed;
[0049] S300: the machine learning program compares the similarity of the unanalyzed data and the analyzed data through the first classification algorithm, and sends the similar unanalyzed data to the warehousing program;
[0050] S400: the warehousing program generates a new analysis rule and inserts it into the database through the data analysis program if the field can be automatically decomposed, otherwise inserts it into the to-be-configured list and lets the user manually configure the analysis rule to realize intelligent data analysis;
[0051] It should be noted that, in order to solve the problems of the existing log analysis method relying on manual rules, poor adaptability, and the traditional machine learning method being sensitive to noise and lacking generalization ability, the embodiment provides an intelligent log analysis method capable of automatically adapting to log format changes, accurately extracting key information, and supporting large-scale real-time processing. The steps S100-S400 establish an intelligent analysis model by using a KNN algorithm, which has significant advantages compared to traditional analysis methods based on fixed rules. The model can automatically learn data features and intelligently match similar data formats, enabling flexible and accurate field analysis. Through continuous training, the model can dynamically adapt to new data rules, effectively solving the problems of high maintenance cost and poor adaptability of traditional methods. At the same time, it realizes the automatic conversion of unstructured data to structured data, greatly improves the data access efficiency and analysis accuracy, and provides more reliable data support for system monitoring, fault troubleshooting, and other application scenarios.
[0052] Embodiment 2, refer to Figures 2 and 3 Based on the previous embodiment, the embodiment provides a specific implementation of the intelligent log analysis method, which describes the technical means used in the method.
[0053] In the embodiment of the present application, the step S100 of acquiring the security and protection equipment and business system data and performing the first processing includes:
[0054] Specifically, the security and protection equipment and business system data are acquired through the syslog method for data processing.
[0055] Specifically, the first processing includes de-duplication processing and de-noising processing.
[0056] The de-duplication processing is to keep only the last record in repeated records if the same record appears multiple times in a first time interval, with only time difference and other attributes being the same. The de-noising processing is to directly delete data that does not meet the format, number of attributes, and requirements.
[0057] It should be noted that the setting of the first time interval is based on the requirement of system business scenarios for data real-time performance and the typical frequency of device log generation, and is usually set as the minimum time unit (such as 5 minutes or 1 hour) for repeated log generation during normal operation of the device.
[0058] In an optional embodiment, the first processing further includes data standardization processing, i.e., format unification of the collected original data, such as converting time stamps of different time zones into a standard time format, or unifying numerical values of different units into the same dimension, to ensure consistency in subsequent processing.
[0059] In another optional embodiment, the first processing further includes key information extraction, that is, extracting core fields from the original data through regular expressions or keyword matching, filtering irrelevant content, reducing data redundancy, and improving parsing efficiency.
[0060] It should be noted that the above step S100 realizes unified collection and preliminary cleaning of multi-source heterogeneous data, and lays a high-quality data foundation for subsequent intelligent parsing. This processing effectively reduces noise interference in the original data, improves the consistency and availability of the data, and supports compatibility processing of different formats of data.
[0061] In the embodiments of the present application, the above step S200 summarizes the parsing rules according to the data after the first processing, and parses the corresponding data for each parsing rule through real-time stream computing, and processes the data that is not normally parsed, including:
[0062] Specifically, more than 400 parsing rules are summarized according to each technical defense device and system data, and the parsing rules are loaded into the memory, and the corresponding data is parsed through real-time stream computing for each parsing rule.
[0063] Specifically, the step of parsing the corresponding data for each parsing rule through real-time stream computing includes: real-time parsing of data flowing into the business system to generate real-time results; by loading the parsing rules into the memory, converting data in various expression forms into a unified description form, and normalizing fields including IP address, port, alarm level and alarm name.
[0064] It should be noted that real-time stream computing is a technology for processing data streams, which can perform real-time processing and analysis on data flowing into the system to generate real-time results, and is mainly used for processing real-time data. Real-time stream computing processes data through an event-based architecture that can automatically trigger processing, so that data is processed immediately when it arrives.
[0065] In the embodiments of the present application, the processing of the data that is not normally parsed includes:
[0066] All data that is not normally parsed is sent to a data analysis engine for machine learning, and a copy is sent to a message queue for analysis by a data parsing program and then inserted into a database.
[0067] It should be noted that the above step S200 combines the rule engine with stream computing, which can efficiently process massive real-time data streams, and automatically isolate and process abnormal data, significantly improving the real-time performance and fault tolerance of the system.
[0068] In the embodiment of the present application, the step S300 of comparing the similarity of the unanalyzed data and the analyzed data by the machine learning program through the first classification algorithm includes:
[0069] The unanalyzed data similar to the analyzed data is found through the first classification algorithm by using the configured analysis rule data in combination with the real-time generated unanalyzed data, and the unanalyzed data similar to the analyzed data is sent to the warehousing program.
[0070] Specifically, the step of comparing the similarity of the unanalyzed data and the analyzed data by the first classification algorithm includes:
[0071] The real-time unanalyzed data is converted into a sample {X1, X2,..., X i ,...X N} after vectorization processing, where X i ={keys, md5(keys)}, each sample contains multi-dimensional features, and the result obtained after normalization processing will fall within the range of 0 to 1 in each dimension.
[0072] For any sample Y to be classified = [y1, y2,...y n ], the Manhattan distance of each sample in the training set is calculated, and the K nearest neighbors are selected according to the calculated distance.
[0073] The classification labels of the neighbors are counted.
[0074] The class of the sample to be classified is determined through a voting mechanism, that is, the label with the highest frequency is selected as the classification result, and the classification process is completed.
[0075] It should be noted that the data input includes two parts, one part is the dimension table data, that is, the configured analysis rule, which is used to provide the training data set; the other part is the source data, that is, the unanalyzed data obtained from the specified index of ES every hour.
[0076] In the embodiment of the present application, the similarity of the unanalyzed data and the analyzed data is compared through the k-nearest neighbor classification (KNN) algorithm, which has the advantages of simplicity, no need to estimate parameters, no need to train, etc., and is particularly suitable for multi-classification problems (the object has multiple class labels). The idea of the algorithm is to find the K samples most similar to the sample to be classified in the known samples, then check the class to which the K known samples belong, find the class with the most samples, and finally classify the sample to be classified into this class. As shown in the following table, the classification process based on the KNN algorithm in the embodiment is as follows: Figure 2
[0077] First, select a K value (select the number of neighbors);
[0078] Read the dimension table data, and convert the dimension table data into samples {X1, X2,..., X i ,...X N}, wherein Xi={keys, md5(keys)}, the training data set sample is as follows:
[0079]
[0080] Unresolved data is:
[0081] {"user_name,ugname,umac,uip,appname,appgname,up,down,create_time,end_time",""}
[0082] "User_name, ugname, umac, uip, appname, appgname, up, down, create_time, end_time" is all the key values of the unresolved data, separated by commas; "2f3c89ab9280587ab1ab130498a76261" is the hash operation of all key values by the md5 function, as the category or label of the text data;
[0083] Further, the application uses a bag-of-words model or Word2Vec to convert text into numerical data; for a string:
[0084] {"user_name, ugname, umac, uip, appname",
[0085] "9c6a89ab9140587ab1ab130418b76261"} is extracted and converted, and the text is converted into numerical data; the above data points form a two-dimensional vector, and the normalized result will fall within the range of 0 to 1 in each dimension, facilitating classification analysis;
[0086] For any sample Y to be classified = [y1, y2,..., y n ], the Manhattan distance between it and each sample in the training set is calculated, and the formula is:
[0087] c=|x1-x2|+|y1-y2|
[0088] Wherein, X, Y represent two different samples, x i ,y i respectively represent the i-th feature value of the feature vector of the sample X, Y;
[0089] According to the calculated distance, the K nearest neighbors are selected, and the classification labels of the neighbors are counted;
[0090] The category of the sample to be classified is determined through a voting mechanism, i.e., the label with the highest frequency of occurrence is selected as the classification result, and the classification process is completed.
[0091] In an optional embodiment, the first classification algorithm can also be a random forest algorithm, which performs ensemble learning by constructing multiple decision trees, uses Bootstrap sampling and feature random selection to construct a population of differentiated decision trees, and determines the final classification result through a voting mechanism. The advantage is that it is naturally suitable for processing high-dimensional features and has strong robustness to noise data, and can automatically evaluate feature importance.
[0092] In another optional embodiment, the first classification algorithm can also be a support vector machine algorithm, which realizes data classification by finding the optimal classification hyperplane, uses kernel functions to process nonlinear feature space, is particularly suitable for small sample scenarios and has excellent generalization ability, has a solid mathematical theoretical foundation and is not sensitive to local extrema.
[0093] It should be noted that the system can flexibly select or combine the use of these algorithms according to the actual data characteristics and business needs. Random forest is more suitable for processing large-scale high-dimensional data, while support vector machine performs better in small sample scenarios. Both can effectively support intelligent parsing and automatic classification of log data, and together with KNN algorithm, they constitute an algorithm solution that adapts to different business scenarios.
[0094] It should be noted that the above step S300 can automatically identify potential associations between data patterns, intelligently classify similar data, effectively solve the problem of insufficient processing of rare data patterns by traditional methods, and greatly improve the system's adaptive ability to new data formats.
[0095] In the embodiments of the present application, as shown in Figure 3 The above step S400 storage procedure automatically decomposes the fields through the data parsing procedure, generates new parsing rules and inserts them into the database, or inserts them into the configuration list for manual configuration of the parsing rules, to realize intelligent data parsing, including:
[0096] The storage procedure consumes data in the message queue "topic_data_manage", and according to the data parsing procedure, if the fields can be automatically decomposed, new parsing rules are generated and inserted into the database, otherwise they are inserted into the configuration list for manual configuration of the parsing rules by the user in the interface;
[0097] The storage procedure obtains the result data of the machine learning procedure, and generates new parsing rules according to the processing logic of the data parsing procedure and inserts them into the database.
[0098] It should be noted that the above step S400 automatically generates new rules and updates the knowledge base for the analyzable data, continuously optimizes the system intelligence level; for complex data, it is configured manually to ensure the accuracy of analysis. Both realize the independent evolution of rules and ensure the reliability of the system in complex scenarios, and finally achieve efficient conversion of unstructured data to structured data.
[0099] In embodiment 3, a log intelligent analysis system is provided, comprising:
[0100] The data processing module is configured to acquire the security equipment and business system data and perform first processing.
[0101] The real-time computing module is configured to summarize the analysis rules according to the first processed data, analyze the corresponding data through real-time stream computing for each analysis rule, and process the data that is not normally analyzed.
[0102] The machine learning module is configured to compare the similarity of the unanalyzed data and the analyzed data through the first classification algorithm, and send the similar unanalyzed data to the warehousing program.
[0103] The warehousing program analysis module is configured to automatically generate new analysis rules and insert them into the database through the data analysis program if the fields can be automatically decomposed, or insert them into the configuration list for manual configuration of the analysis rules by the user to realize intelligent analysis of the data.
[0104] It should be noted that the technical scheme of the log intelligent analysis system belongs to the same concept as the technical scheme of the log intelligent analysis method described above, and the details of the technical scheme of the log intelligent analysis system in this embodiment are not described in detail. can be seen from the description of the technical scheme of the log intelligent analysis method.
[0105] The above each unit module can be embedded in or independent of the processor in the electronic device in hardware form, or can be stored in the memory in the electronic device in software form, so as to call and execute the operation corresponding to each module by the processor.
[0106] The embodiment also provides an electronic device, which comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved through WIFI, an operator network, NFC (Near Field Communication) or other technologies. The computer program is executed by the processor to implement a log intelligent analysis method. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the electronic device, or an external keyboard, touchpad or mouse, etc.
[0107] The embodiment also provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the method proposed in the above embodiment.
[0108] The storage medium proposed in the embodiment belongs to the same inventive concept as the method proposed in the above embodiment. The technical details not described in the embodiment can be referred to the above embodiment, and the embodiment has the same beneficial effects as the above embodiment.
[0109] From the above description about the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and necessary universal hardware, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a floppy disk, a ROM, a RAM, a FLASH, a hard disk or an optical disc, and includes a number of instructions to make an electronic device (which can be a personal computer, a server or a network device, etc.) execute the method of the embodiments of the present application.
[0110] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application rather than limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, and all of them should be covered in the scope of the claims of the present application.
Claims
1. A log intelligent parsing method, characterized in that: include: Obtain technical defense equipment and business system data and perform primary processing; Summarizing parsing rules based on the first processed data, parsing corresponding data using real-time stream computing for each parsing rule, and processing data that is not parsed normally; The machine learning program compares the similarity between unparsed data and parsed data through the first classification algorithm and sends similar unparsed data to the storage program; If the field can be automatically decomposed through the data analysis program, the warehousing program generates a new analysis rule and inserts it into the database; otherwise, it is inserted into the list to be configured, allowing the user to manually configure the analysis rule to achieve intelligent data analysis.
2. A log intelligent parsing method according to claim 1, characterized in that: Parsing the corresponding data by real-time stream computing for each parsing rule includes: Analyze data flowing into business systems in real time to produce real-time results; By loading the parsing rules into the memory, data in various expressions are converted into a unified description form, and the normalized fields include IP address, port, alarm level and alarm name.
3. A log intelligent parsing method according to claim 2, characterized in that: The processing of the data that is not parsed normally includes: For all data that is not parsed normally, one copy is sent to the data analysis engine for machine learning use, and one copy is sent to the message queue for analysis by the data parsing program and then inserted into the database.
4. A log intelligent parsing method according to claim 3, characterized in that: By using the configured parsing rule data and the unparsed data generated in real time, similar unparsed data are found through the first classification algorithm, and the similar unparsed data are sent to the warehousing program.
5. A log intelligent parsing method according to claim 4, characterized in that: Comparing the similarity between the unparsed data and the parsed data by the first classification algorithm includes: Real-time unresolved data is vectorized and converted into samples {X1,X2,...,X i ,...X N }, where X i ={keys,md5(keys)}, each sample contains multi-dimensional features, and the results obtained after normalization will fall within the range of 0 to 1 in each dimension; For any sample to be classified Y=[y1,y2,...y n ], calculate the Manhattan distance between it and each sample in the training set, and select the K nearest neighbors based on the calculated distance; Count the classification labels of neighbors; The category of the sample to be classified is determined through a voting mechanism, that is, the label with the highest frequency of occurrence is selected as the classification result to complete the classification process.
6. A log intelligent parsing method according to claim 5, characterized in that: The operations of the warehousing procedure include: The warehousing program consumes the data in the message queue and generates new parsing rules according to the data parsing program. If the fields can be automatically decomposed, new parsing rules are inserted into the database. Otherwise, the parsing rules are inserted into the list to be configured, allowing users to manually configure the parsing rules in the interface. The warehousing program obtains the result data of the machine learning program, and generates new parsing rules according to the processing logic of the data parsing program and inserts them into the database.
7. A log intelligent parsing method according to claim 1, characterized in that: The first processing includes deduplication processing and denoising processing; The deduplication process is to retain only the last record of the duplicate records if there are multiple records with only the time difference and the same other attributes within the first time interval; The denoising process is to directly delete data that is not in compliance with the format or has inconsistent attribute numbers and requirements.
8. A log intelligent analysis system, applying a log intelligent analysis method according to any one of claims 1 to 7, characterized in that: include: The data processing module is used to obtain technical defense equipment and business system data and perform primary processing; A real-time computing module, configured to summarize parsing rules based on the first processed data, parse the corresponding data using a real-time stream computing method for each parsing rule, and process data that is not parsed normally; A machine learning module, used for the machine learning program to compare the similarity between unparsed data and parsed data through a first classification algorithm, and send similar unparsed data to the storage program; The warehousing program parsing module is used to generate new parsing rules and insert them into the database if the fields can be automatically decomposed by the data parsing program; otherwise, it is inserted into the list to be configured, allowing users to manually configure the parsing rules to achieve intelligent data parsing.
9. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store computer-executable instructions, and the processor implements the steps of the log intelligent parsing method according to any one of claims 1 to 7 when executing the computer-executable instructions.
10. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that: When the computer executable instructions are executed by a processor, the steps of the log intelligent parsing method according to any one of claims 1 to 7 are implemented.