Universal log burying point collecting and extracting method and device and storage medium
Through FlinkQL generation common rules and automatic identification mechanism, the cumbersome and inconsistency problems of manually writing code in traditional log point collection methods are solved, efficient and accurate log collection and analysis are achieved, and the impact of system performance is reduced.
Patent Information
- Application Number
- CN202411153974.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-22
AI Technical Summary
The traditional log point acquisition method requires manual writing and configuration code, which is very labor-intensive and error-prone, and lacks consistency and standardization of the acquisition and extraction process.
Use FlinkQL to automatically generate common rules based on regular expressions, combine automatic recognition mechanisms to identify and extract target content in the log, and convert it into structured data for analysis, monitor applications, and troubleshoot and performance optimization.
Reduce the workload of manually burying points, improve the consistency and accuracy of burying points, realize real-time log collection, reduce the impact on system performance, and solve the large workload, error-prone and lack of consistency in traditional methods. The issue of standardization.
Smart Images

Figure CN119003276A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of modern software development and operation and maintenance, and specifically to a universal log point collection and extraction method, a universal log point collection and extraction device, a computer-readable storage medium, and an electronic device. Background Art
[0002] In modern software development and operations, logs are a crucial information resource for monitoring application status, troubleshooting, and performance optimization. Collecting and analyzing this log information typically requires instrumenting the application. This involves inserting code at key locations to output critical log information to log files or other storage systems. However, traditional methods for instrumenting logs have several drawbacks.
[0003] First, traditional log tracking and collection typically requires manual code writing and configuration, which is labor-intensive and error-prone. Developers need to manually insert tracking code at key locations in the application, which not only increases development and maintenance workload but also makes omissions and errors more likely.
[0004] Secondly, the log formats and content vary across applications, leading to a lack of consistency and standardization in the collection and extraction process. Each application has its own log format and fields, making the collection and analysis of these logs difficult and requiring customized processing for each application. Summary of the Invention
[0005] The main purpose of this application is to provide a universal log point collection and extraction method, a universal log point collection and extraction device, a computer-readable storage medium and an electronic device, so as to at least solve the problem that traditional log point collection usually requires manual writing and configuration of code, which is labor-intensive, error-prone, and lacks consistency and standardization in the collection and extraction process.
[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a general log tracking collection and extraction method is provided, including: using FlinkQL and automatically generating general rules based on regular expressions, the general rules are used to identify key information in the log, and the general rules are related to the tracking requirements; based on the automatic identification mechanism and the general rules, identifying the target content in the log, and extracting the target content in the log, the target content is the content that complies with the general rules; converting the target content into structured data to obtain target data; performing data analysis on the log based on the target data to monitor the application corresponding to the log, and troubleshooting and performance optimization of the application.
[0007] Optionally, FlinkQL is used to automatically generate general rules based on regular expressions, including: creating a data storage table built on the Kafka data stream, the data storage table is used to store the filtered data of the log, and the filtered data is obtained by filtering from the original data of the log based on the tracking requirements; creating an extraction engine built on the Kafka data stream, the extraction engine is used to extract the target data from the filtered data stored in the data storage table, and the extraction rules of the extraction engine are established based on the regular expressions; generating the general rules according to the data storage table and the extraction engine.
[0008] Optionally, the log includes multiple streaming data. Before creating a data storage table built on the Kafka data stream, the method also includes: using natural language technology to analyze the text of the log and extract keywords of each streaming data in the log; using a classification model to classify each streaming data according to the keywords of each streaming data to obtain the category corresponding to each streaming data; determining the streaming data whose category is the target category as the filtered data, and removing the streaming data whose category is the non-target category.
[0009] Optionally, before using FlinkQL and automatically generating general rules based on regular expressions, the method also includes: obtaining the tracking requirements provided by the business; inputting the tracking requirements into the machine learning model, and outputting a construction parameter set of at least one initial regular expression, wherein the initial regular expression corresponds one-to-one to the construction parameter set, and one construction parameter set includes at least one parameter for constructing the corresponding initial regular expression; using all the construction parameter sets to construct the corresponding initial regular expression, and determining to generate initial general rules based on all the regular expressions; using the initial general rules to extract data from the log to obtain initial extracted data; and adaptively learning the machine learning model based on the matching degree between the initial extracted data and the tracking requirements to output a target construction parameter set with an accuracy greater than or equal to a preset accuracy.
[0010] Optionally, the tracking point requirement is input into a machine learning model, and a construction parameter set of at least one initial regular expression is output, including: associating at least one preset rule according to the tracking point requirement; performing multiple combinations according to all the preset rules to obtain at least one rule combination, and one rule combination includes at least one preset rule; training is performed according to all the rule combinations, and a construction parameter set of at least one initial regular expression is output, and the initial regular expression, the construction parameter set and the rule combination correspond one-to-one.
[0011] Optionally, the machine learning model is adaptively learned based on the degree of match between the initial extraction data and the burial point requirements, including: scoring the initial extraction data based on the degree of match between the initial extraction data and the burial point requirements to obtain a corresponding first score, wherein the matching degree is positively correlated with the first score of the initial extraction data; scoring the preset rules in the rule combination corresponding to the initial extraction data based on the first score of the initial extraction data to obtain a second score corresponding to each of the preset rules; adjusting the weight of each of the preset rules based on the second score of each of the preset rules to adaptively learn the machine learning model, wherein the weight of the preset rule is positively correlated with the corresponding second score.
[0012] Optionally, the rule combination corresponding to the initial extraction data includes multiple preset rules, and the initial extraction data is scored according to the matching degree between the initial extraction data and the burial point requirements to obtain a corresponding first score, including: determining the current weight of each preset rule in the rule combination corresponding to the initial extraction data and the product of the initial extraction data and the matching degree as a corresponding sub-score, and the sub-score corresponds one-to-one to the preset rule; and determining the sum of all the sub-scores as the first score of the initial extraction data.
[0013] According to another aspect of the present application, a general log tracking collection and extraction device is provided, including: a generation unit, used to use FlinkQL and automatically generate general rules based on regular expressions, the general rules are used to identify key information in the log, and the general rules are related to the tracking requirements; an identification unit, used to identify target content in the log based on an automatic identification mechanism and the general rules, and extract the target content in the log, the target content being content that complies with the general rules; a first processing unit, used to convert the target content into structured data to obtain target data; a second processing unit, used to perform data analysis on the log based on the target data to monitor the application corresponding to the log, and perform troubleshooting and performance optimization on the application.
[0014] According to another aspect of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the general log point collection and extraction methods.
[0015] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the above-mentioned general log point collection and extraction methods.
[0016] Applying the technical solution of the present application, the above-mentioned general log point collection and extraction method first uses FlinkQL and automatically generates general rules based on regular expressions. The general rules are used to identify key information in the log, and the general rules are related to the point collection requirements; then, based on the automatic recognition mechanism and the general rules, the target content is identified in the log, and the target content in the log is extracted. The target content is the content that complies with the general rules; then the target content is converted into structured data to obtain the target data; finally, the log is analyzed based on the target data to monitor the application corresponding to the log, and the application is troubleshooted and performance optimized. This method reduces the workload of manual point collection, improves the consistency and accuracy of point collection, facilitates the expansion and upgrade of the system, realizes real-time log collection, reduces the impact on system performance, and solves the problem that traditional log point collection usually requires manual writing and configuration of code, which is labor-intensive, error-prone, and lacks consistency and standardization in the collection and extraction process. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:
[0018] Figure 1 A hardware structure block diagram of a mobile terminal that performs a general log embedding collection and extraction method provided in an embodiment of the present application is shown;
[0019] Figure 2 A schematic diagram of a general log point collection and extraction method provided in accordance with an embodiment of the present application is shown;
[0020] Figure 3 A structural block diagram of a general log point collection and extraction device provided according to an embodiment of the present application is shown.
[0021] The above drawings include the following reference numerals:
[0022] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. DETAILED DESCRIPTION
[0023] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0024] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.
[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0026] As described in the background, the traditional method of manually writing and configuring log tracking code is labor-intensive and error-prone. Developers need to manually insert tracking code in key locations within the application, which increases development and maintenance workload and is prone to omissions and errors. The log formats and content vary from application to application, resulting in a lack of consistency and standardization in the collection and extraction process. Each application has its own log format and fields, making the collection and analysis of these logs difficult and requiring customized processing for each application.
[0027] In order to solve the problems that traditional log point collection usually requires manual code writing and configuration, which is labor-intensive, error-prone, and lacks consistency and standardization in the collection and extraction process, the embodiments of the present application provide a universal log point collection and extraction method, a universal log point collection and extraction device, a computer-readable storage medium and an electronic device.
[0028] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.
[0029] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure diagram of a mobile terminal for a general log point collection and extraction method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.
[0030] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the general log embedding collection and extraction method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned networks may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0031] In this embodiment, a general log point collection and extraction method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0032] Figure 2 This is a flow chart of a general log point collection and extraction method according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:
[0033] Step S201: Use FlinkQL and regular expressions to automatically generate general rules. These general rules are used to identify key information in logs and are related to tracking requirements.
[0034] Specifically, the general rules are based on keywords, grammatical structures, context, etc. For example, rules can be designed to identify specific API calls, error messages, performance indicators, etc.
[0035] The business scenario of this embodiment is built on mobile banking monitoring logs and business logs. The definition template of the general rule is built based on Flink's QL syntax.
[0036] The specific method is to use FlinkQL to define general log collection and extraction rules. FlinkQL is a SQL-like query language provided by Apache Flink for querying and transforming streaming data. You can use FlinkQL to define rules for log streams to extract and process logs.
[0037] Using FlinkQL and automatically generating general rules based on regular expressions involves the following steps:
[0038] Step S301: Create a data storage table built on the Kafka data stream. The data storage table is used to store the filtered data of the log. The filtered data is obtained by filtering the original data of the log based on the tracking requirements.
[0039] Step S302: creating an extraction engine built on the Kafka data stream, wherein the extraction engine is used to extract the target data from the filtered data stored in the data storage table, and the extraction rules of the extraction engine are established based on the regular expression;
[0040] Step S303: Generate the general rules according to the data storage table and the extraction engine.
[0041] Specifically, FlinkQL is used to define rules for log streams to extract and process logs.
[0042] The following is a simple example that shows how to use FlinkQL to define log collection rules:
[0043] CREATE TABLE raw_logs(
[0044] log_time TIMESTAMP,
[0045] log_level STRING,
[0046] message STRING
[0047] )WITH(
[0048] 'connector' = 'kafka',
[0049] 'topic' = 'your_topic',
[0050] 'properties.bootstrap.servers'='your_bootstrap_servers' );
[0052] The above steps are used to create a data storage table built on the Kafka data stream.
[0053]
[0054] The above steps are also used to create a data storage table built on the Kafka data stream.
[0055]
[0056] The above steps are used to create an extraction engine built on the Kafka data stream. The specific general rules are established based on REGEXP regular expressions.
[0057] The log includes multiple stream data. Before creating a data storage table built on the Kafka data stream, the method further includes the following steps:
[0058] Step S401: Analyze the text of the log using natural language technology to extract keywords from each of the streaming data in the log;
[0059] Step S402: using a classification model to classify each of the streaming data according to the keywords of each of the streaming data, to obtain a category corresponding to each of the streaming data;
[0060] Step S403 : determining the streaming data of the target category as the filtered data, and removing the streaming data of the non-target category.
[0061] Specifically, this allows for a preliminary screening of the data in the logs, and the use of a classification model to automatically classify the data in the logs with a high accuracy rate.
[0062] Machine learning algorithms are used to analyze the message content of log streams to automatically identify and extract logs. Using fields as features, a large matrix is constructed to train a neural network for classification. Nine categories are identified and automatically categorized based on message content, achieving an accuracy rate of 99.8%. Messages within a specific category can be automatically selected for collection.
[0063] In some instances, the categories of data content include: basic, public, account, loan, financial management, insurance, fund, precious metals, and digital currency.
[0064] It should be noted that the above data categories are modified according to business needs and are not limited to the above categories.
[0065] To implement common rules, you can set up an automatic recognition mechanism that automatically identifies and marks content in logs that meets the defined rules. This automatic recognition is achieved through three methods: text analysis, pattern matching, and machine learning.
[0066] For automatic recognition mechanisms, text analysis, pattern matching, and machine learning are used. The following are related technical implementation methods:
[0067] Text analysis: Use natural language processing technology to analyze log text and identify entities, keywords, and key phrases. This helps you extract important information from the logs for further processing and analysis.
[0068] Pattern matching: Use regular expressions to match and extract log data. You can design matching rules based on the specific format and structure of the logs to identify and extract content that meets the rules.
[0069] Machine Learning: Use machine learning algorithms to train models to identify specific patterns or key information in logs. You can collect labeled log data and train models to automatically identify and tag content that meets your rules.
[0070] Before using FlinkQL to automatically generate general rules based on regular expressions, the above method also includes the following steps:
[0071] Step S501: Obtain the above-mentioned tracking point requirements provided by the business;
[0072] Step S502: Input the tracking requirement into a machine learning model, and output at least one construction parameter set for an initial regular expression. The initial regular expression corresponds to the construction parameter set one-to-one, and each construction parameter set includes at least one parameter for constructing the corresponding initial regular expression.
[0073] The above tracking requirements are input into the machine learning model, and the output is a construction parameter set for at least one initial regular expression, which includes the following steps:
[0074] Step S601: Associating at least one preset rule according to the tracking requirement.
[0075] Step S602: performing multiple combinations based on all the above preset rules to obtain at least one rule combination, wherein the rule combination includes at least one of the above preset rules;
[0076] Step S603 : Training is performed according to all the above rule combinations to output at least one construction parameter set of an initial regular expression, wherein the above initial regular expression, the above construction parameter set and the above rule combinations correspond one to one.
[0077] Specifically, through the adaptive learning mechanism of the machine learning model, the rules can be continuously optimized and updated according to the actual log data and usage, thereby improving the accuracy and adaptability of log extraction.
[0078] Step S503: construct the corresponding initial regular expressions using all the above construction parameter sets, and determine to generate initial general rules based on all the above regular expressions;
[0079] Step S504: extract data from the log using the initial general rules to obtain initial extracted data.
[0080] Step S505: Based on the degree of match between the initial extraction data and the tracking requirements, the machine learning model is adaptively learned to output a target construction parameter set with an accuracy greater than or equal to a preset accuracy.
[0081] Specifically, the above steps implement an adaptive learning mechanism that continuously optimizes and updates rules based on actual log data and usage to adapt to changing log formats and content, improving the system's accuracy and adaptability. Furthermore, through manually-intervened adaptive learning, after the system has been running for a period of time, a portion of the data is manually used to fine-tune the system model and adjust parameters, achieving system adaptive optimization.
[0082] According to the matching degree between the initial extracted data and the tracking requirements, the machine learning model is adaptively learned, including the following steps:
[0083] Step S5051: Score the initially extracted data based on the degree of match between the initially extracted data and the tracking requirement to obtain a corresponding first score, wherein the degree of match is positively correlated with the first score of the initially extracted data.
[0084] Step S5052: scoring the preset rules in the rule combination corresponding to the initial extracted data according to the first score of the initial extracted data to obtain a second score corresponding to each of the preset rules;
[0085] Step S5053: Adjust the weight of each of the above-mentioned preset rules according to the second score of each of the above-mentioned preset rules to perform adaptive learning on the above-mentioned machine learning model, and the weight of the above-mentioned preset rules is positively correlated with the corresponding second score.
[0086] Specifically, by scoring the initial extracted data, the preset rules are scored, and the weights of the preset rules are adjusted under the corresponding tracking requirements, thereby achieving adaptive learning, continuously optimizing and updating the rules to adapt to the ever-changing log format and content.
[0087] The initial extracted data can be scored manually or by machine. Scoring is based on the degree of match between the initial extracted data and the tracking requirements. A higher match results in a higher score. High initial extracted data leads to a higher score for the pre-defined rule that generated the initial extracted data, which in turn increases the weight of the corresponding rule. Generally, the score ranges from -10 to 10.
[0088] The rule combination corresponding to the initial extracted data includes a plurality of the preset rules. Scoring the initial extracted data according to the degree of match between the initial extracted data and the tracking point requirements to obtain a corresponding first score includes the following steps:
[0089] Step S701: Determine a corresponding sub-score by multiplying the current weight of each of the preset rules in the rule combination corresponding to the initial extracted data and the matching degree, wherein the sub-score corresponds to each of the preset rules one by one;
[0090] Step S702: The sum of all the above sub-scores is determined as the first score of the above initially extracted data.
[0091] Specifically, the automatic rule generation algorithm can reduce manual intervention and improve the efficiency and accuracy of log extraction. Real-time monitoring and optimization strategies can ensure that the system continuously optimizes rules during operation, thereby maintaining efficient log extraction capabilities.
[0092] Step S202: identifying target content in the log based on the automatic recognition mechanism and the general rules, and extracting the target content from the log, where the target content is content that complies with the general rules;
[0093] Specifically, by defining rules, we achieve universal support for multiple applications, reducing the workload of manual tracking and improving tracking consistency and accuracy. The flexibility of rule definition allows for customized configuration based on the needs of different applications, while also facilitating system expansion and upgrades.
[0094] In addition, by embedding the collection code in the application, real-time log collection is achieved, reducing the impact on system performance.
[0095] Step S203, converting the target content into structured data to obtain target data;
[0096] Specifically, the application of data collection and analysis technologies can enable the system to better understand and utilize log data, providing more reliable support for adaptive learning mechanisms.
[0097] Step S204: performing data analysis on the log based on the target data to monitor the application corresponding to the log, and performing troubleshooting and performance optimization on the application.
[0098] Specifically, by defining a set of rules, universal support for a variety of applications is achieved, reducing the workload of manually writing and configuring log embedding code. Developers do not need to manually insert embedding code, but instead automatically identify and extract key log information by defining rules, thereby achieving automation and standardization of log embedding. The rule definition is flexible and can be customized according to the needs of different applications. Developers can define appropriate rules based on the characteristics and requirements of the application to achieve the extraction of specific log formats and content. At the same time, the flexibility of rule definition also facilitates the expansion and upgrade of the system. By embedding the collection code in the application, real-time log collection is achieved, reducing the impact on system performance. Compared with the traditional method of manually writing and configuring code, the automated collection and extraction process of the present invention is more efficient and can improve the efficiency and performance of log processing.
[0099] By laying out relevant rule collection methods based on business log streams and implementing relevant business scenarios through relevant rule definitions, we have defined different levels of API abstraction for application development. From top to bottom, the API hierarchy is from high to low. The higher the level of abstraction at the top, the more convenient it is to use, the less flexible it is, and the better the performance. Although this patented invention sacrifices some performance, it returns great business convenience and configuration flexibility, enhancing the system's versatility and scalability.
[0100] The above-mentioned general log tracking collection and extraction method of the present application first uses FlinkQL and automatically generates general rules based on regular expressions. The general rules are used to identify key information in the log, and the general rules are related to the tracking requirements; then, based on the automatic identification mechanism and the general rules, the target content is identified in the log, and the target content in the log is extracted. The target content is the content that complies with the general rules; the target content is then converted into structured data to obtain the target data; finally, the log is analyzed based on the target data to monitor the application corresponding to the log, and troubleshoot and optimize the performance of the application. This method reduces the workload of manual tracking, improves the consistency and accuracy of tracking, facilitates the expansion and upgrade of the system, realizes real-time log collection, reduces the impact on system performance, and solves the problem that traditional log tracking collection usually requires manual writing and configuration of code, which is labor-intensive, error-prone, and lacks consistency and standardization in the collection and extraction process.
[0101] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0102] The embodiment of the present application also provides a general log embedding point collection and extraction device. It should be noted that the general log embedding point collection and extraction device of the embodiment of the present application can be used to execute the general log embedding point collection and extraction method provided by the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation methods, and the details that have been explained will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceived.
[0103] The following introduces the general log point collection and extraction device provided in the embodiment of the present application.
[0104] Figure 3 This is a schematic diagram of a general log point collection and extraction device according to an embodiment of the present application. Figure 3 As shown, the device includes: a generation unit 10, an identification unit 20, a first processing unit 30 and a second processing unit 40. The generation unit 10 is used to use FlinkQL and automatically generate general rules based on regular expressions. The above general rules are used to identify key information in the log, and the above general rules are related to the tracking requirements; the identification unit 20 is used to identify the target content in the above log based on the automatic identification mechanism and the above general rules, and extract the above target content in the above log. The above target content is the content that complies with the above general rules; the first processing unit 30 is used to convert the above target content into structured data to obtain target data; the second processing unit 40 is used to perform data analysis on the above log based on the above target data to monitor the application corresponding to the above log, and perform troubleshooting and performance optimization on the above application.
[0105] The above-mentioned general log point collection and extraction device of the present application includes: a generation unit, an identification unit, a first processing unit and a second processing unit. The generation unit is used to use FlinkQL and automatically generate general rules based on regular expressions. The general rules are used to identify key information in the log, and the general rules are related to the point-of-storage requirements; the identification unit is used to identify the target content in the log based on the automatic identification mechanism and the general rules, and extract the target content in the log. The target content is the content that complies with the general rules; the first processing unit is used to convert the target content into structured data to obtain the target data; the second processing unit is used to perform data analysis on the log based on the target data to monitor the application corresponding to the log, and to troubleshoot and optimize the performance of the application. The device reduces the workload of manual point-of-storage, improves the consistency and accuracy of the point-of-storage, facilitates the expansion and upgrade of the system, realizes real-time log collection, reduces the impact on system performance, and solves the problem that traditional log point-of-storage collection usually requires manual writing and configuration of code, which is labor-intensive, error-prone, and lacks consistency and standardization in the collection and extraction process.
[0106] In some instances, the generation unit includes a first creation module, a second creation module, and a first processing module. The first creation module is used to create a data storage table built on the Kafka data stream, and the data storage table is used to store the filtered data of the log, and the filtered data is obtained by filtering from the original data of the log based on the above-mentioned embedding requirements; the second creation module is used to create an extraction engine built on the Kafka data stream, and the extraction engine is used to extract the target data from the filtered data stored in the data storage table, and the extraction rules of the extraction engine are established based on the regular expression; the first processing module is used to generate the general rules according to the data storage table and the extraction engine. Using FlinkQL to define rules for log streams can realize the extraction and processing of logs.
[0107] In some instances, the log includes multiple stream data. Before creating a data storage table built on the Kafka data stream, the device further includes a second processing module, a first classification module, and a first determination module. The second processing module is used to analyze the text of the log using natural language technology to extract keywords of each of the stream data in the log; the first classification module is used to classify each of the stream data according to the keywords of each of the stream data using a classification model to obtain a category corresponding to each of the stream data; the first determination module is used to determine the stream data that is classified as the target category as the filtered data, and remove the stream data that is classified as the non-target category. In this way, a preliminary screening of the data in the log can be performed first, and the classification model can be used to automatically classify the data in the log with a high accuracy rate.
[0108] In some instances, before using FlinkQL and automatically generating general rules based on regular expressions, the above-mentioned device also includes a first acquisition module, a third processing module, a first construction module, a fourth processing module and a fifth processing module. The first acquisition module is used to obtain the above-mentioned tracking requirements provided by the business; the third processing module is used to input the above-mentioned tracking requirements into the machine learning model, and output a construction parameter set of at least one initial regular expression. The above-mentioned initial regular expression corresponds one-to-one to the above-mentioned construction parameter set, and one of the above-mentioned construction parameter sets includes at least one parameter for constructing the corresponding above-mentioned initial regular expression; the first construction module is used to use all of the above-mentioned construction parameter sets to construct the corresponding above-mentioned initial regular expression, and determine to generate initial general rules based on all of the above-mentioned regular expressions; the fourth processing module is used to use the above-mentioned initial general rules to extract data from the above-mentioned logs to obtain initial extracted data; the fifth processing module is used to adaptively learn the above-mentioned machine learning model according to the matching degree between the above-mentioned initial extraction data and the above-mentioned tracking requirements, so as to output a target construction parameter set with an accuracy greater than or equal to a preset accuracy. The above steps can implement an adaptive learning mechanism to continuously optimize and update rules based on actual log data and usage to adapt to the ever-changing log format and content, thereby improving the accuracy and adaptability of the system.
[0109] In some instances, the third processing module includes a first association module, a first combination module, and a first training module. The first association module is used to associate at least one preset rule according to the above-mentioned tracking requirements; the first combination module is used to make multiple combinations based on all of the above-mentioned preset rules to obtain at least one rule combination, and one of the above-mentioned rule combinations includes at least one of the above-mentioned preset rules; the first training module is used to train according to all of the above-mentioned rule combinations and output a construction parameter set of at least one initial regular expression. The above-mentioned initial regular expression, the above-mentioned construction parameter set, and the above-mentioned rule combination correspond one-to-one. Through the adaptive learning mechanism of the machine learning model, the rules can be continuously optimized and updated according to the actual log data and usage, thereby improving the accuracy and adaptability of log extraction.
[0110] In some instances, the fifth processing module includes a first scoring module, a second scoring module and a sixth processing module, the first scoring module is used to score the above-mentioned initial extraction data according to the matching degree between the above-mentioned initial extraction data and the above-mentioned burying point requirements, and obtain a corresponding first score, wherein the above-mentioned matching degree is positively correlated with the first score of the above-mentioned initial extraction data; the second scoring module is used to score the preset rules in the rule combination corresponding to the above-mentioned initial extraction data according to the first score of the above-mentioned initial extraction data, and obtain a second score corresponding to each of the above-mentioned preset rules; the sixth processing module is used to adjust the weight of each of the above-mentioned preset rules according to the second score of each of the above-mentioned preset rules, so as to adaptively learn the above-mentioned machine learning model, and the weight of the above-mentioned preset rules is positively correlated with the corresponding second score. By scoring the initial extraction data, the preset rules are scored, and the weight of the preset rules under the corresponding burying point requirements is adjusted, thereby realizing adaptive learning, continuously optimizing and updating the rules to adapt to the ever-changing log format and content.
[0111] In some instances, the first scoring module includes a seventh processing module and a second determination module, the seventh processing module is used to determine the current weight of each of the above-mentioned preset rules in the above-mentioned rule combination corresponding to the above-mentioned initial extraction data and the product of the above-mentioned initial extraction data and the above-mentioned matching degree as the corresponding sub-score, and the above-mentioned sub-score corresponds one-to-one to the above-mentioned preset rules; the second determination module is used to determine the sum of all the above-mentioned sub-scores as the first score of the above-mentioned initial extraction data. By automatically generating an algorithm for rules, manual intervention can be reduced and the efficiency and accuracy of log extraction can be improved. Real-time monitoring and optimization strategies can ensure that the system continuously optimizes rules during operation, thereby maintaining efficient log extraction capabilities.
[0112] The general log embedding collection and extraction device includes a processor and memory. The generation unit and other components are stored as program units in the memory, and the processor executes the program units stored in the memory to implement the corresponding functions. The modules are all located in the same processor; alternatively, the modules can be located in different processors in any combination.
[0113] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured. By adjusting kernel parameters, traditional log point collection often requires manual code writing and configuration, which is labor-intensive, error-prone, and lacks consistency and standardization in the collection and extraction process.
[0114] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0115] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is running, the device where the computer-readable storage medium is located is controlled to execute the general log point collection and extraction method.
[0116] Specifically, common log point collection and extraction methods include:
[0117] Step S201: Use FlinkQL and regular expressions to automatically generate general rules. These general rules are used to identify key information in logs and are related to tracking requirements.
[0118] Specifically, the general rules are based on keywords, grammatical structures, context, etc. For example, rules can be designed to identify specific API calls, error messages, performance indicators, etc.
[0119] Step S202: identifying target content in the log based on the automatic recognition mechanism and the general rules, and extracting the target content from the log, where the target content is content that complies with the general rules;
[0120] Specifically, by defining rules, we achieve universal support for multiple applications, reducing the workload of manual tracking and improving tracking consistency and accuracy. The flexibility of rule definition allows for customized configuration based on the needs of different applications, while also facilitating system expansion and upgrades.
[0121] Step S203, converting the target content into structured data to obtain target data;
[0122] Specifically, the application of data collection and analysis technologies can enable the system to better understand and utilize log data, providing more reliable support for adaptive learning mechanisms.
[0123] Step S204: performing data analysis on the log based on the target data to monitor the application corresponding to the log, and performing troubleshooting and performance optimization on the application.
[0124] Specifically, by defining a set of rules, universal support for multiple applications is achieved, reducing the workload of manually writing and configuring log tracking code.
[0125] Optionally, FlinkQL is used to automatically generate general rules based on regular expressions, including: creating a data storage table built on the Kafka data stream, the data storage table is used to store the filtered data of the above-mentioned logs, and the above-mentioned filtered data is obtained by filtering from the original data of the above-mentioned logs based on the above-mentioned embedding requirements; creating an extraction engine built on the Kafka data stream, the above-mentioned extraction engine is used to extract the above-mentioned target data from the above-mentioned filtered data stored in the above-mentioned data storage table, and the extraction rules of the above-mentioned extraction engine are established based on the above-mentioned regular expressions; generating the above-mentioned general rules according to the above-mentioned data storage table and the above-mentioned extraction engine.
[0126] Optionally, the above-mentioned log includes multiple streaming data. Before creating a data storage table built on the Kafka data stream, the above-mentioned method also includes: using natural language technology to analyze the text of the above-mentioned log, and extracting keywords of each of the above-mentioned streaming data in the above-mentioned log; using a classification model to classify each of the above-mentioned streaming data according to the keywords of each of the above-mentioned streaming data, and obtain the category corresponding to each of the above-mentioned streaming data; determining the above-mentioned streaming data with the above-mentioned category as the target category as the above-mentioned filtered data, and removing the above-mentioned streaming data with the above-mentioned category as the non-target category.
[0127] Optionally, before using FlinkQL and automatically generating general rules based on regular expressions, the above method also includes: obtaining the above-mentioned tracking requirements provided by the business; inputting the above-mentioned tracking requirements into the machine learning model, and outputting a construction parameter set of at least one initial regular expression, the above-mentioned initial regular expression corresponds one-to-one to the above-mentioned construction parameter set, and one of the above-mentioned construction parameter sets includes at least one parameter for constructing the corresponding above-mentioned initial regular expression; using all of the above-mentioned construction parameter sets to construct the corresponding above-mentioned initial regular expression, and determining to generate initial general rules based on all of the above-mentioned regular expressions; using the above-mentioned initial general rules to extract data from the above-mentioned logs to obtain initial extracted data; based on the matching degree between the above-mentioned initial extracted data and the above-mentioned tracking requirements, adaptively learning the above-mentioned machine learning model to output a target construction parameter set with an accuracy greater than or equal to a preset accuracy.
[0128] Optionally, the above-mentioned tracking requirements are input into a machine learning model, and a construction parameter set of at least one initial regular expression is output, including: associating at least one preset rule according to the above-mentioned tracking requirements; making multiple combinations according to all of the above-mentioned preset rules to obtain at least one rule combination, and one of the above-mentioned rule combinations includes at least one of the above-mentioned preset rules; training is performed according to all of the above-mentioned rule combinations, and a construction parameter set of at least one initial regular expression is output, and the above-mentioned initial regular expression, the above-mentioned construction parameter set and the above-mentioned rule combination correspond one-to-one.
[0129] Optionally, the machine learning model is adaptively learned based on the degree of match between the initial extraction data and the point-of-sale requirements, including: scoring the initial extraction data based on the degree of match between the initial extraction data and the point-of-sale requirements to obtain a corresponding first score, wherein the degree of match is positively correlated with the first score of the initial extraction data; scoring the preset rules in the rule combination corresponding to the initial extraction data based on the first score of the initial extraction data to obtain a second score corresponding to each of the preset rules; adjusting the weight of each of the preset rules based on the second score of each of the preset rules to adaptively learn the machine learning model, wherein the weight of the preset rule is positively correlated with the corresponding second score.
[0130] Optionally, the rule combination corresponding to the initial extraction data includes multiple preset rules. According to the matching degree between the initial extraction data and the point-of-sale requirements, the initial extraction data is scored to obtain a corresponding first score, including: the current weight of each preset rule in the rule combination corresponding to the initial extraction data and the product of the initial extraction data and the matching degree are determined as corresponding sub-scores, and the sub-scores correspond one-to-one to the preset rules; and the sum of all the sub-scores is determined as the first score of the initial extraction data.
[0131] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the above-mentioned general log point collection and extraction method when running.
[0132] Specifically, common log point collection and extraction methods include:
[0133] Step S201: Use FlinkQL and regular expressions to automatically generate general rules. These general rules are used to identify key information in logs and are related to tracking requirements.
[0134] Specifically, the general rules are based on keywords, grammatical structures, context, etc. For example, rules can be designed to identify specific API calls, error messages, performance indicators, etc.
[0135] Step S202: identifying target content in the log based on the automatic recognition mechanism and the general rules, and extracting the target content from the log, where the target content is content that complies with the general rules;
[0136] Specifically, by defining rules, we achieve universal support for multiple applications, reducing the workload of manual tracking and improving tracking consistency and accuracy. The flexibility of rule definition allows for customized configuration based on the needs of different applications, while also facilitating system expansion and upgrades.
[0137] Step S203, converting the target content into structured data to obtain target data;
[0138] Specifically, the application of data collection and analysis technologies can enable the system to better understand and utilize log data, providing more reliable support for adaptive learning mechanisms.
[0139] Step S204: performing data analysis on the log based on the target data to monitor the application corresponding to the log, and performing troubleshooting and performance optimization on the application.
[0140] Specifically, by defining a set of rules, universal support for multiple applications is achieved, reducing the workload of manually writing and configuring log tracking code.
[0141] Optionally, FlinkQL is used to automatically generate general rules based on regular expressions, including: creating a data storage table built on the Kafka data stream, the data storage table is used to store the filtered data of the above-mentioned logs, and the above-mentioned filtered data is obtained by filtering from the original data of the above-mentioned logs based on the above-mentioned embedding requirements; creating an extraction engine built on the Kafka data stream, the above-mentioned extraction engine is used to extract the above-mentioned target data from the above-mentioned filtered data stored in the above-mentioned data storage table, and the extraction rules of the above-mentioned extraction engine are established based on the above-mentioned regular expressions; generating the above-mentioned general rules according to the above-mentioned data storage table and the above-mentioned extraction engine.
[0142] Optionally, the above-mentioned log includes multiple streaming data. Before creating a data storage table built on the Kafka data stream, the above-mentioned method also includes: using natural language technology to analyze the text of the above-mentioned log, and extracting keywords of each of the above-mentioned streaming data in the above-mentioned log; using a classification model to classify each of the above-mentioned streaming data according to the keywords of each of the above-mentioned streaming data, and obtain the category corresponding to each of the above-mentioned streaming data; determining the above-mentioned streaming data with the above-mentioned category as the target category as the above-mentioned filtered data, and removing the above-mentioned streaming data with the above-mentioned category as the non-target category.
[0143] Optionally, before using FlinkQL and automatically generating general rules based on regular expressions, the above method also includes: obtaining the above-mentioned tracking requirements provided by the business; inputting the above-mentioned tracking requirements into the machine learning model, and outputting a construction parameter set of at least one initial regular expression, the above-mentioned initial regular expression corresponds one-to-one to the above-mentioned construction parameter set, and one of the above-mentioned construction parameter sets includes at least one parameter for constructing the corresponding above-mentioned initial regular expression; using all of the above-mentioned construction parameter sets to construct the corresponding above-mentioned initial regular expression, and determining to generate initial general rules based on all of the above-mentioned regular expressions; using the above-mentioned initial general rules to extract data from the above-mentioned logs to obtain initial extracted data; based on the matching degree between the above-mentioned initial extracted data and the above-mentioned tracking requirements, adaptively learning the above-mentioned machine learning model to output a target construction parameter set with an accuracy greater than or equal to a preset accuracy.
[0144] Optionally, the above-mentioned tracking requirements are input into a machine learning model, and a construction parameter set of at least one initial regular expression is output, including: associating at least one preset rule according to the above-mentioned tracking requirements; making multiple combinations according to all of the above-mentioned preset rules to obtain at least one rule combination, and one of the above-mentioned rule combinations includes at least one of the above-mentioned preset rules; training is performed according to all of the above-mentioned rule combinations, and a construction parameter set of at least one initial regular expression is output, and the above-mentioned initial regular expression, the above-mentioned construction parameter set and the above-mentioned rule combination correspond one-to-one.
[0145] Optionally, the machine learning model is adaptively learned based on the degree of match between the initial extraction data and the point-of-sale requirements, including: scoring the initial extraction data based on the degree of match between the initial extraction data and the point-of-sale requirements to obtain a corresponding first score, wherein the degree of match is positively correlated with the first score of the initial extraction data; scoring the preset rules in the rule combination corresponding to the initial extraction data based on the first score of the initial extraction data to obtain a second score corresponding to each of the preset rules; adjusting the weight of each of the preset rules based on the second score of each of the preset rules to adaptively learn the machine learning model, wherein the weight of the preset rule is positively correlated with the corresponding second score.
[0146] Optionally, the rule combination corresponding to the initial extraction data includes multiple preset rules. According to the matching degree between the initial extraction data and the point-of-sale requirements, the initial extraction data is scored to obtain a corresponding first score, including: the current weight of each preset rule in the rule combination corresponding to the initial extraction data and the product of the initial extraction data and the matching degree are determined as corresponding sub-scores, and the sub-scores correspond one-to-one to the preset rules; and the sum of all the sub-scores is determined as the first score of the initial extraction data.
[0147] An embodiment of the present invention provides a device, comprising a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are performed:
[0148] Step S201: Use FlinkQL and regular expressions to automatically generate general rules. These general rules are used to identify key information in logs and are related to tracking requirements.
[0149] Step S202: identifying target content in the log based on the automatic recognition mechanism and the general rules, and extracting the target content from the log, where the target content is content that complies with the general rules;
[0150] Step S203, converting the target content into structured data to obtain target data;
[0151] Step S204: performing data analysis on the log based on the target data to monitor the application corresponding to the log, and performing troubleshooting and performance optimization on the application.
[0152] Optionally, FlinkQL is used to automatically generate general rules based on regular expressions, including: creating a data storage table built on the Kafka data stream, the data storage table is used to store the filtered data of the above-mentioned logs, and the above-mentioned filtered data is obtained by filtering from the original data of the above-mentioned logs based on the above-mentioned embedding requirements; creating an extraction engine built on the Kafka data stream, the above-mentioned extraction engine is used to extract the above-mentioned target data from the above-mentioned filtered data stored in the above-mentioned data storage table, and the extraction rules of the above-mentioned extraction engine are established based on the above-mentioned regular expressions; generating the above-mentioned general rules according to the above-mentioned data storage table and the above-mentioned extraction engine.
[0153] Optionally, the above-mentioned log includes multiple streaming data. Before creating a data storage table built on the Kafka data stream, the above-mentioned method also includes: using natural language technology to analyze the text of the above-mentioned log, and extracting keywords of each of the above-mentioned streaming data in the above-mentioned log; using a classification model to classify each of the above-mentioned streaming data according to the keywords of each of the above-mentioned streaming data, and obtain the category corresponding to each of the above-mentioned streaming data; determining the above-mentioned streaming data with the above-mentioned category as the target category as the above-mentioned filtered data, and removing the above-mentioned streaming data with the above-mentioned category as the non-target category.
[0154] Optionally, before using FlinkQL and automatically generating general rules based on regular expressions, the above method also includes: obtaining the above-mentioned tracking requirements provided by the business; inputting the above-mentioned tracking requirements into the machine learning model, and outputting a construction parameter set of at least one initial regular expression, the above-mentioned initial regular expression corresponds one-to-one to the above-mentioned construction parameter set, and one of the above-mentioned construction parameter sets includes at least one parameter for constructing the corresponding above-mentioned initial regular expression; using all of the above-mentioned construction parameter sets to construct the corresponding above-mentioned initial regular expression, and determining to generate initial general rules based on all of the above-mentioned regular expressions; using the above-mentioned initial general rules to extract data from the above-mentioned logs to obtain initial extracted data; based on the matching degree between the above-mentioned initial extracted data and the above-mentioned tracking requirements, adaptively learning the above-mentioned machine learning model to output a target construction parameter set with an accuracy greater than or equal to a preset accuracy.
[0155] Optionally, the above-mentioned tracking requirements are input into a machine learning model, and a construction parameter set of at least one initial regular expression is output, including: associating at least one preset rule according to the above-mentioned tracking requirements; making multiple combinations according to all of the above-mentioned preset rules to obtain at least one rule combination, and one of the above-mentioned rule combinations includes at least one of the above-mentioned preset rules; training is performed according to all of the above-mentioned rule combinations, and a construction parameter set of at least one initial regular expression is output, and the above-mentioned initial regular expression, the above-mentioned construction parameter set and the above-mentioned rule combination correspond one-to-one.
[0156] Optionally, the machine learning model is adaptively learned based on the degree of match between the initial extraction data and the point-of-sale requirements, including: scoring the initial extraction data based on the degree of match between the initial extraction data and the point-of-sale requirements to obtain a corresponding first score, wherein the degree of match is positively correlated with the first score of the initial extraction data; scoring the preset rules in the rule combination corresponding to the initial extraction data based on the first score of the initial extraction data to obtain a second score corresponding to each of the preset rules; adjusting the weight of each of the preset rules based on the second score of each of the preset rules to adaptively learn the machine learning model, wherein the weight of the preset rule is positively correlated with the corresponding second score.
[0157] Optionally, the rule combination corresponding to the initial extraction data includes multiple preset rules. According to the matching degree between the initial extraction data and the point-of-sale requirements, the initial extraction data is scored to obtain a corresponding first score, including: the current weight of each preset rule in the rule combination corresponding to the initial extraction data and the product of the initial extraction data and the matching degree are determined as corresponding sub-scores, and the sub-scores correspond one-to-one to the preset rules; and the sum of all the sub-scores is determined as the first score of the initial extraction data.
[0158] The devices in this article can be servers, PCs, PADs, mobile phones, etc.
[0159] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program for initializing at least the following method steps:
[0160] Step S201: Use FlinkQL and regular expressions to automatically generate general rules. These general rules are used to identify key information in logs and are related to tracking requirements.
[0161] Step S202: identifying target content in the log based on the automatic recognition mechanism and the general rules, and extracting the target content from the log, where the target content is content that complies with the general rules;
[0162] Step S203, converting the target content into structured data to obtain target data;
[0163] Step S204: performing data analysis on the log based on the target data to monitor the application corresponding to the log, and performing troubleshooting and performance optimization on the application.
[0164] Optionally, FlinkQL is used to automatically generate general rules based on regular expressions, including: creating a data storage table built on the Kafka data stream, the data storage table is used to store the filtered data of the above-mentioned logs, and the above-mentioned filtered data is obtained by filtering from the original data of the above-mentioned logs based on the above-mentioned embedding requirements; creating an extraction engine built on the Kafka data stream, the above-mentioned extraction engine is used to extract the above-mentioned target data from the above-mentioned filtered data stored in the above-mentioned data storage table, and the extraction rules of the above-mentioned extraction engine are established based on the above-mentioned regular expressions; generating the above-mentioned general rules according to the above-mentioned data storage table and the above-mentioned extraction engine.
[0165] Optionally, the above-mentioned log includes multiple streaming data. Before creating a data storage table built on the Kafka data stream, the above-mentioned method also includes: using natural language technology to analyze the text of the above-mentioned log, and extracting keywords of each of the above-mentioned streaming data in the above-mentioned log; using a classification model to classify each of the above-mentioned streaming data according to the keywords of each of the above-mentioned streaming data, and obtain the category corresponding to each of the above-mentioned streaming data; determining the above-mentioned streaming data with the above-mentioned category as the target category as the above-mentioned filtered data, and removing the above-mentioned streaming data with the above-mentioned category as the non-target category.
[0166] Optionally, before using FlinkQL and automatically generating general rules based on regular expressions, the above method also includes: obtaining the above-mentioned tracking requirements provided by the business; inputting the above-mentioned tracking requirements into the machine learning model, and outputting a construction parameter set of at least one initial regular expression, the above-mentioned initial regular expression corresponds one-to-one to the above-mentioned construction parameter set, and one of the above-mentioned construction parameter sets includes at least one parameter for constructing the corresponding above-mentioned initial regular expression; using all of the above-mentioned construction parameter sets to construct the corresponding above-mentioned initial regular expression, and determining to generate initial general rules based on all of the above-mentioned regular expressions; using the above-mentioned initial general rules to extract data from the above-mentioned logs to obtain initial extracted data; based on the matching degree between the above-mentioned initial extracted data and the above-mentioned tracking requirements, adaptively learning the above-mentioned machine learning model to output a target construction parameter set with an accuracy greater than or equal to a preset accuracy.
[0167] Optionally, the above-mentioned tracking requirements are input into a machine learning model, and a construction parameter set of at least one initial regular expression is output, including: associating at least one preset rule according to the above-mentioned tracking requirements; making multiple combinations according to all of the above-mentioned preset rules to obtain at least one rule combination, and one of the above-mentioned rule combinations includes at least one of the above-mentioned preset rules; training is performed according to all of the above-mentioned rule combinations, and a construction parameter set of at least one initial regular expression is output, and the above-mentioned initial regular expression, the above-mentioned construction parameter set and the above-mentioned rule combination correspond one-to-one.
[0168] Optionally, the machine learning model is adaptively learned based on the degree of match between the initial extraction data and the point-of-sale requirements, including: scoring the initial extraction data based on the degree of match between the initial extraction data and the point-of-sale requirements to obtain a corresponding first score, wherein the degree of match is positively correlated with the first score of the initial extraction data; scoring the preset rules in the rule combination corresponding to the initial extraction data based on the first score of the initial extraction data to obtain a second score corresponding to each of the preset rules; adjusting the weight of each of the preset rules based on the second score of each of the preset rules to adaptively learn the machine learning model, wherein the weight of the preset rule is positively correlated with the corresponding second score.
[0169] Optionally, the rule combination corresponding to the initial extraction data includes multiple preset rules. According to the matching degree between the initial extraction data and the point-of-sale requirements, the initial extraction data is scored to obtain a corresponding first score, including: the current weight of each preset rule in the rule combination corresponding to the initial extraction data and the product of the initial extraction data and the matching degree are determined as corresponding sub-scores, and the sub-scores correspond one-to-one to the preset rules; and the sum of all the sub-scores is determined as the first score of the initial extraction data.
[0170] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.
[0171] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0172] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0173] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0175] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.
[0176] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0177] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.
[0178] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.
[0179] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:
[0180] 1) The above-mentioned general log tracking collection and extraction method of the present application first uses FlinkQL and automatically generates general rules based on regular expressions. The general rules are used to identify key information in the log, and the general rules are related to the tracking requirements; then, based on the automatic identification mechanism and the general rules, the target content is identified in the log, and the target content in the log is extracted. The target content is the content that complies with the general rules; then the target content is converted into structured data to obtain the target data; finally, the log is analyzed based on the target data to monitor the application corresponding to the log, and troubleshoot and optimize the performance of the application. This method reduces the workload of manual tracking, improves the consistency and accuracy of tracking, facilitates the expansion and upgrade of the system, realizes real-time log collection, reduces the impact on system performance, and solves the problem that traditional log tracking collection usually requires manual writing and configuration of code, which is labor-intensive, error-prone, and lacks consistency and standardization in the collection and extraction process.
[0181] 2) The above-mentioned general log point collection and extraction device of the present application includes: a generation unit, an identification unit, a first processing unit and a second processing unit. The generation unit is used to use FlinkQL and automatically generate general rules based on regular expressions. The general rules are used to identify key information in the log, and the general rules are related to the point collection requirements; the identification unit is used to identify the target content in the log based on the automatic identification mechanism and the general rules, and extract the target content in the log. The target content is the content that complies with the general rules; the first processing unit is used to convert the target content into structured data to obtain the target data; the second processing unit is used to perform data analysis on the log based on the target data to monitor the application corresponding to the log, and perform troubleshooting and performance optimization on the application. The device reduces the workload of manual point collection, improves the consistency and accuracy of point collection, facilitates the expansion and upgrade of the system, realizes real-time log collection, reduces the impact on system performance, and solves the problem that traditional log point collection usually requires manual writing and configuration of code, which is labor-intensive, error-prone, and lacks consistency and standardization in the collection and extraction process.
[0182] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.
Claims
1. A general log point collection and extraction method, characterized in that: include: Use FlinkQL and regular expressions to automatically generate general rules, which are used to identify key information in logs and are related to tracking requirements. Based on the automatic recognition mechanism and the general rules, target content is identified in the log, and the target content in the log is extracted, wherein the target content is content that complies with the general rules; Convert the target content into structured data to obtain target data; The log is analyzed based on the target data to monitor the application corresponding to the log, and troubleshoot and optimize the performance of the application.
2. The method according to claim 1, characterized in that Use FlinkQL to automatically generate common rules based on regular expressions, including: Create a data storage table built on the Kafka data stream, the data storage table is used to store the filtered data of the log, the filtered data is obtained by filtering from the original data of the log based on the embedding requirement; Creating an extraction engine built on the Kafka data stream, the extraction engine is used to extract the target data from the filtered data stored in the data storage table, and the extraction rule of the extraction engine is established based on the regular expression; The general rule is generated according to the data storage table and the extraction engine.
3. The method according to claim 2, characterized in that The log includes a plurality of stream data. Before creating a data storage table built on the Kafka data stream, the method further includes: Analyzing the text of the log using natural language technology to extract keywords of each of the streaming data in the log; Using a classification model to classify each of the streaming data according to a keyword of each of the streaming data, to obtain a category corresponding to each of the streaming data; The streaming data whose category is a target category is determined as the filtered data, and the streaming data whose category is a non-target category is removed.
4. The method according to claim 1, characterized in that Before using FlinkQL and automatically generating common rules based on regular expressions, the method further includes: Obtain the tracking point requirements provided by the business; Input the tracking point requirement into the machine learning model, and output at least one construction parameter set of an initial regular expression, wherein the initial regular expression corresponds to the construction parameter set one by one, and one construction parameter set includes at least one parameter for constructing the corresponding initial regular expression; Using all of the construction parameter sets to construct the corresponding initial regular expressions, and determining to generate initial general rules based on all of the regular expressions; Extracting data from the log using the initial general rule to obtain initial extracted data; According to the matching degree between the initial extraction data and the embedding point requirements, the machine learning model is adaptively learned to output a target construction parameter set with an accuracy greater than or equal to a preset accuracy.
5. The method according to claim 4, characterized in that The tracking point requirement is input into the machine learning model, and at least one initial regular expression construction parameter set is output, including: Associating at least one preset rule according to the tracking point requirement; Perform multiple combinations according to all the preset rules to obtain at least one rule combination, wherein one rule combination includes at least one preset rule; Training is performed according to all the rule combinations to output at least one construction parameter set of an initial regular expression, wherein the initial regular expression, the construction parameter set and the rule combinations correspond one to one.
6. The method according to claim 4, characterized in that According to the matching degree between the initial extraction data and the tracking point requirement, the machine learning model is adaptively learned, including: According to the matching degree between the initial extraction data and the tracking point requirement, the initial extraction data is scored to obtain a corresponding first score, wherein the matching degree is positively correlated with the first score of the initial extraction data; According to the first score of the initial extracted data, scoring the preset rules in the rule combination corresponding to the initial extracted data to obtain a second score corresponding to each of the preset rules; According to the second score of each of the preset rules, the weight of each of the preset rules is adjusted to perform adaptive learning on the machine learning model, and the weight of the preset rule is positively correlated with the corresponding second score.
7. The method according to claim 6, characterized in that The rule combination corresponding to the initial extraction data includes a plurality of the preset rules, and the initial extraction data is scored according to the matching degree between the initial extraction data and the tracking point requirement to obtain a corresponding first score, including: Determine the product of the current weight of each of the preset rules in the rule combination corresponding to the initial extraction data and the initial extraction data and the matching degree as a corresponding sub-score, wherein the sub-score corresponds to the preset rule one by one; The sum of all the sub-scores is determined as a first score of the initial extracted data.
8. A general log point collection and extraction device, characterized in that: include: A generation unit, used to use FlinkQL and automatically generate general rules based on regular expressions, where the general rules are used to identify key information in the logs and are related to tracking requirements; an identification unit, configured to identify target content in the log based on an automatic identification mechanism and the general rule, and extract the target content in the log, wherein the target content is content that complies with the general rule; A first processing unit, configured to convert the target content into structured data to obtain target data; The second processing unit is used to perform data analysis on the log based on the target data to monitor the application corresponding to the log, and to troubleshoot and optimize the performance of the application.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the general log embedding point collection and extraction method described in any one of claims 1 to 7.
10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the general log point collection and extraction method described in any one of claims 1 to 7.
Citation Information
Cited By
Log burying point acquisition method and system
CN120407656A
