Log data processing method and device, electronic equipment, storage medium and program

Abstract

Embodiments of the invention disclose a log data processing method and apparatus, an electronic device, a storage medium and a program. The method comprises the steps of obtaining a to-be-stored log; when it is determined that the to-be-stored log is a storm log, storing the to-be-stored log according to a text file; and when it is determined that the to-be-stored log is a non-storm log, storing the to-be-stored log to a target database. According to the technical scheme, the stability of log management can be improved, and the maintenance cost of logs is reduced.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular, to a log data processing method, device, electronic device, storage medium and program. Background Art

[0002] Log management systems usually configure corresponding computing and storage resources according to the number of logs that need to be processed daily. In real-world scenarios, there are a large number of sporadic and uncontrollable log storms. When a log storm occurs, it often leads to unstable situations such as log data loss and processing delays due to insufficient processing performance of the log platform.

[0003] At present, there are two common ways to deal with log storms in related technologies. The first way is to suspend collection until the processing queue of the log platform is reduced to a normal value. The second way is to execute the pause / discard strategy before storage until the overall amount of data written to the database is reduced to a normal value.

[0004] In the process of implementing the present invention, the inventors found that the prior art has the following defects: the method of suspending log collection when a log storm occurs will lead to a large maintenance workload, and the log collection strategy needs to be adjusted every time. At the same time, manual intervention is also required. From discovering the storm to adjusting the collection strategy, the processing time required is relatively long, resulting in delay problems. The method of executing the suspension strategy before warehousing when a log storm occurs will cause the log platform to have no data updates for a long time in the scenario where the log storm lasts for a long time; the method of executing the discard strategy will damage the integrity of normal log data. Summary of the invention

[0005] The embodiments of the present invention provide a log data processing method, device, electronic device, storage medium and program, which can improve the stability of log management and reduce the maintenance cost of logs.

[0006] According to one aspect of the present invention, a log data processing method is provided, comprising:

[0007] Get the logs to be stored;

[0008] When it is determined that the log to be stored is a storm log, storing the log to be stored as a text file;

[0009] When it is determined that the log to be stored is a non-storm log, the log to be stored is stored in a target database.

[0010] According to another aspect of the present invention, there is provided a log data processing device, comprising:

[0011] A module for acquiring logs to be stored, used for acquiring logs to be stored;

[0012] A first log storage module, configured to store the log to be stored as a text file when it is determined that the log to be stored is a storm log;

[0013] The second log storage module is used to store the log to be stored in the target database when it is determined that the log to be stored is a non-storm log.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log data processing method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log data processing method described in any embodiment of the present invention when executed.

[0019] According to another aspect of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the log data processing method according to any embodiment of the present invention is implemented.

[0020] The embodiment of the present invention solves the problems of high log data maintenance cost and poor log management stability in the existing storm log processing method by storing the log to be stored as a text file when it is determined that the acquired log to be stored is a storm log, and storing the log to be stored in a target database when it is determined that the acquired log to be stored is a non-storm log. The stability of log management can be improved and the maintenance cost of the log can be reduced.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a log data processing method provided by Embodiment 1 of the present invention;

[0024] Figure 2 is a flow chart of a log data processing method provided by Embodiment 2 of the present invention;

[0025] Figure 3 This is a flow chart of a log management platform processing logs provided by Embodiment 2 of the present invention;

[0026] Figure 4 This is a schematic diagram of the internal processing flow of a preprocessing module in a log management platform provided by Embodiment 2 of the present invention;

[0027] Figure 5 is a schematic diagram of a log data processing device provided by Embodiment 3 of the present invention;

[0028] Figure 6 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 This is a flow chart of a log data processing method provided by the first embodiment of the present invention. This embodiment is applicable to the case of differentiated hierarchical storage of storm logs and non-storm logs. The method can be executed by a log data processing device, which can be implemented by software and / or hardware, and can generally be integrated in an electronic device. The electronic device can be a terminal device or a server device. As long as it can execute the log data processing method, the embodiment of the present invention does not limit the specific device type of the electronic device. Figure 1 As shown, the method includes the following operations:

[0033] S110: Obtain logs to be stored.

[0034] The logs to be stored may be logs collected by the log management platform and need to be stored.

[0035] In an embodiment of the present invention, the data processing and management process of the log can be realized through the log management platform. Optionally, the log management platform can be configured with a collection module, and the log data of each device can be configured and collected through the collection module to obtain the log to be stored. Optionally, the log management platform can perform log management on one or more different devices, and the embodiment of the present invention does not limit the type of device that the log management platform is connected to.

[0036] S120: When it is determined that the log to be stored is a storm log, store the log to be stored as a text file.

[0037] Among them, the storm log can be a type of log generated when a log storm problem occurs. Log storm refers to the phenomenon that a large number of logs are continuously generated under abnormal system conditions. It is understandable that the data volume of the storm log is usually large.

[0038] Correspondingly, after obtaining the log to be stored, the log management platform can determine the type of log to be stored. If it is determined that the log to be stored is a storm log, it can be stored as a text file. The so-called storage as a text file means that the stored log is treated as a text file, and the original text of the storm log is directly stored at the operating system level. Since the text file storage method has a higher storage compression ratio for log data, usually 1:20, and the ordinary database storage method has a relatively low storage compression ratio for log data, usually 1:2, storing storm logs with large data volumes in the form of text files can save storage space for log data and increase the speed of writing log data.

[0039] S130: When it is determined that the log to be stored is a non-storm log, store the log to be stored in a target database.

[0040] The non-storm log may be a log with a small amount of data generated by the device under normal circumstances. The target database may be a database for storing log data, as long as it can store log data, and the embodiment of the present invention does not limit the database type of the target database.

[0041] It can be seen that the above technical solution stores the logs as text files when it identifies the logs to be stored as storm logs, and stores them conventionally through the target database when it identifies the logs to be stored as non-storm logs, thereby realizing differentiated hierarchical management capabilities for log data. It can meet diverse log management needs by distinguishing log sources, distinguishing features in log sources, and distinguishing log storage objects. It can effectively store storm logs and non-storm logs while reducing storage and maintenance costs to ensure the integrity of normal log data.

[0042] The embodiment of the present invention solves the problems of high log data maintenance cost and poor log management stability in the existing storm log processing method by storing the log to be stored as a text file when it is determined that the acquired log to be stored is a storm log, and storing the log to be stored in a target database when it is determined that the acquired log to be stored is a non-storm log. The stability of log management can be improved and the maintenance cost of the log can be reduced.

[0043] Embodiment 2

[0044] Figure 2 is a flow chart of a log data processing method provided by the second embodiment of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, multiple specific optional implementation methods for determining whether the log to be stored is a storm log are provided. Figure 2 As shown, the method of this embodiment may include:

[0045] S210: Obtain the log to be stored, and calculate the quantitative index data of the log to be stored.

[0046] The quantitative indicator data may be data for quantifying indicators related to the logs to be stored.

[0047] Figure 3 FIG. 1 is a flow chart of a log management platform processing logs provided by Embodiment 2 of the present invention. In a specific example, Figure 3As shown, after the log management platform collects the logs to be stored through the collection module, it can send the logs to be stored to the preprocessing module. The preprocessing module can be used to extract key information from the logs to be stored and assign corresponding management attributes, etc., for subsequent log display, storage and calculation.

[0048] In an embodiment of the present invention, the preprocessing module in the log management platform is mainly responsible for parsing, labeling and other data enrichment processing of the logs to be stored, so that the logs to be stored complete the relevant pre-processing before being stored. Among them, the labeling processing can be to distinguish the storm logs and mark them. Other data enrichment processing contents may include but are not limited to supplementing some additional attribute management fields to the logs to be stored, such as the management party of the IP (Internet Protocol, a protocol for interconnection between networks) address, etc.

[0049] Figure 4 Schematic diagram of the internal processing flow of a preprocessing module in a log management platform provided by the second embodiment of the present invention. Figure 4As shown, the preprocessing module can complete the preprocessing process of the logs to be stored through the Source thread, Task queue, Worker thread, Sink queue and Sink thread. Specifically, the Source thread can be used to capture and push the logs to be stored. The collection module in the log management platform can be configured with multiple collectors, and the collector is responsible for the collection and acquisition of logs. The Source thread can be connected to each collector to obtain the log data to be stored collected by each collector. Optionally, a collector can be configured with multiple Source threads for log collection. For example, 10 Source threads can collect log data sent by a collector, etc. The Task queue can be used to aggregate the logs to be stored collected by each Source thread. Exemplarily, assuming that a collector corresponds to 10 Source threads for log collection, the log data of a collector collected by multiple Source threads can be aggregated into a Task queue. The Worker thread can be a thread used to execute background tasks or process parallel calculations in multi-threaded programming. In an embodiment of the present invention, the EPS (Event per Second) calculation process can be performed by the Worker thread in the preprocessing module to calculate the quantitative indicator data of the log to be stored. Among them, EPS calculation can be the amount of data that calculates the number of logs processed by the Worker thread every second. In addition, the Worker thread can also perform summary content extraction operations to extract important character content in the log to be stored. At the same time, the Worker thread can also perform subsequent Storm vocabulary matching processes to complete the Storm log judgment process. It is understandable that the Worker thread can also perform other general processing functions, such as parsing IP addresses and log line wrapping processing. The Sink queue can be a component used for log data transmission and processing, and its main function is to write log data to the corresponding system. For example, the Sink queue can be used to determine the storage path of the log to be stored, that is, whether to store it normally in the database or in the form of a text file. Correspondingly, the Sink thread can perform operations such as writing to the database or writing files to complete the final storage process of the log to be stored.

[0050] S220 , determining whether the quantitative index data of the log to be stored meets the log storm determination condition, if so, executing S240 , otherwise, executing S230 .

[0051] The log storm judgment condition may be a condition for judging whether the log to be stored is a storm log according to the quantitative index data of the log to be stored. Optionally, the log storm judgment condition may include at least one judgment condition.

[0052] In an embodiment of the present invention, the preprocessing module in the log management platform can determine whether the quantitative index data of the log to be stored meets the log storm judgment condition based on the calculated quantitative index data of the log to be stored, so as to quickly identify whether the log storm phenomenon occurs.

[0053] In an optional embodiment of the present invention, the quantitative indicator data may include a quantitative indicator of the number of log acquisition entries, a quantitative indicator of log processing time, a quantitative indicator of log processing writing, and a quantitative indicator of log average time consumption; the determination that the quantitative indicator data of the log data to be stored meets the log storm judgment condition may include: when it is determined that the month-on-month upward change rate of the quantitative indicator of the number of log acquisition entries exceeds a first threshold, the month-on-month upward change rate of the quantitative indicator of log processing time consumption and the month-on-month upward change rate of the quantitative indicator of log processing writing exceeds a second threshold, and the month-on-month upward change rate of the average log time consumption indicator exceeds a third threshold, it is determined that the quantitative indicator data of the log data to be stored meets the log storm judgment condition.

[0054] Among them, the quantitative index of the number of log acquisition entries can be an index of the average number of log data acquired per second within a time window. The quantitative index of log processing time consumption can be the average time consumption for preprocessing each log within a time window. The quantitative index of log processing writing can be the average time consumption for writing processed logs to the corresponding queue within a time window. The average log time consumption index can be the average time consumption for each log when storing logs within a time window. The first threshold, the second threshold and the third threshold can be specific thresholds configured according to the log storm judgment requirements, and their values ​​can be the same or different. The embodiment of the present invention does not limit the specific values ​​of the first threshold, the second threshold and the third threshold.

[0055] For example, Figure 4As shown, the quantitative index of log acquisition can be the average number of log data obtained from the task queue per second within a 10-second window, which can be called 10s_avg_input (eps). The quantitative index of log processing time consumption can be the average time consumption of the worker thread preprocessing each log within a 10-second window, in microseconds / event, which can be called 10s_avg_cost_queue (us / e). The quantitative index of log processing writing can be the average time consumption of the worker thread to complete the writing to the sink queue within a 10-second window, which can be called 10s_avg_cost_sink_queue (us / e). The time consumption of the sink queue processing becomes longer and the value becomes larger, which is used to indicate an increase in the probability of congestion. The average log time consumption index can be the average time consumption of each log in the sink thread within a 10-second window, which can be called 10s_avg_cost_sink (us / e). Optionally, the preprocessing module can collect and cache the corresponding indicator data every 30 seconds in the two processing links of the worker thread and the sink queue.

[0056] Correspondingly, after obtaining the data of each quantitative indicator, it can be further determined whether the quantitative indicator data of the log data to be stored meets the log storm judgment condition. Specifically, it can be determined whether the month-on-month upward change rate of the quantitative indicator of the number of log acquisition entries exceeds the first threshold, and whether the month-on-month upward change rate of the quantitative indicator of the log processing time consumption and the quantitative indicator of the log processing write exceeds the second threshold, and at the same time, it can be determined whether the month-on-month upward change rate of the average log time consumption indicator exceeds the third threshold. If the above judgment conditions are met at the same time, that is, the month-on-month upward change rate of the quantitative indicator of the number of log acquisition entries exceeds the first threshold, the month-on-month upward change rate of the quantitative indicator of the log processing time consumption and the quantitative indicator of the log processing write exceeds the second threshold, and the month-on-month upward change rate of the average log time consumption indicator exceeds the third threshold, then it can be determined that the quantitative indicator data of the log data to be stored meets the log storm judgment condition.

[0057] In a specific example, when the quantitative indicator data of the logs to be stored meet the following conditions at the same time, it can be determined that a log storm has occurred:

[0058] Condition 1: The input log quantity, i.e. the log acquisition number quantitative indicator 10s_avg_input (eps) has a month-on-month upward change rate of more than 5 times;

[0059] Condition 2: The processing time of the worker thread itself is satisfied at the same time: the log processing time quantitative index 10s_avg_cost_queue (us / e) and the log processing writing quantitative index 10s_avg_cost_sink_queue (us / e) of the time consumed by outputting to the target queue have an upward change rate of more than 1 times compared with the previous month;

[0060] Condition 3: The average log processing time indicator 10s_avg_cost_sink (us / e) of the target queue processing time has an upward change rate of more than 1 times compared with the previous month.

[0061] S230: Determine that the log to be stored is a non-storm log, and store the log to be stored in a target database.

[0062] For example, Figure 3 As shown, the log management platform may also include a routing module. The routing module may be used to configure and process the log data sent by the preprocessing module, and may distribute the log data with different tags to the corresponding storage units for differentiated storage. Among them, the database storage method may use an unstructured search engine to store normal log data, i.e., non-Storm logs, in the form of binary files.

[0063] S240: Obtain the target log field of the log to be stored.

[0064] The target log field may be a part of the fields screened out from the log to be stored, and may be used as a reference to determine whether the log to be stored is a storm log.

[0065] Accordingly, when it is determined that the quantitative index data of the log data to be stored meets the log storm judgment condition, it is determined that a log storm has occurred. In order to further filter out storm logs from the logs to be stored, the target log field of each log to be stored can be obtained to perform a subsequent storm log judgment process.

[0066] In an optional embodiment of the present invention, obtaining the target log field of the log to be stored may include: determining the target reference field in the log to be stored, and counting the total number of characters in the target reference field; and taking the first set number of target reference fields with the largest total number of characters as the target log field.

[0067] The target reference field may be a partial field selected from the log to be stored, and may be used to determine the target log field of the log to be stored. The first set number may be set according to actual needs, for example, 2, 3 or 5, etc. The embodiment of the present invention does not limit the specific value of the first set number.

[0068] Specifically, in order to obtain the target log field of the log to be stored, the target reference field that can be used as the basis for storm log judgment can be first screened from the log to be stored, and the total number of characters in each target reference field can be counted. It can be understood that different target reference fields may have the same or different total number of characters. Furthermore, a set number of target reference fields with the largest total number of characters can be used as target log fields. For example, the first three target reference fields with the largest total number of characters can be used as target log fields.

[0069] In an optional embodiment of the present invention, determining the target reference field in the log to be stored may include: replacing the time and date content of the log to be stored to obtain a fuzzy processed log; performing content segmentation on the fuzzy processed log by field to obtain an original word segmentation field; selecting a set number of alternative reference fields for the original word segmentation field; performing pattern recognition on the alternative reference fields, and deleting the pattern fields in the alternative reference fields to obtain the target reference field.

[0070] Among them, the obfuscated log may be the log data obtained after replacing the time and date content of the log to be stored. The original word segmentation field may be the field data obtained after the obfuscated log is segmented by field. The second set number can be set according to actual needs, for example, it can be 10 or 20, etc. The embodiment of the present invention does not limit the specific numerical value of the second set number. The alternative reference field may be a partial field filtered out from the original word segmentation field. The pattern field may be a field type with low reference significance for storm log judgment.

[0071] In an embodiment of the present invention, when it is necessary to determine the target reference field in the log to be stored, the time and date content of the log to be stored can be replaced to obtain a fuzzy log. Further, the fuzzy log is segmented by field to obtain the original word segmentation fields included in the fuzzy log. After obtaining the original word segmentation fields, some candidate reference fields with reference value can be further selected from the original word segmentation fields, and pattern recognition can be performed on the candidate reference fields to delete the pattern fields in the candidate reference fields, thereby obtaining the desired target reference field.

[0072] In a specific example, the logs to be stored generated during the log storm period can be identified according to the matching rules of "time / date", the "time / date" content in each log data to be stored can be identified, and the Make a replacement; the following is an example:

[0073] Assume that the original log to be stored is as follows:

[0074] <190> Oct 0717:08:542809109150008447(root)4424361f Traffic@FLOW:SESSION:10.219.70.195:50630->220.181.7.165:80(TCP),interface ethernet0 / 2,vrtrust-vr,policy 16,user-@-,host-,policy deny.

[0075] After matching the "time / date" rule, the above logs to be stored can be converted into the following fuzzy logs:

[0076] <190> 2809109150008447(root)4424361f Traffic@FLOW:SESSION:10.219.70.195:50630->220.181.7.165:80(TCP),interface ethernet0 / 2,vr trust-vr,policy 16,user-@-,host-,policy deny.

[0077] Furthermore, for each obfuscated log obtained by processing, the content can be segmented according to the symbols such as "space", ",", "<" and ">", and the content of the obfuscated log can be divided into word segmentation fields one by one. Exemplarily, the above obfuscated log can obtain a list of original word segmentation fields as shown in Table 1.

[0078] Table 1 List of original word segmentation fields

[0079]

[0080]

[0081] Furthermore, remove "time / date" from the original word segmentation field above. After the fields are selected, the first 10 fields are selected as candidate reference fields to obtain a list of candidate reference fields as shown in Table 2.

[0082] Table 2 List of alternative reference fields

[0083]

[0084] Furthermore, pattern recognition is performed on the field contents of the screened candidate reference fields, and pattern fields such as serial numbers and IP addresses that have low reference significance for storm log judgment are identified and deleted, so as to obtain the target reference fields in each storage log.

[0085] Correspondingly, after removing the pattern fields that hit "pattern recognition" from the candidate reference fields, the total number of characters included in the other candidate reference fields with character content is statistically ranked, and the top three fields with the highest ranking are taken as the target log fields, which can violate the "suspected storm content" field list. Table 3 is a target reference field list obtained by statistically ranking the total number of characters after the target reference fields are screened in Table 2 above. As shown in Table 3, based on the target reference field list, the top two fields with the highest ranking, namely Field5 and Field6, are taken as the target log fields.

[0086] Table 3 Target reference field list

[0087] Serial number Field Name Field Value Statistics 4 Field4 4424361f 10 5 Field5 Traffic@FLOW: 1000 6 Field6 SESSION: 1000 9 Field9 interface 10 10 Field10 ethernet0 / 2 10 11 Field11 vr 1

[0088] S250, determining whether the target log field matches the preset storm word library, if so, executing S260, otherwise, executing S270.

[0089] The preset storm word library may be a pre-built word library, which may be used to match the field type in the log to determine whether the log is a storm log.

[0090] S260: Determine that the log to be stored is the storm log, and store the log to be stored as a text file.

[0091] S270: Store the target log field into the preset storm word library, and store the log to be stored into the target database.

[0092] Correspondingly, after obtaining the target log field of the log to be stored, the target log field can be matched with the preset storm word library. If the target log field hits certain fields in the preset storm word library, it indicates that the target log field matches the preset storm word library, and the target log field is determined to be a log storm keyword. At this time, the corresponding log to be stored can be marked with a storm feature, such as adding a storm_flag=y label to the log to be stored to determine that it is a storm log. Further, the log to be stored that is determined to be a storm log can be stored as a text file. If the target log field does not hit certain fields in the preset storm word library, it indicates that the target log field does not match the preset storm word library, and the target log field screened out this time can be stored in the preset storm word library as a reference field for the next judgment of the storm log, but this batch of log data is not marked with the storm log label and can be stored normally in the target database. It can be understood that since the amount of data of the storm log that appears and is identified for the first time is usually not particularly large, storing it in the target database will not cause adverse effects.

[0093] In a specific example, Figure 3 As shown, the log management platform may also include a routing module. Optionally, the routing module may be responsible for configuring and processing log data to be stored with different labels, ensuring that the log data to be stored is correctly distributed to the corresponding storage unit. Specifically, if the log to be stored contains a storm label such as storm_flag=y, the storm routing rules may be executed to route to the operating system "text file" storage and store it as a text file. If the log to be stored does not contain a storm label, it will be directly stored in the target database as a normal log.

[0094] In a specific example, Figure 3 As shown, the log management platform may further include a presentation module for presenting stored log data, including simultaneously presenting normal log data stored in a conventional database storage and storm log data stored in an operating system via a text file.

[0095] The above technical solution has newly added a function of rapid identification of storm log content in the preprocessing module of the log management platform, realizing the automatic discovery capability of storm log content with high repetition rate; at the same time, a preset storm vocabulary is introduced in the preprocessing module, which improves the accuracy of log storm identification. At the same time, the preset storm vocabulary also has the function of automatic content update, which can reduce the operation and maintenance workload. The log management platform is equipped with a data routing function, which realizes the differentiated hierarchical management capability of log data, including distinguishing sources, distinguishing features in sources, and distinguishing target storage objects. The log data processing adopted by the above log management platform can ensure that when a log storm occurs, the overall availability of the log management platform and the integrity rate of log data remain at a normal availability level, while ensuring the integrity of log data, greatly reducing the processing and storage costs of storm logs.

[0096] Compared with the "pause collection method" and "pause / discard strategy method before storage" provided by the existing log management platform, the embodiment of the present invention fully summarizes the characteristic rules of log data storm generation. Through the combination strategy of "preset storm word library", "log content overview preprocessing" and "differential storage", it can automatically determine whether there is a log storm in the log management platform, which type of log data belongs to the storm log, and which belongs to the normal log data, and reasonably divert the log data of the storm part, so that the log management platform can achieve differentiated management of normal logs and storm logs without manual intervention, and achieve the effect of continuous and stable operation of the platform. Specifically, through the preprocessing process of the log to be stored, the log storm characteristics and storm content can be automatically identified, and then the differentiated storage of the storm log data can be achieved through routing. The normal log data still maintains the original preprocessing, storage and display data flow functions, so that the resource consumption of the log platform by the log storm is reduced to a controllable level, and the stability of the log management platform in managing logs is improved. At the same time, the initial version of the preset storm word library configured for the log management platform in the embodiment of the present invention can be created by industry experience value, and can be continuously and automatically updated later. The automatic update mechanism of the preset storm word library can be completed through the automatic incremental accumulation of the preprocessing module on the one hand, and can also be completed through the Web (World Wide Web, Global Wide Area Network) online editing method provided by the log management platform on the other hand. In other words, the daily maintenance of the preset storm word library does not require recompiling the program or modifying the logic code, and the overall maintenance cost is reduced.

[0097] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0098] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present invention.

[0099] Embodiment 3

[0100] Figure 5 is a schematic diagram of a log data processing device provided by Embodiment 3 of the present invention, such as Figure 5 As shown, the device includes: a to-be-stored log acquisition module 310, a first log storage module 320 and a second log storage module 330, wherein:

[0101] The log to be stored acquisition module 310 is used to acquire the log to be stored;

[0102] The first log storage module 320 is used to store the log to be stored as a text file when it is determined that the log to be stored is a storm log;

[0103] The second log storage module 330 is configured to store the log to be stored in a target database when it is determined that the log to be stored is a non-storm log.

[0104] The embodiment of the present invention solves the problems of high log data maintenance cost and poor log management stability in the existing storm log processing method by storing the log to be stored as a text file when it is determined that the acquired log to be stored is a storm log, and storing the log to be stored in a target database when it is determined that the acquired log to be stored is a non-storm log. The stability of log management can be improved and the maintenance cost of the log can be reduced.

[0105] Optionally, the first log storage module 320 is also used to: calculate the quantitative index data of the log to be stored; when it is determined that the quantitative index data of the log to be stored meets the log storm judgment condition, obtain the target log field of the log to be stored; when it is determined that the target log field matches the preset storm vocabulary, determine that the log to be stored is the storm log.

[0106] Optionally, the quantitative indicator data includes a quantitative indicator of the number of log acquisition entries, a quantitative indicator of log processing time, a quantitative indicator of log processing writing, and an average log time consumption indicator; the first log storage module 320 is also used to: when it is determined that the month-on-month upward change rate of the quantitative indicator of the number of log acquisition entries exceeds a first threshold, the month-on-month upward change rate of the quantitative indicator of log processing time consumption and the month-on-month upward change rate of the quantitative indicator of log processing writing exceeds a second threshold, and the month-on-month upward change rate of the average log time consumption indicator exceeds a third threshold, determine that the quantitative indicator data of the log data to be stored meets the log storm judgment condition.

[0107] Optionally, the first log storage module 320 is further used to: determine the target reference field in the log to be stored, and count the total number of characters in the target reference field; and use a first set number of target reference fields with the largest total number of characters as the target log field.

[0108] Optionally, the first log storage module 320 is also used to: replace the time and date content of the log to be stored to obtain a fuzzy processed log; perform content segmentation on the fuzzy processed log by field to obtain an original word segmentation field; select a second set number of alternative reference fields for the original word segmentation field; perform pattern recognition on the alternative reference field, and delete the pattern field in the alternative reference field to obtain the target reference field.

[0109] Optionally, the above device may further include a target log field storage module, which is used to store the target log field into the preset storm word library when it is determined that the target log field does not match the preset storm word library.

[0110] The above-mentioned log data processing device can execute the log data processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the log data processing method provided by any embodiment of the present invention.

[0111] Since the log data processing device introduced above is a device that can execute the log data processing method in the embodiment of the present invention, based on the log data processing method introduced in the embodiment of the present invention, the technical personnel in this field can understand the specific implementation of the log data processing device in this embodiment and its various variations, so how the log data processing device implements the log data processing method in the embodiment of the present invention is not described in detail here. As long as the technical personnel in this field implement the device used by the log data processing method in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0112] Embodiment 4

[0113] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0114] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0115] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0116] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a log data processing method.

[0117] Optionally, the log data processing method may include: obtaining the log to be stored; if it is determined that the log to be stored is a storm log, storing the log to be stored as a text file; if it is determined that the log to be stored is a non-storm log, storing the log to be stored in a target database.

[0118] In some embodiments, the log data processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the log data processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the log data processing method in any other appropriate manner (e.g., by means of firmware).

[0119] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0121] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0124] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0126] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A log data processing method, characterized in that: include: Get the logs to be stored; When it is determined that the log to be stored is a storm log, storing the log to be stored as a text file; When it is determined that the log to be stored is a non-storm log, the log to be stored is stored in a target database.

2. The method according to claim 1, characterized in that: The determining that the log to be stored is a storm log includes: Calculating the quantitative index data of the log to be stored; When it is determined that the quantitative index data of the log to be stored meets the log storm judgment condition, obtaining the target log field of the log to be stored; When it is determined that the target log field matches the preset storm vocabulary, the log to be stored is determined to be the storm log.

3. The method according to claim 2, characterized in that The quantitative index data includes a quantitative index of the number of log acquisition entries, a quantitative index of the log processing time consumption, a quantitative index of the log processing writing, and an average log time consumption index; The determining that the quantitative index data of the log data to be stored meets the log storm judgment condition includes: When it is determined that the month-on-month upward change rate of the quantitative indicator of the number of log acquisition entries exceeds the first threshold, the month-on-month upward change rate of the quantitative indicator of the log processing time consumption and the quantitative indicator of the log processing write exceeds the second threshold, and the month-on-month upward change rate of the average log time consumption indicator exceeds the third threshold, it is determined that the quantitative indicator data of the log data to be stored meets the log storm judgment condition.

4. The method according to claim 2, characterized in that: The step of obtaining the target log field of the log to be stored includes: Determine a target reference field in the log to be stored, and count the total number of characters in the target reference field; A first set number of target reference fields having the largest total number of characters are used as the target log fields.

5. The method according to claim 4, characterized in that The determining of the target reference field in the log to be stored includes: The time and date content of the log to be stored is replaced to obtain a fuzzy processed log; Segment the obfuscated log by field to obtain original segmentation fields; Selecting a second set number of candidate reference fields for the original word segmentation field; Pattern recognition is performed on the candidate reference fields, and pattern fields in the candidate reference fields are deleted to obtain the target reference field.

6. The method according to claim 2, characterized in that The method further comprises: When it is determined that the target log field does not match the preset storm word library, the target log field is stored in the preset storm word library.

7. A log data processing device, characterized in that: include: A module for obtaining logs to be stored, used to obtain logs to be stored; A first log storage module, configured to store the log to be stored as a text file when it is determined that the log to be stored is a storm log; The second log storage module is used to store the log to be stored in the target database when it is determined that the log to be stored is a non-storm log.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log data processing method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log data processing method described in any one of claims 1 to 6 when executed.

10. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the log data processing method according to any one of claims 1 to 6 is implemented.

Images