Log data processing method, electronic equipment, storage medium and program

Abstract

The embodiment of the invention discloses a log data processing method, electronic equipment, a storage medium and a program, and the method comprises the steps: screening target type log data from original log data; screening a target storage field for the target type log data; and merging the target storage fields to obtain a merged storage log, and storing the merged storage log. According to the technical scheme, the log data storage cost and maintenance cost can be reduced, and meanwhile the utilization value of the log data is guaranteed.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular, to a log data processing method, device, electronic device, storage medium and program. Background Art

[0002] Log data is mainly generated by applications at runtime. When an application performs a specific operation, corresponding log data is generated, which records the application's operating status, error information, user operations, and other information. Log data is generated in a variety of ways, depending on the application's development language and framework.

[0003] Generally, an IT (Information Technology) system can generate many different types of logs, such as but not limited to security protection, application, system, and mail service logs. Different types of logs correspond to different amounts of data, and their requirements for storage resources are also different. Some types of logs, such as security protection logs, have a large amount of data, usually accounting for about 90% of the total amount of data in the entire IT system log. This type of log data, when managed in accordance with conventional methods, has the problem of occupying a large storage space and having low value utilization. At present, for log types with large data volumes, two mainstream storage methods are usually used. One of them is on-demand filtering, collection, and storage. That is, the default is not to collect data at ordinary times. When analysis is required, the collection and analysis process is turned on, and then the collection process is turned off. The second method is to expand the storage space to store complete log data. That is, the same management strategy is adopted for this type of log as other logs, and the collection and analysis of this part of the log is completed by expanding the storage space.

[0004] In the process of implementing the present invention, the inventors found that the prior art has the following defects: the on-demand filtering, collection and storage method for managing a certain type of log has a large maintenance workload, the collection strategy needs to be adjusted each time the log is collected, and the integrity of the log data cannot be guaranteed. The conventional method of managing a certain type of log by expanding the storage space is high in overall cost and consumes a large amount of storage resources. Summary of the invention

[0005] The embodiments of the present invention provide a log data processing method, device, electronic device, storage medium and program, which can reduce the log data storage cost and maintenance cost while ensuring the utilization value of the log data.

[0006] According to one aspect of the present invention, a log data processing method is provided, comprising:

[0007] Filter target type log data from raw log data;

[0008] Filtering target storage fields for the target type log data;

[0009] The target storage fields are merged to obtain a merged storage log, and the merged storage log is stored.

[0010] According to another aspect of the present invention, there is provided a log data processing device, comprising:

[0011] A target type log filtering module is used to filter target type log data from original log data;

[0012] A target storage field screening module, used for screening target storage fields for the target type log data;

[0013] The merged storage log storage module is used to merge the target storage fields to obtain a merged storage log, and store the merged storage log.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log data processing method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log data processing method described in any embodiment of the present invention when executed.

[0019] According to another aspect of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the log data processing method according to any embodiment of the present invention is implemented.

[0020] The embodiment of the present invention solves the problems of high storage cost, large maintenance workload and difficulty in ensuring log data integrity when storing log types with large data volumes by filtering target type log data from original log data, filtering target storage fields for target type log data, further merging the target storage fields to obtain a merged storage log, and storing the merged storage log. The embodiment of the present invention can reduce log data storage cost and maintenance cost while ensuring the utilization value of log data.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a log data processing method provided by Embodiment 1 of the present invention;

[0024] Figure 2 This is a flow chart of a method for providing application information provided by Embodiment 2 of the present invention;

[0025] Figure 3 This is a flow chart of a log management platform processing logs provided by Embodiment 2 of the present invention;

[0026] Figure 4 This is a schematic diagram of the effect of a first storage configuration rule list provided by Embodiment 2 of the present invention;

[0027] Figure 5 This is a schematic diagram of the effect of a first storage configuration rule configuration interface provided by Embodiment 2 of the present invention;

[0028] Figure 6 This is a schematic diagram of part of the original log data provided by the second embodiment of the present invention;

[0029] Figure 7 is a schematic diagram of a routing matching rule list provided in Embodiment 2 of the present invention;

[0030] Figure 8 This is a schematic diagram of the effect of a routing matching rule configuration interface provided by Embodiment 2 of the present invention;

[0031] Fig. 9 This is a schematic diagram of a mapping relationship list of merged primary keys and corresponding target storage fields provided by the second embodiment of the present invention;

[0032] Fig.10 It is a schematic diagram of a merged storage log list provided by the second embodiment of the present invention;

[0033] Fig.11 This is a schematic diagram of the effect of performing a log query based on a log query instruction provided by the second embodiment of the present invention;

[0034] Fig.12 is a schematic diagram of a log data processing device provided by Embodiment 3 of the present invention;

[0035] Fig.13 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] Embodiment 1

[0039] Figure 1 This is a flow chart of a log data processing method provided by the first embodiment of the present invention. This embodiment is applicable to the case where storage fields are filtered out from some types of filtered logs and then merged and stored. The method can be executed by a log data processing device, which can be implemented by software and / or hardware and can generally be integrated in an electronic device. The electronic device can be a terminal device or a server device. As long as the log data processing method can be executed, the embodiment of the present invention does not limit the specific device type of the electronic device. Accordingly, Figure 1 As shown, the method includes the following operations:

[0040] S110: Filter target type log data from original log data.

[0041] The original log data may be the log data originally collected by the log management platform from each device. The target type log data may be the log data that needs to selectively store some field contents. The embodiment of the present invention does not limit the log type of the target type log data.

[0042] It is understandable that log data has the characteristics of huge data volume and high repetitive content. Among the original log data collected by the log management platform, some types of log data may account for a relatively large proportion of the total log data, but the storage value is relatively low. For example, statistical analysis of the operation log data of the IT system can draw the following conclusions: 1. 90% of the log data is generated by security protection equipment, typically firewall equipment; 2. Among the log data of security protection equipment, 90% are traffic Session logs. For this type of log data, such as traffic Session logs, if they are managed according to conventional methods or the extended storage space method, there are problems of large storage space occupation and low value utilization. On-demand filtering and collection of such logs also have problems such as large maintenance workload and difficulty in ensuring the integrity of log data. Therefore, traffic Session logs can be filtered as target type log data.

[0043] S120: Filter target storage fields for the target type log data.

[0044] The target storage field may be a part of the fields to be stored that are screened out from the target type log data.

[0045] Since the storage value of target type log data is low, after obtaining the target type log data, the target type log data can be further filtered for target storage fields with higher storage value. It is understandable that the target storage field is usually an important field in the target type log data, and the key log information of the target type log data can be retained. Optionally, a target type log data can filter out multiple target storage fields. The field types of the target storage fields filtered by each target type log data can be the same.

[0046] S130: Merge the target storage fields to obtain a merged storage log, and store the merged storage log.

[0047] The merging and storing logs may be a log obtained by merging and processing the target storage fields filtered by the target type log data.

[0048] In the embodiment of the present invention, after the target storage field is obtained by screening, the target storage field can be merged and sorted according to a certain field merging rule to obtain a merged storage log. Correspondingly, after the merged storage log is obtained, the merged storage log can be stored.

[0049] Exemplarily, assuming that the target storage fields filtered out by target type log data A and target type log data B are field a and field b, and the field values ​​of field a filtered out by target type log data A and field a filtered out by target type log data B are the same, and the field values ​​of field b filtered out by target type log data A and field b filtered out by target type log data B are the same, then the target storage fields of target type log data A and target type log data B can be merged to obtain a merged storage log. Assuming that the target storage fields filtered out by target type log data A and target type log data B are field a and field b, and the field values ​​of field a filtered out by target type log data A and field a filtered out by target type log data B are different, and / or, the field values ​​of field b filtered out by target type log data A and field b filtered out by target type log data B are different, then the field a and field b filtered out by target type log data A can be merged and collated to obtain a merged storage log, and the field a and field b filtered out by target type log data B can be merged and collated to obtain a merged storage log.

[0050] It can be seen that the above technical solution does not use the storage method of the full amount of data for the selected target type log data, but selects the target storage fields with storage value for storage, thereby reducing the storage space of the target type log data. In addition, the embodiment of the present invention does not simply store the selected target storage fields, but merges the target storage fields before storing them, so as to store the key information in the target type log data and further significantly reduce the storage space requirement of the target type log data, thereby reducing the log data storage cost and maintenance cost, while ensuring the utilization value of the log data.

[0051] The embodiment of the present invention solves the problems of high storage cost, large maintenance workload and difficulty in ensuring log data integrity when storing log types with large data volumes by filtering target type log data from original log data, filtering target storage fields for target type log data, further merging the target storage fields to obtain a merged storage log, and storing the merged storage log. The embodiment of the present invention can reduce log data storage cost and maintenance cost while ensuring the utilization value of log data.

[0052] Embodiment 2

[0053] Figure 2This is a flow chart of a method for providing application information provided by the second embodiment of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, multiple specific optional implementation methods are provided for filtering target type log data from original log data, filtering target storage fields for target type log data, and merging and storing target storage fields. Figure 2 As shown, the method of this embodiment may include:

[0054] S210: Filter target type log data from original log data.

[0055] Figure 3 Schematic diagram of a flow chart of a log management platform processing logs provided by the second embodiment of the present invention. Figure 3 As shown, the log management platform can configure a collection module, and configure, collect and receive the log data of each device through the collection module to obtain the original log data. The log management platform can also configure a label module to assign corresponding label management attributes to different types of log numbers for subsequent display, storage and calculation, etc. Exemplarily, for firewall logs, the labels that can be assigned are "appname=firewall", tag=session, and for operating system logs, the labels that can be assigned are "appname=os, tag=secure". The log management platform can also configure a routing module to configure and process log data with labels, and distribute log data with different labels to corresponding storage units for storage. If it is conventional log data, conventional storage can be performed, such as storing it in the corresponding database. If it is a filtered target type log data, it can be clipped and stored. The log management platform can maintain a merge strategy module to configure the data source, merge rules and result output configuration for merging. When it is necessary to clip and store the target type log data, the target storage field can be merged in combination with the merge strategy module to obtain a merged storage log. The log management platform can also be configured with a presentation module for presenting data, and can present log data in conventional storage and log data in clipped storage at the same time. Optionally, the log data in clipped storage can include original target type log data and merged storage logs.

[0056] In an optional embodiment of the present invention, before filtering the target type log data from the original log data, it may also include: obtaining a first storage configuration rule for the original log data; wherein the first storage configuration rule includes a first segmentation time and a first data retention time; storing the original log data in different data storage units according to the first segmentation time; and trimming the original log data stored in the data storage unit according to the first data retention time.

[0057] The first storage configuration rule may be configuration data for storing the original log data. The first segmentation time may be the time for segmenting and storing the original log data, which may be used to centrally store the original log data within a certain time period. The first data retention time may be the time range for retaining the original log data.

[0058] In the embodiment of the present invention, optionally, the first storage configuration rules of the original log data may be managed in a list form, and each row in the list represents a first storage configuration rule. Figure 4 It is a schematic diagram of the effect of a first storage configuration rule list provided in Embodiment 2 of the present invention. Figure 5 Schematic diagram of the effect of a first storage configuration rule configuration interface provided by the second embodiment of the present invention. Figure 4 As shown in the figure, each first storage configuration rule may be configured with configuration data such as index name, total duration / maximum duration, used size and maximum limit. Figure 5 The first storage configuration rule configuration interface shown is used to configure the specific rule content of the corresponding first storage configuration rule. Figure 5 As shown, the split time in the first storage configuration rule configuration interface can be the first split time, and the deletion phase time can be the first data retention time. Specifically, the first split time and the first data retention time can be configured. It can be understood that the specific values ​​of the first split time and the first data retention time configured corresponding to different first storage configuration rules can be the same or different, and the embodiment of the present invention does not limit this.

[0059] Correspondingly, after obtaining the original log data, the original log data can be first matched with each first storage configuration rule, for example, a certain field in each original log data can be matched with the index name of the first storage configuration rule to determine the first storage configuration rule that the original log data matches. After determining the first storage configuration rule that the original log data matches, the set storage time period of the original log data can be determined by the first segmentation time in the first storage configuration rule to distribute the original log data to different storage buckets, so that the original log data is stored in different data storage units. In this way, data within the same period is stored in continuous storage units to optimize the search and deletion operations of different time ranges. For example, assuming that the first segmentation time is 1 day, a certain type of original log data generated within the first day can be stored in data storage unit A, and a certain type of original log data generated within the second day can be stored in data storage unit B. At the same time, the original log data stored in the data storage unit can also be trimmed according to the first data retention time in the matching first storage configuration rule. The first data retention time is also the retention time set for the original log data, and the original log data that exceeds the first data retention time can be marked as deleted. When the disk space used to store log data is lower than a certain threshold, such as when the idle rate is lower than 80%, the original log data with a deletion mark can be deleted in chronological order from old to new. The system time and the time of the oldest log data in the storage unit can be compared. If the time of the oldest log data in the storage unit is greater than the sum of the system time and the first data retention time, the original log data stored in the storage unit is marked with a "deletion mark". When the disk space is lower than a certain threshold, the original log data stored in the storage unit with a "deletion mark" is deleted in chronological order from old to new to free up disk space. When deleting the original log data of the storage unit with a "deletion mark", the original log data of a time period can be quickly deleted according to the log data stored at the first split time, thereby improving the efficiency of log deletion.

[0060] In an optional embodiment of the present invention, filtering target type log data from original log data may include: obtaining at least one routing matching rule for filtering the target type log data; wherein the routing matching rule includes a field matching rule and a feature matching rule; matching the original log data with the field matching rule in the routing matching rule; upon determining that the field content of the first target field in the original log data matches the field matching rule in the target routing matching rule, matching the target character feature of the original log data with the feature matching rule in the target routing matching rule; upon determining that the target character feature of the original log data matches the feature matching rule in the target routing matching rule, determining that the original log data is the target type log data.

[0061] Among them, the route matching rule can be a rule configured by the routing module in the log management platform, which can be used to filter the target type log data from the original log data. The field matching rule can be a rule for matching related fields. The feature matching rule can be a rule for matching related character features. The first target field can be a field that matches the field matching rule of one of the route matching rules. The target route matching rule can be a route matching rule whose field matching rule matches the first target field. The target character feature can be a character feature in the original log data that needs to be matched with the route matching rule.

[0062] In an embodiment of the present invention, a plurality of routing matching rules can be configured for the routing module of the log management platform to filter the target type log data from the original log data according to the routing matching rules. Optionally, the routing matching rules can include two types of sub-rules: field matching rules and feature matching rules. The field matching rules can be used to filter out the log type of a certain device type or a certain manufacturer's device. The feature matching rules can be used to match the key features of the target type log data. When performing routing matching, the original log data can be first matched with the field matching rules in each routing matching rule. If the field content of the first target field in the original log data matches the field matching rule in the target routing matching rule, the target character feature of the original log data can be further obtained, and the target character feature of the original log data can be matched again with the feature matching rule in the target routing matching rule. If the target character feature of the original log data matches the feature matching rule in the target routing matching rule, it can be determined that the original log data is the target type log data. Correspondingly, if the field content of the first target field in the original log data does not match the field matching rules in each routing matching rule, there is no need to match the character feature again, and it can be directly determined that the original log data is not the target type log data. If the field content of the first target field in the original log data matches the field matching rule in the target routing matching rule, but its target character feature does not match the feature matching rule in the target routing matching rule, it can also be determined that the original log data is not target type log data.

[0063] Figure 6 is a schematic diagram of part of the data in the original log data provided by the second embodiment of the present invention. In a specific example, Figure 6 As shown, each piece of original log data can be used as a record, and each field in the record can be called an "object". The object content in the original log data can be used as the reference data for matching the routing matching rules. Figure 7 is a schematic diagram of a routing matching rule list provided in Embodiment 2 of the present invention. Figure 8 Schematic diagram of the effect of a routing matching rule configuration interface provided by the second embodiment of the present invention. Figure 7 As shown in the figure, you can manage each route matching rule in the form of a list. Each line in the route matching rule list represents a route matching rule. After clicking "Edit" for each route matching rule, you can enter the following Figure 8 The routing matching rule configuration interface shown is used to configure rules, including but not limited to field matching rules and character feature matching rules. Optionally, the field matching rule can determine the field object and its reference value for matching, which is used to match the character content of the field. For example, appname=firewall can filter out the original log data with the appname field value of firewall. The character feature matching rule can determine the character features used to match the original log, and can be configured in the form of regular expressions. Exemplarily, the field object is fixed to "raw_message", and the default field value is empty, indicating that all content is matched, such as ".*FLOW.*". In the routing matching rule, field matching and feature matching have a cascading relationship, and character feature matching will be further performed only after the field matching is successful. Optionally, the routing matching rule can also configure the target storage area corresponding to the log data to specify the storage object name, such as the index name "firewall_session". The routing matching rules in the routing matching rule list can be matched sequentially. Once the original log data meets the first routing matching rule, routing distribution is performed, and subsequent routing matching rules are no longer matched.

[0064] S220: Storing the target type log data in the original data storage area.

[0065] The raw data storage area may be used to store the target type log data that has been preliminarily screened and has not undergone any data processing.

[0066] In a specific example, Figure 3 As shown, in addition to storing normal logs in the database for conventional storage, corresponding storage space can also be reserved in the database for tailored storage. Optionally, a raw data storage area can be reserved from the database first, and after the target type log data is filtered out from the raw log data, the target type log data can be first stored in the raw data storage area.

[0067] S230: Filter target storage fields for the target type log data.

[0068] For example, Figure 3 As shown, the target type log data can be first dynamically read from the original data storage area, and the target storage field can be filtered for the read target type log data.

[0069] S240: Concatenate and combine the target storage fields to obtain a merged primary key.

[0070] The merged primary key may be the field content obtained by concatenating and combining the target storage fields.

[0071] Accordingly, after filtering the target storage field for the target type log data, the target storage field can be spliced ​​and combined according to certain rules to obtain the merged primary key, so as to execute the subsequent target storage field merging process through the merged primary key. Optionally, the execution time interval for each target storage field splicing and combination task, the number of failed retries, the specific calculation formula of the task, and the storage location of the task result can be defined in the form of a scheduled task.

[0072] S250: Count the target storage fields with the same merge primary key to obtain a set of storage fields to be merged.

[0073] The storage field set to be merged may be a set for storing the storage fields to be merged. Optionally, the storage field set to be merged may include at least two storage fields to be merged; each storage field to be merged includes a target storage field corresponding to a target type log data.

[0074] It is understandable that a target storage field corresponding to a target type log data can generate a corresponding merged primary key, and the merged primary key can represent the core information of the target storage field corresponding to a target type log data. The same merged primary key can indicate that the target storage field corresponding to the merged primary key is the same, and at the same time indicates that the corresponding target type log data is likely to be duplicate log data with the same content. Therefore, after obtaining the merged primary key corresponding to each target storage field, the target storage fields with the same merged primary key can be counted to obtain the merged statistical number. Exemplarily, assuming that the merged statistical number of a merged primary key is 3, it indicates that the merged primary key corresponds to 3 types of storage fields to be merged, the field contents of the 3 types of storage fields to be merged are the same, and the original target type log data corresponding to the 3 types of storage fields to be merged are likely to be duplicate log data with the same content. Accordingly, the target storage fields with the same merged primary key can be added to the set of storage fields to be merged as storage fields to be merged.

[0075] Exemplarily, assuming that the contents of each field of a target storage field of a target type log data are field A1, field B1 and field C1, the merged primary key obtained by splicing and combining the target storage fields is A1-B1-C1. The contents of each field of a target storage field of another target type log data are field a1, field b1 and field c1, and the merged primary key obtained by splicing and combining the target storage fields is a1-b1-c1. If the contents of the merged primary key A1-B1-C1 and the merged primary key a1-b1-c1 are the same, then field A1, field B1 and field C1 can be used as one storage field to be merged, and field a1, field b1 and field c1 can be used as another storage field to be merged, and the above two storage fields to be merged are added to the set of storage fields to be merged corresponding to the merged primary key A1-B1-C1 and the merged primary key a1-b1-c1.

[0076] S260: Determine the log start time and the log end time according to the storage field set to be merged.

[0077] The log start time may be the earliest log generation time corresponding to the storage field to be merged in the storage field set to be merged, and the log end time may be the latest log generation time corresponding to the storage field to be merged in the storage field set to be merged.

[0078] It is understandable that the number of the storage field sets to be merged may be zero, one or more. Accordingly, when it is determined that there are storage field sets to be merged, the log generation time corresponding to each storage field to be merged in each storage field set to be merged may be analyzed to determine the earliest log time in each storage field set to be merged as the log start time, and determine the latest log time in each storage field set to be merged as the log end time.

[0079] S270: Calculate the difference between the log end time and the log start time to obtain the log continuous occurrence time.

[0080] The log continuous occurrence time may be a time period consisting of the log end time and the log start time.

[0081] Correspondingly, after determining the log start time and log end time of each storage field set to be merged, the difference between the log start time and log end time of each storage field set to be merged can be calculated respectively to obtain the log continuous occurrence time corresponding to each storage field set to be merged.

[0082] S280. Construct the merged storage log according to the merged primary key, the target storage field, the merged statistical quantity of the merged primary key, the log start time, the log end time, and the log continuous occurrence time.

[0083] The merged statistics quantity is the statistics quantity of the same merged primary key.

[0084] In the embodiment of the present invention, a merged storage log may be constructed and generated based on a target storage field corresponding to a merged primary key, a merged statistical quantity, a log start time, a log end time, and a log continuous occurrence time.

[0085] In a specific example, five target storage fields, namely source IP (src_ip), target IP (dst_ip), source port (src_port), target port (dst_port) and action (action), can be extracted from the target type log data, and the five target storage field values ​​can be concatenated into a merged primary key using a string concatenation method. The target storage fields can be separated by “:”, for example:

[0086] src_ip:10.219.70.195

[0087] src_port:51554

[0088] dst_ip:220.181.7.165

[0089] dst_port:80

[0090] action:Block

[0091] Then, the merged primary key Dedup_key generated by the corresponding splicing combination of the above target storage fields can be: Dedup_key=10.219.70.195:51554:220.181.7.165:80:Block.

[0092] Fig. 9 Schematic diagram of a mapping relationship list of merged primary keys and corresponding target storage fields provided by the second embodiment of the present invention. Fig. 9 As shown, according to the task execution time interval of the target storage field splicing and combining task as the statistical time range, the target storage field of each target type log data within the statistical time range is spliced ​​and combined to generate the merged primary key field Dedup_key. At the same time, the log generation time corresponding to each merged primary key can also be determined: Timestamp.

[0093] Furthermore, the field Dedup_key of the merged primary key is used as the statistical grouping field, and the Dedup_key with the same merged primary key field value is calculated to form the number of merged statistical quantities of each merged primary key, and the target storage field with the same merged primary key is added to the storage field set to be merged as the storage field to be merged. The earliest log generation time in the storage field set to be merged corresponding to the same merged primary key Dedup_key is read as the log start time start, and the latest log generation time is the log end time end, and end is subtracted from start, and the difference result is assigned as the log duration time and stored in the duration field. Fig.10 is a schematic diagram of a merged storage log list provided by the second embodiment of the present invention. Fig.10 As shown in the figure, a merged storage log can be constructed based on the merged primary key, each target storage field (i.e., src_ip, src_port, dst_ip, dst_port, action), the merged statistical quantity of the merged primary key, the log start time start, the log end time end, and the log duration duration to achieve deduplication of the target storage field. Fig.10 As shown, each row can represent a merged storage log.

[0094] S290. Store the combined storage log in a combined data storage area.

[0095] The merged data storage area may be used to store the merged storage log.

[0096] In the embodiment of the present invention, Figure 3 As shown, after the target type log data is clipped to generate a merged storage log, the merged storage log can be stored in the merged data storage area. The merged data storage area can be another data storage area that is reserved from the database and is different from the original data storage area. Optionally, the calculation results of each round of merged storage logs can be saved in the data storage unit space corresponding to the merged data storage area in a tail append manner through a scheduled task.

[0097] It should be noted that when storing the merged storage log in the merged data storage area, the target type log data corresponding to the merged storage log can be deleted from the original data storage area to ensure that the original data storage area and the merged data storage area do not store the same log information at the same time.

[0098] In an optional embodiment of the present invention, storing the merged storage log may include: obtaining a second storage configuration rule for the merged storage log; wherein the second storage configuration rule includes a second split time and a second data retention time; storing the merged storage log to different data storage units according to the second split time; the above method may also include: trimming the merged storage log stored in the data storage unit according to the second data retention time.

[0099] The second storage configuration rule may be configuration data for storing the merged storage log. The second segmentation time may be the time for segmenting and storing the merged storage log, which may be used to centrally store the merged storage log within a certain time period. The second data retention time may be the time range for retaining the merged storage log.

[0100] Specifically, when storing the merged storage log, the merged storage log can be first matched with each second storage configuration rule, for example, a certain field in each merged storage log can be matched with the index name of the second storage configuration rule to determine the second storage configuration rule that the merged storage log matches. After determining the second storage configuration rule that the merged storage log matches, the set storage time period of the merged storage log can be determined by the second split time in the second storage configuration rule to allocate the merged storage log to different storage buckets, so that the merged storage log is stored in different data storage units. In this way, data within the same period is stored in continuous storage units to optimize search and deletion operations in different time ranges. For example, assuming that the second split time is 1 day, a certain type of merged storage log generated within the first day can be stored in data storage unit A, and a certain type of merged storage log generated within the second day can be stored in data storage unit B. At the same time, the merged storage log stored in the data storage unit can also be trimmed according to the second data retention time in the matching second storage configuration rule. The second data retention time is also the retention time set for the merged storage log, and the merged storage log that exceeds the second data retention time can be marked as deleted. When the disk space used to store log data is lower than a certain threshold, such as when the idle rate is lower than 80%, the merged storage logs with deletion marks can be deleted in chronological order from old to new. The system time and the time of the oldest log data in the storage unit can be compared. If the time of the oldest log data in the storage unit is greater than the sum of the system time and the second data retention time, the merged storage log stored in the storage unit is marked with a "deletion mark". When the disk space is lower than a certain threshold, the merged storage logs stored in the storage unit with the "deletion mark" are deleted in chronological order from old to new to free up disk space. When deleting the merged storage logs stored in the storage unit with the "deletion mark", the merged storage logs of a time period can be quickly deleted according to the log data stored at the second split time, thereby improving the efficiency of log deletion.

[0101] In an optional embodiment of the present invention, after storing the merged storage log, it may also include: obtaining a log query instruction input by a user; wherein the log query instruction includes a query start time and a query end time; the log query instruction is used to indicate a query target type log data; when it is determined that the query start time is greater than the latest time of the log data in the merged data storage area, reading the target query log data from the original data storage area; when it is determined that the query end time is less than the earliest time of the log data in the original data storage area, reading the target query log data from the merged data storage area; when it is determined that the query start time is less than or equal to the latest time of the log data in the merged data storage area, and when it is determined that the query end time is greater than or equal to the earliest time of the log data in the original data storage area, reading the target query log data from the merged data storage area and the original data storage area.

[0102] Among them, the log query instruction can be used to query relevant logs from the target type log data indication stored in the log management platform. The query start time is also the starting time point input to the log management platform. The query end time is also the end time point input to the log management platform. The target query log data can be the target type log data that meets the query time requirement in the log query instruction.

[0103] In the embodiment of the present invention, Figure 3 As shown, the log management platform can also be configured with a display module to display the log data queried by the user, and can simultaneously display the normal logs stored conventionally by the user, and can also query the target type log data. When the user sends a log query instruction to the log management platform to indicate the query target type log data, the log management platform can automatically determine and integrate the original target type log data stored in the original data storage area and / or the merged storage log stored in the merged data storage area.

[0104] Fig.11 Schematic diagram of the effect of performing log query based on log query instruction provided by the second embodiment of the present invention. Fig.11As shown, when the user inputs a log query instruction for instructing to query the target type log data into the log management platform, the query start time and the query end time in the log query instruction can be analyzed. Specifically, if the query start time is greater than the latest time of the log data in the merged data storage area, the target type log data within the time range of the query start time and the query end time can be read from the original data storage area as the target query log data, and the target query log data is fed back to the user for display. If the query end time is less than the earliest time of the log data in the original data storage area, the target type log data within the time range of the query start time and the query end time can be read from the merged data storage area as the target query log data, and the target query log data is fed back to the user for display. If the query start time is less than or equal to the latest time of the log data in the merged data storage area, and the query end time is greater than or equal to the earliest time of the log data in the original data storage area, the target type log data within the time range of the query start time and the query end time can be read from the merged data storage area and the original data storage area as the target query log data, and the target query log data is fed back to the user for display. It can be understood that, as Fig.11 As shown, the later the time in the data storage area, the newer (now) the log generation time is; the earlier the time in the data storage area, the older (old) the log generation time is. Optionally, the query log data of the query can be presented to the user in the form of a data table or text list by appending the result at the end. Exemplarily, the data in rows 1 to 5 of the table of user feedback are the target query log data of the original data storage area corresponding to the query, and rows 6 to 20 are the target query log data of the merged data storage area corresponding to the query.

[0105] The above technical solution sets up a routing function in the log management platform, realizing the differentiated hierarchical management capability of log data, including distinguishing the source, distinguishing the characteristics in the source, and distinguishing the target storage object. At the same time, a storage clipping function is also set up to realize the differentiated management capability of storage policies, including distinguishing the storage duration, the parameters for deleting data, and the matching of storage policies and routing policies. The log management platform also sets up a data merging function to realize the merging management capability of highly duplicate data, including the merging rules, the content composition of the merged data, and the storage parameter management of the merged data. The log management platform also expands the data presentation function to realize the merged presentation of data from different storage units, including the simultaneous presentation of the original details, the merged data, and the data in other storage units.

[0106] Compared with the "on-demand filtering and collection method" and "expanded storage space method" provided by the current log management platform, the embodiment of the present invention performs differentiated management on highly repeated logs, extracts repeated features, and sets targeted merging / compression processing for repeated features, which has the following advantages: 1. The overall storage cost of log data is reduced. Taking "security class traffic session log data" as an example, the security five-tuple information therein is used as the standard for judging the repetition rate, that is, the five field values ​​of source IP (src_ip), target IP (dst_ip), source port (src_port), target port (dst_port), and action (action) are the same, that is, the merging condition is met. Through statistical practice data, the five-tuple repetition rate of security class traffic session logs is about 30%, that is, the storage space can be directly reduced by about 30%. In some scenarios where the source port does not need to be paid attention to, the four fields of source IP (src_ip), target IP (dst_ip), target port (dst_port), and action (action) are used as unique merging conditions, and the repetition rate can be increased to 80%, that is, the storage space is reduced by 80%. The data is converted from the original log into key five-tuple data and stored together, resulting in a reduction in data storage space. According to statistical practice data, a single original log is about 200 characters, and the five-tuple data is about 150 characters, which can reduce the storage space by about 25%. At the same time, the log data processing method provided by the embodiment of the present invention is provided through the Web (World Wide Web) interactive page method. Users do not need to master conventional development languages ​​and configuration scripts and other coding skills to store and manage special log types on demand, thereby improving the flexibility, convenience and user experience of log data processing.

[0107] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information (such as log data) are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0108] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present invention.

[0109] Embodiment 3

[0110] Fig.12 is a schematic diagram of a log data processing device provided by Embodiment 3 of the present invention, such as Fig.12 As shown, the device includes: a target type log screening module 310, a target storage field screening module 320 and a merged storage log storage module 330, wherein:

[0111] A target type log screening module 310 is used to screen target type log data from raw log data;

[0112] A target storage field screening module 320 is used to screen the target storage field of the target type log data;

[0113] The merged storage log storage module 330 is used to merge the target storage fields to obtain a merged storage log, and store the merged storage log.

[0114] The embodiment of the present invention solves the problems of high storage cost, large maintenance workload and difficulty in ensuring log data integrity when storing log types with large data volumes by filtering target type log data from original log data, filtering target storage fields for target type log data, further merging the target storage fields to obtain a merged storage log, and storing the merged storage log. The embodiment of the present invention can reduce log data storage cost and maintenance cost while ensuring the utilization value of log data.

[0115] Optionally, the target type log filtering module 310 is also used to: obtain at least one routing matching rule for filtering the target type log data; wherein the routing matching rule includes a field matching rule and a feature matching rule; match the original log data with the field matching rule in the routing matching rule; when it is determined that the field content of the first target field in the original log data matches the field matching rule in the target routing matching rule, match the target character feature of the original log data with the feature matching rule in the target routing matching rule; when it is determined that the target character feature of the original log data matches the feature matching rule in the target routing matching rule, determine that the original log data is the target type log data.

[0116] Optionally, the above-mentioned device also includes an original log data storage module, which is used to obtain a first storage configuration rule for the original log data; wherein the first storage configuration rule includes a first split time and a first data retention time; the original log data is stored in different data storage units according to the first split time; and a first cropping processing module is used to: crop the original log data stored in the data storage unit according to the first data retention time.

[0117] Optionally, the merged storage log storage module 330 is also used to: concatenate and combine the target storage fields to obtain a merged primary key; count the target storage fields with the same merged primary key to obtain a set of storage fields to be merged; wherein the set of storage fields to be merged includes at least 2 storage fields to be merged; each of the storage fields to be merged includes a target storage field corresponding to a target type log data; determine the log start time and the log end time according to the set of storage fields to be merged; calculate the difference between the log end time and the log start time to obtain the log continuous occurrence time; construct the merged storage log according to the merged primary key, the target storage field, the merged statistical number of the merged primary key, the log start time, the log end time and the log continuous occurrence time.

[0118] Optionally, the merged storage log storage module 330 is also used to: obtain a second storage configuration rule for the merged storage log; wherein the second storage configuration rule includes a second split time and a second data retention time; store the merged storage log in different data storage units according to the second split time; and a second trimming processing module is used to: trim the merged storage log stored in the data storage unit according to the second data retention time.

[0119] Optionally, the above device also includes a target type log data storage module, which is used to: store the target type log data in the original data storage area; the merged storage log storage module 330 is also used to: store the merged storage log in the merged data storage area.

[0120] Optionally, the above-mentioned device also includes a log query module, which is used to: obtain a log query instruction input by a user; wherein the log query instruction includes a query start time and a query end time; the log query instruction is used to indicate the query target type log data; when it is determined that the query start time is greater than the latest time of the log data in the merged data storage area, read the target query log data from the original data storage area; when it is determined that the query end time is less than the earliest time of the log data in the original data storage area, read the target query log data from the merged data storage area; when it is determined that the query start time is less than or equal to the latest time of the log data in the merged data storage area, and when it is determined that the query end time is greater than or equal to the earliest time of the log data in the original data storage area, read the target query log data from the merged data storage area and the original data storage area.

[0121] The above-mentioned log data processing device can execute the log data processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the log data processing method provided by any embodiment of the present invention.

[0122] Since the log data processing device introduced above is a device that can execute the log data processing method in the embodiment of the present invention, based on the log data processing method introduced in the embodiment of the present invention, the technical personnel in this field can understand the specific implementation of the log data processing device in this embodiment and its various variations, so how the log data processing device implements the log data processing method in the embodiment of the present invention is not described in detail here. As long as the technical personnel in this field implement the device used by the log data processing method in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0123] Embodiment 4

[0124] Fig.13 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0125] like Fig.13 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0126] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0127] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a log data processing method.

[0128] Optionally, the log data processing method may include: filtering target type log data from original log data; filtering target storage fields for the target type log data; merging the target storage fields to obtain a merged storage log, and storing the merged storage log.

[0129] In some embodiments, the log data processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the log data processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the log data processing method in any other appropriate manner (e.g., by means of firmware).

[0130] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0132] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0134] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0135] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0136] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0137] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A log data processing method, characterized in that: include: Filter target type log data from raw log data; Filtering target storage fields for the target type log data; The target storage fields are merged to obtain a merged storage log, and the merged storage log is stored.

2. The method according to claim 1, characterized in that The step of filtering target type log data from original log data includes: Acquire at least one routing matching rule for filtering the target type log data; wherein the routing matching rule includes a field matching rule and a feature matching rule; Matching the original log data with the field matching rule in the routing matching rule; In the case where it is determined that the field content of the first target field in the original log data matches the field matching rule in the target routing matching rule, matching the target character feature of the original log data with the feature matching rule in the target routing matching rule; In the case where it is determined that the target character feature of the original log data matches the feature matching rule in the target routing matching rule, the original log data is determined to be the target type log data.

3. The method according to claim 1, characterized in that Before filtering the target type log data from the original log data, the method further includes: Acquire a first storage configuration rule for the original log data; wherein the first storage configuration rule includes a first segmentation time and a first data retention time; storing the original log data in different data storage units according to the first split time; The original log data stored in the data storage unit is trimmed according to the first data retention time.

4. The method according to claim 1, characterized in that The step of merging the target storage fields to obtain a merged storage log includes: Concatenate and combine the target storage fields to obtain a merged primary key; Counting the target storage fields with the same merge primary key to obtain a set of storage fields to be merged; wherein the set of storage fields to be merged includes at least 2 storage fields to be merged; each storage field to be merged includes a target storage field corresponding to a target type log data; Determine the log start time and log end time according to the storage field set to be merged; Calculate the difference between the log end time and the log start time to obtain the log continuous occurrence time; The merged storage log is constructed according to the merged primary key, the target storage field, the merged statistical quantity of the merged primary key, the log start time, the log end time and the log continuous occurrence time.

5. The method according to claim 1, characterized in that The storing and merging the storage logs includes: Obtaining a second storage configuration rule for the merged storage log; wherein the second storage configuration rule includes a second segmentation time and a second data retention time; storing the merged storage log into different data storage units according to the second split time; The method further comprises: The merged storage log stored in the data storage unit is pruned according to the second data retention time.

6. The method according to claim 1, characterized in that After filtering the target type log data from the original log data, the method further includes: storing the target type log data in the original data storage area; The storing and merging the storage logs includes: The combined storage log is stored in a combined data storage area.

7. The method according to claim 6, characterized in that After storing the merged storage log, the method further includes: Obtaining a log query instruction input by a user; wherein the log query instruction includes a query start time and a query end time; the log query instruction is used to indicate the query target type log data; When it is determined that the query start time is greater than the latest time of the log data in the merged data storage area, reading the target query log data from the original data storage area; When it is determined that the query end time is less than the earliest time of the log data in the original data storage area, reading the target query log data from the merged data storage area; When it is determined that the query start time is less than or equal to the latest time of the log data in the merged data storage area, and when it is determined that the query end time is greater than or equal to the earliest time of the log data in the original data storage area, target query log data is read from the merged data storage area and the original data storage area.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log data processing method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log data processing method described in any one of claims 1 to 7 when executed.

10. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the log data processing method according to any one of claims 1 to 7 is implemented.

Images