Log generation method and device

Abstract

The invention provides a log generation method and device, and the method comprises the steps: responding to a selection operation of a target field in historical data, and generating a regular expression; based on the regular expression, extracting a target field from the historical data; generating a configuration operation interface containing the target field; generating a mapping rule of the target field according to the configuration operation of the user on the target field on the configuration operation interface; and generating a log generation rule according to the regular expression and the mapping rule. According to the method, technicians are liberated from repeated log generation rule configuration work, and meanwhile, the technical threshold of the log generation rule configuration work is reduced, so that common users can also configure log generation rules by themselves.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a log generation method and device. Background Art

[0002] There are many types of security products and network devices, and there is no unified standard for the log formats of different products. When collecting device logs, we mainly rely on manual adaptation of the log data format of each device based on the log samples provided by the manufacturer. The adaptation work is complicated and repetitive. This log solution has some shortcomings. On the one hand, it needs to be adapted repeatedly for different manufacturers and different products, and there is a lot of repetitive work; on the other hand, the technical threshold is high, and ordinary users cannot define the adaptation rules themselves. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present application provides a log generation method and device.

[0004] According to a first aspect of an embodiment of the present application, a log generation method is provided, the method comprising:

[0005] In response to a selection operation on a target field in the historical data, a regular expression is generated;

[0006] Based on the regular expression, extract the target field from the historical data; generate a configuration operation interface including the target field;

[0007] Generate a mapping rule for the target field according to the configuration operation of the user on the configuration operation interface;

[0008] A log generation rule is generated according to the regular expression and the mapping rule.

[0009] According to a second aspect of an embodiment of the present application, a log generating device is provided, the device comprising:

[0010] An expression module, for generating a regular expression in response to a selection operation on a target field in the historical data;

[0011] An interface module, used to extract the target field from the historical data based on the regular expression; and generate a configuration operation interface including the target field;

[0012] A mapping module, used to generate a mapping rule for the target field according to the configuration operation of the user on the target field in the configuration operation interface;

[0013] The log module is used to generate a log generation rule according to the regular expression and the mapping rule.

[0014] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including:

[0015] A memory and one or more processors; the memory is coupled to the processor; wherein the memory stores computer program code, the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method as described above.

[0016] According to a fourth aspect of an embodiment of the present application, there is provided a computer-readable storage medium, comprising computer instructions, which, when executed on an electronic device, enables the electronic device to execute the method as described above.

[0017] According to a fifth aspect of an embodiment of the present application, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the method described above.

[0018] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0019] This application provides a log generation method, which frees technical personnel from repetitive log generation rule configuration work, improves log generation efficiency, and lowers the technical threshold for configuring log generation rules, so that ordinary users can also configure log generation rules by themselves.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in the specification and constitute a part of this application, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0022] Figure 1 A first flow chart of the log generation method provided in the embodiment of the present application;

[0023] Figure 2 A second flow chart of the log generation method provided in the embodiment of the present application;

[0024] Figure 3 A third flow chart of the log generation method provided in the embodiment of the present application;

[0025] Figure 4 A schematic diagram of a log generation device provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application.

[0028] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0029] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0030] Next, the embodiments of the present application are described in detail.

[0031] The present application embodiment provides a log generation method, such as Figure 1 As shown, the method may include the following steps:

[0032] Step 110: generating a regular expression in response to a selection operation on a target field in the historical data;

[0033] Step 120: extract the target field from the historical data based on the regular expression; generate a configuration operation interface including the target field;

[0034] Step 130: Generate a mapping rule for the target field according to the configuration operation of the user on the configuration operation interface;

[0035] Step 140: Generate log generation rules according to the regular expression and the mapping rule.

[0036] Specifically, historical data is displayed on the interface, and the historical data includes multiple fields. In this embodiment, the field selected by the user from the historical data is called the target field, and the number of target fields is not limited in this embodiment. That is, in response to the user's selection operation on the target field in the historical data, this embodiment generates a regular expression, and the regular expression is used to extract the target field from the original data including but not limited to the historical data.

[0037] The purpose of this embodiment is to generate log generation rules, which are subsequently used as the basis for generating log data based on original data. Original data can be divided into historical data and target data according to the time sequence. This embodiment generates log generation rules based on historical data, and subsequently generates log data based on target data according to the log generation rules. Of course, log data can also be generated based on historical data according to the log generation rules.

[0038] Afterwards, this embodiment extracts the data content of the target field from the historical data based on the regular expression, and displays the target field in the configuration operation interface, prompting the user to perform configuration operations on the target field. Specifically, the target field includes a field name and a field value, and this embodiment supports the user to perform configuration operations on the field name and / or field value.

[0039] This embodiment generates a mapping rule for the target field according to the user's configuration operation on the target field in the configuration operation interface. The mapping rule here can be a direct assignment rule or a mapping assignment rule according to the user's configuration operation. The direct assignment rule refers to directly assigning a value to the field name / field value, and the mapping assignment rule refers to assigning a value to the field name / field value according to a certain mapping rule. For example, the value range of the field value is 0 to 10, where 0 to 3 is assigned to low, 4 to 7 is assigned to medium, and 8 to 10 is assigned to high.

[0040] In actual applications, the data types of the historical data of the device may be various. For different data types, this embodiment can generate corresponding regular expressions.

[0041] For example, when the data type of historical data is a key-value pair mode, the separator and connector are determined according to the characters before and after the target field, and the position of each target field after the connector is calculated. The matching group of the target field can be determined by the position, and finally the overall regular expression is obtained.

[0042] For another example, when the data type of historical data is a fixed separation mode, the separator is determined based on the characters before and after the target field, and the position of each target field after the separator is calculated. The matching group of the target field can be determined by the position, and finally the overall regular expression is obtained.

[0043] Therefore, if Figure 2As shown, as a preferred implementation, this embodiment can generate regular expressions in different ways according to the data type:

[0044] Step 210: determine the data type of the historical data;

[0045] Step 220: If the data type of the historical data is a key-value pair mode, the separator and connector are determined by the characters before and after the target field, and the process proceeds to step 230;

[0046] Step 230, calculating the position of the target field at the connector to generate a regular expression;

[0047] Step 240: If the data type of the historical data is a delimiter mode, the delimiter is determined by the target field, and the process proceeds to step 250;

[0048] Step 250: Calculate the position of the delimiter in the target field to generate a regular expression.

[0049] As a specific implementation method, for the convenience of description, when generating the log generation rule, the description information of the log generation rule can be generated at the same time, such as the rule name of the log generation rule, the asset type that generates the original data, the manufacturer, the device model and the log type, etc.

[0050] As described above, after generating the log generation rule, this embodiment may further include the following steps: Figure 3 As shown:

[0051] Step 310: Obtain device information of the target device;

[0052] Step 320: Match the device information with the description information of the log generation rule to obtain the target log generation rule;

[0053] Step 330: Generate log data of the target device according to the target log generation rule and the original data of the target device.

[0054] It can be seen from the above technical solutions that this embodiment aims to provide a log generation method, freeing technicians from the repetitive log generation rule configuration work, improving log generation efficiency, and at the same time lowering the threshold for configuring log generation rules, allowing ordinary users to configure log generation rules themselves.

[0055] The following is an introduction to the log generation method of this application by taking a practical application as an example.

[0056] The log generation method of this embodiment includes three steps: adaptation rule setting, automatic regular expression generation, and field matching. Each step is introduced below.

[0057] Rule settings: Set the description information of the log generation rule, including rule name, asset type, manufacturer, model and log type;

[0058] Automatic regular expression generation: Analyze the data type of given historical data, calculate the location of the target field connector and separator, and automatically generate regular expressions;

[0059] Field matching: Use automatically generated regular expressions to parse historical data, extract the field content of the target field, including the field name and field value, and configure a normalized mapping relationship for it.

[0060] The automatic regular generation process is as follows:

[0061] 11) Given historical data;

[0062] 12) Analyze the data type to determine whether it is a key-value pair or a fixed separation mode;

[0063] Key-value pair mode log example:

[0064] <187> Oct30

[0065] 11:24:152023FW-5040%%10ATK / 3 / ATK_IP4_UDP_FLOOD_SZ:SrcZoneName(1025)=DMZ;DstIPAddr(1007)=182.9. 1.3; RcvVPNInstance(1042)=; UpperLimit(1049)=1; Action(1053)=logging; BeginTime_c(1011)=1698635895

[0066] Fixed separation mode log sample:

[0067] nsfocus,V1,flow,HTTP,10.10.157.104,6948-5A1E-FCDE-F715,UTS,6629afe44721f8c27fc7d3cd08,1714008036,111.38.17.179,,00:50:56:b3:56:74,50702,TCP,http,Website browsing,10.202. 8.22,,b4:43:26:52:09:ec,9080,2024-04-2509:20:36,,,GET, / taf / common / taf / globalC onst.js,10.202.8.22:9080,,BIGipServerWwLTi+I89lRMhdHnNMcsNA=704956938.1575.0000

[0068] 13) If it is a key-value pair mode, the separator and connector are determined by the characters before and after the target field, and the position of each target field after the connector is calculated. The matching group of the target field can be determined by the position, and finally the overall regular expression is formed.

[0069] Example:

[0070] <188> Jul 25 17:25:22 2024JD_K05_17-18U_Nonrnd_T5080 %%10IPS / 4 / IPS_IPV4_INTERZONE:Protocol(1001)=TCP; Application(1002)=http; SrcIPAddr(1003)=10.88.132.58; SrcPort(1004)=38749; DstIPAddr(1007)=192.168.0.3; DstPort(1008)=8666; RcvVPNInstance(1042)=--; SrcZoneName(1025)=Trust; DstZoneName(1035)=Untrust; UserName(1113)=jason; PolicyName(1079)=passall;AttackName(1088 )=Sensitive_Information_On_Page(Apache_Tomcat_Error_Report1); AttackID(1089)=24678; Category(1090)=InformationDisc losure;Protection(1091)=WebServer;SubProtection(1092)=Tomcat;Severity(1087)=LOW;Action(1053)=Permit&Logging;CVE( 1075)=--;BID(1076)=--;MSB(1077)=--;HitDirection(1115)=reply;RealSrcIP(1100)=;SubCategory(1124)=RiskConfiguration

[0071] The above example selects the fields 10.88.132.58 and 38749. From the preceding and following strings, we can determine that the delimiter is ";" and the connector is "=". The first selected field 10.88.132.58 is after the third connector. The string before 10.88.132.58 is matched with (?:[^=\n]*=){3}, and the content of 10.88.132.58 itself is matched with ([^;]+). The second selected field 38749 is after the fourth connector, one position away from the first selected field. The string before 38749 is matched with (?:[^=\n]*=), and the content of 38749 itself is matched with ([^;]+).

[0072] 14) If it is a fixed delimiter mode, the delimiter is determined by the characters before and after the target field, and the position of each target field after the delimiter is calculated. The matching group of the target field can be determined by the position, and finally the overall regular expression is formed.

[0073] Example:

[0074] nsfocus,V1,flow,HTTP,10.10.157.104,6948-5A1E-FCDE-F715,UTS,6629afe44721f8c27fc7d3cd08,1714008036,111.38.17.179,,00:50:56:b3:56:74,50702,TCP,http,Website browsing,10.202.8.22,,b4:43:26:52:09:ec,9080,2024-04-2509:20:36,,,,,GET, / taf / common / taf / globalConst.js

[0075] In the above example, we can judge that the delimiter is "," by the preceding and following strings. The first selected field 111.38.17.179 is after the 9th delimiter. The string before 111.38.17.179 is matched with (?:[^,\n]*,){9}, and the content of 111.38.17.179 itself is matched with ([^,]+). The second selected field website browsing is after the 15th delimiter, 6 positions away from the first selected field. The string before website browsing is matched with (?:[^,\n]*,){6}, and the content of website browsing itself is matched with ([^,]+).

[0076] 15) If it is in JSON format, you can use the following general regular expression to parse out each field name and field value.

[0077] Example:

[0078] (?:\"|\")([^\"]+)(?:\"|\")\s*(?:\:|\∶)\s*(?:\"|\")?([^",]*)(?:,|(?:\"|\"),|(?:\"|\")}|\n}|"\n})

[0079] The field matching process is as follows:

[0080] 21) Parse historical data using the generated regular expression to obtain field names and field values.

[0081] 22) Configure the corresponding mapped field name and field value method for each field.

[0082] Direct assignment of values, such as copying the IP address directly to the source IP field;

[0083] The value mapping method is as follows: 1 is mapped to severe risk, and 2 is mapped to high risk.

[0084] 23) Save the log generation rules to the database, collect the original data, adapt it according to the manufacturer, device and model information, and select the corresponding regular analysis and field mapping.

[0085] Based on the same inventive concept, the present application also provides a log generation device, the structural diagram of which is as follows: Figure 4 As shown, the device comprises:

[0086] An expression module 410, for generating a regular expression in response to a selection operation on a target field in the historical data;

[0087] The interface module 420 is used to extract the target field from the historical data based on the regular expression and generate a configuration operation interface including the target field;

[0088] A mapping module 430, configured to generate a mapping rule for the target field according to a configuration operation performed by a user on the target field in the configuration operation interface;

[0089] The log module 440 is used to generate a log generation rule according to the regular expression and the mapping rule.

[0090] As a specific implementation, the target field includes a field name and a field value, and the mapping module generates a mapping rule for the target field in the following manner:

[0091] Generate a mapping rule for the field name and / or a mapping rule for the field value in the target field.

[0092] As a specific implementation, the expression module generates a regular expression in the following manner:

[0093] The data type of the historical data is determined, and a regular expression is generated according to the data type, where the data type is a key-value pair mode or a fixed separation mode.

[0094] An embodiment of the present application provides an electronic device, which may include: a memory and one or more processors. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device may perform various functions or steps of the above method embodiment.

[0095] The structure of the electronic device can refer to Figure 5 The structure of the electronic device 100 is shown.

[0096] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0097] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes each function or step of the above method embodiment.

[0098] The computer-readable storage medium includes but is not limited to any one of the following: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0099] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to perform each function or step of the above method embodiment.

[0100] Among them, the electronic device, computer-readable storage medium, and computer program product provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0101] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0102] In the several embodiments provided in the present application, it should be understood that the disclosed method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of modules or units, which can be electrical, mechanical or other forms.

[0103] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0104] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A log generation method, characterized in that: The method comprises: In response to a selection operation on a target field in the historical data, a regular expression is generated; Based on the regular expression, extract the target field from the historical data; generate a configuration operation interface including the target field; Generate a mapping rule for the target field according to the configuration operation of the user on the configuration operation interface; A log generation rule is generated according to the regular expression and the mapping rule.

2. The method according to claim 1, characterized in that The target field includes a field name and a field value. The method specifically generates a mapping rule for the target field in the following manner: Generate a mapping rule for the field name and / or a mapping rule for the field value in the target field.

3. The method according to claim 1, characterized in that The method specifically generates a regular expression in the following manner: The data type of the historical data is determined, and a regular expression is generated according to the data type, where the data type is a key-value pair mode or a fixed separation mode.

4. The method according to claim 1, characterized in that: The method further comprises: Generate description information of the log generation rule, the description information including at least one of the following: rule name, asset type, manufacturer, device model, and log type.

5. The method according to claim 4, characterized in that The method further comprises: Get the device information of the target device; Matching the device information with the description information of the log generation rule to obtain a target log generation rule; Generate log data of the target device according to the target log generation rule.

6. The method according to claim 1, characterized in that The mapping rule is a direct assignment rule or a mapping assignment rule.

7. A log generating device, characterized in that: The device comprises: An expression module, for generating a regular expression in response to a selection operation on a target field in the historical data; An interface module, used to extract the target field from the historical data based on the regular expression; and generate a configuration operation interface including the target field; A mapping module, used to generate a mapping rule for the target field according to the configuration operation of the user on the target field in the configuration operation interface; The log module is used to generate a log generation rule according to the regular expression and the mapping rule.

8. The device according to claim 7, characterized in that The target field includes a field name and a field value. The mapping module generates a mapping rule for the target field in the following manner: Generate a mapping rule for the field name and / or a mapping rule for the field value in the target field.

9. The device according to claim 7, characterized in that The expression module specifically generates a regular expression in the following manner: The data type of the historical data is determined, and a regular expression is generated according to the data type, where the data type is a key-value pair mode or a fixed separation mode.

10. An electronic device, characterized in that: include: A memory and one or more processors; the memory is coupled to the processor; wherein the memory stores computer program code, the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-6.

11. A computer-readable storage medium comprising computer instructions, characterized in that: When the computer instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 6.

12. A computer program product, characterized in that When the computer program product is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images