Log data processing method and device, electronic equipment and storage medium

By automatically extracting and analyzing the key field sets and intercepting times in the business log, identifying the business types that intercept exceptions, solving the problems of low efficiency and poor accuracy of log data processing in the existing technology, and achieving efficient and accurate log data processing .

CN119046098APending Publication Date: 2024-11-29BEIJING YOUTEJIE INFORMATION TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411181708.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In the prior art, in the face of large and complex business logs with large data volumes, there is a lack of effective analytical means, resulting in low detection efficiency, high labor and time costs, and there are mis-detection and missed detection, and the detection accuracy is poor.

Method used

By obtaining the business data in the business log, extracting the key field set, calculating the number of intercepted times of each business type, and identifying the business type of intercepting exceptions based on the historical intercept data, automatically displaying the corresponding interception management rules to achieve automatic detection.

Benefits of technology

It reduces the labor and time cost of log data processing, improves processing efficiency, avoids mis-checking and missed inspections, and improves detection accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046098A_ABST
    Figure CN119046098A_ABST
Patent Text Reader

Abstract

The invention discloses a log data processing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that after service data of all service events in a service log are obtained, key field sets corresponding to all the service events are obtained according to the service data of all the service events; then according to the key field set of each business event, obtaining the number of intercepted times of each business type; and finally, according to the historical interception data and the number of intercepted times of each service type, obtaining a target service type with interception abnormity, and displaying a target interception management rule corresponding to the target service type. Therefore, automatic detection and acquisition of the business type with the interception abnormity in the log data are realized, the labor cost and the time cost consumed for processing the log data are reduced, the log data processing efficiency is improved, the phenomena of wrong detection and missing detection are avoided, and the detection precision of the business type with the interception abnormity is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a log data processing method, device, electronic device and storage medium. Background Art

[0002] With the widespread popularity of the Internet, business systems have become a part of supporting daily operations of various institutions. As an important part of business systems, the speed and scale of business log generation are also increasing.

[0003] In the existing technology, there is a lack of effective parsing methods in the face of huge and complex business logs, and usually it still relies on manual analysis to obtain business events and business types with interception anomalies, and then update the interception management rules for business types with interception anomalies.

[0004] However, this analysis method not only consumes high manpower and time costs, but also has low data processing efficiency. In addition, due to the large detection errors of manual analysis methods, false detections or missed detections often occur, and the detection accuracy is poor. Summary of the invention

[0005] To solve the above technical problems, embodiments of the present invention provide a log data processing method, device, electronic device and storage medium to solve the problem of low detection efficiency of abnormal business events in business logs.

[0006] In a first aspect, an embodiment of the present invention provides a log data processing method, including:

[0007] Obtain the business data of each business event in the business log, obtain the key field sets corresponding to each business event based on the business data of each business event, obtain the number of interceptions of each business type based on the key field sets of each business event, wherein the interception of business events is related to the interception management rules, obtain the target business type with interception anomalies based on the historical interception data and the number of interceptions of each business type, and display the target interception management rules corresponding to the target business type.

[0008] In a second aspect, an embodiment of the present invention provides a log data processing device, including:

[0009] The business data acquisition module is used to obtain the business data of each business event in the business log. The key field set acquisition module is used to obtain the key field set corresponding to each business event based on the business data of each business event. The interception number acquisition module is used to obtain the number of interceptions of each business type based on the key field set of each business event, wherein the interception of business events is related to the interception management rules. The abnormal business acquisition module is used to obtain the target business type with interception exceptions based on historical interception data and the number of interceptions of each business type, and display the target interception management rules corresponding to the target business type.

[0010] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0011] At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log data processing method of any embodiment of the present invention.

[0012] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the log data processing method of any embodiment of the present invention when executed by a processor.

[0013] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the log data processing method of any embodiment of the present invention when executed by a processor.

[0014] The technical solution of the embodiment of the present invention, after obtaining the business data of each business event in the business log, first obtains the key field set corresponding to each business event according to the business data of each business event; then obtains the number of times each business type is intercepted according to the key field set of each business event; finally, according to the historical interception data and the number of times each business type is intercepted, obtains the target business type with interception anomalies, and displays the target interception management rules corresponding to the target business type. In this way, the automatic detection and acquisition of the business type with interception anomalies in the log data is realized, which not only reduces the manpower cost and time cost of processing log data, improves the efficiency of log data processing, but also avoids the occurrence of false detection and missed detection, and improves the detection accuracy of the business type with interception anomalies.

[0015] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flow chart of a log data processing method provided in Embodiment 1 of the present invention;

[0018] Figure 2 A flow chart of a log data processing method provided in Embodiment 2 of the present invention;

[0019] Figure 3 A flow chart of a log data processing method provided in Embodiment 3 of the present invention;

[0020] Figure 4 A flowchart of a log data processing method provided in Embodiment 4 of the present invention;

[0021] Figure 5 A structural diagram of a log data processing device provided in Embodiment 5 of the present invention;

[0022] Figure 6 A schematic diagram of the structure of an electronic device for implementing the log data processing method of embodiment 6 of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] Embodiment 1

[0026] Figure 1 This is a flow chart of a log data processing method provided in the first embodiment of the present invention. This embodiment can be applied to extract intercepted business events in business logs and display corresponding interception rules. This method can be executed by the log data processing device in any embodiment of the present invention. The log data processing device can be implemented in the form of hardware and / or software, and the log data processing device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0027] S101. Obtain business data of each business event in a business log.

[0028] In the embodiment of the present invention, the business system can be specifically understood as: a computer system or software application that supports daily operations and core business processes. These systems are usually used to automate, optimize and manage key business activities. The business log can be specifically understood as: a detailed activity record of business events recorded by the business system during operation, such as user operations, system events, error messages, etc. Business events can be specifically understood as: specific activities or changes with business significance related to business processes or business rules recorded in the business log, such as transaction records, process status, user behavior, system response, etc. Business data can be specifically understood as: various data generated in business events, which are specific information describing business events. For example, when an operation and maintenance event is intercepted, specific information about the interception will be recorded in its business data: for example, whether it is an overauthorized operation by non-operation and maintenance personnel, whether the number of operation and maintenance times reaches the threshold, whether the operation and maintenance event is during working hours, etc.

[0029] Specifically, after the actual business operation is generated, the business log recorded in the business system is obtained, so as to obtain the detailed activity record of all business events in the business log. The business data in the detailed activity record can provide data support for business personnel to maintain business processes and monitor abnormal business status. For a business system in which the business log is grouped according to the event identifier of the business event for data recording, all business data under a business event can be obtained in sequence in a sequential acquisition manner. Each business data occupies one or more data rows, and multiple data under the business event appear continuously. The first business data of the business event begins with a trigger time identifier containing year, month, day, hour, minute, second, millisecond and microsecond, which indicates the business processing starting point of the current business event in the business system. Other non-first business data begins with a trigger time identifier containing month, day, hour, minute, second, millisecond and microsecond, which indicates the processing time of the current business event in different processing nodes in the system. After all business data of a business event are displayed, all business data of the next business event will continue to be displayed. The time trigger identifier of the first business data is used as the sorting and distinction basis between each business event.

[0030] For a business system that records data in business logs according to system time, the business data of each business event may appear interspersed. Since each piece of business data includes the event identifier of its corresponding business event, all business data of the business event can be obtained by combining the event identifier with the trigger time identifier. That is, through the business event identifier, all business data under the current business event can be obtained by traversal query, and then the sorting result of each business data under the current business event is determined according to the trigger time identifier of each business data, so as to obtain all business data under the current business event; wherein, the event identifier can be specifically understood as: an identifier used to uniquely identify each business event or operation in the business log. The event identifier can help quickly retrieve and associate related records in the log, such as a unique ID or code, which can be used to distinguish different business events and quickly locate specific events in a large amount of log data.

[0031] Optionally, in an embodiment of the present invention, business data of each business event in a business log is obtained, including: obtaining continuous data items of each business event in sequence according to a trigger time identifier in the business log; if it is determined that a target business event has missing data items, obtaining the missing data items according to the event identifier of the target business event; and merging the continuous data items and the missing data items of the target business event to obtain complete business data of the target business event.

[0032] Specifically, in some business systems, part of the business data of a business event may appear continuously in the manner based on the event identifier grouping described above. The above continuously appearing business data is the continuous data item of the business event. At this time, the continuous data items of the above business event can be obtained in sequence according to the trigger time identifier in the business log; and other business data under the business event may appear in the manner based on the system time arrangement described above. At this time, for a business event, the continuous data item cannot fully reflect all the business information of the business event, that is, there is data missing. The above other business data arranged based on the system time, that is, the business data interspersed and distributed in different positions of the business log, can be regarded as the missing data item of the business event.

[0033] In addition, for business systems that record data in groups according to the event identifiers of business events, it is possible that due to system anomalies and other reasons, business data that should have appeared continuously does not appear continuously, but is instead interspersed and distributed in the business log in other ways. At this time, other business data other than the continuous data items are regarded as missing data items of the business event. Therefore, after obtaining the continuous data items of the current business event based on the trigger time identifier, if it is determined that the business event has missing data items, it is necessary to perform a global traversal query of the business log based on the event identifier of the current business event to obtain all missing data items related to the event identifier, and then merge the continuous data items and the missing data items based on the trigger time identifier to obtain the complete business data of the current business event. Therefore, when there are missing data items in the business event, the data integrity of the business event can be ensured based on the event identifier and the trigger time identifier, thereby avoiding the loss of business data. In particular, determining whether there is missing data in the current business event may include the following methods: determining the business type based on the event identifier of the current business event, and then determining whether the character length of the continuous data items of the current business event is greater than or equal to a preset minimum character threshold based on the business type; if it is greater than or equal to the minimum character threshold, it indicates that the business data has been completely acquired, that is, there are no data missing items; if it is less than the minimum character threshold, it indicates that the business data has not been acquired completely, that is, there are data missing items; or determining whether there is a processing end mark or an intermediate node mark in the continuous data items of the current business event based on the business type; if it is determined that there is a processing end mark or an intermediate node mark, it indicates that there are no data missing items; if it is determined that there is no processing end mark and an intermediate node mark, it indicates that there are data missing items.

[0034] S102: Acquire key field sets corresponding to respective business events according to business data of respective business events.

[0035] In the embodiment of the present invention, the key field set can be understood as the log entries and data information in the business log that are crucial for understanding the business process, monitoring business behavior, analyzing data quality, and performing abnormality troubleshooting, such as business code, product name, business type, business time, abnormal risk level, abnormal record, etc. Extracting the key field set from the business data of the current business event can be achieved by constructing a global regular expression; wherein the global regular expression is composed of multiple sub-regular expressions, each of which is used to extract data information under a key field, for example, a sub-regular expression is when traversing to " / TriggerTime" in the business data, extracting the 10 characters thereafter as the specific trigger time under the key field "trigger time", thereby pre-configuring the arrangement order of each sub-regular expression according to the order of appearance of each key field in the business data, and then obtaining the data information of all key fields of the current business event in sequence according to the global regular expression.

[0036] S103: Obtain the number of interceptions of each service type according to the key field set of each service event, wherein the interception of a service event is related to an interception management rule.

[0037] In the embodiments of the present invention, the business type can be specifically understood as: the classification of specific business activities or transactions performed. For example, in financial services, business types may include deposits, withdrawals, transfers, etc.; in e-commerce, they may include product browsing, ordering, payments, etc. The number of times a business type is intercepted can be specifically understood as: the number of times a business event under this business type is intercepted by the system because it does not comply with pre-established interception management rules. The interception management rules can be specifically understood as: a set of predefined policies or standards for automatically detecting and intercepting business events that do not comply with the above policies or standards.

[0038] Specifically, according to the key field set of each business event, data information of one or more key fields related to the execution status of the business event is obtained, and then business events with intercepted execution status are screened and obtained, and then according to the key field set of the current business event, data information of one or more key fields related to the business type is obtained to determine the business type to which the current business event belongs. Therefore, according to the business types to which all the intercepted business events belong, the number of interceptions of each business type in the business log can be obtained.

[0039] S104. According to the historical interception data and the number of interceptions of each service type, the target service type with interception anomalies is obtained, and the target interception management rules corresponding to the target service type are displayed.

[0040] In the embodiment of the present invention, the historical interception data can be specifically understood as: the records of the past business operations that were intercepted by the system due to non-compliance with the interception management rules. The target business type can be specifically understood as: the type corresponding to the specific business event that was intercepted. The target interception management rule can be specifically understood as: the interception rule triggered by the specific business event that was intercepted.

[0041] Specifically, the historical interception data includes the number of interceptions within a unit detection cycle (for example, one day) for each business type, and thus, through the interception number change curve or the change in the number of interceptions within adjacent unit detection cycles, the maximum daily change in the number of interceptions for each business type under normal interception status can be known. If the number of interceptions of the current business type, compared with the historical interception data, changes in the number of interceptions per day is greater than the above-mentioned maximum daily change, it indicates that there is an interception anomaly in the current business type, that is, the current business type is the target business type with interception anomaly; if the number of interceptions of the current business type, compared with the historical interception data, changes in the number of interceptions per day is less than the above-mentioned maximum daily change, it indicates that there is no interception anomaly in the current business type.

[0042] For each business type, its corresponding interception management rules are often composed of multiple interception management sub-rules. Taking the business type of modifying the main title of the website page as an example, its corresponding sub-interception management rules may include: if the user identity identifier that issues the modification operation is not an administrator, the current modification operation request is intercepted; if the modification time of the modification operation is not within the preset modification time, the current modification operation request is intercepted; if the web page identifier that issues the modification operation is not within the preset web page identifier, the current modification operation request is intercepted. Therefore, after detecting and obtaining the target business type with interception anomalies, the target interception management rules corresponding to the target business type are displayed to be intuitively displayed to business personnel in a visual manner. Optionally, in an embodiment of the present invention, the target interception management rules corresponding to the target business type are displayed, including: according to the interception management sub-rules that are hit respectively according to each intercepted business event under the target business type, each interception management sub-rule in the target interception management rule is displayed in turn.

[0043] Specifically, the interception management sub-rules can be understood as more detailed rules formulated for specific situations or specific types of business activities. They are specific implementation details under the interception management rules. When a business event is intercepted, it may hit one or more interception management sub-rules in the interception management rules corresponding to the current business type. The interception management sub-rule hit by the business event is processed by adding one to the number of effective interception times, and then the hit values ​​of the interception management sub-rules hit by other business events are obtained. After all business events of the current business type are counted, the number of effective interception times of each interception management sub-rule in the interception management rules corresponding to the business type can be obtained, and then the interception management sub-rules in the target interception management rules are displayed in order from most to least according to the number of effective interception times. This realizes the sorting display based on the number of effective interception times, and gives priority to the business personnel to display the interception management sub-rules that are most likely to cause interception anomalies, thereby improving the display effect of the interception management rules.

[0044] The technical solution of the embodiment of the present invention, after obtaining the business data of each business event in the business log, first obtains the key field set corresponding to each business event according to the business data of each business event; then obtains the number of times each business type is intercepted according to the key field set of each business event; finally, according to the historical interception data and the number of times each business type is intercepted, obtains the target business type with interception anomalies, and displays the target interception management rules corresponding to the target business type. In this way, the automatic detection and acquisition of the business type with interception anomalies in the log data is realized, which not only reduces the manpower cost and time cost of processing log data, improves the efficiency of log data processing, but also avoids the occurrence of false detection and missed detection, and improves the detection accuracy of the business type with interception anomalies.

[0045] Embodiment 2

[0046] Figure 2 This is a flow chart of a log data processing method provided by Embodiment 2 of the present invention. The relationship between this embodiment and the above embodiment is that when obtaining each key field through a global regular expression, there is a second key field that cannot be obtained. Figure 2 As shown, the method includes:

[0047] S201. Obtain business data of each business event in the business log.

[0048] S202: According to the global regular expression, obtain key fields of the first business event in sequence.

[0049] S203: If the second key field is not obtained in the business data of the specified number of digits after the first key field, the second key field is configured as a null value, and other key fields of the first business event are continuously obtained, wherein the second key field is a downstream key field adjacent to the first key field.

[0050] As described in the above technical solution, key fields can be obtained based on a global regular expression. However, in a traditional global regular expression, if one of the sub-regular expressions is written incorrectly and cannot be used to obtain matching key field information, then other sub-regular expressions following the sub-regular expression in the regular sequence will also fail to take effect, because the global regular expression will always obtain the corresponding key field information based on the current sub-regular expression until all business data of the current business event is traversed.

[0051] In order to solve the above problem, a corresponding traversal range can be configured between each key field, that is, between each sub-regular expression. For example, a specified number of bits is configured as the traversal range between the first key field and the second key field. After the data information of the first key field is obtained through the first sub-regular expression, the second key field is traversed and queried only within the specified number of bits after the position of the first key field through the second sub-regular expression. If the second key field is obtained, the data information of the subsequent key fields is continued to be obtained according to the third sub-regular expression; if the second key field is not obtained within the above-mentioned specified number of bits, the second key field is configured as a null value, and the second key field is skipped, and the third sub-regular expression is continued to be used to traverse and obtain the key field from the position where the specified number of bits ends.

[0052] This avoids the blocking of subsequent key field log information acquisition due to a log record error in a key field, and improves the flexibility and efficiency of key field information acquisition.

[0053] S204. Traverse the business data again by using the target local regular expression that matches the second key field to obtain the second key field.

[0054] A local regular expression is used to extract data information under a key field, and different key fields correspond to different local regular expressions; in particular, a local regular expression is different from the corresponding sub-regular expression in the global regular expression, and the two are constructed in different ways. Therefore, when the data information under the key field cannot be obtained through the sub-regular expression in the global regular expression, it does not mean that the data information under the key field cannot be obtained through the corresponding local regular expression; taking the above technical solution as an example, a sub-regular expression in the global regular expression is, when traversing to " / TriggerTime" in the business data, extract the subsequent 10 characters as the specific trigger time under the key field "trigger time", and the local regular expression corresponding to the sub-regular expression can be constructed as, when traversing to " / Trigger" in the business data, after an interval of 4 characters, extract the subsequent 10 characters as the specific trigger time under the key field "trigger time".

[0055] S205: Obtain the number of interceptions of each business type according to the key field set of each business event, wherein the interception of a business event is related to an interception management rule.

[0056] S206: According to the historical interception data and the number of interceptions of each service type, the target service type with interception anomalies is obtained, and the target interception management rules corresponding to the target service type are displayed.

[0057] The technical solution of the embodiment of the present invention is that after obtaining each key field of the first business event in sequence according to the global regular expression, if the second key field is not obtained in the business data of the specified number of digits after the first key field, the second key field is configured as a null value, and other key fields of the first business event are continuously obtained, and finally the business data is traversed again through the target local regular expression that matches the second key field to obtain the second key field. In this way, the data integrity of the obtained key fields is ensured, and the log information of subsequent key fields is blocked due to incorrect construction of the global regular expression or inaccurate log records of the key fields, thereby improving the flexibility of obtaining key field information.

[0058] Embodiment 3

[0059] Figure 3 A flowchart of a log data processing method provided in the third embodiment of the present invention. The relationship between this embodiment and the above embodiments is that when there are multiple data segments satisfying the key field trigger corresponding regular expression in the log file of the business event, and the data segment that abnormally triggers the key field regular expression is located upstream of the data segment that normally triggers the key field regular expression, resulting in erroneous acquisition of key field information, the key field information is obtained. Figure 3As shown, the method includes:

[0060] S301. Obtain business data of each business event in the business log.

[0061] S302: Acquire key field sets corresponding to respective business events according to business data of respective business events.

[0062] S303: According to the global regular expression, obtain key fields of the first business event in sequence.

[0063] S304: If it is determined that there is data anomaly in the third key field in the current key field set, the third key field is acquired again according to the location of the abnormal data of the third key field and the business data between the location of the fourth key field, wherein the fourth key field is a downstream key field adjacent to the third key field.

[0064] Specifically, when it is determined that there is a data anomaly in the third key field in the current key field set, it indicates that it may be because the error field appears before the third key field, resulting in the error field being traversed and obtained before traversing to the third key field, and the data information under the error field is used as the third key field, which causes the data information obtained for the third key field to be incorrect.

[0065] Exemplarily, if the third key field of the target business event is the modification time, the trigger flag of the corresponding sub-regular expression is " / ModificationTime", that is, the preset number of characters after " / ModificationTime" is used as the modification time obtained, and the fourth key field is the number of modifications, and the trigger flag of the corresponding sub-regular expression is " / ModificationNum", that is, the preset number of characters after " / ModificationNum" is used as the modification time obtained; if after the upstream key field of the third key field is obtained, when continuing to obtain the third key field, there is another field of the same " / ModificationTime" before the correct third key field " / ModificationTime", then it is obvious that the preset number of characters under this field will be used as the actual data information obtained.

[0066] But actually the third key field that you want to obtain refers to the latter " / ModificationTime". At this time, there is an exception in data acquisition under the third key field, that is, the data information of the previous " / ModificationTime" is mistakenly regarded as the data information of the third key field. At this time, since the above-mentioned erroneously obtained data information is located before the third key field, and the third key field is located before the adjacent downstream key field (that is, the fourth key field), then the third key field must be located between the above-mentioned erroneous data obtained and the fourth key field. Therefore, within the above-mentioned position interval, the third key field can be continued to be obtained through a sub-regular expression or a local regular expression that matches the third key field; if there is still an exception in the acquisition result, continue to acquire between the current erroneous data and the fourth key field through a sub-regular expression or a local regular expression that matches the third key field, until the data of the third key field obtained meets the preset data rules.

[0067] Among them, judging whether there are any abnormalities in the data of the third key field obtained can be performed based on the pre-configured preset data rules. The preset data rules include the digit characteristics and character characteristics of the data under the third key field. For example, if the third key field is the modification time, its beginning needs to be presented in the form of year, month and day, so the beginning of its data must be a number; if the third key field is address information, then its data must contain keywords such as "province" and "city", so that the preset data rules are used as evaluation criteria for whether there are any abnormalities in the data under the key field obtained.

[0068] S305: Obtain the number of interceptions of each business type according to the key field set of each business event, wherein the interception of a business event is related to an interception management rule.

[0069] S306: According to the historical interception data and the number of interceptions of each service type, the target service type with interception anomalies is obtained, and the target interception management rules corresponding to the target service type are displayed.

[0070] The technical solution of the embodiment of the present invention is that when there is a data anomaly in the third key field in the key field set, the third key field is obtained again according to the location of the abnormal data of the third key field and the business data between the location of the fourth key field, thereby avoiding mistaking the interference field before the third key field as the third key field, ensuring the accuracy of the key field acquisition result, and at the same time, according to the location of the abnormal data of the third key field and the location of the fourth key field, the third key field is accurately located, thereby improving the acquisition efficiency of the key fields.

[0071] Embodiment 4

[0072] Figure 4This is a flow chart of a log data processing method provided by the fourth embodiment of the present invention. The relationship between this embodiment and the above embodiments is that after obtaining the number of interceptions of each business type according to the key field set of each business event, an interception curve is drawn according to the historical interception data and the number of interceptions of each business type, and a dynamic change threshold is obtained to determine whether there is an interception anomaly in the current business type. Figure 4 As shown, the method includes:

[0073] S401. Obtain business data of each business event in the business log.

[0074] S402: Acquire key field sets corresponding to respective business events according to business data of respective business events.

[0075] S403: Obtain the number of interceptions of each business type according to the key field set of each business event, wherein the interception of a business event is related to an interception management rule.

[0076] S404: Draw an interception change curve based on the historical interception data of the current service type to determine whether there is an interception anomaly in the current service type.

[0077] The interception change curve reflects the correlation between the number of interceptions and time. According to the interception change curve, the dynamic change threshold of the number of interceptions is obtained, and according to the dynamic change threshold and the current number of interceptions, it is determined whether there is an interception anomaly in the current business type.

[0078] In the embodiment of the present invention, the interception change curve can be specifically understood as: a trend graph showing the number of times or frequency of interception of business operations over time within a certain time range. The horizontal axis of the curve is the interception time, and the vertical axis is the number of interceptions. The dynamic change threshold can be specifically understood as: the allowable fluctuation range of the parameters actually set for the parameters included in the business according to business needs.

[0079] Through the interception change curve, we can not only know the number of daily interceptions under the current business type, but also obtain the change value of the interception number between two adjacent days, and then multiply the average value or the maximum value of each change value by the preset weight coefficient (for example, the weight coefficient is greater than or equal to 1) to calculate the dynamic change threshold; the dynamic change threshold is used as an evaluation standard for whether there is an abnormality in the current number of interceptions of the current business type, that is, if the difference between the current number of interceptions and the number of yesterday's interceptions is greater than or equal to the dynamic change threshold, it indicates that there is an interception abnormality in the current business type; if the difference between the current number of interceptions and the number of yesterday's interceptions is less than the dynamic change threshold, it indicates that there is no interception abnormality in the current business type.

[0080] S405: If it is determined that there is an interception anomaly in the current service type, the target interception management rule corresponding to the target service type is displayed.

[0081] S406: If it is determined that there is no interception exception for the current business type, no display is performed.

[0082] The technical solution of the embodiment of the present invention draws an interception change curve based on the historical interception data of the current business type, and obtains the dynamic change threshold of the number of interceptions based on the interception change curve, and then judges whether there is an interception anomaly in the current business type based on the dynamic change threshold and the current number of interceptions. Therefore, by obtaining the interception change curve under the current business type, an accurate dynamic change threshold is obtained, so that the dynamic change threshold truly reflects the changing pattern of the number of interceptions of the current business type, and then judges whether there is an interception anomaly in the current business type based on the dynamic change threshold, which greatly improves the accuracy of the judgment result.

[0083] Embodiment 5

[0084] Figure 5 : is a structural block diagram of a log data processing device provided by Embodiment 5 of the present invention, the device specifically comprises:

[0085] The business data acquisition module 501 is used to acquire the business data of each business event in the business log;

[0086] The key field acquisition module 502 is used to acquire the key field set corresponding to each business event according to the business data of each business event;

[0087] The interception number acquisition module 503 is used to obtain the number of interceptions of each business type according to the key field set of each business event; wherein the interception of the business event is related to the interception management rule;

[0088] The abnormal business acquisition module 504 is used to acquire the target business type with interception abnormality according to the historical interception data and the number of interceptions of each business type, and display the target interception management rules corresponding to the target business type.

[0089] The technical solution of the embodiment of the present invention, after obtaining the business data of each business event in the business log, first obtains the key field set corresponding to each business event according to the business data of each business event; then obtains the number of times each business type is intercepted according to the key field set of each business event; finally, according to the historical interception data and the number of times each business type is intercepted, obtains the target business type with interception anomalies, and displays the target interception management rules corresponding to the target business type. In this way, the automatic detection and acquisition of the business type with interception anomalies in the log data is realized, which not only reduces the manpower cost and time cost of processing log data, improves the efficiency of log data processing, but also avoids the occurrence of false detection and missed detection, and improves the detection accuracy of the business type with interception anomalies.

[0090] Optionally, in an embodiment of the present invention, the business data acquisition module 501 is specifically used to: acquire continuous data items of each business event in sequence according to the trigger time identifier in the business log; if it is determined that there are missing data items in the target business event, acquire the missing data items according to the event identifier of the target business event; merge the continuous data items and the missing data items of the target business event to obtain complete business data of the target business event.

[0091] Optionally, in the embodiment of the present invention, the key field acquisition module 502 is specifically used to:

[0092] According to the global regular expression, each key field of the first business event is obtained in turn; if the second key field is not obtained in the business data of the specified number of bits after the first key field, the second key field is configured as a null value, and other key fields of the first business event are continued to be obtained; wherein the second key field is a downstream key field adjacent to the first key field; through the target local regular expression that matches the second key field, the business data is traversed again to obtain the second key field.

[0093] Optionally, in the embodiment of the present invention, the key field acquisition module 502 is further specifically used for:

[0094] If it is determined that there is a data anomaly in the third key field in the current key field set, the third key field is obtained again based on the location of the abnormal data of the third key field and the business data between the location of the fourth key field; wherein the fourth key field is a downstream key field adjacent to the third key field.

[0095] Optionally, in the embodiment of the present invention, the abnormal service acquisition module 504 is specifically used to:

[0096] According to each intercepted business event under the target business type, the interception management sub-rules hit respectively, and each interception management sub-rule in the target interception management rule is displayed in sequence.

[0097] Optionally, in the embodiment of the present invention, the abnormal service acquisition module 504 is further specifically used for:

[0098] Based on the historical interception data of the current business type, draw an interception change curve; wherein the interception change curve reflects the correlation between the number of interceptions and time; according to the interception change curve, obtain the dynamic change threshold of the number of interceptions, and based on the dynamic change threshold and the current number of interceptions, determine whether there is an interception anomaly in the current business type.

[0099] The above device can execute the log data processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the log data processing method provided by any embodiment of the present invention.

[0100] Embodiment 6

[0101] The invention also provides an electronic device, a storage medium and a computer program product. Figure 6 A schematic diagram of the structure of an electronic device for implementing the log data processing method of the fifth embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0102] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the ROM 12 or the computer program loaded from the storage unit 18 to the RAM 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0103] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0104] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a log data processing method.

[0105] In some embodiments, the log data processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on a heterogeneous hardware accelerator via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the log data processing method described above may be performed. Alternatively, in other embodiments, the processor may be configured to execute the log data processing method by any other appropriate means (e.g., by means of firmware).

[0106] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or electronic device.

[0108] In the context of the present invention, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein may be implemented on a heterogeneous hardware accelerator having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the heterogeneous hardware accelerator. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0110] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data electronic device), or a computing system that includes middleware components (e.g., an application electronic device), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), a blockchain network, and the Internet.

[0111] The computing system may include a client and an electronic device. The client and the electronic device are generally remote from each other and usually interact through a communication network. The relationship between the client and the electronic device is generated by computer programs running on corresponding computers and having a client-electronic device relationship with each other. The electronic device may be a cloud electronic device, also known as a cloud computing electronic device or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log data processing method, characterized in that: include: Obtain business data for each business event in the business log; According to the business data of each business event, obtain the key field set corresponding to each business event; According to the key field set of each business event, the number of interceptions of each business type is obtained; wherein the interception of the business event is related to the interception management rule; According to the historical interception data and the number of times each business type is intercepted, the target business type with interception anomalies is obtained, and the target interception management rules corresponding to the target business type are displayed.

2. The method according to claim 1, characterized in that The obtaining of the business data of each business event in the business log includes: According to the trigger time identifier in the business log, sequentially obtain the continuous data items of each business event; If it is determined that a data item is missing from the target business event, obtaining the missing data item according to the event identifier of the target business event; The continuous data items and missing data items of the target business event are merged to obtain the complete business data of the target business event.

3. The method according to claim 1, characterized in that The step of obtaining key field sets corresponding to each business event according to the business data of each business event includes: According to the global regular expression, each key field of the first business event is obtained in sequence; If the second key field is not obtained in the business data of the specified number of digits after the first key field, the second key field is configured as a null value, and other key fields of the first business event are continuously obtained; wherein the second key field is a downstream key field adjacent to the first key field; The business data is traversed again by using the target local regular expression that matches the second key field to obtain the second key field.

4. The method according to claim 3, characterized in that After obtaining the key fields of the first business event in sequence according to the global regular expression, the method further includes: If it is determined that there is a data anomaly in the third key field in the current key field set, the third key field is obtained again based on the location of the abnormal data of the third key field and the business data between the location of the fourth key field; wherein the fourth key field is a downstream key field adjacent to the third key field.

5. The method according to claim 1, characterized in that The target interception management rule corresponding to the target service type is displayed, including: According to each intercepted business event under the target business type, the interception management sub-rules hit respectively, and each interception management sub-rule in the target interception management rule is displayed in sequence.

6. The method according to claim 1, characterized in that The method of obtaining the target service type with interception anomaly according to the historical interception data and the number of interceptions of each service type includes: Draw an interception change curve based on the historical interception data of the current service type; wherein the interception change curve reflects the correlation between the number of interceptions and time; According to the interception change curve, a dynamic change threshold of the number of interceptions is obtained, and according to the dynamic change threshold and the current number of interceptions, it is determined whether there is an interception anomaly in the current service type.

7. A log data processing device, characterized in that: include: The business data acquisition module is used to obtain the business data of each business event in the business log; The key field acquisition module is used to obtain the key field set corresponding to each business event according to the business data of each business event; The interception times acquisition module is used to obtain the interception times of each business type according to the key field set of each business event; wherein the interception of the business event is related to the interception management rule; The abnormal business acquisition module is used to obtain the target business type with interception anomalies based on historical interception data and the number of interceptions of each business type, and display the target interception management rules corresponding to the target business type.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log data processing method according to any one of claims 1 to 6 when executed.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the log data processing method according to any one of claims 1 to 6.