Method, device, equipment and medium for detecting abnormality of indicator data in call chain log

Abstract

The present invention discloses an abnormality detection method, device, equipment and medium for indicator data under call chain logs. The method includes: collecting business logs, and aggregating each business log into a call chain aggregate log corresponding to each call chain; performing field value data processing on each business chain aggregate log to obtain multiple indicator data and indicator types corresponding to each call chain aggregate log; clustering each indicator data of the same indicator type in the same business chain aggregate log to obtain an indicator data set corresponding to each indicator type; inputting each indicator data set into a pre-trained isolation forest model to identify abnormal indicator data sets, and performing abnormal alarms according to the target indicator type. The embodiment of the present invention first aggregates the call chain log and then performs abnormality detection on the indicator data under the aggregate log, thereby realizing rapid positioning and abnormal diagnosis of abnormal indicator data of the entire chain, and realizing efficient business quality monitoring.

Description

Technical Field

[0001] The present invention relates to the field of call chain log analysis, and in particular to a method, device, equipment and medium for detecting abnormality of indicator data in a call chain log. Background Art

[0002] A complete business process is composed of a series of business function units connected in series. Each business is implemented by a small functional module logic. When performing business quality monitoring, the existing technology often needs to find a unified business ID, and then find all related call records under a business request to form a call chain to realize the series connection of multiple functional modules under the microservice.

[0003] However, with the launch of containers and microservice architectures in the third-generation system, a business transaction may involve multiple functional modules, which may be developed by different teams, implemented in different programming languages, and distributed on multiple servers or even across multiple data centers. This results in the lack of a unified business ID between functional modules, making the quality monitoring, abnormal diagnosis and location of a certain business extremely complicated, and the operation and maintenance response time is getting longer and longer.

[0004] Therefore, there is an urgent need for a method to detect anomalies in indicator data under the call chain log, so as to establish an end-to-end business link and thus build a complete business quality monitoring process under a microservice architecture. Summary of the invention

[0005] Based on this, the present invention provides a method, device, equipment and medium for detecting anomalies in indicator data under a call chain log, so as to solve the problem that in the process of business quality monitoring, some logs lack identifying business IDs, resulting in complex business quality monitoring and anomaly diagnosis and long operation and maintenance response time.

[0006] In a first aspect, an embodiment of the present invention provides a method for detecting abnormality of indicator data in a call chain log, the method comprising:

[0007] During the detection period, business logs are collected in each business function module corresponding to the target business function, and each business log is aggregated into a call chain aggregate log corresponding to each call chain;

[0008] Perform field value data processing on each business chain aggregation log to obtain multiple indicator data corresponding to each call chain aggregation log and indicator types corresponding to each indicator data;

[0009] Clustering the indicator data of the same indicator type in the same business chain aggregation log to obtain an indicator data set corresponding to each indicator type;

[0010] Each indicator data set corresponding to each business chain aggregation log is input into the pre-trained isolation forest model to identify abnormal indicator data sets, and abnormal alarms are issued according to the target indicator type to which each abnormal indicator data set belongs.

[0011] In a second aspect, an embodiment of the present invention further provides a device for detecting abnormality of indicator data in a call chain log, the device comprising:

[0012] A call chain log aggregation module is used to collect business logs in each business function module corresponding to the target business function during the detection period, and aggregate each business log into a call chain aggregation log corresponding to each call chain;

[0013] The field value data processing module is used to process the field value data of each business chain aggregation log respectively, and obtain multiple indicator data corresponding to each call chain aggregation log, and the indicator type corresponding to each indicator data;

[0014] The indicator data aggregation module is used to cluster the indicator data of the same indicator type in the same business chain aggregation log to obtain the indicator data set corresponding to each indicator type;

[0015] The abnormal alarm module is used to input each indicator data set corresponding to each business chain aggregation log into a pre-trained isolation forest model, identify abnormal indicator data sets, and issue abnormal alarms according to the target indicator type to which each abnormal indicator data set belongs.

[0016] In a third aspect, an embodiment of the present invention further provides a device, the device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a method for detecting anomalies in indicator data under a call chain log as described in any embodiment of the present invention.

[0020] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a method for detecting anomalies in indicator data under a call chain log as described in any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention first aggregates multiple business logs under the same business chain, and then processes the field values ​​in all business logs to obtain a number of indicator data, and determines the specific indicator data with abnormalities by calculating the abnormal score value of the indicator data, thereby realizing the rapid positioning and abnormal diagnosis of abnormal indicator data in the entire chain, and providing an efficient solution for business quality monitoring under the microservice architecture.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 This is a flowchart of a method for detecting abnormality of indicator data in a call chain log according to Embodiment 1 of the present invention;

[0025] Figure 2 is a flowchart of another method for detecting abnormality of indicator data in a call chain log provided in Embodiment 2 of the present invention;

[0026] Figure 3 It is a reference diagram of sudden increase and change of average time consumption indicator data under a time consumption indicator type applicable to an embodiment of the present invention;

[0027] Figure 4 It is a reference graph of the increase in the number of errors of error number indicator data under an error type indicator type applicable to an embodiment of the present invention;

[0028] Figure 5 It is a structural diagram of a device for detecting abnormality of indicator data in a call chain log provided according to Embodiment 3 of the present invention;

[0029] Figure 6 It is a structural schematic diagram of an electronic device that implements a method for detecting anomaly of indicator data under a call chain log according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] Figure 1 This is a flowchart of a method for detecting abnormality of indicator data in a call chain log provided by the first embodiment of the present invention. This embodiment can be applied to the case of detecting abnormality of indicator data in a call chain log under a microservice architecture. The method can be executed by a device for detecting abnormality of indicator data in a call chain log. The device can be implemented in the form of hardware and / or software. The device can be configured in an electronic device with data processing function. Figure 1 As shown, the method includes:

[0034] S110 . During the detection period, business logs are collected in each business function module corresponding to the target business function, and each business log is aggregated into a call chain aggregate log corresponding to each call chain.

[0035] The target business function can be understood as the complete logical process of handling a specific business. Under the microservice architecture, the handling process of the specific business is divided into multiple distributed microservices that work together to complete it. Since each distributed microservice is implemented by a corresponding business function module, and each business function module contains its own exclusive business log, when quality monitoring of a certain business is carried out within a certain detection period, it is first necessary to collect all the business function modules included in the business function implementation and all the business log information under each business function module, and then aggregate the log modules belonging to the same call chain according to the common characteristics in the business log information, that is, restore the end-to-end execution process of the business, and then construct a complete business process under each call chain.

[0036] Optionally, the business logs are aggregated into call chain aggregate logs corresponding to the call chains, which may include:

[0037] Identify adjacent first and second modules in each business function module, and obtain each first business log collected in the first module and each second business log collected in the second module;

[0038] According to the first module and the second module, a target field name mapping pair set is obtained from the pre-constructed field name mapping pair sets under each adjacent business module;

[0039] Extracting a field value matching pair set corresponding to a target field name mapping pair set from a combination of each first business log and the second business log;

[0040] When it is determined that two field values ​​in each field value matching pair in the field value matching pair set of the target first business log and the target second business log are the same, it is determined that the target first business log and the target second business log belong to the same call chain;

[0041] When it is determined that the log matching operation for all adjacent business function modules is completed, all business logs belonging to each call chain are aggregated into a call chain aggregate log corresponding to the call chain.

[0042] The first module and the second module respectively refer to two current functional modules that are judged to belong to the same call chain according to the content of their business logs among the numerous collected business modules. The first business log refers to all the log information contained in the first functional module; the second business log refers to all the log information contained in the second functional module, wherein the content in each business log is composed of a field name and a field value that matches the field name. Therefore, the pre-constructed field name mapping pair set contains a variety of matching relationships consisting of several field names in the first business log and several field names in the second business log. According to the field name mapping pair set, the field value corresponding to the field name in each business log can be found, and then a variety of matching relationships consisting of several field values ​​in the first business log and several field values ​​in the second business log, that is, a field value matching pair set, can be obtained.

[0043] By identifying and comparing the specific field values ​​of each pair of field value matching pairs in the field value matching pair set, when the field values ​​in the field value matching pairs are the same, it means that the first business log and the second business log contain the same content, and then the first business log and the second business log are considered to be two logs under the same call chain. Further, the above operation is performed on the business logs under all functional modules, so that the business log modules under the same call chain are aggregated to obtain the log information under the complete call chain.

[0044] Furthermore, before obtaining the target field name mapping pair set from the pre-built field name mapping pair sets under each adjacent business module according to the first module and the second module, the method may further include:

[0045] Obtain historical normal business logs collected in each business function module within the set historical detection period;

[0046] Cluster each historical normal business log according to the business function module to which it belongs, and obtain a module log set corresponding to each business function module;

[0047] Perform field name analysis on each module log in each module log set, determine a key field name set corresponding to each module log set, and extract a key field value set corresponding to each module log according to the key field name set;

[0048] The key field value sets corresponding to the two adjacent business function modules are matched for consistency, and based on the matching results, a field name mapping pair set under each adjacent business module is generated.

[0049] The historical normal business log refers to the non-abnormal business log collected within a certain time period without receiving abnormal alarms. First, all business logs are classified according to the functional modules to which the historical normal business logs belong, so that each module log set belongs to the same functional module. Specifically, field name analysis of each module log in each module log set refers to identifying each field name information in each module log, and the key field name set refers to the important field names that can be used to construct a field name mapping pair set that are screened out from all field name information. When the key field name is determined, the key field value corresponding to the key field name can be determined, and a key field value set is constructed. The consistency matching is to identify the key value in the key value set under two adjacent business function modules. If the key values ​​are consistent, the key name field corresponding to a certain pair of key value fields that are currently consistent is searched, and a field name mapping pair is obtained from the key name field. All field name mapping pairs are combined to obtain a field name mapping pair set.

[0050] Optionally, performing field name analysis on each module log in each module log set to determine a key field name set corresponding to each module log set may include:

[0051] Get each current module log in the current module log set, and get the field value set corresponding to the same field name in each current module log;

[0052] Calculate the similarity measurement value of each field value set, and use the field names corresponding to the field value sets whose similarity measurement values ​​are less than a preset similarity threshold as candidate field names;

[0053] Combine each candidate field name with the matching field value to obtain multiple feature variables, and use the principal component analysis method to process each feature variable to obtain the contribution value of each candidate field name;

[0054] In descending order of contribution value, filter out the key field name set corresponding to the current module log set from each candidate field name.

[0055] Continuing with the above embodiment, the specific implementation process of the key field names screened out from all field name information for constructing a field name mapping pair set is as follows: Calculate the similarity measurement value of each field value set in the acquired field value set, wherein the similarity measurement value can be understood as the number of times a certain field value appears repeatedly in a field value set. If a certain field value appears repeatedly 5 times, the similarity measurement value of the field value set can be recorded as 5, that is, the more times a certain field value appears repeatedly, the greater the similarity measurement value of the field value set; conversely, if a certain field value appears repeatedly less frequently, the similarity measurement value of the field value set is said to be low. The preset similarity threshold is a fixed constant. When the similarity measurement value of the field value set is less than the preset similarity threshold, it means that the number of occurrences of the field value set is small. At this time, all field value sets less than the preset similarity threshold are retained, and the field names corresponding to the field values ​​in the retained field value set are called candidate field names.

[0056] Furthermore, the principal component analysis method is used to calculate the variance of each pair of associated field names and field values, and the proportion of the variance of each pair of field names and field values ​​in the total variance is counted, where each pair of field names and field values ​​can be called a group of principal components. The larger the proportion, the higher the contribution of the group of principal components. The field names in the group of principal components are called candidate field names. Since the candidate field names are sorted in order of contribution, the key field names are obtained by selecting a certain number of field names from the candidate field names.

[0057] S120: Perform field value data processing on each service chain aggregation log to obtain a plurality of indicator data corresponding to each call chain aggregation log, and an indicator type corresponding to each indicator data.

[0058] Since in the above embodiment, the log information under the same call chain has been aggregated by determining that the first business log and the second business log are log data under the same call chain, in this embodiment, after extracting the field values ​​in the business log under the aggregated complete call chain, all indicator data under the call chain can be obtained by processing the field values. Among them, the indicator data can be: average time consumption indicator, maximum time consumption indicator, number of errors, etc., and each indicator data can be classified into a certain indicator type, such as: average time consumption indicator and maximum time consumption indicator belong to the time consumption indicator type, and the error number indicator data belongs to the error indicator type.

[0059] S130: Cluster the indicator data of the same indicator type in the same business chain aggregation log to obtain an indicator data set corresponding to each indicator type.

[0060] Traverse the log information under the same business chain after aggregation, and classify all the extracted indicator data into the corresponding indicator types. Each indicator type corresponds to an indicator data set, such as an error indicator data set or an indicator indicator data set.

[0061] S140: Input each indicator data set corresponding to each business chain aggregation log into a pre-trained isolation forest model, identify abnormal indicator data sets, and issue an abnormal alarm according to the target indicator type to which each abnormal indicator data set belongs.

[0062] The isolation forest model is used to detect anomalies in continuous data, and the data judged as abnormal in the isolation forest model is displayed as "isolated outliers" in the graph. Therefore, after all indicator data sets are input into the isolation forest model, it is possible to identify which indicator data set has abnormal indicators, and by matching the indicator type to which the abnormal indicator data belongs, generate exclusive alarm information for a certain category.

[0063] The technical solution of the embodiment of the present invention collects business logs in each business function module corresponding to the target business function during the detection time period, and aggregates each business log into a call chain aggregate log corresponding to each call chain; processes the field value data of each business chain aggregate log to obtain multiple indicator data corresponding to each call chain aggregate log, and the indicator type corresponding to each indicator data; clusters the indicator data of the same indicator type in the same business chain aggregate log to obtain an indicator data set corresponding to each indicator type; inputs each indicator data set corresponding to each business chain aggregate log into a pre-trained isolation forest model to identify abnormal indicator data sets, and performs abnormal alarms according to the target indicator type to which each abnormal indicator data set belongs. The technical solution of the embodiment of the present invention first aggregates multiple business logs under the same business chain, and then processes the field values ​​in all business logs to obtain a number of indicator data, and determines the specific indicator data with abnormalities by calculating the abnormal score value of the indicator data, thereby realizing the rapid positioning and abnormal diagnosis of abnormal indicator data of the entire chain, and providing an efficient solution for business quality monitoring under the microservice architecture.

[0064] Embodiment 2

[0065] Figure 2 This is a flowchart of another method for detecting abnormal indicator data in a call chain log provided by Embodiment 2 of the present invention. This embodiment is based on the above embodiment and is refined. Figure 2 As shown, the method includes:

[0066] S210 . During the detection period, business logs are collected in each business function module corresponding to the target business function, and each business log is aggregated into a call chain aggregate log corresponding to each call chain.

[0067] S220: Input each field value contained in each business chain aggregation log into a pre-trained word segmentation model to obtain multiple indicator data and an indicator type corresponding to each indicator data.

[0068] The indicator types include time-consuming indicators and error indicators.

[0069] In an embodiment of the present invention, since a business chain log text after aggregation contains a large number of unprocessed field values, if the field values ​​are not classified and processed, valid indicator data cannot be extracted subsequently. Therefore, the keyword extraction of the field values ​​in the log text is realized by inputting each field value into a word segmentation model to identify the field values. The extracted keywords can be classified as indicator data into the corresponding indicator type.

[0070] S230: Cluster the indicator data of the same indicator type in the same business chain aggregation log to obtain an indicator data set corresponding to each indicator type.

[0071] S240: Input each indicator data set corresponding to each business chain aggregation log into a pre-trained isolation forest model to obtain an abnormal score value corresponding to each indicator data set.

[0072] In the isolation forest model, the data set is randomly split recursively until all sample points are isolated. Under this random splitting strategy, outliers that are easily isolated can be understood as points that are sparsely distributed and far away from dense groups. In statistical terms, in the data space, sparsely distributed areas indicate that the probability of data occurring in this area is very low, so the data falling in these areas can be considered abnormal. Therefore, the isolation forest model can be used to detect whether there are abnormal points in the data indicator set, that is, abnormal indicator data. Among them, the abnormal score value calculated by the isolation forest model is between (0,1).

[0073] S250: Identify the indicator data set whose abnormal score value falls within the abnormal range as an abnormal indicator data set.

[0074] The abnormal range is a predefined score interval of abnormal points. When the abnormal score value falls into the score interval, it indicates that there is abnormality in the indicator data in the abnormal indicator data set.

[0075] S260. According to the target indicator type to which each abnormal indicator data set belongs, taking the detection time period as a starting point, trace back to obtain a normal historical monitoring time period.

[0076] Furthermore, if you want to view the specific abnormal indicator data in the abnormal indicator data set, you first need to extract the various indicator types in the abnormal indicator data set, and the normal historical monitoring time period refers to a period of time that occurs before the current detection time and when there is no abnormality in the indicator data.

[0077] S270: Obtain an average value of each historical indicator data under the target indicator type within the normal historical monitoring time period.

[0078] The average value of each historical indicator data within the normal historical monitoring period can be calculated as the control value of the abnormal indicator data.

[0079] S280, calculating the difference between each abnormal indicator data set and the average value of each historical indicator data, and obtaining target indicator data of the abnormal difference according to the calculation result.

[0080] By comparing the indicator data of each indicator type in the abnormal indicator data set with the corresponding average value of historical indicator data, when the difference value exceeds the preset threshold, it can be determined which specific indicator data is abnormal.

[0081] S290: Issue an alarm for the target indicator data, and visualize the target indicator data of each call chain with time as the dimension.

[0082] Among them, the target indicator data refers to the indicator data determined as abnormal indicator data, and the visual display refers to the data indicator change chart of the target indicator data within a certain time length, with the time dimension as the horizontal axis and the data value as the vertical axis, which provides abnormal alarms for the front end and is easy to understand. Figure 3 A reference chart for sudden changes in average time-consuming indicator data under the time-consuming indicator type; Figure 4 Added a reference graph for the number of errors in the error number indicator data under the error indicator type.

[0083] The technical solution of the embodiment of the present invention improves the method for detecting abnormal indicator data under the call chain log by refining the overall solution. Specifically, multiple business logs under the same business chain are first aggregated, and then the field values ​​in the log are processed using the word segmentation model to obtain a number of indicator data. The indicator data is input into the isolation forest model to obtain abnormal indicator data, and finally the abnormal indicator data is abnormally alarmed and visualized. It not only achieves the distinction and aggregation of different call chain data, but also achieves the rapid and accurate identification of abnormal indicator data.

[0084] Embodiment 3

[0085] Figure 5 This is a schematic diagram of the structure of a device for detecting abnormal indicator data in a call chain log provided by Embodiment 3 of the present invention. Figure 5 As shown, the device comprises:

[0086] The call chain log aggregation module 510 is used to collect the business logs in each business function module corresponding to the target business function during the detection period, and aggregate the business logs into call chain aggregation logs corresponding to each call chain;

[0087] The field value data processing module 520 is used to process the field value data of each business chain aggregation log respectively, and obtain multiple indicator data corresponding to each call chain aggregation log, and the indicator type corresponding to each indicator data;

[0088] The indicator data aggregation module 530 is used to cluster the indicator data of the same indicator type in the same business chain aggregation log to obtain an indicator data set corresponding to each indicator type;

[0089] The abnormal alarm module 540 is used to input each indicator data set corresponding to each business chain aggregation log into a pre-trained isolation forest model, identify abnormal indicator data sets, and issue abnormal alarms according to the target indicator type to which each abnormal indicator data set belongs.

[0090] The technical solution of the embodiment of the present invention, the call chain log aggregation module 510, is used to collect business logs respectively in each business function module corresponding to the target business function within the detection time period, and aggregate each business log into a call chain aggregation log corresponding to each call chain;

[0091] The field value data processing module 520 processes the field value data of each business chain aggregation log respectively, and obtains multiple indicator data corresponding to each call chain aggregation log, and the indicator type corresponding to each indicator data; the indicator data aggregation module 530 clusters the indicator data of the same indicator type in the same business chain aggregation log, and obtains the indicator data set corresponding to each indicator type; the abnormal alarm module 540 inputs the indicator data set corresponding to each business chain aggregation log into the pre-trained isolation forest model, identifies the abnormal indicator data set, and performs abnormal alarm according to the target indicator type to which each abnormal indicator data set belongs. The technical solution of the embodiment of the present invention first aggregates multiple business logs under the same business chain, and then processes the field values ​​in all business logs to obtain a number of indicator data, and determines the specific indicator data with abnormalities by calculating the abnormal score value of the indicator data, thereby realizing the rapid positioning and abnormal diagnosis of abnormal indicator data of the whole chain, and providing an efficient solution for business quality monitoring under the microservice architecture.

[0092] Based on the above embodiments, the call chain log aggregation module 510 may include:

[0093] A business log collection unit, used to identify adjacent first modules and second modules in each business function module, and obtain each first business log collected in the first module and each second business log collected in the second module;

[0094] A target field name mapping pair set acquisition unit, configured to acquire a target field name mapping pair set from pre-built field name mapping pair sets under each adjacent business module according to the first module and the second module;

[0095] A field value matching pair set extraction unit, used to extract a field value matching pair set corresponding to a target field name mapping pair set from a combination of each first business log and a second business log;

[0096] a same call chain determining unit, configured to determine that the target first business log and the target second business log belong to the same call chain when it is determined that two field values ​​in each field value matching pair in the field value matching pair set of the target first business log and the target second business log are the same;

[0097] The call chain log aggregation unit is used to aggregate all business logs belonging to each call chain into a call chain aggregation log corresponding to the call chain when it is determined that the log matching operation for all adjacent business function modules is completed.

[0098] On the basis of the above embodiments, according to the first module and the second module, before obtaining the target field name mapping pair set from the pre-built field name mapping pair sets under each adjacent business module, the following may also be included:

[0099] The historical normal business log mobile unit is used to obtain the historical normal business logs collected in each business function module within the set historical detection period;

[0100] A module log set acquisition unit is used to cluster each historical normal business log according to the business function module to which it belongs, and obtain a module log set corresponding to each business function module;

[0101] The key field value set extraction unit is used to perform field name analysis on each module log in each module log set, determine the key field name set corresponding to each module log set, and extract the key field value set corresponding to each module log according to the key field name set;

[0102] The field name mapping pair set generation unit is used to perform consistency matching on each key field value set corresponding to each two adjacent business function modules, and generate a field name mapping pair set under each adjacent business module according to the matching result.

[0103] Based on the above embodiment, the key field value set extraction unit may further include:

[0104] Get each current module log in the current module log set, and get the field value set corresponding to the same field name in each current module log;

[0105] Calculate the similarity measurement value of each field value set, and use the field names corresponding to the field value sets whose similarity measurement values ​​are less than a preset similarity threshold as candidate field names;

[0106] Combine each candidate field name with the matching field value to obtain multiple feature variables, and use the principal component analysis method to process each feature variable to obtain the contribution value of each candidate field name;

[0107] In descending order of contribution value, filter out the key field name set corresponding to the current module log set from each candidate field name.

[0108] Based on the above embodiments, the field value data processing module 520 may include:

[0109] An indicator data and indicator type acquisition unit, used to input each field value contained in each business chain aggregation log into a pre-trained word segmentation model to obtain multiple indicator data and an indicator type corresponding to each indicator data;

[0110] The indicator types include time-consuming indicators and error indicators.

[0111] Based on the above embodiments, the abnormal alarm module 540 may include:

[0112] An abnormal score value acquisition unit is used to input each indicator data set corresponding to each business chain aggregation log into a pre-trained isolation forest model to obtain an abnormal score value corresponding to each indicator data set;

[0113] The abnormal indicator data set identification unit is used to identify the indicator data set whose abnormal score value falls into the abnormal range as the abnormal indicator data set.

[0114] Based on the above embodiments, the abnormal alarm module 540 may further include:

[0115] A normal historical monitoring time period tracing unit is used to trace back the normal historical monitoring time period based on the target indicator type to which each abnormal indicator data set belongs, starting from the detection time period;

[0116] A historical indicator data average value acquisition unit, used to acquire the average value of each historical indicator data under the target indicator type within the normal historical monitoring time period;

[0117] A target indicator data acquisition unit is used to calculate the difference between each abnormal indicator data set and the average value of each historical indicator data, and obtain the target indicator data of the abnormal difference according to the calculation result;

[0118] The visualization unit is used to issue an alarm for the target indicator data and to visualize the target indicator data of each call chain based on time.

[0119] An apparatus for detecting anomalies of indicator data under a call chain log provided by an embodiment of the present invention can execute an anomaly detection method for detecting anomalies of indicator data under a call chain log provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0120] Embodiment 4

[0121] Figure 6A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0122] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0123] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0124] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for detecting abnormality of indicator data under a call chain log.

[0125] That is, during the detection period, business logs are collected in each business function module corresponding to the target business function, and each business log is aggregated into a call chain aggregate log corresponding to each call chain;

[0126] Perform field value data processing on each business chain aggregation log to obtain multiple indicator data corresponding to each call chain aggregation log and indicator types corresponding to each indicator data;

[0127] Clustering the indicator data of the same indicator type in the same business chain aggregation log to obtain an indicator data set corresponding to each indicator type;

[0128] Each indicator data set corresponding to each business chain aggregation log is input into the pre-trained isolation forest model to identify abnormal indicator data sets, and abnormal alarms are issued according to the target indicator type to which each abnormal indicator data set belongs.

[0129] In some embodiments, a method for detecting anomalies in indicator data under a call chain log may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for detecting anomalies in indicator data under a call chain log described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute a method for detecting anomalies in indicator data under a call chain log in any other appropriate manner (e.g., by means of firmware).

[0130] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0132] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0134] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0135] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0136] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0137] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting abnormality of indicator data in a call chain log, characterized in that: include: During the detection period, business logs are collected in each business function module corresponding to the target business function, and each business log is aggregated into a call chain aggregate log corresponding to each call chain; Perform field value data processing on each business chain aggregation log respectively to obtain multiple indicator data corresponding to each call chain aggregation log and indicator types corresponding to each indicator data; Clustering the indicator data of the same indicator type in the same business chain aggregation log to obtain an indicator data set corresponding to each indicator type; Input each indicator data set corresponding to each business chain aggregation log into the pre-trained isolation forest model, identify abnormal indicator data sets, and issue abnormal alarms based on the target indicator type to which each abnormal indicator data set belongs; The business logs are aggregated into call chain aggregate logs corresponding to the call chains, including: Identify adjacent first and second modules in each business function module, and obtain each first business log collected in the first module and each second business log collected in the second module; According to the first module and the second module, a target field name mapping pair set is obtained from the pre-constructed field name mapping pair sets under each adjacent business module; Extracting a field value matching pair set corresponding to a target field name mapping pair set from a combination of each first business log and the second business log; When it is determined that two field values ​​in each field value matching pair in the field value matching pair set of the target first business log and the target second business log are the same, it is determined that the target first business log and the target second business log belong to the same call chain; When it is determined that the log matching operation for all adjacent business function modules is completed, all business logs belonging to each call chain are aggregated into a call chain aggregate log corresponding to the call chain.

2. The method according to claim 1, characterized in that Before obtaining a target field name mapping pair set from the pre-built field name mapping pair sets under each adjacent business module according to the first module and the second module, the method further includes: Obtain historical normal business logs collected in each business function module within the set historical detection period; Cluster each historical normal business log according to the business function module to which it belongs, and obtain a module log set corresponding to each business function module; Perform field name analysis on each module log in each module log set, determine a key field name set corresponding to each module log set, and extract a key field value set corresponding to each module log according to the key field name set; The key field value sets corresponding to the two adjacent business function modules are matched for consistency, and based on the matching results, a field name mapping pair set under each adjacent business module is generated.

3. The method according to claim 2, characterized in that Perform field name analysis on each module log in each module log set to determine the key field name set corresponding to each module log set, including: Get each current module log in the current module log set, and get the field value set corresponding to the same field name in each current module log; Calculate the similarity measurement value of each field value set, and use the field names corresponding to the field value sets whose similarity measurement values ​​are less than a preset similarity threshold as candidate field names; Combine each candidate field name with the matching field value to obtain multiple feature variables, and use the principal component analysis method to process each feature variable to obtain the contribution value of each candidate field name; In descending order of contribution value, filter out the key field name set corresponding to the current module log set from each candidate field name.

4. The method according to any one of claims 1 to 3, characterized in that: The field value data of each business chain aggregation log is processed respectively to obtain multiple indicator data corresponding to each call chain aggregation log, and the indicator type corresponding to each indicator data, including: Input each field value contained in each business chain aggregation log into a pre-trained word segmentation model to obtain multiple indicator data and an indicator type corresponding to each indicator data; The indicator types include time-consuming indicators and error indicators.

5. The method according to any one of claims 1 to 3, characterized in that: Input each indicator data set corresponding to each business chain aggregation log into the pre-trained isolation forest model to identify abnormal indicator data sets, including: Input each indicator data set corresponding to each business chain aggregation log into the pre-trained isolation forest model to obtain the anomaly score value corresponding to each indicator data set; The indicator data set whose abnormal score value falls into the abnormal range is identified as an abnormal indicator data set.

6. The method according to any one of claims 1 to 3, characterized in that: According to the target indicator type to which each abnormal indicator data set belongs, abnormal alarms are issued, including: According to the target indicator type to which each abnormal indicator data set belongs, starting from the detection time period, trace back to obtain the normal historical monitoring time period; Obtaining the average value of each historical indicator data under the target indicator type during the normal historical monitoring time period; Calculate the difference between each abnormal indicator data set and the average value of each historical indicator data, and obtain the target indicator data of the abnormal difference based on the calculation result; Alerts are issued for target indicator data, and the target indicator data of each call chain is visualized with time as the dimension.

7. A device for detecting abnormality of indicator data under a call chain log, characterized in that: include: A call chain log aggregation module is used to collect business logs in each business function module corresponding to the target business function during the detection period, and aggregate each business log into a call chain aggregation log corresponding to each call chain; The field value data processing module is used to process the field value data of each business chain aggregation log respectively, and obtain multiple indicator data corresponding to each call chain aggregation log, and the indicator type corresponding to each indicator data; The indicator data aggregation module is used to cluster the indicator data of the same indicator type in the same business chain aggregation log to obtain the indicator data set corresponding to each indicator type; The abnormal alarm module is used to input each indicator data set corresponding to each business chain aggregation log into the pre-trained isolation forest model, identify the abnormal indicator data set, and issue an abnormal alarm according to the target indicator type to which each abnormal indicator data set belongs; Among them, the call chain log aggregation module also includes: A business log collection unit, used to identify adjacent first modules and second modules in each business function module, and obtain each first business log collected in the first module and each second business log collected in the second module; A target field name mapping pair set acquisition unit, configured to acquire a target field name mapping pair set from pre-built field name mapping pair sets under each adjacent business module according to the first module and the second module; A field value matching pair set extraction unit, used to extract a field value matching pair set corresponding to a target field name mapping pair set from a combination of each first business log and a second business log; a same call chain determining unit, configured to determine that the target first business log and the target second business log belong to the same call chain when it is determined that two field values ​​in each field value matching pair in the field value matching pair set of the target first business log and the target second business log are the same; The call chain log aggregation unit is used to aggregate all business logs belonging to each call chain into a call chain aggregation log corresponding to the call chain when it is determined that the log matching operation for all adjacent business function modules is completed.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a method for detecting anomalies in indicator data under a call chain log as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a method for detecting anomalies in indicator data under a call chain log according to any one of claims 1 to 6 when executed.

Images