Data monitoring method and device, computer device and storage medium

Abstract

The application relates to a data monitoring method and device, computer equipment and a storage medium, and relates to the technical field of big data. The method comprises the following steps: processing each original transaction data in an original transaction data set according to at least one monitoring index to obtain target transaction data corresponding to each original transaction data; constructing an isolated forest corresponding to each monitoring index based on an isolated forest algorithm according to each target transaction data; determining a comprehensive health value of each target transaction data according to the isolated forest corresponding to each monitoring index; and determining abnormal transaction data in each original transaction data according to the comprehensive health value of each target transaction data. The method can simplify the data monitoring process.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data monitoring method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the gradual development of the financial market, a large number of trading businesses with different characteristics have emerged. In order to ensure the accuracy of trading data, data monitoring methods have been developed. Current data monitoring methods set different monitoring rules for each trading task, monitor the data according to the monitoring rules, and then send the abnormal trading data detected directly to the operations and maintenance personnel for processing.

[0003] However, due to the rapid growth in the types of transaction businesses, the current data monitoring methods require reconfiguring the corresponding monitoring rules for newly added transaction businesses, making the entire data monitoring process quite cumbersome. Summary of the Invention

[0004] Therefore, it is necessary to provide a data monitoring method, apparatus, computer equipment, and storage medium to address the aforementioned technical problems and simplify the data monitoring process.

[0005] Firstly, this application provides a data monitoring method. The method includes:

[0006] Based on at least one monitoring indicator, process each original transaction data in the original transaction dataset to obtain the target transaction data corresponding to each original transaction data.

[0007] Based on the isolated forest algorithm, an isolated forest corresponding to each monitoring indicator is constructed according to the transaction data of each target.

[0008] Based on the isolated forest corresponding to each monitoring indicator, determine the comprehensive health value of each target transaction data;

[0009] Based on the overall health value of each target transaction data, abnormal transaction data in each original transaction data is identified.

[0010] In one embodiment, the original transaction data in the original transaction dataset is processed according to at least one monitoring indicator to obtain the target transaction data corresponding to each original transaction data, including:

[0011] For each piece of raw transaction data in the original transaction dataset, determine the indicator data of that raw transaction data under each monitoring indicator; then concatenate the indicator data of that raw transaction data under each monitoring indicator to obtain the target transaction data corresponding to that raw transaction data.

[0012] In one embodiment, based on the isolated forest algorithm, an isolated forest corresponding to each monitoring indicator is constructed according to the target transaction data, including:

[0013] For each monitoring indicator, an indicator dataset is constructed based on the indicator data of each target transaction data under that monitoring indicator. Different indicator data are selected from the indicator dataset as split points, and the indicator dataset is split using the split points to obtain the isolated forest corresponding to that monitoring indicator.

[0014] In one embodiment, the comprehensive health value of each target transaction data is determined based on the isolated forest corresponding to each monitoring indicator, including:

[0015] For each target transaction data, the health value of the target transaction data under each monitoring indicator is determined based on the isolated forest corresponding to each monitoring indicator; the sum of the health values ​​of the target transaction data under each monitoring indicator is taken as the comprehensive health value of the target transaction data.

[0016] In one embodiment, the indicator health value of the target transaction data under each monitoring indicator is determined based on the isolated forest corresponding to each monitoring indicator, including:

[0017] Based on the isolated forest corresponding to each monitoring indicator, determine the average height of the target transaction data under each monitoring indicator; based on the weight of each monitoring indicator and the average height of the target transaction data under each monitoring indicator, determine the indicator health value of the target transaction data under each monitoring indicator.

[0018] In one embodiment, the average height of the target transaction data under each monitoring metric is determined based on the isolated forest corresponding to each monitoring metric, including:

[0019] For each monitoring indicator, the height value of the target transaction data under each isolated tree in the isolated forest corresponding to the monitoring indicator is determined; the average height value of the target transaction data under each isolated tree is taken as the average height of the target transaction data under the monitoring indicator.

[0020] In one embodiment, the method further includes:

[0021] The health values ​​of the target transaction data corresponding to the abnormal transaction data under each monitoring indicator are compared with the health thresholds corresponding to each monitoring indicator; the monitoring indicators whose health values ​​are less than the corresponding health thresholds are regarded as abnormal factors corresponding to the abnormal transaction data.

[0022] Secondly, this application also provides a data monitoring device. The device includes:

[0023] The data processing module is used to process each original transaction data in the original transaction dataset according to at least one monitoring indicator to obtain the target transaction data corresponding to each original transaction data.

[0024] The forest building module is used to construct isolated forests corresponding to various monitoring indicators based on the isolated forest algorithm and the transaction data of each target.

[0025] The health value determination module is used to determine the comprehensive health value of each target transaction data based on the isolated forest corresponding to each monitoring indicator.

[0026] The anomaly detection module is used to identify abnormal transaction data in each original transaction data based on the comprehensive health value of each target transaction data.

[0027] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0028] Based on at least one monitoring indicator, process each original transaction data in the original transaction dataset to obtain the target transaction data corresponding to each original transaction data.

[0029] Based on the isolated forest algorithm, an isolated forest corresponding to each monitoring indicator is constructed according to the transaction data of each target.

[0030] Based on the isolated forest corresponding to each monitoring indicator, determine the comprehensive health value of each target transaction data;

[0031] Based on the overall health value of each target transaction data, abnormal transaction data in each original transaction data is identified.

[0032] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0033] Based on at least one monitoring indicator, process each original transaction data in the original transaction dataset to obtain the target transaction data corresponding to each original transaction data.

[0034] Based on the isolated forest algorithm, an isolated forest corresponding to each monitoring indicator is constructed according to the transaction data of each target.

[0035] Based on the isolated forest corresponding to each monitoring indicator, determine the comprehensive health value of each target transaction data;

[0036] Based on the overall health value of each target transaction data, abnormal transaction data in each original transaction data is identified.

[0037] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0038] Based on at least one monitoring indicator, process each original transaction data in the original transaction dataset to obtain the target transaction data corresponding to each original transaction data.

[0039] Based on the isolated forest algorithm, an isolated forest corresponding to each monitoring indicator is constructed according to the transaction data of each target.

[0040] Based on the isolated forest corresponding to each monitoring indicator, determine the comprehensive health value of each target transaction data;

[0041] Based on the overall health value of each target transaction data, abnormal transaction data in each original transaction data is identified.

[0042] The aforementioned data monitoring methods, devices, computer equipment, and storage media, by introducing monitoring indicators, process each original transaction data in the original transaction dataset to extract the data that needs to be monitored from each original transaction data, i.e., the target transaction data corresponding to each original transaction data. This ensures that when a new transaction occurs, the data that needs to be monitored in the original transaction data corresponding to the new transaction can be extracted solely through monitoring indicators, without the need to reconfigure monitoring rules, thus simplifying the data monitoring process. Furthermore, by employing the isolated forest algorithm, each target transaction data is analyzed from multiple monitoring indicators to determine the health value of each target transaction data. Based on the health value of each target transaction data, abnormal transaction data in each original transaction data can be accurately identified, providing a method for accurately extracting abnormal transaction data from a large amount of original transaction data. Attached Figure Description

[0043] Figure 1 This is a diagram illustrating the application environment of a data monitoring method in one embodiment.

[0044] Figure 2A This is a flowchart illustrating a data monitoring method in one embodiment;

[0045] Figure 2B Here is a structural diagram of an isolated tree in one embodiment;

[0046] Figure 3 This is a schematic diagram of the process for determining a comprehensive health value in one embodiment;

[0047] Figure 4 This is a flowchart illustrating the process of determining the health value of an indicator in one embodiment;

[0048] Figure 5 This is a flowchart illustrating the process of identifying anomalous factors in one embodiment;

[0049] Figure 6 This is a flowchart illustrating the data monitoring method in another embodiment;

[0050] Figure 7 This is a structural block diagram of a data monitoring device in one embodiment;

[0051] Figure 8 This is a structural block diagram of the data monitoring device in another embodiment;

[0052] Figure 9 This is a structural block diagram of the data monitoring device in another embodiment;

[0053] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] The data monitoring method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process, such as raw transaction data. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. For example, server 104 processes each raw transaction data in the raw transaction dataset according to at least one monitoring indicator to obtain target transaction data corresponding to each raw transaction data; based on the isolated forest algorithm, it constructs isolated forests corresponding to each monitoring indicator based on each target transaction data; based on the isolated forests corresponding to each monitoring indicator, it determines the comprehensive health value of each target transaction data; further, based on the comprehensive health value of each target transaction data, it identifies abnormal transaction data in each raw transaction data; and finally, server 104 sends the abnormal transaction data to terminal 102. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and IoT devices. Server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0056] With the gradual development of the financial market, a large number of trading businesses with different characteristics have emerged. In order to ensure the accuracy of the acquired trading data, data monitoring methods have been developed. Current data monitoring methods set different monitoring rules for each trading task, monitor the data according to the monitoring rules, and then send the monitored abnormal trading data directly to the operations and maintenance personnel for processing.

[0057] However, due to the rapid growth in the types of transaction businesses, the current data monitoring methods require reconfiguring the corresponding monitoring rules for newly added transaction businesses, making the entire data monitoring process quite cumbersome.

[0058] Based on this, in one embodiment, such as Figure 2A As shown, a data monitoring method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0059] S201, based on at least one monitoring indicator, process each original transaction data in the original transaction dataset to obtain the target transaction data corresponding to each original transaction data.

[0060] The monitoring metrics are pre-set indicators used to monitor key data in the raw transaction data. Optionally, in the quotation system, the monitoring metrics can be one or more of the following: CPU utilization, memory utilization, disk I / O, network traffic, packet loss rate, transaction volume, response time, and transaction success rate.

[0061] Raw transaction data refers to the unprocessed data generated during transactions in various businesses; raw transaction dataset is a collection of raw transaction data, containing at least two raw transaction datasets; target transaction data refers to the data obtained after processing the raw transaction data, which includes data corresponding to monitoring indicators.

[0062] Optionally, since the types of transaction businesses vary, the resulting raw transaction data also differ. To standardize the format of the raw transaction data for each type of transaction business, monitoring indicators can be pre-set based on the data format of the raw transaction data for each type of transaction business. It is understood that the raw transaction data for each type of transaction business includes the indicator data corresponding to the monitoring indicators.

[0063] Optionally, a data processing model including at least one monitoring indicator can be constructed, and then each original transaction data in the original transaction dataset can be sequentially input into the trained data processing model. Based on the input original transaction data and the parameters of the data processing model itself, the data processing model directly outputs the target transaction data corresponding to each original transaction data.

[0064] Alternatively, for each piece of original transaction data in the original transaction dataset, determine the indicator data of that original transaction data under each monitoring indicator; then concatenate the indicator data of that original transaction data under each monitoring indicator to obtain the target transaction data corresponding to that original transaction data.

[0065] Among them, indicator data refers to the data corresponding to each monitoring indicator contained in the original transaction data.

[0066] Specifically, after acquiring the original transaction dataset, each original transaction data point is analyzed to directly determine its corresponding indicator data under various monitoring metrics. Once the indicator data for each monitoring metric is obtained, it can be concatenated according to a predefined format to obtain the target transaction data corresponding to the original transaction data.

[0067] For example, in a quotation system, the monitoring metrics are trading volume, response time, and trading success rate. The original trading data obtained is "Acquisition time, 2023-03-06 05:16; Trading volume, 60; Response time, 10ms; Trading success rate, 100%". By concatenating the metric data under each monitoring metric for this original trading data, the target trading data "Trading volume, 60; Response time, 10ms; Trading success rate, 100%" can be obtained.

[0068] S202, based on the Isolation Forest algorithm, constructs an Isolation Forest corresponding to each monitoring indicator based on the transaction data of each target.

[0069] Optionally, after obtaining the target transaction data, the target transaction data can be directly input into the forest construction model trained based on the isolated forest algorithm. The forest construction model can directly construct the isolated forest corresponding to each monitoring indicator based on the input target transaction data and the parameters of the forest construction model itself.

[0070] Alternatively, for each monitoring metric, an metric dataset can be constructed based on the metric data of each target transaction data under that monitoring metric; and an isolated forest corresponding to that monitoring metric can be constructed based on the isolated forest algorithm. Specifically, constructing the isolated forest corresponding to the monitoring metric based on the isolated forest algorithm can be achieved by selecting different metric data as split points from the metric dataset and using these split points to segment the metric dataset, thus obtaining the isolated forest corresponding to that monitoring metric.

[0071] Specifically, after obtaining the target transaction data, for each monitoring indicator, the indicator data under that monitoring indicator can be extracted to form a corresponding indicator dataset. Then, a random split point can be generated. Based on the comparison between the split point and each data point in the indicator dataset, the indicator dataset is divided into a left child node set and a right child node set. Further, the datasets in the left and right child node sets with non-zero data points are further divided using the same method until the data point in each child node set is zero. This constructs the isolated tree corresponding to the monitoring indicator. It is understandable that, since the splitting is random, each splitting method can produce a corresponding isolated tree; therefore, each monitoring indicator can construct a corresponding isolated forest.

[0072] For example, refer to Figure 2B The method for constructing an isolated tree in an isolated forest for the indicator dataset {61,60,59,55,42}, which is a monitoring indicator for trading volume, is as follows: In the current dataset, indicator data 55 is selected as the cut point P. Indicator data smaller than the cut point P are placed into the right child node set, and the remaining indicator data are placed into the left child node set, resulting in the left child node set {61,60,59,55} and the right child node set {42}. Then, indicator data 60 is selected as the cut point P1, resulting in the left child node set {61,60} and the right child node set {59,55}. For the left child node set {61,60}, indicator data 61 is selected as the cut point P2, resulting in the left child node set {61} and the right child node set {60}. For the right child node set {59,55}, indicator data 59 is selected as the cut point P3, resulting in the left child node set {59} and the right child node set {55}. At this point, the isolated tree is constructed.

[0073] S203 determines the overall health value of each target transaction data based on the isolated forest corresponding to each monitoring indicator.

[0074] The so-called comprehensive health value is an indicator that comprehensively measures whether there are any anomalies in the target transaction data from multiple monitoring indicators. Furthermore, the higher the comprehensive health value, the lower the possibility of anomalies in the target transaction data, that is, the higher the health level of the data.

[0075] Optionally, after constructing the isolated forest corresponding to each monitoring indicator, for each isolated forest, the indicator health value of each target transaction data under the monitoring indicator corresponding to that isolated forest can be determined according to the structure of each isolated tree in the isolated forest; furthermore, for each target transaction data, the comprehensive health value of the target transaction data can be determined according to the indicator health value of the target transaction data under each monitoring indicator.

[0076] S204. Based on the comprehensive health value of each target transaction data, identify the abnormal transaction data in each original transaction data.

[0077] Specifically, after obtaining the overall health value of each target transaction data, the overall health value of each target transaction data can be compared with a preset health threshold. If the overall health value of a target transaction data is less than the health threshold, then the original transaction data is abnormal transaction data; otherwise, the original transaction data is normal transaction data.

[0078] Furthermore, after identifying abnormal transaction data, the identified abnormal transaction data can be directly sent to the terminal held by the operation and maintenance personnel, prompting them to handle it in a timely manner.

[0079] Understandably, since the monitoring metrics are preset based on the data format of the original transaction data, for each new transaction, the original transaction data of that transaction contains the corresponding metric data for each monitoring metric. Therefore, when a new transaction occurs, there is no need to reconfigure new monitoring metrics for the new transaction. In other words, this method is applicable to all transaction scenarios.

[0080] The aforementioned data monitoring method introduces monitoring indicators to process each original transaction data in the original transaction dataset, extracting the data that needs to be monitored from each original transaction data, i.e., the target transaction data corresponding to each original transaction data. This ensures that when a new transaction occurs, the data that needs to be monitored in the original transaction data corresponding to the new transaction can be extracted simply by using the monitoring indicators, without reconfiguring the monitoring rules, thus simplifying the data monitoring process. Furthermore, the isolated forest algorithm is used to analyze each target transaction data from multiple monitoring indicators to determine the health value of each target transaction data. Based on the health value of each target transaction data, abnormal transaction data in each original transaction data can be accurately identified, providing a way to accurately extract abnormal transaction data from a large amount of original transaction data.

[0081] To ensure the accuracy of identifying abnormal transaction data, this embodiment provides a method for determining a comprehensive health value, based on the above embodiments. Figure 3 As shown, the specific steps include:

[0082] S301, for each target transaction data, determine the indicator health value of the target transaction data under each monitoring indicator based on the isolated forest corresponding to each monitoring indicator.

[0083] Optionally, after constructing the isolated forests corresponding to each monitoring indicator, for each target transaction data, the height value of each indicator data in each isolated tree of the corresponding isolated forest can be determined based on the structure of each isolated tree in the isolated forest corresponding to each monitoring indicator.

[0084] For each monitoring metric, the health value of the target transaction data under that monitoring metric is determined based on the height value of the corresponding metric data in each isolated tree of the corresponding isolated forest. For example, if the height values ​​of a target transaction data under the monitoring metric of trading volume are 3, 3, and 4, the maximum value can be selected as the health value of the trading volume metric, i.e., the health value of the target transaction data under the trading volume metric is 4.

[0085] S302, the sum of the health values ​​of the target transaction data under each monitoring indicator is taken as the comprehensive health value of the target transaction data.

[0086] Furthermore, after determining the health values ​​of the target transaction data under each monitoring indicator, the health values ​​of the target transaction data under each monitoring indicator can be added together, and the result can be used as the comprehensive health value of the target transaction data.

[0087] In this embodiment, by introducing the health values ​​of the target transaction data under various monitoring indicators, the security of the target transaction data can be analyzed from multiple dimensions, ensuring the accuracy of the determination of the comprehensive health value of each target transaction data, and thus ensuring the accuracy of the determination of abnormal transaction data.

[0088] To ensure the accuracy of the health values ​​of the target transaction data under various monitoring indicators, this embodiment provides a method for determining the health values ​​of the indicators, based on the above embodiments. Figure 4 As shown, the specific steps include:

[0089] S401, based on the isolated forest corresponding to each monitoring indicator, determine the average height of the target transaction data under each monitoring indicator.

[0090] Optionally, for each monitoring indicator, the height value of the target transaction data under each isolated tree in the isolated forest corresponding to the monitoring indicator is determined; the average height value of the target transaction data under each isolated tree is taken as the average height of the target transaction data under the monitoring indicator.

[0091] Specifically, for each monitoring metric, the height value of the target transaction data under each isolated tree in the isolated forest corresponding to that monitoring metric can be determined based on the tree structure of each isolated tree and the position of the target transaction data within that isolated tree. For example, see [link to relevant documentation]. Figure 2B In the isolated tree, the height of the target transaction data corresponding to indicator data 42 in the isolated tree is 1; the height of the target transaction data corresponding to indicator data 61, 60, 59, and 55 in the isolated tree is 3.

[0092] Furthermore, the height value of the target transaction data under each isolated tree can be calculated, and then the average height value of the target transaction data under each isolated tree can be used as the average height of the target transaction data under the monitoring indicator.

[0093] S402, based on the weight of each monitoring indicator and the average height of the target transaction data under each monitoring indicator, determine the indicator health value of the target transaction data under each monitoring indicator.

[0094] Understandably, given the varying importance and volatility of each monitoring indicator, to ensure the accuracy of the target transaction data's health values ​​under each indicator, the weight of each monitoring indicator can be determined based on its importance and the volatility of the corresponding data. For example, in a quotation system, the transaction success rate is highly important and less prone to fluctuation, thus it can be assigned a higher weight; while response time is easily affected by network and quotation system fluctuations, therefore it can be assigned a lower weight.

[0095] Optionally, after determining the weights of each monitoring indicator, for each target transaction data point, the weight of each monitoring indicator is multiplied by the average height of that target transaction data under each monitoring indicator to obtain the indicator health value of that target transaction data under each monitoring indicator. For example, in a quotation system, the weight of trading volume is 0.4, the weight of response time is 0.1, and the weight of transaction success rate is 0.5; in the target transaction data, the indicator health value corresponding to trading volume is A1, the indicator health value corresponding to response time is A2, and the indicator health value corresponding to transaction success rate is A3. Therefore, the comprehensive health value of the target transaction data is 0.4*A1 + 0.1*A2 + 0.5*A3.

[0096] Optionally, since the average height of the target transaction data varies greatly under each monitoring indicator, it has a certain impact on the determination of the health value of the subsequent indicators. Therefore, formula (1) can be used to normalize the average height of each indicator data in the isolated tree. Then, for each target transaction data, the indicator health value of the target transaction data under each monitoring indicator is determined according to the weight of each monitoring indicator and the average height of the target transaction data after normalization under each monitoring indicator.

[0097]

[0098] Among them, H i,j h represents the average height of the i-th target transaction data after normalization under monitoring indicator j; i,j This represents the average height of the i-th target transaction data under monitoring indicator j; This represents the maximum average height among all target transaction data under monitoring indicator j; This represents the minimum average height among all target transaction data under monitoring indicator j.

[0099] In this embodiment, by introducing the weights of each monitoring indicator, the health value of the target transaction data under each monitoring indicator can be determined based on information such as the importance of each monitoring indicator, thereby ensuring the accuracy of the health value of the target transaction data under each monitoring indicator.

[0100] To improve the timeliness of handling transaction data anomalies, after identifying the abnormal transaction data, this embodiment provides a method for determining anomaly factors based on the above embodiments, such as... Figure 5 As shown, the specific steps include:

[0101] S501 compares the health values ​​of the target transaction data corresponding to the abnormal transaction data under each monitoring indicator with the health thresholds corresponding to each monitoring indicator.

[0102] Optionally, for each monitoring indicator, a health threshold can be preset based on the range of historical health values ​​for each indicator. Historical health values ​​refer to the health values ​​of historical target transaction data under each monitoring indicator.

[0103] Furthermore, after identifying abnormal transaction data, the health values ​​of the target transaction data corresponding to the abnormal transaction data under each monitoring indicator can be compared with the health thresholds corresponding to each monitoring indicator to obtain the comparison results.

[0104] S502 identifies monitoring indicators whose health values ​​are below the corresponding health thresholds as abnormal factors corresponding to abnormal transaction data.

[0105] Among them, abnormal factors can include monitoring indicators corresponding to indicator data that cause abnormalities in the original transaction data; furthermore, abnormal factors can include only one monitoring indicator or multiple monitoring indicators.

[0106] Specifically, for each abnormal transaction data, after obtaining the comparison results of the indicator health value of the target transaction data corresponding to the abnormal transaction data under each monitoring indicator and the corresponding health threshold of each monitoring indicator, the monitoring indicators whose indicator health value is less than the corresponding health threshold are regarded as abnormal factors in the abnormal transaction data.

[0107] Furthermore, after identifying the abnormal factors in the abnormal transaction data, the abnormal transaction data and the abnormal factors within it can be sent simultaneously to the operations and maintenance personnel's terminals for easy viewing.

[0108] In this embodiment, by comparing the health values ​​of each monitoring indicator in the abnormal transaction data with the health thresholds corresponding to each monitoring indicator, abnormal factors are introduced, which makes it easier for operation and maintenance personnel to quickly locate the cause of the abnormality, thereby improving the timeliness of transaction data abnormality handling.

[0109] Figure 6 This is a flowchart illustrating a data monitoring method in another embodiment. Based on the above embodiments, this embodiment provides an optional example of a data monitoring method. (Combined with...) Figure 6 The specific implementation process is as follows:

[0110] S601, for each piece of original transaction data in the original transaction dataset, determine the indicator data of the original transaction data under each monitoring indicator, and concatenate the indicator data of the original transaction data under each monitoring indicator to obtain the target transaction data corresponding to the original transaction data.

[0111] S602: For each monitoring indicator, construct an indicator dataset based on the indicator data of each target transaction data under that monitoring indicator, and construct the corresponding isolated forest based on the isolated forest algorithm.

[0112] Among them, based on the isolated forest algorithm, constructing the isolated forest corresponding to the monitoring indicator can be achieved by selecting different indicator data as split points from the indicator dataset and using the split points to segment the indicator dataset to obtain the isolated forest corresponding to the monitoring indicator.

[0113] S603: For each target transaction data, determine the indicator health value of the target transaction data under each monitoring indicator based on the isolated forest corresponding to each monitoring indicator, and take the sum of the indicator health values ​​of the target transaction data under each monitoring indicator as the comprehensive health value of the target transaction data.

[0114] Specifically, for each monitoring indicator, the height value of the target transaction data under each isolated tree in the isolated forest corresponding to the monitoring indicator is determined; the average height value of the target transaction data under each isolated tree is taken as the average height of the target transaction data under the monitoring indicator; based on the weight of each monitoring indicator and the average height of the target transaction data under each monitoring indicator, the indicator health value of the target transaction data under each monitoring indicator is determined.

[0115] S604, based on the comprehensive health value of each target transaction data, identify abnormal transaction data in each original transaction data.

[0116] S605 compares the health values ​​of the target transaction data corresponding to the abnormal transaction data under each monitoring indicator with the health thresholds corresponding to each monitoring indicator. The monitoring indicators whose health values ​​are less than the corresponding health thresholds are regarded as abnormal factors corresponding to the abnormal transaction data.

[0117] The specific processes of S601-S605 described above can be found in the description of the above method embodiments. Their implementation principles and technical effects are similar and will not be repeated here.

[0118] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0119] Based on the same inventive concept, this application also provides a data monitoring device for implementing the data monitoring method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more data monitoring device embodiments provided below can be found in the limitations of the data monitoring method described above, and will not be repeated here.

[0120] In one embodiment, such as Figure 7 As shown, a data monitoring device 1 is provided, including: a data processing module 10, a forest construction module 20, a health value determination module 30, and an anomaly determination module 40, wherein:

[0121] Data processing module 10 is used to process each original transaction data in the original transaction dataset according to at least one monitoring indicator to obtain the target transaction data corresponding to each original transaction data.

[0122] Forest building module 20 is used to build isolated forests corresponding to each monitoring indicator based on the isolated forest algorithm and the transaction data of each target.

[0123] The health value determination module 30 is used to determine the comprehensive health value of each target transaction data based on the isolated forest corresponding to each monitoring indicator.

[0124] The anomaly determination module 40 is used to determine the abnormal transaction data in each original transaction data based on the comprehensive health value of each target transaction data.

[0125] In one embodiment, the data processing module 10 is specifically used for:

[0126] For each piece of raw transaction data in the original transaction dataset, determine the indicator data of that raw transaction data under each monitoring indicator; then concatenate the indicator data of that raw transaction data under each monitoring indicator to obtain the target transaction data corresponding to that raw transaction data.

[0127] In one embodiment, such as Figure 8 As shown, the forest building block 20 includes:

[0128] Data set construction unit 21 is used to construct an indicator dataset for each monitoring indicator based on the indicator data of each target transaction data under that monitoring indicator.

[0129] Forest building unit 22 is used to select different indicator data as split points from the indicator dataset, and use the split points to split the indicator dataset to obtain the isolated forest corresponding to the monitoring indicator.

[0130] In one embodiment, such as Figure 9 As shown, in Figure 7 Based on this, the health value determination module 30 includes:

[0131] The first determining unit 31 is used to determine the indicator health value of each target transaction data under each monitoring indicator based on the isolated forest corresponding to each monitoring indicator.

[0132] The second determining unit 32 is used to take the sum of the health values ​​of the target transaction data under each monitoring indicator as the comprehensive health value of the target transaction data.

[0133] In one embodiment, the first determining unit 31 includes:

[0134] The average height determination sub-unit is used to determine the average height of the target transaction data under each monitoring indicator based on the isolated forest corresponding to each monitoring indicator;

[0135] The health value determination subunit is used to determine the health value of the target transaction data under each monitoring indicator based on the weight of each monitoring indicator and the average height of the target transaction data under each monitoring indicator.

[0136] In one embodiment, the average height determining subunit is specifically used for:

[0137] For each monitoring indicator, the height value of the target transaction data under each isolated tree in the isolated forest corresponding to the monitoring indicator is determined; the average height value of the target transaction data under each isolated tree is taken as the average height of the target transaction data under the monitoring indicator.

[0138] In one embodiment, the data monitoring device 1 further includes a factor determination module, which is specifically used for:

[0139] The health values ​​of the target transaction data corresponding to the abnormal transaction data under each monitoring indicator are compared with the health thresholds corresponding to each monitoring indicator; the monitoring indicators whose health values ​​are less than the corresponding health thresholds are regarded as abnormal factors corresponding to the abnormal transaction data.

[0140] Each module in the aforementioned data monitoring device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0141] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores raw transaction data and other data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data monitoring method.

[0142] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0143] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0144] Based on at least one monitoring indicator, process each original transaction data in the original transaction dataset to obtain the target transaction data corresponding to each original transaction data.

[0145] Based on the isolated forest algorithm, an isolated forest corresponding to each monitoring indicator is constructed according to the transaction data of each target.

[0146] Based on the isolated forest corresponding to each monitoring indicator, determine the comprehensive health value of each target transaction data;

[0147] Based on the overall health value of each target transaction data, abnormal transaction data in each original transaction data is identified.

[0148] In one embodiment, when the processor executes the logic in the computer program that processes each original transaction data in the original transaction dataset according to at least one monitoring indicator to obtain the target transaction data corresponding to each original transaction data, the following steps are specifically implemented:

[0149] For each piece of raw transaction data in the original transaction dataset, determine the indicator data of that raw transaction data under each monitoring indicator; then concatenate the indicator data of that raw transaction data under each monitoring indicator to obtain the target transaction data corresponding to that raw transaction data.

[0150] In one embodiment, when the processor executes the logic in the computer program that constructs an isolated forest corresponding to each monitoring indicator based on each target transaction data using the isolated forest algorithm, the specific steps are as follows:

[0151] For each monitoring indicator, an indicator dataset is constructed based on the indicator data of each target transaction data under that monitoring indicator. Different indicator data are selected from the indicator dataset as split points, and the indicator dataset is split using the split points to obtain the isolated forest corresponding to that monitoring indicator.

[0152] In one embodiment, when the processor executes the logic in the computer program to determine the overall health value of each target transaction data based on the isolated forest corresponding to each monitoring indicator, the following steps are specifically implemented:

[0153] For each target transaction data, the health value of the target transaction data under each monitoring indicator is determined based on the isolated forest corresponding to each monitoring indicator; the sum of the health values ​​of the target transaction data under each monitoring indicator is taken as the comprehensive health value of the target transaction data.

[0154] In one embodiment, when the processor executes the logic in the computer program to determine the health value of the target transaction data under each monitoring indicator based on the isolated forest corresponding to each monitoring indicator, the following steps are specifically implemented:

[0155] Based on the isolated forest corresponding to each monitoring indicator, determine the average height of the target transaction data under each monitoring indicator; based on the weight of each monitoring indicator and the average height of the target transaction data under each monitoring indicator, determine the indicator health value of the target transaction data under each monitoring indicator.

[0156] In one embodiment, when the processor executes the logic in the computer program to determine the average height of the target transaction data under each monitoring indicator based on the isolated forest corresponding to each monitoring indicator, the following steps are specifically implemented:

[0157] For each monitoring indicator, the height value of the target transaction data under each isolated tree in the isolated forest corresponding to the monitoring indicator is determined; the average height value of the target transaction data under each isolated tree is taken as the average height of the target transaction data under the monitoring indicator.

[0158] In one embodiment, when the processor executes a computer program, it specifically implements the following steps:

[0159] The health values ​​of the target transaction data corresponding to the abnormal transaction data under each monitoring indicator are compared with the health thresholds corresponding to each monitoring indicator; the monitoring indicators whose health values ​​are less than the corresponding health thresholds are regarded as abnormal factors corresponding to the abnormal transaction data.

[0160] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0161] Based on at least one monitoring indicator, process each original transaction data in the original transaction dataset to obtain the target transaction data corresponding to each original transaction data.

[0162] Based on the isolated forest algorithm, an isolated forest corresponding to each monitoring indicator is constructed according to the transaction data of each target.

[0163] Based on the isolated forest corresponding to each monitoring indicator, determine the comprehensive health value of each target transaction data;

[0164] Based on the overall health value of each target transaction data, abnormal transaction data in each original transaction data is identified.

[0165] In one embodiment, when the code logic in a computer program that processes each raw transaction data in the raw transaction dataset according to at least one monitoring indicator to obtain the target transaction data corresponding to each raw transaction data is executed by the processor, the following steps are specifically implemented:

[0166] For each piece of raw transaction data in the original transaction dataset, determine the indicator data of that raw transaction data under each monitoring indicator; then concatenate the indicator data of that raw transaction data under each monitoring indicator to obtain the target transaction data corresponding to that raw transaction data.

[0167] In one embodiment, when the processor executes the code logic in the computer program that constructs an isolated forest corresponding to each monitoring indicator based on each target transaction data using the isolated forest algorithm, the specific steps are as follows:

[0168] For each monitoring indicator, an indicator dataset is constructed based on the indicator data of each target transaction data under that monitoring indicator. Different indicator data are selected from the indicator dataset as split points, and the indicator dataset is split using the split points to obtain the isolated forest corresponding to that monitoring indicator.

[0169] In one embodiment, when the processor executes the code logic in the computer program that determines the overall health value of each target transaction data based on the isolated forest corresponding to each monitoring indicator, the following steps are specifically implemented:

[0170] For each target transaction data, the health value of the target transaction data under each monitoring indicator is determined based on the isolated forest corresponding to each monitoring indicator; the sum of the health values ​​of the target transaction data under each monitoring indicator is taken as the comprehensive health value of the target transaction data.

[0171] In one embodiment, when the processor executes the code logic in the computer program that determines the health value of the target transaction data under each monitoring indicator based on the isolated forest corresponding to each monitoring indicator, the specific steps are as follows:

[0172] Based on the isolated forest corresponding to each monitoring indicator, determine the average height of the target transaction data under each monitoring indicator; based on the weight of each monitoring indicator and the average height of the target transaction data under each monitoring indicator, determine the indicator health value of the target transaction data under each monitoring indicator.

[0173] In one embodiment, when the processor executes the code logic in the computer program that determines the average height of the target transaction data under each monitoring indicator based on the isolated forest corresponding to each monitoring indicator, the specific steps are as follows:

[0174] For each monitoring indicator, the height value of the target transaction data under each isolated tree in the isolated forest corresponding to the monitoring indicator is determined; the average height value of the target transaction data under each isolated tree is taken as the average height of the target transaction data under the monitoring indicator.

[0175] In one embodiment, when the code logic in a computer program is executed by a processor, the following steps are specifically implemented:

[0176] The health values ​​of the target transaction data corresponding to the abnormal transaction data under each monitoring indicator are compared with the health thresholds corresponding to each monitoring indicator; the monitoring indicators whose health values ​​are less than the corresponding health thresholds are regarded as abnormal factors corresponding to the abnormal transaction data.

[0177] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0178] Based on at least one monitoring indicator, process each original transaction data in the original transaction dataset to obtain the target transaction data corresponding to each original transaction data.

[0179] Based on the isolated forest algorithm, an isolated forest corresponding to each monitoring indicator is constructed according to the transaction data of each target.

[0180] Based on the isolated forest corresponding to each monitoring indicator, determine the comprehensive health value of each target transaction data;

[0181] Based on the overall health value of each target transaction data, abnormal transaction data in each original transaction data is identified.

[0182] In one embodiment, when a computer program is executed by a processor to process each piece of raw transaction data in the raw transaction dataset according to at least one monitoring indicator to obtain the target transaction data corresponding to each piece of raw transaction data, the following steps are specifically implemented:

[0183] For each piece of raw transaction data in the original transaction dataset, determine the indicator data of that raw transaction data under each monitoring indicator; then concatenate the indicator data of that raw transaction data under each monitoring indicator to obtain the target transaction data corresponding to that raw transaction data.

[0184] In one embodiment, when a computer program is executed by a processor to construct an isolated forest corresponding to each monitoring indicator based on the isolated forest algorithm according to each target transaction data, the following steps are specifically implemented:

[0185] For each monitoring indicator, an indicator dataset is constructed based on the indicator data of each target transaction data under that monitoring indicator. Different indicator data are selected from the indicator dataset as split points, and the indicator dataset is split using the split points to obtain the isolated forest corresponding to that monitoring indicator.

[0186] In one embodiment, when a computer program is executed by a processor to determine the overall health value of each target transaction data based on the isolated forest corresponding to each monitoring indicator, the following steps are specifically implemented:

[0187] For each target transaction data, the health value of the target transaction data under each monitoring indicator is determined based on the isolated forest corresponding to each monitoring indicator; the sum of the health values ​​of the target transaction data under each monitoring indicator is taken as the comprehensive health value of the target transaction data.

[0188] In one embodiment, when a computer program is executed by a processor to determine the health value of the target transaction data under each monitoring indicator based on the isolated forest corresponding to each monitoring indicator, the following steps are specifically implemented:

[0189] Based on the isolated forest corresponding to each monitoring indicator, determine the average height of the target transaction data under each monitoring indicator; based on the weight of each monitoring indicator and the average height of the target transaction data under each monitoring indicator, determine the indicator health value of the target transaction data under each monitoring indicator.

[0190] In one embodiment, when a computer program is executed by a processor to determine the average height of the target transaction data under each monitoring metric based on the isolated forest corresponding to each monitoring metric, the following steps are specifically implemented:

[0191] For each monitoring indicator, the height value of the target transaction data under each isolated tree in the isolated forest corresponding to the monitoring indicator is determined; the average height value of the target transaction data under each isolated tree is taken as the average height of the target transaction data under the monitoring indicator.

[0192] In one embodiment, when a computer program is executed by a processor, the following steps are specifically implemented:

[0193] The health values ​​of the target transaction data corresponding to the abnormal transaction data under each monitoring indicator are compared with the health thresholds corresponding to each monitoring indicator; the monitoring indicators whose health values ​​are less than the corresponding health thresholds are regarded as abnormal factors corresponding to the abnormal transaction data.

[0194] It should be noted that the data involved in this application (including but not limited to original transaction data, monitoring indicators, etc.) are all data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0195] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0196] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0197] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data monitoring method, characterized in that, The method includes: Based on at least one monitoring indicator, process each original transaction data in the original transaction dataset to obtain the target transaction data corresponding to each original transaction data. Based on the isolated forest algorithm, an isolated forest corresponding to each monitoring indicator is constructed according to the transaction data of each target. Based on the isolated forest corresponding to each monitoring indicator, determine the comprehensive health value of each target transaction data; Based on the overall health value of each target transaction data, abnormal transaction data in each original transaction data is identified; The process of determining the overall health value of each target transaction data based on the isolated forest corresponding to each monitoring indicator includes: For each target transaction data point, the average height of that target transaction data point under each monitoring indicator is determined based on the isolated forest corresponding to each monitoring indicator. Based on the weight of each monitoring indicator and the average height of the target transaction data under each monitoring indicator, determine the indicator health value of the target transaction data under each monitoring indicator. The sum of the health values ​​of the target transaction data under each monitoring indicator is taken as the comprehensive health value of the target transaction data.

2. The method according to claim 1, characterized in that, The step of processing each original transaction data in the original transaction dataset according to at least one monitoring indicator to obtain the target transaction data corresponding to each original transaction data includes: For each piece of raw transaction data in the raw transaction dataset, determine the indicator data for that raw transaction data under each monitoring indicator; The indicator data under each monitoring indicator of the original transaction data are spliced ​​together to obtain the target transaction data corresponding to the original transaction data.

3. The method according to claim 1, characterized in that, The method based on the isolated forest algorithm constructs isolated forests corresponding to each monitoring indicator based on the target transaction data, including: For each monitoring metric, construct a metric dataset based on the metric data of each target transaction under that monitoring metric; Different indicator data are selected from the indicator dataset as split points, and the indicator dataset is split using the split points to obtain the isolated forest corresponding to the monitoring indicator.

4. The method according to claim 1, characterized in that, The step of determining the average height of the target transaction data under each monitoring indicator based on the isolated forest corresponding to each monitoring indicator includes: For each monitoring indicator, the height value of the target transaction data under each isolated tree is determined based on the tree structure of each isolated tree in the isolated forest corresponding to that monitoring indicator. The average height of the target transaction data under each isolated tree is taken as the average height of the target transaction data under this monitoring indicator.

5. The method according to claim 1, characterized in that, The method further includes: The health values ​​of the target transaction data corresponding to the abnormal transaction data under each monitoring indicator are compared with the health thresholds corresponding to each monitoring indicator. Monitoring indicators whose health values ​​are lower than the corresponding health thresholds are considered as abnormal factors corresponding to the abnormal transaction data.

6. A data monitoring device, characterized in that, The device includes: The data processing module is used to process each original transaction data in the original transaction dataset according to at least one monitoring indicator to obtain the target transaction data corresponding to each original transaction data. The forest building module is used to construct isolated forests corresponding to various monitoring indicators based on the isolated forest algorithm and the transaction data of each target. The health value determination module is used to determine the comprehensive health value of each target transaction data based on the isolated forest corresponding to each monitoring indicator. The anomaly detection module is used to identify abnormal transaction data in each original transaction data based on the comprehensive health value of each target transaction data. The health value determination module is specifically used for: For each target transaction data point, the average height of that target transaction data point under each monitoring indicator is determined based on the isolated forest corresponding to each monitoring indicator. Based on the weight of each monitoring indicator and the average height of the target transaction data under each monitoring indicator, determine the indicator health value of the target transaction data under each monitoring indicator. The sum of the health values ​​of the target transaction data under each monitoring indicator is taken as the comprehensive health value of the target transaction data.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images