A data monitoring processing method and device

Abstract

The application provides a data monitoring processing method and device, relates to the technical field of data monitoring, and can be applied to the financial field or other technical fields.The method comprises the following steps: acquiring multidimensional data and derived features reflecting system performance risks; the multidimensional data comprises application system basic environment information, virtual machine configuration information and test log information; the derived features are the in-memory memory rising speed, the garbage collection frequency rising speed and the garbage collection time consumption rising speed in the life cycle; the multidimensional data and the derived features are monitored based on a preset monitoring model to obtain a monitoring result; wherein the preset monitoring model is obtained by training a decision tree algorithm model according to multidimensional sample data and derived feature sample data; the device executes the above method.The data monitoring processing method and device provided in the application embodiment can comprehensively and accurately monitor data, and then timely perform system risk prevention and control.

Description

Technical Field

[0001] This invention relates to the field of data monitoring technology, and specifically to a data monitoring and processing method and apparatus. Background Technology

[0002] With the transformation of IT architecture and the gradual advancement of digital transformation, more and more application systems are being transformed into distributed system frameworks based on x86 servers. The application of new system service frameworks and distributed database servers has brought a series of problems to daily technical testing work and has also made it difficult to monitor JVM data. Summary of the Invention

[0003] To address the problems in the prior art, embodiments of the present invention provide a data monitoring and processing method and apparatus, which can at least partially solve the problems existing in the prior art.

[0004] On one hand, the present invention proposes a data monitoring and processing method, comprising:

[0005] The system acquires multi-dimensional data and derived features reflecting system performance risks. The multi-dimensional data includes basic application system environment information, virtual machine configuration information, and test log information. The derived features are the rate of increase in heap memory, the rate of increase in garbage collection count, and the rate of increase in garbage collection time during the system's lifecycle.

[0006] The multi-dimensional data and derived features are monitored based on a preset monitoring model to obtain monitoring results;

[0007] The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data.

[0008] The acquisition of the multi-dimensional sample data includes:

[0009] Acquire initial multi-dimensional data, perform data cleaning on the data fields of the initial multi-dimensional data to obtain model feature indicator system data; the model feature indicator system data includes application system environment indicator system data, virtual machine basic parameter indicator system data, and test log information indicator system data.

[0010] Data cleaning is performed on the data field content of the model feature index system data to obtain test anomaly data during the testing period;

[0011] The test anomaly data is sequentially discretized, normalized, and vectorized, and then labeled to obtain the multi-dimensional sample data.

[0012] The process of obtaining the derived feature sample data includes:

[0013] Get the changes in old generation heap memory data, the changes in garbage collection counts, and the changes in garbage collection time;

[0014] Calculate the ratios of the changes in the old generation heap memory data, the changes in the number of garbage collections, and the changes in the garbage collection time to the preset monitoring period;

[0015] The ratio results are sequentially discretized, normalized, and vectorized, and then labeled to obtain the derived feature sample data.

[0016] The decision tree algorithm model is a distributed gradient boosting framework; correspondingly, training the decision tree algorithm model based on multi-dimensional sample data and derived feature sample data includes:

[0017] Initialize and set the training parameters of the distributed gradient boosting framework;

[0018] The training parameters are adjusted, and the distributed gradient boosting framework is repeatedly trained until the tree depth, leaf node sample weights, and learning weights that avoid overfitting are obtained.

[0019] The data monitoring and processing method further includes:

[0020] After repeated training of the distributed gradient boosting framework, the generalization ability of the distributed gradient boosting framework is tested using both non-cross-validation and cross-validation methods to obtain the preset monitoring model.

[0021] The data monitoring and processing method further includes:

[0022] If the monitoring result is determined to be an abnormal monitoring result, then the weight values ​​of each feature of the preset monitoring model are obtained;

[0023] Arrange the feature weights in descending order and extract the top k feature weights.

[0024] The data monitoring and processing method further includes:

[0025] The training data in the training dataset is updated based on the monitoring results. The training data includes the multi-dimensional sample data and the derived feature sample data.

[0026] On one hand, the present invention proposes a data monitoring and processing device, comprising:

[0027] The acquisition unit is used to acquire multi-dimensional data and derived features reflecting system performance risks; the multi-dimensional data includes basic environment information of the application system, virtual machine configuration information, and test log information; the derived features are the rate of increase in heap memory, the rate of increase in garbage collection times, and the rate of increase in garbage collection time during the life cycle.

[0028] The monitoring unit is used to monitor the multi-dimensional data and the derived features based on a preset monitoring model, and obtain monitoring results;

[0029] The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data.

[0030] In another aspect, embodiments of the present invention provide an electronic device, including: a processor, a memory, and a bus, wherein,

[0031] The processor and the memory communicate with each other via the bus;

[0032] The memory stores program instructions that can be executed by the processor, and the processor can execute the following methods by calling the program instructions:

[0033] The system acquires multi-dimensional data and derived features reflecting system performance risks. The multi-dimensional data includes basic application system environment information, virtual machine configuration information, and test log information. The derived features are the rate of increase in heap memory, the rate of increase in garbage collection count, and the rate of increase in garbage collection time during the system's lifecycle.

[0034] The multi-dimensional data and derived features are monitored based on a preset monitoring model to obtain monitoring results;

[0035] The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data.

[0036] This invention provides a non-transitory computer-readable storage medium, comprising:

[0037] The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the following methods:

[0038] The system acquires multi-dimensional data and derived features reflecting system performance risks. The multi-dimensional data includes basic application system environment information, virtual machine configuration information, and test log information. The derived features are the rate of increase in heap memory, the rate of increase in garbage collection count, and the rate of increase in garbage collection time during the system's lifecycle.

[0039] The multi-dimensional data and derived features are monitored based on a preset monitoring model to obtain monitoring results;

[0040] The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data.

[0041] The data monitoring and processing method and apparatus provided in this invention acquire multi-dimensional data and derived features reflecting system performance risks. The multi-dimensional data includes basic application system environment information, virtual machine configuration information, and test log information. The derived features are the rate of increase in heap memory, the rate of increase in garbage collection times, and the rate of increase in garbage collection time during the lifecycle. The multi-dimensional data and the derived features are monitored based on a preset monitoring model to obtain monitoring results. The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data, which can comprehensively and accurately monitor the data and thus promptly prevent and control system risks. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0043] Figure 1 This is a flowchart illustrating a data monitoring and processing method provided in an embodiment of the present invention.

[0044] Figure 2 This is a flowchart illustrating a data monitoring and processing method provided in another embodiment of the present invention.

[0045] Figure 3 This is a schematic flowchart of the modular data monitoring and processing method provided in the embodiments of the present invention.

[0046] Figure 4 This is a schematic diagram of the structure of a data monitoring and processing device provided in an embodiment of the present invention.

[0047] Figure 5 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.

[0049] Figure 1 This is a flowchart illustrating a data monitoring and processing method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the data monitoring and processing method provided in this embodiment of the invention includes:

[0050] Step S1: Obtain multi-dimensional data and derived features reflecting system performance risks; the multi-dimensional data includes basic environment information of the application system, virtual machine configuration information, and test log information; the derived features are the rate of increase in heap memory, the rate of increase in garbage collection times, and the rate of increase in garbage collection time during the life cycle.

[0051] Step S2: Monitor the multi-dimensional data and the derived features based on the preset monitoring model to obtain monitoring results;

[0052] The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data.

[0053] In step S1 above, the device acquires multi-dimensional data and derived features reflecting system performance risks; the multi-dimensional data includes basic application system environment information, virtual machine configuration information, and test log information; the derived features are the rate of increase in heap memory usage, the rate of increase in garbage collection counts, and the rate of increase in garbage collection time over the lifetime. The device can be a computer device that executes the method, such as a server. It should be noted that the acquisition and analysis of data involved in this embodiment of the invention are authorized by the user.

[0054] The basic environment information of the application system may include relevant physical information during system setup, including but not limited to application name, operating system type, test system environment, test cluster nodes, version number, release time, number of CPUs, memory, etc.

[0055] Virtual machine configuration information can specifically refer to Java Virtual Machine configuration information, which is the Java Virtual Machine configuration information generated by system services during initialization. This includes, but is not limited to, initial heap memory, maximum heap memory, minimum heap memory, initial young generation memory, maximum young generation memory, Eden space / Survivor space ratio, old generation / young generation ratio, old generation memory, permanent generation memory, GC processing mechanism, throughput throughpu, pause, etc.

[0056] Test log information records the response status of all services in a single test process. The content includes, but is not limited to, statistical time, process ID, CPU utilization, memory status, GC type, heap information (survivor space size, Eden space size, old generation size), GC start time, GC end time, number of garbage collections, garbage collection time, and total GC time.

[0057] The derived features reflect system performance risks, as explained below:

[0058] The faster the heap memory increases during its lifecycle, the greater the risk of system performance anomalies; the slower the heap memory increases during its lifecycle, the smaller the risk of system performance anomalies.

[0059] The faster the rate of increase in garbage collection frequency, the greater the risk of system performance anomalies; the slower the rate of increase in garbage collection frequency, the lower the risk of system performance anomalies. Garbage collection (GC) is a process of garbage collection.

[0060] The faster the garbage collection time increases, the greater the risk of system performance anomalies; the slower the garbage collection time increases, the smaller the risk of system performance anomalies.

[0061] In step S2 above, the device monitors the multi-dimensional data and the derived features based on a preset monitoring model to obtain monitoring results; the multi-dimensional data and derived features can be integrated and input into the preset monitoring model as a whole, and the output of the preset monitoring model can be used as the monitoring result.

[0062] Monitoring results can include normal monitoring results, indicating that all data monitoring results are normal.

[0063] Monitoring results can include abnormal monitoring results, indicating that there are anomalies in the data monitoring results.

[0064] The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data. Acquiring multi-dimensional sample data includes:

[0065] Initial multi-dimensional data is acquired, and the data fields of the initial multi-dimensional data are cleaned to obtain model feature indicator system data; the model feature indicator system data includes application system environment indicator system data, virtual machine basic parameter indicator system data, and test log information indicator system data; such as Figure 2 As shown:

[0066] Step 1: This embodiment of the invention involves three main entities: the application system, the database, and the Java Virtual Machine (JVM). Treating an application system as a whole, starting from the application system, data such as its system version, database, JVM, and daily test logs are collected. This can be broken down into three dimensions: basic application system environment information, JVM configuration information, and test log information.

[0067] The basic environment information of the application system may include, but is not limited to, data such as application name, operating system type, test system environment, and test cluster nodes.

[0068] Virtual machine configuration information may include, but is not limited to, initial heap memory, maximum heap memory, minimum heap memory, GC (Garbage Collection) processing mechanism, and other data.

[0069] Test log information refers to the specific processing information of the relevant links involved in a test.

[0070] Step 2: Construction of the model feature indicator system data. Based on different data source dimensions, this is broken down into three parts: application system environment indicator system data, virtual machine basic parameter indicator system data, and test log information indicator system data. Specific indicator system data is shown in Table 1.

[0071] Table 1

[0072]

[0073]

[0074] (1) Application system environment indicator system: The application system environment mainly refers to the relevant physical information when the system is built, including but not limited to application name, operating system type, test system environment, test cluster nodes, version number, release time, number of CPUs, memory, etc.

[0075] (2) JVM basic parameter index system: JVM basic parameters mainly refer to the Java Virtual Machine configuration information generated by the system service during the initialization process, including but not limited to initial heap memory, maximum heap memory, minimum heap memory, initial young generation memory, maximum young generation memory, Eden space / Survivor space ratio, old generation / young generation ratio, old generation memory, permanent generation memory, GC processing mechanism, throughput throughpu, pause, etc.

[0076] (3) Test Log Information Indicator System: Test log information is recorded on a per-test basis, documenting the response status of all service information. To obtain key log information, natural language processing technology is needed to identify keywords in the log text, locate the position and content of keywords, and extract target data information. This content includes, but is not limited to, statistical time, process ID, CPU utilization, memory status, GC type, heap information (survivor space size, Eden space size, old generation size), GC start time, GC end time, number of garbage collections, garbage collection time, and total GC time.

[0077] Step 3: Feature engineering mainly involves processing and selecting raw data collected from multiple dimensions. However, the initial dataset contains various types of data, including structured and unstructured data. Furthermore, due to the large number of data sources, there is a problem of not being able to match them one by one. Therefore, it is necessary to process different types of data in different ways and finally summarize them into the initial training set required for model training.

[0078] Data cleaning is performed on the data fields of the initial multi-dimensional data to remove data fields other than the model feature index system data, thereby achieving preliminary dimensionality reduction of the data and obtaining the model feature index system data.

[0079] Key information can be extracted from test logs to obtain test log information indicator system data, as explained below:

[0080] a. Divide the obtained log files according to the unique test ID generated during each test, and there are a total of N test information entries, namely B0={b1,b2,b3,…,bN};

[0081] b. Perform text splitting, parsing, and data extraction on B0. Identify the application name, test system environment, test time, GC type, GC time, and other related information in each test log record to obtain target feature values. Traverse all log instances in the log file text to obtain the log-related feature vector BsN = {test ID, application name list, test system environment list, GC type list, ...}.

[0082] c. Repeat steps a and b N times until the traversal is complete, forming the log information feature matrix, T = {Bs1, Bs2, ..., BsN}.

[0083] Data cleaning is performed on the data fields of the model feature index system to obtain abnormal test data during the testing period; the specific rules for data cleaning are as follows:

[0084] if (GC type = NULL) —> This test record is normal and is not an object to be analyzed.

[0085] if (GC type != NULL) —> This test record is abnormal and is an object to be analyzed.

[0086] Define the data retrieval time interval as: the time when the last test ID occurred - the time when the first test ID occurred.

[0087] if (data acquisition time interval >= 'set threshold') —> This test record is abnormal and is not the object to be analyzed.

[0088] if (data acquisition time interval < 'set threshold') —> the test record is normal and is the object to be analyzed.

[0089] The abnormal test data is sequentially discretized, normalized, and vectorized, and then labeled to obtain the multi-dimensional sample data. The discretization process is explained below:

[0090] The data will be divided according to a set threshold and represented using Boolean values, such as initial heap memory, maximum heap memory, minimum heap memory, etc.

[0091] if (field value < 'set threshold') —> field is assigned a value of 1, otherwise it is assigned a value of 0.

[0092] The normalization is explained as follows:

[0093] This involves mapping data values ​​to [0,1] to eliminate the influence of units on subsequent model construction, such as the number of CPUs, memory, garbage collection counts, garbage collection time, and total GC time. The rules are as follows:

[0094] W* = (W - Wmin) / (Wmax - Wmin).

[0095] The model feature index system data constructed in step 2 and the derived features in step 3 are treated as a whole. Each record is vectorized, and then all records are normalized. That is, the real value matrix composed of each field in the index dimension is vectorized to obtain the real feature matrix T.

[0096] Step 4: Based on the relevant conceptual standards of the JVM garbage collection mechanism, define and label the GC types that are commonly generated in the current system. Since there is no single GC type, the negative analysis objects initially screened are further refined and processed according to GC type. The greater the impact of GC on performance, the higher the value is assigned. The specific standardization method is shown in Table 2:

[0097] Table 2

[0098] Minor GC (Young Generation Garbage Collection) 1 Major GC (Old Generation GC) 2 Global GC (full GC) 3

[0099] Further, derived features under different GC types are identified. Different basic thresholds are set based on the daily occurrence frequency of different GC types. Test IDs that may have performance issues are marked as 1, and those that are not are marked as 0. Taking Full GC as an example, the specific rules are as follows:

[0100] Rule 1: if (Full GC frequency >= 'base threshold 1') —> Yes: assign a value of 1, no: assign a value of 0;

[0101] Rule 2: if (the rate of increase of Full GC count >= 'base threshold 2') —> Yes: assign a value of 1, no: assign a value of 0;

[0102] Rule 3: if (Full GC time increase rate >= 'base threshold 3') —> Yes: assign 1, no: assign 0;

[0103] Rule 4: if (old generation heap memory growth rate during lifetime >= 'base threshold 4') —> Yes: assign a value of 1, no: assign a value of 0;

[0104] Taking into account all GC types, each GC type corresponds to four judgment rules. Finally, for the target test ID, a discriminant value matrix is ​​constructed, namely G = {rule 1, rule 2, rule 3, ...}.

[0105] Output the test anomaly judgment. If the test ID does not meet any of the rules, it is judged as no anomaly; if the test ID meets one or more judgment rules, it is judged as an anomaly. The more judgment rules are met, the more serious the anomaly. The specific judgment rules are shown in Table 3.

[0106] Table 3

[0107] Test1 G1＝{0,0,0，…} 0 Test2 G2＝{1,0,0，…} 1 Test 3 G3＝{1,1,0，…} 2 Test4 G3＝{1,0,1，…} 2 Test5 G3＝{1,1,1，…} 3 …… …… ……

[0108] Obtaining the derived feature sample data includes:

[0109] Get the changes in old generation heap memory data, the changes in garbage collection counts, and the changes in garbage collection time;

[0110] Calculate the ratios of the changes in old generation heap memory data, the changes in garbage collection frequency, and the changes in garbage collection time to the preset monitoring period. The preset monitoring period can be set independently according to actual conditions. Calculate the results of each ratio using the following formulas:

[0111] Heap memory growth rate over lifecycle = Change in old generation heap memory data / Preset monitoring period;

[0112] The rate of increase in waste collection frequency = change in waste collection frequency / preset monitoring cycle;

[0113] The rate of increase in waste collection time = change in waste collection time / preset monitoring cycle;

[0114] The ratio results are sequentially discretized, normalized, and vectorized, and then labeled to obtain the derived feature sample data. This can be referred to the above description and will not be repeated here.

[0115] Step 5: The decision tree algorithm model is a distributed gradient boosting framework, namely LightGBM (LightGradient Boosting Machine).

[0116] The model feature index system data in step 2 above is integrated with the discrimination rules shown in Table 2 in step 4 to obtain the initial feature matrix of the model to be trained.

[0117] The real feature matrix T from step 3 and the discriminant matrix G from step 4 are integrated and used as the input to the model to be trained.

[0118] Accordingly, a decision tree algorithm model is trained based on multi-dimensional sample data and derived feature sample data, including:

[0119] Initialize the training parameters of the distributed gradient boosting framework; define the basic XGBoost model and initialize the general type parameters, booster parameters, and learning task parameters.

[0120] The training parameters are adjusted, and the distributed gradient boosting framework is repeatedly trained until the optimal tree depth, leaf node sample weights, and learning weights to avoid overfitting are obtained. The model is then repeatedly trained by adjusting the parameters until the optimal settings for these parameters are determined, thus fine-tuning the model.

[0121] The entire training process can be implemented in the following way:

[0122] a. Define the basic XGBoost model and initialize the general type parameters, booster parameters, and learning task parameters;

[0123] b. Based on the Histogram algorithm, find the optimal split point, with a constant number of candidate split points.

[0124] c. Change the sampling method from random sampling to sampling samples with smaller absolute gradient values ​​according to a certain proportion, while retaining the one-sided sampling method for samples with larger absolute gradient values.

[0125] d. Bind features that cannot simultaneously take the same value to reduce the dimensionality of the data features;

[0126] e. By adjusting the parameters, repeatedly train the model until the optimal settings for parameters such as tree depth, leaf node sample weights, and learning weights are determined to avoid overfitting, and then optimize the model.

[0127] f. The generalization ability of the model is tested using both non-cross-validation and cross-validation methods, and the optimal training model LightGBM is finally obtained.

[0128] Step 6, Final Monitoring Analysis and Anomaly Feedback and Early Warning, mainly consists of the following steps:

[0129] First, auxiliary analysis. Based on the model constructed in step 5, it is believed that the larger the feature weight value of a single feature, the greater its influence. Therefore, the feature weights that affect the degree of anomalies (the feature weight value of each feature is one of the contents of the model training output) are sorted, and the weights from largest to smallest are denoted as TOP={(H1: weight(H1)),H2: weight(H2)),…,Hn: weight(Hn))}. The top 5 with the largest weight values ​​are selected as the key indicator set REC={H1,H2,H3,H4,H5} to assist subsequent manual analysis.

[0130] Second, monitoring and anomaly early warning. Based on the target test content, the latest test data and related JVM data within the system are obtained. An anomaly early warning training model is used to determine whether new test data carries the risk of OutOfMemoryError (OOM) or any anomalies, and the judgment result is output. Simultaneously, the relevant weighted index values ​​of the key indicator set REC are fed back to assist technical personnel in taking timely emergency measures, such as adjusting memory size and application system settings, to promptly avoid performance risks and system anomalies.

[0131] Step 7: Continuous Optimization. Based on the model feedback, add new training data to the training dataset to iteratively optimize the training model.

[0132] like Figure 3 As shown, the method of this invention can be implemented based on modularity, specifically including:

[0133] Data acquisition module: used to acquire multi-dimensional data and derived features that reflect system performance risks.

[0134] Data Feature System Construction Module: Used to construct the model feature indicator system data.

[0135] Feature engineering processing module: Used to obtain multi-dimensional sample data and derived feature sample data through feature engineering processing.

[0136] JVM Monitoring and Anomaly Warning Model Training Module: Used to train the decision tree algorithm model to obtain the preset monitoring model.

[0137] Results Feedback and Model Optimization Module: Used to optimize the data in the model training set based on the output results of the model application.

[0138] The data monitoring and processing method provided in this invention acquires multi-dimensional data and derived features reflecting system performance risks. The multi-dimensional data includes basic application system environment information, virtual machine configuration information, and test log information. The derived features are the rate of increase in heap memory, the rate of increase in garbage collection counts, and the rate of increase in garbage collection time during the lifecycle. The multi-dimensional data and the derived features are monitored based on a preset monitoring model to obtain monitoring results. The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data, which can comprehensively and accurately monitor the data and thus promptly prevent and control system risks.

[0139] Further, obtaining the multi-dimensional sample data includes:

[0140] Obtain initial multi-dimensional data, perform data cleaning on the data fields of the initial multi-dimensional data to obtain model feature indicator system data; the model feature indicator system data includes application system environment indicator system data, virtual machine basic parameter indicator system data, and test log information indicator system data; refer to the above description, and will not be repeated here.

[0141] Data cleaning is performed on the data fields of the model feature index system to obtain abnormal test data during the testing period; please refer to the above description, which will not be repeated here.

[0142] The abnormal test data is sequentially discretized, normalized, and vectorized, and then labeled to obtain the multi-dimensional sample data. This can be referred to the above description and will not be repeated here.

[0143] The data monitoring and processing method provided in this embodiment of the invention achieves data dimensionality reduction through data processing, which can improve model training efficiency.

[0144] Further, obtaining the derived feature sample data includes:

[0145] To obtain the changes in old generation heap memory data, the number of garbage collections, and the time spent on garbage collection, please refer to the above explanation; further details are omitted here.

[0146] Calculate the ratios of the changes in old generation heap memory data, the changes in the number of garbage collections, and the changes in garbage collection time to the preset monitoring period; refer to the above description, and will not be repeated here.

[0147] The ratio results are sequentially discretized, normalized, and vectorized, and then labeled to obtain the derived feature sample data. This can be referred to the above description and will not be repeated here.

[0148] The data monitoring and processing method provided in this embodiment of the invention achieves data dimensionality reduction through data processing, which can improve model training efficiency.

[0149] Furthermore, the decision tree algorithm model is a distributed gradient boosting framework; correspondingly, training the decision tree algorithm model based on multi-dimensional sample data and derived feature sample data includes:

[0150] Initialize and set the training parameters of the distributed gradient boosting framework; refer to the above instructions, which will not be repeated here.

[0151] Adjust the training parameters and repeatedly train the distributed gradient boosting framework until you obtain the appropriate tree depth, leaf node sample weights, and learning weights to avoid overfitting. Refer to the above description; further details are omitted.

[0152] The data monitoring and processing method provided in this embodiment of the invention can avoid model overfitting.

[0153] Furthermore, the data monitoring and processing method also includes:

[0154] After repeated training of the distributed gradient boosting framework, its generalization ability is tested using both non-cross-validation and cross-validation methods to obtain the preset monitoring model. This can be referred to the above description and will not be repeated here.

[0155] The data monitoring and processing method provided in this embodiment of the invention can improve the generalization ability of the model.

[0156] Furthermore, the data monitoring and processing method also includes:

[0157] If the monitoring result is determined to be an abnormal monitoring result, then the weight values ​​of each feature of the preset monitoring model are obtained; this can be referred to the above description and will not be repeated here.

[0158] Arrange the feature weights in descending order and extract the top k feature weights. Refer to the above explanation; further details are omitted.

[0159] The data monitoring and processing method provided in this embodiment of the invention facilitates users in analyzing the influence of model features on abnormal monitoring results.

[0160] Furthermore, the data monitoring and processing method also includes:

[0161] The training data in the training dataset is updated based on the monitoring results. The training data includes the multi-dimensional sample data and the derived feature sample data. This can be referred to the above description and will not be repeated here.

[0162] The data monitoring and processing method provided in this embodiment of the invention can improve the accuracy of model monitoring by updating the training dataset.

[0163] It should be noted that the data monitoring and processing method provided in this embodiment of the invention can be used in the financial field, or in any technical field other than the financial field. This embodiment of the invention does not limit the application field of the data monitoring and processing method.

[0164] Figure 4 This is a schematic diagram of the structure of a data monitoring and processing device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the data monitoring and processing device provided in this embodiment of the invention includes an acquisition unit 401 and a monitoring unit 402, wherein:

[0165] The acquisition unit 401 is used to acquire multi-dimensional data and derived features reflecting system performance risks; the multi-dimensional data includes basic environment information of the application system, virtual machine configuration information, and test log information; the derived features are the rate of increase in heap memory, the rate of increase in garbage collection times, and the rate of increase in garbage collection time during the lifetime; the monitoring unit 402 is used to monitor the multi-dimensional data and the derived features based on a preset monitoring model to obtain monitoring results; wherein, the preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data.

[0166] Specifically, the acquisition unit 401 in the device is used to acquire multi-dimensional data and derived features reflecting system performance risks; the multi-dimensional data includes basic environment information of the application system, virtual machine configuration information, and test log information; the derived features are the rate of increase in heap memory, the rate of increase in garbage collection times, and the rate of increase in garbage collection time during the lifetime; the monitoring unit 402 is used to monitor the multi-dimensional data and the derived features based on a preset monitoring model to obtain monitoring results; wherein, the preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data.

[0167] The data monitoring and processing device provided in this embodiment of the invention acquires multi-dimensional data and derived features reflecting system performance risks. The multi-dimensional data includes basic environment information of the application system, virtual machine configuration information, and test log information. The derived features are the rate of increase in heap memory, the rate of increase in garbage collection times, and the rate of increase in garbage collection time during the lifecycle. The device monitors the multi-dimensional data and the derived features based on a preset monitoring model to obtain monitoring results. The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data, which can comprehensively and accurately monitor the data and thus promptly prevent and control system risks.

[0168] Furthermore, the data monitoring and processing device is also used for:

[0169] Acquire initial multi-dimensional data, perform data cleaning on the data fields of the initial multi-dimensional data to obtain model feature indicator system data; the model feature indicator system data includes application system environment indicator system data, virtual machine basic parameter indicator system data, and test log information indicator system data.

[0170] Data cleaning is performed on the data field content of the model feature index system data to obtain test anomaly data during the testing period;

[0171] The test anomaly data is sequentially discretized, normalized, and vectorized, and then labeled to obtain the multi-dimensional sample data.

[0172] The data monitoring and processing device provided in this embodiment of the invention achieves data dimensionality reduction through data processing, which can improve model training efficiency.

[0173] Furthermore, the data monitoring and processing device is also used for:

[0174] Get the changes in old generation heap memory data, the changes in garbage collection counts, and the changes in garbage collection time;

[0175] Calculate the ratios of the changes in the old generation heap memory data, the changes in the number of garbage collections, and the changes in the garbage collection time to the preset monitoring period;

[0176] The ratio results are sequentially discretized, normalized, and vectorized, and then labeled to obtain the derived feature sample data.

[0177] The data monitoring and processing device provided in this embodiment of the invention achieves data dimensionality reduction through data processing, which can improve model training efficiency.

[0178] Furthermore, the decision tree algorithm model is a distributed gradient boosting framework; correspondingly, the data monitoring and processing device is also used for:

[0179] Initialize and set the training parameters of the distributed gradient boosting framework;

[0180] The training parameters are adjusted, and the distributed gradient boosting framework is repeatedly trained until the tree depth, leaf node sample weights, and learning weights that avoid overfitting are obtained.

[0181] The data monitoring and processing device provided in this embodiment of the invention can avoid model overfitting.

[0182] Furthermore, the data monitoring and processing device is also used for:

[0183] After repeated training of the distributed gradient boosting framework, the generalization ability of the distributed gradient boosting framework is tested using both non-cross-validation and cross-validation methods to obtain the preset monitoring model.

[0184] The data monitoring and processing device provided in this embodiment of the invention can improve the generalization ability of the model.

[0185] Furthermore, the data monitoring and processing device is also used for:

[0186] If the monitoring result is determined to be an abnormal monitoring result, then the weight values ​​of each feature of the preset monitoring model are obtained;

[0187] Arrange the feature weights in descending order and extract the top k feature weights.

[0188] The data monitoring and processing device provided in this embodiment of the invention facilitates users in analyzing the influence of model features on anomaly monitoring results.

[0189] Furthermore, the data monitoring and processing device is also used for:

[0190] The training data in the training dataset is updated based on the monitoring results. The training data includes the multi-dimensional sample data and the derived feature sample data.

[0191] The data monitoring and processing device provided in this embodiment of the invention can improve the accuracy of model monitoring by updating the training dataset.

[0192] The embodiments of the data monitoring and processing device provided in this invention can be used to execute the processing flow of the above-described method embodiments. Its functions will not be repeated here, but can be referred to the detailed description of the above-described method embodiments.

[0193] Figure 5 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 5 As shown, the electronic device includes: a processor 501, a memory 502, and a bus 503;

[0194] The processor 501 and the memory 502 communicate with each other via the bus 503.

[0195] The processor 501 is used to call program instructions in the memory 502 to execute the methods provided in the above-described method embodiments, including, for example:

[0196] The system acquires multi-dimensional data and derived features reflecting system performance risks. The multi-dimensional data includes basic application system environment information, virtual machine configuration information, and test log information. The derived features are the rate of increase in heap memory, the rate of increase in garbage collection count, and the rate of increase in garbage collection time during the system's lifecycle.

[0197] The multi-dimensional data and derived features are monitored based on a preset monitoring model to obtain monitoring results;

[0198] The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data.

[0199] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can perform the methods provided in the above-described method embodiments, such as:

[0200] The system acquires multi-dimensional data and derived features reflecting system performance risks. The multi-dimensional data includes basic application system environment information, virtual machine configuration information, and test log information. The derived features are the rate of increase in heap memory, the rate of increase in garbage collection count, and the rate of increase in garbage collection time during the system's lifecycle.

[0201] The multi-dimensional data and derived features are monitored based on a preset monitoring model to obtain monitoring results;

[0202] The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data.

[0203] This embodiment provides a computer-readable storage medium storing a computer program that causes the computer to execute the methods provided in the above-described method embodiments, including, for example:

[0204] The system acquires multi-dimensional data and derived features reflecting system performance risks. The multi-dimensional data includes basic application system environment information, virtual machine configuration information, and test log information. The derived features are the rate of increase in heap memory, the rate of increase in garbage collection count, and the rate of increase in garbage collection time during the system's lifecycle.

[0205] The multi-dimensional data and derived features are monitored based on a preset monitoring model to obtain monitoring results;

[0206] The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data.

[0207] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0208] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0209] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0210] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0211] In the description of this specification, the references to terms such as "an embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0212] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data monitoring processing method, characterized by, include: Obtain multi-dimensional data and derived features that reflect system performance risks; The multi-dimensional data includes basic application system environment information, virtual machine configuration information, and test log information; the derived features are the rate of increase in heap memory, the rate of increase in garbage collection times, and the rate of increase in garbage collection time during the lifecycle. The multi-dimensional data and derived features are monitored based on a preset monitoring model to obtain monitoring results; The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data; The acquisition of the multi-dimensional sample data includes: Acquire initial multi-dimensional data, perform data cleaning on the data fields of the initial multi-dimensional data to obtain model feature indicator system data; the model feature indicator system data includes application system environment indicator system data, virtual machine basic parameter indicator system data, and test log information indicator system data. Data cleaning is performed on the data field content of the model feature index system data to obtain test anomaly data during the testing period; The abnormal test data is sequentially discretized, normalized, and vectorized, and then labeled to obtain the multi-dimensional sample data. The discrimination markers include: Classify and assign values ​​to waste recycling types; Set corresponding basic threshold values ​​for derived features based on different levels of waste recycling types; A discriminant matrix is ​​generated based on whether each derived feature exceeds the corresponding basic threshold. The anomaly level is determined by the number of rules in the discriminant matrix that satisfy the threshold condition; the more rules in the discriminant matrix that satisfy the threshold condition, the higher the anomaly level.

2. The data monitoring and processing method according to claim 1, characterized in that, Obtaining the derived feature sample data includes: Get the changes in old generation heap memory data, the changes in garbage collection counts, and the changes in garbage collection time; Calculate the ratios of the changes in the old generation heap memory data, the changes in the number of garbage collections, and the changes in the garbage collection time to the preset monitoring period; The ratio results are sequentially discretized, normalized, and vectorized, and then labeled to obtain the derived feature sample data.

3. The data monitoring and processing method according to any one of claims 1 to 2, characterized in that, The decision tree algorithm model is a distributed gradient boosting framework; correspondingly, the decision tree algorithm model is trained based on multi-dimensional sample data and derived feature sample data, including: Initialize and set the training parameters of the distributed gradient boosting framework; The training parameters are adjusted, and the distributed gradient boosting framework is repeatedly trained until the tree depth, leaf node sample weights, and learning weights that avoid overfitting are obtained.

4. The data monitoring and processing method according to claim 3, characterized in that, The data monitoring and processing method further includes: After repeated training of the distributed gradient boosting framework, the generalization ability of the distributed gradient boosting framework is tested using both non-cross-validation and cross-validation methods to obtain the preset monitoring model.

5. The data monitoring and processing method according to claim 1, characterized in that, The data monitoring and processing method further includes: If the monitoring result is determined to be an abnormal monitoring result, then the weight values ​​of each feature of the preset monitoring model are obtained; Arrange the feature weights in descending order and extract the top k feature weights.

6. The data monitoring and processing method according to claim 1, characterized in that, The data monitoring and processing method further includes: The training data in the training dataset is updated based on the monitoring results. The training data includes the multi-dimensional sample data and the derived feature sample data.

7. A data monitoring and processing device, characterized in that, include: The acquisition unit is used to acquire multi-dimensional data and derived features that reflect system performance risks; The multi-dimensional data includes basic application system environment information, virtual machine configuration information, and test log information; the derived features are the rate of increase in heap memory, the rate of increase in garbage collection times, and the rate of increase in garbage collection time during the lifecycle. The monitoring unit is used to monitor the multi-dimensional data and the derived features based on a preset monitoring model, and obtain monitoring results; The preset monitoring model is obtained by training a decision tree algorithm model based on multi-dimensional sample data and derived feature sample data; The device is also used for: Acquire initial multi-dimensional data, perform data cleaning on the data fields of the initial multi-dimensional data to obtain model feature indicator system data; the model feature indicator system data includes application system environment indicator system data, virtual machine basic parameter indicator system data, and test log information indicator system data. Data cleaning is performed on the data field content of the model feature index system data to obtain test anomaly data during the testing period; The abnormal test data is sequentially discretized, normalized, and vectorized, and then labeled to obtain the multi-dimensional sample data. The discrimination markers include: Classify and assign values ​​to waste recycling types; Set corresponding basic threshold values ​​for derived features based on different levels of waste recycling types; A discriminant matrix is ​​generated based on whether each derived feature exceeds the corresponding basic threshold. The anomaly level is determined by the number of rules in the discriminant matrix that satisfy the threshold condition; the more rules in the discriminant matrix that satisfy the threshold condition, the higher the anomaly level.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images