GC early warning method, system and storage medium based on gradient boosting regression

Abstract

The present invention relates to artificial intelligence, and provides a GC early warning method based on gradient boosting regression. By performing sample processing, a GC prediction model based on gradient boosting regression is used to determine whether a major pause occurs, and then an attribution analysis model based on an unsupervised clustering algorithm is used to perform attribution analysis on GC indicator data determined to have a major pause, and then the attribution analysis result is output and an early warning is issued. The GC early warning method based on gradient boosting regression of the present invention realizes intelligent control of GC operation and maintenance, and achieves the technical effects of fast processing speed, high accuracy, small memory usage, and strong business availability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more particularly to a GC early warning method, system, electronic device and computer-readable storage medium based on gradient boosting regression. Background Art

[0002] With the popularization and enhancement of functions of smart terminals such as smartphones, tablets, and smart home appliances, users have higher and higher requirements for the performance of smart terminals, and therefore, higher requirements for AI intelligent operation and maintenance. However, most of the current research on AI intelligent operation and maintenance focuses on CPU and memory, while there is a lack of effective control over the GC (Garbage Collection) operation and maintenance link, which is more important in intelligent operation and maintenance.

[0003] The disadvantages of the current GC operation and maintenance scenarios are as follows:

[0004] 1) When obtaining operation and maintenance data, it is difficult to obtain all field data in a similar time period because the sampling frequencies of each process are different;

[0005] 2) There are many element fields in the operation and maintenance data, and the correlation between fields is large. It is a big problem to filter and retain the interpretable field information;

[0006] 3) In the process of modeling using operation and maintenance data, the proportion of negative samples is too small. For example, samples with Major Pause only account for 0.1%, which causes the model to be extremely biased towards positive samples and poorly identify negative samples.

[0007] Therefore, a GC early warning method with higher accuracy is urgently needed. Summary of the invention

[0008] The present invention provides a GC early warning method, system, electronic device and computer-readable storage medium based on gradient boosting regression, which are used to solve the problem of low GC early warning accuracy in the prior art.

[0009] To achieve the above object, the present invention provides a GC early warning method based on gradient boosting regression, the method comprising:

[0010] Obtain the original GC indicator data through the JAVA-level API interface;

[0011] Normalizing and characterizing the original GC index data to obtain a GC feature vector;

[0012] Inputting the obtained GC feature vector into a pre-trained GC prediction model based on gradient boosting regression; determining whether a pause exceeding a preset time threshold occurs by using the GC prediction model based on gradient boosting regression;

[0013] An early warning is issued for a pause that is determined to exceed a preset time threshold.

[0014] Further, preferably, the method further comprises: performing attribution analysis on GC indicator data determined to have paused for more than a preset time threshold, and displaying the attribution analysis result.

[0015] Further, preferably, the training process of the GC prediction model based on gradient boosting regression includes:

[0016] The GC data source data is subjected to a data extraction method based on time fuzzy matching to form a GC training data set based on time matching;

[0017] Performing new sample synthesis on the undersampled samples in the time-matching-based GC training data set until the ratio of positive and negative samples in the GC training data set reaches a set value;

[0018] Performing data normalization and feature processing on the GC training data set to obtain a GC feature vector;

[0019] The GC feature vector is input into the original GC prediction model, the number of iteration steps, the large gradient data sampling rate, and the small gradient data sampling rate are set, and the model is trained with MSE as the loss function to obtain a trained GC prediction model based on gradient boosting regression.

[0020] Further, preferably, the GC data source data is subjected to a data extraction method based on time fuzzy matching to form a GC training data set based on time matching, including:

[0021] Obtain the connection primary key of each data table through the data table field of the GC data source;

[0022] Constructing a data table sequence with time difference according to the time field of the connection primary key;

[0023] Extract data from the data table sequence with time difference to obtain data samples within adjacent time periods;

[0024] The data samples within the adjacent time are used to form a GC training data set based on time matching.

[0025] Further, preferably, synthesizing new samples from under-sampled samples in the time-matching-based GC training data set includes:

[0026] The distance between each sample in the minority class and other minority class samples is calculated using the Euclidean distance as the standard;

[0027] Determine a set Xn of K nearest neighbors of each minority class x by using the distance, where n=1, 2...K;

[0028] Determine the sampling ratio N according to the preset sample imbalance ratio;

[0029] For each minority class x, select its k nearest neighbors in turn, repeat N-1 times; synthesize new samples according to the new sample formula; where,

[0030] The new sample formula is as follows:

[0031] x new =x+rand(0,1)*|xx n |

[0032] Among them, x is the selected minority class sample; x n is a sample randomly selected from the k nearest neighbors of x.

[0033] Further, preferably, the calculation of the distance between each sample in the minority class and other minority class samples based on the Euclidean distance is implemented by the following formula:

[0034]

[0035] Among them, X is the selected minority class sample set; Y is the other minority class sample set; x i is the i-th dimension feature of x; y i is the i-th dimension feature of y; n is the total number of dimension features.

[0036] Further, preferably, in the step of performing data normalization processing on the GC training data set and performing feature processing to obtain a GC feature vector, feature processing is performed on the GC feature vector by a feature processing algorithm; the feature processing algorithm includes a feature histogram, gradient-based one-sided sampling, and mutually exclusive feature bundling.

[0037] In order to solve the above problems, the present invention also provides a GC prediction and early warning system based on gradient boosting regression, the system comprising:

[0038] The data acquisition unit is used to obtain the original GC indicator data through the JAVA-level API interface;

[0039] A GC feature vector acquisition unit performs normalization and characterization processing on the original GC index data to obtain a GC feature vector;

[0040] The GC prediction model processing unit based on gradient boosting regression inputs the obtained GC feature vector into a pre-trained GC prediction model based on gradient boosting regression; and determines whether a pause exceeding a preset time threshold occurs by using the GC prediction model based on gradient boosting regression;

[0041] The early warning unit issues an early warning for a pause that is determined to exceed a preset time threshold.

[0042] In order to solve the above problems, the present invention also provides an electronic device, which includes: a memory storing at least one instruction; and a processor executing the instructions stored in the memory to implement the steps in the above-mentioned GC early warning method based on gradient boosting regression.

[0043] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is executed by a processor in an electronic device to implement the above-mentioned GC early warning method based on gradient boosting regression.

[0044] The present invention provides a GC prediction and early warning method, system, electronic device and storage medium based on gradient boosting regression, which obtains original GC index data through the API interface at the JAVA level; normalizes and characterizes the original GC index data to obtain GC feature vectors; inputs the obtained GC feature vectors into the pre-trained GC prediction model based on gradient boosting regression; determines whether a pause exceeding a preset time threshold occurs through the GC prediction model based on gradient boosting regression; and issues an early warning for pauses determined to exceed the preset time threshold. The beneficial effects are as follows: by characterizing and normalizing the original GC index data, field data of similar time periods are obtained; and interpretable field information is screened and retained; pauses exceeding the preset time threshold are screened out through the GC prediction model based on gradient boosting regression; and then the root cause analysis task is further performed on the pauses exceeding the preset time threshold, so that the cause can be found when GC occurs, so as to solve the corresponding problem quickly later. It can be applied to large-scale GC early warning projects in business. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a flow chart of a GC early warning method based on gradient boosting regression provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a module of a GC prediction and early warning system based on gradient boosting regression provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of the internal structure of an electronic device for implementing a GC early warning method based on gradient boosting regression provided by an embodiment of the present invention;

[0048] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0050] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0051] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0052] The GC early warning method based on gradient boosting regression of the present invention is applied to the GC operation and maintenance process of the garbage collector. Taking the PS (Parallel Scavenge) collector as an example, when the PS collector executes GC (Garbage Collection), GC data is obtained through GC callback. The GC data contains rich information such as GC type, GC occurrence time, GC duration, memory usage of each partition of the new generation and the old generation before and after GC occurs, etc., based on which the measurement indicators are constructed and calculated. Among them, GC types include FGC (FullGC, referred to as FGC) and YGC (YoungGC, referred to as YGC).

[0053] Reference Figure 1 FIG. 1 is a flow chart of a GC early warning method based on gradient boosting regression provided by an embodiment of the present invention. The method can be executed by a system, which can be implemented by software and / or hardware.

[0054] In this embodiment, the GC early warning method based on gradient boosting regression includes S110 to S140:

[0055] S110, obtaining original GC indicator data through a JAVA-level API interface;

[0056] The original GC indicator data can be obtained by using PrintGCStats to obtain the GC log, and then parsing the GC log. The GC of the process can be obtained through Processinfo and rancher info.

[0057] The GC early warning method based on gradient boosting regression can be applied to scenarios including but not limited to the Serial collector, ParNew collector, Parallel Scavenge collector, CMS collector, and G1 collector.

[0058] S120, normalizing and characterizing the original GC index data to obtain a GC feature vector; that is, normalizing and characterizing the original GC index data to determine parameter data for input into a pre-trained GC prediction model based on gradient boosting regression.

[0059] The training process of the GC prediction model based on gradient boosting regression includes: characterizing the original GC index data to obtain new GC index data, normalizing the new GC index data to form a GC data set; dividing the GC data set into a training set and a test set; training the gradient boosting regression GBR model with the training set to obtain a GC prediction model based on gradient boosting regression; inputting the test set into the GC prediction model based on gradient boosting regression to obtain a preliminary prediction value of GC data, using the preliminary prediction value of GC data to obtain a final prediction value of GC data through a linear regression method, and using a loss function to adjust the parameters of the GC prediction model based on gradient boosting regression until the loss function value meets a preset condition, thereby obtaining a trained GC prediction model based on gradient boosting regression. Specifically, it includes steps S121 to S124.

[0060] S121, subjecting the GC data source data to a data extraction method based on time fuzzy matching to form a GC training data set based on time matching.

[0061] The GC data sources include the PrintGCStats data table, the Processinfo data table, and the rancher info data table.

[0062] The method of extracting GC data source data through a data extraction method based on time fuzzy matching to form a GC training data set based on time matching includes:

[0063] S1211, obtaining the connection primary key of each data table through the data table field of the GC data source;

[0064] S1212, constructing a data table sequence with a time difference according to the time field of the connection primary key;

[0065] S1213, extracting data from the data table sequence with time differences to obtain data samples within adjacent time periods;

[0066] S1214: Use data samples in adjacent time periods to form a GC training data set based on time matching.

[0067] As an improvement of this embodiment, after the time-matching-based GC training data set is formed, the method further includes preprocessing the time-matching-based GC training data set for missing value and outlier analysis.

[0068] S122, synthesizing new samples from the undersampled samples in the time-matching-based GC training data set until the ratio of positive and negative samples in the GC training data set reaches a set value. In a specific implementation process, the set value is 10:1.

[0069] It should be noted that in the existing technology, when using operation and maintenance data for modeling, the proportion of negative samples is too small, for example, the samples with Major Pause only account for 0.1%, which leads to the problem that the model is extremely biased towards positive samples and poorly recognizes negative samples. Therefore, a combination of undersampling and oversampling is proposed. Random undersampling is used for the majority class, but random oversampling of the minority class is likely to cause overfitting, which will make the information learned by the model too specific and not generalized enough. Therefore, it is necessary to analyze the minority class samples and artificially synthesize new samples.

[0070] Specifically, the method for synthesizing new samples from undersampled samples in the time-matching-based GC training data set includes:

[0071] S1221. Calculate the distance between each sample in the minority class and samples in other minority classes using the Euclidean distance as the standard;

[0072] The calculation of the distance between each sample in the minority class and other minority class samples based on the Euclidean distance is implemented by the following formula:

[0073]

[0074] Among them, X is the selected minority class sample set; Y is the other minority class sample set; x i is the i-th dimension feature of x; y i is the i-th dimension feature of y; n is the total number of dimension features.

[0075] S1222, determining a set Xn of K nearest neighbors of each minority class x by using the distance, where n=1, 2...K;

[0076] S1223, determining a sampling ratio N according to a preset sample imbalance ratio;

[0077] S1224. Select k nearest neighbors of each minority class x in turn, and repeat N-1 times; synthesize new samples according to the new sample formula; the new sample formula is as follows:

[0078] x new =x+rand(0,1)*|xx n |

[0079] Among them, x is the selected minority class sample; x n is a sample randomly selected from the k nearest neighbors of x.

[0080] A GC training dataset with a positive-negative sample ratio of 10:1 was constructed in the above way. The results showed that the sample imbalance was effectively alleviated.

[0081] S123, performing data normalization processing and feature processing on the GC training data set to obtain a GC feature vector;

[0082] In the step of performing data normalization processing on the GC training data set and performing feature processing to obtain the GC feature vector, the GC feature vector is feature processed by a feature processing algorithm; the feature processing algorithm includes a feature histogram, gradient-based single-side sampling, and mutually exclusive feature bundling. Specifically, the method of performing feature processing on the GC feature vector by a feature processing algorithm includes: determining the data volume and value of the original attribute data; determining the feature processing algorithm according to the data volume and value; if the data volume and value of the original attribute data exceed the set threshold, then using a feature histogram, gradient-based single-side sampling, and mutually exclusive feature bundling to perform feature processing on the features corresponding to the original attribute data; if the data volume and value of the original attribute data do not exceed the set threshold, then using a feature histogram and gradient-based single-side sampling to perform feature processing on the features corresponding to the original attribute data.

[0083] Among them, the GC training data set is subjected to data normalization, that is, the target variable is normalized first and then restored. Normalization first can speed up the gradient descent to find the optimal solution and improve the accuracy. The implementation method is as follows: (1) Find the fields with discrete feature values ​​and percentages, and keep them without normalization. (2) Perform MaxAbsscaler normalization on the remaining variables. (3) Set an upper threshold limit for the target variable to ensure accurate restoration when receiving new data.

[0084] The method of performing mutually exclusive bundling processing on GC feature vectors through a feature processing algorithm includes: performing mutually exclusive bundling processing on GC feature vectors; first inputting feature F, maximum number of conflicts K, and graph G. Constructing a weighted graph, the weights correspond to the total conflicts between the features. Arranging the features in descending order according to their degrees in the graph. Checking each feature in the list in turn, assigning it to an existing bundle set with small conflicts, or creating a new bundle set. Performing operations on each feature to obtain the final feature bundle set bundles. In addition, by using mutually exclusive feature bundling to process feature information, instead of scanning all features to obtain the best split point, some features are bundled to achieve the purpose of reducing the feature dimension, thereby reducing the consumption of finding the best split point.

[0085] The method of performing histogram feature processing on the GC feature vector through the feature processing algorithm includes: using the histogram algorithm, putting the discretized features into the histogram, and only needing to find the optimal segmentation point according to the discrete value of the histogram during traversal, which not only simplifies the expression of the data and reduces the use of memory, but also brings a certain regularization effect, which can prevent the model we make from overfitting and have better generalizability. The algorithm is implemented as follows: the continuous features are discretized into K integers. A histogram with a width of K is constructed. When traversing the data, the discretized values ​​are used as indexes for statistics. Finally, the optimal segmentation point is found according to the statistics of the histogram.

[0086] S124, inputting the GC feature vector into the original GC prediction model, setting the number of iteration steps, large gradient data sampling rate, and small gradient data sampling rate, performing model training with MSE as the loss function, and obtaining a trained GC prediction model based on gradient boosting regression.

[0087] In the specific implementation process, the classifier effect is not good during the modeling process. By selecting the decision tree-based gradient boosting for regression, the gradient-based unilateral sampling algorithm is used, that is, instead of using the sample points used to calculate the gradient, the sample is sampled to calculate the gradient. Specifically, first input the training data, the number of iterations d, the large gradient data sampling rate m, the small gradient data sampling rate n, use MSE as the loss function, and the weak learner is defined as the decision tree; calculate the absolute value of the gradient of each sample point, and sort the absolute value of the gradient of each sample point in descending order; select the first m*100% of the samples after sorting to generate a subset M of large gradient sample points; randomly select n*(1-m)*100% of the sample points from the remaining samples as the set N of small gradient sample points. Merge the sets M and N to get the set A, and multiply the small gradient samples in N by the weight coefficient (1-m) / n. Use the set A to learn a new weak learner, repeat until convergence, and finally output the trained strong learner.

[0088] In short, the original gradient boosting decision tree (GBDT) algorithm adopts the idea that "the negative gradient of the loss function is approximately equal to the residual". Compared with the LightGBM algorithm, other tree model algorithms based on the Boosting framework use a random sampling strategy to extract a certain number of samples to participate in the construction of the decision tree and perform gradient updates each time a decision tree is constructed. The LightGBM algorithm uses a gradient unilateral sampling strategy to specifically use samples with larger gradients to construct decision trees, thereby ensuring the integrity of the data distribution of the samples, and at the same time randomly sampling samples with smaller gradients to construct decision trees.

[0089] The Gradient Boosting Regression (GBR) method can quickly and accurately predict time series data. Gradient boosting constructs trees in a continuous manner, and each tree attempts to correct the errors of the previous tree. By default, gradient boosting constructs trees in a continuous manner, and each tree attempts to correct the errors of the previous tree. By default, there is no randomization in the gradient boosting regression tree, but strong pre-pruning is used. Gradient boosting trees usually use trees with a very small depth (between 1 and 5), so that the model takes up less memory and the prediction speed is faster.

[0090] S130, inputting the obtained GC feature vector into a pre-trained GC prediction model based on gradient boosting regression; and determining whether a pause exceeding a preset time threshold occurs by using the GC prediction model based on gradient boosting regression.

[0091] S140: issuing an early warning for a pause that is determined to exceed a preset time threshold.

[0092] S150, performing attribution analysis on GC indicator data determined to have paused for more than a preset time threshold, and displaying the attribution analysis results; wherein the attribution analysis is implemented by an attribution analysis model based on an unsupervised clustering algorithm.

[0093] Specifically, the attribution analysis of GC indicator data that is determined to have a major pause is implemented through an attribution analysis model based on an unsupervised clustering algorithm. The training method of the attribution analysis model based on an unsupervised clustering algorithm is to extract negative samples from the prediction results to form a new training set, and use an unsupervised clustering algorithm. The algorithm is implemented as follows: 1) Randomly generate K cluster centers within the sample space. 2) Calculate the distance from each sample point to each center and assign it to the class corresponding to the nearest center. 3) Recalculate the centers of each class that has been obtained. 4) Iterate steps 2) and 3) until the new center is equal to or less than the previous centroid. In the specific implementation process, it is usually set to K = 3, 4, and 5, and field analysis is performed on the results corresponding to different K values. By visualizing the clustering effect, K = 3 is finally selected, corresponding to three types of reasons: CPU usage, heap memory usage, and number of threads. After the GC warning, it can help locate the cause of the occurrence and quickly solve the problem.

[0094] The GC early warning method based on gradient boosting regression of the present invention performs sample processing, determines whether a major pause occurs by using the GC prediction model based on gradient boosting regression, and performs attribution analysis on GC indicator data determined to have a major pause by using an attribution analysis model based on an unsupervised clustering algorithm. The attribution analysis result is output, and an early warning is issued. The GC early warning method based on gradient boosting regression of the present invention has the characteristics of fast processing speed, high accuracy, small memory usage, and strong business availability.

[0095] Specifically, the improvement of the algorithm reduces the sample time complexity, thereby speeding up the model processing speed, and can meet the industry's task time requirements during GC warning. The GC-prediction and warning model established by the present invention achieves an F1 value of 0.8501 on the stock test data, reaching a relatively high level; and the model framework is lightweight, has low memory consumption, and can be used for large data parallel processing. In addition, the root cause analysis task completed by the model can find out the cause when GC occurs, so as to quickly solve the corresponding problem later, which fully meets business needs and can be applied to large-scale GC warning projects in business.

[0096] The GC early warning method based on gradient boosting regression of the present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0097] like Figure 2 As shown, the present invention provides a GC prediction and early warning system 200 based on gradient boosting regression, which can be installed in an electronic device. According to the functions implemented, the GC prediction and early warning system 200 based on gradient boosting regression can include a data acquisition unit 210, a GC feature vector acquisition unit 220, a GC prediction model processing unit 230 based on gradient boosting regression, and an early warning unit 240. The unit described in the present invention can also be called a module, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0098] In this embodiment, the functions of each module / unit are as follows:

[0099] The data acquisition unit 210 is used to acquire the original GC indicator data through the JAVA-level API interface;

[0100] The GC feature vector acquisition unit 220 performs normalization and characterization processing on the original GC index data to obtain a GC feature vector;

[0101] The GC prediction model processing unit 230 based on gradient boosting regression inputs the obtained GC feature vector into the pre-trained GC prediction model based on gradient boosting regression; and determines whether a pause exceeding a preset time threshold occurs by using the GC prediction model based on gradient boosting regression;

[0102] The warning unit 240 issues a warning for a pause that is determined to exceed a preset time threshold.

[0103] Furthermore, the GC prediction model processing unit 220 based on gradient boosting regression includes a GC feature vector acquisition module (not shown in the figure), a GC prediction model training module based on gradient boosting regression (not shown in the figure) and a GC feature vector input module (not shown in the figure); the GC feature vector acquisition module is used to normalize and characterize the original GC indicator data to obtain a GC feature vector; the GC prediction model training module based on gradient boosting regression is used to train and obtain a GC prediction model training module based on gradient boosting regression; the GC feature vector input module is used to input the obtained GC feature vector into a pre-trained GC prediction model based on gradient boosting regression.

[0104] The GC prediction model training module based on gradient boosting regression includes a GC training data set acquisition submodule, a positive and negative sample ratio adjustment submodule, a GC feature vector acquisition submodule and a GC prediction model training submodule based on gradient boosting regression;

[0105] Among them, the GC training data set acquisition submodule is used to extract the GC data source data through a data extraction method based on time fuzzy matching to form a GC training data set based on time matching; the positive and negative sample ratio adjustment submodule is used to synthesize new samples for the undersampled samples in the GC training data set based on time matching until the positive and negative sample ratio in the GC training data set reaches a set value; the GC feature vector acquisition submodule is used to perform data normalization and feature processing on the GC training data set to obtain a GC feature vector; the GC prediction model training submodule based on gradient boosting regression is used to input the GC feature vector into the original GC prediction model, set the number of iteration steps, large gradient data sampling rate, and small gradient data sampling rate, and perform model training with MSE as the loss function to obtain a trained GC prediction model based on gradient boosting regression.

[0106] The GC prediction and early warning system 200 based on gradient boosting regression of the present invention reduces the sample time complexity by improving the algorithm, thereby speeding up the model processing speed, and can meet the industry's task time requirements during GC early warning. The GC-prediction and early warning model established by the present invention achieves an F1 value of 0.8501 on the stock test data, reaching a relatively high level; and the model framework is lightweight, has low memory consumption, and can be used for large data parallel processing. In addition, the root cause analysis task completed by the model can find out the cause when GC occurs, so as to quickly solve the corresponding problem later, fully meet business needs, and achieve the technical effect of projects that can be applied to large-scale GC early warning in business.

[0107] like Figure 3 As shown, the present invention provides an electronic device 3 of a GC early warning method based on gradient boosting regression.

[0108] The electronic device 3 may include a processor 30, a memory 31 and a bus, and may also include a computer program stored in the memory 31 and executable on the processor 30, such as a GC prediction and early warning program 32 based on gradient boosting regression. The memory 31 may also include both an internal storage unit of the GC prediction and early warning system based on gradient boosting regression of artificial intelligence and an external storage device. The memory 31 may be used not only to store application software and various data installed in the artificial intelligence auxiliary device, such as the code of the artificial intelligence auxiliary program, but also to temporarily store data that has been output or is to be output.

[0109] The memory 31 includes at least one type of readable storage medium, including flash memory, mobile hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 31 may be an internal storage unit of the electronic device 3, such as a mobile hard disk of the electronic device 3. In other embodiments, the memory 31 may also be an external storage device of the electronic device 3, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card, etc. equipped on the electronic device 3. Further, the memory 31 may also include both an internal storage unit and an external storage device of the electronic device 3. The memory 31 may not only be used to store application software and various types of data installed in the electronic device 3, such as the code of the GC prediction and early warning program based on gradient boosting regression, but also be used to temporarily store data that has been output or is to be output.

[0110] The processor 30 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 30 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 31 (such as GC prediction and early warning programs based on gradient boosting regression, etc.), and calls data stored in the memory 31 to execute various functions of the electronic device 3 and process data.

[0111] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 31 and at least one processor 30, etc.

[0112] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the electronic device 3, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0113] For example, although not shown, the electronic device 3 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 30 through a power management system, so that the power management system can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators. The electronic device 3 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0114] Furthermore, the electronic device 3 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 3 and other electronic devices.

[0115] Optionally, the electronic device 3 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 3 and to display a visual user interface.

[0116] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0117] The GC prediction and warning program 32 based on gradient boosting regression stored in the memory 31 in the electronic device 3 is a combination of multiple instructions. When running in the processor 30, it can achieve: obtaining original GC indicator data through the JAVA-level API interface; normalizing and characterizing the original GC indicator data to obtain a GC feature vector; inputting the obtained GC feature vector into a pre-trained GC prediction model based on gradient boosting regression; determining whether a pause exceeding a preset time threshold occurs through the GC prediction model based on gradient boosting regression; and issuing a warning for a pause determined to exceed the preset time threshold.

[0118] Further, preferably, the training process of the GC prediction model based on gradient boosting regression includes: obtaining original GC indicator data through a JAVA-level API interface; normalizing and characterizing the original GC indicator data to obtain a GC feature vector; inputting the obtained GC feature vector into a pre-trained GC prediction model based on gradient boosting regression; determining whether a pause exceeding a preset time threshold occurs through the GC prediction model based on gradient boosting regression; performing attribution analysis on GC indicator data determined to have a pause exceeding the preset time threshold; outputting the attribution analysis result and issuing an early warning.

[0119] Furthermore, the training process of the GC prediction model based on gradient boosting regression includes: subjecting the GC data source data to a data extraction method based on time fuzzy matching to form a GC training data set based on time matching; synthesizing new samples for the undersampled samples in the GC training data set based on time matching until the ratio of positive and negative samples in the GC training data set reaches a set value; performing data normalization and feature processing on the GC training data set to obtain a GC feature vector; inputting the GC feature vector into the original GC prediction model, setting the number of iteration steps, large gradient data sampling rate, and small gradient data sampling rate, and performing model training with MSE as the loss function to obtain a trained GC prediction model based on gradient boosting regression.

[0120] Further, preferably, the method of subjecting the GC data source data to a data extraction method based on time fuzzy matching to form a GC training data set based on time matching includes: obtaining the connection primary key of each data table through the data table field of the GC data source; constructing a data table sequence with a time difference according to the time field of the connection primary key; performing data extraction on the data table sequence with a time difference to obtain data samples within adjacent time; and forming a GC training data set based on time matching using the data samples within adjacent time.

[0121] Further, preferably, the method of synthesizing new samples from the undersampled samples in the time-matching-based GC training data set until the ratio of positive and negative samples in the GC training data set reaches a set value comprises: calculating the distance from each sample in the minority class to other minority class samples based on the Euclidean distance;

[0122] Determine a set Xn of K nearest neighbors of each minority class x by using the distance, where n=1, 2...K;

[0123] Determine the sampling ratio N according to the preset sample imbalance ratio;

[0124] For each minority class x, select its k nearest neighbors in turn, and repeat N-1 times;

[0125] The formula for generating new samples is as follows:

[0126] x new =x+rand(0,1)*|xx n |

[0127] Among them, x is the selected minority class sample; x n is a sample randomly selected from the k nearest neighbors of x.

[0128] Further, preferably, the calculation of the distance between each sample in the minority class and other minority class samples based on the Euclidean distance is implemented by the following formula:

[0129]

[0130] Among them, X is the selected minority class sample set; Y is the other minority class sample set; x i is the i-th dimension feature of x; y i is the i-th dimension feature of y; n is the total number of dimension features.

[0131] Further, preferably, in the step of performing data normalization processing on the GC training data set and performing feature processing to obtain a GC feature vector, feature processing is performed on the GC feature vector by a feature processing algorithm; the feature processing algorithm includes a feature histogram, gradient-based one-sided sampling, and mutually exclusive feature bundling.

[0132] Further, preferably, the GC data source includes a PrintGCStats data table, a Processinfo data table, and a rancher info data table.

[0133] Specifically, the specific implementation method of the processor 30 for the above instructions can refer to Figure 1The description of the relevant steps in the corresponding embodiment is not repeated here. It should be emphasized that in order to further ensure the privacy and security of the above-mentioned GC prediction and early warning program based on gradient boosting regression, the above-mentioned database high-availability processing data is stored in the node of the blockchain where the server cluster is located.

[0134] Furthermore, if the module / unit integrated in the electronic device 3 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or system capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0135] An embodiment of the present invention also provides a computer-readable storage medium, which may be non-volatile or volatile, and stores a computer program. When the computer program is executed by a processor, it implements: obtaining original GC indicator data through a JAVA-level API interface; normalizing and characterizing the original GC indicator data to obtain a GC feature vector; inputting the obtained GC feature vector into a pre-trained GC prediction model based on gradient boosting regression; determining whether a pause exceeding a preset time threshold occurs through the GC prediction model based on gradient boosting regression; and issuing an early warning for a pause determined to exceed the preset time threshold.

[0136] Furthermore, the training process of the GC prediction model based on gradient boosting regression includes: subjecting the GC data source data to a data extraction method based on time fuzzy matching to form a GC training data set based on time matching; synthesizing new samples for the undersampled samples in the GC training data set based on time matching until the ratio of positive and negative samples in the GC training data set reaches a set value; performing data normalization and feature processing on the GC training data set to obtain a GC feature vector; inputting the GC feature vector into the original GC prediction model, setting the number of iteration steps, large gradient data sampling rate, and small gradient data sampling rate, and performing model training with MSE as the loss function to obtain a trained GC prediction model based on gradient boosting regression.

[0137] Further, preferably, the method of subjecting the GC data source data to a data extraction method based on time fuzzy matching to form a GC training data set based on time matching includes: obtaining the connection primary key of each data table through the data table field of the GC data source; constructing a data table sequence with a time difference according to the time field of the connection primary key; performing data extraction on the data table sequence with a time difference to obtain data samples within adjacent time; and forming a GC training data set based on time matching using the data samples within adjacent time.

[0138] Further, preferably, the method of synthesizing new samples from the undersampled samples in the time-matching-based GC training data set until the ratio of positive and negative samples in the GC training data set reaches a set value comprises: calculating the distance from each sample in the minority class to other minority class samples based on the Euclidean distance;

[0139] Determine a set Xn of K nearest neighbors of each minority class x by using the distance, where n=1, 2...K;

[0140] Determine the sampling ratio N according to the preset sample imbalance ratio;

[0141] For each minority class x, select its k nearest neighbors in turn, and repeat N-1 times;

[0142] The formula for generating new samples is as follows:

[0143] x new =x+rand(0,1)*|xx n |

[0144] Among them, x is the selected minority class sample; x n is a sample randomly selected from the k nearest neighbors of x.

[0145] Further, preferably, the calculation of the distance between each sample in the minority class and other minority class samples based on the Euclidean distance is implemented by the following formula:

[0146]

[0147] Among them, X is the selected minority class sample set; Y is the other minority class sample set; x i is the i-th dimension feature of x; y i is the i-th dimension feature of y; n is the total number of dimension features.

[0148] Further, preferably, in the step of performing data normalization processing on the GC training data set and performing feature processing to obtain a GC feature vector, feature processing is performed on the GC feature vector by a feature processing algorithm; the feature processing algorithm includes a feature histogram, gradient-based one-sided sampling, and mutually exclusive feature bundling.

[0149] Further, preferably, the GC data source includes a PrintGCStats data table, a Processinfo data table, and a rancher info data table.

[0150] Specifically, the specific implementation method when the computer program is executed by the processor can refer to the description of the relevant steps in the GC early warning method based on gradient boosting regression in the embodiment, which will not be repeated here.

[0151] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0152] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0154] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0155] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0156] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0157] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in a system claim can also be implemented by one unit or system through software or hardware. Secondary words are used to indicate names, but not to indicate any particular order.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A GC early warning method based on gradient boosting regression, characterized in that: Methods include: Obtain the original GC indicator data through the JAVA-level API interface; Normalizing and characterizing the original GC index data to obtain a GC feature vector; Inputting the obtained GC feature vector into a pre-trained GC prediction model based on gradient boosting regression; determining whether a pause exceeding a preset time threshold occurs by using the GC prediction model based on gradient boosting regression; Issue an early warning for a pause that exceeds a preset time threshold; The training process of the GC prediction model based on gradient boosting regression includes: The GC data source data is subjected to a data extraction method based on time fuzzy matching to form a GC training data set based on time matching; Performing new sample synthesis on the undersampled samples in the time-matching-based GC training data set until the ratio of positive and negative samples in the GC training data set reaches a set value; Performing data normalization and feature processing on the GC training data set to obtain a GC feature vector; Input the GC feature vector into the original GC prediction model, set the number of iteration steps, large gradient data sampling rate, and small gradient data sampling rate, perform model training with MSE as the loss function, and obtain a trained GC prediction model based on gradient boosting regression; The method of extracting the GC data source data through a data extraction method based on time fuzzy matching to form a GC training data set based on time matching includes: Obtain the connection primary key of each data table through the data table field of the GC data source; Constructing a data table sequence with time difference according to the time field of the connection primary key; Extract data from the data table sequence with time difference to obtain data samples within adjacent time periods; The data samples within the adjacent time are used to form a GC training data set based on time matching.

2. The GC early warning method based on gradient boosting regression as claimed in claim 1, characterized in that: The method further comprises: An attribution analysis is performed on GC indicator data that is determined to have paused for more than a preset time threshold, and the attribution analysis results are displayed.

3. The GC early warning method based on gradient boosting regression as claimed in claim 1, characterized in that: The synthesizing new samples from the under-sampled samples in the time-matching-based GC training data set includes: The distance between each sample in the minority class and other minority class samples is calculated using the Euclidean distance as the standard; Determine each minority class by the distance x of K The set of nearest neighbors Xn ,in, n = 1, 2…K ; Determine the sampling ratio based on the preset sample imbalance ratio N ; For each minority class x Select them in turn k Neighbors, repeat N-1 times; synthesize new samples according to the new sample formula; where, The new sample formula is as follows: in, x is the selected minority class sample; x n for x of k A sample randomly selected from the set of neighbors.

4. The GC early warning method based on gradient boosting regression as claimed in claim 3, characterized in that: The calculation of the distance between each sample in the minority class and other minority class samples based on the Euclidean distance is implemented by the following formula: in, X is the selected minority class sample set; Y is a set of other minority class samples; x i for x No. i Dimensional features; y i for y No. i Dimensional features; n is the total number of dimension features.

5. The GC early warning method based on gradient boosting regression according to any one of claims 1 to 4, characterized in that: In the step of performing data normalization processing on the GC training data set and performing feature processing to obtain a GC feature vector, feature processing is performed on the GC feature vector by a feature processing algorithm; the feature processing algorithm includes a feature histogram, gradient-based single-side sampling, and mutually exclusive feature bundling.

6. A GC prediction and early warning system based on gradient boosting regression, characterized in that: The system comprises: The data acquisition unit is used to obtain the original GC indicator data through the JAVA-level API interface; A GC feature vector acquisition unit performs normalization and characterization processing on the original GC index data to obtain a GC feature vector; The GC prediction model processing unit based on gradient boosting regression inputs the obtained GC feature vector into a pre-trained GC prediction model based on gradient boosting regression; and determines whether a pause exceeding a preset time threshold occurs by using the GC prediction model based on gradient boosting regression; The training process of the GC prediction model based on gradient boosting regression includes: subjecting the GC data source data to a data extraction method based on time fuzzy matching to form a GC training data set based on time matching; synthesizing new samples for undersampled samples in the GC training data set based on time matching until the ratio of positive and negative samples in the GC training data set reaches a set value; performing data normalization processing and feature processing on the GC training data set to obtain a GC feature vector; inputting the GC feature vector into the original GC prediction model, setting the number of iteration steps, large gradient data sampling rate, and small gradient data sampling rate, and performing model training with MSE as the loss function to obtain a trained GC prediction model based on gradient boosting regression; subjecting the GC data source data to a data extraction method based on time fuzzy matching to form a GC training data set based on time matching, including: obtaining the connection primary key of each data table through the data table field of the GC data source; constructing a data table sequence with time difference according to the time field of the connection primary key; performing data extraction on the data table sequence with time difference to obtain data samples in adjacent time; and forming a GC training data set based on time matching using the data samples in adjacent time; The early warning unit issues an early warning for a pause that is determined to exceed a preset time threshold.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the steps in the GC early warning method based on gradient boosting regression as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the GC early warning method based on gradient boosting regression as described in any one of claims 1 to 5 is implemented.

Images