A problem localization method and apparatus

Abstract

The application discloses a problem positioning method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: obtaining a training set according to a labeled and unlabeled log sample set, performing first training on a problem classification model; taking logs of a preset number of problem scenarios as a test set, performing consistency evaluation on the problem classification model after the first training, performing second training on the problem classification model after the first training according to the consistency evaluation result, repeating the consistency evaluation and the second training process until the final consistency evaluation result meets a preset condition to obtain a trained problem classification model, and determining a problem category in log data of a problem to be positioned. The embodiment can realize automatic and accurate positioning and classification of bugs, help testers avoid low-level problems, enhance the independence of testers, save the cost of repairing problems for developers, has low requirements on test resources, has low computation cost, and has strong self-adaptability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a problem positioning method and device. BACKGROUND

[0002] At present, the bug positioning in the background of test environment is manually searched by the test personnel, and the bug positioning usually depends on the technical level and experience accumulation of the test personnel. When the test personnel cannot position the bug, the test personnel will give the abnormal log screenshot to the R&D for processing. The developer needs to analyze the code for positioning. When a system is responsible for by multiple persons and the logic is complex, the difficulty and cost of problem positioning will be greatly increased. Therefore, the automatic bug positioning in the background of test environment is particularly important.

[0003] The existing bug detection based on path analysis and iterative metamorphic testing mainly makes the test case achieve full path coverage for bug detection based on the white box criterion. This method needs to continuously iterate the test case, and has high requirements for test resources. The software fault positioning model based on BP (back propagation) neural network has high algorithm calculation cost and poor self-adaptive ability. The sorting method of the text similarity between the source code and the defect report does not fully utilize the structural information of the source code, and the accuracy of bug positioning is low. The software defect positioning method based on the code structure information derived from the above method has certain improvement in accuracy, but it still tends to be the defect diagnosis of the code logic, structure and framework of the developer, and can be applied to the self-test link of the development, and can be applied to the code analysis after the test personnel submits the specific bug. The method for automatically positioning the bug in the test environment has not been involved.

[0004] In the process of implementing the present application, the inventors have found that the prior art at least has the following problems:

[0005] The bug cannot be automatically and accurately positioned in the test environment, and the method has high requirements for test resources, high algorithm calculation cost, poor self-adaptive ability and other defects. SUMMARY

[0006] Therefore, the embodiments of the present application provide a problem positioning method and device, which can automatically and accurately position the bug, help the test personnel to avoid some low-level problems, enhance the independence of the test personnel, save the cost of repairing the problem for the developer, and have low requirements for test resources, small calculation cost and strong self-adaptive ability.

[0007] To achieve the above object, according to one aspect of the embodiments of the present application, a problem positioning method is provided.

[0008] The problem positioning method comprises the following steps: obtaining a training set according to a first set of labeled log samples and a second set of unlabeled log samples, performing first training on a problem classification model, and the label is used to indicate a problem category; taking logs of a preset number of problem scenarios as a test set, performing consistency evaluation on the first trained problem classification model by using the test set, and performing second training on the first trained problem classification model according to the consistency evaluation result; the process of the consistency evaluation and the second training is repeated until the final consistency evaluation result meets a preset condition, and the second training is no longer continued, thereby obtaining a trained problem classification model; and inputting log data of a problem to be positioned into the trained problem classification model to determine a problem category in the log data.

[0009] Optionally, the problem classification model is a clustering model, and the step of obtaining a training set according to a first set of labeled log samples and a second set of unlabeled log samples and performing first training on a problem classification model comprises the following steps: determining a plurality of initial clustering centers according to the first set of log samples, calculating the Euclidean distance from each log sample in the first set of log samples and the second set of log samples to each initial clustering center to cluster the log samples, and redetermining the clustering center of each cluster; continuously iterating the process of calculating the Euclidean distance, clustering the log samples, and redetermining the clustering center until the difference between the sum of the Euclidean distances obtained in the k+1 iteration and the sum of the Euclidean distances obtained in the k iteration is less than a set accuracy, and stopping the iteration to complete the first training, and the sum of the Euclidean distances is the sum of the Euclidean distances from the log samples to the clustering centers after the iteration.

[0010] Optionally, the step of performing consistency evaluation on the first trained problem classification model by using the test set and performing second training on the first trained problem classification model according to the consistency evaluation result comprises the following steps: clustering the logs in the test set by using the first trained problem classification model to obtain a problem clustering result; generating a cross-classification table by using the problem clustering result and the expected problem classification result of the logs in the test set; calculating a consistency coefficient according to the cross-classification table, and the consistency coefficient reflects the consistency between the problem clustering result and the expected problem classification result; if the consistency coefficient does not meet the preset condition, the logs in the test set that are inconsistent between the problem clustering result and the expected problem classification result are added to the training set, and the first trained problem classification model is subjected to second training.

[0011] Optionally, the first set of labeled log samples and the second set of unlabeled log samples are obtained by extracting a plurality of abnormal log samples with defect keywords from the set of valid log samples, the abnormal log samples being the valid log samples with problems; extracting a plurality of features of the defect keywords from each abnormal log sample, and performing scalarization and normalization on each feature to obtain a feature vector corresponding to each abnormal log sample; obtaining the first set of log samples according to the feature vectors of the abnormal log samples with known problem categories, and obtaining the second set of log samples according to the feature vectors of the abnormal log samples with unknown problem categories.

[0012] Optionally, the method further comprises: configuring a system requiring log monitoring to collect logs from the system; and filtering the collected logs to obtain the set of valid log samples including a plurality of valid log samples.

[0013] According to another aspect of the embodiments of the present application, a problem positioning device is provided.

[0014] A problem positioning device comprises a first training module, a consistency evaluation module, a second training module, and a problem category prediction module, wherein: the first training module is configured to obtain a training set according to a first set of labeled log samples and a second set of unlabeled log samples, perform first training on a problem classification model, and use the labels to indicate problem categories; the consistency evaluation module is configured to use logs of a preset number of problem scenarios as a test set, perform consistency evaluation on the problem classification model after the first training by using the test set; the second training module is configured to perform second training on the problem classification model after the first training according to the consistency evaluation result; the consistency evaluation module and the second training module repeat the consistency evaluation and the second training until the final consistency evaluation result meets a preset condition, and the second training is stopped, thereby obtaining a trained problem classification model; and the problem category prediction module is configured to input log data of a problem to be positioned into the trained problem classification model to determine a problem category in the log data.

[0015] Optionally, the problem classification model is a clustering model, and the first training module is further configured to: determine a plurality of initial clustering centers according to the first log sample set, calculate the Euclidean distances from each log sample in the first log sample set and the second log sample set to each initial clustering center, cluster each log sample according to the Euclidean distances, and redetermine the clustering center of each cluster; iteratively perform the above processes of calculating the Euclidean distances, clustering each log sample, and redetermining the clustering center until a difference between a sum of Euclidean distances obtained in the k+1th iteration and a sum of Euclidean distances obtained in the kth iteration is less than a preset accuracy, and stop the iteration to complete the first training, wherein the sum of Euclidean distances is a sum of the Euclidean distances from each log sample to each clustering center after the iteration.

[0016] Optionally, the consistency evaluation module is further configured to: use the first trained problem classification model to cluster the logs in the test set to obtain a problem clustering result; use the problem clustering result and the expected problem classification result of the logs in the test set to generate a cross-classification table; calculate a consistency coefficient according to the cross-classification table, wherein the consistency coefficient reflects the consistency between the problem clustering result and the expected problem classification result; and if the consistency coefficient does not satisfy the preset condition, add the logs in the test set, for which the problem clustering result is inconsistent with the expected problem classification result, to the training set, and use the second training module to perform second training on the first trained problem classification model.

[0017] Optionally, the method further includes a training set generation module configured to obtain a first log sample set with labels and a second log sample set without labels, wherein: a plurality of abnormal log samples with defect keywords are extracted from the effective log sample set, the abnormal log samples are effective log samples with problems; a plurality of features of the defect keywords are extracted from each abnormal log sample, and each feature is scalarized and normalized to obtain a feature vector corresponding to each abnormal log sample; the first log sample set is obtained according to the feature vectors of the abnormal log samples with known problem categories, and the second log sample set is obtained according to the feature vectors of the abnormal log samples with unknown problem categories.

[0018] Optionally, the method further includes a system configuration module configured to configure a system that needs to be monitored, a log collection module configured to collect logs from the system, and a log cleaning module configured to filter the collected logs to obtain the effective log sample set including a plurality of effective log samples.

[0019] According to still another aspect of the embodiments of the present application, an electronic device is provided.

[0020] An electronic device includes one or more processors; a memory storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement a problem positioning method provided by the present application.

[0021] According to yet another aspect of the embodiments of the present application, a computer readable medium is provided.

[0022] A computer readable medium having stored thereon a computer program that, when executed by a processor, implements a problem positioning method provided by the present application.

[0023] One embodiment of the above-mentioned application has the following advantages or beneficial effects: the problem classification model is first trained, and the logs of a preset number of problem scenarios are used as a test set to evaluate the consistency of the first trained problem classification model, the first trained problem classification model is second trained according to the consistency evaluation result, the process of consistency evaluation and second training is repeated until the final consistency evaluation result meets the preset condition, and a trained problem classification model is obtained to determine the problem category in the log data of the problem to be positioned. The automatic and accurate positioning and classification of bugs can be realized, the test personnel can be helped to avoid some low-level problems, the independence of the test personnel can be enhanced, the cost of repairing problems can be saved for the developers, the requirement for test resources is not high, the calculation cost is small, and the self-adaptation ability is strong.

[0024] Further effects of the above-mentioned non-conventional optional mode will be described in the following in combination with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings are used to better understand the present application and do not constitute an improper limitation on the present application. Among them:

[0026] Figure 1 is a schematic diagram of the main steps of a problem positioning method according to an embodiment of the present application;

[0027] Figure 2 is a schematic diagram of a problem classification model training process according to an embodiment of the present application;

[0028] Figure 3 is a schematic diagram of a consistency evaluation process of a problem classification model according to an embodiment of the present application;

[0029] Figure 4 is a schematic diagram of the main modules of a problem positioning device according to an embodiment of the present application;

[0030] Figure 5 is a schematic diagram of the overall framework of a problem positioning device according to an embodiment of the present application;

[0031] Figure 6 is a classification diagram of testing environment bugs according to an embodiment of the present application;

[0032] Figure 7 is an exemplary system architecture diagram to which embodiments of the present application can be applied;

[0033] Figure 8 is a structural diagram of a computer system of a terminal device or a server suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0034] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, in which various specific details are set forth to assist in a thorough understanding of the embodiments of the present application. It will be understood by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the spirit and scope of the present application. Also, for the purpose of clarity and the brevity of description, the description below omits the description of well-known functions and structures.

[0035] Those skilled in the art will recognize that the embodiments of the present application can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) or a combination of hardware and software.

[0036] Figure 1 is a main step diagram of a problem positioning method according to an embodiment of the present application.

[0037] As shown in Figure 1 , the problem positioning method according to an embodiment of the present application mainly includes the following steps S101 to S103.

[0038] Step S101: obtaining a training set according to the first set of labeled log samples and the second set of unlabeled log samples, and performing first training on the problem classification model.

[0039] The label is used to indicate the problem category.

[0040] The system requiring monitoring logs can be configured to collect logs from the system, and filter the collected logs to obtain a set of valid log samples including a plurality of valid log samples. The set of valid log samples includes abnormal log samples, and the abnormal log samples are valid log samples with problems (i.e., bugs). Some of the abnormal log samples are of known problem categories, and some of the abnormal log samples are of unknown problem categories. The abnormal log samples of known problem categories are determined by manually analyzing the bugs to determine the problem categories.

[0041] By extracting defect keywords from the effective log samples, abnormal log samples can be obtained. Each type of problem can be identified according to the corresponding defect keyword. The defect keyword is used to represent a bug. The presence of a defect keyword indicates the presence of a bug. For example, the defect keyword java.lang.NullPointerException indicates a null pointer problem, and No aliveprovider indicates a connection problem of upstream interface calling. When a certain effective log sample contains a defect keyword, the effective log sample can be extracted as an abnormal log sample.

[0042] The first set of labeled log samples and the second set of unlabeled log samples can be obtained in the following way: extracting a plurality of abnormal log samples containing defect keywords from the effective log sample set; extracting a plurality of features of the defect keywords from each abnormal log sample, and performing scalarization and normalization processing on each feature to obtain a feature vector corresponding to each abnormal log sample; obtaining the first set of log samples according to the feature vectors of the abnormal log samples of the known problem categories, and obtaining the second set of log samples according to the feature vectors of the abnormal log samples of the unknown problem categories.

[0043] The step of performing scalarization and normalization processing on each feature specifically includes: converting all the features of the defect keywords into scalars, such as defect keyword feature numbering. Each number corresponds to a scalar, thereby scalarizing the features to obtain a scalar data set A. The obtained scalar data set A is normalized to [0, 1] using the min-max method, which can be normalized according to the following formula:

[0044]

[0045] Where x' is the scalar value of x after normalization processing. The features of each abnormal log sample after scalarization and normalization processing constitute the feature vector of the abnormal log sample.

[0046] The problem classification model can be a clustering model, specifically a k-means clustering algorithm model, or other clustering algorithm models.

[0047] The step S101 can specifically include: determining a plurality of initial clustering centers according to the first log sample set, calculating the Euclidean distance of each log sample in the first log sample set and the second log sample set to each initial clustering center to cluster each log sample and redetermine the clustering center of each cluster; continuously iterating the process of calculating the Euclidean distance, clustering each log sample and redetermining the clustering center until the difference between the sum of the Euclidean distances obtained in the k+1 iteration and the sum of the Euclidean distances obtained in the k iteration is less than the set accuracy, and stopping the iteration to complete the first training, wherein the sum of the Euclidean distances is the sum of the Euclidean distances of each log sample to each clustering center after the iteration.

[0048] The step S102: taking the logs of a preset number of problem scenarios as a test set, using the test set to evaluate the consistency of the first trained problem classification model, and according to the consistency evaluation result, performing a second training on the first trained problem classification model, repeating the process of consistency evaluation and second training until the final consistency evaluation result meets the preset condition, and no longer continuing the second training to obtain the trained problem classification model.

[0049] The step of using the test set to evaluate the consistency of the first trained problem classification model, and according to the consistency evaluation result, performing a second training on the first trained problem classification model, can specifically include: using the first trained problem classification model to cluster the logs in the test set to obtain a problem clustering result; using the obtained problem clustering result and the expected problem classification result of the logs in the test set to generate a cross-classification table; calculating a consistency coefficient according to the cross-classification table, the consistency coefficient reflecting the consistency of the problem clustering result and the expected problem classification result; if the consistency coefficient does not meet the preset condition, adding the logs in the test set whose problem clustering result and expected problem classification result are inconsistent to the training set to perform a second training on the first trained problem classification model.

[0050] The preset condition met by the final consistency evaluation result, i.e., the preset condition met by the consistency coefficient, is specifically: the consistency coefficient after each training meets the probability of making the problem clustering result and the expected problem classification result completely consistent reaching a preset value, for example, 95%, i.e., being in the 95% confidence interval of kappa=1, and kappa is the consistency coefficient.

[0051] The second training has the same steps as the first training, except that the training set of the second training also adds part of the test set based on the training set of the first training. The specific steps of the second training can be referred to the introduction of the steps of the first training, which will not be described here.

[0052] The step S103: inputting the log data of the problem to be positioned into the trained problem classification model to determine the problem category in the log data.

[0053] The embodiment of the present application can also include synchronizing the located problems and problem categories to the current test personnel for timely processing.

[0054] The problems in a period of time can also be analyzed and summarized, and the problem distribution in a period of time can be visually and specifically displayed by generating a graph, so that the test personnel and the R&D personnel can jointly grow through analysis and summary.

[0055] The related logs of the located problems and the corresponding problem categories can also be summarized and concluded, and a problem knowledge base can be established to facilitate the test personnel to quickly and specifically understand the bug.

[0056] The located problems and their category information and complete log information in a month are stored in a distributed manner to provide data support for the log report generation and the problem knowledge base.

[0057] Figure 2 is a problem classification model training process diagram according to an embodiment of the present application.

[0058] Since the causes of the bugs in the test environment are various and cannot be enumerated, an embodiment of the present application uses k-means clustering to classify the bug categories, and a problem classification model is constructed based on the learning of a small amount of known classified bug logs for semi-supervised learning, which overcomes the defects of high cost and the need for a large amount of historical data in supervised learning, and overcomes the defects of no supervision and unsatisfactory learning effect in unsupervised learning, which easily causes defect misclassification.

[0059] As shown in Figure 2 , the problem classification model training process includes steps S201 to S206.

[0060] Step S201: Obtain a first log sample set L with labels and a second log sample set U without labels.

[0061] The first log sample set L = [X1, Y1], and the second log sample set U = [X2]. Wherein, X = X1 + X2, X represents a set of feature vectors of abnormal log samples, X1 is a set of feature vectors of abnormal log samples of known problem categories, X2 is a set of feature vectors of abnormal log samples of unknown problem categories, and Y1 is a set of labels (i.e. problem categories) of each feature vector in X1.

[0062] Step S202: Configure the number of clusters.

[0063] The number of clusters is l = len(unique(Y1)), where len(unique(Y1)) represents the number of distinct values ​​of Y1. Step S202 can be performed before step S201.

[0064] Step S203: Determine multiple initial cluster centers and obtain an initial cluster set.

[0065] In the k-means algorithm, different initial cluster centers lead to different final clustering results, and the clustering accuracy is significantly affected by the initial cluster centers. In unsupervised learning k-means algorithms, the initial cluster centers are randomly selected, which can easily lead to defective misclassifications. This embodiment of the invention determines the initial cluster centers based on X1. The initial cluster centers are as follows:

[0066]

[0067] In this formula, the numerator represents the sum of the feature vectors of all abnormal log samples in the category, and the denominator represents the number of abnormal log samples in the category. Where x i Let y represent the i-th value in X1. i Let L(x) represent the i-th value in Y1, i.e., the i-th category. j )=y i This indicates an abnormal log sample that falls into this category.

[0068] Initial y i All x with the same value i The values ​​are grouped into one category, resulting in an initial cluster set Φ = [C1, C2, ..., C...]. l C1,C2,...,C l This represents l initial clusters.

[0069] Step S204: Calculate the Euclidean distance from each point in the set X of feature vectors of the abnormal log samples to each cluster center.

[0070] Each point in X corresponds to a point x in X. i value.

[0071] Step S205: Adjust the cluster centers of each cluster according to the Euclidean distance and then cluster them.

[0072] The process of calculating Euclidean distance, clustering each log sample, and redetermining cluster centers is iterated continuously, wherein, according to D... k The minimum principle adjusts the cluster centers of each cluster, D. k This represents the sum of the Euclidean distances from each log sample to each cluster center after the k-th iteration.

[0073] For each iteration, the sum of the Euclidean distance of each log sample to each cluster center D is as follows:

[0074]

[0075] Step S206: Determine whether the difference between the sum of the Euclidean distance obtained in the k+1 iteration and the sum of the Euclidean distance obtained in the k iteration is less than the set accuracy delta, if not, return to step S204, if yes, stop iteration, that is, Φ k+1 k k+1 k Φk+1 and Φk respectively represent the cluster set obtained in the k+1 iteration and the k iteration.

[0076] Figure 3 is a consistency evaluation process schematic diagram of a problem classification model according to an embodiment of the application.

[0077] The embodiment of the application continuously optimizes the problem classification model through consistency evaluation to improve the clustering accuracy, thereby improving the accuracy of bug positioning. Figure 3 As shown in the figure, the consistency evaluation process includes steps S301 to S304.

[0078] Step S301: Collect n problem scenarios logs in a local environment as a test set, and manually set the problem classification expected results of each log in the test set.

[0079] The problem classification expected result is the expected bug classification.

[0080] After the test set is collected, the test set can also be preprocessed, such as cleaning, and the preprocessed test set is used for the operation of step S302.

[0081] Step S302: Cluster the logs in the test set using the first trained problem classification model to obtain the problem clustering result.

[0082] The first trained problem classification model is a problem classification model trained by using the first log sample set L with labels and the second log sample set U without labels as a training set, through the training process shown in the figure. Figure 2 The test set is input into the first trained problem classification model, and the bug classification result of the test set, that is, the problem clustering result, is output.

[0083] Step S303: Form a cross-classification table by combining the problem clustering result obtained with the problem classification expected result of the logs in the test set, and calculate the consistency coefficient kappa.

[0084] The consistency coefficient, that is, the kappa coefficient, is calculated according to the following formula: ​​​

[0085]

[0086] wherein N is the total number of cells in the cross classification table, A ii is the element in the diagonal position of the cross classification table, A i. is the sum of the elements in the i-th row, A .j is the sum of the elements in the j-th column.

[0087] Generally, the closer the value of the kappa coefficient is to 1, the better the consistency between the problem clustering result and the problem classification expected result is. The form of the cross classification table is shown in Table 1, wherein A ij (1≤i≤5, 1≤j≤5) represents the frequency of various cases, for example, A 11 represents the frequency of the problems whose prediction and expectation are both null pointers, wherein the prediction refers to the problem clustering result, and the expectation refers to the problem classification expected result.

[0088] Table 1

[0089]

[0090] Step S304: determining whether the consistency coefficient meets a preset condition, if not, adding the logs in the n problem scenes that are inconsistent with the problem classification expected result into the training set, and continuing the second training until the second training is terminated when the preset condition is met.

[0091] Determining whether the consistency coefficient meets a preset condition, i.e., determining whether the current is in the 95% confidence interval of kappa = 1, if not, adding the logs (specifically in the form of feature vectors) in the n problem scenes that are inconsistent with the problem classification expected result into the training set, and continuing the training (i.e., the second training) according to the training process of Figure 2 . The second training is terminated until the determination result is in the 95% confidence interval of kappa = 1. The determination result being in the 95% confidence interval of kappa = 1 means that the probability of the problem clustering result being completely consistent with the problem classification expected result reaches 95%, i.e., 95 times out of 100 times of training reach kappa = 1.

[0092] The embodiment of the present application effectively improves the accuracy of the k-means clustering result by semi-supervised learning and consistency evaluation, realizes accurate classification of bugs, and is not only suitable for automatic positioning of bugs in a test environment, but also can be popularized to a generation environment to provide help for positioning of online problems for operation and maintenance personnel. The embodiment of the present application has low requirements for test resources, small calculation cost, and strong self-adaptation ability.

[0093] Figure 4 is the main module schematic diagram of the problem positioning device according to an embodiment of the present application.

[0094] like Figure 4 As shown, a problem localization device 400 according to an embodiment of the present invention mainly includes: a first training module 401, a consistency evaluation module 402, a second training module 403, and a problem category prediction module 404.

[0095] The first training module 401 is used to obtain a training set based on a labeled first log sample set and an unlabeled second log sample set, and to perform the first training on the problem classification model. The labels are used to indicate the problem category.

[0096] The problem location device 400 may also include a system configuration module for configuring the systems whose logs need to be monitored. Specifically, this may include configuring information such as the IP address of the system currently requiring log monitoring, and the log rollback period. For a detailed description of the functions of this system configuration module 502, please refer to the following section.

[0097] The problem location device 400 may also include a log collection module for collecting logs from the system whose logs are to be monitored. For example, a syslog (system log) service can be set up using a Linux server to collect logs. See the detailed description of the log collection module 503 below for its functionality.

[0098] The problem location device 400 may also include a log cleaning module for filtering the collected logs to obtain a valid log sample set containing multiple valid log samples. For details on the function of this log cleaning module, please refer to the detailed description of the log cleaning module 504 below.

[0099] The problem localization device 400 may further include a training set generation module for obtaining a labeled first log sample set and an unlabeled second log sample set, wherein: multiple abnormal log samples containing defect keywords are extracted from the valid log sample set, and the abnormal log samples are valid log samples with problems; multiple features of defect keywords are extracted from each abnormal log sample, and each feature is scalarized and normalized to obtain a feature vector corresponding to each abnormal log sample; the first log sample set is obtained based on the feature vectors of abnormal log samples with known problem categories, and the second log sample set is obtained based on the feature vectors of abnormal log samples with unknown problem categories.

[0100] The problem classification model can be a clustering model, specifically a k-means clustering algorithm model, or other clustering algorithm models.

[0101] The first training module 401 can be specifically configured to determine a plurality of initial clustering centers according to the first log sample set, calculate the Euclidean distances of each log sample in the first log sample set and the second log sample set to each initial clustering center, cluster each log sample, and redetermine the clustering center of each cluster; iteratively perform the above processes of calculating the Euclidean distances, clustering each log sample, and redetermining the clustering center until the difference between the sum of the Euclidean distances obtained in the k+1 iteration and the sum of the Euclidean distances obtained in the k iteration is less than the set accuracy, and stop the iteration to complete the first training. The sum of the Euclidean distances is the sum of the Euclidean distances of each log sample to each clustering center after the iteration.

[0102] The consistency evaluation module 402 is configured to use logs of a preset number of problem scenarios as a test set to evaluate the consistency of the problem classification model after the first training.

[0103] The consistency evaluation module 402 can be specifically configured to cluster the logs in the test set by using the problem classification model after the first training to obtain a problem clustering result, generate a cross-classification table by using the problem clustering result and the expected problem classification result of the logs in the test set, calculate a consistency coefficient according to the cross-classification table, and reflect the consistency between the problem clustering result and the expected problem classification result. If the consistency coefficient does not meet the preset condition, the logs in the test set that are inconsistent between the problem clustering result and the expected problem classification result are added to the training set to perform the second training on the problem classification model after the first training by the second training module.

[0104] The second training module 403 is configured to perform the second training on the problem classification model after the first training according to the consistency evaluation result.

[0105] The consistency evaluation module 402 and the second training module 403 repeatedly perform the processes of consistency evaluation and second training until the final consistency evaluation result meets the preset condition, and the second training is stopped to obtain the trained problem classification model.

[0106] The problem category prediction module 404 is configured to input the log data of a problem to be located into the trained problem classification model to determine the problem category in the log data.

[0107] Figure 5 It is a schematic diagram of the overall framework of the problem positioning device according to an embodiment of the present application

[0108] As Figure 5As shown, the overall framework of the problem positioning device of one embodiment of the present application includes: a UI (user interface) interaction module 501, a system configuration module 502, a log collection module 503, a log cleaning module 504, a bug identification module 505, an intelligent analysis module 506, and a distributed storage module 507.

[0109] The UI interaction module 501 mainly involves abnormal monitoring and alarm, report generation, bug knowledge base, etc. The abnormal monitoring and alarm function is mainly to synchronize the captured bug and its classification type to the current tester for timely processing. The report generation function mainly provides the user with a bug summary analysis function within a period of time, and displays the bug distribution within a period of time in a specific manner through the generation of a graph. Through analysis and summary, the tester and the developer can grow together. The bug knowledge base is an induction summary of the bug-related logs and their corresponding classification, which facilitates the tester to quickly and specifically understand the bug.

[0110] The system configuration module 502 mainly includes user management, log source system related configuration, parameter setting, etc. The tester can set the system IP information of the current log to be monitored in this module, and can also set the log rollback period of the system. In general, the positioning of the problem requires real-time, and the summary of the problem is asynchronous, so it is not necessary to completely store a large amount of log data. Therefore, in the embodiment of the present application, only the complete log information of the last month is retained, and all specific bug-related information is stored.

[0111] The log collection module 503 can use a Linux server to build a syslog service to collect logs, and provide data sources for subsequent intelligent analysis of logs. It also supports uploading of log files. In the training process of the problem classification model, a large number of abnormal log samples of known problem categories are needed. A large number of historical records and expected results obtained by analysis can be uploaded as data sources of the training set.

[0112] The log cleaning module 504 cleans the fixed output system logs through a filter. Since the logs obtained by the log collection module 503 have many repeated system logs, and sometimes the proportion of the logs in the logs is much larger than that of the effective logs, the fixed output system logs are cleaned through log cleaning, so that the effective logs are obtained, which can reduce the data base participating in intelligent analysis and improve the processing efficiency. These effective logs can be used as effective log samples to constitute an effective log sample set.

[0113] The bug identification module 505 can be used as a training set generation module (the training set generation module has been described in detail above). The bug identification module 505 is mainly used to identify and extract logs with bugs, and the extracted logs with bugs can be used as abnormal log samples. The identification of log bugs in a test environment is strongly dependent on the output control of the logs by the developers during coding, and therefore needs to be agreed with the developers to perform complete log printing as much as possible, so as to ensure that the problem can be diagnosed through log analysis when the problem occurs. The classification of test environment bugs is as shown in Figure 6 Figure 6 Only several common classifications are exemplarily shown. Each type of bug can be identified according to the corresponding defect keyword, for example, java.lang.NullPointerException is a null pointer problem, and No alive provider is a connection problem of upstream and downstream interface calls.

[0114] The embodiment of the application mainly uses the defect keyword extraction method to extract and analyze the defect keywords in the effective log sample set to obtain a plurality of abnormal log samples. Then, all bug information of the abnormal log samples is preprocessed to change the features of all defect keywords into scalars.

[0115] In order to facilitate the calculation of the clustering algorithm later, the min-max method is used to normalize the obtained scalar data set A to [0, 1], and the formula is as follows:

[0116]

[0117] Where x' is the scalar value of x after normalization processing, and the normalized data set can be used as a direct data source of the intelligent analysis module 506. Each feature of the abnormal log sample after scalarization and normalization processing constitutes a feature vector, and the feature vector of the abnormal log sample according to the known problem category can obtain a first log sample set with labels, and the feature vector of the abnormal log sample according to the unknown problem category can obtain a second log sample set without labels to obtain a training set.

[0118] ​The intelligent analysis module 506 mainly uses semi-supervised k-means clustering to classify bugs, first learns a small amount of log of known bug categories to build a problem classification model, and then continuously optimizes the problem classification model through consistency evaluation. The effective output of the log is a key factor to measure the maintainability of a system, whether in a production environment or in a test environment, the log is crucial for system monitoring and problem analysis and positioning, and the embodiment of the application mainly positions and classifies bugs based on intelligent analysis of the log. The intelligent analysis module 506 is specifically subdivided into a first training module, a consistency evaluation module, a second training module, and a problem category prediction module. The functions of the four modules are the same as the first training module 401, the consistency evaluation module 402, the second training module 403, and the problem category prediction module 404 described above, and will not be repeated here.

[0119] The distributed storage module 507 adopts a distributed storage mode to save the collected log information. Since the data volume of the log information is large, the content to be stored mainly includes the located bug and its category information and the complete log information within a month, which can provide data support for log report generation and bug knowledge base.

[0120] In addition, the specific implementation content of the problem positioning device in the embodiment of the application has been described in detail in the problem positioning method described above, and therefore the repeated content will not be described here.

[0121] Figure 7 An exemplary system architecture 700 to which the problem positioning method or problem positioning device of the embodiment of the application can be applied is shown.

[0122] As shown in Figure 7 The system architecture 700 can include terminal devices 701, 702, 703, a network 704, and a server 705. The network 704 is used to provide a communication link medium between the terminal devices 701, 702, 703 and the server 705. The network 704 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0123] A user can use the terminal devices 701, 702, 703 to interact with the server 705 through the network 704 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 701, 702, 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0124] The terminal devices 701, 702, and 703 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0125] The server 705 can be a server providing various services, such as a background management server supporting a shopping website browsed by a user using the terminal device 701, 702, or 703 (only as an example). The background management server can analyze and process received product information query requests and the like, and feed back the processing results (such as target push information, product information - only as an example) to the terminal device.

[0126] It should be noted that the problem positioning method provided by the embodiments of the present application is generally executed by the server 705, and accordingly, the problem positioning apparatus is generally arranged in the server 705.

[0127] It should be understood that, Figure 7 The number of terminal devices, networks, and servers in the system is only illustrative. Any number of terminal devices, networks, and servers can be provided according to implementation needs.

[0128] Reference is made below to Figure 8 which shows a structural schematic diagram of a computer system 800 suitable for implementing the terminal device or server of the embodiments of the present application. Figure 8 The terminal device or server shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0129] As shown in Figure 8 , the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 802 or programs loaded from a storage portion 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the system 800 are also stored in the RAM 803. The CPU 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0130] The following components are connected to the I / O interface 805: an input part 806 including a keyboard, a mouse, etc.; an output part 807 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 808 including a hard disk, etc.; and a communication part 809 including a network interface card such as a LAN card, a modem, etc. The communication part 809 performs communication processing via a network such as the Internet. The drive 610 is also connected to the I / O interface 605 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 610 as necessary, so that a computer program read out therefrom is installed in the storage part 608 as necessary.

[0131] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with the embodiments disclosed herein. For example, the embodiments disclosed herein include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 809 and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-described functions defined in the system of the present application are executed.

[0132] It should be noted that the computer-readable medium shown in the application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or instrument, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, electrical connections with one or more conductive wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or instrument. In this application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, device or instrument. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0133] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0134] The modules described in the embodiments of the present application can be implemented in the form of software or hardware. The described modules can also be arranged in a processor, for example, a processor can include a first training module, a consistency evaluation module, a second training module, and a problem category prediction module. In some cases, the names of these modules do not constitute a limitation on the modules themselves, for example, the first training module can also be described as "a module for performing first training on a problem classification model according to a first set of labeled log samples and a second set of unlabeled log samples".

[0135] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, which, when executed by the device, cause the device to include: obtaining a training set according to a first set of labeled log samples and a second set of unlabeled log samples, performing first training on a problem classification model, the labels being used to indicate problem categories; taking logs of a preset number of problem scenarios as a test set, performing consistency evaluation on the problem classification model after the first training using the test set, and performing second training on the problem classification model after the first training according to the consistency evaluation result, repeating the consistency evaluation and the second training until the final consistency evaluation result meets a preset condition, and obtaining a trained problem classification model; inputting log data of a problem to be located into the trained problem classification model to determine the problem category in the log data.

[0136] According to the technical scheme of the embodiments of the present application, the problem classification model is first trained, logs of a preset number of problem scenarios are taken as a test set, consistency evaluation is performed on the problem classification model after the first training, second training is performed on the problem classification model after the first training according to the consistency evaluation result, the consistency evaluation and the second training are repeated until the final consistency evaluation result meets a preset condition, and a trained problem classification model is obtained to determine the problem category in log data of a problem to be located. The automatic and accurate positioning and classification of bugs can be realized, the test personnel can be helped to avoid some low-level problems, the independence of the test personnel can be enhanced, the cost of repairing problems for the developer can be saved, the requirement for test resources is not high, the calculation cost is small, and the self-adaptation ability is strong.

[0137] The foregoing detailed description has set forth various embodiments of the devices and / or processes via the use of specific terminology. However, embodiments thereof can be practiced with the exact description not being presented in detail. The term "device" should be understood to encompass devices operating in various modes, such as active mode, sleep mode, hibernate mode, and the like. The terms "coupled" and "connected," along with their derivatives, can be used. It should be understood that these terms are not intended as synonyms for each other. Rather, particular circuitry that can be said to be coupled or connected can be coupled and connected via some transmission medium.

Claims

1. A method for locating background problems in a test environment, the method comprising: The application comprises the following steps: According to the labeled first log sample set and the unlabeled second log sample set, a training set is obtained, and a problem classification model is first trained, wherein the label is used to indicate a problem category; Logs of a preset number of problem scenarios are used as a test set, the first trained problem classification model is evaluated for consistency by using the test set, and the first trained problem classification model is second trained according to a consistency evaluation result, the consistency evaluation and the second training are repeated until the final consistency evaluation result meets a preset condition, the second training is stopped, and a trained problem classification model is obtained; Log data of a problem to be positioned is input into the trained problem classification model to determine a problem category in the log data; Problems in a period of time are summarized and analyzed, and a problem distribution in the period of time is displayed in a graphical manner; the log of the positioned problem and the corresponding problem category are summarized, and the positioned problem, the category information and complete log information in a month are stored in a distributed manner to establish a problem knowledge base; The problem classification model is a clustering model, and the first training of the problem classification model according to the labeled first log sample set and the unlabeled second log sample set comprises the following steps: a plurality of initial clustering centers are determined according to the first log sample set, the Euclidean distances of each log sample in the first log sample set and the second log sample set to each initial clustering center are calculated to cluster the log samples, and the clustering centers of each cluster are re-determined; the calculation of the Euclidean distances, the clustering of the log samples and the re-determination of the clustering centers are iterated until the difference between the sum of the Euclidean distances obtained in the k+1 iteration and the sum of the Euclidean distances obtained in the k iteration is less than a set accuracy, the iteration is stopped, and the first training is completed, wherein the sum of the Euclidean distances is the sum of the Euclidean distances of the log samples to the clustering centers after the iteration.

2. The method of claim 1, wherein, The first trained problem classification model is evaluated for consistency by using the test set, and the first trained problem classification model is second trained according to a consistency evaluation result, and the steps comprise: The first trained problem classification model is used to cluster the logs in the test set to obtain a problem clustering result; The problem clustering result and the expected problem classification result of the logs in the test set are used to generate a cross-classification table; A consistency coefficient is calculated according to the cross-classification table, and the consistency coefficient reflects the consistency between the problem clustering result and the expected problem classification result; If the consistency coefficient does not meet the preset condition, the logs in the test set, for which the problem clustering result and the expected problem classification result are inconsistent, are added to the training set, and the first trained problem classification model is second trained.

3. The method of claim 1, wherein, The labeled first log sample set and the unlabeled second log sample set are obtained by the following method: extracting a plurality of abnormal log samples with defect keywords from the effective log sample set, the abnormal log samples being the effective log samples with problems; extracting a plurality of features of the defect keywords for each abnormal log sample, and performing scalarization and normalization processing on each feature to obtain a feature vector corresponding to each abnormal log sample; obtaining the first log sample set according to the feature vectors of the abnormal log samples with known problem categories, and obtaining the second log sample set according to the feature vectors of the abnormal log samples with unknown problem categories.

4. The method of claim 3, wherein, Further comprising: configuring a system requiring monitored logs to collect logs from the system; filtering the collected logs to obtain the effective log sample set including a plurality of effective log samples.

5. A test environment background problem positioning apparatus characterized by comprising: Comprising: a first training module, a consistency evaluation module, a second training module, and a problem category prediction module, wherein: the first training module is configured to obtain a training set according to the first log sample set with labels and the second log sample set without labels, perform first training on a problem classification model, and the labels are used to indicate problem categories; the consistency evaluation module is configured to use logs of a preset number of problem scenarios as a test set to perform consistency evaluation on the first trained problem classification model; the second training module is configured to perform second training on the first trained problem classification model according to the consistency evaluation result; the process of the consistency evaluation and the second training is repeated through the consistency evaluation module and the second training module until the final consistency evaluation result meets a preset condition, the second training is stopped, and a trained problem classification model is obtained; the problem category prediction module is configured to input log data of a problem to be located into the trained problem classification model to determine a problem category in the log data; the device is further configured to: perform problem aggregation analysis within a period of time, display problem distribution within the period of time by generating a graph; and summarize the located problem and the corresponding problem category, store the located problem, the category information, and complete log information within a month by a distributed storage manner to establish a problem knowledge base; the problem classification model is a clustering model, and the first training module is further configured to: determine a plurality of initial clustering centers according to the first log sample set, calculate Euclidean distances of each log sample in the first log sample set and the second log sample set to each initial clustering center to cluster the log samples, and redetermine a clustering center of each cluster; continuously iterate the processes of calculating the Euclidean distances, clustering the log samples, and redetermining the clustering centers until a difference between a sum of Euclidean distances obtained in a k+1 iteration and a sum of Euclidean distances obtained in a k iteration is less than a set precision, stop the iteration, and complete the first training, the sum of Euclidean distances being a sum of Euclidean distances of the log samples to the clustering centers after the iteration.

6. The apparatus of claim 5, wherein, the consistency evaluation module is further configured to: Clustering the logs in the test set by using the first trained question classification model to obtain a question clustering result; Generating a cross-classification table by using the question clustering result and the question classification expected result of the logs in the test set; Calculating a consistency coefficient according to the cross-classification table, the consistency coefficient reflecting consistency between the question clustering result and the question classification expected result; If the consistency coefficient does not satisfy the preset condition, adding the logs in the test set, for which the question clustering result is inconsistent with the question classification expected result, to the training set to perform second training on the first trained question classification model by the second training module.

7. An electronic device, comprising: comprising: one or more processors; memory for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method according to any one of claims 1-4.

8. A computer readable medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method according to any one of claims 1-4.

Images