Service quality detection model training data set acquisition method and device

Abstract

The application provides an acquisition method and device of a training data set of a service quality detection model, and the method comprises the following steps: evaluating the hour granularity index data of a first index set based on a non-real-time index QoE model to obtain a target hour time point; evaluating the minute granularity index data of a second index set in the target hour time point based on the non-real-time index QoE model to obtain a target minute time point; acquiring the index data of a third index set of the target minute time point, and labeling the index data of the third index set to obtain a labeled target data set. The non-real-time service QoE model is used to screen the quality difference and quality optimization minute time points of a real-time quality detection model, and the improvement from unsupervised to supervised in the construction of the real-time quality detection model is realized, so that the accuracy of the real-time quality detection model can be significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a method and device for obtaining training data set of a service quality defect detection model, an electronic device and a storage medium. BACKGROUND

[0002] To improve user satisfaction with mobile Internet service, it is necessary to obtain real-time perception of the service by the user, and to detect real-time service quality defects, so as to quickly feedback and intervene in the problem, eliminate the problem and provide protection for high-quality service.

[0003] The existing quality defect detection method has problems of high cost, great operation difficulty and low quality defect detection accuracy. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0005] To this end, a first object of the present application is to provide a method for obtaining training data set of a service quality defect detection model, so as to obtain a labeled data set for training a real-time service quality defect detection model, and solve the problem that in the related art, mobile Internet service real-time KPI (Key Performance Indicator) / KQI (Key Quality Indicator) index data is difficult to obtain reliable and reliable quality defect labels.

[0006] A second object of the present application is to provide a device for obtaining training data set of a service quality defect detection model.

[0007] A third object of the present application is to provide an electronic device.

[0008] A fourth object of the present application is to provide a computer-readable storage medium.

[0009] A fifth object of the present application is to provide a computer program product.

[0010] To achieve the above objects, a first aspect of the present application provides a method for obtaining training data set of a service quality defect detection model, comprising:

[0011] evaluating the hour granularity index data of the first index set based on a non-real-time index QoE model to obtain a target hour time point;

[0012] evaluating the minute granularity index data of the second index set in the target hour time point based on the non-real-time index QoE model to obtain a target minute time point;

[0013] obtain the target minute time point by evaluating the minute granularity index data of the second index set based on the non-real-time index QoE model.

[0014] In some implementations, the non-real-time index QoE model is used to evaluate the hour granularity index data of the first index set to obtain a target hour time point, including:

[0015] obtain the hour granularity index data of the first index set and the index historical value sequence of the corresponding hour;

[0016] generate a non-real-time index QoE model of the corresponding hour based on the index historical value sequence;

[0017] evaluate the hour granularity index data of the first index set by using the non-real-time index QoE model to obtain an evaluation result of the hour granularity index data;

[0018] obtain the target hour time point based on the evaluation result of the hour granularity index data and a first preset service threshold.

[0019] In some implementations, the non-real-time index QoE model is used to evaluate the minute granularity index data of the second index set in the target hour time point to obtain a target minute time point, including:

[0020] obtain the minute granularity index data of the second index set in the target hour time point;

[0021] evaluate the minute granularity index data of the second index set based on the dynamic parameter of the non-real-time index QoE model to obtain an evaluation result of the minute granularity index data of the second index set;

[0022] obtain the target minute time point based on the evaluation result of the minute granularity index data of the second index set and a second preset service threshold.

[0023] In some implementations, after obtaining the target minute time point, the method further includes:

[0024] obtain the dynamic parameter of the historical non-real-time index QoE model of the target minute time point;

[0025] evaluate the minute granularity index data of the second index set of the target minute time point based on the dynamic parameter of the historical non-real-time index QoE model to obtain an evaluation result of the minute granularity index data of the second index set of the target minute time point;

[0026] Filter the target minute time point based on the evaluation result of the minute granularity index data of the second index set of the target minute time point and the second preset service threshold, to obtain an updated target minute time point.

[0027] In some implementations, after the target minute time point is obtained, the method further includes:

[0028] Obtaining index data of a fourth index set of the target minute time point;

[0029] Detecting the index data of the fourth index set by an unsupervised anomaly detection model to obtain an anomaly detection result;

[0030] Filtering the target minute time point based on the anomaly detection result, to obtain an updated target minute time point.

[0031] In some implementations, the training of the service quality defect detection model by the labeled target data set includes:

[0032] Supervised training of a real-time service quality defect detection model based on the target data set, to obtain a minute granularity service quality defect detection model.

[0033] In some implementations, the target hour time point includes a quality good hour time point and a quality defect hour time point, and the label includes a quality good label or a quality defect label.

[0034] To achieve the above purpose, a second aspect embodiment of the present application provides an acquisition device of a training data set of a service quality defect detection model, including:

[0035] A first evaluation module is configured to evaluate the hour granularity index data of the first index set based on a non-real-time index quality of experience (QoE) model, to obtain a target hour time point.

[0036] A second evaluation module is configured to evaluate the minute granularity index data of a second index set in the target hour time point based on the non-real-time index QoE model, to obtain a target minute time point.

[0037] A label processing module is configured to obtain index data of a third index set of the target minute time point, and assign a label to the index data of the third index set, to obtain a labeled target data set, and train a service quality defect detection model by the labeled target data set.

[0038] In some implementations, the first evaluation module is specifically configured to:

[0039] Obtain the hour granularity index data of the first index set and the corresponding hour index historical value sequence.

[0040] generate a non-real-time index QoE model corresponding to the hour based on the sequence of historical values of the index;

[0041] evaluate the hour-granularity index data of the first index set through the non-real-time index QoE model to obtain an evaluation result of the hour-granularity index data;

[0042] obtain a target hour time point based on the evaluation result of the hour-granularity index data and a first preset service threshold.

[0043] In some implementations, the second evaluation module is specifically configured to:

[0044] obtain minute-granularity index data of a second index set in the target hour time point;

[0045] evaluate the minute-granularity index data of the second index set based on the dynamic parameter of the non-real-time index QoE model to obtain an evaluation result of the minute-granularity index data of the second index set;

[0046] obtain a target minute time point based on the evaluation result of the minute-granularity index data of the second index set and a second preset service threshold.

[0047] In some implementations, the second evaluation module is further configured to:

[0048] obtain a dynamic parameter of a historical non-real-time index QoE model of the target minute time point;

[0049] evaluate the minute-granularity index data of the second index set of the target minute time point based on the dynamic parameter of the historical non-real-time index QoE model to obtain an evaluation result of the minute-granularity index data of the second index set of the target minute time point;

[0050] filter the target minute time point based on the evaluation result of the minute-granularity index data of the second index set of the target minute time point and the second preset service threshold to obtain an updated target minute time point.

[0051] In some implementations, the second evaluation module is further configured to:

[0052] obtain index data of a fourth index set of the target minute time point;

[0053] detect the index data of the fourth index set through an unsupervised anomaly detection model to obtain an anomaly detection result;

[0054] filter the target minute time point based on the anomaly detection result to obtain an updated target minute time point.

[0055] In some implementations, the apparatus further includes a model training module configured to:

[0056] perform supervised training on the real-time service quality detection model based on the target data set, to obtain a minute-granularity service quality detection model.

[0057] In some implementations, the target hourly time point includes a high-quality hourly time point and a low-quality hourly time point, and the label includes a high-quality label or a low-quality label.

[0058] To achieve the above object, a third aspect of the present application provides an electronic device, comprising: a processor, and a memory connected to the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method of the first aspect.

[0059] To achieve the above object, a fourth aspect of the present application provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the method of the first aspect.

[0060] To achieve the above object, a fifth aspect of the present application provides a computer program product, comprising a computer program, which is executed by a processor to implement the method of the first aspect.

[0061] The service quality detection model provided by the present application uses the method for acquiring a training data set, an apparatus, an electronic device and a storage medium. The non-real-time service QoE model is used to select low-quality and high-quality minute time points for a real-time quality detection model, and to construct an index data set with low-quality and high-quality labels. The improvement from unsupervised to supervised is realized in the construction of the real-time quality detection model, which can significantly improve the accuracy of the real-time quality detection model.

[0062] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0063] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings.

[0064] Figure 1 A flowchart of a method for acquiring a training data set for a service quality detection model provided by the first embodiment of the present application;

[0065] Figure 2 A flowchart of a method for acquiring a training data set for a service quality detection model provided by the second embodiment of the present application;

[0066] Figure 3 A flowchart of a method for obtaining a training data set for a service quality defect detection model according to an embodiment of the present application is shown in FIG. 3.

[0067] Figure 4 A flowchart of a method for obtaining a training data set for a service quality defect detection model according to an embodiment of the present application is shown in FIG. 3.

[0068] Figure 5 A flowchart of a method for obtaining a training data set for a service quality defect detection model according to an embodiment of the present application is shown in FIG. 3.

[0069] Figure 6 A block diagram of an apparatus for obtaining a training data set for a service quality defect detection model according to an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0070] Embodiments of the present application are described in detail below with reference to examples shown in the attached drawings, wherein the same or similar reference numerals designate the same or similar elements throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0071] In the related art, a service quality defect detection is generally implemented by using a dialing test technology. The cost of using this technology is very high. To fully master the quality defect status, a large number of mobile test terminals need to be deployed, and different service dialing test data needs to be collected and uploaded continuously. Since there is a lack of a high-quality real-time QoE model based on real-time indexes (such as minute granularity indexes) in mobile Internet services, it is difficult to optimize the QoE evaluation system directly using real-time indexes to construct a QoE evaluation system. Since the target period to be evaluated is too short, it is not convenient to form a feedback loop with real user experience or expert verification. Therefore, it is difficult to optimize. There is a large error when a QoE evaluation system based on non-real-time indexes (such as hour granularity or day granularity index data) is directly applied to real-time index (such as minute granularity) QoE evaluation. Directly using an unsupervised anomaly detection method can only detect relatively extreme anomalies, and the results lack reliability and stability.

[0072] To solve the problem that it is difficult to obtain a reliable and reliable quality defect label in the related art, embodiments of the present application provide a method and apparatus for obtaining a training data set for a service quality defect detection model to realize real-time service quality defect detection model training data set acquisition.

[0073] The method and apparatus for obtaining a training data set for a service quality defect detection model of the embodiments of the present application are described below with reference to the accompanying drawings.

[0074] Figure 1This is a flowchart illustrating a method for obtaining a training dataset for a poor business quality detection model provided in an embodiment of this application.

[0075] It should be noted that the execution subject of the method for obtaining the training dataset for the poor service quality detection model in this application embodiment is the device for obtaining the training dataset for the poor service quality detection model in this application embodiment. The device for obtaining the training dataset for the poor service quality detection model can be configured in an electronic device so that the electronic device can perform the function of obtaining the training dataset for the poor service quality detection model.

[0076] like Figure 1 As shown, the method for obtaining the training dataset for this poor business quality detection model includes the following steps:

[0077] Step 101: Evaluate the hourly granularity index data of the first index set based on the non-real-time index QoE (Quality of Experience) model to obtain the target hourly time point.

[0078] As one implementation method, a series of hourly QoE models are generated based on the historical sequence data of the first indicator set. The QoE scores of the hourly granular indicator data of the first indicator set in non-real-time are calculated using the generated series of hourly QoE models to obtain the QoE scores of each hour. Then, the target hourly time point is obtained based on the QoE scores of each hour and the threshold.

[0079] It can be understood that, since the QoE model in this application embodiment is generated through non-real-time hourly granular index data, the generated QoE model is a non-real-time index QoE model.

[0080] It should be noted that the target hour time point includes both high-quality and low-quality hour time points. Obtaining the target hour time point means completing the screening of low-quality and high-quality hour time points.

[0081] It should also be noted that although there are over 100 metrics related to mobile internet services, considering that most metrics do not reflect user experience, the metric set in this application embodiment can be established by focusing on combinations of business metrics that conform to the QoE model. Furthermore, appropriate metric sets should be selected based on specific scenarios. For example, for the common scenario of detecting poor quality in mobile internet services such as gaming, the selected business metrics include general KPI metrics and gaming KQI metrics, i.e., constructing a metric set that includes both general KPI metrics and gaming KQI metrics. For the scenario of detecting poor quality in web services, the selected business metrics include general KPI metrics and web KQI metrics. Specific examples of business metrics are shown in Table 1 below:

[0082]

[0083] The first index set (named A h ) is a multi-KPI / KQI index combination concerned by the QoE model.

[0084] In step 102, the minute granularity index data of the second index set in the target hour time point is evaluated based on the non-real-time index QoE model, and a target minute time point is obtained.

[0085] After the screening of the poor quality and good quality hour time points is completed, the minute time points in the screened hour time points are screened in this step.

[0086] As an implementation manner, the minute granularity index data of the second index set in the target hour time point is obtained; the minute granularity index data of the second index set is evaluated based on the dynamic parameters of the non-real-time index QoE model, and an evaluation result of the minute granularity index data of the second index set is obtained; and the target minute time point is obtained based on the evaluation result of the minute granularity index data of the second index set and a second preset service threshold.

[0087] It should be noted that the second index set (named A m ) is a multi-KPI / KQI index combination concerned by the QoE model. The index items of the second index set are the same as those of the first index set, but the time granularity is minutes.

[0088] In this step, the non-real-time QoE model is used for internal screening in the poor quality and good quality hour time points, and the screening of the poor quality and good quality minute time points is completed.

[0089] In step 103, the index data of the third index set of the target minute time point is obtained, and the index data of the third index set is labeled to obtain a labeled target data set.

[0090] It can be understood that the index data corresponding to the third index set of the screened poor quality and good quality minute time points is obtained, and is labeled as poor quality or good quality to form a labeled data set. The data set can be used to train a real-time service quality defect detection model.

[0091] The third index set (named Cm) is a multi-KPI / KQI index combination concerned by the supervised service quality defect anomaly detection model, the index data is of minute granularity, and the index items can be selected according to the needs of the algorithm.

[0092] As an implementation manner, when the service quality defect detection model is trained, the real-time service quality defect detection model is supervised and trained based on the target data set, and a minute granularity service quality defect detection model is obtained.

[0093] Based on the above labeled data set, a supervised machine learning algorithm such as LightGBM (Light Gradient Boosting Machine), RF (Random Forest) can be used, and the current index value of the real-time third index set is used as the model input, and the quality difference and quality label are used as the target output, and the model is trained.

[0094] Taking the random forest model as an example, first, h training data sets are randomly sampled from the input data index set C m with replacement, and then K features are randomly selected from each training data set (K is less than the number of index items in the index set C m , then Y decision trees are established according to the K features of the random forest, and then each decision tree is used to predict the result, and all predicted results are saved, and finally the classification model is voted, the vote number of each prediction result is calculated, and the model with the highest vote number is selected as the final decision.

[0095] The trained model can be used for real-time quality difference detection, and the input of the model is the index value of the minute KPI / KQI index set C m , and the output is the quality difference detection result.

[0096] The service quality difference detection model training data set acquisition method provided by the embodiment of the application uses a non-real-time service QoE model to select quality difference and quality minute time points for a real-time quality difference detection model, realizes the improvement from unsupervised to supervised in the construction of the real-time quality difference detection model, and can significantly improve the accuracy of the real-time quality difference detection model, so that the rapid detection of service quality difference can be realized, the user satisfaction of online users can be improved, the work efficiency can be improved, the cost can be reduced, the user stickiness can be improved through the improvement of user satisfaction, more business value can be created while providing high-quality services to users. And the method is simple, the relative difficulty is small, and the cost is low.

[0097] The following will be described in detail in the manner of the specific matters of the target hour time point based on the step 101 of the above embodiment, which is the evaluation of the hour granularity index data of the first index set based on the non-real-time index QoE model.

[0098] Figure 2 The flowchart of the service quality difference detection model training data set acquisition method provided by another embodiment of the application is shown. That is, step 101 further includes the following steps:

[0099] Step 201, obtaining the hour granularity index data of the first index set and the corresponding hour index historical value sequence.

[0100] It can be understood that for the hour granularity index data, the index data of each business index can be obtained at each hour point. The index history value sequence of the first index set is obtained, that is, the index history value sequence corresponding to each business index in the first index set is obtained, and each index history value sequence includes a plurality of index data in a period of time, for example, the hour granularity index data of index A corresponding to 1 o'clock every day in a month. For the hour granularity index data, the corresponding index history value sequence and the index data of the current hour of each business index can be obtained at each hour point.

[0101] In step 202, a non-real-time index QoE model corresponding to the hour is generated based on the index history value sequence.

[0102] As an implementation manner, for each hour point, the dynamic threshold and the dynamic weight of each index in the index set are obtained based on the index history value sequence of the hour point, and the non-real-time index QoE model corresponding to the hour point is generated based on the dynamic threshold and the dynamic weight. After the non-real-time index QoE model corresponding to the hour is generated, the model parameters of the non-real-time index QoE model corresponding to each hour point are saved for use in the following steps; the model parameters include the dynamic threshold and the dynamic weight of each index and other information.

[0103] It should be noted here that the dynamic threshold refers to a single-index dynamic threshold, and the time series prediction method is adopted to obtain the single-index dynamic threshold based on the index history value sequence. The specific type of the time series prediction method used in this embodiment is not limited.

[0104] As a possible implementation manner, the manner of obtaining the dynamic threshold includes: calculating the sigma value of the fluctuation of a single index based on the index history value sequence of the single index; predicting the current prediction value of the single index based on the index history value sequence of the single index; and determining the dynamic threshold of the single index based on the current prediction value and the sigma value. For example, the current prediction value plus or minus 3 times the sigma value is taken as the dynamic threshold of the index.

[0105] As a possible implementation manner, the objective weight assignment method (CRITIC) is used to determine the dynamic weight of each index in the index set. The objective weight assignment method calculates the weight of each index according to the correlation and volatility between sequences, which belongs to the prior art and will not be described here.

[0106] In step 203, the hour granularity index data of the first index set is evaluated by the non-real-time index QoE model to obtain the evaluation result of the hour granularity index data.

[0107] After obtaining the non-real-time index QoE model corresponding to the hour point, the hour granularity index data of the first index set can be evaluated through the non-real-time index QoE model to obtain the evaluation result of the hour granularity index data. The evaluation result of the embodiment is the QoE evaluation score of each hour point.

[0108] As an implementation manner, the evaluation score of the hour granularity index data of the first index set is obtained through the following formula:

[0109]

[0110] Wherein, S h is the QoE evaluation score of the hour point; S hi represents the score of the i-th index in the first index set; W i represents the corresponding dynamic weight of the i-th index in the first index set, and n is the number of indexes in the first index set.

[0111] Wherein, different types of indexes have different QoE scoring formulas, and exemplary QoE scoring formulas of single indexes are as follows:

[0112]

[0113] Wherein, A hi represents the current value of the i-th index of the first index set; base is the passing value of the index, goal is the excellent value of the index, and the base value and the goal value can be set and adjusted according to expert experience or determined based on the dynamic threshold of each index. The embodiment is determined based on the dynamic threshold of each index; y is the index QoE score corresponding to the passing index value (the value of y is 60 by default).

[0114] That is, at the current hour point, the hour granularity index data of the first index set at the current hour point and the index historical value sequence are obtained. Based on the index historical value sequence at the current hour point, the dynamic threshold and the dynamic weight of each index are obtained, based on the dynamic threshold and the dynamic weight, the non-real-time index QoE model of the current hour point is obtained, and the evaluation score of the hour granularity index data of the first index set at the current hour point is calculated through the non-real-time index QoE model.

[0115] It should be noted that after obtaining the evaluation score of each hour point, the QoE score S h of each hour point is saved; the model parameters of the QoE model of each hour point are saved, and the model parameters include the dynamic threshold of each index, the dynamic weight of each index, etc.

[0116] In step 204, based on the evaluation result of the hour granularity index data and the first preset service threshold, the target hour point is obtained.

[0117] After obtaining the evaluation scores of each hour time point, according to the QoE scores and the first preset service threshold, the poor quality and high quality hour time points can be screened out.

[0118] For example, the first preset service threshold includes a service poor quality threshold and a service high quality threshold, the QoE scores are compared with the service poor quality threshold T h_bad and the service high quality threshold T h_good The hour time point with the QoE score greater than the service high quality threshold is taken as the target hour time point, specifically, a high quality hour time point; the hour time point with the QoE score less than the service poor quality threshold is taken as the target hour time point, specifically, a poor quality hour time point. That is, the hour time point with the score S h less than the service poor quality threshold T h_bad is the poor quality hour time point, and the hour time point with the score S h greater than the service high quality threshold T h_good is the high quality hour time point.

[0119] By implementing the embodiment, the non-real-time service QoE model is used to screen the accurate poor quality and high quality hour time points for the real-time poor quality detection model.

[0120] The following describes in detail the evaluation of the minute granularity index data of the second index set in the target hour time point based on the non-real-time index QoE model according to the step 102 of the above embodiment in a specific matter manner.

[0121] Figure 3 The following describes in detail the evaluation of the minute granularity index data of the second index set in the target hour time point based on the non-real-time index QoE model according to the step 102 of the above embodiment in a specific matter manner.

[0122] Step 301, obtaining the minute granularity index data of the second index set in the target hour time point.

[0123] After determining the target hour time point, the minute granularity index data of the second index set in the target hour time point is obtained.

[0124] Step 302, evaluating the minute granularity index data of the second index set based on the dynamic parameters of the non-real-time index QoE model to obtain the evaluation result of the minute granularity index data of the second index set.

[0125] For the hour time point of poor quality and the hour time point of good quality, the model parameters of the QoE model corresponding to the hour time point are used to evaluate the index set data of each minute time point in the hour time point of poor quality and the hour time point of good quality, to obtain the evaluation score. That is, the dynamic threshold and the dynamic weight corresponding to the hour time point are used to evaluate the index set data of each minute time point in the hour time point of poor quality and the hour time point of good quality, to obtain the evaluation score.

[0126] The evaluation formula of this step is as follows:

[0127] S mi = (A mi -A mi_min ) / (A mi_max -A mi_min )

[0128]

[0129] Wherein, S m is the QoE evaluation score of the minute time point; S mi represents the evaluation score of the i-th index of the second index set; W i represents the dynamic weight corresponding to the i-th index of the second index set; A mi represents the current value of the i-th index of the second index set; A mi_max and A mi_min represent the maximum value and the minimum value of the i-th index of the first index set, that is, the dynamic threshold. The dynamic weight and the dynamic threshold here are the dynamic parameters of the non-real-time index QoE model of the corresponding hour time point which are obtained and saved before.

[0130] Step 303, based on the evaluation result of the minute granularity index data of the second index set and the second preset service threshold, the target minute time point is obtained.

[0131] It can be understood that the minute time point with the score S m lower than the service poor quality threshold T h_bad is regarded as the poor quality minute time point, and the minute time point with the score S m higher than the service good quality threshold T h_good is regarded as the good quality minute time point, and the poor quality minute time point and the good quality minute time point are preliminarily screened out.

[0132] Step 304, the dynamic parameters of the historical non-real-time index QoE model of the target minute time point are obtained.

[0133] For example, the dynamic parameters of the historical non-real-time index QoE model are the model parameters of the QoE model of the same hour time point of the previous day.

[0134] It can be understood that, in order to reduce the influence of the continuous deterioration of the index on the QoE score, the model parameters of the QoE model corresponding to other time periods (such as the same hour time point of the previous day) are used to calculate the score corresponding to the minute granularity index data of the target minute time point.

[0135] In step 305, based on the dynamic parameters of the historical non-real-time index QoE model, the minute granularity index data of the second index set of the target minute time point is evaluated to obtain the evaluation result of the minute granularity index data of the second index set of the target minute time point.

[0136] For example, for the poor quality and good quality hour time points, the model parameters of the QoE model corresponding to the same hour time point of the previous day are used to evaluate the index set data of each minute time point within the poor quality and good quality hour time points, respectively, to obtain the evaluation score S m2 .

[0137] That is, in this embodiment, by the above steps, for the poor quality and good quality hour time points, the model parameters of the QoE model corresponding to the hour time point and the model parameters of the QoE model corresponding to the same hour time point of the previous day are used to evaluate the index set data of each minute time point within the poor quality and good quality hour time points, respectively.

[0138] In step 306, based on the evaluation result of the minute granularity index data of the second index set of the target minute time point and the second preset service threshold, the target minute time point is screened to obtain an updated target minute time point.

[0139] It can be understood that, if S m2 is also lower than the poor quality threshold T h_bad , the poor quality minute time point is retained; if it is higher than the threshold, the poor quality minute time point is removed. If S m2 is also higher than the good quality threshold T h_good , the good quality minute time point is retained; if it is lower than the threshold, the good quality minute time point is removed.

[0140] By implementing this embodiment, the non-real-time service QoE model is used to screen accurate poor quality and good quality minute time points for the real-time poor quality detection model, so as to obtain more accurate labeled training data set for training the service poor quality detection model.

[0141] On the basis of any of the above embodiments, in order to obtain more accurate poor quality and good quality minute time points, an unsupervised anomaly detection method and a QoE model method can be used to mutually verify each other to obtain the target minute time point. This will be described in detail below. Figure 4A flowchart of a method for obtaining a training data set of a service quality difference detection model according to another embodiment of the present application is shown. The method for obtaining the training data set of the service quality difference detection model can include the following steps:

[0142] In step 401, the hour granularity index data of the first index set is evaluated based on the non-real-time index QoE model to obtain a target hour time point.

[0143] It should be noted that the implementation of this step can refer to steps 101 or 201-204 in the above embodiments for details, and the principle is the same, which will not be repeated here.

[0144] In step 402, the minute granularity index data of the second index set in the target hour time point is evaluated based on the non-real-time index QoE model to obtain a target minute time point.

[0145] It should be noted that the implementation of this step can refer to steps 102 or 301-306 in the above embodiments for details, and the principle is the same, which will not be repeated here.

[0146] In step 403, the index data of the fourth index set of the target minute time point is obtained, and the index data of the fourth index set is detected by an unsupervised anomaly detection model to obtain an anomaly detection result, and the target minute time point is filtered based on the anomaly detection result to obtain an updated target minute time point.

[0147] It can be understood that the unsupervised anomaly detection model is used to detect the index data corresponding to the preliminary filtered quality difference and quality minute time points, and the quality difference and quality minute time points are re-filtered.

[0148] For the target minute time point preliminarily filtered in the above step, the minute index data corresponding to the fourth index set at the minute time point is obtained, and the unsupervised anomaly detection model is used for anomaly judgment to further filter the quality difference and quality minute time points.

[0149] For example, the fourth index set (named Bm) is a multi-KPI / KQI index combination concerned by the unsupervised anomaly detection model, and the index value is of minute granularity. The index item can be selected according to the algorithm needs, and can be different from the index set Am.

[0150] It should be noted that the index items of the third index set Cm can be selected according to the algorithm needs, and can be different from the index set Am or the index set Bm.

[0151] There are many choices for unsupervised anomaly detection models, and there are currently many unsupervised anomaly detection models, including commonly used k-means clustering, isolation forest (IForest), local outlier factor (LOF), and collaborative filtering (Collaborative Filtering), etc. The embodiment selects the "Robust Random Cut Forest (RRCF)" algorithm model trained in advance based on the historical data set of the indicator set Bm.

[0152] The core idea of the random cut forest algorithm is to randomly select a dimension among all d dimensions of the data points, then randomly select two cut values in this dimension, and divide the data points into three parts (i.e. left node, right node and split node). Next, the above cutting is recursively performed on the left and right nodes until the depth of the tree reaches the maximum depth or the number of nodes reaches the maximum value. Finally, the scores of all nodes are determined to determine the abnormal points.

[0153] Step 404, obtaining the indicator data of the third indicator set at the target minute time point, and assigning a label to the indicator data of the third indicator set to obtain a labeled target data set.

[0154] It should be noted that the implementation of the present step can refer to the step 104 in the above embodiments for details, and the principle is the same, which will not be repeated here.

[0155] The business quality difference detection model of the embodiment of the present application uses the training data set acquisition method, which uses the unsupervised anomaly detection method and the QoE model method to mutually confirm the results, screens the quality difference and quality excellent minute time points, and constructs an indicator data set with quality difference and quality excellent labels, further improves the accuracy of the training data set, and thus improves the accuracy of the real-time quality difference detection model.

[0156] In order to clearly illustrate the above embodiments, specific examples will be described. Figure 5 A flowchart of a business quality difference detection model training data set acquisition method provided by the embodiment of the present application is shown in FIG. 1. Figure 5 As shown in the figure, the business quality difference detection model training data set acquisition method of the present application comprises:

[0157] S1: generating a non-real-time indicator QoE model, and performing business quality difference evaluation on the hour granularity non-real-time indicator data.

[0158] S2: saving the QoE scores S h of each hour granularity, saving the QoE model parameters of each hour granularity, and the QoE model parameters include dynamic threshold values of each indicator, dynamic weights of each indicator, etc.

[0159] S3: QoE scores S of each hour granularity h and service quality poor threshold T h_bad and service quality good threshold T h_good Screening quality poor and good hour time points, scores S h below the service quality poor threshold T h_bad The hour time point is the quality poor hour time point, the score S h above the service quality good threshold T h_good The hour time point is the quality good hour time point.

[0160] S4: Using non-real-time QoE model, screening within quality poor and good hour time points, completing preliminary screening of quality poor and good minute time points.

[0161] S4-1: Processing for quality poor hour time points:

[0162] S4-1-1: Using QoE model parameters (dynamic threshold and dynamic weight) corresponding to the period in the hour service scoring system to calculate index set A m The service score S corresponding to the index data of each minute granularity in each quality poor hour point m1 .

[0163] S4-1-2: Find all minute time points with service scores below the quality poor threshold T h_bad The minute time point is the quality poor minute time point.

[0164] S4-1-3: For quality poor minute time points, using QoE model parameters (dynamic threshold and dynamic weight) corresponding to other periods (such as the same period the day before) in the hour service scoring system to calculate the service total score S m2 corresponding to the minute granularity data.

[0165] S4-1-4: If S m2 is also below the quality poor threshold T h_bad , the quality poor minute time point is retained; if it is above the threshold, the quality poor minute time point is removed.

[0166] S4-1-5: Obtain index set B m The index values corresponding to the quality poor minute time points are processed using the "random forest cutting" algorithm model trained based on the historical data set of index set B m , appropriately relaxing the threshold for abnormality judgment to make a prediction judgment. If it is judged to be abnormal, the quality poor minute time point is retained.

[0167] S4-2: Processing for quality good hour time points:

[0168] S4-2-1: Calculate the index set A using the QoE model parameters (dynamic threshold and dynamic weight) corresponding to the time period in the hourly service scoring system m The service score S corresponding to the index data of each minute granularity within each hour quality point m1 .

[0169] S4-2-2: Find all minute time points with service scores higher than the quality threshold T h_good , which is the quality minute time point.

[0170] S4-2-3: For the quality minute time point, calculate the service total score S m2 corresponding to the minute granularity data using the QoE model parameters (dynamic threshold and dynamic weight) corresponding to other time periods (such as the same period of the previous day) in the hourly service scoring system.

[0171] S4-2-4: If S m2 is also higher than the quality threshold T h_good , keep the quality minute time point, and if it is lower than the threshold, remove the quality minute time point.

[0172] S4-2-5: Obtain index set B m , and use the "random forest cutting" algorithm model trained in advance based on the historical data set of index set B m to predict and judge the index data at the quality minute time point. If it is normal, keep the quality minute time point.

[0173] S5: Organize and summarize the filtered quality difference and quality minute time points, and combine index set C m to form a data set with quality difference and quality labels corresponding to the index values at each minute time point.

[0174] S6: Train the real-time quality difference detection model based on the data set, and output the quality difference anomaly detection model based on the real-time minute granularity index data.

[0175] The service quality detection model of the embodiment of the application uses a training data set acquisition method, uses a non-real-time service QoE model, and screens quality difference and quality optimal minute time points for a real-time quality detection model. Meanwhile, the results of the unsupervised anomaly detection method and the QoE model method are mutually verified to screen quality difference and quality optimal minute time points, and an index data set with quality difference and quality optimal labels is constructed. The improvement from unsupervised to supervised in the construction of the real-time quality detection model is realized, and the accuracy of the real-time quality detection model can be significantly improved. A fast and relatively accurate quality detection model is obtained, the efficiency and effect of actively discovering mobile Internet service quality degradation problems are enhanced, and the problem discovery time is reduced by about 60 minutes (from the hour level to the 5-minute level). On this basis, the problems are timely processed, the total number of service complaints is reduced, and the user satisfaction is significantly improved.

[0176] In the context of rich mobile Internet service (such as game, WEB browsing service) scenarios and various types of indicators, the embodiment of the application can be widely applied to communication equipment manufacturers, operators and the like to construct operation and maintenance products. The service quality can be effectively distinguished, the service perception can be measured, and thus the optimization capability of network operation and maintenance to construct difference customization services can be supported, the Internet user complaints can be effectively reduced, and the user perception can be improved.

[0177] To implement the above embodiment, the application further provides a device for acquiring a training data set of a service quality detection model. Figure 6 A structural schematic diagram of a device for acquiring a training data set of a service quality detection model provided by the embodiment of the application is shown in FIG. 1. Figure 6 As shown in the figure, the device for acquiring a training data set of a service quality detection model can include a first evaluation module 601, a second evaluation module 602 and a label processing module 603.

[0178] The first evaluation module 601 is configured to evaluate the hour granularity index data of the first index set based on a non-real-time index QoE model to obtain a target hour time point.

[0179] The second evaluation module 602 is configured to evaluate the minute granularity index data of the second index set in the target hour time point based on the non-real-time index QoE model to obtain a target minute time point.

[0180] The label processing module 603 is configured to acquire the index data of the third index set of the target minute time point, and assign labels to the index data of the third index set to obtain a labeled target data set; and train a service quality detection model by using the labeled target data set.

[0181] Further, in a possible implementation manner of the embodiment of the application, the first evaluation module 601 is specifically configured to:

[0182] obtain hour granularity index data of the first index set and a sequence of index historical values corresponding to the hour;

[0183] generate a non-real-time index QoE model corresponding to the hour based on the sequence of index historical values;

[0184] evaluate the hour granularity index data of the first index set through the non-real-time index QoE model to obtain an evaluation result of the hour granularity index data;

[0185] obtain a target hour time point based on the evaluation result of the hour granularity index data and a first preset service threshold.

[0186] Further, in a possible implementation manner of the embodiment of the application, the second evaluation module 602 is specifically configured to:

[0187] obtain minute granularity index data of a second index set in the target hour time point;

[0188] evaluate the minute granularity index data of the second index set based on the dynamic parameter of the non-real-time index QoE model to obtain an evaluation result of the minute granularity index data of the second index set;

[0189] obtain a target minute time point based on the evaluation result of the minute granularity index data of the second index set and a second preset service threshold.

[0190] Further, in a possible implementation manner of the embodiment of the application, the second evaluation module 602 is further configured to:

[0191] obtain a dynamic parameter of a historical non-real-time index QoE model of the target minute time point;

[0192] evaluate the minute granularity index data of the second index set of the target minute time point based on the dynamic parameter of the historical non-real-time index QoE model to obtain an evaluation result of the minute granularity index data of the second index set of the target minute time point;

[0193] filter the target minute time point based on the evaluation result of the minute granularity index data of the second index set of the target minute time point and the second preset service threshold to obtain an updated target minute time point.

[0194] Further, in a possible implementation manner of the embodiment of the application, the second evaluation module 602 is further configured to:

[0195] obtain index data of a fourth index set of the target minute time point;

[0196] detect the index data of the fourth index set through an unsupervised anomaly detection model to obtain an anomaly detection result;

[0197] Based on the abnormality detection result, the target minute time point is screened to obtain an updated target minute time point.

[0198] Further, in a possible implementation manner of the embodiment of the application, the target hour time point includes a good hour time point and a poor hour time point, and the label includes a good label or a poor label.

[0199] It should be noted that the foregoing explanation and description of the method embodiment for acquiring the training data set of the service quality detection model also applies to the device for acquiring the training data set of the service quality detection model, which will not be described here again.

[0200] Based on the foregoing embodiments, the application further provides a possible implementation manner of a device for acquiring a training data set of a service quality detection model. On the basis of the last embodiment, the device for acquiring the training data set of the service quality detection model further includes a model training module 604, configured to: supervise training of a real-time service quality detection model based on a target data set to obtain a minute-granularity service quality detection model.

[0201] The device for acquiring the training data set of the service quality detection model according to the embodiments of the application uses a non-real-time service QoE model to screen poor and good minute time points for a real-time quality detection model. Meanwhile, the unsupervised anomaly detection method and the QoE model method are used to verify each other, to screen poor and good minute time points, and to construct an index data set with poor and good labels. The improvement from unsupervised to supervised in the construction of the real-time quality detection model is realized, which can significantly improve the accuracy of the real-time quality detection model.

[0202] To implement the foregoing embodiments, the application further provides an electronic device, including a processor and a memory connected with the processor in communication; the memory stores computer execution instructions; and the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0203] To implement the foregoing embodiments, the application further provides a computer readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, the computer execution instructions are used to implement the method provided in the foregoing embodiments.

[0204] To implement the foregoing embodiments, the application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the computer program implements the method provided in the foregoing embodiments.

[0205] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the application comply with relevant laws and regulations and do not violate public order and good customs.

[0206] It is important to note that user's personal information shall be collected for legitimate and reasonable uses of the functionality and not shared or sold outside of those legitimate uses. Further, such collection / sharing shall occur after receiving the consent of the users, including but not limited to informing the users to read the user agreement / user notice before using the functionality and signing the agreement / authorization including authorization of relevant user information. In addition, any necessary steps shall be taken to safeguard and secure access to such personal information data and ensure that others with access to the personal information data adhere to their privacy policies and procedures.

[0207] The present application contemplates providing an implementation in which the user has the option to opt in or opt out of permitting the collection and / or access to personal information data. That is, the present disclosure contemplates providing user- controlled privacy settings that allow the user to restrict the use of personal information data. For example, users can opt in or opt out of certain data collection, use, or sharing by the present application. In particular, users can opt in or opt out of the collection and use of personal information data for certain purposes at the time of sign-up, at the time of login, or at any time by editing their settings. It will be appreciated that the present application might collect certain metadata regarding a user's interactions with the application and use of the application without the user's opt-in consent but with the user's prior agreement to the application's terms and conditions.

[0208] In the foregoing detailed description of embodiments of the application, various specific terminology is used. For example, the terms "one or more embodiments," "some embodiments,” “an example,” “a specific example,” or “some examples” are used. Such terminology refers, and is used in reference to, one or more particular embodiments or examples of the application for purposes of articulating the description of the particular embodiments or examples. This use of such terminology in the detailed description is not intended to in any way limit the scope of the application, but rather is intended to help describe or identify the embodiments or examples for which the particular features are described. Thus, the terminology used in the detailed description is used for describing particular embodiments or examples, but is not intended to limit the scope of the application. Moreover, the use of the terms first, second, etc. do not denote any quantity or importance, but rather are used to distinguish one element from another. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner on an embodiment or example by example basis.

[0209] In addition, the terms "first", "second", etc. are used herein only to describe various steps in a flow chart or describe various embodiments, and do not imply a relative importance or a specific order. Thus, a feature described as a "first" feature can later be described as a "second" feature, and vice versa. Moreover, the use of the terms "first", "second", etc., are used herein to identify individual features of the application for purposes of description, and are not intended to limit the scope of the application.

[0210] Any processes or methods described in the flow charts or elsewhere in this specification can be understood as representing code or modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions (or steps in a method). The various embodiments of the application can include additional or fewer processes or methods, as desired. Moreover, the operations of the processes or methods can be rearranged or reordered in other embodiments. The various processes or methods described herein can be implemented using hardware, software, or a combination thereof. When implemented in software, the software structure can include modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions (or steps in a method). The results of the various processes or methods described herein can be stored in a memory or other storage location.

[0211] The logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be considered as a sequence of instructions to implement logic functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a machine-readable storage device (e.g., magnetic, optical or other storage device), a machine-readable storage substrate, a machine-readable signal, or any combination thereof. Other, specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electrical) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). Note that the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner into an electronic computer readable medium, then stored in the computer memory.

[0212] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. As such, in some embodiments, memory (including computer-readable storage media) can store software or firmware for use by the instruction execution system. In other embodiments, hardware logic (including programmable logic for use with a programmable logic device) can be used to implement at least some of the steps or methods.

[0213] Those of skill in the art would understand that information and signals can be represented using any of a variety of technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0214] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0215] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for obtaining a training dataset for a business quality poor detection model, characterized in that, Includes the following steps: The hourly granularity index data of the first index set is evaluated based on the non-real-time index experience quality QoE model to obtain the target hourly time point; The minute-level index data of the second index set within the target hour time point are evaluated based on the non-real-time index QoE model to obtain the target minute time point; After obtaining the target minute time point, the method further includes: acquiring indicator data of the fourth indicator set of the target minute time point; detecting the indicator data of the fourth indicator set through an unsupervised anomaly detection model to obtain anomaly detection results; and filtering the target minute time point based on the anomaly detection results to obtain updated target minute time points. Obtain the indicator data of the third indicator set at the target minute time point, and assign labels to the indicator data of the third indicator set to obtain a labeled target dataset; train a business quality poor detection model using the labeled target dataset.

2. The method according to claim 1, characterized in that, The evaluation of hourly granular index data of the first index set based on the non-real-time index QoE model to obtain the target hourly time point includes: Obtain hourly granular indicator data and the corresponding hourly historical value sequence of indicators for the first indicator set; Based on the historical value sequence of the aforementioned indicator, a non-real-time indicator QoE model for the corresponding hour is generated; The hourly granularity index data of the first index set is evaluated using the non-real-time index QoE model to obtain the evaluation results of the hourly granularity index data. Based on the evaluation results of the hourly granularity index data and the first preset business threshold, the target hourly time point is obtained.

3. The method according to claim 1, characterized in that, The evaluation of minute-level index data of the second index set within the target hourly time point based on the non-real-time index QoE model yields the target minute time point; including: Obtain minute-level index data of the second index set within the target hour time point; Based on the dynamic parameters of the non-real-time indicator QoE model, the minute-level indicator data of the second indicator set is evaluated to obtain the evaluation results of the minute-level indicator data of the second indicator set. Based on the evaluation results of the minute-level index data of the second index set and the second preset business threshold, the target minute time point is obtained.

4. The method according to claim 3, characterized in that, After obtaining the target minute time point, the process also includes: Obtain the dynamic parameters of the historical non-real-time indicator QoE model for the target minute time point; Based on the dynamic parameters of the historical non-real-time indicator QoE model, the minute-level indicator data of the second indicator set at the target minute time point are evaluated to obtain the evaluation results of the minute-level indicator data of the second indicator set at the target minute time point. Based on the evaluation results of the minute-level index data of the second set of indicators for the target minute time point and the second preset business threshold, the target minute time point is filtered to obtain the updated target minute time point.

5. The method according to claim 1, characterized in that, The step of training a poor service quality detection model using the labeled target dataset includes: Based on the target dataset, a supervised training method for detecting poor service quality is performed on the real-time service quality detection model to obtain a service quality detection model with minute-level granularity.

6. The method according to claim 1, characterized in that, The target hour time point includes high-quality hour time points and low-quality hour time points, and the label includes a high-quality label or a low-quality label.

7. A device for acquiring a training dataset for a business quality poor detection model, characterized in that, include: The first evaluation module is used to evaluate the hourly granular indicator data of the first indicator set based on the non-real-time indicator experience quality QoE model to obtain the target hour time point; The second evaluation module is used to evaluate the minute-level index data of the second index set within the target hour time point based on the non-real-time index QoE model, so as to obtain the target minute time point. After obtaining the target minute time point, the method further includes: acquiring indicator data of the fourth indicator set of the target minute time point; detecting the indicator data of the fourth indicator set through an unsupervised anomaly detection model to obtain anomaly detection results; and filtering the target minute time point based on the anomaly detection results to obtain updated target minute time points. The label processing module is used to acquire the indicator data of the third indicator set at the target minute time point, and assign labels to the indicator data of the third indicator set to obtain a labeled target dataset; and to train a business quality poor detection model using the labeled target dataset.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

Images