Index threshold determination method and device, electronic equipment and storage medium

Abstract

The invention relates to an index threshold determination method and device, electronic equipment and a computer readable storage medium, relates to the technical field of network operation, and can be applied to a scene of screening a target service form according to a determined service index. The method comprises the following steps: determining an index dynamic threshold, wherein the index dynamic threshold is used for screening a target business form from business forms to be processed; the index dynamic threshold is generated based on a threshold determination model, and the threshold determination model is obtained by training one or more of the time dimension, the geographic dimension, the user dimension, the network dimension and the service dimension of the historical service form and is dynamically updated based on the real-time service form. The invention provides a scheme for automatically generating the dynamic index thresholds, the dynamic index thresholds of various indexes are automatically determined and adjusted through the threshold determination model, the network operation capability is improved, manual intervention can be reduced, and the working efficiency is improved.

Description

[0001] This application claims priority to Chinese patent application number 202311320233.5, filed on October 12, 2023, and entitled “Indicator Threshold Determination Method and Device, Electronic Device and Storage Medium”, the entire contents of which are incorporated herein by reference. Background Art

[0003] Network failure refers to the state where the network cannot provide normal services or the service quality is reduced due to hardware problems, software vulnerabilities, virus intrusion, etc. Before locating the root cause of the network failure, it is usually possible to set thresholds for indicators such as network quality, network performance, and service quality based on expert experience, screen out poor quality call records, and then use artificial intelligence and big data analysis and other technical means to identify and segment the root cause of the poor quality.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide an indicator threshold determination method, an indicator threshold determination device, an electronic device and a computer-readable storage medium, thereby at least to a certain extent overcoming the problem that the service indicator threshold configuration that causes network failures needs to be manually set and cannot be adaptively adjusted according to real-time changes in the network.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present invention.

[0007] According to a first aspect of the present disclosure, a method for determining an indicator threshold is provided, comprising: determining an indicator dynamic threshold, wherein the indicator dynamic threshold is used to screen a target business form from business forms to be processed; the indicator dynamic threshold is generated based on a threshold determination model, wherein the threshold determination model is trained based on one or more dimensions of a time dimension, a geographic dimension, a user dimension, a network dimension, and a business dimension of historical business forms, and is dynamically updated based on real-time business forms.

[0008] In an exemplary embodiment of the present disclosure, the above method also includes: collecting network performance data, service quality data, configuration data and business operation data from the network through deep packet inspection probes and a production system to generate an initial historical business form; the production system includes one or more of a network management system, a customer relationship management system, a work order system, and a customer service system.

[0009] In an exemplary embodiment of the present disclosure, after generating the initial historical business form, the method further includes: reviewing the initial historical business form to obtain the historical business form.

[0010] In an exemplary embodiment of the present disclosure, the threshold determination model is generated by the following steps: using the historical business form with classification labels, model training is performed on a pre-built initial model to obtain the threshold determination model, the threshold determination model uses probability density analysis and n*σ criterion to determine the indicator dynamic threshold of each business-related indicator in the historical business form, and the indicator dynamic threshold is used to determine the classification label of the business form to be processed.

[0011] In an exemplary embodiment of the present disclosure, the use of probability density analysis and n*σ criterion to determine the indicator dynamic threshold of each of the business-related indicators includes: using the n*σ criterion to determine the indicator threshold upper limit and the indicator threshold lower limit of the business-related indicator; determining the indicator dynamic threshold according to the indicator threshold upper limit and the indicator threshold lower limit; the indicator threshold upper limit and the indicator threshold lower limit are determined based on the indicator probability distribution of the business-related indicator, and the indicator probability distribution is determined based on the indicator mean, indicator variance and probability density function of the business-related indicator.

[0012] In an exemplary embodiment of the present disclosure, determining the indicator dynamic threshold according to the indicator threshold upper limit and the indicator threshold lower limit includes: determining the indicator normal distribution of the business-related indicator, the indicator normal distribution is determined based on the indicator threshold upper limit, the indicator threshold lower limit and the indicator grouping interval of the business-related indicator, the indicator grouping interval is determined according to the indicator threshold upper limit, the indicator threshold lower limit and the number of threshold groups; determining the indicator dynamic threshold according to the inverse function of the cumulative distribution function of the indicator normal distribution.

[0013] In an exemplary embodiment of the present disclosure, the method further comprises: using an association mining algorithm to perform secondary inspection and optimization processing on the indicator dynamic threshold to obtain the optimized indicator dynamic threshold.

[0014] In an exemplary embodiment of the present disclosure, the association mining algorithm is used to perform secondary inspection and optimization processing on the indicator dynamic threshold to obtain the optimized indicator dynamic threshold, including: classifying the historical business forms according to business type to obtain frequent 1-item sets; determining the minimum support threshold and the minimum confidence threshold corresponding to each of the classified business indicators; and determining whether to trigger a threshold update operation for the indicator dynamic threshold based on the frequent 1-item set, the minimum support threshold and the minimum confidence threshold to obtain the optimized indicator dynamic threshold.

[0015] In an exemplary embodiment of the present disclosure, the determining whether to trigger a threshold update operation on the indicator dynamic threshold based on the frequent 1-item set, the minimum support threshold and the minimum confidence threshold to obtain the optimized indicator dynamic threshold includes: performing a frequent item set determination operation based on the frequent 1-item set to obtain all frequent k-item sets in the historical business form; calculating association rules based on all the frequent k-item sets to obtain indicator confidence; when the indicator confidence is greater than or equal to the minimum confidence threshold, using the indicator dynamic threshold as the optimized indicator dynamic threshold; when the indicator confidence is less than the minimum confidence threshold, triggering the update operation of the indicator dynamic threshold to obtain the optimized indicator dynamic threshold.

[0016] In an exemplary embodiment of the present disclosure, the frequent item set determination operation is performed based on the frequent 1-item set to obtain all frequent k-item sets in the historical business form, including: scanning the frequent 1-item set, determining frequent k-item sets and the support of each of the frequent k-item sets based on the scanning result of the frequent 1-item set; determining frequent k-item sets according to the minimum support among the supports of multiple frequent k-item sets; and repeatedly performing the above steps until the number of determined frequent k+j item sets is less than or equal to 1, thereby obtaining all the frequent k-item sets.

[0017] In an exemplary embodiment of the present disclosure, the above method also includes: using the real-time business form to perform model update processing on the threshold determination model to obtain an updated threshold determination model, wherein the updated threshold determination model is used to adjust the dynamic threshold of the indicator, and the real-time business form is a business form obtained by periodic sampling based on the time dimension.

[0018] According to a second aspect of the present disclosure, there is provided an indicator threshold determination device, comprising: an indicator threshold determination module, for determining an indicator dynamic threshold, wherein the indicator dynamic threshold is used to screen target business forms from business forms to be processed, wherein the indicator dynamic threshold is generated based on a threshold determination model, wherein the threshold determination model is trained based on one or more dimensions of a time dimension, a geographic dimension, a user dimension, a network dimension, and a business dimension of historical business forms, and is dynamically updated based on real-time business forms.

[0019] In an exemplary embodiment of the present disclosure, the indicator threshold determination device also includes an initial historical form collection module, which is used to: collect network performance data, service quality data, configuration data and business operation data from the network through deep packet inspection probes and production systems to generate initial historical business forms; the production system includes one or more of a network management system, a customer relationship management system, a work order system, and a customer service system.

[0020] In an exemplary embodiment of the present disclosure, the indicator threshold determination device further includes a historical form generation module, which is used to: review the initial historical business form to obtain the historical business form.

[0021] In an exemplary embodiment of the present disclosure, the indicator threshold determination device also includes a threshold model generation module, which is used to: use the historical business form with classification labels to perform model training on a pre-built initial model to obtain the threshold determination model, and the threshold determination model uses probability density analysis and n*σ criterion to determine the indicator dynamic threshold of each business-related indicator in the historical business form, and the indicator dynamic threshold is used to determine the classification label of the business form to be processed.

[0022] In an exemplary embodiment of the present disclosure, the threshold model generation module includes a threshold determination unit, which is used to: adopt the n*σ criterion to determine the indicator threshold upper limit and the indicator threshold lower limit of the business-related indicator; determine the indicator dynamic threshold according to the indicator threshold upper limit and the indicator threshold lower limit; the indicator threshold upper limit and the indicator threshold lower limit are determined based on the indicator probability distribution of the business-related indicator, and the indicator probability distribution is determined based on the indicator mean, indicator variance and probability density function of the business-related indicator.

[0023] In an exemplary embodiment of the present disclosure, the threshold determination unit includes a threshold determination sub-unit, which is used to: determine the indicator normal distribution of the business-related indicator, the indicator normal distribution is determined based on the indicator threshold upper limit, the indicator threshold lower limit and the indicator grouping interval of the business-related indicator, and the indicator grouping interval is determined according to the indicator threshold upper limit, the indicator threshold lower limit and the number of threshold groups; determine the indicator dynamic threshold according to the inverse function of the cumulative distribution function of the indicator normal distribution.

[0024] In an exemplary embodiment of the present disclosure, the indicator threshold determination device also includes a threshold updating module, which is used to: use an association mining algorithm to perform secondary inspection and optimization processing on the indicator dynamic threshold to obtain the optimized indicator dynamic threshold.

[0025] In an exemplary embodiment of the present disclosure, the threshold update module includes a threshold update unit, which is used to: classify the historical business forms according to the business type to obtain a frequent 1-item set; determine the minimum support threshold and the minimum confidence threshold corresponding to each of the classified business indicators; and determine whether to trigger a threshold update operation for the indicator dynamic threshold based on the frequent 1-item set, the minimum support threshold and the minimum confidence threshold to obtain the optimized indicator dynamic threshold.

[0026] In an exemplary embodiment of the present disclosure, the threshold updating unit includes a threshold updating subunit, which is used to: perform a frequent item set determination operation based on the frequent 1-item set to obtain all frequent k-item sets in the historical business form; calculate association rules based on all the frequent k-item sets to obtain indicator confidence; when the indicator confidence is greater than or equal to the minimum confidence threshold, use the indicator dynamic threshold as the optimized indicator dynamic threshold; when the indicator confidence is less than the minimum confidence threshold, trigger the update operation of the indicator dynamic threshold to obtain the optimized indicator dynamic threshold.

[0027] In an exemplary embodiment of the present disclosure, the threshold updating subunit includes a frequent item set determination subunit, which is used to: scan the frequent 1-item set, determine the frequent k-item set and the support of each of the frequent k-item set based on the scanning result of the frequent 1-item set; determine the frequent k-item set according to the minimum support among the supports of multiple frequent k-item sets; repeat the above steps until the number of determined frequent k+j item sets is less than or equal to 1, and obtain all the frequent k-item sets.

[0028] In an exemplary embodiment of the present disclosure, the indicator threshold determination device also includes a model update module, which is used to: use the real-time business form to perform model update processing on the threshold determination model to obtain an updated threshold determination model, and the updated threshold determination model is used to adjust the indicator dynamic threshold. The real-time business form is a business form obtained by periodic sampling based on the time dimension.

[0029] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the method for determining an indicator threshold according to any one of the above items is implemented.

[0030] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for determining an indicator threshold according to any one of the above items is implemented.

[0031] The technical solution provided by the present disclosure may have the following beneficial effects:

[0032] In the exemplary embodiment of the present disclosure, the indicator threshold determination method, on the one hand, is based on the historical business form training to obtain the threshold generation model, the threshold generation model can adapt to multiple dimensions in the historical business form, obtain the indicator dynamic threshold of various indicators, and improve the network operation capability. On the other hand, because the threshold generation model can be dynamically updated according to the real-time business form, the indicator dynamic threshold can be automatically adjusted, reducing manual intervention, improving work efficiency, and reducing operation and maintenance costs.

[0033] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0035] Figure 1 A flowchart of a method for determining an indicator threshold according to an exemplary embodiment of the present disclosure is schematically shown;

[0036] Figure 2 A system architecture diagram for implementing an indicator threshold determination method according to an exemplary embodiment of the present disclosure is schematically shown;

[0037] Figure 3 A flowchart of determining the dynamic threshold of each service-related indicator by using probability density analysis and n*σ criterion according to an exemplary embodiment of the present disclosure is schematically shown;

[0038] Figure 4 A flowchart of determining an indicator dynamic threshold according to an indicator threshold upper limit and an indicator threshold lower limit according to an exemplary embodiment of the present disclosure is schematically shown;

[0039] Figure 5 A block diagram schematically shows an indicator threshold determination device according to an exemplary embodiment of the present disclosure;

[0040] Figure 6 A block diagram schematically shows an electronic device according to an exemplary embodiment of the present disclosure;

[0041] Figure 7 A schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0042] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0043] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations, materials or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0044] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0045] The relevant technical solutions have the following technical problems in the configuration of indicator thresholds: (1) The setting of indicator thresholds is mainly based on fixed thresholds given by expert experience, which cannot make accurate judgments and rapid adjustments in a timely manner according to changes in the network and customer service experience, which leads to evaluation results that are inconsistent with reality, and problems are missed or there are excessive negative reviews, which ultimately leads to misjudgment or omission of factors that affect customers, and there is a risk of inaccurate fault location.

[0046] (2) Experience thresholds are often considered from a holistic perspective, which results in a one-size-fits-all approach. They are usually trained and modeled based on historical data from a certain time series, without taking into account factors such as unbalanced regional development, asynchronous network construction maturity, and differentiated customer service usage.

[0047] (3) The configuration of the experience threshold needs to be manually processed and cannot be adjusted adaptively, which not only consumes manpower costs but also affects network operation efficiency.

[0048] Based on this, according to the embodiments of the present disclosure, a method for determining an indicator threshold, an apparatus for determining an indicator threshold, a computer-readable storage medium, and an electronic device are proposed.

[0049] In this article, the variable x in a service detailed record (xDR) represents different service types, such as voice service, data service, etc. The xDR records the service data in detail, including time dimension, geographical dimension, network / network element dimension, user dimension, service dimension, terminal dimension, etc.

[0050] In this example embodiment, a method for determining an indicator threshold is first provided. The method for determining an indicator threshold of the present disclosure may be implemented using a server, or the method described in the present disclosure may be implemented using a terminal device, wherein the terminal described in the present disclosure may include mobile terminals such as mobile phones, tablet computers, laptop computers, PDAs, and personal digital assistants (PDAs), as well as fixed terminals such as desktop computers. Figure 1 The following is a schematic diagram of a method for determining an indicator threshold according to some embodiments of the present disclosure. Figure 1 , the indicator threshold determination method may include the following steps:

[0051] Step S110, determine the dynamic threshold of the indicator, which is used to screen the target business form from the business forms to be processed; the dynamic threshold of the indicator is generated based on the threshold determination model, and the threshold determination model is trained based on one or more dimensions of the time dimension, geographical dimension, user dimension, network dimension, and business dimension of the historical business form, and is dynamically updated based on the real-time business form.

[0052] According to some exemplary embodiments of the present disclosure, the dynamic threshold of the indicator may be a reference threshold corresponding to the indicator value of different dimensions, and the dynamic threshold of the indicator may be used to measure whether a certain indicator in the business form meets the requirements. The pending business form may be a business form to be classified using the dynamic threshold of the indicator. The target business form may be a business form that does not meet specific requirements and is screened out from the pending business forms through screening. The threshold determination model may be an intelligent algorithm model for determining the dynamic threshold of the indicator. The historical business form may be a business form generated within a specific time period in the past. The real-time business form may be a new business form generated within the current time period.

[0053] In order to determine the corresponding dynamic thresholds for different indicators in the business form, the historical business form can be used as training data for model training to obtain a threshold determination model. The historical business form can be a call form of various business types collected from multiple different systems, and the data dimensions in the historical business form are formed by comprehensive processing of the data dimensions involved in multiple systems. The historical business form can contain multi-dimensional data such as time dimension, geographical dimension, user dimension, network dimension, and business dimension.

[0054] Specifically, the time dimension may be relevant information representing the time period in the historical service form. The geographic dimension may be relevant information about the geographical area to which the historical service form belongs. The user dimension may be the specific category of users included in the historical service form. The network dimension may be network-related information included in the historical service form. For example, the network dimension may include relevant information such as network quality, network element information, and network performance. The service dimension may be the specific service category involved in the historical service form. The service category may include multiple services such as video, game, and voice.

[0055] The historical business forms containing the above data dimensions are used as training data for model training to obtain a threshold determination model. The threshold determination model can adapt to the business indicators of multiple different data dimensions and determine the dynamic thresholds of indicators corresponding to each of the multiple business indicators, so that the business forms to be processed can be classified and processed according to the dynamic thresholds of indicators. If the business indicator of the business form to be processed is less than the dynamic threshold of the indicator, the business form to be processed is used as the target business form, that is, the business forms that do not meet the specific business quality requirements are screened out, so that the target business forms can be subsequently demarcated, segmented, and analyzed to locate the root cause of the poor quality of such forms.

[0056] In order to adapt to the ever-changing actual network environment and determine the dynamic threshold of indicators that are more in line with the real-time status, the business forms obtained in real time (i.e., real-time business forms) can be used as model training data, and the threshold determination model can be incrementally optimized and updated. The updated threshold determination model can be used to automatically adjust the dynamic threshold of indicators corresponding to each business indicator, and the adjusted dynamic threshold of indicators can be used as the basis for screening target business forms. According to the adjusted dynamic threshold of indicators, poor-quality business forms are screened out, and the poor-quality business forms are used as input data for locating the root cause of poor quality, thereby improving the accuracy of locating the root cause of poor quality.

[0057] According to the indicator threshold determination method in this example embodiment, on the one hand, a threshold determination model is obtained based on historical business form training, and the threshold determination model can adapt to multiple dimensions in the historical business form to obtain dynamic thresholds of indicators for various indicators, thereby improving network operation capabilities. On the other hand, since the threshold generation model can be dynamically updated according to real-time business forms, the dynamic threshold of the indicator can be automatically adjusted, reducing manual intervention, improving work efficiency, and reducing operation and maintenance costs.

[0058] The following further describes the indicator threshold determination method in this example embodiment.

[0059] In an exemplary embodiment of the present disclosure, network performance data, service quality data, configuration data and business operation data are collected from the network through deep packet inspection probes and a production system to generate an initial historical business form; the production system includes one or more of a network management system, a customer relationship management system, a work order system, and a customer service system.

[0060] Among them, the Deep Packet Inspection (DPI) probe can be a traffic detection and control technology based on the application layer, which adds application protocol identification of application layer data, data packet content detection and deep decoding on top of the traditional Internet Protocol (IP) data packet detection technology. The production system can refer to an information system that supports the daily business operations of the unit under normal circumstances. Network performance data can be indicator data for measuring network performance, including jitter, packet loss rate, latency, etc. Service quality data can be quality feedback data of related customer services provided by communication service providers to customers. Service quality data can include video download rate of video services, page opening latency of browsing services, etc. Configuration data can be relevant system configuration data in various systems. Business operation data can be network management data, customer service data, etc. The initial historical business form can be data collected directly through the deep packet inspection probe or the production system.

[0061] refer to Figure 2 , Figure 2 A system architecture diagram for implementing an indicator threshold determination method according to an exemplary embodiment of the present disclosure is schematically shown. When determining the historical business form, the xDR form data related to the customer experience can be first collected from the network management system, customer relationship management system, work order system, customer service system, etc. through the data collection module 210. For example, the data collection module 210 can use DPI probes and various types of production systems to collect network performance data, service quality data, configuration data, and business operation data from the network. After the above data is collected, the above data is merged into a data table to obtain the corresponding xDR form data wide table as the initial historical business form 211.

[0062] Specifically, a network management system (NMS) can be a system that adjusts the network status by combining software and hardware, and is responsible for the management and operation of network resources such as core networks, wireless networks, and transmission networks. A customer relationship management system (CRM) can help companies collect, organize, update, and store customer information, share and collaborate on customer data, and improve the accuracy and real-time nature of customer information. A work order system can be used to record, process, and track the completion of a job. A customer service system can be a system that coordinates personnel, business processes, technology, and strategy. Based on the user complaint data provided by the customer service system, operations such as user complaint processing and user behavior analysis can be performed.

[0063] Through the above processing steps, an initial historical business form including multiple dimensions such as time dimension, geographical dimension, user dimension, network dimension, and business dimension is obtained, so as to generate a historical business form based on the initial historical business form. For example, network performance data, service quality data, configuration data, and service operation data obtained from the network can be business data related to customer experience. After the above business data related to customer experience is collected from the network, customer experience analysis can be performed based on the above data.

[0064] In an exemplary embodiment of the present disclosure, the initial historical business form is reviewed to obtain the historical business form.

[0065] Continue to refer Figure 2 In order to obtain form data that meets the standard data specifications, after obtaining the initial historical business form, the initial historical business form 211 is input into the data review module 220, and the data integrity and accuracy of the initial historical business form 211 are automatically verified and improved by the data review module 220, and the processed business form is used as the historical business form. When the data review module 220 reviews the initial historical business form 211, some business forms that are missing specific values ​​can be deleted or mean-added to obtain a historical business form that meets the requirements. Through the above steps, the historical business form obtained can be used as model training data for the threshold determination model and used in the model training process.

[0066] In an exemplary embodiment of the present disclosure, a threshold determination model is generated by the following steps: using historical business forms with classification labels, a pre-built initial model is trained to obtain a threshold determination model, the threshold determination model uses probability density analysis and n*σ criterion to determine the indicator dynamic threshold of each business-related indicator in the historical business form, and the indicator dynamic threshold is used to determine the classification label of the business form to be processed.

[0067] The business-related index may be an index specifically related to a specific business of the network service. The classification label may be a label indicating whether a business form meets the business requirements. If a business form does not meet the specific business requirements, its classification label is configured as a non-compliant form.

[0068] Continue to refer Figure 2 The threshold training module 230 obtains a pre-built initial model, which may be a pre-built artificial intelligence (AI) model. The model training module 231 uses the obtained historical business form to perform model training on the initial model to obtain a threshold determination model. During the model training process, the pre-built initial model determines the indicator dynamic threshold of each business-related indicator in the historical business form by using probability density analysis and n*σ criterion.

[0069] After obtaining the indicator dynamic threshold of each business-related indicator, the indicator dynamic threshold is used to classify the historical business forms. For example, if the indicator value of a business-related indicator of the historical business form is less than the indicator dynamic threshold, the classification label of the historical business form is set to 1, indicating that it does not meet specific business requirements; if the indicator value of a business-related indicator of the historical business form is greater than or equal to the indicator dynamic threshold, the classification label of the historical business form is set to 0, indicating that it meets specific business requirements.

[0070] During the use of the model, the threshold determination model also uses probability density analysis and n*σ criteria to determine the dynamic thresholds of the indicators of each business-related indicator in the business form to be processed, and uses the dynamic thresholds of the indicators as reference values ​​to determine the classification labels of the business form to be processed, thereby obtaining the classification labels of each business form to be processed. The threshold determination model obtained through the above processing steps can adapt to multiple different data dimensions and obtain the dynamic thresholds of indicators of different business indicators for the classification process of the business form to be processed.

[0071] In an exemplary embodiment of the present disclosure, probability density analysis and n*σ criterion are used to determine the indicator dynamic threshold of each business-related indicator, including: using the n*σ criterion to determine the indicator threshold upper limit and the indicator threshold lower limit of the business-related indicator; determining the indicator dynamic threshold according to the indicator threshold upper limit and the indicator threshold lower limit; the indicator threshold upper limit and the indicator threshold lower limit are determined based on the indicator probability distribution of the business-related indicator, and the indicator probability distribution is determined based on the indicator mean, indicator variance and probability density function of the business-related indicator.

[0072] The upper limit of the indicator threshold may be the maximum value that a certain indicator can obtain. The lower limit of the indicator threshold may be the minimum value that a certain indicator can obtain. The indicator probability distribution may be the probability distribution of all values ​​of a certain indicator.

[0073] refer to Figure 3 , Figure 3 A flowchart for determining the dynamic threshold values ​​of indicators for each business-related indicator using probability density analysis and the n*σ criterion according to an exemplary embodiment of the present disclosure is schematically shown. The reviewed historical business forms are input into the pre-built initial model, and the data dimensions of the historical business forms cover multiple dimensions such as time, region, user, network, and business. For example, according to the regional dimension, the historical business forms (the total number of orders is N) of the same quarter are subjected to correlation analysis, and the probability density analysis of each indicator value related to customer experience is performed. In step S310, the n*σ criterion is used to determine the upper limit and lower limit of the indicator threshold of the business-related indicator. The n*σ (n is an integer between 1 and 6, and is adaptively adjusted according to the indicator data) criterion is used to determine the reasonable value range of each business-related indicator, including the upper limit and lower limit of the indicator threshold of the business-related indicator.

[0074] For each business-related indicator, the indicator mean and indicator variance corresponding to the business-related indicator can be calculated respectively, and the probability density function of the business-related indicator can be calculated. According to the calculated indicator mean, indicator variance and probability density function, the indicator probability distribution of the business-related indicator is determined. After obtaining the corresponding indicator probability distribution, the indicator threshold upper limit and indicator threshold lower limit of the business-related indicator can be determined.

[0075] In step S320, based on the indicator threshold upper limit and the indicator threshold lower limit, the indicator dynamic threshold of the business-related indicator is determined. For example, for a single business-related indicator, the historical data of the business-related indicator is obtained, and the corresponding indicator mean μ and indicator variance σ are determined according to the historical data. Analyze its probability distribution and use the n*σ criterion to obtain the upper limit of the indicator threshold T1=μ+n*σ and the lower limit of the indicator threshold T0=μ-n*σ of the reasonable value of the indicator.

[0076] According to the dynamic threshold of the indicator, the historical business forms can be labeled and classified to obtain the classification labels corresponding to the historical business forms. The historical business forms with classification labels are used as model training data to train the initial model to obtain the threshold determination model. The threshold determination model trained through the above steps can adaptively determine the dynamic threshold of indicators for different business-related indicators to classify the business forms to be processed.

[0077] In an exemplary embodiment of the present disclosure, a dynamic threshold of an indicator is determined according to an upper limit of an indicator threshold and a lower limit of an indicator threshold, including: determining an indicator normal distribution of a business-related indicator, the indicator normal distribution is determined based on an upper limit of an indicator threshold, a lower limit of an indicator threshold and an indicator grouping interval of the business-related indicator, the indicator grouping interval is determined according to an upper limit of an indicator threshold, a lower limit of an indicator threshold and the number of threshold groups; determining the dynamic threshold of the indicator according to an inverse function of a cumulative distribution function of the indicator normal distribution.

[0078] The index grouping distance is the distance between the highest value and the lowest value of each group after the index values ​​are grouped. The threshold grouping quantity can be the quantity obtained by grouping the values ​​of the upper limit and lower limit of the index threshold.

[0079] refer to Figure 4 , Figure 4 The flowchart of determining the dynamic threshold of an indicator according to the upper limit of the indicator threshold and the lower limit of the indicator threshold according to an exemplary embodiment of the present disclosure is schematically shown. In step S410, the indicator normal distribution of the business-related indicator is determined. For a single business-related indicator, after calculating the upper limit of the indicator threshold and the lower limit of the indicator threshold of the indicator, the indicator grouping interval of the business-related indicator can be further determined. Specifically, according to the upper limit of the indicator threshold and the lower limit of the indicator threshold, the upper and lower limit intervals of the indicator threshold of the business-related indicator are equally divided into x groups, that is, the number of threshold groups is determined, and then the corresponding indicator grouping interval is determined according to the number of threshold groups, such as the indicator grouping interval d=(T1-T0) / (x-1); wherein T1 can represent the upper limit of the indicator threshold, T0 can represent the lower limit of the indicator threshold, and x can represent the number of threshold groups. After the indicator grouping interval is calculated, the normal distribution of the indicator is obtained.

[0080] In step S420, the indicator dynamic threshold is determined according to the inverse function of the cumulative distribution function of the normal distribution of the indicator. The inverse function of the business-related indicator is used to calculate the indicator dynamic threshold T of the business-related indicator. For business forms whose indicator values ​​are lower than the indicator dynamic threshold T, a poor quality label is added to obtain the corresponding classification label. Similarly, all business-related indicators are marked according to the above method to obtain a historical business form with a label. Through the above processing steps, the business form can be marked according to the indicator dynamic threshold to complete the classification processing for the business form.

[0081] In an exemplary embodiment of the present disclosure, the method further comprises: using an association mining algorithm to perform secondary inspection and optimization processing on the indicator dynamic threshold to obtain an optimized indicator dynamic threshold.

[0082] Among them, the secondary inspection and optimization processing can be a process of adjusting and optimizing the dynamic threshold of the indicator.

[0083] Continue to refer Figure 2 For historical business forms with classification labels, the indicator association rules of dimensions such as cells, users, and slices are further mined through an improved association rule algorithm (such as the Apriori algorithm) to verify the rationality of the indicator dynamic threshold. For the determined indicator dynamic threshold, the threshold adjustment module 232 can use an association mining algorithm to perform secondary inspection and optimization processing on the indicator dynamic threshold to obtain an optimized indicator dynamic threshold. Through the above processing steps, the indicator threshold that meets the association rule can be obtained.

[0084] In an exemplary embodiment of the present disclosure, an association mining algorithm is used to perform secondary inspection and optimization processing on the dynamic threshold of the indicator to obtain an optimized dynamic threshold of the indicator, including: classifying historical business forms according to business types to obtain frequent 1-item sets; determining the minimum support threshold and the minimum confidence threshold corresponding to each classified business indicator; and determining whether to trigger a threshold update operation on the dynamic threshold of the indicator based on the frequent 1-item set, the minimum support threshold and the minimum confidence threshold to obtain an optimized dynamic threshold of the indicator.

[0085] The business category may be the category of all specific businesses involved in the historical business form. The minimum support threshold may be the minimum support value corresponding to the frequent item set. The minimum confidence threshold may be the minimum confidence value of the association rule contained in the frequent item set. The threshold update operation may be an automated operation for dynamically updating the dynamic threshold of the optimization indicator.

[0086] For a historical business form data set with a total number of business forms N, the historical business forms can be classified according to the business type to obtain frequent 1-item sets. For example, the indicators are divided into A, B, C, D, ... categories according to the business type, and the indicator characteristics of each type of business are represented by "business category + number", such as the indicators of Class A business are represented by Ai (i = 1, 2, 3, ...). Scan the data set to obtain all the indicators that have appeared, classify the indicators according to the business, and use the business single category containing multiple business indicators as a candidate frequent 1-item set, such as the frequent 1-item set of Class A business is {A} = {A1, A2, A3, ...}.

[0087] An association rule is an implication of the form A=>B, where A and B are subsets of all item sets and are not empty, and A intersects B. The support of an association rule is defined as: support(A=>B)=P(A∪B). The confidence of an association rule is defined as: For each classified business indicator obtained by the division, the minimum support threshold and the minimum confidence threshold corresponding to each classified business indicator can be determined. The minimum support threshold can be marked as Ts, and the minimum confidence threshold can be marked as Tc.

[0088] Then, according to the determined frequent 1-item set, and the minimum support threshold and minimum confidence threshold corresponding to the frequent 1-item set, it is determined whether to trigger the threshold update operation of the indicator dynamic threshold to obtain the optimized indicator dynamic threshold. The improved Apriori algorithm is mainly reflected in the overall consideration of frequent item sets according to the business dimension, without considering a single indicator, which greatly reduces the overhead caused by permutations and combinations and scanning data sets, and improves the efficiency of calculation while ensuring the accurate positioning of frequent item sets.

[0089] In an exemplary embodiment of the present disclosure, based on the frequent 1-item set, the minimum support threshold and the minimum confidence threshold, it is determined whether to trigger a threshold update operation on the indicator dynamic threshold to obtain an optimized indicator dynamic threshold, including: performing a frequent item set determination operation based on the frequent 1-item set to obtain all frequent k-item sets in the historical business form; calculating association rules based on all frequent k-item sets to obtain indicator confidence; when the indicator confidence is greater than or equal to the minimum confidence threshold, using the indicator dynamic threshold as the optimized indicator dynamic threshold; when the indicator confidence is less than the minimum confidence threshold, triggering an update operation on the indicator dynamic threshold to obtain the optimized indicator dynamic threshold.

[0090] The association rule may be a specific rule that indicates an association relationship between frequent itemsets, and the indicator confidence may be the confidence of the association rule between different indicators.

[0091] Based on the frequent 1-item set, the frequent item set determination operation is performed to obtain all the frequent k-item sets in the historical business form, where the value of k can be 2, 3, 4, …, N. All association rules are calculated from the frequent k-item set, and the indicator confidence is calculated. If the indicator confidence is greater than or equal to the minimum confidence threshold Tc, the indicator threshold of the items contained in the frequent item set is reasonable, and the indicator dynamic threshold can be used as the optimized indicator dynamic threshold. If the indicator confidence is less than the minimum confidence threshold Tc, the currently calculated indicator dynamic threshold is considered unreasonable, and the update operation of the indicator dynamic threshold is triggered until the updated indicator dynamic threshold meets the conditions and the optimized indicator dynamic threshold is obtained. In the process of optimizing the indicator dynamic threshold, the association rules between multiple business-related indicators are taken into account, so that the calculated optimized indicator dynamic threshold is more reasonable.

[0092] In an exemplary embodiment of the present disclosure, a frequent item set determination operation is performed based on a frequent 1-item set to obtain all frequent k-item sets in a historical business form, including: scanning the frequent 1-item set, determining the frequent k-item set and the support of each candidate frequent k-item set based on the scanning result of the frequent 1-item set; determining the frequent k-item set according to the minimum support among the supports of multiple frequent k-item sets; and repeatedly performing the above steps until the number of determined frequent k+j item sets is less than or equal to 1, thereby obtaining all frequent k-item sets.

[0093] Assume that the set itemset = {item_1, item_2, ..., item_m} is the set of all items, where item_k (k = 1, 2, ..., m) is called an item, the set of items is called an itemset, and the itemset containing k items is called a k-itemset. Frequent itemsets can be all itemsets that meet the minimum support. The properties of frequent itemsets include: (1) all non-empty subsets of frequent itemsets are also frequent itemsets; (2) if item set A is not a frequent item set, then other itemsets or the union of transactions and item set A are not frequent itemsets. Frequent k-itemsets can be all k-itemsets that meet the minimum support. After scanning the entire business form data set classified by business type, all form data that have appeared and contain a single business type are regarded as candidate frequent 1-itemsets, and then frequent k-itemsets are mined based on the candidate frequent 1-itemsets.

[0094] The specific process of mining frequent k-itemsets based on frequent 1-itemsets is as follows: Scan the data and calculate the support of candidate frequent k-itemsets. Consider that all subsets of frequent k-itemsets are also frequent itemsets, apply pruning rules, and obtain all candidate frequent k-itemsets. Scan the data set for candidate frequent k-itemsets to obtain the support of all candidate frequent k-itemsets, and calculate the minimum support threshold Ts based on the support of all candidate frequent k-itemsets. According to the minimum support threshold Ts, the frequent k+1 itemset is obtained. Specifically, the candidate frequent k-itemsets whose support is less than the minimum support threshold Ts are deleted to obtain the frequent k+1 itemset.

[0095] Scan the data set, obtain each item, generate the candidate 1-item set C1, then count each item, delete the unsatisfied items from C1 according to the minimum support, and obtain the frequent 1-item set L1; execute the pruning strategy on the set generated by the self-connection of L1 to generate the candidate 2-item set C2, then scan all transactions and count each item in the set C2. Similarly, delete the unsatisfied items from C2 according to the minimum support, and obtain the frequent 2-item set L2. And so on, to obtain the frequent k-item set.

[0096] Repeat the above steps until the number of frequent item sets generated in the k+j layer is 0 or 1, thereby obtaining all frequent k-item sets, where the value of k can be 2, 3, 4, ..., N; the value of j can be 0, 1, 2, 3, 4, ..., N. Through the above processing steps, all frequent k-item sets contained in the historical business form can be obtained, so that the determination process of the dynamic threshold of the indicator takes into account the association rules between multiple indicators and is more reasonable.

[0097] In an exemplary embodiment of the present disclosure, a real-time business form is used to perform model update processing on a threshold determination model to obtain an updated threshold determination model. The updated threshold determination model is used to adjust the dynamic threshold of the indicator. The real-time business form is a business form obtained by periodic sampling based on the time dimension.

[0098] The updated threshold determination model may be a model obtained by optimizing and updating the threshold determination model using a real-time business form as model training data.

[0099] Continue to refer Figure 2 In the process of network operation, new business forms will be continuously generated. The data acquisition module 210 can perform periodic data sampling operations based on the time dimension, such as performing data sampling according to a pre-configured time period to obtain a real-time business form. The time period can be configured as 1 week, one month or three months. After obtaining the real-time business form, the real-time business form is used as model training data for the threshold determination model. The threshold adjustment module 231 can further perform model update processing on the threshold determination model based on the real-time business form to obtain an updated threshold determination model.

[0100] After obtaining the updated threshold determination model, the updated threshold determination model is used to adjust the indicator dynamic threshold of the business-related indicator, and the form classification module 240 classifies the pending business form according to the adjusted indicator dynamic threshold, and determines the classification label corresponding to the pending business form. The threshold determination model is updated using the real-time business form, and the indicator threshold determination operation is performed using the updated threshold determination model, so that the determined indicator dynamic threshold can be more accurate and reasonable.

[0101] In summary, the indicator threshold determination method disclosed in the present invention determines the indicator dynamic threshold, and the indicator dynamic threshold is used to screen the target business form from the business form to be processed; the indicator dynamic threshold is generated based on the threshold determination model, and the threshold determination model is trained based on one or more dimensions of the time dimension, geographical dimension, user dimension, network dimension, and business dimension of the historical business form, and is dynamically updated based on the real-time business form. On the one hand, the threshold determination model is obtained based on the historical business form training, and the threshold determination model can adapt to multiple dimensions in the historical business form to obtain the indicator dynamic thresholds of various indicators, thereby improving the network operation capability. On the other hand, since the threshold generation model can be dynamically updated according to the real-time business form, the indicator dynamic threshold can be automatically adjusted, reducing manual intervention, improving work efficiency, and reducing operation and maintenance costs. On the other hand, the threshold determination model adopts the n*σ criterion to flexibly respond to the differential changes of different indicator thresholds, so that the generated threshold is more reasonable and effective. On the other hand, when the indicator dynamic threshold is optimized twice, the frequent item sets are comprehensively considered according to the business dimension, without considering a single indicator, which greatly reduces the overhead caused by the permutation and combination and scanning of the data set, and improves the efficiency of calculation while ensuring the accurate positioning of the frequent item sets.

[0102] It should be noted that, although the steps of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0103] In addition, in this exemplary embodiment, a device for determining an indicator threshold is also provided. Figure 5 , the indicator threshold determination device 500 may include: an indicator threshold determination module 510.

[0104] Specifically, the indicator threshold determination module 510 is used to determine the dynamic threshold of the indicator. The dynamic threshold of the indicator is used to screen the target business form from the business forms to be processed. The dynamic threshold of the indicator is generated based on the threshold determination model. The threshold determination model is trained based on one or more dimensions of the time dimension, geographic dimension, user dimension, network dimension, and business dimension of the historical business form, and is dynamically updated based on the real-time business form.

[0105] In an exemplary embodiment of the present disclosure, the indicator threshold determination device 500 also includes an initial historical form collection module, which is used to: collect network performance data, service quality data, configuration data and business operation data from the network through deep packet detection probes and production systems to generate initial historical business forms; the production system includes one or more of a network management system, a customer relationship management system, a work order system, and a customer service system.

[0106] In an exemplary embodiment of the present disclosure, the indicator threshold determination device 500 further includes a historical form generation module, which is used to: review the initial historical business form to obtain a historical business form.

[0107] In an exemplary embodiment of the present disclosure, the indicator threshold determination device 500 also includes a threshold model generation module, which is used to: use historical business forms with classification labels to perform model training on a pre-built initial model to obtain a threshold determination model, and the threshold determination model uses probability density analysis and n*σ criterion to determine the indicator dynamic threshold of each business-related indicator in the historical business form, and the indicator dynamic threshold is used to determine the classification label of the business form to be processed.

[0108] In an exemplary embodiment of the present disclosure, the threshold model generation module includes a threshold determination unit, which is used to: adopt the n*σ criterion to determine the indicator threshold upper limit and the indicator threshold lower limit of the business-related indicator; determine the indicator dynamic threshold according to the indicator threshold upper limit and the indicator threshold lower limit; the indicator threshold upper limit and the indicator threshold lower limit are determined based on the indicator probability distribution of the business-related indicator, and the indicator probability distribution is determined based on the indicator mean, indicator variance and probability density function of the business-related indicator.

[0109] In an exemplary embodiment of the present disclosure, the threshold determination unit includes a threshold determination sub-unit, which is used to: determine the normal distribution of indicators of business-related indicators, the normal distribution of indicators is determined based on the upper limit of the indicator threshold, the lower limit of the indicator threshold and the indicator grouping interval of the business-related indicators, the indicator grouping interval is determined according to the upper limit of the indicator threshold, the lower limit of the indicator threshold and the number of threshold groups; determine the dynamic threshold of the indicator according to the inverse function of the cumulative distribution function of the normal distribution of the indicator.

[0110] In an exemplary embodiment of the present disclosure, the indicator threshold determination device 500 also includes a threshold update module, which is used to: use an association mining algorithm to perform secondary inspection and optimization processing on the indicator dynamic threshold to obtain an optimized indicator dynamic threshold.

[0111] In an exemplary embodiment of the present disclosure, the threshold update module includes a threshold update unit, which is used to: classify historical business forms according to business types to obtain frequent 1-item sets; determine the minimum support threshold and the minimum confidence threshold corresponding to each classified business indicator; and determine whether to trigger a threshold update operation on the indicator dynamic threshold based on the frequent 1-item set, the minimum support threshold and the minimum confidence threshold to obtain an optimized indicator dynamic threshold.

[0112] In an exemplary embodiment of the present disclosure, the threshold updating unit includes a threshold updating subunit, which is used to: perform a frequent item set determination operation based on a frequent 1-item set to obtain all frequent k-item sets in a historical business form; calculate association rules based on all frequent k-item sets to obtain an indicator confidence; when the indicator confidence is greater than or equal to a minimum confidence threshold, use the indicator dynamic threshold as an optimized indicator dynamic threshold; when the indicator confidence is less than the minimum confidence threshold, trigger an update operation of executing the indicator dynamic threshold to obtain an optimized indicator dynamic threshold.

[0113] In an exemplary embodiment of the present disclosure, the threshold updating subunit includes a frequent item set determination subunit, which is used to: scan the frequent 1-item set, determine the frequent k-item set and the support of each frequent k-item set based on the scanning result of the frequent 1-item set; determine the frequent k-item set according to the minimum support among the supports of multiple frequent k-item sets; repeat the above steps until the number of determined frequent k+j item sets is less than or equal to 1, and all frequent k-item sets are obtained.

[0114] In an exemplary embodiment of the present disclosure, the indicator threshold determination device 500 also includes a model update module, which is used to: use a real-time business form to perform model update processing on the threshold determination model to obtain an updated threshold determination model, and the updated threshold determination model is used to adjust the dynamic threshold of the indicator. The real-time business form is a business form obtained by periodic sampling based on the time dimension.

[0115] The specific details of the virtual modules of each of the above-mentioned indicator threshold determination devices have been described in detail in the corresponding indicator threshold determination method, so they will not be repeated here.

[0116] It should be noted that, although several modules or units of the indicator threshold determination device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0117] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0118] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0119] Reference below Figure 6 hereinafter describes an electronic device 600 according to such an embodiment of the present disclosure. Figure 6 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0120] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), and a display unit 640.

[0121] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0122] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 621 and / or a cache memory unit 622 , and may further include a read-only memory unit (ROM) 623 .

[0123] The storage unit 620 may include a program / utility 624 having a set (at least one) of program modules 625, such program modules 625 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0124] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0125] The electronic device 600 may also communicate with one or more external devices 670 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0126] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0127] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0128] refer to Figure 7 As shown, a program product 700 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0129] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0130] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0131] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0132] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0133] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0134] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0135] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for determining an indicator threshold, characterized in that: include: Determine a dynamic threshold of an indicator, wherein the dynamic threshold of the indicator is used to screen a target business form from the business forms to be processed; The dynamic threshold of the indicator is generated based on a threshold determination model, and the threshold determination model is trained based on one or more dimensions of the time dimension, geographical dimension, user dimension, network dimension, and business dimension of the historical business form, and is dynamically updated based on the real-time business form.

2. The method according to claim 1, characterized in that The method further comprises: Through deep packet inspection probes and production systems, network performance data, service quality data, configuration data and business operation data are collected from the network to generate an initial historical business form; the production system includes one or more of a network management system, a customer relationship management system, a work order system, and a customer service system.

3. The method according to claim 2, characterized in that After generating the initial historical business form, the method further includes: The initial historical business form is reviewed to obtain the historical business form.

4. The method according to claim 1, characterized in that The threshold determination model is generated by the following steps: The historical business form with classification labels is used to train the pre-built initial model to obtain the threshold determination model. The threshold determination model uses probability density analysis and n*σ criterion to determine the indicator dynamic threshold of each business-related indicator in the historical business form. The indicator dynamic threshold is used to determine the classification label of the business form to be processed.

5. The method according to claim 4, characterized in that The method of using probability density analysis and n*σ criterion to determine the dynamic threshold of each business-related indicator in the historical business form includes: Adopting the n*σ criterion, determining the upper limit and the lower limit of the indicator threshold of the business-related indicator; Determining the indicator dynamic threshold according to the indicator threshold upper limit and the indicator threshold lower limit; The indicator threshold upper limit and the indicator threshold lower limit are determined based on the indicator probability distribution of the business-related indicator, and the indicator probability distribution is determined based on the indicator mean, indicator variance and probability density function of the business-related indicator.

6. The method according to claim 5, characterized in that The determining the indicator dynamic threshold according to the indicator threshold upper limit and the indicator threshold lower limit includes: Determine an indicator normal distribution of the business-related indicator, wherein the indicator normal distribution is determined based on the indicator threshold upper limit, the indicator threshold lower limit, and the indicator grouping interval of the business-related indicator, and the indicator grouping interval is determined according to the indicator threshold upper limit, the indicator threshold lower limit, and the number of threshold groups; The indicator dynamic threshold is determined according to the inverse function of the cumulative distribution function of the normal distribution of the indicator.

7. The method according to claim 1, characterized in that The method further comprises: The association mining algorithm is used to perform secondary inspection and optimization processing on the indicator dynamic threshold to obtain the optimized indicator dynamic threshold.

8. The method according to claim 7, characterized in that The adopting of the association mining algorithm to perform secondary inspection and optimization processing on the indicator dynamic threshold to obtain the optimized indicator dynamic threshold includes: Classify the historical business forms according to business types to obtain frequent 1-item sets; Determine the minimum support threshold and the minimum confidence threshold corresponding to each of the classification business indicators; According to the frequent 1-item set, the minimum support threshold and the minimum confidence threshold, it is determined whether to trigger a threshold update operation on the indicator dynamic threshold to obtain the optimized indicator dynamic threshold.

9. The method according to claim 8, characterized in that The determining whether to trigger a threshold update operation on the indicator dynamic threshold according to the frequent 1-item set, the minimum support threshold and the minimum confidence threshold to obtain the optimized indicator dynamic threshold includes: Performing a frequent item set determination operation based on the frequent 1-item set to obtain all frequent k-item sets in the historical business form; Calculate association rules based on all the frequent k-item sets to obtain indicator confidence; When the indicator confidence is greater than or equal to the minimum confidence threshold, using the indicator dynamic threshold as the optimized indicator dynamic threshold; When the indicator confidence is less than the minimum confidence threshold, an update operation of the indicator dynamic threshold is triggered to obtain the optimized indicator dynamic threshold.

10. The method according to claim 9, characterized in that The performing of a frequent item set determination operation based on the frequent 1 item set to obtain all frequent k item sets in the historical business form includes: Scan the frequent 1-itemset, and determine the frequent k-itemset and the support of each of the frequent k-itemset based on the scanning result of the frequent 1-itemset; Determining a frequent k-item set according to a minimum support among the supports of the multiple frequent k-item sets; The above steps are repeated until the number of determined frequent k+j item sets is less than or equal to 1, and all the frequent k item sets are obtained.

11. The method according to any one of claims 1 to 10, characterized in that: The method further comprises: The real-time business form is used to perform model update processing on the threshold determination model to obtain an updated threshold determination model, and the updated threshold determination model is used to adjust the dynamic threshold of the indicator. The real-time business form is a business form obtained by periodic sampling based on the time dimension.

12. A device for determining an indicator threshold, characterized in that: include: The indicator threshold determination module is used to determine the dynamic threshold of the indicator, and the dynamic threshold of the indicator is used to screen the target business form from the business forms to be processed. The dynamic threshold of the indicator is generated based on the threshold determination model, and the threshold determination model is trained based on one or more dimensions of the time dimension, geographical dimension, user dimension, network dimension, and business dimension of the historical business form, and is dynamically updated based on the real-time business form.

13. An electronic device, characterized in that: include: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the indicator threshold determination method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for determining an indicator threshold value according to any one of claims 1 to 11 is implemented.

Images