A method, device and readable storage medium for user complaint early warning

Abstract

The application provides a user complaint early warning method, device and readable storage medium, the method comprises the following steps: obtaining feature data of a user to be predicted; inputting the feature data of the user to be predicted into trained 1 plus N complaint prediction models respectively, and outputting a complaint probability predicted by each complaint prediction model; wherein the 1 plus N complaint prediction models comprise one Trace data complaint prediction model constructed based on wireless signaling Trace data key indicators, and N application service complaint prediction models jointly constructed based on the Trace data key indicators and N terminal application corresponding performance perception rating indicator data; calculating a complaint expectation of the user to be predicted according to the predicted complaint probability; if the complaint expectation of the user to be predicted is greater than or equal to a preset tolerance threshold, user complaint early warning is performed. The method, device and readable storage medium can solve the problem that the existing user complaint early warning method is prone to insufficient accuracy of user complaint early warning.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a method, apparatus, and readable storage medium for user complaint early warning. Background Technology

[0002] In mobile networks, user complaint early warning and handling have always been a focus for telecommunications operators, who have consistently strived to reduce complaints. User complaints arise when an operator's network equipment malfunctions or the network quality fails to meet specific user service needs and exceeds their tolerance threshold. Complaint early warning aims to identify users with poor experience (who are likely to complain) in advance, providing reassurance before network optimization is complete, thus preventing complaints, reducing the number of complaints, and maintaining user loyalty.

[0003] However, existing user complaint early warning methods typically use complaint prediction models that differentiate between business categories (e.g., one model for games, another for videos). Due to the wide variety of terminal services and applications, the varying network performance requirements of different applications (e.g., data services such as different games, short videos, and video-on-demand), and the unique characteristics of users with different preferences for application types, usage frequencies, and sensitivity levels, the accuracy of user complaint early warnings is often insufficient. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art by providing a method, apparatus and readable storage medium for user complaint early warning, so as to solve the problem that the existing user complaint early warning methods are prone to insufficient accuracy in user complaint early warning.

[0005] In a first aspect, the present invention provides a method for user complaint early warning, the method

[0006] include:

[0007] Obtain the feature data of the user to be predicted;

[0008] The feature data of the user to be predicted is input into 1 plus N complaint prediction models, and the complaint probability predicted by each complaint prediction model is output. The 1 plus N complaint prediction models include 1 Trace data complaint prediction model based on key indicators of wireless signaling Trace data, and N application service complaint prediction models jointly constructed based on the key indicators of Trace data and the performance perception rating indicators corresponding to N terminal applications, where N is greater than or equal to 1.

[0009] Calculate the expected complaint rate of the user to be predicted based on the predicted complaint probability;

[0010] If the expected complaint from the user to be predicted is greater than or equal to the preset tolerance threshold, a user complaint warning will be issued.

[0011] Furthermore, before obtaining the feature data of the user to be predicted, the method further includes:

[0012] Acquire trace data, deep packet inspection (DPI) signaling details (XDR) data, terminal application OTT data, and user complaint information within the local network area;

[0013] Based on the Trace data, XDR data, OTT data, and user complaint information, 1 plus N feature datasets are obtained;

[0014] The 1+N feature datasets are used to train 1+N pre-set complaint prediction models respectively, and the complaint probability output by each complaint prediction model is obtained;

[0015] The complaint probabilities output by the 1+N complaint prediction models are fused to obtain the user's expected complaint value Y.

[0016] The Y values ​​of positive samples in the feature dataset are sorted in ascending order to obtain the tolerance threshold for user complaints after comprehensive evaluation.

[0017] Furthermore, the step of obtaining 1 plus N feature datasets based on the Trace data, XDR data, OTT data, and user complaint information specifically includes:

[0018] Obtain key metrics of the Trace data at each time granularity within a preset time window;

[0019] The user identifier corresponding to the Trace data is obtained by associating the preset association fields in the XDR data with the Trace data;

[0020] Obtain the key indicators corresponding to N terminal applications in the OTT data at each time granularity within a preset time window;

[0021] Based on the key indicators corresponding to the N terminal applications, performance perception rating index data is generated for each terminal application. The performance perception rating index data includes the duration and percentage corresponding to multiple performance perception rating levels.

[0022] Based on the user identifier, the key indicators of the Trace data and the performance perception rating indicators of N terminal applications are correlated on the preset time window to obtain the feature data of each application business complaint prediction model.

[0023] Based on the complainant's mobile phone number and historical complaint time period in the user complaint information, and by associating them with the feature data of each application service complaint prediction model, N feature datasets are obtained from the 1 plus N feature datasets; and...

[0024] Based on the complainant's mobile phone number, historical complaint time period, and key indicators associated with the Trace data from the user's identifier in the user complaint information, one feature dataset is obtained from the 1 plus N feature datasets.

[0025] Furthermore, the step of generating performance-aware rating index data for each of the N terminal applications based on their respective key indicators specifically includes:

[0026] The key indicators corresponding to each terminal application at each time granularity are scored using an interval scoring method.

[0027] Based on the scores of each key indicator for each terminal application at each time granularity, the comprehensive perception score for each terminal application at each time granularity is obtained.

[0028] Based on the comprehensive perception score of each terminal application at each time granularity, a rating is generated on the preset time window to produce performance perception rating index data corresponding to each terminal application.

[0029] Furthermore, the overall perception score for each terminal application is calculated according to the following formula:

[0030]

[0031] In the formula, W i The importance weighting factor for the i-th key indicator of the terminal application. Let be the score of the i-th key indicator of the terminal application, and n be the total number of key indicators of the terminal application.

[0032] Furthermore, the performance perception rating level includes four levels: excellent, good, average, and poor.

[0033] The performance perception rating index data includes the duration of excellent rating, good rating, average rating, poor rating, percentage of excellent rating, percentage of good rating, percentage of average rating, percentage of poor rating, monthly average daily usage frequency, and monthly average daily usage duration.

[0034] Furthermore, the process of fusing the complaint probabilities output by the 1+N complaint prediction models to obtain the user's expected complaint value Y specifically includes:

[0035] Calculate the impact factor corresponding to each of the complaint prediction models;

[0036] The user's expected complaint value Y is calculated based on the influencing factors corresponding to each complaint prediction model and the output complaint probability.

[0037] Furthermore, the calculation of the impact factor corresponding to each of the complaint prediction models is specifically performed according to the following formula:

[0038]

[0039] In the formula, δ i This represents the influence factor of the i-th complaint prediction model. When i = 0, The duration of the user's connection state within a preset time window, when i≠0. The usage duration of the i-th terminal application within the preset time window;

[0040] The user's expected complaint value Y is calculated based on the influencing factor and the output complaint probability corresponding to each complaint prediction model, specifically according to the following formula:

[0041]

[0042] In the formula, Y represents the expectation of the complaint, y i This represents the complaint probabilities output by 1 plus N complaint prediction models.

[0043] Furthermore, before obtaining the feature data of the user to be predicted, the method further includes:

[0044] The user to be predicted is obtained from the operator's B domain data;

[0045] The method further includes, simultaneously with or after issuing user complaint warnings:

[0046] The users to be predicted were reassured.

[0047] Secondly, the present invention provides a device for user complaint early warning, comprising:

[0048] The feature data acquisition module is used to acquire feature data of the user to be predicted.

[0049] The complaint probability acquisition module, connected to the feature data acquisition module, is used to input the feature data of the user to be predicted into 1 plus N pre-trained complaint prediction models and output the complaint probability predicted by each complaint prediction model; wherein, the 1 plus N complaint prediction models include 1 Trace data complaint prediction model constructed based on key indicators of wireless signaling Trace data, and N application service complaint prediction models jointly constructed based on the key indicators of the Trace data and the performance perception rating indicators corresponding to N terminal applications, wherein N is greater than or equal to 1;

[0050] The complaint expectation calculation module, connected to the complaint probability acquisition module, is used to calculate the complaint expectation of the user to be predicted based on the predicted complaint probability.

[0051] The complaint warning module, connected to the complaint expectation calculation module, is used to issue a user complaint warning if the complaint expectation of the user to be predicted is greater than or equal to a preset tolerance threshold.

[0052] Thirdly, the present invention provides a device for user complaint early warning, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the user complaint early warning method described in the first aspect above.

[0053] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the user complaint early warning method described in the first aspect.

[0054] This invention provides a method, apparatus, and readable storage medium for user complaint early warning. First, feature data of the user to be predicted is acquired. Then, the feature data of the user to be predicted is input into 1 plus N pre-trained complaint prediction models, and the complaint probability predicted by each model is output. The 1 plus N complaint prediction models include one Trace data complaint prediction model constructed based on key indicators of wireless signaling Trace data, and N application service complaint prediction models jointly constructed based on the key indicators of the Trace data and N performance perception rating indicators corresponding to terminal applications, where N is greater than or equal to 1. Next, the complaint expectation of the user to be predicted is calculated based on the predicted complaint probability. If the complaint expectation of the user to be predicted is greater than or equal to a preset tolerance threshold, a user complaint early warning is issued. This invention integrates one plus N user-level complaint prediction models to comprehensively evaluate and determine user complaint expectations. The application business complaint prediction model does not use business category distinctions (such as one model for games, another for videos, etc.), but instead uses the business distinctions provided by specific terminal applications, and considers the usage habits of each user and each application (i.e., performance perception rating index data). This not only improves the accuracy of user complaint warnings, but also reduces the number of user complaints, solving the problem that existing user complaint warning methods are prone to insufficient accuracy. Attached Figure Description

[0055] Figure 1 This is a flowchart of a user complaint early warning method according to Embodiment 1 of the present invention;

[0056] Figure 2 The process for obtaining the tolerance threshold in this embodiment of the invention;

[0057] Figure 3 This is a schematic diagram of a user complaint early warning device according to Embodiment 2 of the present invention;

[0058] Figure 4 This is a schematic diagram of a user complaint early warning device according to Embodiment 3 of the present invention. Detailed Implementation

[0059] To enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0060] It is understood that the specific embodiments and accompanying drawings described herein are merely for explaining the invention and are not intended to limit the invention.

[0061] It is understood that, without conflict, the various embodiments and features in the embodiments of the present invention can be combined with each other.

[0062] It is understood that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, while the parts unrelated to the present invention are not shown in the drawings.

[0063] It is understood that each unit or module involved in the embodiments of the present invention may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.

[0064] It is understood that the terms "first," "second," etc., in the embodiments of the present invention are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0065] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this invention may occur in a different order than that marked in the accompanying drawings.

[0066] It is understood that the flowcharts and block diagrams of this invention illustrate the possible architecture, functions, and operations of systems, apparatuses, devices, and methods according to various embodiments of this invention. Each block in the flowchart or block diagram may represent a unit, module, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagram and flowchart can be implemented using a hardware-based system to achieve the specified function, or using a combination of hardware and computer instructions.

[0067] It is understood that the units and modules involved in the embodiments of the present invention can be implemented by software or by hardware. For example, the units and modules can be located in a processor.

[0068] Example 1:

[0069] This embodiment provides a method for user complaint early warning, such as... Figure 1 As shown, the method includes:

[0070] Step S101: Obtain the feature data of the user to be predicted.

[0071] In this embodiment, the feature data of the user to be predicted includes key indicators of wireless signaling trace data and performance perception rating indicators corresponding to N terminal applications.

[0072] Step S102: Input the feature data of the user to be predicted into the trained 1 plus N complaint prediction models respectively, and output the complaint probability predicted by each complaint prediction model; wherein, the 1 plus N complaint prediction models include 1 Trace data complaint prediction model constructed based on key indicators of wireless signaling Trace data, and N application service complaint prediction models jointly constructed based on the key indicators of the Trace data and the performance perception rating indicators corresponding to N terminal applications, wherein N is greater than or equal to 1;

[0073] Step S103: Calculate the expected complaint rate of the user to be predicted based on the predicted complaint probability;

[0074] Step S104: If the expected complaint from the user to be predicted is greater than or equal to the preset tolerance threshold, then issue a user complaint warning.

[0075] This invention integrates 1+N user-level complaint prediction models to comprehensively evaluate and determine users' complaint expectations, thereby improving the accuracy of user complaint early warning and reducing the number of user complaints.

[0076] Optionally, before obtaining the feature data of the user to be predicted, the method further includes:

[0077] Acquire trace data, DPI (Deep Packet Inspection) signaling details (XDR) data, terminal application OTT (Over The Top) data, and user complaint information within the local network area;

[0078] Based on the Trace data, XDR data, OTT data, and user complaint information, 1 plus N feature datasets are obtained;

[0079] The 1+N feature datasets are used to train 1+N pre-set complaint prediction models respectively, and the complaint probability output by each complaint prediction model is obtained;

[0080] The complaint probabilities output by the 1+N complaint prediction models are fused to obtain the user's expected complaint value Y.

[0081] The Y values ​​of positive samples in the feature dataset are sorted in ascending order to obtain the tolerance threshold for user complaints after comprehensive evaluation.

[0082] In this embodiment, by selecting key indicators from different data sources, and aggregating, combining, and associating them at the user level, business level, and time level, one plus N feature datasets are constructed.

[0083] Optionally, obtaining 1 plus N feature datasets based on the Trace data, XDR data, OTT data, and user complaint information specifically includes:

[0084] Obtain key metrics of the Trace data at each time granularity within a preset time window;

[0085] The user identifier corresponding to the Trace data is obtained by associating the preset association fields in the XDR data with the Trace data;

[0086] Obtain the key indicators corresponding to N terminal applications in the OTT data at each time granularity within a preset time window;

[0087] Based on the key indicators corresponding to the N terminal applications, performance perception rating index data is generated for each terminal application. The performance perception rating index data includes the duration and percentage corresponding to multiple performance perception rating levels.

[0088] Based on the user identifier, the key indicators of the Trace data and the performance perception rating indicators of N terminal applications are correlated on the preset time window to obtain the feature data of each application business complaint prediction model.

[0089] Based on the complainant's mobile phone number and historical complaint time period in the user complaint information, and by associating them with the feature data of each application service complaint prediction model, N feature datasets are obtained from the 1 plus N feature datasets; and...

[0090] Based on the complainant's mobile phone number, historical complaint time period, and key indicators associated with the Trace data from the user's identifier in the user complaint information, one feature dataset is obtained from the 1 plus N feature datasets.

[0091] In this embodiment, wireless signaling (Trace) data can be subscribed to by the operator's network management system and periodically (e.g., at 15-minute intervals) obtained by the base station. After parsing by the parsing server, a Trace indicator report is generated, and key indicator fields (fields are optional) of the Trace data are extracted, as shown in Table 1, which provides an example of key indicators selected for 4G LTE Trace data.

[0092] Table 1: Examples of Key Metrics for Trace Data

[0093]

[0094] In this embodiment, the DPI signaling detail record (XDR) data can be provided by the DPI signaling acquisition system. The data can be associated with key fields in the Trace data to attach user identifiers (such as IMSI, International Mobile Subscriber Identity, MSISDN mobile phone number) and timestamps to these key fields. The associated fields are shown in Table 2. Specifically, the associated fields for 4G radio-side Trace data and XDR include: MME Group ID (Mobility Management Entity Group Identifier), MME Code (Mobility Management Entity Code), MME UE S1AP ID (Mobility Management Entity and User S1 Interface Identifier), and Timestamp. The associated fields for 5G radio-side Trace data and XDR include: AMF Region ID (Access and Mobility Management Entity Region Identifier), AMF Set ID (Access and Mobility Management Entity Set Identifier), AMF UE NGAP ID (Access and Mobility Management Entity and User NG Interface Identifier), and Timestamp.

[0095] Table 2: Correlation Fields Between 4G / 5G Wireless Side Trace Data and XDR

[0096] 4G field 5G field IMSI, MSISDN IMSI, MSISDN MME Group ID (Associated) AMF Region ID (Association) MME Code (Association) AMF Set ID (Association) MME UE S1AP ID (Associated) AMF UE NGAP ID (Associated) Time Stamp (related) Time Stamp (related)

[0097] In this embodiment, the terminal application (OTT) data is provided by internet service providers (such as Tencent, Alibaba, Baidu, Xiaomi, etc.) and can be divided into four types of applications: browsers, videos, games, and instant messaging. The key performance indicators (KPIs) for each type of terminal application are independent. For example, different games have different performance requirements and user preferences. This invention does not use the average KPI data for all game-related services.

[0098] Specifically, the key metrics corresponding to the terminal application can be shown in Tables 3 to 6. The granularity of the start and end times can be selected (e.g., 15-minute granularity), and the usage duration represents the total time the application is used within the start and end times.

[0099] Table 3: Examples of Key Metrics for Browser Applications

[0100]

[0101]

[0102] Table 4: Examples of Key Metrics for Video Applications

[0103]

[0104] Table 5: Examples of 3 Key Metrics for Game Applications

[0105]

[0106] Table 6: Examples of 4 Key Metrics for Instant Messaging Applications

[0107]

[0108]

[0109] Optionally, generating performance-aware rating index data for each of the N terminal applications based on their respective key indicators specifically includes:

[0110] The key indicators corresponding to each terminal application at each time granularity are scored using an interval scoring method.

[0111] Based on the scores of each key indicator for each terminal application at each time granularity, the comprehensive perception score for each terminal application at each time granularity is obtained.

[0112] Based on the comprehensive perception score of each terminal application at each time granularity, a rating is generated on the preset time window to produce performance perception rating index data corresponding to each terminal application.

[0113] In this embodiment, to improve the model's accuracy, an interval scoring method is used to score the key indicators corresponding to each terminal application at each time granularity (the maximum score can be 100, and the interval step size for each key indicator can be obtained through actual testing). For example, the webpage response success rate score for application 1 can be scored according to the following formula:

[0114]

[0115] Optionally, the overall perception score for each terminal application is calculated according to the following formula:

[0116]

[0117] In the formula, W iLet be the importance weight factor for the i-th key indicator of the terminal application, and let the sum of the importance weight factors of all key indicators be 1. Let be the score of the i-th key indicator of the terminal application, and n be the total number of key indicators of the terminal application.

[0118] Specifically, the performance perception rating level includes four levels: excellent, good, medium, and poor; the performance perception rating index data includes the duration of excellent rating, good rating, medium rating, poor rating, percentage of excellent rating duration, percentage of good rating duration, percentage of medium rating duration, percentage of poor rating duration, monthly average daily usage frequency, and monthly average daily usage duration.

[0119] In this embodiment, the perception of each terminal application is rated based on the comprehensive perception score of each terminal application at each time granularity, and can be divided into four levels: excellent, good, average, and poor. For example, when 90≤F A The rating is excellent; when 80 ≤ F A If the score is <90, the rating is Good; if the score is 60≤F, the rating is Good. A If the score is less than 80, the rating is medium; if the score is less than 80, the rating is medium. A If the score is less than 60, the rating is poor.

[0120] In this embodiment, each terminal application is rated on a time window (e.g., 15 minutes, 30 minutes, 1 hour), generating performance-aware rating index data for each terminal application, as shown in Table 7. Other applications obtain performance-aware rating index data similarly, where the usage duration is the sum of the usage duration at each time granularity within the time window.

[0121] Table 7: Example of performance perception rating metrics for Application 1

[0122]

[0123]

[0124] In this embodiment, key indicators of trace data and performance-aware rating indicators of various terminal applications are combined and correlated to construct a feature dataset. This dataset can be correlated using time start and end dates (e.g., the same 15-minute time interval) and user identifier (IMSI) fields to construct feature data for machine learning models (e.g., neural network models). Table 8 shows an example of the model feature dataset fields for Application 1.

[0125] Table 8: Examples of feature set fields for Application 1

[0126]

[0127]

[0128] Similarly, using the same method, the model feature dataset fields for key indicators of single trace data are constructed (represented by application 0 here), as shown in Table 9. The interval scores and ratings for each indicator are defined by reference to the experience values ​​of routine road network assessments by telecommunications operators and the experience values ​​of local network trace data evaluations.

[0129] Table 9: Examples of Model Feature Set Fields for Key Metrics in Single Trace Data

[0130]

[0131]

[0132] The final model consists of a Trace data complaint prediction model constructed from key indicators of single Trace data, and an application business complaint prediction model constructed from performance perception rating indicators and key indicators of Trace data for N terminal applications. This results in 1+N complaint prediction models, where N depends on the number of applications provided by the OTT application.

[0133] Optionally, fusing the complaint probabilities output by the 1+N complaint prediction models to obtain the user's expected complaint value Y specifically includes:

[0134] Calculate the impact factor corresponding to each of the complaint prediction models;

[0135] The user's expected complaint value Y is calculated based on the influencing factors corresponding to each complaint prediction model and the output complaint probability.

[0136] Specifically, the calculation of the impact factor corresponding to each of the complaint prediction models is performed according to the following formula:

[0137]

[0138] In the formula, δ i This represents the influence factor of the i-th complaint prediction model. When i = 0, The duration of the user's connection state within a preset time window, when i≠0. The usage duration of the i-th terminal application within the preset time window;

[0139] The user's expected complaint value Y is calculated based on the influencing factor and the output complaint probability corresponding to each complaint prediction model, specifically according to the following formula:

[0140]

[0141] In the formula, Y represents the expectation of the complaint, y i This represents the complaint probabilities output by 1 plus N complaint prediction models.

[0142] In this embodiment, the minimum value of Y after ascending order can be taken as the tolerance threshold for user complaints after comprehensive evaluation. It should be noted that the tolerance threshold of existing user complaint early warning methods is usually defined manually or set after considering the average of network performance indicators, which is not refined enough. The tolerance threshold obtained by the present invention can improve the accuracy of user complaint early warning.

[0143] Specifically, the process for obtaining the tolerance threshold is as follows: Figure 2 As shown, the Trace sample dataset includes key metrics of Trace data, the terminal application 1 sample dataset includes key metrics of Trace data and performance-aware rating metrics data corresponding to terminal application 1, the terminal application 2 sample dataset includes key metrics of Trace data and performance-aware rating metrics data corresponding to terminal application 2, and so on.

[0144] In this embodiment, the feature data of the user to be predicted is input into 1+N complaint prediction models. The complaint probability of each model is output, along with the expected complaint rate of the user. This expected rate is compared with a tolerance threshold. If the expected complaint rate is greater than or equal to the tolerance threshold, a user warning is issued. Taking four terminal applications as an example (i.e., N=4), the prediction results of the 1+4 complaint prediction models are shown in Table 10:

[0145] Table 10: Examples of prediction results from the 1+4 complaint prediction models

[0146]

[0147] Optionally, before obtaining the feature data of the user to be predicted, the method further includes:

[0148] The user to be predicted is obtained from the operator's B domain data;

[0149] The method further includes, simultaneously with or after issuing user complaint warnings:

[0150] The users to be predicted were reassured.

[0151] In this embodiment, priority can be given to ensuring the perception of high-value users or VIP users (who can be obtained from the operator's B domain data), and to providing complaint warnings and reassurance (such as SMS reassurance, advance compensation for call charges, points, data traffic, etc.).

[0152] In one specific embodiment, the user complaint alert method may include the following steps:

[0153] 1. Obtain multi-source data within the local network area: including wireless signaling (Trace) data, DPI signaling details (XDR) data, terminal application (OTT) data, user complaint information, and B domain data.

[0154] 2. Select key indicators from different data sources, and aggregate, combine, and correlate them at the user, business, and time levels to construct a feature dataset.

[0155] (1) Wireless signaling (Trace) data can be subscribed to by the operator's network management system and obtained periodically by the base station (e.g., at 15-minute intervals). After being parsed by the parsing server, Trace indicator reports are generated, and key indicator fields (fields are optional) of the Trace data are extracted. As shown in Table 1, this table provides an example of the key indicators selected for 4G LTE Trace data.

[0156] (2) DPI signaling detail record (XDR) data can be provided by the DPI signaling acquisition system. The data can be associated with key fields in the Trace data to attach user identifiers (such as IMSI, International Mobile Subscriber Identity, MSISDN mobile phone number) and timestamps to the key fields of the Trace data. The associated fields are shown in Table 2 (including 4G and 5G).

[0157] (3) Terminal application (OTT) data is provided by Internet service providers (such as Tencent, Alibaba, Baidu, Xiaomi, etc.). For example, Tencent provides data on video lag and game lag in Honor of Kings to operators. In recent years, more and more application service providers have cooperated with telecommunications operators to effectively improve the user experience of applications, expand the scope of user service use, and achieve a win-win situation in the network and market.

[0158] The key personalized metrics for terminal applications are shown in Tables 3 to 6, with examples of key metrics for four types of applications (browser, video, game, and instant messaging). The granularity of the start and end times is selectable (e.g., 15-minute granularity), and usage duration represents the total time the application is used within the start and end times.

[0159] Specifically, the key performance indicators (KPIs) for various terminal applications are independent. For example, in the gaming industry, different games have different performance requirements for key KPIs, and user preferences vary. This invention does not use average data for all gaming business KPIs.

[0160] Furthermore, performance perception ratings for personalized key metrics across different terminal applications are implemented. The data input to the prediction models for each terminal application does not use the average of specific key metrics over a given time period. Instead, it uses a scoring system that differentiates the duration and percentage of time users perceive good or bad performance, thus improving model accuracy. The key metric scores for each terminal application can be obtained using an interval scoring method (with a maximum score of 100, and the interval step size for each key metric can be obtained through actual testing). Examples can be given using Formulas 1 and 2 (Formula 1 represents the webpage response success rate score for application 1, and Formula 2 represents the webpage response latency score for application 1). Formula 1 is an example of a positive metric (the larger the value, the higher the score), and Formula 2 is an example of a negative metric (the smaller the value, the higher the score).

[0161]

[0162]

[0163] The overall perceptual score for Application 1 can be expressed by Formula 3. Based on the score, the perception of Application 1 is rated as excellent, good, average, or poor. For example: when... Then the rating is excellent; when The rating is then good; when The rating is then medium; when The rating is then poor. To apply the importance weight factor of key indicator i, the sum of the importance weight factors of all key indicators is 1.

[0164]

[0165] After rating the terminal applications at different time windows (e.g., 15 minutes, 30 minutes, 1 hour), performance-aware rating index data for each terminal application is generated, as shown in Table 7. Other applications are similarly evaluated to obtain performance-aware rating index data, where the usage duration is the sum of the usage duration at each time granularity within the time window.

[0166] (4) Combine and correlate key indicators of trace data and performance perception rating indicators of each terminal application to construct a feature dataset. This can be correlated by time start and end (e.g., the same 15-minute granular time period) and user identifier (IMSI) fields to construct feature data for machine learning models (e.g., neural network models can be used). Table 8 shows an example of the model feature dataset fields for application 1.

[0167] Similarly, using the same method, the model feature dataset fields for key indicators of single trace data are constructed (represented by application 0 here), as shown in Table 9. The interval scores and ratings for each indicator are defined by reference to the experience values ​​of routine road network assessments by telecommunications operators and the experience values ​​of local network trace data evaluations.

[0168] The final model consists of a Trace data complaint prediction model constructed from key indicators of single Trace data, and an application business complaint prediction model constructed from performance perception rating indicators and key indicators of Trace data for N terminal applications. This results in 1+N complaint prediction models, where N depends on the number of applications provided by the OTT application.

[0169] (5) User complaint information, including the mobile phone number of the complaining user, the time when the user described the poor experience, which is generally vague (such as around 3 p.m.), and the application service described by the user that the user had a poor experience.

[0170] Based on the user's mobile phone number and the time period of historical complaints, a feature training set for the complaint prediction model is constructed by associating the model's feature data. For example, if the user's complaint time period is from 3 PM to 4 PM, then the start and end time of the time window in the 1+N complaint prediction model is from 3 PM to 4 PM.

[0171] 3. Train the model using the historical complaint feature dataset to generate a 1+N complaint prediction model. When a user complaint contains specific application services mentioned in the complaint information (which can be multiple application services), construct positive samples (representing complained samples) for the corresponding application service complaint prediction model. Construct negative complaint samples (representing no complaints) from other time windows where this application service was used. When the historical complaint information does not record specific application services mentioned in the complaint (e.g., the user does not wish to mention them), the complaint samples are only positive samples for the single trace data complaint prediction model.

[0172] Train 1+N complaint prediction models separately, then fuse the complaint probabilities output by each model, and output the expected value Y of the final user complaint using Formula 4. Where y i δ represents the output values ​​of 1+N complaint prediction models (the output results of the complaint probability predicted by each complaint prediction model). i The influencing factors for each complaint prediction model can be given by Formula 5. Let i be the usage duration of the i-th terminal application within the time window. When i = 0, This refers to the duration of a user's connection state within a time window, based on Trace data statistics.

[0173]

[0174]

[0175] After training with all the large datasets, the Y values ​​of all positive samples can be sorted in ascending order to obtain the tolerance threshold Y for user complaints after comprehensive evaluation. h (For example, the minimum value after ascending order can be taken).

[0176] 4. Input the user-level business characteristic data to be predicted into 1+N complaint prediction models, output the complaint probability of each complaint prediction model, output the user complaint expectation, and compare it with the tolerance threshold Y. h Compare the results; if the expected complaint output is greater than Y... h If the user is not properly informed, a warning will be issued, and the user will be reassured (e.g., by sending a text message, offering advance compensation for phone bills, points, or data).

[0177] Preferred options include prioritizing the perception of high-value users or VIP users (whose data can be obtained from the operator's B-domain data) and providing complaint warnings and reassurance.

[0178] The user complaint early warning method provided in this embodiment of the invention first obtains the feature data of the user to be predicted; then, the feature data of the user to be predicted is input into 1 plus N pre-trained complaint prediction models, and the complaint probability predicted by each complaint prediction model is output; wherein, the 1 plus N complaint prediction models include one Trace data complaint prediction model constructed based on key indicators of wireless signaling Trace data, and N application service complaint prediction models jointly constructed based on the key indicators of the Trace data and N performance perception rating indicators corresponding to terminal applications, wherein N is greater than or equal to 1; then, the complaint expectation of the user to be predicted is calculated according to the predicted complaint probability; if the complaint expectation of the user to be predicted is greater than or equal to a preset tolerance threshold, a user complaint early warning is issued. This invention integrates one plus N user-level complaint prediction models to comprehensively evaluate and determine user complaint expectations. The application business complaint prediction model does not use business category distinctions (such as one model for games, another for videos, etc.), but instead uses the business distinctions provided by specific terminal applications, and considers the usage habits of each user and each application (i.e., performance perception rating index data). This not only improves the accuracy of user complaint warnings, but also reduces the number of user complaints, solving the problem that existing user complaint warning methods are prone to insufficient accuracy.

[0179] Example 2:

[0180] like Figure 3 As shown, this embodiment provides a device for user complaint early warning, used to execute the above-mentioned user complaint early warning method, including:

[0181] Feature data acquisition module 11 is used to acquire feature data of the user to be predicted;

[0182] The complaint probability acquisition module 12, connected to the feature data acquisition module 11, is used to input the feature data of the user to be predicted into 1 plus N complaint prediction models and output the complaint probability predicted by each complaint prediction model; wherein, the 1 plus N complaint prediction models include 1 Trace data complaint prediction model constructed based on key indicators of wireless signaling Trace data, and N application service complaint prediction models jointly constructed based on the key indicators of the Trace data and the performance perception rating indicators corresponding to N terminal applications, wherein N is greater than or equal to 1;

[0183] The complaint expectation calculation module 13 is connected to the complaint probability acquisition module 12 and is used to calculate the complaint expectation of the user to be predicted based on the predicted complaint probability.

[0184] The complaint warning module 14 is connected to the complaint expectation calculation module 13 and is used to issue a user complaint warning if the complaint expectation of the user to be predicted is greater than or equal to a preset tolerance threshold.

[0185] Optionally, the device further includes:

[0186] The data source acquisition module is used to acquire Trace data, Deep Packet Inspection (DPI) signaling details (XDR) data, terminal application OTT data, and user complaint information within the local network area.

[0187] The dataset acquisition module is used to obtain 1 plus N feature datasets based on the Trace data, XDR data, OTT data, and user complaint information;

[0188] The model training module is used to train 1+N pre-set complaint prediction models using the 1+N feature datasets respectively, and obtain the complaint probability output by each complaint prediction model;

[0189] The probability fusion module is used to fuse the complaint probabilities output by the 1 plus N complaint prediction models to obtain the user's expected complaint value Y.

[0190] The tolerance threshold acquisition module is used to sort the Y values ​​of positive samples in the feature dataset in ascending order to obtain the tolerance threshold for user complaints after comprehensive evaluation.

[0191] Optionally, the dataset acquisition module specifically includes:

[0192] The first acquisition unit is used to acquire key indicators of the Trace data at each time granularity within a preset time window;

[0193] The first association unit is used to associate the XDR data with the Trace data according to the preset association field in the XDR data to obtain the user identifier corresponding to the Trace data.

[0194] The second acquisition unit is used to acquire the key indicators corresponding to N terminal applications in the OTT data at each time granularity within a preset time window.

[0195] The indicator data generation unit is used to generate performance perception rating indicator data for each terminal application based on the key indicators corresponding to the N terminal applications. The performance perception rating indicator data includes the duration and percentage corresponding to multiple performance perception rating levels.

[0196] The second association unit is used to associate the key indicators of the Trace data and the performance perception rating indicators of N terminal applications on the preset time window according to the user identifier, so as to obtain the feature data of each application business complaint prediction model.

[0197] The third association unit is used to associate the feature data of each application business complaint prediction model with the mobile phone number of the complaining user and the historical complaint time period in the user complaint information, so as to obtain N feature datasets in the 1 plus N feature datasets.

[0198] The fourth association unit is used to obtain one feature dataset from the 1 plus N feature datasets by associating the key indicators of the Trace data with the complainant's mobile phone number, historical complaint time period and user identifier in the user complaint information.

[0199] Optionally, the indicator data generation unit specifically includes:

[0200] The scoring unit is used to score the key indicators corresponding to each terminal application at each time granularity using an interval scoring method;

[0201] The integration unit is used to obtain the comprehensive perception score of each terminal application at each time granularity based on the scores of each key indicator of each terminal application at each time granularity.

[0202] The rating unit is used to rate each terminal application based on the comprehensive perception score of each terminal application at each time granularity on the preset time window, and generate performance perception rating index data corresponding to each terminal application.

[0203] Optionally, the overall perception score for each terminal application is calculated according to the following formula:

[0204]

[0205] In the formula, W i The importance weighting factor for the i-th key indicator of the terminal application. Let be the score of the i-th key indicator of the terminal application, and n be the total number of key indicators of the terminal application.

[0206] Optionally, the performance perception rating level includes four levels: excellent, good, average, and poor.

[0207] The performance perception rating index data includes the duration of excellent rating, good rating, average rating, poor rating, percentage of excellent rating, percentage of good rating, percentage of average rating, percentage of poor rating, monthly average daily usage frequency, and monthly average daily usage duration.

[0208] Optionally, the probability fusion module specifically includes:

[0209] The first calculation unit is used to calculate the impact factor corresponding to each of the complaint prediction models;

[0210] The second calculation unit is used to calculate the user's expected complaint value Y based on the influencing factors corresponding to each complaint prediction model and the output complaint probability.

[0211] Optionally, the first calculation unit calculates the impact factor corresponding to each complaint prediction model according to the following formula:

[0212]

[0213] In the formula, δ i This represents the influence factor of the i-th complaint prediction model. When i = 0, The duration of the user's connection state within a preset time window, when i≠0. The usage duration of the i-th terminal application within the preset time window;

[0214] The second calculation unit calculates the user's expected complaint value Y according to the following formula:

[0215]

[0216] In the formula, Y represents the expectation of the complaint, y i This represents the complaint probabilities output by 1 plus N complaint prediction models.

[0217] Optionally, the device further includes:

[0218] The user acquisition module is used to acquire the user to be predicted from the operator's B domain data;

[0219] The reassurance module is used to reassure the user to be predicted.

[0220] Example 3:

[0221] refer to Figure 4 This embodiment provides a user complaint warning device, including a memory 21 and a processor 22. The memory 21 stores a computer program, and the processor 22 is configured to run the computer program to execute the user complaint warning method in Embodiment 1.

[0222] The memory 21 is connected to the processor 22. The memory 21 can be a flash memory, a read-only memory or other memory, and the processor 22 can be a central processing unit or a microcontroller.

[0223] Example 4:

[0224] This embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the user complaint warning method in Embodiment 1 above.

[0225] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules, or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.

[0226] In summary, the user complaint early warning method, apparatus, and readable storage medium provided in this embodiment of the invention first acquire the feature data of the user to be predicted; then, the feature data of the user to be predicted is input into 1 plus N pre-trained complaint prediction models, and the complaint probability predicted by each complaint prediction model is output; wherein, the 1 plus N complaint prediction models include one Trace data complaint prediction model constructed based on key indicators of wireless signaling Trace data, and N application service complaint prediction models jointly constructed based on the key indicators of the Trace data and N performance perception rating indicators corresponding to terminal applications, wherein N is greater than or equal to 1; then, the complaint expectation of the user to be predicted is calculated according to the predicted complaint probability; if the complaint expectation of the user to be predicted is greater than or equal to a preset tolerance threshold, a user complaint early warning is issued. This invention integrates one plus N user-level complaint prediction models to comprehensively evaluate and determine user complaint expectations. The application business complaint prediction model does not use business category distinctions (such as one model for games, another for videos, etc.), but instead uses the business distinctions provided by specific terminal applications, and considers the usage habits of each user and each application (i.e., performance perception rating index data). This not only improves the accuracy of user complaint warnings, but also reduces the number of user complaints, solving the problem that existing user complaint warning methods are prone to insufficient accuracy.

[0227] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for user complaint early warning, characterized in that, include: Obtain the feature data of the user to be predicted; The feature data of the user to be predicted is input into 1 plus N complaint prediction models, and the complaint probability predicted by each complaint prediction model is output. The 1 plus N complaint prediction models include 1 Trace data complaint prediction model based on key indicators of wireless signaling Trace data, and N application service complaint prediction models jointly constructed based on the key indicators of Trace data and the performance perception rating indicators corresponding to N terminal applications, where N is greater than or equal to 1. Calculate the expected complaint rate of the user to be predicted based on the predicted complaint probability; If the expected complaint from the user to be predicted is greater than or equal to the preset tolerance threshold, a user complaint warning will be issued. Before obtaining the feature data of the user to be predicted, the method further includes: Train 1+N pre-set complaint prediction models using 1+N feature datasets respectively, and obtain the complaint probability output by each complaint prediction model; The complaint probabilities output by the 1+N complaint prediction models are fused to obtain the user's expected complaint value Y. The Y values ​​of positive samples in the feature dataset are sorted in ascending order to obtain the tolerance threshold for user complaints after comprehensive evaluation. The process of fusing the complaint probabilities output by the 1+N complaint prediction models to obtain the user's expected complaint value Y specifically includes: Calculate the impact factor corresponding to each of the complaint prediction models; The user's expected complaint value Y is calculated based on the influencing factors corresponding to each complaint prediction model and the output complaint probability. The calculation of the impact factor corresponding to each of the complaint prediction models is specifically based on the following formula: In the formula, δ i This represents the influence factor of the i-th complaint prediction model. When i = 0, The duration of the user's connection state within a preset time window, when i≠0. The usage duration of the i-th terminal application within the preset time window; The user's expected complaint value Y is calculated based on the influencing factor and the output complaint probability corresponding to each complaint prediction model, specifically according to the following formula: In the formula, Y represents the expectation of the complaint, y i This represents the complaint probabilities output by 1 plus N complaint prediction models.

2. The method according to claim 1, characterized in that, Before obtaining the feature data of the user to be predicted, the method further includes: Acquire trace data, deep packet inspection (DPI) signaling details (XDR) data, terminal application OTT data, and user complaint information within the local network area; Based on the Trace data, XDR data, OTT data, and user complaint information, 1 plus N feature datasets are obtained.

3. The method according to claim 2, characterized in that, The process of obtaining 1 plus N feature datasets based on the Trace data, XDR data, OTT data, and user complaint information specifically includes: Obtain key metrics of the Trace data at each time granularity within a preset time window; The user identifier corresponding to the Trace data is obtained by associating the preset association fields in the XDR data with the Trace data; Obtain the key indicators corresponding to N terminal applications in the OTT data at each time granularity within a preset time window; Based on the key indicators corresponding to the N terminal applications, performance perception rating index data is generated for each terminal application. The performance perception rating index data includes the duration and percentage corresponding to multiple performance perception rating levels. Based on the user identifier, the key indicators of the Trace data and the performance perception rating indicators of N terminal applications are correlated on the preset time window to obtain the feature data of each application business complaint prediction model. Based on the complainant's mobile phone number and historical complaint time period in the user complaint information, and by associating them with the feature data of each application service complaint prediction model, N feature datasets are obtained from the 1 plus N feature datasets; and... Based on the complainant's mobile phone number, historical complaint time period, and key indicators associated with the Trace data from the user's identifier in the user complaint information, one feature dataset is obtained from the 1 plus N feature datasets.

4. The method according to claim 3, characterized in that, The step of generating performance-aware rating index data for each terminal application based on the key indicators corresponding to the N terminal applications specifically includes: The key indicators corresponding to each terminal application at each time granularity are scored using an interval scoring method. Based on the scores of each key indicator for each terminal application at each time granularity, the comprehensive perception score for each terminal application at each time granularity is obtained. Based on the comprehensive perception score of each terminal application at each time granularity, a rating is generated on the preset time window to produce performance perception rating index data corresponding to each terminal application.

5. The method according to claim 4, characterized in that, The overall perception score for each terminal application is calculated according to the following formula: In the formula, W i The importance weighting factor for the i-th key indicator of the terminal application. Let be the score of the i-th key indicator of the terminal application, and n be the total number of key indicators of the terminal application.

6. The method according to claim 4, characterized in that, The performance perception rating level includes four levels: excellent, good, average, and poor. The performance perception rating index data includes the duration of excellent rating, good rating, average rating, poor rating, percentage of excellent rating, percentage of good rating, percentage of average rating, percentage of poor rating, monthly average daily usage frequency, and monthly average daily usage duration.

7. The method according to claim 1, characterized in that, Before obtaining the feature data of the user to be predicted, the method further includes: The user to be predicted is obtained from the operator's B domain data; The method further includes, simultaneously with or after issuing user complaint warnings: The users to be predicted were reassured.

8. A device for early warning of user complaints, characterized in that, include: The feature data acquisition module is used to acquire feature data of the user to be predicted. The complaint probability acquisition module, connected to the feature data acquisition module, is used to input the feature data of the user to be predicted into 1 plus N pre-trained complaint prediction models and output the complaint probability predicted by each complaint prediction model; wherein, the 1 plus N complaint prediction models include 1 Trace data complaint prediction model constructed based on key indicators of wireless signaling Trace data, and N application service complaint prediction models jointly constructed based on the key indicators of the Trace data and the performance perception rating indicators corresponding to N terminal applications, wherein N is greater than or equal to 1; The complaint expectation calculation module, connected to the complaint probability acquisition module, is used to calculate the complaint expectation of the user to be predicted based on the predicted complaint probability. The complaint warning module is connected to the complaint expectation calculation module and is used to issue a user complaint warning if the complaint expectation of the user to be predicted is greater than or equal to a preset tolerance threshold. The device further includes: The model training module is used to train 1+N pre-set complaint prediction models using 1+N feature datasets respectively, and obtain the complaint probability output by each complaint prediction model; The probability fusion module is used to fuse the complaint probabilities output by the 1 plus N complaint prediction models to obtain the user's expected complaint value Y. The tolerance threshold acquisition module is used to sort the Y values ​​of positive samples in the feature dataset in ascending order to obtain the tolerance threshold for user complaints after comprehensive evaluation. The probability fusion module specifically includes: The first calculation unit is used to calculate the impact factor corresponding to each of the complaint prediction models; The second calculation unit is used to calculate the user's expected complaint value Y based on the influencing factor corresponding to each complaint prediction model and the output complaint probability. The first calculation unit calculates the impact factor corresponding to each complaint prediction model according to the following formula: In the formula, δ i This represents the influence factor of the i-th complaint prediction model. When i = 0, The duration of the user's connection state within a preset time window, when i≠0. The usage duration of the i-th terminal application within the preset time window; The second calculation unit calculates the user's expected complaint value Y according to the following formula: In the formula, Y represents the expectation of the complaint, y i This represents the complaint probabilities output by 1 plus N complaint prediction models.

9. A device for user complaint early warning, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to implement the method for user complaint early warning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the user complaint early warning method as described in any one of claims 1-7.

Images