Mobile network evaluation method and system based on user perception
By collecting and preprocessing user data, building machine learning models, and clustering with geographic grid dimensions, the limitations and lag problems of network evaluation methods in the existing technology in user perception are solved, and more accurate and efficient Network evaluation and optimization.
Patent Information
- Application Number
- CN202411124573.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-26
AI Technical Summary
The existing network evaluation methods have limitations in evaluating users' real perception, which cannot objectively and accurately reflect the user experience, and there is a lag in handling after user complaints, which fails to fully reflect the user's real perception and data island problems.
By collecting user data, preprocessing and feature selection, a machine learning model is built to predict user complaint probability, extract the root causes of quality poorness, and cluster them in combination with geographic grid dimensions to achieve more objective and close to user perception network evaluation and optimization.
It realizes more accurate user-perceived network evaluation, discovers problems in advance and handles them, reduces user complaint rates, breaks data silos, and improves the fairness and efficiency of evaluations.
Smart Images

Figure CN119025993A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of network communication technology, and in particular to a mobile network evaluation method based on user perception, a mobile network evaluation system based on user perception, an electronic device, and a computer-readable storage medium. Background Art
[0002] As competition in the telecommunications market becomes increasingly fierce, optimizing network performance and improving service levels to deepen the experience and satisfaction of 5G users are undoubtedly key measures to improve user stickiness and win market favor. Currently, common mobile network quality and service level evaluation indicators mainly include: mobile network indicators, Ministry of Industry and Information Technology satisfaction, user complaint rate, etc. The method used is to use the currently popular big data and AI technology to mine and analyze events that affect user perception based on user signaling and user plane data, and conduct full network user analysis and network evaluation.
[0003] Problems in the prior art:
[0004] 1. Existing network evaluation methods have limitations in evaluating users' real perception and cannot objectively and accurately reflect users' actual experience: although network KPI (Key Performance Indicator) indicators can reflect the service quality of the entire network, they are still different from the perception of individual users; user complaints are often subjective, and the complaint standards of different users are not uniform, so it is difficult to objectively quantify; the data sample of the Ministry of Industry and Information Technology's satisfaction survey is limited, and the respondents have different understandings of the events, resulting in strong randomness in the feedback results and a lack of sufficient fairness and objectivity.
[0005] 2. Currently, there is a lack of proactive evaluation of the network from the perspective of user perception, and when problems arise, the existing methods are mostly to maintain customer relations after complaints, which is lagging behind.
[0006] 3. The existing evaluation methods involve a narrow user dimension and do not solve problems such as data silos and inaccurate data. Therefore, the evaluation results obtained cannot fully reflect the user's true perception. Summary of the invention
[0007] In order to at least solve the problems that the existing network evaluation methods in the prior art have limitations in evaluating the real perception of users, and there is a lag in processing after user complaints, the present disclosure provides a mobile network evaluation method based on user perception, a mobile network evaluation system based on user perception, an electronic device, and a computer-readable storage medium, which can obtain a more objective and user-perceived network evaluation, and more accurately evaluate and optimize the user behavior and network quality of the entire network; and
[0008] It can handle problems and appease users in advance, reducing the user complaint rate.
[0009] In a first aspect, the present disclosure provides a mobile network evaluation method based on user perception, the method comprising:
[0010] Collect user data, including basic user information, complaint data, and user perception data;
[0011] Preprocess the collected user data, including data auditing, data cleaning and data transformation, to transform the user data into indicator data that can be processed by the model;
[0012] Perform influencing feature selection on each indicator of the processed user data;
[0013] After completing the selection of influencing features, construct positive and negative sample data, which come from random users who have not complained and users who have complained, respectively. Take the influencing feature data extracted from the samples as input, select a preset machine learning model for training and testing, and obtain a user complaint prediction model;
[0014] Based on the trained user complaint prediction model, the user data to be identified is input to predict the complaint probability of the sub-healthy user population, and the details of high-probability complaint users are obtained.
[0015] Furthermore, the method further comprises:
[0016] Based on the complaint data, the root causes of complaints are extracted and the root causes of poor quality are marked;
[0017] Based on the results of poor quality root cause annotation, user data is sorted and classified according to voice indicators and data indicators;
[0018] The annotated root causes of poor quality are used as user labels for training data, and model training is performed on two types of behaviors: voice complaints and data complaints, respectively, to obtain user voice complaint prediction models and user data complaint prediction models.
[0019] Furthermore,
[0020] The user basic information includes user number, age, gender, terminal, package, call fee, voice and traffic usage information;
[0021] The complaint data is the complaint data of existing network users due to network reasons obtained from the complaint system, including user number, complaint time, complaint location, customer complaint description, complaint problem location and service category;
[0022] The user perception data is data related to user perception in the user XDR (Extended Detection and Response), IMS (IP Multimedia Subsystem) XDR, and call MOS (mean opinion score) values obtained from the upstream system, including user number, time, service type, key indicators, and the cell where the user is located.
[0023] Furthermore, the preprocessing also includes data reduction and data exploration;
[0024] The data reduction includes completing the indicator dimension reduction by analyzing the correlation between various indicators of the user data;
[0025] The data exploration includes determining the distribution of user voice indicators and data indicators and excluding useless indicators.
[0026] Furthermore, the method further comprises:
[0027] Select the following six events as the poor quality events perceived by users: dropped calls, voice network failure, poor voice quality, weak coverage, abnormal video usage, and abnormal web page usage;
[0028] Combined with the cell where the poor quality evaluation event occurred, the clustering of single-user poor quality services in the grid dimension is realized, including:
[0029] Divide the geographic space on the map into a series of uniform grid systems, define the size of each grid as M*N, and use the longitude and latitude of a rectangle to represent the location of each grid;
[0030] Based on the pre-processed user data, the location of the cell where the quality difference event occurred is converted into longitude and latitude and placed on a grid system map;
[0031] Calculate the number of users and times of poor quality events in each grid, as well as the proportion of poor quality events to all events;
[0032] Whether the user passes through the poor quality area and the frequency of passing through the poor quality area are used as the relationship indicators between the user and the poor quality area.
[0033] Furthermore, the influencing feature selection of each indicator of the processed user data includes:
[0034] After data preprocessing and clustering in the grid dimension, various features to be selected are obtained from various indicators, including user age, gender, package, network time, network type, voice usage, Internet usage, frequency of occurrence of poor quality events on the day and in the past three days, frequency of complaints in the month and in the past six months, and the relationship between users and poor quality areas;
[0035] The influencing features of N models are selected from various features to be selected through a preset feature selection method.
[0036] Furthermore, the method further comprises:
[0037] Output the grid clustering result table based on the number of users and times of poor quality events in each grid and the proportion of poor quality events to all events;
[0038] The network quality is evaluated according to the grid clustering result table.
[0039] Furthermore, the impact feature data extracted from the sample is used as input, a preset machine learning model is selected for training and testing, and a user complaint prediction model is obtained, including:
[0040] Compare the accuracy, recall and f1-score evaluation indicators of several algorithms, including logistic regression, neural network, LightGBM (Light Gradient Boosting Machine), multi-layer perceptron and random forest algorithm, on sample data. According to the comparison results, determine the weights of various algorithms, and use the integrated algorithm that combines the above algorithms and weights as the preset machine learning model.
[0041] The optimal parameters are selected for training. During the training process, the cross-validation method is used to evaluate and tune the machine learning model to obtain the user complaint model.
[0042] In a second aspect, the present disclosure provides a mobile network evaluation system based on user perception, the system comprising:
[0043] A collection module, which is configured to collect user data, including basic user information, complaint data, and user perception data;
[0044] A preprocessing module is configured to preprocess the collected user data, including data auditing, data cleaning and data transformation, so as to transform the user data into indicator data that can be processed by the model;
[0045] A selection module configured to select influencing features for each indicator of the processed user data;
[0046] A training module is configured to construct positive and negative sample data after the selection module completes the selection of influencing features, wherein the positive and negative sample data are respectively from random non-complaining users and complaining users, and the influencing feature data extracted from the samples is used as input, and a preset machine learning model is selected for training and testing to obtain a user complaint prediction model;
[0047] The prediction module is set to predict the complaint probability of sub-healthy users based on the trained user complaint prediction model, and inputs the user data to be identified to predict the complaint probability of the sub-healthy user population, and obtains the details of high-probability complaint users.
[0048] In a third aspect, the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the user perception-based mobile network evaluation method as described in any one of the first aspects.
[0049] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the user-perception-based mobile network evaluation method described in any one of the first aspects above is implemented.
[0050] Beneficial effects:
[0051] The user-perception-based mobile network evaluation method, user-perception-based mobile network evaluation system, electronic device and storage medium provided by the present disclosure integrate multiple data sources such as user basic information, signaling plane data, user plane data and complaint data, conduct multi-dimensional mining and analysis, break data silos, fully mine user behavior events based on various data sources, and accurately complete feature selection after data cleaning and a series of indicator selection work, which can not only reflect user perception more realistically and objectively but also ensure the simplicity and efficiency of the algorithm. After selecting appropriate influencing features to train a user complaint prediction model, a high-probability complaint user list can be obtained, and according to the user details and corresponding problems, problem handling and user appeasement can be carried out in advance to reduce the user complaint rate. In addition, the network can be evaluated based on data such as the user complaint probability, so as to achieve a more objective and user-perception-closer network evaluation. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 A flowchart of a mobile network evaluation method based on user perception provided in the first embodiment of the present disclosure;
[0053] Figure 2 A result diagram of bringing a test set into a model for verification provided in an embodiment of the present disclosure;
[0054] Figure 3 A precision-recall curve diagram of the model test results provided by the embodiment of the present disclosure;
[0055] Figure 4 The ROC curve diagram of the model test results provided by the embodiment of the present disclosure;
[0056] Figure 5A flowchart of a mobile network evaluation method based on user perception provided in Embodiment 2 of the present disclosure;
[0057] Figure 6 A grid map of a grid clustering result provided in the second embodiment of the present disclosure;
[0058] Figure 7 A complaint prediction result display diagram provided in the second embodiment of the present disclosure;
[0059] Figure 8 An architecture diagram of a mobile network evaluation system based on user perception provided in Embodiment 3 of the present disclosure;
[0060] Fig. 9 This is an architecture diagram of an electronic device provided in Embodiment 4 of the present disclosure. DETAILED DESCRIPTION
[0061] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further described in detail below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention, rather than to limit the present invention.
[0062] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence; and, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be arbitrarily combined with each other.
[0063] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms of "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings.
[0064] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present disclosure, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.
[0065] At present, the common mobile network quality and service level evaluation indicators include mobile network indicators, MIIT satisfaction, and user complaint rate; among them, the network KPI indicators formulated by each operator include: network connection rate, call drop rate, network registration success rate, PDU (Protocol Data Unit) session establishment success rate and other indicators; the data source is generated by the professional network management of the equipment, and is counted in units of network elements, cities, and provinces. MIIT satisfaction: Horizontal benchmarking is carried out through the MIIT's scoring and ranking of each operator's network quality satisfaction; the data source is the MIIT. User complaint rate: Statistics and analysis of the number of user complaints and appeals during the evaluation period, and user satisfaction is measured by year-on-year and month-on-month changes; the data source is the customer service platform.
[0066] However, network evaluation based on these indicators has limitations in assessing users' real perception, and cannot analyze actual user calls and Internet events to conduct network evaluation, detect problems in advance and reduce complaints, and there is a lag in handling user complaints.
[0067] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems existing in the prior art are described in detail below with specific embodiments. It is understandable that in the embodiments of the present application, the execution subject can perform some or all of the steps in the embodiments of the present application, and these steps or operations are only examples. The embodiments of the present application can also perform other operations or deformations of various operations. In addition, each step can be performed in a different order presented in the embodiments of the present application, and it is possible not to perform all the operations in the embodiments of the present application. And, the following several specific embodiments can be combined with each other, and may not be repeated in some embodiments for the same or similar concepts or processes.
[0068] Figure 1 A flowchart of a mobile network evaluation method based on user perception is provided in the first embodiment of the present disclosure, such as Figure 1 As shown, the method includes:
[0069] Step S101: Collect user data, including basic user information, complaint data and user perception data;
[0070] Step S102: pre-processing the collected user data, including data auditing, data cleaning and data transformation, and transforming the user data into index data that can be processed by the model;
[0071] Step S103: Select influencing features for each indicator of the processed user data;
[0072] Step S104: After completing the selection of influencing features, construct positive and negative sample data, wherein the positive and negative sample data are respectively from random non-complaining users and complaining users, and use the influencing feature data extracted from the samples as input, select a preset machine learning model for training and testing, and obtain a user complaint prediction model;
[0073] Step S105: Based on the trained user complaint prediction model, the user data to be identified is input to predict the complaint probability of the sub-healthy user group, and the details of the high-probability complaint users are obtained.
[0074] Since the existing network evaluation methods have limitations in evaluating the real perception of users, the disclosed embodiment starts from the user's perspective, collects user data from all aspects, breaks the data island, integrates multi-dimensional data sources such as user basic information, signaling data, user data and complaint data, and conducts multi-dimensional mining and analysis, and then pre-processes the data, such as data audit, completes the verification of the legality, consistency, integrity, etc. of the data; data cleaning, cleans the audited data, including missing value processing, duplicate value processing, abnormal value processing, etc.; and data transformation, transforms the data of special types of indicators to obtain features that can be understood by the model, such as using word2vec to convert character features into numerical features, such as: gender, package, etc.; and completes basic statistical analysis of existing data. Then, the various indicators of the processed user data are selected for influencing features, specifically by combining data exploration results and expert experience, and through feature selection methods, the influencing features are selected from various indicators, and the original data is converted into potential problems and features that can better represent the prediction model, so as to select the most relevant features for subsequent model training. After the influence feature selection, positive and negative sample data are constructed from the collected user data. The positive and negative sample data come from random non-complaint users and complaining users (limited by a certain time period, such as complaints on the same day, within three days or within a week). Due to the uneven distribution of complaints and non-complaints, SMOTE (Synthetic Minority Over-sampling Technique) oversampling is used to solve the sample imbalance problem; the SMOTE method synthesizes minority class samples to increase their number in the data set, thereby achieving sample balance and improving the generalization ability and accuracy of the model. The influence feature data extracted from the sample is used as input, and the user complaint prediction model is obtained through training and testing through the preset machine learning model. The machine learning model can be constructed through various algorithms such as logistic regression, neural network, LightGBM, multi-layer perceptron, etc. The model construction and training process can be implemented through various existing methods, which will not be repeated here. After the model training is completed, based on the trained user complaint prediction model, the user data to be identified is input to predict the complaint probability of the sub-healthy user population, and the details of the high-probability complaint users are obtained. The sub-healthy user population is preliminarily judged by the user's indicator data in the early stage, and the user's health is scored based on the user's perception index to determine whether the user is in a sub-healthy state. The data is processed in the same way to extract features and then input into the model, so that the details of users with high probability of complaints can be obtained.
[0075] Furthermore, by regularly collecting complaint lists, updating sample data, and training, the latest model is generated.
[0076] Based on the trained model, the user index data is input to predict the complaint probability of sub-healthy users every day, and the details of users with high probability of complaints are obtained. According to the user details and the corresponding problems found, the problems are handled in advance and the users are comforted to reduce the user complaint rate.
[0077] The disclosed embodiment breaks the data island, fully mines user behavior events based on various data sources, and accurately completes feature selection after data cleaning and a series of indicator selection work, which can not only reflect user perception more realistically and objectively, but also ensure the simplicity and efficiency of the algorithm. After selecting appropriate influencing features to train the user complaint prediction model, it is possible to obtain high-probability complaint user details, and according to the user details and corresponding problems, the problem is handled in advance and the user is appeased, thereby reducing the user complaint rate. In addition, the network can be evaluated based on data such as the user complaint probability, achieving a more objective and user-perceived network evaluation.
[0078] Furthermore, the method further comprises:
[0079] Based on the complaint data, the root causes of complaints are extracted and the root causes of poor quality are marked;
[0080] Based on the results of poor quality root cause annotation, user data is sorted and classified according to voice indicators and data indicators;
[0081] The annotated root causes of poor quality are used as user labels for training data, and model training is performed on two types of behaviors: voice complaints and data complaints, respectively, to obtain user voice complaint prediction models and user data complaint prediction models.
[0082] Since users' use of the network can be divided into voice and data types, voice communication and data communication have obvious differences in communication objects, transmission reliability requirements, average duration, and communication establishment request response time. The usage scenarios and problems encountered are generally quite different. When users complain, the root causes of complaints are extracted based on the recorded user complaints, such as poor signals in underground garages and basements, or network freezes, and the specific causes are marked to determine the specific causes, and divided into voice complaints, data complaints, or both. Then, for user data with complaints, they are sorted and classified according to voice indicators and data indicators according to the complaint type. The user labels of the training data are used to obtain two types of data sets (voice training set and data training set). The two data sets are used to train models for voice complaints and data complaints, respectively, to obtain user voice complaint prediction models and user data complaint prediction models. By classifying and training the data separately, models of two complaint types are obtained. In subsequent predictions, the two models are used for prediction, so that the probability of users complaining in voice or online can be obtained. This makes the model training more targeted and the model prediction more accurate.
[0083] Furthermore,
[0084] The user basic information includes user number, age, gender, terminal, package, call fee, voice and traffic usage information;
[0085] The complaint data is the complaint data of existing network users due to network reasons obtained from the complaint system, including user number, complaint time, complaint location, customer complaint description, complaint problem location and service category;
[0086] The user perception data is the data related to user perception in the user XDR, IMS XDR and call MOS value obtained from the upstream system, including user number, time, service type, key indicators and the cell where the user is located.
[0087] By correlating and analyzing user basic information, user complaint data, and user perception data, we can establish a more objective network evaluation that is closer to user perception. After data cleaning and a series of feature selection work, accurate feature selection can not only reflect user perception more realistically and objectively, but also ensure the simplicity and efficiency of the algorithm.
[0088] Furthermore, the preprocessing also includes data reduction and data exploration;
[0089] The data reduction includes completing the indicator dimension reduction by analyzing the correlation between various indicators of the user data;
[0090] The data exploration includes determining the distribution of user voice indicators and data indicators and excluding useless indicators.
[0091] Data reduction, by analyzing the correlation between indicators to complete the initial dimension reduction of indicators, for example, if the Pearson correlation coefficient between the uplink mos performance and the downlink mos performance is 1, then one of them is selected as the model input. This reduces the number of indicators and obtains better and more representative features. Through data exploration, the distribution of data indicators can be obtained. For some indicators, such as almost all users are normal or abnormal, they can be excluded. The selected influencing features can better complete the model training and ensure the model training effect.
[0092] Furthermore, the method further comprises:
[0093] Select the following six events as the poor quality events perceived by users: dropped calls, voice network failure, poor voice quality, weak coverage, abnormal video usage, and abnormal web page usage;
[0094] Combined with the cell where the poor quality evaluation event occurred, the clustering of single-user poor quality services in the grid dimension is realized, including:
[0095] Divide the geographic space on the map into a series of uniform grid systems, define the size of each grid as M*N, and use the longitude and latitude of a rectangle to represent the location of each grid;
[0096] Based on the pre-processed user data, the location of the cell where the quality difference event occurred is converted into longitude and latitude and placed on a grid system map;
[0097] Calculate the number of users and times of poor quality events in each grid, as well as the proportion of poor quality events to all events;
[0098] Whether the user passes through the poor quality area and the frequency of passing through the poor quality area are used as the relationship indicators between the user and the poor quality area.
[0099] After data processing, the embodiment of the present disclosure finally selects six events, namely, call drop, voice network failure, poor voice quality, weak coverage, abnormal video usage, and abnormal web page usage, as user-perceived evaluation events. The evaluation criteria for each type of poor quality event are shown in Table 1. For different data types, Hive+Spark computing technology is used to implement data analysis and mining.
[0100] Table 1 Evaluation criteria for poor quality events
[0101]
[0102]
[0103] After obtaining whether a poor quality event has occurred, the clustering of single-user poor quality services in the grid dimension is further realized in combination with the cell where the event occurred. First, the geographic space is divided into a series of uniform grid systems, and the size of each grid is defined as M*N. The position of each grid can be represented by the longitude and latitude of any point in the rectangle, and the longitude and latitude of the lower left corner (i.e., the northwest corner) can be used. Based on the processed poor quality data, the location of the cell where the poor quality event occurred is converted into longitude and latitude and placed on the map. Then calculate the number of users and times of poor quality events in each grid, as well as the proportion of poor quality events to all events. At this point, if the number of poor quality events is graded and represented by different colors, a grid map is obtained.
[0104] By combining user service call records with the grid, the specific location of each poor quality service event can be accurately located. The distribution pattern of the problem area can be effectively identified, and the analysis can be performed to determine whether the user has passed through the poor quality area and the frequency of passing through. The indicator of the relationship between the user and the poor quality area can be increased to prepare for subsequent complaint prediction and analysis.
[0105] Furthermore, the influencing feature selection of each indicator of the processed user data includes:
[0106] After data preprocessing and clustering in the grid dimension, various features to be selected are obtained from various indicators, including user age, gender, package, network time, network type, voice usage, Internet usage, frequency of occurrence of poor quality events on the day and in the past three days, frequency of complaints in the month and in the past six months, and the relationship between users and poor quality areas;
[0107] The influencing features of N models are selected from various features to be selected through a preset feature selection method.
[0108] After data preprocessing and clustering in the grid dimension, feature selection and extraction are performed on the processed data. By combining data exploration results and expert experience, based on user age, gender, package, online time, network type, voice usage (usage) on the same day and in the past three days, Internet usage (usage), six frequency of occurrence of poor quality events, complaint frequency in the current month and in the past six months, and the relationship between users and poor quality areas, feature selection is performed through a preset feature selection method. The preset feature selection method includes L1 regularization, t-Test, decision tree feature importance ranking, and nonlinear feature construction methods. After combining expert experience, paper research and multiple training verifications, N features are finally selected as the influencing features of the model. Among them, generally N ≥ 5. Feature selection can be performed separately for the user voice complaint prediction model and the user data complaint prediction model. The feature selection process is the same as the current feature selection method, which will not be repeated here.
[0109] Furthermore, the method further comprises:
[0110] Output the grid clustering result table based on the number of users and times of poor quality events in each grid and the proportion of poor quality events to all events;
[0111] The network quality is evaluated according to the grid clustering result table.
[0112] The output raster clustering result table can be shown in Table 2.
[0113] Table 2 Main output fields of raster clustering
[0114]
[0115] Table 2 shows the main output fields of grid clustering, listing the grid ID, longitude and latitude, city and cell, and counting the number of poor quality events and users with poor quality in the grid, as well as whether the grid has been a poor quality grid for three consecutive days. User details include all user details of the grid with poor service quality. Combining geographic grids with user call records and clustering abnormal events can more accurately identify network problems. The complaint model prediction and grid clustering result table can more accurately evaluate and optimize the user behavior and network quality of the entire mobile network.
[0116] Furthermore, the impact feature data extracted from the sample is used as input, a preset machine learning model is selected for training and testing, and a user complaint prediction model is obtained, including:
[0117] Compare the accuracy, recall and f1-score evaluation indicators of logistic regression, neural network, LightGBM, multi-layer perceptron and random forest algorithm on sample data. According to the comparison results, determine the weights of various algorithms, and use the integrated algorithm that combines the above algorithms and weights as the preset machine learning model.
[0118] The optimal parameters are selected for training. During the training process, the cross-validation method is used to evaluate and tune the machine learning model to obtain the user complaint model.
[0119] Logistic regression is a statistical method for binary classification problems. It maps the output of linear regression to between 0 and 1 through a logical function and is often used to solve classification problems. Neural networks, especially neural networks in deep learning, are computational models that simulate the structure of neurons in the human brain and predict or classify by learning the complex relationship between input and output. LightGBM is an algorithm framework that includes models such as GBDT models, random forests, and logistic regression, and is usually used in scenarios such as binary classification, multi-classification, and sorting. Multilayer perceptron (MLP) is a type of neural network that includes at least one hidden layer containing multiple neurons for processing more complex nonlinear problems. Random forest is an ensemble learning method that improves the accuracy and stability of predictions by constructing multiple decision trees and outputting the mode of the class prediction results of these trees. These algorithms have their own characteristics. After an overall evaluation of the embodiment of the present disclosure, the final model selects an integrated algorithm that integrates the above five algorithms and uses random forest as a higher weight. According to the comparison results, the weights of various algorithms are determined, and the integrated algorithm that integrates the above various algorithms and weights is used as a preset machine learning model.
[0120] In another embodiment, a voting strategy can also be adopted to obtain the final prediction result. That is, for a given sample, a probability threshold is set, and each model outputs the prediction probability in turn. The number of models whose prediction probability is greater than the threshold is counted. If more than half (three or more) of the prediction probabilities are greater than the threshold, the final prediction result is a positive class (with a tendency to complain), otherwise it is predicted to be a negative class (no tendency to complain). The optimal parameters are selected through traversal for training, and the cross-validation method is used to evaluate and tune the model during the training process. Finally, the accuracy of bringing the test set into verification is 93%, and the verification results are shown in Figure 2 The effect of the integrated classifier is shown in Figure 3 The precision-recall curve of Figure 4 ROC curve of . Figure 3 The precision and recall are shown by Figure 3 It can be seen that AUPRC (Area Under the Precision-Recall Curve) = 0.71, F1-score = 0.66. Figure 4 The ROC curve (Receiver Operating Characteristic Curve) is shown. Figure 4 It can be seen that the area under the ROC curve is 0.76, that is, AUC = 0.76; Fβ-Score = 0.65.
[0121] By comparing multiple classification algorithms, we continuously weighed and tried, and then selected the algorithm with better results for integration and designed weights, which effectively improved the prediction accuracy of the model.
[0122] After model training, the details of users with high probability of complaints can be obtained according to the two complaint prediction models. The complaint prediction result table is shown in Table 3. Table 3 shows the main output fields of complaint prediction and lists the probability of users making complaints in voice or online. Problem analysis can also be performed based on the corresponding data indicators of these users.
[0123] Table 3 Main output fields of complaint prediction
[0124] user City Predicting complaint types Predicting the probability of complaints 13xxxxx xx voice 95% 13xxxxx xx Internet Access 87% 13xxxxx xx Internet Access 62%
[0125] According to the user details and corresponding problems found, the problems are handled and the users are comforted in advance to reduce the user complaint rate.
[0126] The disclosed embodiment breaks down data silos, integrates multiple data sources such as basic user information, signaling data, user data, and complaint data, conducts multi-dimensional mining and analysis, and achieves a more objective network evaluation that is closer to user perception. Based on various data sources, it fully mines user behavior events, analyzes and calculates user personal characteristics, usage habits, and the occurrence of poor quality events, and comprehensively analyzes the network situation in dimensions such as the location of occurrence, number of occurrences, and frequency of occurrence. In addition, through big data and artificial intelligence, self-developed poor quality algorithms are used to establish grid clustering analysis methods and complaint prediction models to more accurately evaluate and optimize the user behavior and network quality of the entire mobile network. A more objective network evaluation that is closer to user perception is achieved.
[0127] like Figure 5 As shown, the second embodiment of the present disclosure further provides a mobile network evaluation method based on user perception, including:
[0128] 1. Data Collection
[0129] (1) User basic information: Collect user characteristic data, including user number, age, gender, terminal, package, call charges, voice and data usage, etc.
[0130] (2) Complaint data: The complaint system regularly obtains all complaint information from existing users due to network issues. The fields to be stored include user number, complaint time, complaint location, customer complaint description, complaint problem location, and service category.
[0131] (3) User perception data: 5G user XDR, IMS XDR, call MOS value and other data closely related to user perception are obtained from the upstream system. The stored fields include user number, time, service type, key indicators, user's cell, etc.
[0132] 2. Data preprocessing
[0133] Data preprocessing mainly includes data auditing, data cleaning, data transformation, data reduction and data exploration.
[0134] (1) Data audit, to verify the legality, consistency, and integrity of data;
[0135] (2) Clean the audited data, including missing value processing, duplicate value processing, and outlier processing;
[0136] (3) Perform data transformation on special types of indicators to obtain features that can be understood by the model. Use word2vec to convert character features into numerical features, such as gender, package, etc. Then reduce the data and complete the initial dimensionality reduction of the indicators by analyzing the correlation between the indicators. For example, if the Pearson correlation coefficient between the upstream MOS performance and the downstream MOS performance is 1, then select one of them as the model input.
[0137] (4) Based on the complaint data, the root causes of the complaints are extracted and expert annotation of the root causes of poor quality is completed.
[0138] (5) Complete basic statistical analysis and exploration of existing data, understand the distribution of user voice indicators and data indicators, and make preliminary sorting and classification of the data to prepare for subsequent data preprocessing and feature engineering.
[0139] Through exploration, we finally selected six events, namely, call drop, voice network failure, voice quality, poor coverage, video usage, and web page usage, as user-perceived evaluation events. The evaluation criteria for each type of poor quality event are shown in Table 1 in Example 1. For different data types, Hive+Spark computing technology is used to implement data analysis and mining.
[0140] 3. Raster cluster analysis
[0141] This step realizes the clustering of single-user poor quality services in the grid dimension in combination with the cell where the event occurred. First, the geographic space is divided into a series of uniform grid systems, and the size of each grid is defined as M*N. The position of each grid can be represented by the longitude and latitude of any point in the rectangle. Here, the longitude and latitude of the lower left corner (i.e., northwest corner) are used for definition. Based on the preprocessed poor quality data, the location of the cell where the poor quality event occurred is converted into longitude and latitude and placed on the map. Then calculate the number of users and times of poor quality events in each grid, as well as the proportion of poor quality events to all events. At this point, if the number of poor quality events is divided into grades and represented by different colors, a grid map is obtained.
[0142] By combining user service call records with the grid, the specific location of each poor quality service event can be accurately located. The distribution pattern of the problem area can be effectively identified, and the analysis can be performed to determine whether the user has passed through the poor quality area and the frequency of passing through, so as to prepare for subsequent complaint prediction and analysis.
[0143] 4. Establishment of complaint prediction model
[0144] Feature selection and extraction were performed on the pre-processed data. By combining data exploration results and expert experience, based on user age, gender, package, online time, network type, voice usage and Internet usage on the same day and in the past three days, the frequency of occurrence of six poor quality events, the frequency of complaints in the current month and in the past six months, and the relationship between users and poor quality areas, L1 regularization, t-Test, decision tree feature importance ranking, and nonlinear feature construction methods were used. After combining expert experience, paper research, and multiple training verifications, N features were finally selected. The root causes of poor quality marked by experts were used as user labels for training data, and model training was performed for two types of behaviors: voice complaints and data complaints.
[0145] After completing the feature engineering, the generated vector data is used as input, and the positive and negative sample data come from random non-complaint users and complaining users, respectively. Due to the uneven distribution of complaints and non-complaints, SMOT oversampling is used to solve the sample imbalance problem. By comparing the accuracy, recall rate, f1-score and other evaluation indicators of several algorithms such as logistic regression, neural network, LightGBM, multi-layer perceptron and random forest algorithm on sample data, the final model selection integrates the above five algorithms after overall evaluation. And a voting strategy is adopted to obtain the final prediction result. That is, for a given sample, a probability threshold is set, and each model outputs the prediction probability in turn. The number of models with prediction probabilities greater than the threshold is counted. If more than half (three or more) of the prediction probabilities are greater than the threshold, the final prediction result is a positive class (with a tendency to complain), otherwise it is predicted to be a negative class (no tendency to complain). The optimal parameters are selected for training through traversal, and the cross-validation method is used to evaluate and tune the model during the training process. Finally, the accuracy of bringing the test set into verification reached 93%. See the verification results. Figure 2 The effect of the integrated classifier is shown in Figure 3 The precision-recall curve of Figure 4 ROC curve of . Figure 3 The precision and recall are shown by Figure 3 It can be seen that AUPRC = 0.71, F1-score = 0.66. Figure 4 The ROC curve is shown. Figure 4 It can be seen that the area under the ROC curve is 0.76, that is, AUC = 0.76; Fβ-Score = 0.65.
[0146] By regularly collecting complaint lists, updating sample data, and training to generate the latest model. Based on the trained model, the user indicator data is input to predict the complaint probability of sub-healthy users every day, and the details of high-probability complaint users are obtained. According to the user details and corresponding problems found, the problems are handled in advance and the users are appeased to reduce the user complaint rate.
[0147] 5. Result output
[0148] (1) Grid clustering result table
[0149] The grid clustering result table is shown in Table 2 in Example 1. Table 2 shows the main output fields of grid clustering, lists the grid ID, longitude and latitude, city of origin and included cells, counts the number of poor quality events and the number of poor quality users that occurred in the grid, and whether the grid has been a poor quality grid for three consecutive days. The user details include all user details data that appear in the grid and have poor service quality.
[0150] (2) Complaint prediction results table
[0151] The complaint prediction result table is shown in Table 3. Table 3 shows the main output fields of complaint prediction and lists the probability of users making complaints via voice or Internet.
[0152] The method of the disclosed embodiment has been applied to the grid map and complaint prediction analysis in the integrated network management system of the northern region. Figure 6 The raster map shown and Figure 7 Complaint prediction analysis results shown.
[0153] 1. Grid map: The system uses a 300m*300m grid to display the distribution of sub-healthy users. Different colors distinguish the number of abnormal users, which is convenient for the network optimization department to carry out accurate network optimization layout according to the region. It supports exporting the global grid distribution and the user situation in a single grid.
[0154] 2. Complaint prediction analysis: Daily input of user index data is used to predict the probability of complaints from the user population, and a list of users with a high probability of complaints is obtained. Based on the user details and corresponding problems found, problem handling and user appeasement are carried out in advance to reduce the user complaint rate.
[0155] The third embodiment of the present disclosure also provides a mobile network evaluation system based on user perception, such as Figure 8 As shown, the system comprises:
[0156] A collection module 11 is configured to collect user data, including basic user information, complaint data, and user perception data;
[0157] A preprocessing module 12, which is configured to preprocess the collected user data, including data auditing, data cleaning and data transformation, and transforming the user data into index data that can be processed by the model;
[0158] A selection module 13, which is configured to select influencing features for each indicator of the processed user data;
[0159] A training module 14 is configured to construct positive and negative sample data after the selection module completes the selection of influencing features, wherein the positive and negative sample data are respectively from random non-complaining users and complaining users, and the influencing feature data extracted from the samples is used as input, and a preset machine learning model is selected for training and testing to obtain a user complaint prediction model;
[0160] The prediction module 15 is configured to predict the complaint probability of the sub-healthy user population based on the trained user complaint prediction model, input the user data to be identified, and obtain the details of the high-probability complaint users.
[0161] Furthermore, the system further comprises a classification module 16:
[0162] The classification module is configured to extract the root causes of complaints from complaint data and complete the labeling of root causes of poor quality; and
[0163] Based on the results of poor quality root cause annotation, user data is sorted and classified according to voice indicators and data indicators;
[0164] The training module 14 is also configured to use the marked root causes of poor quality as user labels for training data, and to perform model training on two types of behaviors, voice complaints and data complaints, respectively, to obtain a user voice complaint prediction model and a user data complaint prediction model.
[0165] Furthermore,
[0166] The user basic information includes user number, age, gender, terminal, package, call fee, voice and traffic usage information;
[0167] The complaint data is the complaint data of existing network users due to network reasons obtained from the complaint system, including user number, complaint time, complaint location, customer complaint description, complaint problem location and service category;
[0168] The user perception data is the data related to user perception in the user XDR, IMS XDR and call MOS value obtained from the upstream system, including user number, time, service type, key indicators and the cell where the user is located.
[0169] Furthermore, the preprocessing module 12 is also configured to perform data reduction and data exploration;
[0170] The data reduction includes completing the indicator dimension reduction by analyzing the correlation between various indicators of the user data;
[0171] The data exploration includes determining the distribution of user voice indicators and data indicators and excluding useless indicators.
[0172] Furthermore, the system further comprises a grid clustering module 17:
[0173] The grid clustering module 17 is configured to select six events, namely, call drop, voice network failure, poor voice quality, weak coverage, abnormal video usage, and abnormal web page usage, as poor quality events perceived by users; and
[0174] Combined with the cell where the poor quality evaluation event occurred, the clustering of single-user poor quality services in the grid dimension is realized, including:
[0175] Divide the geographic space on the map into a series of uniform grid systems, define the size of each grid as M*N, and use the longitude and latitude of a rectangle to represent the location of each grid;
[0176] Based on the pre-processed user data, the location of the cell where the quality difference event occurred is converted into longitude and latitude and placed on a grid system map;
[0177] Calculate the number of users and times of poor quality events in each grid, as well as the proportion of poor quality events to all events;
[0178] Whether the user passes through the poor quality area and the frequency of passing through the poor quality area are used as the relationship indicators between the user and the poor quality area.
[0179] Furthermore, the selection module 13 is specifically configured as follows:
[0180] After data preprocessing and clustering in the grid dimension, various features to be selected are obtained from various indicators, including user age, gender, package, network time, network type, voice usage, Internet usage, frequency of occurrence of poor quality events on the day and in the past three days, frequency of complaints in the month and in the past six months, and the relationship between users and poor quality areas;
[0181] The influencing features of N models are selected from various features to be selected through a preset feature selection method.
[0182] Furthermore, the grid clustering module 17 is also configured as follows:
[0183] Output the grid clustering result table based on the number of users and times of poor quality events in each grid and the proportion of poor quality events to all events; and
[0184] The network quality is evaluated according to the grid clustering result table.
[0185] Furthermore, the training module 14 is specifically configured as follows:
[0186] Compare the accuracy, recall and f1-score evaluation indicators of logistic regression, neural network, LightGBM, multi-layer perceptron and random forest algorithm on sample data. According to the comparison results, determine the weights of various algorithms, and use the integrated algorithm that combines the above algorithms and weights as the preset machine learning model.
[0187] The optimal parameters are selected for training. During the training process, the cross-validation method is used to evaluate and tune the machine learning model to obtain the user complaint model.
[0188] The user-perceived mobile network evaluation system of the disclosed embodiment is used to implement the user-perceived mobile network evaluation method in method embodiment 1 and embodiment 2, so the description is relatively simple. For details, please refer to the relevant description in the previous method embodiment, which will not be repeated here.
[0189] In addition, if Fig. 9 As shown, the fourth embodiment of the present disclosure further provides an electronic device, including a memory 100 and a processor 200, wherein the memory 100 stores a computer program, and when the processor 200 runs the computer program stored in the memory 100, the processor 200 executes the above-mentioned various possible methods.
[0190] The memory 100 is connected to the processor 200. The memory 100 may be a flash memory, a read-only memory or other memory, and the processor 200 may be a central processing unit or a single-chip microcomputer.
[0191] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to perform the above-mentioned various possible methods.
[0192] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), Digital Video Disc (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.
[0193] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.
Claims
1. A mobile network evaluation method based on user perception, characterized in that: The method comprises: Collect user data, including basic user information, complaint data, and user perception data; Preprocess the collected user data, including data auditing, data cleaning and data transformation, to transform the user data into indicator data that can be processed by the model; Perform influencing feature selection on each indicator of the processed user data; After completing the selection of influencing features, construct positive and negative sample data, which come from random users who have not complained and users who have complained, respectively. Take the influencing feature data extracted from the samples as input, select a preset machine learning model for training and testing, and obtain a user complaint prediction model; Based on the trained user complaint prediction model, the user data to be identified is input to predict the complaint probability of the sub-healthy user population, and the details of high-probability complaint users are obtained.
2. The method according to claim 1, characterized in that The method further comprises: Based on the complaint data, the root causes of complaints are extracted and the root causes of poor quality are marked; Based on the results of poor quality root cause annotation, user data is sorted and classified according to voice indicators and data indicators; The annotated root causes of poor quality are used as user labels for training data, and model training is performed on two types of behaviors: voice complaints and data complaints, respectively, to obtain user voice complaint prediction models and user data complaint prediction models.
3. The method according to claim 1, characterized in that The user basic information includes user number, age, gender, terminal, package, call fee, voice and traffic usage information; The complaint data is the complaint data of existing network users due to network reasons obtained from the complaint system, including user number, complaint time, complaint location, customer complaint description, complaint problem location and service category; The user perception data is data related to user perception in the user scalable threat detection and response XDR, IP multimedia system IMS XDR, and call mean opinion score MOS value obtained from the upstream system, including user number, time, service type, key indicators and the cell where the user is located.
4. The method according to claim 1, characterized in that: The preprocessing also includes data reduction and data exploration; The data reduction includes completing the indicator dimension reduction by analyzing the correlation between various indicators of the user data; The data exploration includes determining the distribution of user voice indicators and data indicators and excluding useless indicators.
5. The method according to any one of claims 2 to 4, characterized in that: The method further comprises: Select the following six events as the poor quality events perceived by users: dropped calls, voice network failure, poor voice quality, weak coverage, abnormal video usage, and abnormal web page usage; Combined with the cell where the poor quality evaluation event occurred, the clustering of single-user poor quality services in the grid dimension is realized, including: Divide the geographic space on the map into a series of uniform grid systems, define the size of each grid as M*N, and use the longitude and latitude of a rectangle to represent the location of each grid; Based on the pre-processed user data, the location of the cell where the quality difference event occurred is converted into longitude and latitude and placed on a grid system map; Calculate the number of users and times of poor quality events in each grid, as well as the proportion of poor quality events to all events; Whether the user passes through the poor quality area and the frequency of passing through the poor quality area are used as the relationship indicators between the user and the poor quality area.
6. The method according to claim 5, characterized in that The selecting of influencing features for each indicator of the processed user data includes: After data preprocessing and clustering in the grid dimension, various features to be selected are obtained from various indicators, including user age, gender, package, network time, network type, voice usage, Internet usage, frequency of occurrence of poor quality events on the day and in the past three days, frequency of complaints in the month and in the past six months, and the relationship between users and poor quality areas; The influencing features of N models are selected from various features to be selected through a preset feature selection method.
7. The method according to claim 5, characterized in that The method further comprises: Output the grid clustering result table based on the number of users and times of poor quality events in each grid and the proportion of poor quality events to all events; The network quality is evaluated according to the grid clustering result table.
8. The method according to claim 1, characterized in that The impact feature data extracted from the sample is used as input, a preset machine learning model is selected for training and testing, and a user complaint prediction model is obtained, including: Compare the accuracy, recall and F1-score evaluation indicators of logistic regression, neural network, LightGBM, multi-layer perceptron and random forest algorithm on sample data. According to the comparison results, determine the weights of various algorithms, and use the integrated algorithm that combines the above algorithms and weights as the preset machine learning model. The optimal parameters are selected for training. During the training process, the cross-validation method is used to evaluate and tune the machine learning model to obtain the user complaint model.
9. A mobile network evaluation system based on user perception, characterized in that: The system comprises: A collection module, which is configured to collect user data, including basic user information, complaint data, and user perception data; A preprocessing module is configured to preprocess the collected user data, including data auditing, data cleaning and data transformation, so as to transform the user data into indicator data that can be processed by the model; A selection module configured to select influencing features for each indicator of the processed user data; A training module is configured to construct positive and negative sample data after the selection module completes the selection of influencing features, wherein the positive and negative sample data are respectively from random non-complaining users and complaining users, and the influencing feature data extracted from the samples is used as input, and a preset machine learning model is selected for training and testing to obtain a user complaint prediction model; The prediction module is set to predict the complaint probability of sub-healthy users based on the trained user complaint prediction model, and inputs the user data to be identified to predict the complaint probability of the sub-healthy user population, and obtains the details of high-probability complaint users.
10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the mobile network evaluation method based on user perception as described in any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for evaluating a mobile network based on user perception as claimed in any one of claims 1 to 8 is implemented.
