A service-aware evaluation model training method, a service-aware evaluation method, and related devices

Abstract

This application discloses a service awareness assessment model training method, a service awareness assessment method, and related equipment. The training method includes: acquiring XDR call detail records (CDRs) of multiple user services to form a first sample set; obtaining the service latency and downlink rate of each user service based on the first sample set; labeling the first sample set with positive and negative samples based on the service latency and downlink rate to obtain a second sample set; extracting features from the second sample set to obtain a third sample set; and training the model using the third sample set to obtain a service awareness assessment model. Using the embodiments of this application, accurate user awareness labels are provided for the sample set, enabling the model to effectively learn the features of positive and negative samples and improve the model's assessment accuracy.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a service awareness assessment model training method, a service awareness assessment method, and related equipment. Background Technology

[0002] With the development of mobile internet, users' demands for mobile communication service quality are increasing. To improve user experience and reduce complaints, operators need to understand users' service perception and quickly locate and resolve problems. Currently, intelligent models are mainly used to evaluate user service perception; however, the training samples of these models generally lack clear user perception labels, making it difficult to accurately reflect users' actual service perception. Summary of the Invention

[0003] This application provides a training method for a business perception assessment model, a business perception assessment method, and related equipment to solve the problem that the lack of clear user perception labels in the prior art makes it difficult to truly reflect the user's actual business perception.

[0004] To achieve the above objectives, embodiments of this application provide a method for training a business-aware evaluation model, comprising: Obtain XDR call detail records for multiple user services to form the first sample set; Based on the first sample set, the service latency and downlink rate of each user service are obtained; The first sample set is labeled with positive and negative samples based on the service latency and the downlink rate to obtain the second sample set; Feature extraction is performed on the second sample set to obtain the third sample set; The business perception evaluation model is obtained by using the third sample set for model training.

[0005] As an improvement to the above scheme, the step of labeling the first sample set with positive and negative samples based on the service latency and the downlink rate to obtain the second sample set includes: The first sample set is classified to obtain one or more first sub-sample sets; For each of the first subsets, the coordinate points within the first subset are fitted using service latency and downlink rate as coordinate points to obtain the fitted curve of the first subset. Based on the slope of the tangent line of the fitted curve of the first subsample set, the network perception anomalies of the fitted curve of the first subsample set are obtained. Based on the network perception anomalies in the fitted curve of the first sub-sample set, positive and negative sample labels are applied to the first sub-sample set to obtain the second sub-sample set. One or more of the second sub-sample sets are used to construct the second sample set.

[0006] As an improvement to the above scheme, the step of obtaining the network-aware anomalies of the fitted curve of the first sub-sample set based on the slope of the tangent line of the fitted curve of the first sub-sample set includes: The average downlink rate of the fitted curve of the first subsample set is obtained by averaging the highest and lowest downlink rates. The corresponding coordinate point is then used as the target coordinate point. For the fitted curve of the first subsample set, calculate the tangent slope of the coordinate points located below the target coordinate point, and select the coordinate point corresponding to the largest tangent slope as the network perception anomaly point of the fitted curve of the first subsample set.

[0007] As an improvement to the above scheme, before labeling the first sample set with positive and negative samples based on the service latency and the downlink rate to obtain the second sample set, the service awareness evaluation model training method further includes: Based on the first sample set, the wireless side latency ratio of each user service is obtained; Based on the distribution of the wireless side latency proportions of all the aforementioned user services, the proportion threshold is obtained; Based on the aforementioned percentage threshold, XDR call detail records that do not meet the preset percentage threshold are deleted from the first sample set.

[0008] As an improvement to the above scheme, the step of obtaining the percentage threshold based on the distribution of the wireless side latency percentages of all the user services includes: Plot the wireless latency percentage of all the user services on a number axis; The number axis is divided into multiple equal-width intervals, and the upper limit of the equal-width interval with the highest wireless side latency ratio is used as the ratio threshold.

[0009] As an improvement to the above solution, the step of obtaining XDR call detail records (CDRs) of multiple user services to form a first sample set includes: For different scenarios, use base stations covering the scenario to identify resident users in different scenarios; The XDRs of user services for multiple resident users in different scenarios are obtained to form the first sample set.

[0010] As an improvement to the above solution, the business perception evaluation model is a random forest model; The step of using the third sample set to train the model and obtain the business perception evaluation model includes: Samples with replacement are drawn from the third sample set as training data for the decision tree. The training data of the decision tree is used as the root node, and the decision tree is recursively constructed until a preset stopping condition is reached to obtain the decision tree. Multiple decision trees are integrated to form the random forest model.

[0011] As an improvement to the above solution, the training method for the business perception evaluation model further includes: The hyperparameters of the random forest model are optimized to obtain the target parameters; Based on the target parameters and the random forest model, a target random forest model is obtained; wherein, the target random forest model is used for user business perception evaluation.

[0012] To achieve the above objectives, embodiments of this application also provide a business awareness assessment method, including: Obtain the XDR call detail records (CDRs) of the user's services to be evaluated; Feature extraction is performed on the XDR call detail records to obtain the target features; The target features are input into the business perception evaluation model, and the perception evaluation results of the user's business to be evaluated are output. The business perception assessment model is trained using the business perception assessment training method described above.

[0013] To achieve the above objectives, embodiments of this application also provide a business awareness evaluation model training apparatus, comprising: The first acquisition module is used to acquire XDR call detail records of multiple user services to form the first sample set; The calculation module is used to obtain the service latency and downlink rate of each user service based on the first sample set. The annotation module is used to annotate the first sample set with positive and negative samples based on the service latency and the downlink rate to obtain a second sample set; The first extraction module is used to extract features from the second sample set to obtain the third sample set; The training module is used to train the model using the third sample set to obtain the business perception evaluation model.

[0014] To achieve the above objectives, embodiments of this application also provide a business awareness assessment device, comprising: The second acquisition module is used to acquire the XDR call detail records of the user's services to be evaluated; The second extraction module is used to extract features from the XDR call detail record to obtain target features; The evaluation module is used to input the target features into the business perception evaluation model and output the perception evaluation result of the user's business to be evaluated. The business perception evaluation model is trained by the aforementioned business perception evaluation model training device.

[0015] To achieve the above objectives, embodiments of this application also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the service awareness evaluation model training method or the service awareness evaluation method described above.

[0016] To achieve the above objectives, embodiments of this application also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the above-described service awareness evaluation model training method, or the above-described service awareness evaluation method.

[0017] To achieve the above objectives, embodiments of this application also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the business awareness evaluation model training method or the business awareness evaluation method described above.

[0018] Compared with existing technologies, the present application provides a service perception assessment model training method, service perception assessment method, and related equipment. This involves acquiring XDR call detail records (CDRs) of multiple user services to form a first sample set; obtaining the service latency and downlink rate of each user service based on the first sample set; labeling the first sample set with positive and negative samples based on the service latency and downlink rate to obtain a second sample set; extracting features from the second sample set to obtain a third sample set; and training the model using the third sample set to obtain a service perception assessment model for evaluating user service perception. Therefore, the present application's method of labeling positive and negative samples with service latency and downlink rate provides accurate user perception labels for the sample set, enabling the model to effectively learn the features of positive and negative samples. This not only improves model training efficiency but also enhances the model's assessment accuracy, ultimately reflecting the user's actual service perception. Attached Figure Description

[0019] Figure 1 This is a flowchart of a business perception evaluation model training method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a three-way handshake provided in an embodiment of this application; Figure 3This is a schematic diagram of a fitting curve provided in an embodiment of this application; Figure 4 This is another schematic diagram of a fitting curve provided in an embodiment of this application; Figure 5 This is a flowchart of a business perception assessment method provided in an embodiment of this application; Figure 6 This is a structural block diagram of a business perception evaluation model training device provided in an embodiment of this application; Figure 7 This is a structural block diagram of a service awareness assessment device provided in an embodiment of this application; Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0021] In the description of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0022] In this application description, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0023] In this application description, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." The term "and / or" means at least one of the connected objects, such as A and / or B, indicating three cases: including only A, only B, and both A and B. Unless otherwise stated, the term "multiple" means two or more.

[0024] See Figure 1 , Figure 1 This is a flowchart of a business perception evaluation model training method provided in an embodiment of this application. The business perception evaluation model training method includes: S11. Obtain XDR call detail records for multiple user services to form the first sample set; It is worth noting that a user service generates one XDR (User Signaling Data) call detail record. By obtaining XDR call detail records of multiple user services, a first sample set is formed; each sample in this first sample set is an XDR call detail record.

[0025] In one specific embodiment, each XDR call detail record contains the following information: start time, end time, cell ID, user IMSI (International Mobile Subscriber Identity), service type, number of uplink IP (Internet Protocol) packets, number of downlink IP packets, number of uplink TCP (Transmission Control Protocol) out-of-order packets, number of downlink TCP out-of-order packets, number of uplink TCP retransmission packets, number of downlink TCP retransmission packets, TCP connection establishment response delay, TCP connection establishment confirmation delay, total delay from successful TCP connection establishment to the first transaction request, delay of the last HTTP (Hypertext Transfer Protocol) content packet, etc.

[0026] S12. Based on the first sample set, obtain the service latency and downlink rate of each user service; It is worth noting that each XDR call detail record contains corresponding service metrics related to the user's business. By analyzing these service metrics, the service latency and downlink speed of the corresponding user's business can be obtained.

[0027] Specifically, when a user initiates a service request, a TCP connection must be established first. Data transmission can only begin after the connection is successfully established. Service latency refers to the time from the TCP connection establishment response to the last HTTP content packet.

[0028] like Figure 2 This diagram illustrates the TCP three-way handshake process. UE represents the user equipment, such as a mobile phone, computer, or smartwatch. N3 is the interface between the core network and the RAN (Radio Access Network), where the latency counter is located. SP is the application server, providing services to the user. SYN stands for Synchronize Sequence Numbers, SYN.ACK represents the Synchronize Sequence Numbers Acknowledgement character, ACK represents the Acknowledgement character, GET represents a GET request, and TCP.No.1 (200 OK) represents the first transaction request. To clearly represent the latency segment, the latency of the three-way handshake is defined as follows, based on the object counted by the latency counter: The first and second handshakes: N3->SP->N3, are used to provide feedback on the status of 5GC (core network) to the application server, corresponding to the TCP connection establishment response latency in the XDR call detail record; The second and third handshakes: N3->UE->N3, are used to provide feedback on the actual situation on the NG RAN (Radio Access Network) side, corresponding to the TCP connection establishment confirmation delay in the XDR call detail record.

[0029] In one specific embodiment, service latency In the formula, TCP connection establishment response latency; TCP connection establishment confirmation delay The total latency from successful TCP connection establishment to the first transaction request. This is the delay for the last HTTP content packet.

[0030] Specifically, the downlink rate is obtained based on the number of downlink IP packets and the latency of the last HTTP content packet recorded in the XDR call detail record; where downlink rate = downlink traffic × 8 / latency of the last HTTP content packet. Downlink traffic can be obtained from the number of downlink IP packets, for example, downlink traffic = number of downlink IP packets × average IP packet size.

[0031] S13. Label the first sample set with positive and negative samples according to the service latency and the downlink rate to obtain the second sample set; It's worth noting that the direct indicators affecting service perception are primarily latency and speed, while the indicators on XDR call detail records cannot determine the user's perception. Although latency and speed are not directly correlated, in air interface scenarios, both change synchronously with channel quality: deteriorating air interface channel quality means increased latency and slower download speeds. Based on this, this application's embodiment uses service latency and downlink speed as anchor points, utilizing the indicator anomalies caused by changes in air interface quality to label each sample in the first sample set with positive and negative samples. These labeled samples form the second sample set.

[0032] S41. Extract features from the second sample set to obtain the third sample set; It is worth noting that each XDR call detail record (CDR) has over a hundred fields on its service side, making model training using XDR CDRs extremely labor-intensive. Therefore, this embodiment extracts features from each sample in the second sample set, and these extracted sample features constitute the third sample set. In other words, each sample in the third sample set includes sample features and corresponding positive and negative sample labels.

[0033] Features extracted from XDR call detail records include, but are not limited to: start time, end time, cell ID, user IMSI, service type, number of uplink IP packets, number of downlink IP packets, number of uplink out-of-order TCP packets, number of downlink out-of-order TCP packets, number of uplink TCP retransmission packets, number of downlink TCP retransmission packets, TCP connection establishment response latency, TCP connection establishment confirmation latency, total latency from successful TCP connection establishment to the first transaction request, and latency of the last HTTP content packet.

[0034] S15. Use the third sample set to train the model and obtain the business perception evaluation model.

[0035] In this embodiment, the third sample set is ultimately used for model training to obtain the business perception evaluation model. The model used for training is not specifically limited; it can be a traditional machine learning model such as logistic regression or random forest, or a deep learning model such as a multilayer perceptron or long short-term memory network. Taking a multilayer perceptron as an example, this structure includes an input layer, a hidden layer, and an output layer. The sample features from the third sample set are input through the input layer, passed through the hidden layer, and output as predicted values ​​through the output layer. These predicted values ​​are compared with the positive and negative sample labels corresponding to the sample features. Through iterative training, a trained neural network model is obtained, which is the business perception evaluation model.

[0036] This application embodiment uses service latency and downlink rate to label positive and negative samples, providing accurate user-perceived labels for the sample set. This enables the model to effectively learn the features of positive and negative samples, improving not only the model training efficiency but also the model evaluation accuracy, ultimately reflecting the user's actual service perception.

[0037] In an optional embodiment, the step of labeling the first sample set with positive and negative samples based on the service latency and the downlink rate to obtain a second sample set includes: The first sample set is classified to obtain one or more first sub-sample sets; For each of the first subsets, the coordinate points within the first subset are fitted using service latency and downlink rate as coordinate points to obtain the fitted curve of the first subset. Based on the slope of the tangent line of the fitted curve of the first subsample set, the network perception anomalies of the fitted curve of the first subsample set are obtained. Based on the network perception anomalies in the fitted curve of the first sub-sample set, positive and negative sample labels are applied to the first sub-sample set to obtain the second sub-sample set. One or more of the second sub-sample sets are used to construct the second sample set.

[0038] This application pre-classifies the first sample set, so that each first sub-sample set has its own network perception anomalies. This provides differentiated annotations for different types of first sub-sample sets, accurately identifying which data points represent abnormal situations in user business perception and avoiding perception bias. For example, the first sample set can be classified according to at least one or more of scenario, business type, and business time, which can effectively avoid perception bias caused by differences in business attributes, scenario attributes, time attributes, etc., and lay an accurate and reliable data foundation for subsequent model training.

[0039] In one specific embodiment, services are categorized according to their type: small packet services and large packet services. Small packet services include instant messaging (downlink traffic <500kb), web browsing (downlink traffic <500kb), games, and payment services. Large packet services include instant messaging (downlink traffic ≥500kb), web browsing (downlink traffic ≥500kb), and video services. These two types of services have different requirements for downlink speed and service latency, so the judgment criteria (i.e., network-aware anomalies) are also different.

[0040] Next, using service latency and downlink rate as coordinate points, the coordinate points within the first subset are fitted to obtain the fitting curve of the first subset. The fitting curve of the first subset is the downlink rate-service latency curve.

[0041] The fitted curve of the first subset can reflect the downlink speed and latency of the user when performing business in this scenario. Under normal circumstances, the downlink speed and latency will remain in a relatively stable range. However, as the air interface quality deteriorates, the downlink speed and latency also change. Therefore, this embodiment of the application then uses the tangent slope to find the inflection point of air interface quality deterioration as a network perception anomaly point, and uses this network perception anomaly point for positive and negative sample labeling.

[0042] In an optional embodiment, obtaining the network-aware anomalies of the fitted curve of the first subset of samples based on the slope of the tangent line of the fitted curve of the first subset of samples includes: The average downlink rate of the fitted curve of the first subsample set is obtained by averaging the highest and lowest downlink rates. The corresponding coordinate point is then used as the target coordinate point. For the fitted curve of the first subsample set, calculate the tangent slope of the coordinate points located below the target coordinate point, and select the coordinate point corresponding to the largest tangent slope as the network perception anomaly point of the fitted curve of the first subsample set.

[0043] In order to avoid misjudgment caused by a rapid decrease in rate under normal fluctuations, the embodiments of this application only calculate the tangent slope for coordinate points below the average downlink rate, which can effectively filter out the interference caused by normal fluctuations and improve the accuracy and reliability of anomaly judgment in business perception assessment.

[0044] In one specific embodiment, for any first subset of samples, the service latency and corresponding downlink rate of each sample within it are used as coordinate points and plotted in a Cartesian coordinate system. It is assumed that each coordinate point is defined as... ,in Indicates service latency. This represents the corresponding downlink rate. It is expressed using an m-th order polynomial. Fit these coordinate points:

[0045] in, , , ..., , These are the coefficients of the polynomial.

[0046] The coefficients of the polynomial are solved using the least squares method, as follows: Constructing a design matrix and observation vector : ,

[0047] Based on the coefficient vector A = =

[0048] Find the polynomial coefficients , ,…, Substitute into the polynomial :

[0049] The fitted curve for the first sample set is obtained, as follows: Figure 3 .

[0050] Find the highest and lowest downlink rates on the fitted curve of the first sample set, calculate the average downlink rate, and use the corresponding coordinate points as the target coordinate points. For example Figure 3 The highest downlink rate is 0.5, and the lowest downlink rate is 0. Therefore, the average downlink rate = (0.5 + 0) / 2 = 0.25. Draw a line parallel to the x-axis at the target coordinate point, as shown below. Figure 4 The slope of the tangent line at the coordinate point below the target coordinate point is obtained by differentiating the curve function below the dashed line:

[0051] For each Calculate the slope of the tangent line at each point on the curve. Finally, the maximum tangent slope Max( The point with the largest tangent slope reflects the point where the downward velocity changes the most. Based on the largest tangent slope, the corresponding... Marked as This coordinate point is the network-aware anomaly point within the first subset of samples: For each XDR call detail record (CDR) in the first subset, positive and negative sample labels are applied. For example, if the following conditions are met... > and < The XDR call detail records are labeled as abnormal, otherwise they are marked as normal. Each XDR call detail record is a sample, and the positive and negative sample labeling of the first subset is now complete.

[0052] In an optional embodiment, before labeling the first sample set with positive and negative samples based on the service latency and the downlink rate to obtain the second sample set, the service awareness evaluation model training method further includes: Based on the first sample set, the wireless side latency ratio of each user service is obtained; Based on the distribution of the wireless side latency proportions of all the aforementioned user services, the proportion threshold is obtained; Based on the aforementioned percentage threshold, XDR call detail records that do not meet the preset percentage threshold are deleted from the first sample set.

[0053] In order to reduce the number of analyses, this application embodiment uses the wireless side latency ratio to delete XDR call detail records that do not meet the preset conditions from the first sample set. This can effectively reduce the data size and focus on the data subset that may have network air interface anomalies, thereby improving the efficiency and relevance of subsequent analyses.

[0054] Specifically, the delay generated by the second and third handshakes is taken as the analysis object, and the proportion of the delay of the second and third handshakes in the total delay of the three handshakes is calculated, that is: wireless side delay proportion = TCP connection establishment confirmation delay / (TCP connection establishment response delay + TCP connection establishment confirmation delay).

[0055] In one optional embodiment, obtaining the percentage threshold based on the distribution of the wireless side latency percentages of all the user services includes: Plot the wireless latency percentage of all the user services on a number axis; The number axis is divided into multiple equal-width intervals, and the upper limit of the equal-width interval with the highest wireless side latency ratio is used as the ratio threshold.

[0056] This application embodiment performs initial screening based on the wireless side latency ratio, and performs secondary screening through curve fitting, thereby efficiently screening and accurately locating XDR call detail records that may affect user service perception from massive XDR data, ensuring a low false positive rate.

[0057] In one specific embodiment, the wireless latency percentage is plotted on a number axis, which is divided into 10 equally wide intervals: [0, 10%, 20%, 30%...100%]. The upper limit of the interval with the highest frequency of wireless latency percentage occurrences is selected as the percentage threshold. For example, if the wireless latency percentage occurs most frequently in the interval [30%, 40%), the value corresponding to 40% is taken as the percentage threshold. If the wireless latency percentage is greater than this threshold, it is considered that there is an anomaly in the current network wireless air interface, and it is retained. Wireless latency percentages less than or equal to this threshold are deleted. That is, wireless latency percentages to the left of 40% on the number axis are deleted, and wireless latency percentages to the right of 40% are retained as the first sample set.

[0058] In one optional embodiment, obtaining XDR call detail records (CDRs) of multiple user services to form a first sample set includes: For different scenarios, use base stations covering the scenario to identify resident users in different scenarios; The XDRs of user services for multiple resident users in different scenarios are obtained to form the first sample set.

[0059] It is worth noting that this application embodiment utilizes scene layers and base station latitude and longitude coordinates, and obtains base stations covering various scenes through geospatial calculations using a Geographic Information System (GIS). It identifies resident users of base stations through the Measurement Report (MR) submitted by the terminal. For example, when a terminal accesses a base station for a duration exceeding a preset time, it can be determined that the terminal's user is a resident user. The resident users of a base station are those resident users within the scene corresponding to that base station. For example, if there are base stations 1 and 2 covering scene 1, the resident users of base station 1 and base station 2 constitute the resident users of scene 1.

[0060] The scenarios described in this application include, but are not limited to, universities, urban villages, industrial parks, shopping malls, and residential areas. By importing the scene layer and base station latitude and longitude coordinates into a GIS platform, cell names with scene labels can be exported, thus obtaining the base stations covering that scene.

[0061] Because some macro base stations at the scene boundaries do not fully cover the entire scene, taking all users under these base stations would include user service perceptions from outside the scene, failing to accurately reflect the actual user service perceptions within that scene. Furthermore, mobile users within the scene only temporarily enter the scene, reflecting only their temporary perceptions and not the long-term status of the wireless base stations, thus failing to reflect the long-term user experience. Therefore, this embodiment utilizes the XDR call detail records (CDRs) of resident users within the scene to characterize the current user group whose service perceptions can be accurately reflected.

[0062] XDR (Extended Call Detail Record) records the signaling data from the start to the end of each user's service request. Each service request is recorded as a separate call detail record (CDR). The service plane data includes signaling information related to the user's service usage, such as IMSI, cell ID, service type, time, traffic, latency, and rate. By matching the IMSI and cell ID of resident users within a scenario, XDR CDRs can be obtained for the user services of resident users in different scenarios.

[0063] In one optional embodiment, the business-aware evaluation model is a random forest model; The step of using the third sample set to train the model and obtain the business perception evaluation model includes: Samples with replacement are drawn from the third sample set as training data for the decision tree. The training data of the decision tree is used as the root node, and the decision tree is recursively constructed until a preset stopping condition is reached to obtain the decision tree. Multiple decision trees are integrated to form the random forest model.

[0064] It's worth noting that perception thresholds vary across different scenarios and services. Even within the same scenario and service, perception thresholds (i.e., the aforementioned network perception anomalies) can differ due to variations in location and user demographics. To improve the model's adaptability to different scenarios, service types, and geographical locations, further refinement of the evaluation model is necessary. Random forests, by constructing multiple decision trees, can capture subtle changes in data and flexibly adapt to different perception thresholds. They excel at handling high-dimensional and non-linear data, automatically discovering complex relationships between features, and improving prediction stability and reducing overfitting through majority voting mechanisms. Furthermore, random forests support incremental learning, allowing for rapid model updates as new data is added, ensuring the model remains up-to-date to reflect changes in user behavior. This approach enhances the model's generalization ability and provides tools for understanding the decision-making process, making it highly suitable for addressing diverse perception threshold challenges.

[0065] Specifically, when recursively constructing any decision tree, multiple features are randomly selected at each node and split according to Gini Impurity or Information Gain until a preset stopping condition is met, resulting in a complete decision tree.

[0066] In an optional embodiment, the business-aware evaluation model training method further includes: The hyperparameters of the random forest model are optimized to obtain the target parameters; Based on the target parameters and the random forest model, a target random forest model is obtained; wherein, the target random forest model is used for user business perception evaluation.

[0067] The embodiments of this application can improve the model evaluation accuracy by optimizing the hyperparameters of the random forest model.

[0068] To facilitate understanding, the training process of a specific random forest model will be explained below: 1. Model Initialization The main parameters of the random forest model include: the number of decision trees. Maximum depth of each tree Minimum number of sample splits Minimum number of leaf node samples The number of features randomly selected at each node .

[0069] 2. Model Training (1) Sampling Assume the total number of samples in the third sample set is Label vector For each decision tree ( ), drawn with replacement from the training set The nth sample is used as the training data for this tree. Let the nth sample be... The training data for each decision tree is and .

[0070] (2) Constructing a decision tree For each decision tree Recursively build the tree until a stopping condition (such as maximum depth) is met. or minimum sample size The specific steps are as follows: a. Feature selection At each node, select randomly There are several features, and the optimal splitting feature and splitting point are selected from them. Assume the current node has... There are samples, and the feature set is... ,but:

[0071] in Indicates the first One characteristic, It is the number of features selected randomly.

[0072] For classification problems, the following settings are typically used:

[0073] in It represents the total number of features.

[0074] b. Divide Criteria When selecting the optimal split point at each node, it is necessary to calculate the change in label distribution before and after the split. Commonly used splitting criteria are Gini impurity or information gain. Taking Gini impurity as an example, let's assume the label distribution of the current node is... The impurity of Gini is defined as:

[0075] in, It is a category The proportion in the current node, It represents the total number of categories.

[0076] Choose the feature and split point that maximizes the reduction of Gini impurity. Assume the optimal splitting feature is... The split point is ,but:

[0077] in, and These are the label distributions of the left and right child nodes after the split. and It represents the number of samples in the left and right child nodes.

[0078] c. Recursive construction Based on the selected splitting features and splitting points, the sample is divided into left and right child nodes, and the tree is recursively constructed until the preset stopping condition is met.

[0079] The preset stop conditions include at least one of the following: Reaching maximum depth ; Reaching the minimum number of sample splits ; Reaching the minimum number of leaf node samples ; All samples in a node belong to the same category.

[0080] d. Leaf node prediction When the recursive construction reaches a leaf node, a prediction is made based on the label distribution in that node. For classification problems, the majority class is typically chosen as the predicted value for that node. Assume the leaf node contains... A normal sample and If there are anomaly samples, the predicted value for that leaf node is:

[0081] (3) Integrated prediction The final prediction of a random forest is a majority vote of the predictions from all decision trees. Let the... A decision tree for a given sample The prediction is Then the prediction of the random forest is:

[0082] in, ( ) is an indicator function that takes the value 1 when the condition is true and 0 otherwise.

[0083] (4) Model optimization Set different combinations of hyperparameters in the random forest (such as maximum depth) Minimum number of sample splits Minimum number of leaf node samples and the number of features randomly selected at each node Random forest models were trained separately. This was achieved through initial trials, gradual adjustments, and automated tuning. The optimal combination of hyperparameters was found through cross-validation and performance saturation point analysis.

[0084] (5) Model validation Input the validation set into the model when a new data point is added. When a data point arrives, it is passed to each decision tree in the random forest. Each decision tree casts a vote (normal or anomalous), and the final result is determined by majority vote. That is, if more than half of the trees consider the data point normal, the output is normal; otherwise, the output is anomalous. The output is verified to be consistent with the labels on the validation set; otherwise, it needs to be re-evaluated.

[0085] The following data serves as an example: The XDR call detail record (CDR) of a user within a specific university setting is obtained, input into the user service perception model for identification, and the output results are shown in Table 1. Table 1

[0086] In Table 1, whether the user's business perception is normal is indicated by "0" or "1", which indicates that the user's perception in the university setting is deteriorated and there is a possibility of complaint.

[0087] See Figure 5 , Figure 5 This is a flowchart of a business awareness assessment method provided in an embodiment of this application. The business awareness assessment method includes: S21. Obtain the XDR call detail records of the user's services to be evaluated; S22. Extract features from the XDR call detail record to obtain target features; S23. Input the target features into the business perception evaluation model and output the perception evaluation result of the user's business to be evaluated. The business perception assessment model is trained using the business perception assessment training method described above.

[0088] The method for obtaining the business perception evaluation model in this application embodiment can refer to the relevant description in the above-described business perception evaluation training method, and can achieve the same beneficial effects. To avoid repetition, it will not be repeated here.

[0089] See Figure 6 , Figure 6 This is a structural block diagram of a business perception evaluation model training device 10 provided in an embodiment of this application. The business perception evaluation model training device 10 includes: The first acquisition module 11 is used to acquire XDR call detail records of multiple user services to form a first sample set; The calculation module 12 is used to obtain the service latency and downlink rate of each user service based on the first sample set. The annotation module 13 is used to annotate the first sample set with positive and negative samples according to the service latency and the downlink rate to obtain a second sample set; The first extraction module 14 is used to extract features from the second sample set to obtain the third sample set; Training module 15 is used to train the model using the third sample set to obtain the business perception evaluation model.

[0090] Optionally, the annotation module 13 is specifically used for: The first sample set is classified to obtain one or more first sub-sample sets; For each of the first subsets, the coordinate points within the first subset are fitted using service latency and downlink rate as coordinate points to obtain the fitted curve of the first subset. Based on the slope of the tangent line of the fitted curve of the first subsample set, the network perception anomalies of the fitted curve of the first subsample set are obtained. Based on the network perception anomalies in the fitted curve of the first sub-sample set, positive and negative sample labels are applied to the first sub-sample set to obtain the second sub-sample set. One or more of the second sub-sample sets are used to construct the second sample set.

[0091] Optionally, the annotation module 13 is further configured to: The average downlink rate of the fitted curve of the first subsample set is obtained by averaging the highest and lowest downlink rates. The corresponding coordinate point is then used as the target coordinate point. For the fitted curve of the first subsample set, calculate the tangent slope of the coordinate points located below the target coordinate point, and select the coordinate point corresponding to the largest tangent slope as the network perception anomaly point of the fitted curve of the first subsample set.

[0092] Optionally, the annotation module 13 is further configured to: Based on the first sample set, the wireless side latency ratio of each user service is obtained; Based on the distribution of the wireless side latency proportions of all the aforementioned user services, the proportion threshold is obtained; Based on the aforementioned percentage threshold, XDR call detail records that do not meet the preset percentage threshold are deleted from the first sample set.

[0093] Optionally, the annotation module 13 is further configured to: Plot the wireless latency percentage of all the user services on a number axis; The number axis is divided into multiple equal-width intervals, and the upper limit of the equal-width interval with the highest wireless side latency ratio is used as the ratio threshold.

[0094] Optionally, the first acquisition module 11 is specifically used for: For different scenarios, use base stations covering the scenario to identify resident users in different scenarios; The XDRs of user services for multiple resident users in different scenarios are obtained to form the first sample set.

[0095] Optionally, the business perception evaluation model is a random forest model; The training module 15 is specifically used for: Samples with replacement are drawn from the third sample set as training data for the decision tree. The training data of the decision tree is used as the root node, and the decision tree is recursively constructed until a preset stopping condition is reached to obtain the decision tree. Multiple decision trees are integrated to form the random forest model.

[0096] Optionally, the training module 15 is further configured to: The hyperparameters of the random forest model are optimized to obtain the target parameters; Based on the target parameters and the random forest model, a target random forest model is obtained; wherein, the target random forest model is used for user business perception evaluation.

[0097] It is worth noting that the working process of each module in the business perception evaluation model training device 10 described in this application embodiment can refer to the working process of the business perception evaluation model training method described in the above embodiment and achieve the same beneficial effect, and will not be repeated here.

[0098] See Figure 7 , Figure 7 This is a structural block diagram of a service awareness assessment device 20 provided in an embodiment of this application. The service awareness assessment device 20 includes: The second acquisition module 21 is used to acquire the XDR call detail records of the user's services to be evaluated; The second extraction module 22 is used to extract features from the XDR call detail record to obtain target features; Evaluation module 23 is used to input the target features into the business perception evaluation model and output the perception evaluation result of the user business to be evaluated; The business perception evaluation model is trained by the aforementioned business perception evaluation model training device.

[0099] It is worth noting that the working process of each module in the business perception assessment device 20 described in this application embodiment can refer to the working process of the business perception assessment method described in the above embodiment and achieve the same beneficial effect, and will not be repeated here.

[0100] Furthermore, this application also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the service awareness evaluation model training method as described in any of the above embodiments.

[0101] Furthermore, this application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the business awareness evaluation model training method as described in any of the above embodiments.

[0102] See Figure 8 , Figure 8 This is a structural block diagram of an electronic device 30 provided in an embodiment of this application. The electronic device 30 includes: a processor 31, a memory 32, and a computer program stored in the memory 32 and executable on the processor 31. When the processor 31 executes the computer program, it implements the steps in the above-described embodiments of the service awareness evaluation model training method. Alternatively, when the processor 31 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments.

[0103] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device 30.

[0104] The electronic device 30 may include, but is not limited to, a processor 31 and a memory 32. Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 30 and does not constitute a limitation on the electronic device 30. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device 30 may also include input / output devices, network access devices, buses, etc.

[0105] The processor 31 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 31 is the control center of the electronic device 30, connecting various parts of the electronic device 30 via various interfaces and lines.

[0106] The memory 32 can be used to store the computer programs and / or modules. The processor 31 implements various functions of the electronic device 30 by running or executing the computer programs and / or modules stored in the memory 32 and calling the data stored in the memory 32. The memory 32 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0107] If the modules / units integrated in the electronic device 30 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 31, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0108] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0109] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A method for training a business perception evaluation model, characterized in that, include: Obtain XDR call detail records for multiple user services to form the first sample set; Based on the first sample set, the service latency and downlink rate of each user service are obtained; The first sample set is labeled with positive and negative samples based on the service latency and the downlink rate to obtain the second sample set; Feature extraction is performed on the second sample set to obtain the third sample set; The business perception evaluation model is obtained by using the third sample set for model training.

2. The business perception evaluation model training method as described in claim 1, characterized in that, The step of labeling the first sample set with positive and negative samples based on the service latency and the downlink rate to obtain the second sample set includes: The first sample set is classified to obtain one or more first sub-sample sets; For each of the first subsets, the coordinate points within the first subset are fitted using service latency and downlink rate as coordinate points to obtain the fitted curve of the first subset. Based on the slope of the tangent line of the fitted curve of the first subsample set, the network perception anomalies of the fitted curve of the first subsample set are obtained. Based on the network perception anomalies in the fitted curve of the first sub-sample set, positive and negative sample labels are applied to the first sub-sample set to obtain the second sub-sample set. One or more of the second sub-sample sets are used to construct the second sample set.

3. The training method for the business perception evaluation model as described in claim 2, characterized in that, The step of obtaining network-perceptual outliers of the fitted curve of the first sub-sample set based on the slope of the tangent line of the fitted curve of the first sub-sample set includes: The average downlink rate of the fitted curve of the first subsample set is obtained by averaging the highest and lowest downlink rates. The corresponding coordinate point is then used as the target coordinate point. For the fitted curve of the first subsample set, calculate the tangent slope of the coordinate points located below the target coordinate point, and select the coordinate point corresponding to the largest tangent slope as the network perception anomaly point of the fitted curve of the first subsample set.

4. The training method for the business perception evaluation model as described in any one of claims 1 to 3, characterized in that, Before labeling the first sample set with positive and negative samples based on the service latency and the downlink rate to obtain the second sample set, the service awareness evaluation model training method further includes: Based on the first sample set, the wireless side latency ratio of each user service is obtained; Based on the distribution of the wireless side latency proportions of all the aforementioned user services, the proportion threshold is obtained; Based on the aforementioned percentage threshold, XDR call detail records that do not meet the preset percentage threshold are deleted from the first sample set.

5. The training method for the business perception evaluation model as described in claim 4, characterized in that, The step of obtaining the percentage threshold based on the distribution of wireless side latency percentages for all user services includes: Plot the wireless latency percentage of all the user services on a number axis; The number axis is divided into multiple equal-width intervals, and the upper limit of the equal-width interval with the highest wireless side latency ratio is used as the ratio threshold.

6. The training method for the business perception evaluation model as described in claim 1, characterized in that, The step of obtaining XDR call detail records (CDRs) for multiple user services to form a first sample set includes: For different scenarios, use base stations covering the scenario to identify resident users in different scenarios; The XDRs of user services for multiple resident users in different scenarios are obtained to form the first sample set.

7. The training method for the business perception evaluation model as described in claim 1, characterized in that, The business perception evaluation model is a random forest model; The step of using the third sample set to train the model and obtain the business perception evaluation model includes: Samples with replacement are drawn from the third sample set as training data for the decision tree. The training data of the decision tree is used as the root node, and the decision tree is recursively constructed until a preset stopping condition is reached to obtain the decision tree. Multiple decision trees are integrated to form the random forest model.

8. The training method for the business perception evaluation model as described in claim 7, characterized in that, The training method for the business perception assessment model also includes: The hyperparameters of the random forest model are optimized to obtain the target parameters; Based on the target parameters and the random forest model, a target random forest model is obtained; wherein, the target random forest model is used for user business perception evaluation.

9. A business perception assessment method, characterized in that, include: Obtain the XDR call detail records (CDRs) of the user's services to be evaluated; Feature extraction is performed on the XDR call detail records to obtain the target features; The target features are input into the business perception evaluation model, and the perception evaluation results of the user's business to be evaluated are output. The business perception assessment model is trained by the business perception assessment training method according to any one of claims 1 to 7.

10. A training device for a business perception evaluation model, characterized in that, include: The first acquisition module is used to acquire XDR call detail records of multiple user services to form the first sample set; The calculation module is used to obtain the service latency and downlink rate of each user service based on the first sample set. The annotation module is used to annotate the first sample set with positive and negative samples based on the service latency and the downlink rate to obtain a second sample set; The first extraction module is used to extract features from the second sample set to obtain the third sample set; The training module is used to train the model using the third sample set to obtain the business perception evaluation model.

11. A business perception assessment device, characterized in that, include: The second acquisition module is used to acquire the XDR call detail records of the user's services to be evaluated; The second extraction module is used to extract features from the XDR call detail record to obtain target features; The evaluation module is used to input the target features into the business perception evaluation model and output the perception evaluation result of the user's business to be evaluated. The business perception evaluation model is trained by the business perception evaluation model training device described in claim 10.

12. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the business awareness assessment model training method as described in any one of claims 1 to 8, or the business awareness assessment method as described in claim 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the service awareness assessment model training method as described in any one of claims 1 to 8, or the service awareness assessment method as described in claim 9.

14. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the business awareness assessment model training method as described in any one of claims 1 to 8, or the business awareness assessment method as described in claim 9.

Images