Network performance evaluation method and related device

Abstract

The embodiment of the invention discloses a network performance evaluation method and a related device. By analyzing the expression difference of various types of terminals under similar network conditions, the common characteristics of the terminals which may cause common network faults are identified and summarized into a fault map. And then, obtaining a target type parameter of a target terminal accessing the network, and determining whether a network fault corresponding to the target type parameter exists by querying the fault map. And carrying out network performance evaluation on the target terminal according to a query result. It can be seen that the situation that different types of terminals may have inconsistent network experiences under similar network conditions is considered, so that the network performance of the terminal can be reasonably evaluated without obtaining real-time network use data from the terminal by constructing the fault map as an important basis for network performance evaluation. Therefore, the network performance evaluation has more dimensions and higher accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication, and in particular to a network performance evaluation method and related apparatus. BACKGROUND

[0002] With the development of wireless network technology, users have higher and higher requirements for wireless network quality. Network operation and maintenance focuses on customer satisfaction and user experience assurance, and accurate user experience evaluation is the basis of experience assurance.

[0003] Traditional network experience evaluation techniques are mainly based on network performance indicators such as signal strength and connection speed. Considering that users use the network in various scenarios, from indoor office to outdoor activities, from simple data browsing to high-definition video conference, each use has different requirements for the network, and terminal behavior data of different types and categories are quite different. The traditional network experience evaluation techniques do not always accurately reflect the actual experience of users, have poor scene adaptation capability, and have low accuracy. SUMMARY

[0004] The embodiments of the present application provide a network performance evaluation method and related apparatus, which make the dimension of network performance evaluation more and the accuracy higher.

[0005] In a first aspect, the embodiments of the present application provide a network performance evaluation method. Specifically, a fault atlas is obtained, wherein the fault atlas is used to represent network faults corresponding to terminals with at least one same type parameter, which is equivalent to that the fault atlas includes a corresponding relationship between the type parameter of the terminal and the network fault. Then, a target type parameter of a target terminal accessing a network is obtained. Further, the target terminal is evaluated in terms of network performance according to the fault atlas and the target type parameter, which is equivalent to querying whether there is a network fault corresponding to the target type parameter in the fault atlas, and then evaluating the target terminal in terms of network performance based on the query result.

[0006] In this embodiment, the fault atlas can be determined through data clustering and common problem analysis, which is equivalent to analyzing the performance differences of terminals of different types under similar network conditions, identifying common features of terminals that may cause common network faults, and summarizing them into a fault atlas. Considering that different types of terminals may have inconsistent network experience under similar network conditions, the fault atlas is constructed as an important basis for network performance evaluation, without obtaining real-time network data from the terminal, the network performance of the terminal can also be reasonably evaluated. Thus, the dimension of network performance evaluation is more and the accuracy is higher.

[0007] In some possible implementation manners, the network performance evaluation on the target terminal according to the fault atlas and the target type parameter comprises: determining a target network fault corresponding to the target type parameter according to the fault atlas; and performing network performance evaluation on the target terminal according to the target network fault. That is, if it is determined through the query of the fault atlas that the target network fault corresponding to the target type parameter exists, it indicates that the target terminal itself has a defect affecting its network performance, and a lower network performance score should be given in the dimension of the terminal side. For the target terminal with the defect, the network performance can be more accurately evaluated.

[0008] In some possible implementation manners, the obtaining of the fault atlas comprises: obtaining type parameters of terminals of multiple types, and performing network fault detection on the terminals of the multiple types; and generating the fault atlas according to a result of the network fault detection and the type parameters of the terminals of the multiple types. Here, a specific manner of generating the fault atlas is provided, which needs to perform data clustering on the type parameters of the terminals of the multiple types, and perform common problem analysis on the terminals of the multiple types, so as to obtain the fault atlas. That is, the fault atlas can summarize a large amount of data, and supports continuous updating, and has a good actual application effect.

[0009] In some possible implementation manners, the type parameter of the terminal comprises at least one of a model of the terminal, an operating system version of the terminal, and a chip model of the terminal. That is, in the fault atlas, the type parameters of the terminal described above can have corresponding network faults respectively, or any combination of the type parameters of the terminal described above can have corresponding network faults, which is beneficial to enrich the content recorded by the fault atlas.

[0010] In some possible implementation manners, the type parameter of the terminal further comprises a system log of the terminal. In the fault atlas, the system log of the terminal does not necessarily have a corresponding network fault alone, but the system log of the terminal in combination with any other type parameter of the terminal described above can have a corresponding network fault, which is beneficial to further enrich the content recorded by the fault atlas.

[0011] In some possible implementation manners, the method further comprises: obtaining network side data from a network device and / or application side data from a server; performing feature classification on the network side data and / or the application side data; and performing model training according to the network side data and / or the application side data after the feature classification to generate a classification model, the classification model being used for data classification according to a use scenario of the terminal. It should be understood that the process of generating the classification model can be regarded as an offline process, and the purpose is to classify the data collected in real time in an online process through the classification model, so as to facilitate corresponding processing according to actual requirements on the basis of data classification.

[0012] In some possible implementation manners, the method further includes: obtaining network side data and / or application side data related to the target terminal; filtering the network side data and / or the application side data related to the target terminal according to the classification model to filter out data irrelevant to user experience of the target terminal; and performing network performance evaluation on the network side and / or the application side according to the filtered data. It should be understood that, by filtering out the data irrelevant to user experience of the target terminal through the classification model before performing the network performance evaluation, the efficiency and accuracy of the network performance evaluation can be improved.

[0013] In some possible implementation manners, the data irrelevant to user experience of the target terminal includes network side data and / or application side data generated by autonomous behavior of the target terminal, for example, data generated in a scenario in which the target terminal automatically connects to Wi-Fi, or data generated in a scenario in which the target terminal automatically updates an application in a black screen mode.

[0014] In some possible implementation manners, performing the network performance evaluation on the application side according to the filtered data includes: obtaining an application type used by the target terminal; and performing the network performance evaluation on the application side according to the filtered data and the application type. That is, in similar network conditions, the network performance evaluation on the application side also needs to be performed according to different application types, which is beneficial to improving the accuracy of the application side score. For example, when the target terminal uses a game application, the target terminal is very sensitive to packet loss and time delay, and packet loss type and time delay type data have a greater impact on the application side score; when the target terminal uses a chat application, the target terminal is not sensitive to packet loss and time delay, and the packet loss type and time delay type data have a smaller impact on the application side score.

[0015] In some possible implementation manners, the method further includes: obtaining a solution graph, the solution graph being used to represent a solution corresponding to each network fault in the fault graph; and determining a target solution corresponding to a target network fault corresponding to the target type parameter according to the solution graph, if the target network fault corresponding to the target type parameter exists in the fault graph. That is, based on the specific network fault determined in the fault graph, a corresponding solution can be quickly found in the solution graph, which is beneficial to quickly solving the network fault.

[0016] In some possible implementation manners, the method further includes: if the target network fault corresponding to the target type parameter exists in the fault graph, performing an alarm.

[0017] In a second aspect, an embodiment of the present application provides a network performance evaluation apparatus, which comprises an acquisition unit and a processing unit. The acquisition unit is configured to acquire a fault atlas, wherein the fault atlas is used to represent network faults corresponding to terminals with at least one same type parameter; and acquire a target type parameter of a target terminal accessing a network. The processing unit is configured to perform network performance evaluation on the target terminal according to the fault atlas and the target type parameter.

[0018] In some possible implementation manners, the processing unit is specifically configured to determine a target network fault corresponding to the target type parameter according to the fault atlas; and perform network performance evaluation on the target terminal according to the target network fault.

[0019] In some possible implementation manners, the acquisition unit is specifically configured to acquire type parameters of terminals of multiple types, and perform network fault detection on the terminals of the multiple types; and generate the fault atlas according to a result of the network fault detection and the type parameters of the terminals of the multiple types.

[0020] In some possible implementation manners, the type parameter of the terminal comprises at least one of a model number of the terminal, an operating system version of the terminal, and a chip model number of the terminal.

[0021] In some possible implementation manners, the type parameter of the terminal further comprises a system log of the terminal.

[0022] In some possible implementation manners, the acquisition unit is further configured to acquire network-side data from a network device and / or application-side data from a server. The processing unit is further configured to perform feature classification on the network-side data and / or the application-side data; and perform model training according to the network-side data and / or the application-side data after the feature classification to generate a classification model, wherein the classification model is used to perform data classification according to a use scenario of the terminal.

[0023] In some possible implementation manners, the acquisition unit is further configured to acquire network-side data and / or application-side data related to the target terminal. The processing unit is further configured to filter, according to the classification model, the network-side data and / or the application-side data related to the target terminal to filter out data irrelevant to user experience of the target terminal; and perform network performance evaluation on the network side and / or the application side according to the filtered data.

[0024] In some possible implementation manners, the data irrelevant to the user experience of the target terminal comprises network-side data and / or application-side data generated by autonomous behavior of the target terminal.

[0025] In some possible implementation manners, the processing unit is specifically configured to acquire an application type used by the target terminal; and perform network performance evaluation on the application side according to the filtered data and the application type.

[0026] In some possible implementation, the obtaining unit is further configured to obtain a solution atlas, the solution atlas being used to represent a solution corresponding to each network fault in the fault atlas. If the target network fault corresponding to the target type parameter exists in the fault atlas, the processing unit is further configured to determine a target solution corresponding to the target network fault according to the solution atlas.

[0027] In some possible implementation, if the target network fault corresponding to the target type parameter exists in the fault atlas, the processing unit is further configured to perform an alarm.

[0028] In a third aspect, an embodiment of the present application provides a network performance evaluation apparatus, the network performance evaluation apparatus comprising an interface and a processor, the interface being configured to receive and send data, and the processor being configured to execute the method according to any of the embodiments of the first aspect.

[0029] In a fourth aspect, an embodiment of the present application provides a network performance evaluation system, the network performance evaluation system comprising a terminal, a network device and the network performance evaluation apparatus according to the third aspect, the terminal being configured to communicate with the network device, and the network device being configured to communicate with the network performance evaluation apparatus.

[0030] In some possible implementation, the network performance evaluation system further comprises a server, the terminal being configured to communicate with the server, and the server being configured to communicate with the network performance evaluation apparatus.

[0031] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing instructions, the instructions being executed by a computer to implement the method according to any of the embodiments of the first aspect.

[0032] In a sixth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising program instructions, the program instructions being executed to implement the method according to any of the embodiments of the first aspect.

[0033] In the embodiments of the present application, the fault atlas can be determined through data clustering and common problem analysis, and the fault atlas is used to represent the network failure corresponding to the terminal with at least one same type parameter. It is equivalent to analyzing the performance difference of each type of terminal under similar network conditions, identifying the common characteristics of the terminal that may cause common network failure, and summarizing as a fault atlas. Then, the target type parameter of the target terminal accessing the network is obtained, and whether there is a network failure corresponding to the target type parameter is determined by querying the fault atlas. Further, the network performance of the target terminal is evaluated according to the query result. It can be seen that the embodiments of the present application take into account the possible inconsistent network experience of different types of terminals under similar network conditions, so the fault atlas is constructed as an important basis for network performance evaluation, without obtaining real-time network data from the terminal, and the network performance of the terminal can be reasonably evaluated. For example, according to the target type parameter of the target terminal in the fault atlas, the corresponding network failure can be found, even if the current network condition is good, the network performance comprehensive score of the target terminal should be adjusted accordingly, so that the dimension of network performance evaluation is more, and the accuracy is higher. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 An application scenario of the network performance evaluation method provided by the embodiments of the present application is shown;

[0035] Figure 2 Another application scenario of the network performance evaluation method provided by the embodiments of the present application is shown;

[0036] Figure 3 Another application scenario of the network performance evaluation method provided by the embodiments of the present application is shown;

[0037] Figure 4 A flowchart of a network performance evaluation method in the embodiments of the present application is shown;

[0038] Figure 5 A schematic diagram of a fault atlas in the embodiments of the present application is shown;

[0039] Figure 6 A flowchart of another network performance evaluation method in the embodiments of the present application is shown;

[0040] Figure 7 An implementation diagram of a data filtering method based on model training in the embodiments of the present application is shown;

[0041] Figure 8 An embodiment diagram of network performance comprehensive evaluation in the embodiments of the present application is shown;

[0042] Figure 9 An interface display diagram of network performance comprehensive evaluation in the embodiments of the present application is shown;

[0043] Figure 10 This is a schematic diagram of a network performance evaluation device in an embodiment of this application;

[0044] Figure 11 This is another structural schematic diagram of the network performance evaluation device in the embodiments of this application. Detailed Implementation

[0045] This application provides a network performance evaluation method and related apparatus, which enables network performance evaluation to have more dimensions and higher accuracy.

[0046] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three possibilities: A exists alone, B exists alone, or A and B exist simultaneously. The naming or numbering of steps in the embodiments of this application does not imply that the steps in the method flow must be executed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in the embodiments of this application is a logical division. In actual applications, there may be other division methods. For example, multiple units may be combined into or integrated into another system, or some features may be ignored or not executed. In addition, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms. None of these are limited in the embodiments of this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution in this application.

[0047] Figure 1 This is a schematic diagram illustrating an application scenario of the network performance evaluation method provided in this application embodiment. For example...Figure 1 As shown, this application scenario includes a network management device and multiple access points (APs). All APs can establish communication connections with the network management device. Furthermore, this application scenario may also include at least one terminal, which is associated with an AP to obtain the wireless access services provided by that AP. The network management device, also known as a network analyzer, can be deployed in the cloud or as a physical device. For example, the network management device can be a wireless controller, such as an access controller (AC). Alternatively, the network management device can be a server independent of the AC but capable of communicating with it. APs can be network devices such as routers or switches that support wireless local area networks (WLANs). Terminals, also known as stations (STAs), can be WLAN-enabled devices including, but not limited to, mobile phones, laptops, wearable devices, or smart home devices.

[0048] Figure 2 This is a schematic diagram illustrating another application scenario of the network performance evaluation method provided in the embodiments of this application. For example... Figure 2 As shown, the network management device has a unified southbound interface. This unified southbound interface can be used to receive data from APs and / or terminals. For example, APs and / or terminals can report data to this unified southbound interface via Hypertext Transfer Protocol (HTTP) or next-generation HTTP (HTTP 2). The reported data includes, but is not limited to, network-side data, application-side data, and terminal-side data. The data reporting method can be active or passive. The network management device also includes a big data computing platform. The unified southbound interface can transmit the received data to the big data computing platform. This big data computing platform can then use big data analysis and storage components such as Spark, Kafka, the Hadoop Distributed File System (HDFS), and Druid to support its data analysis and storage. Among them, Spark is a fast and general-purpose computing engine designed for large-scale data processing, Kafka is a high-throughput distributed publish-subscribe messaging system, and Druid is an efficient data query system.

[0049] Continue to refer to Figure 2On the one hand, network management devices can obtain fault maps by analyzing terminal-side data through a big data computing platform. Specifically, terminal-side data can include terminal type parameters. The fault map represents network faults corresponding to terminals with at least one shared type parameter. This is equivalent to analyzing the performance differences of various terminal types under similar network conditions, identifying common characteristics of terminals that may lead to common network faults, and summarizing them into a fault map. Then, the target type parameters of the target terminals accessing the network are obtained, and the target type parameters are matched against the fault map to determine whether a network fault corresponding to the target type parameter exists. Furthermore, network performance is evaluated for the target terminal based on the matching results, achieving terminal-side scoring. On the other hand, network management devices can perform network-side and / or application-side scoring based on data analysis from a big data computing platform, thus integrating multiple dimensions of scoring and making network performance evaluation more accurate.

[0050] Figure 3 This is a schematic diagram illustrating another application scenario of the network performance evaluation method provided in the embodiments of this application. For example... Figure 3 As shown, this application scenario includes a user network and a network management device. The network management device includes a data analysis engine and a visualization module for presenting data and analysis results. The user network can be a WLAN or a local area network (LAN), and user networks can be connected via a wide area network (WAN). Access points (APs) in each user network can access the WAN through network devices such as switches, firewalls, and routers. The network management device has device management and data acquisition functions. The network management device can add devices via the Simple Network Management Protocol (SNMP) for unified management and analysis of the added devices (i.e., APs in the user network). Devices can report key performance indicator (KPI) data via telemetry technology, and the reported data can be uniformly encoded using the protocol buffer (ProtoBuf) format. Furthermore, the device can also report log data via the system log (Syslog) protocol. KPI data can include network performance-related data such as traffic, packet loss, latency, signal strength, uplink bandwidth, downlink bandwidth, experience rate, access mode, channel utilization, interference rate, and number of access users; log data can include terminal roaming data.

[0051] Continue to refer to Figure 3The data analysis engine in the network management device can include a data analysis component, a big data analysis platform, and an artificial intelligence (AI) engine. The data analysis component can provide data collection, data subscription and distribution, and data analysis task management, etc. The big data analysis platform can provide data analysis and storage functions through big data analysis and storage components such as Kafka, Spark, HDFS, and Druid. The AI engine can provide a machine learning framework that supports analyzing collected data through artificial intelligence and machine learning, and can provide a typical machine learning algorithm library for business analysis, for example, the typical algorithm library can be catBoost or xgboost, etc. The visual module in the network management device for presenting data and analysis results can present the results of network performance evaluation from multiple dimensions such as network analysis, application analysis, terminal analysis, and tuning, etc.

[0052] It should be noted that the network performance evaluation method provided by the embodiments of the present application can be applied to the network management device as shown in Figure 1 、 Figure 2 or Figure 3 The network performance evaluation method provided by the embodiments of the present application will be described in detail below.

[0053] Figure 4 A flowchart of a network performance evaluation method in an embodiment of the present application. As shown in Figure 4 , the network performance evaluation method is mainly for terminal-side network performance evaluation, and specifically includes the following steps.

[0054] In the embodiments of the present application, the fault atlas is used to represent the network faults corresponding to the terminals with at least one same type parameter, which is equivalent to that the fault atlas includes the correspondence between the type parameters of the terminals and the network faults. The type parameters of the terminals include but are not limited to at least one of the model of the terminal, the operating system version of the terminal, and the chip model of the terminal. That is, in the fault atlas, the above-mentioned type parameters of the terminals can have corresponding network faults respectively, or any combination of the above-mentioned type parameters of the terminals can have corresponding network faults. Alternatively, the type parameters of the terminals can also include the system log of the terminal. In the fault atlas, the system log of the terminal does not necessarily have a separate corresponding network fault, but the system log of the terminal combined with any other type parameter of the terminal can have a corresponding network fault.

[0055] Figure 5 A schematic diagram of a fault atlas in an embodiment of the present application. As shown in Figure 5 , the correspondence between the type parameters of the terminals and the network faults is shown. It should be understood that Figure 5The provided is only an example of a fault atlas, and the specific content in the fault atlas depends on the actual situation after data clustering and common problem analysis, which is not limited here. For example, terminals using a certain type of network card driver are prone to active logout problems. By comparing the type of network card and the performance under the same AP, the common problem of this type of terminal can be identified, and the network fault corresponding to the network card type is recorded in the fault atlas. For another example, a certain type of terminal is limited by its own configuration, and in the scenario of having both 2.4G frequency Wi-Fi signal and 5G frequency Wi-Fi signal, this type of terminal will preferentially connect to the 2.4G frequency Wi-Fi signal, and cannot preferentially connect to the 5G frequency Wi-Fi signal. The model of this type of terminal and the corresponding network fault can be recorded in the fault atlas.

[0056] In a possible implementation, the network management device collects type parameters of multiple types of terminals. For example, the AP can obtain type parameters of multiple types of terminals associated with the AP, and report the type parameters of multiple types of terminals to the network management device. Further, the network management device performs data clustering on the type parameters of multiple types of terminals, and performs common problem analysis on multiple types of terminals, to obtain a fault atlas. That is, the performance differences of terminals of different types under similar network conditions are analyzed, the common characteristics of terminals that may cause common network faults are identified, and the fault atlas is summarized. For example, the network management device can detect network faults of multiple types of terminals respectively, to identify whether there is a common network fault under similar network conditions. The data used for network fault detection can include network side data from network devices such as APs and / or application side data from application servers.

[0057] 102. Obtain a target type parameter of a target terminal accessing a network.

[0058] Specifically, for a target terminal accessing a network, a target AP associated with the target terminal can obtain a target type parameter of the target terminal. Further, the target AP reports the target type parameter of the target terminal to the network management device, so that the network management device performs network performance evaluation on the target terminal based on the target type parameter of the target terminal.

[0059] 103. Perform network performance evaluation on the terminal side according to the fault atlas and the target type parameter.

[0060] Specifically, the network management device matches the target type parameter according to the fault atlas, that is, queries whether the network fault corresponding to the target type parameter exists in the fault atlas. One result of the matching is that the network fault corresponding to the target type parameter does not exist in the fault atlas, which indicates that the target terminal itself has no defects affecting its network performance, and a higher network performance score can be given in the dimension of the terminal side. Another result of the matching is that the network fault corresponding to the target type parameter exists in the fault atlas, which indicates that the target terminal itself has defects affecting its network performance, and a lower network performance score should be given in the dimension of the terminal side. It should be understood that the reference proportion of the terminal side network performance evaluation to different network faults in the fault atlas can also be different. For example, the network fault that has a greater impact on user experience has a greater impact on the scoring of the terminal side network performance evaluation.

[0061] In one possible scenario, the network management device can also pre-configure a solution atlas corresponding to the fault atlas, and the solution atlas includes a solution corresponding to each network fault in the fault atlas. The network management device can quickly find the solution corresponding to the specific network fault in the solution atlas according to the specific network fault determined in the fault atlas, which is beneficial to realize the rapid solution of the network fault. Specifically, the solution can be realized by adjusting the configuration parameters of the network device associated with the terminal, such as the AP. For example, more bandwidth is reserved for the target terminal by the AP to improve the network performance of the target terminal. In another possible scenario, if the network management device queries that the network fault corresponding to the target type parameter exists in the fault atlas, the network management device can also perform an alarm.

[0062] Figure 6 A flowchart of another network performance evaluation method in the embodiments of the present application is shown in FIG. 6. As shown in FIG. 6, the network performance evaluation method mainly aims at the network side and / or application side network performance evaluation, and specifically includes the following steps. Figure 6

[0063] 201、Obtain network side data and / or application side data.

[0064] ​Specifically, the network management device can obtain network side data from network devices such as APs, and the types of network side data include but are not limited to traffic, packet loss, latency, signal strength, uplink bandwidth, downlink bandwidth, experience rate, access mode, channel utilization, interference rate, and the number of access users, etc. The network management device can obtain application side data from application servers, where the application side data refers to data related to the application running on the terminal, for example, data related to A application comes from the corresponding A application server, and data related to B application comes from the corresponding B application server. The types of application side data are similar to those of network side data, with the difference being that the application side data is about the traffic, packet loss, latency, etc. of specific applications.

[0065] 202. Generate a classification model according to the network side data and / or application side data.

[0066] The network management device can perform model training based on the collected network side data and / or application side data, so that the model can continuously learn the patterns and rules in the data through algorithms, thereby generating a classification model that can classify data according to the use scenarios of the terminal. Figure 7 An embodiment of the present application is shown in the embodiment of the present application. In the offline process shown in Figure 7 In the offline process shown in the embodiment of the present application, the collected network side data and / or application side data can be divided into scenario-labeled data and unlabeled data. The scenario-labeled data is used for model training, and is usually manually labeled with scenarios, for example, the data can be labeled with tags such as "mobile phone", "laptop", "wearable device", "used with network", "not used with network", "slow roaming", "fast roaming", etc. The unlabeled data is used to evaluate the performance of the model to ensure that the model can also perform well on unseen data. Then, the scenario-labeled data is classified according to features to distinguish the features of the data, which can be achieved according to feature importance, feature-label correlation, and expert experience, etc. For example, the scenario-labeled data can be divided into traffic class, latency class, packet loss class, etc. Further, the data after feature classification is used for model training to generate a classification model, for example, supervised training can be performed based on xgboost, that is, labeled data is needed in the model training process, each input data corresponds to a correct output classification, and the relationship between the input data and the output classification is learned to obtain the classification model.

[0067] 203. Filter the network side data and / or application side data related to the target terminal according to the classification model.

[0068] It should be understood that in order to quickly and accurately assess network performance on the network side and / or application side based on massive amounts of data, the primary consideration should be data related to user experience. Therefore, a major purpose of using classification models is to filter out data that is irrelevant to user experience before conducting network performance assessments. Specifically, network management devices acquire network-side data and / or application-side data related to the target terminal; then, they filter this data using a classification model to remove data irrelevant to the user experience of the target terminal; finally, network performance assessments on the network side and / or application side are performed based on the filtered data.

[0069] As an example, in Figure 7 During the online process shown, the network management device collects real-time data, which may include network-side data and / or application-side data related to the target terminal. A classification model generated based on model training can identify the type of real-time data to distinguish real-time data related to different terminal types. Figure 7 The terminal types 1 and 2 shown are just examples of terminal type classification. In practical applications, more terminal types can be classified, such as mobile phones, laptops, and wearable devices. For example, if the target terminal is a mobile phone, the corresponding real-time data belongs to the mobile phone category; if the target terminal is a laptop, the corresponding real-time data belongs to the laptop category. The real-time data after type identification is then post-processed according to the classification model to distinguish between data related to user experience and data unrelated to user experience. This allows for the filtering out of data unrelated to user experience. It should be understood that the data types related to user experience will differ for different types of terminals. Therefore, performing type identification before post-processing allows for more targeted filtering of real-time data related to various terminal types, thus more accurately identifying user experience-related data. In some possible scenarios, type identification of the collected real-time data can be skipped, and post-processing can be performed directly to filter out data unrelated to user experience.

[0070] It should be noted that whether the data is related to user experience can depend on multiple aspects. As an example, whether the data is related to user experience can depend on whether the data is generated due to user active behavior or terminal automatic behavior. For data generated due to user active behavior, i.e. data generated by user through operating the terminal, such as data generated only when the user is using the terminal, it is data related to user experience. For data generated due to terminal automatic behavior, i.e. data generated without user using the terminal, such as data generated in the scenario of terminal automatically connecting Wi-Fi or in the scenario of terminal automatically updating application in black screen, it is data not related to user experience. As another example, for data that can be generated whether the user is using the terminal or not, it can also be distinguished whether the data is related to user experience according to whether the terminal is actually in use, such as roaming data generated in the process of terminal in use is data related to user experience, and roaming data generated in the process of terminal not in use (e.g. in black screen) is data not related to user experience. As yet another example, whether the data is related to user experience can also depend on the specific application currently used by the user, such as the user is more sensitive to network latency when playing games than when sending messages, thus latency data when the user is playing games is related to user experience, and latency data when the user is sending messages is not related to user experience.

[0071] 204. Perform network performance evaluation on network side and / or application side according to the filtered data.

[0072] On one hand, the network management device can perform network performance evaluation on network side according to the filtered network side data. On the other hand, the network management device can perform network performance evaluation on application side according to the filtered application side data. It should be noted that under similar network conditions, network performance evaluation on application side also needs to be considered according to different application types. That is, the network side score of the target terminal has no difference, but the application side score of the target terminal when using different applications can have great difference. For example, the target terminal is very sensitive to packet loss and latency when using game applications, and packet loss and latency data have great influence on the application side score; the target terminal is not sensitive to packet loss and latency when using chat applications, and packet loss and latency data have little influence on the application side score. Therefore, for network performance evaluation on application side, the scoring weight of the same type of data for different types of applications can be different, which depends on the actual application scenario, which is not limited here.

[0073] The way of combining terminal side, network side and application side network performance comprehensive evaluation is introduced below through a specific embodiment. Figure 8An embodiment of the present application is shown for the network performance comprehensive evaluation. It should be understood that the embodiment mainly introduces the process of scoring the comprehensive network performance of the target terminal of the access network by the network management device in the online process, and the specific way of generating the fault atlas and the classification model in the offline process can be referred to the related introduction of the above embodiment, which will not be described here.

[0074] As shown in Figure 8 , in a first aspect, the network management device collects network side data and application side data, and inputs the network side data and the application side data into the classification model to filter the collected network side data and the application side data through the classification model. Among them, the filtered data is mainly the data irrelevant to user experience, and the corresponding retained data after filtering is the data related to user experience. As an example, it is found that the user experience of the target terminal in a certain key protection area is poor, and through the playback of the network journey, it is found that in the process of the user from the elevator to the workstation, the target terminal automatically attempts to connect a certain Wi-Fi, but fails to connect due to certificate errors, and is successful only after the user walks to the workstation and manually connects. Then, based on the classification model, it can be identified that the log of the target terminal attempting to connect Wi-Fi failure in the moving process of the user is the autonomous behavior of the target terminal, and the user actually does not use it, and the data related to it is irrelevant to user experience and needs to be filtered. Further, the network side network performance score is calculated according to the filtered network side data, and the application side network performance score is calculated according to the filtered application side data. It should be understood that the network side score mainly depends on the actual network performance of the AP and other network devices accessed by the target terminal, and the application side score mainly depends on the actual network performance of the specific application used by the target terminal.

[0075] As shown in Figure 8 , in a second aspect, the network management device collects terminal side data, which specifically refers to the type parameters of the target terminal, such as at least one of the model, operating system version and chip model of the target terminal. Then, the terminal side data is used to query the fault atlas to determine whether there is a corresponding network fault. Further, the terminal side network performance score is calculated according to the query result. It should be understood that the terminal side score mainly depends on whether the target terminal itself has defects affecting the network performance. For example, if the corresponding network fault can be queried through the fault atlas, the terminal side score should be lowered. It should be understood that the above-mentioned Figure 8 introduced is the specific way to realize the network side score, the application side score and the terminal side score, on the basis of which, the network side score, the application side score and the terminal side score can also be visually displayed through the interface. The following provides a possible interface display diagram.

[0076] Figure 9An interface display diagram for network performance comprehensive evaluation in embodiments of the present application. In a first aspect, as shown in Figure 9 , the interface displays network performance scores in multiple dimensions, which is equivalent to displaying the score results of the network side score, the application side score and the terminal side score introduced in Figure 8 . For example, the network side score is 100, the application side score is 100, and the terminal side score is 64. It can be seen intuitively through the interface that the user experience is poor due to the terminal side. In some possible scenarios, the interface can also display security aspect scores, such as the security aspect score 100 shown in Figure 9 , wherein the security aspect score mainly refers to data related to security performance, such as firewall data, etc. It should be understood that the interface can also display a comprehensive score based on the scores in each dimension, such as the comprehensive score 78 shown in Figure 9 . In a second aspect, as shown in Figure 9 , the interface can also display specific reasons for poor terminal side score, such as displaying signal strength fluctuations of the terminal in a period of time (such as signal strength fluctuations between -80 dBm and -65 dBm), and the average value of the signal strength in this period of time is -71 dBm. On this basis, it is also shown through event analysis that the weak signal time ratio exceeds 80%, and through fault library identification and network comparison, the terminal model is relatively old, the terminal performance and antenna gain are weak, and the weak signal time ratio exceeds the average value by 130%, which is equivalent to displaying specific reasons for the fault. It is suggested to adjust the terminal position or upgrade the firmware and equipment, which is equivalent to also displaying solutions corresponding to the fault.

[0077] Figure 10 A structural schematic diagram of a network performance evaluation device in embodiments of the present application. As shown in Figure 10 , the network performance evaluation device comprises an acquisition unit 301 and a processing unit 302. In a possible implementation manner, the acquisition unit 301 is configured to perform the operations of steps 101 and 102 in the embodiment shown in Figure 4 , and the processing unit 302 is configured to perform the operation of step 103 in the embodiment shown in Figure 4 . In another possible implementation manner, the acquisition unit 301 is configured to perform the operation of step 201 in the embodiment shown in Figure 6 , and the processing unit 302 is configured to perform the operations of steps 202-204 in the embodiment shown in Figure 6 .

[0078] It should be understood that the network performance evaluation apparatus provided in the embodiments of this application can also be implemented in other ways. For example, the unit division in the above apparatus is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system. In addition, the functional units in the various embodiments of this application may be integrated into one processing unit, or they may be independent physical units, or two or more functional units may be integrated into one processing unit. The integrated units described above may be implemented in hardware or as software functional units.

[0079] Figure 11 This is another structural schematic diagram of the network performance evaluation device in the embodiments of this application. For example... Figure 11 As shown, the network performance evaluation device includes a processor 401 and an interface 402. The interface 402 can be a transceiver or an input / output interface, used to receive signals from other devices and transmit them to the processor 401, or to send signals from the processor 401 to other devices. Optionally, the network performance evaluation device may also include a memory 403, which stores program instructions and data. Specifically, the processor 201 is used to execute the above-described... Figure 4 and Figure 6 The operation in the illustrated embodiment.

[0080] This application also provides a chip. The chip integrates circuitry for implementing the functions of the processor 401 described above and one or more interfaces. As an example, the chip integrates a memory. As another example, when the chip does not integrate a memory, it can be connected to an external memory via an interface. The chip can perform the method steps of any one or more of the foregoing embodiments. Alternatively, the chip can implement the actions performed by the data processing device in the foregoing embodiments based on program code stored in the memory.

[0081] As an example, the chip in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor, any conventional processor, or a processing circuit that implements a specific function.

[0082] The embodiment of the present application further provides a computer readable storage medium comprising a program or instructions, which, when executed on a computer, cause the method performed by the above method embodiment to be implemented.

[0083] It should be understood that the processor mentioned in the embodiment of the present application can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor which is implemented by reading software code stored in a memory. The memory can exist independently and be connected to the processor, or the memory can be integrated with the processor.

[0084] As an example, the processor in the embodiment of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor, or any conventional processor, and can also be a processing circuit implementing specific functions.

[0085] The memory in the embodiment of the present application can be a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and the storage medium can also exist as discrete components in the network device or the terminal device.

[0086] In the above embodiments, the data processing method can be implemented by software, hardware, firmware or any combination thereof, in whole or in part.

[0087] When implemented by hardware, the data processing method provided by the embodiments of the present application can be implemented without reading software codes or instructions, for example, by a CPU, a DSP, an ASIC, an FPGA, other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0088] When implemented by software, the data processing method can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer programs or instructions are loaded and executed on a computer, the processes or functions of the embodiments of the present application are executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a terminal device or other programmable devices. The computer programs or instructions can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server integrating one or more available media. The available medium can be a magnetic medium, for example, a floppy disk, a hard disk, a magnetic tape; an optical medium, for example, a digital versatile disc (DVD); or a semiconductor medium, for example, a solid state disk (SSD).

[0089] Finally, it should be noted that: the above, only for the specific embodiments of the present application, but the protection scope of the present application is not limited to this, any skilled in the art of the technical person in the technical range disclosed in the present application, can easily think of changes or replacement, should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A network performance evaluation method, characterized in that, include: Obtain a fault map, wherein the fault map is used to represent network faults corresponding to terminals having at least one of the same type parameters; Obtain the target type parameter of the target terminal accessing the network; The network performance of the target terminal is evaluated based on the fault map and the target type parameters.

2. The method according to claim 1, characterized in that, The network performance evaluation of the target terminal based on the fault map and the target type parameters includes: Based on the fault map, determine the target network fault corresponding to the target type parameter; The network performance of the target terminal is evaluated based on the target network fault.

3. The method according to claim 1 or 2, characterized in that, Obtaining fault maps includes: Obtain type parameters for various types of terminals and perform network fault detection on these terminals. The fault map is generated based on the network fault detection results and the type parameters of the various types of terminals.

4. The method according to any one of claims 1 to 3, characterized in that, The terminal type parameters include at least one of the terminal model, the terminal operating system version, and the terminal chip model.

5. The method according to claim 4, characterized in that, The terminal type parameter also includes the terminal's system log.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Acquire network-side data from network devices and / or application-side data from servers; Perform feature classification on the network-side data and / or the application-side data; The model is trained based on the network-side data and / or application-side data classified according to the features to generate a classification model, which is used to classify data according to the usage scenario of the terminal.

7. The method according to claim 6, characterized in that, The method further includes: Acquire network-side data and / or application-side data related to the target terminal; The network-side data and / or application-side data related to the target terminal are filtered according to the classification model to remove data that is irrelevant to the user experience of the target terminal. Network performance is evaluated on the network side and / or application side based on the filtered data.

8. The method according to claim 7, characterized in that, Data unrelated to the user experience of the target terminal includes network-side data and / or application-side data generated by the target terminal's autonomous behavior.

9. The method according to claim 7 or 8, characterized in that, The application-side network performance evaluation based on the filtered data includes: Obtain the application type used by the target terminal; The network performance of the application side is evaluated based on the filtered data and the application type.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: Obtain a solution map, which represents the solution corresponding to each network fault in the fault map; If a target network fault corresponding to the target type parameter exists in the fault map, then the target solution corresponding to the target network fault is determined according to the solution map.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: If a target network fault corresponding to the target type parameter exists in the fault map, an alarm will be issued.

12. A network performance evaluation device, characterized in that, It includes an interface and a processor, the interface being used to send and receive data, and the processor being used to perform the method as described in any one of claims 1 to 11.

13. A network performance evaluation system, characterized in that, It includes a terminal, a network device, and a network performance evaluation device as described in claim 12, wherein the terminal communicates with the network device, and the network device communicates with the network performance evaluation device.

14. The network performance evaluation system according to claim 13, characterized in that, The network performance evaluation system also includes a server, the terminal communicates with the server, and the server communicates with the network performance evaluation device.

Images