Multi-type network application service quality assessment system and method

Abstract

This invention provides a multi-type network application service quality assessment system and method, including a network service comprehensive indicator monitoring subsystem, a network service quality fine-grained assessment subsystem, and a service grading assessment visualization subsystem. The network service comprehensive indicator monitoring subsystem is used to: define indicators for different types of network application assessment templates; receive monitoring and assessment instructions sent by the central control node; and monitor the indicators of the assessment templates by establishing a monitoring model. The network service quality fine-grained assessment subsystem is used to: determine the scoring criteria for individual indicators; use the analytic hierarchy process (AHP) to determine the weights between different indicators, formulate a scoring template, and obtain a total score. The service grading assessment visualization subsystem is used for: user interaction with the multi-type network application service quality assessment system, setting application service quality parameters, and storing monitoring and assessment data. The system and method provided by this invention enable the system to comprehensively assess the service quality of applications at different network nodes.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a system and method for evaluating the quality of services for various types of network applications. Background Technology

[0002] With the rapid development of internet technology, various online applications have emerged, greatly facilitating people's lives. Therefore, to provide better application services, it is necessary to conduct service quality assessments for various types of online applications, such as web pages, streaming media, and instant messaging. Quality of Service (QoE) is a user-centric approach to evaluating online application service quality, reflecting the user's intuitive experience. Online application service quality is the result of a series of overlapping factors. To accurately evaluate online application service quality, a multi-indicator, refined QoE model must be established. Furthermore, the key indicators affecting the experience of different online applications vary, necessitating the development of differentiated QoE models for different applications.

[0003] Currently, there are many studies on Quality of Service (QoE) evaluation models for network applications both domestically and internationally, which can be divided into subjective evaluation and objective evaluation. Subjective evaluation methods focus directly on user perception, selecting a certain number of testers to score the quality of the network application's user experience under specific conditions according to scoring criteria. Finally, the network application's service quality is obtained through statistical analysis of the scoring results. While subjective evaluation can directly reflect the user's service experience, its experimental costs are high and the process is cumbersome. Therefore, subjective evaluation is often used to verify the results of objective evaluation. The International Organization for Standardization (ITU-R) BT.500-11 introduces several subjective evaluation methods. [1]: Single Stimulus Method (SSM) and Double Stimulus Impairment Scale (DSIS). In the Single Stimulus Method test, the staff randomly shuffled the test objects and handed them over to the testers for scoring. The same test object may appear multiple times. All subjective scores are statistically evaluated and processed to obtain the average opinion score. In the Double Stimulus Impairment Scale, the same test object will appear continuously. First, the testers will check the unimpaired test objects, and then score the impaired test objects. The score represents the degree of impairment of the impaired test objects. Objective evaluation is to monitor relevant indicators that affect the quality of network application services, establish a service quality evaluation model, and indirectly represent the user experience. Most commonly used objective methods are based on machine learning. By learning from a large number of sample user data, the mapping relationship between each indicator and the QoE of the webpage is obtained. Reference [2] proposes to use decision tree + AdaBoost to establish a mapping model from the QoE of video streaming services to the user experience score, and to predict the actual user experience. However, the service quality evaluation method described in the article can only be used on its self-built simulation platform and cannot be used in actual situations to evaluate real service quality.

[0004] Internationally, there is ongoing research and development to improve web application service quality assessment systems. Lighthouse, launched by Google, is a service quality assessment tool for web pages. It can detect multiple web page performance metrics, analyze web application performance, generate reports, and provide web page performance optimization suggestions. Its advantages include easy installation and local node operation, but its scoring templates are not publicly available, making it impossible to customize the assessment template to suit specific needs. WebPageTest, based in the US, is a professional web page performance analysis tool. It allows for highly customized configuration of the testing and analysis environment, including the physical location of test nodes, device models, browser versions, network conditions, and the number of tests. However, WebPageTest's monitoring is mostly located overseas, resulting in limited data volume and unstable results when monitoring and evaluating domestic websites. Pytomo is a service quality testing tool for YouTube videos. By simulating user usage, it obtains metrics such as YouTube's server response latency, download speed, and buffering time, but it does not provide service quality evaluation or scoring. These tools are all designed for single web applications, with fixed and unpublished assessment templates, failing to adequately meet the basic needs of service quality assessment for various types of web applications and cannot be completely copied. In response, this invention was developed to meet the needs of refined service quality assessment for various types of network applications. It features multi-indicator monitoring, construction of service quality assessment models for various application types, flexible template configuration, and multi-node comprehensive assessment functions, which are of great significance for improving application service quality and enhancing user experience.

[0005] Lighthouse, proposed by Google Chrome, is an open-source web service quality assessment tool. It estimates web performance by setting metric scores and metric weights and using a weighted average method. The metrics in Lighthouse come from Google's "web metrics" [3]. Google selected five metrics to evaluate web service quality, including First Content Render Time (FCP), Speed ​​Index (SI), Maximum Content Render Time (LCP), Total Blocking Time (TBT), and Cumulative Layout Offset Time (CLS). The metric scores are determined by a distribution function obtained from the data provided by Google User Experience Reports, with a score range of 0 to 100. The weights are set as follows: 10%, 10%, 25%, 30%, and 25%. The explanations of each metric are based on Google's "Web Metrics" as follows: FCP is the time from the start of page loading until any part of the page content is rendered on the screen; SI measures the visual speed at which content appears during page loading; LCP is a user-centric metric for measuring perceived loading speed, representing the relative time it takes for the largest visible block of image or text to be rendered; TBT is an important metric for measuring interactive reliability, representing the time the main thread is blocked between FCP and Time to Interact (TTI); CLS is a user-centric metric for measuring visual stability, helping to quantify the frequency with which users experience unexpected layout shifts. Summary of the Invention

[0006] The embodiments of the present invention provide a multi-type network application service quality assessment system and method to solve the technical problems existing in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution.

[0008] A multi-type network application service quality assessment system, including a network service comprehensive indicator monitoring subsystem, a network service quality fine assessment subsystem, and a service grading assessment visualization subsystem;

[0009] The network service comprehensive indicator monitoring subsystem is used to: define indicators for different types of network application evaluation templates; receive monitoring and evaluation instructions sent by the central control node; and monitor the indicators of different types of network application evaluation templates by establishing corresponding monitoring models. Different types of network application evaluation templates include web page indicators, streaming media indicators, and instant messaging indicators.

[0010] The network service quality fine-grained evaluation subsystem is used to: determine the scoring criteria for individual indicators; use the analytic hierarchy process (AHP) to determine the weights between different indicators, develop scoring templates for different network applications, and obtain the overall score for network application service quality.

[0011] The service grading assessment visualization subsystem is used to: provide users with interactive operations with the service quality assessment system for various types of network applications, and to set application service quality parameters; provide the function of storing monitoring and assessment data; and to visualize the assessment results.

[0012] Preferably, the network service quality fine evaluation subsystem includes a monitoring indicator scoring module, a hierarchical analysis module, and a service quality scoring module;

[0013] The monitoring indicator scoring module is used to: determine the scoring criteria for monitoring indicators, and score the results of monitored web page indicators, streaming media indicators, and instant messaging indicators;

[0014] The hierarchical analysis module is used to: establish a hierarchical structure model to classify indicator data into hierarchical levels and determine the weight of each level of indicator; based on the two-level indicator system, compare the indicator data levels pairwise to construct a comparison matrix; calculate the weight vector of the comparison matrix and verify the consistency of the weight vector; and combine the weights of each level of indicator to obtain the indicator weights of all indicator data that have not been classified into hierarchical levels.

[0015] The service quality scoring module is used to: map the scores of the monitoring values ​​input from the indicator monitoring subsystem according to the scoring criteria obtained from the monitoring indicator scoring module; and score the quality of network application services monitored according to the indicators selected by the user, the scoring template, and the weight matrix determined by the hierarchical analysis module.

[0016] Preferably, the monitoring indicator scoring module scores the results of the monitored webpage indicators through the following process:

[0017] A1 reads a piece of obtained URL data and determines whether the number of URLs n in the read URL data is 0. If it is, proceed to step A2; otherwise, save the value of the webpage indicator, decrement the number of URLs n by 1, and then re-execute this step.

[0018] A2 performs data cleaning and processing on the webpage metrics data;

[0019] A3 stores the cleaned webpage metrics data into an xlsx file;

[0020] A4 reads the webpage metrics data after data cleaning and processing to obtain the metric data distribution.

[0021] A5 specifies and saves the scoring criteria for webpage indicators based on the distribution of indicator data.

[0022] The monitoring indicator scoring module scores the results of the monitored streaming media indicators through the following process:

[0023] B1 records videos of the original user experience played by streaming applications;

[0024] B2 changes the metrics of the video multiple times through a tool;

[0025] B3 records the user streaming media playback experience videos under different metrics and different degrees of loss based on the execution result of step B2;

[0026] B4 scores the user streaming media playback experience videos under different metrics and different degrees of loss respectively;

[0027] B5 performs a data cleaning operation on the scoring results of step B4;

[0028] B6 determines the mapping relationship between the metric data and the scoring results based on the execution result of step B5;

[0029] Step B5 specifically includes:

[0030] B51 sets the scoring error range as A, the result of the first scoring as X, and T as the screening intensity;

[0031] B52 If the proportion of the repeated scoring results within the range of [X - A, X + A] is P, and P < T, the scoring result is determined to be valid;

[0032] B53 sets the scoring result of user i as Ri and the screening intensity as T;

[0033] B54 If the scoring result of user i satisfies then the scoring result is determined to be valid;

[0034] The monitoring metric scoring module scores the results of the monitored instant messaging metrics through the following process:

[0035] C1 records the user experience instant messaging application video in the original situation;

[0036] C2 changes the metrics of the user experience instant messaging application video in the original situation multiple times through a tool;

[0037] C3 records the user instant messaging playback experience videos under different metrics and different degrees of loss based on the execution result of step C2;

[0038] C4 scores the user instant messaging playback experience videos under different metrics and different degrees of loss respectively;

[0039] C5 performs a data cleaning operation on the scoring results of step C4;

[0040] C6 determines the mapping relationship between the metric data and the scoring results based on the execution result of step C5.

[0041] Preferably, the service grading assessment visualization subsystem includes: an assessment selection module, a registration and login module, a personal center module, a log auditing module, and a user management module;

[0042] The evaluation selection module is used to: provide different input parameter setting options for network service quality evaluation; send the selected input parameters to the network service comprehensive index monitoring subsystem; and visualize the evaluation results of the network service quality fine evaluation subsystem.

[0043] The registration and login module provides users with system login, registration, and logout functions;

[0044] The user center module is used to view all historical test results of a user's account;

[0045] The log auditing module is used to record all errors and exceptions during system operation;

[0046] The user management module is used to: add registered users, delete registered users, modify registered users, and manage the permissions of registered users.

[0047] Preferably, the network service comprehensive indicator monitoring subsystem includes a webpage indicator monitoring module, a streaming media indicator monitoring module, an instant messaging indicator monitoring module, and a monitoring type selection module;

[0048] The webpage metrics monitoring module is used to: define and evaluate the metrics of webpage type templates by constructing a two-level metric system for webpages; establish a model for monitoring webpage type-related values ​​based on the defined and evaluated webpage metrics; and design a webpage crawler program to crawl webpage data.

[0049] The streaming media metrics monitoring module is used to: define and evaluate the metrics of streaming media types by constructing a two-level metric system for streaming media; and establish a model for monitoring relevant values ​​of streaming media types based on the defined and evaluated streaming media metrics.

[0050] The instant messaging metrics monitoring module is used to: define and evaluate the metrics of instant messaging types by constructing a two-level metric system for instant messaging; and establish a model for monitoring the relevant values ​​of instant messaging types based on the defined and evaluated instant messaging metrics.

[0051] The monitoring type selection module is used to: receive monitoring instructions sent by the master control node, call the corresponding indicator monitoring function, and pass the indicator detection results to the service quality fine evaluation subsystem.

[0052] Preferably, the hierarchical analysis module establishes a hierarchical structure model to classify URL indicator data into hierarchical levels and determines the weight of each level of indicator. This process includes:

[0053] Determine the weights of the primary indicators;

[0054] Determine the weight of secondary indicators relative to primary indicators;

[0055] The weights of all indicators are obtained by multiplying the weight of the primary indicator by the weight of the secondary indicator relative to the primary indicator.

[0056] The hierarchical analysis module, based on a two-level indicator system, performs pairwise comparisons of indicator data levels, constructs a comparison matrix, and calculates the weight vector of the comparison matrix. The process of verifying the consistency of the weight vector includes:

[0057] The first comparison matrix is ​​constructed by comparing each indicator in the primary indicators pairwise.

[0058] By comparing each indicator in the secondary indicators pairwise, a second comparison matrix, a third comparison matrix, and a fourth comparison matrix are constructed respectively.

[0059] Calculate the maximum eigenvalue λ and the corresponding eigenvector W of the first, second, third, and fourth contrast matrices;

[0060] Through

[0061] CI=λ-n / n-1

[0062] The consistency index CI of the first comparison matrix, the second comparison matrix, the third comparison matrix and the fourth comparison matrix are calculated respectively; where λ is the largest eigenvalue and n is the number of indices.

[0063] Through

[0064] CR = CI / RI

[0065] The consistency ratios CR of the first, second, third, and fourth comparison matrices are calculated respectively; where RI is the random consistency index.

[0066] If CR < 0.1, the test passes, and the weight vector w is used as the eigenvector; if CR >= 0.1, the test fails, and the process returns to the sub-step of constructing the comparison matrix.

[0067] The weights of each level of indicators are combined to obtain the indicator weights of all indicator data without hierarchical classification.

[0068] Preferably, in the two-level indicator system of the webpage, the first-level indicators include response time attribute, transmission rate attribute and service robustness; the second-level indicators include: connection establishment time, DNS resolution time, transmission time and first packet time in the response time attribute, and download speed, maximum content rendering rate (LCP) and first content rendering rate (FCP) in the transmission rate attribute.

[0069] The service robustness attributes include packet loss rate, latency jitter, and HTTP status codes;

[0070] In the two-level indicator system of streaming media, the first-level indicators include response time, transmission rate, and viewing attributes; the second-level indicators include: connection establishment time, DNS resolution time, total transmission time, and first packet time in the response time attributes; download speed, bitrate, and frame rate in the transmission rate attributes; and transmission packet loss rate, latency jitter, and frame rate in the viewing attributes.

[0071] In the two-level indicator system of instant messaging, the first-level indicators include response time attributes, interaction quality attributes, and service robustness attributes; the second-level indicators include: login time and dial-up connection time in response time attributes, file transfer speed, message sending latency, image sending rate, and video sending rate in interaction quality attributes, and disconnection rate and error codes in service robustness attributes.

[0072] Secondly, the present invention provides a method for evaluating the quality of service of multiple types of network applications, including:

[0073] S1 Select the type of network application service to be evaluated, select the node to perform the monitoring and evaluation task, enter the URL of the network application service to be tested, select a custom monitoring module or a fixed monitoring template, and call the indicator API to collect initial information.

[0074] Based on the initial information collected, S2 defines the indicators for different types of network application evaluation templates; it receives monitoring and evaluation instructions sent by the central control node, and monitors the indicators for different types of network application evaluation templates by establishing corresponding monitoring models.

[0075] S3 determines the scoring criteria for individual indicators; uses the analytic hierarchy process (AHP) to determine the weights between different indicators, develops scoring templates for different network applications, and obtains the overall score for the quality of network application services.

[0076] S4 provides a visual output of the evaluation results from the network service quality fine-grained evaluation subsystem.

[0077] As can be seen from the technical solutions provided by the embodiments of the present invention above, the multi-type network application service quality assessment system and method provided by the present invention includes a network service comprehensive indicator monitoring subsystem, a network service quality fine assessment subsystem, and a service grading assessment visualization subsystem. The network service comprehensive indicator monitoring subsystem is used to: define indicators for different types of network application assessment templates; receive monitoring and assessment instructions sent by the central control node; and monitor the indicators of different types of network application assessment templates by establishing corresponding monitoring models; the different types of network application assessment templates include web page indicators, streaming media indicators, and instant messaging indicators. The network service quality fine assessment subsystem is used to: determine the scoring standards for individual indicators; use the analytic hierarchy process (AHP) to determine the weights between different indicators, formulate scoring templates for different network applications, and obtain the total score for network application service quality. The service grading assessment visualization subsystem is used for: user interaction with the multi-type network application service quality assessment system, setting application service quality parameters, and storing monitoring and assessment data. The system and method provided by this invention address the problem of coarse-grained evaluation indicators by formulating multiple evaluation indicators to achieve precise detection and perception of application service status at a fine-grained level. To address the issue of fixed and undisclosed evaluation models, multiple indicators are selected to form different evaluation templates for different network applications, while also supporting user-defined templates to achieve diversified evaluation for different applications. Furthermore, to address the problem of single detection nodes originating from overseas, a deployment topology with a central control node and sub-nodes is designed, enabling the system to comprehensively evaluate the service quality of applications across different network nodes, resulting in more realistic and effective evaluation results.

[0078] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0079] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0080] Figure 1 Architecture diagram of the multi-type network application service quality assessment system provided by the present invention;

[0081] Figure 2 Architecture diagram of the network service comprehensive index monitoring subsystem of the multi-type network application service quality evaluation system provided by the present invention;

[0082] Figure 3 Architecture diagram of the network service quality fine evaluation subsystem of the multi-type network application service quality evaluation system provided by the present invention;

[0083] Figure 4 A distribution chart of connection establishment time indicators for the multi-type network application service quality assessment system provided by the present invention;

[0084] Figure 5 A flowchart illustrating the process of obtaining webpage indicator scoring criteria for the multi-type network application service quality evaluation system provided by this invention.

[0085] Figure 6 A flowchart illustrating the process of obtaining the streaming media indicator scoring criteria for the multi-type network application service quality evaluation system provided by this invention.

[0086] Figure 7 Flowchart for obtaining the scoring criteria of instant messaging indicators for the multi-type network application service quality evaluation system provided by the present invention;

[0087] Figure 8 A flowchart for determining the weights of indicators in the hierarchical analysis of the multi-type network application service quality evaluation system provided by this invention;

[0088] Figure 9 The architecture diagram of the service grading assessment visualization subsystem of the multi-type network application service quality assessment system provided by the present invention;

[0089] Figure 10 A flowchart illustrating the usage of the multi-type network application service quality assessment system provided by this invention;

[0090] Figure 11 A flowchart illustrating the process of the multi-type network application service quality assessment method provided by this invention;

[0091] Figure 12 The topology diagram of the multi-type network application service quality assessment system cluster provided by the present invention.

[0092] In the picture:

[0093] 101. Network Service Comprehensive Index Monitoring Subsystem; 102. Network Service Quality Fine-grained Assessment Subsystem; 103. Service Classification Assessment Visualization Subsystem;

[0094] 201. Webpage Indicator Monitoring Module; 202. Streaming Media Indicator Monitoring Module; 203. Instant Messaging Indicator Monitoring Module; 204. Monitoring Type Selection Module;

[0095] 301. Monitoring Indicator Scoring Module; 302. Hierarchical Analysis Module; 303. Service Quality Scoring Module;

[0096] 901. Evaluation and Selection Module; 902. Registration and Login Module; 903. Personal Center Module; 904. Log Audit Module; 905. User Management Module. Detailed Implementation

[0097] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0098] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0099] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0100] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0101] This invention provides a service quality assessment system and method for different network applications, mainly to solve the following technical problems existing in the prior art.

[0102] Existing technology 1, Lighthouse, is an open-source automation tool proposed by Google Chrome. It uses a score range of 0-100 to represent the quality of web page performance and QoE (Quality of Service). Its score for web service quality is equal to the sum of the products of each metric's score and its corresponding weight. Lighthouse's metrics are derived from Google's "web metrics." In Lighthouse, the five selected metrics include First Content Render Time (FCP), Speed ​​Index (SI), Maximum Content Render Time (LCP), Total Blocking Time (TBT), and Cumulative Layout Offset Time (CLS). The score mapping for each metric value is determined by a distribution function obtained from data provided by Google's root user experience report. The score range is 0 to 100, with weight values ​​of 10%, 10%, 25%, 30%, and 25%. The drawback of this existing technology is:

[0103] (1) The evaluation indicators of the technical model are few, the evaluation results of the webpage may be one-sided, and the evaluation granularity is coarse; the evaluation template is fixed, and users cannot choose the indicators they want and change them according to their needs; the service quality evaluation monitoring node is only the host running Lighthouse, and the evaluation results are easily affected by the node's own condition and are not accurate enough.

[0104] (2) The scheme for determining the weight of indicators in Technique 1 was not made public. Instead, the weight results were directly published. The scoring results may not be accurate enough and cannot be convincing.

[0105] Existing technology two, targeting streaming media services under cloud services, selects initial buffer latency, buffer duration, number of buffers, buffer percentage, number of bitrate switches, and average bitrate as objective evaluation indicators. Based on the different levels of importance of each indicator to QoE, it uses the CRITIC weighting method to perform weighted clustering of QoE indicators; it divides the QoE scores into five levels and uses a decision tree + AdaBoost approach to establish a mapping model from QoE of video streaming services to user experience scores, predicting the actual user experience. The drawback of this technical solution is:

[0106] (1) Technique 2 only involves manually controlling parameters in a self-built server environment, lacking evaluation of real business scenarios and verification of model accuracy.

[0107] (2) The service quality assessment results in Technology 2 simply divide the user experience score into five levels, lacking a refined service quality assessment result.

[0108] See Figure 1The present invention provides a multi-type network application service quality assessment system, including a network service comprehensive index monitoring subsystem 101, a network service quality fine assessment subsystem 102, and a service level assessment visualization subsystem 103;

[0109] The Network Service Comprehensive Index Monitoring Subsystem 101 is used to: define the indicators for different types of network application evaluation templates; receive monitoring and evaluation instructions sent by the central control node; and monitor the indicators of different types of network application evaluation templates by establishing corresponding monitoring models; the different types of network application evaluation templates include web page indicators, streaming media indicators, and instant messaging indicators.

[0110] The network service quality fine evaluation subsystem 102 is used to: determine the scoring criteria for individual indicators; use the analytic hierarchy process to determine the weights between different indicators, formulate scoring templates for different network applications, and obtain the total score for network application service quality.

[0111] The service grading assessment visualization subsystem 103 is used to: provide users with interactive operations with the multi-type network application service quality assessment system, and the function of setting application service quality parameters; provide the function of storing monitoring and assessment data; and visualize the assessment results. Users can select the type of application to be tested, application address, scoring and assessment template, and monitoring nodes, and display the indicator monitoring results and total score in the form of charts. A webpage is also provided to help users use the system and view the assessment results.

[0112] The purpose of this invention is to meet the practical business needs for customized and refined service quality assessment of different types of network applications, and to achieve accurate and objective service quality assessment for various types of network applications. This invention addresses the technical problems in current network application service quality assessment, such as coarse-grained assessment methods, fixed and undisclosed assessment models, and a single monitoring node. To this end, it provides a multi-type network application service quality assessment method and system. To address the problem of coarse-grained assessment indicators, multiple assessment indicators are formulated to achieve precise detection and perception of application service status at a fine-grained level. To address the problem of fixed and undisclosed evaluation models, multiple indicators are selected to form different assessment templates for different network applications, while also supporting user-defined templates to achieve diversified assessments for different applications. To address the problem of a single detection node originating from abroad, a deployment topology with a central control node and sub-nodes is designed, enabling the system to comprehensively assess the service quality of applications across different network nodes, resulting in more realistic and effective assessment results.

[0113] Network Service Comprehensive Index Monitoring Subsystem

[0114] In a preferred embodiment provided by the present invention, the network service comprehensive index monitoring subsystem 101 is located at the monitoring node, such as... Figure 2As shown, it consists of a webpage indicator monitoring module 201, a streaming media indicator monitoring module 202, an instant messaging indicator monitoring module 203, and a monitoring type selection module 204. Its functions include defining indicators for evaluating different types of network application templates, designing functions and programs to monitor their relevant values, receiving and parsing monitoring commands sent by the master control node, and returning the monitoring results to the master control node.

[0115] Webpage Indicator Monitoring Module

[0116] (1) Determination of webpage type indicator system

[0117] The webpage metrics monitoring module 201 first needs to identify the key metrics affecting webpage service quality. After analysis, a secondary metric system containing 13 metrics was selected. The primary metrics are: response time attribute, transmission rate attribute, and service robustness attribute. The secondary metrics are the 13 metrics under the primary metrics, including: In the response time attribute, connection establishment time, first packet time, DNS resolution time, network latency, total transmission time, SSL handshake time, and redirection time; in the transmission rate attribute, download speed, maximum content rendering rate (LCP), and first content rendering rate (FCP); and in the service robustness attribute, packet loss rate, latency jitter, and HTTP status codes. The secondary metric system is shown in Table 1.

[0118] Table 1. Webpage Type Index Classification Table

[0119]

[0120] Response time attribute: the speed at which a page loads and the amount of content loaded within a certain time.

[0121] Connection establishment time: By using the timestamp, ACK, SEQ and other fields in the TCP packet, the time spent from the client sending a TCP connection request to receiving the acknowledgment packet returned by the server is calculated, which is the time spent successfully establishing a TCP connection.

[0122] First packet time: refers to the time required from when the client sends an HTTP GET request to when it receives the first HTTP protocol data packet returned by the server.

[0123] SSL handshake time: For HTTPS business requests, the time taken from the client to completing the SSL / SSH connection with the server.

[0124] Network latency: The time it takes for a data packet to travel from the client to the server.

[0125] Transmission time: The time it takes for the client and server to complete all data transmission after establishing a connection.

[0126] DNS resolution time: The time it takes for a client to complete domain name resolution from the moment it sends a DNS request message to the domain name server until it receives the DNS response message.

[0127] Transfer rate attributes: the speed at which resources are downloaded and the speed at which page content is loaded.

[0128] Content download speed: The average download speed at which a client downloads all the data from the website from the server.

[0129] Maximum Content Render Rate (LCP): This is an important user-centric metric used by Google to measure perceived loading speed. It marks the point in time on the page loading timeline when the main content of the page has essentially finished loading.

[0130] First Content Render Rate (FCP): This is an important user-centric metric used by Google to measure perceived loading speed. It marks the point on the page load timeline from the moment a user first sees any content on the screen.

[0131] Service robustness property: The performance of a webpage in providing services under poor network conditions.

[0132] Packet loss rate: The ratio of the number of TCP retransmission packets between the client and the server to the total number of packets transmitted.

[0133] Network jitter: The average of the results of multiple network latency tests over a period of time is calculated by taking the difference between the results before and after the tests.

[0134] HTTP status codes: Numeric codes returned by the server to the client indicating the completion status of the request.

[0135] (2) Implementation of webpage indicator monitoring functions

[0136] By using methods provided in the pycurl library and performing secondary development with Python code, we can monitor connection establishment time, DNS resolution time, total transmission time, first packet time, SSL handshake time, redirection time, HTTP status codes, and download speed metrics.

[0137] We implemented the monitoring of Maximum Content Render Rate (LCP) and First Content Render Rate (FCP) metrics by modifying and rewriting the open-source code of Lighthouse, Google's web metrics scoring tool.

[0138] By combining the tsharke tool with Python code, packet capture statistics are performed on the URL of the website under test. The ratio between the number of retransmitted TCP packets and the total number of transmitted TCP packets within a certain period of time is used to obtain the packet loss rate monitoring results.

[0139] Using Python's ping3 library, ICMP request messages are sent to a specified website. Based on the timestamp of the returned messages, network latency and jitter metrics are statistically analyzed and calculated.

[0140] Streaming Media Metrics Monitoring Module

[0141] (1) Determination of streaming media type indicator system

[0142] The streaming media metrics detection module 202 first needs to identify the key metrics affecting the quality of streaming media services. Based on analysis, we designed and selected a secondary metric system comprising 11 metrics. The primary metrics are: response time attribute, transmission rate attribute, and viewing quality attribute. The secondary metrics are the 11 sub-metrics under the primary metrics, including: connection establishment time, DNS resolution time, total transmission time, and first packet time in the response time attribute; download speed, bitrate, and packet loss rate in the transmission rate attribute; and transmission packet loss rate, latency jitter, HTTP status codes, and buffering rate in the viewing quality attribute. The secondary metric system is shown in Table 2.

[0143] Table 2. Streaming Media Type Index Classification Table

[0144]

[0145] Response time attribute: the speed at which a page loads and the amount of content loaded within a certain time.

[0146] Connection establishment time: By using the timestamp, ACK, SEQ and other fields in the TCP packet, the time spent from the client sending a TCP connection request to receiving the acknowledgment packet returned by the server is calculated, which is the time spent successfully establishing a TCP connection.

[0147] First packet time: refers to the time required from when the client sends an HTTP GET request to when it receives the first HTTP protocol data packet returned by the server.

[0148] Total transmission time: The total time spent from the start of transmission to the completion of all data transmission after the client and server establish a connection.

[0149] DNS resolution time: The time it takes for a client to complete domain name resolution from the moment it sends a DNS request message to the domain name server until it receives the DNS response message.

[0150] Transfer rate attribute: How fast resources are downloaded from the server.

[0151] Content download rate: The rate at which a client downloads streaming media data from a server.

[0152] Bitrate: The amount of data used by a video file per unit of time.

[0153] Frame rate: The number of images played per second.

[0154] Viewing quality attribute: The stability of the service provided by the streaming application

[0155] Packet loss rate: The ratio of the number of TCP retransmission packets between the client and the server to the total number of packets transmitted.

[0156] Network jitter: The average of the results of multiple network latency tests over a period of time is calculated by taking the difference between the results before and after the tests.

[0157] Stuttering rate: When the video's buffering speed is less than its playback speed, and the buffered content has finished playing, the ratio of the total waiting time of the buffered content to the time it takes for the video to play normally is called the stuttering rate.

[0158] (2) Implementation of streaming media metric monitoring function

[0159] By using methods provided in the pycurl library and performing secondary development with Python code, we can monitor metrics such as connection establishment time, DNS resolution time, start / end time, and first packet time.

[0160] The download speed is obtained by dividing the downloaded video size by the download time using Python's requests library.

[0161] Use Python's cv2 library to obtain the video's bitrate, frame rate, and duration. If the video duration is greater than the download time, the stuttering rate is 0; otherwise, the stuttering rate is calculated as (download time - video duration) / video duration.

[0162] By combining the tsharke tool with Python code, packet capture statistics are performed on the URL of the website under test. The ratio between the number of retransmitted TCP packets and the total number of transmitted TCP packets within a certain period of time is used to obtain the packet loss rate monitoring results.

[0163] Using Python's ping3 library, ICMP request messages are sent to a specified website. Based on the timestamp of the returned messages, network latency and jitter metrics are statistically analyzed and calculated.

[0164] Instant Messaging Metrics Monitoring Module

[0165] (1) Determination of Instant Messaging Type Index System

[0166] The instant messaging metric detection module 203 first needs to identify the key metrics affecting the quality of instant messaging services. Based on analysis, we designed and selected a secondary metric system comprising eight metrics. The primary metrics are: response time attribute, interaction quality attribute, and service robustness attribute. The secondary metrics are the eight sub-metrics under the primary metrics, including: login time and dial-up connection time in the response time attribute; message sending latency, file transfer rate, image sending rate, and video sending rate in the interaction quality attribute; and disconnection rate and error codes in the service robustness attribute. The secondary metric system is shown in Table 3.

[0167] Table 3 Instant Messaging Type Index Classification Table

[0168]

[0169] Response time attribute: The time it takes for the instant messaging server to respond after receiving a request.

[0170] Login time: The time it takes for the client to send a login request to the server and complete the login process.

[0171] Dialing connection time: The time taken from when the client sends a voice dialing request to when the phone rings successfully.

[0172] Interaction quality attribute: How quickly an instant messaging application completes its basic functions.

[0173] File transfer rate: The rate at which files are sent and transferred between clients.

[0174] Message sending latency: The time it takes for a client to send a short message and for the recipient to successfully receive it.

[0175] Image sending rate: The rate at which images are sent and transmitted between clients.

[0176] Video transmission rate: The rate at which video is transmitted between clients.

[0177] Service robustness attribute: the stability of an instant messaging application in providing normal service.

[0178] Disconnection rate: Diaox instant messaging client disconnection time divided by normal login online time.

[0179] Error code: The error code returned when the instant messaging server encounters an exception.

[0180] (2) Implementation of instant messaging indicator monitoring function

[0181] For the instant messaging application Telegram, the methods provided by the Python pyrogram library are used to realize the interaction between the client and the Telegram server, complete functions such as login, message sending, voice dialing, and file transfer, and monitor the time spent completing these functions to obtain the metric monitoring results.

[0182] For the instant messaging application WhatsApp, the WhatsApp-web.js library in Node.js is used to realize the interaction between the client and the WhatsApp server, complete functions such as login, message sending, voice dialing, and file transfer, and monitor the time spent completing these functions to obtain the metric monitoring results.

[0183] Monitoring type selection module

[0184] The main function of the monitoring type selection module 204 is for the monitoring node to receive monitoring instructions sent by the master control node, call the corresponding indicator monitoring function, and pass the indicator detection results into the service quality assessment subsystem. The specific process is as follows:

[0185] The master node sends monitoring instructions in JSON format to the monitoring nodes via HTTP protocol. This transmits the monitoring instructions set by the user. In the JSON data, the key "author" corresponds to the username of the user performing the test; the key "apptypes" corresponds to the type of application selected by the user; the key "url" corresponds to the address of the application to be tested; the key "fixed" corresponds to the type of application to be tested; and the key "testmodels" corresponds to the service quality evaluation template selected by the user. After the metric monitoring is completed, the metric results are passed to the evaluation template function selected by the user for subsequent evaluation and scoring.

[0186] Service Quality Fine-Grained Evaluation Subsystem

[0187] In the preferred embodiments provided by the present invention, such as Figure 3 As shown, the service quality fine-grained assessment subsystem is located at the monitoring node and consists of a monitoring indicator scoring module, a hierarchical analysis module, and a service quality scoring module. Its function is to formulate scoring standards for monitoring indicators, determine the weights among each indicator, and obtain the overall service quality score.

[0188] Monitoring indicator scoring module

[0189] The function of the monitoring indicator scoring module 301 is to determine the scoring criteria for the monitoring indicators and score the results of individual monitoring indicators.

[0190] For web-based applications, write a crawler program to crawl a sufficient amount of URL data, and call the web metric monitoring module to monitor the metrics related to the service quality of these websites. Then, clean the obtained metric data by removing outliers and anomalies that interfere with the scoring criteria. Finally, define the metric scoring intervals based on the distribution of the metric monitoring results. For example, establish the distribution of the connection time metric as Figure 4 . The metric scoring range is from 0 to 100 points. Based on the percentage of the monitored metric value in the total metric values, divide the data distribution into 10 equal parts. That is, the data in the first equal part corresponds to 0 - 10 points, the data in the second equal part corresponds to 10 - 20 points, and so on. Among them, the metric data values at the endpoints of each equal part's score interval are taken as the top 10% values of their respective parts. For example, when the endpoint score is 10 points, take the metric data value at 10% of the data distribution. The score mapping for the data within each equal part is a linear mapping obtained based on the metric data values and scores at the endpoints. The process of data acquisition and processing to obtain the scoring criteria is as Figure 5 shown.

[0191] For streaming media type applications, as Figure 6 shown, first record the original streaming media application's playback video, and then use tools such as moviepy and TC to control the change of a single metric of the streaming media application each time, and record the streaming media playback video under the condition of metric degradation; distribute a scoring questionnaire. Users first watch the recorded video of the streaming media playback under normal conditions, and then watch several playback videos under different metric adjustments to experience the change of the service quality of the streaming media application under different metric changes, and score their experience feelings, and collect the scoring results. To further ensure the reliability of the scoring results, clean the collected scoring results. The data cleaning process is as follows: (1) Individual consistency: To improve the data validity, the questionnaire will randomly provide duplicate service quality videos for users to score. Set the scoring error interval as A, the user's first scoring result as X, and T as the screening intensity. If the proportion of the repeated scoring results within the interval [X - A, X + A] is P, when P < T, it is considered that the user's scoring result for this metric is valid. (2) Individual deviation: Individual deviation is used to measure the difference between a single user's score for a video within a single subtask and the average score of other users. Denote the scoring result of user i as Ri and the screening intensity T. When the user's scoring result satisfies when, it is considered that the user's scoring result for this metric is valid. Based on the scoring results of users for the service quality under different metrics, establish the mapping relationship between the monitored metric data and the scores. The above two processes are executed simultaneously.

[0192] For instant messaging type applications, similar to the process of formulating the metric scoring criteria for streaming media applications, as Figure 7As shown, videos of users using the instant messaging application under normal conditions were first recorded. Then, using tools such as TC (Tracking Control), individual metrics were controlled to change each time, and videos of application usage under degraded metrics were recorded separately. A rating questionnaire was distributed. Users first watched a video of the instant messaging application's usage under normal conditions, and then watched several videos under different metric adjustments. They experienced the changes in the instant messaging application's service quality under different metric changes, rated their user experience, and the rating results were collected. To ensure the validity of the rating results, individual consistency and individual bias methods were used to clean the data. Based on the users' service quality ratings under different metrics, a direct mapping relationship between the monitoring metric data and the ratings was established.

[0193] Hierarchical Analysis Module

[0194] The function of the analytic hierarchy process (AHP) module 302 is to use the AHP method to determine the weights among different indicators affecting service quality, such as... Figure 7 As shown, the process of determining each indicator using the analytic hierarchy process is as follows:

[0195] 1. Establish a hierarchical structure model. Due to the large number of indicators, they are divided into two levels based on their type. First, determine the weight of the first-level indicators, then determine the weight of the second-level indicators relative to the first-level indicators, and finally, the weight of all indicators is equal to the weight of the first-level indicators multiplied by the weight of the second-level indicators relative to the first-level indicators.

[0196] 2. Determine the pairwise comparison matrix for the indicators. As mentioned above, since the indicators are divided into two levels, the weights between the first-level indicators are determined first, and then the second-level indicators are determined. Taking a web-based application as an example, this includes three first-level indicators: response time, transmission rate, and service robustness. By comparing them pairwise, a pairwise comparison matrix is ​​constructed. Then, for the second-level indicators under the response time attribute: connection establishment time, DNS resolution time, total transmission time, first packet time, SSL handshake time, network latency, and redirection time, a pairwise comparison matrix is ​​constructed. For the second-level indicators under the transmission rate attribute: download speed, maximum content rendering rate (LCP), and first content rendering rate (FCP), a pairwise comparison matrix is ​​constructed. Finally, for the service robustness attributes: packet loss rate and latency jitter, a pairwise comparison matrix is ​​constructed.

[0197] 3. Calculate the weight vector and perform a consistency check. For each pairwise comparison matrix mentioned above, calculate the largest eigenvalue λ and the corresponding eigenvector W, and perform a consistency check using the consistency index CI, the random consistency index RI, and the consistency ratio CR. Where CI = λ - n / n - 1, where λ is the largest eigenvalue and n is the number of indices; the standard for RI is shown in Table 4 below; CR = CI / RI. If CR < 0.1, the check passes, and the eigenvector (after normalization) is the weight vector w; if CR >= 0.1, the check fails, and the pairwise comparison matrix needs to be reconstructed.

[0198] Table 4 Random Consistency Index (RI)

[0199] n 1 2 3 4 5 6 7 8 … RI 0 0 0.52 0.89 1.12 1.36 1.41 1.46 …

[0200] 4. Combine the weight vectors at each level. Combine the weights of the primary indicators with the weights of the secondary indicators relative to the primary indicators to obtain the indicator weights of all indicators in the system when there are no hierarchical levels, and save them.

[0201] In some other embodiments, machine learning methods can also be used to calculate the index weights.

[0202] Machine learning is a science and technology about learning from data. It helps machines learn patterns from existing complex data to predict future behavioral outcomes and trends. To determine the weights of grading metrics in a fine-grained evaluation model for webpage types, machine learning can also be used. This involves normalizing and labeling the data, and inputting the monitoring metric T. i Output index weight W i This is used to determine the webpage rating. The ratings for the three dimensions and the overall rating of a webpage can be calculated using the following formula.

[0203]

[0204] Where S can represent the rating of the webpage in three dimensions and the total rating, and T... i Represented as the score of indicator i corresponding to the rating, W i This represents the weight of indicator i corresponding to the score.

[0205] There are various ways to represent the input to a machine learning model, such as variables, features, and descriptors, depending on the research question or target requirements. In this project, we typically acquire metrics such as connection time, DNS resolution time, download speed, and packet loss rate from webpage monitoring. The data is processed through data cleaning and normalization, and these metrics are also labeled to indicate webpage quality. The input to the machine learning model should be the processed metrics and their labels, and the output should be the weights of each metric. Based on the metric weights, the weights of the three-dimensional metrics in the hierarchical metric system can be naturally obtained, thus determining the metric weights. Knowing the metric weights and scores, we can ultimately achieve webpage performance scoring.

[0206] Service quality scoring module

[0207] The service quality scoring module 303 scores application service quality according to user selection. First, based on the scoring criteria obtained from the monitoring indicator scoring module, it maps the monitoring values ​​input from the indicator monitoring subsystem to a score of 0-100. Then, based on the user-selected indicators, scoring templates, and the weight matrix determined by the hierarchical analysis module, it scores the monitored network application service quality. For each network application, users can use either the fixed scoring template provided by the system or flexibly select monitoring indicators to define new templates. Through flexible selection and combination of indicators, the system achieves diverse scoring templates, enabling refined evaluation of different types of application services and generating a data dictionary that returns the results data via an interface.

[0208] Service grading assessment visualization subsystem

[0209] In a preferred embodiment of the present invention, the service grading assessment visualization subsystem uses the Django framework as the backend, the Vue framework as the frontend, and MySQL as the database. It interacts with users in a web-based visualization manner. After deploying the website, users only need to access the page and log in to assess the quality of network application services, making it more convenient and efficient for users. Figure 9 As shown, the subsystem includes: evaluation and selection module 901, registration and login module 902, personal center module 903, log auditing module 904, and user management module 905.

[0210] The evaluation selection module 901 allows users to set different parameters to conduct service quality assessments using the system, while simultaneously visualizing the results. The user's process for conducting a service quality assessment is as follows: open the visual scoring webpage—select the application type to be tested and enter the application address—select scoring indicators, nodes, and other parameters, click OK to conduct the assessment—view the indicator monitoring results and scoring data charts. The system also supports batch assessments; users can write all the URLs of the applications to be tested into Word, Excel, or TXT files and upload them to the website for batch assessment.

[0211] Registration and Login Module 902: Provides users with system login, registration, and logout functions. Users click the registration button, enter their username, password, and other information, and complete the registration upon verification. Users can log in to use the system by entering their username and password on the website's initial login page.

[0212] User Center Module 903: After logging in, users can click on User Center Module 903 to view all historical test results for that account.

[0213] Log audit module 904: Records all errors and exceptions during system operation, facilitating subsequent upgrades and optimizations.

[0214] User Management Module 905: System administrators can use this module to add, delete, and modify users; and manage user permissions.

[0215] Figure 10 A diagram illustrating the usage process of the system provided by this invention.

[0216] Secondly, such as Figure 11 As shown, the present invention provides a method for evaluating the quality of service of multiple types of network applications, including:

[0217] S1 Select the type of network application service to be evaluated, select the node to perform the monitoring and evaluation task, enter the URL of the network application service to be tested, select a custom monitoring module or a fixed monitoring template, and call the indicator API to collect initial information.

[0218] Based on the initial information collected, S2 defines the indicators for different types of network application evaluation templates; it receives monitoring and evaluation instructions sent by the central control node, and monitors the indicators for different types of network application evaluation templates by establishing corresponding monitoring models.

[0219] S3 determines the scoring criteria for individual indicators; uses the analytic hierarchy process (AHP) to determine the weights between different indicators, develops scoring templates for different network applications, and obtains the overall score for the quality of network application services.

[0220] S4 provides a visual output of the evaluation results from the network service quality fine-grained evaluation subsystem.

[0221] The assessment results are used to monitor and evaluate the service quality of various types of network applications and provide a basis for the subsequent improvement and upgrading of network applications.

[0222] Thirdly, this invention provides a multi-type network application service quality assessment system cluster, including multiple monitoring nodes and a central control node that controls each of the multiple monitoring nodes. Each monitoring node has the aforementioned assessment system. Specifically, it can be as follows: Figure 12As shown, the central control node is used for system data storage and visualization. It sends monitoring and evaluation commands to the monitoring nodes based on user-selected parameters and receives and presents the evaluation results. However, since network applications typically provide varying service quality to users in different regions, evaluating service quality from a single node cannot accurately reflect the global service quality level. Furthermore, the monitoring results may be affected by the node's own network conditions, leading to inaccurate assessments of application service quality during network anomalies. Therefore, this system deploys monitoring nodes globally to acquire indicator data and evaluate service quality for the application in different regions. Users can flexibly choose to view the monitoring results of a single node or select the average of multiple monitoring node results to comprehensively evaluate the network application service quality, achieving a refined and accurate assessment of the service quality of different network applications.

[0223] The present invention also provides two embodiments to exemplarily illustrate application scenarios of the system provided by the present invention.

[0224] Example 1

[0225] Using the multi-type network application service quality assessment system described in this invention, users can either use the several scoring models provided by the system or customize the scoring model parameters as needed to obtain refined evaluation results of the service quality of different applications.

[0226] A. Enterprise developers can use the system of this invention to monitor the network applications they develop, promptly detect service quality anomalies, identify application problems based on the monitoring results, make appropriate adjustments, and improve user experience.

[0227] B. Alternatively, if enterprise developers want to compare the service quality of their application with that of other vendors, they can use this development system to monitor the application, view the service quality gap between different applications based on the evaluation results, and find directions for future application upgrades and optimizations through the indicator monitoring results.

[0228] C. Alternatively, if a network application user wants to find the application with the best service quality and that best suits their needs among several similar applications, they can use the system of this invention to evaluate the service quality of the application and select the most suitable application.

[0229] Example 2

[0230] Using the multi-type network application service quality assessment system described in this invention, users can select different nodes to assess the application service quality.

[0231] A. Users can select different individual monitoring nodes to view the service quality assessment results of that node, reflecting the user experience at that node location.

[0232] B. Alternatively, users can select several monitoring nodes to monitor the application service quality simultaneously, take the average of the evaluation scores, and obtain the comprehensive service quality evaluation result.

[0233] In summary, the multi-type network application service quality assessment system and method provided by this invention includes a network service comprehensive indicator monitoring subsystem, a network service quality fine-grained assessment subsystem, and a service grading assessment visualization subsystem. The network service comprehensive indicator monitoring subsystem is used to: define indicators for different types of network application assessment templates; receive monitoring and assessment instructions sent by the central control node; and monitor the indicators of different types of network application assessment templates by establishing corresponding monitoring models. These different types of network application assessment templates include webpage indicators, streaming media indicators, and instant messaging indicators. The network service quality fine-grained assessment subsystem is used to: determine the scoring standards for individual indicators; use the analytic hierarchy process (AHP) to determine the weights between different indicators; formulate scoring templates for different network applications; and obtain the overall score for network application service quality. The service grading assessment visualization subsystem is used for: user interaction with the multi-type network application service quality assessment system; setting application service quality parameters; and storing monitoring and assessment data. The system and method provided by this invention address the problem of coarse-grained evaluation indicators by formulating multiple evaluation indicators to achieve precise detection and perception of application service status at a fine-grained level. To address the issue of fixed and undisclosed evaluation models, multiple indicators are selected to form different evaluation templates for different network applications, while also supporting user-defined templates to achieve diversified evaluation for different applications. Furthermore, to address the problem of single detection nodes originating from overseas, a deployment topology with a central control node and sub-nodes is designed, enabling the system to comprehensively evaluate the service quality of applications across different network nodes, resulting in more realistic and effective evaluation results.

[0234] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0235] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0236] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0237] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-type network application service quality assessment system, characterized in that, This includes a network service comprehensive indicator monitoring subsystem, a network service quality fine-grained assessment subsystem, and a service grading assessment visualization subsystem; The network service comprehensive indicator monitoring subsystem is used to: define indicators for different types of network application evaluation templates; receive monitoring and evaluation instructions sent by the central control node; and monitor the indicators of the different types of network application evaluation templates by establishing corresponding monitoring models; the different types of network application evaluation templates include web page indicators, streaming media indicators, and instant messaging indicators. The network service quality fine-grained evaluation subsystem is used to: determine the scoring criteria for individual indicators; use the analytic hierarchy process (AHP) to determine the weights between different indicators, develop scoring templates for different network applications, and obtain the overall score for network application service quality. The service grading assessment visualization subsystem is used to: provide users with interactive operations with the multi-type network application service quality assessment system, and the function of setting application service quality parameters; provide the function of storing monitoring and assessment data; and visualize the assessment results.

2. The system according to claim 1, characterized in that, The network service quality fine evaluation subsystem includes a monitoring indicator scoring module, a hierarchical analysis module, and a service quality scoring module. The monitoring indicator scoring module is used to: determine the scoring criteria for the monitoring indicators, and score the results of the monitored web page indicators, streaming media indicators, and instant messaging indicators. The hierarchical analysis module is used to: establish a hierarchical structure model to classify the indicator data into hierarchical levels and determine the weight of each level of indicator; based on the two-level indicator system, compare the indicator data levels pairwise to construct a comparison matrix; calculate the weight vector of the comparison matrix and verify the consistency of the weight vector; and combine the weights of each level of indicator to obtain the indicator weights of all indicator data that have not been classified into hierarchical levels. The service quality scoring module is used to: map the monitoring values ​​input from the indicator monitoring subsystem to scores based on the scoring criteria obtained by the monitoring indicator scoring module; and score the monitored network application service quality based on the user-selected indicators, scoring templates, and the weight matrix determined by the hierarchical analysis module.

3. The system according to claim 2, characterized in that, The monitoring indicator scoring module scores the results of the monitored webpage indicators through the following process: A1 reads a piece of obtained URL data and determines whether the number of URLs n in the read URL data is 0. If it is, proceed to step A2; otherwise, save the value of the webpage indicator, decrement the number of URLs n by 1, and then re-execute this step. A2 performs data cleaning and processing on the webpage metrics data; A3 stores the cleaned webpage metrics data into an xlsx file; A4 reads the webpage metrics data after data cleaning and processing to obtain the metric data distribution. A5 specifies and saves the scoring criteria for webpage indicators based on the distribution of indicator data. The monitoring indicator scoring module scores the results of the monitored streaming media indicators through the following process: B1 records videos of the original user experience played by streaming applications; B2 repeatedly changed the metrics of the video using tools; Based on the execution results of step B2, B3 records user streaming media playback experience videos under different metrics and different levels of loss; B4 Score the user streaming media playback experience videos under different loss degrees of different indicators respectively; B5 Perform data cleaning operations on the scoring results in step B4; B6 Based on the execution result of step B5, determine the mapping relationship between the indicator data and the scoring results; Step B5 specifically includes: B51 Set the scoring error interval as A, the result of the first scoring as X, and T as the screening intensity; B52 If the proportion of the repeated scoring results within the range of [X - A, X + A] is P, and P < T, determine that the scoring result is valid; B53 Set the scoring result of user i as Ri and the screening intensity as T; B54 If user i's rating result satisfies When the time comes, the scoring result is deemed valid; The monitoring indicator scoring module scores the results of the monitored instant messaging indicators through the following process: C1 Record the user experience instant messaging application video in the original situation; C2 Use tools to change the indicators of the user experience instant messaging application video in the original situation multiple times; C3 Based on the execution result of step C2, record the user instant messaging playback experience videos under different indicators and different loss degrees; C4 Score the user instant messaging playback experience videos under different indicators and different loss degrees respectively; C5 Perform data cleaning operations on the scoring results in step C4; C6 Based on the execution result of step C5, determine the mapping relationship between the indicator data and the scoring results.

4. The system according to claim 1, characterized in that, The service classification evaluation visualization subsystem includes: an evaluation selection module, a registration and login module, a personal center module, a log audit module, and a user management module; The evaluation selection module is used to: provide input parameter setting options for different network service quality evaluations, send the selected input parameters to the network service comprehensive indicator monitoring subsystem; visually output the evaluation results of the network service quality fine evaluation subsystem; The registration and login module is used to provide users with system login, registration, and logout functions; The personal center module is used to view all historical test results of the user account; The log audit module is used to record all error exceptions during the operation of the system; The user management module is used to: add registered users, delete registered users, modify registered users, and manage the permissions of registered users.

5. The system according to claim 2, characterized in that, The network service comprehensive indicator monitoring subsystem includes a web page indicator monitoring module, a streaming media indicator monitoring module, an instant messaging indicator monitoring module, and a monitoring type selection module; The web page indicator monitoring module is used to: define and evaluate the indicators of the web page type template by constructing a two-level indicator system for web pages; establish a model for monitoring the relevant values of the web page type based on the defined and evaluated web page indicators; design a web crawler program, and crawl web page data through the web crawler program; The streaming media indicator monitoring module is used to: define and evaluate the indicators of the streaming media type by constructing a two-level indicator system for streaming media; establish a model for monitoring the relevant values of the streaming media type based on the defined and evaluated streaming media indicators; The instant messaging indicator monitoring module is used to: define and evaluate the indicators of the instant messaging type by constructing a two-level indicator system for instant messaging; establish a model for monitoring the relevant values of the instant messaging type based on the defined and evaluated instant messaging indicators; The monitoring type selection module is used to: receive monitoring instructions sent by the master control node, call the corresponding indicator monitoring function, and transmit the indicator detection results to the service quality fine evaluation subsystem.

6. The system according to claim 5, characterized in that, The hierarchical analysis module establishes a hierarchical structure model to classify URL indicator data into different levels and determines the weight of each level of indicator. The process includes: Determine the weights of the primary indicators; Determine the weight of secondary indicators relative to primary indicators; The weights of all indicators are obtained by multiplying the weight of the primary indicator by the weight of the secondary indicator relative to the primary indicator. The hierarchical analysis module, based on the two-level indicator system, performs pairwise comparisons of the indicator data levels, constructs a comparison matrix, and calculates the weight vector of the comparison matrix. The process of verifying the consistency of the weight vector includes: A first comparison matrix is ​​constructed by comparing each indicator in the first-level indicators pairwise. By comparing each indicator in the secondary indicators pairwise, a second comparison matrix, a third comparison matrix, and a fourth comparison matrix are constructed respectively. Calculate the maximum eigenvalue λ and the corresponding eigenvector W of the first, second, third, and fourth contrast matrices; Through The consistency index (CI) of the first, second, third, and fourth comparison matrices is calculated respectively; where, The largest eigenvalue is n, and the number of indices is n. Through The consistency ratio CR of the first, second, third, and fourth comparison matrices are calculated respectively; where, It is a random consistency indicator; like If the test passes, the weight vector w is used as the feature vector; if If the test fails, return to the sub-step of constructing the comparison matrix; The weights of each level of indicators are combined to obtain the indicator weights of all indicator data without hierarchical classification.

7. The system according to claim 6, characterized in that, The two-level indicator system of the webpage includes the following: the first-level indicators include response time, transmission rate, and service robustness; the second-level indicators include: connection establishment time, DNS resolution time, transmission time, and first packet time in the response time attributes; download speed, maximum content rendering rate (LCP), and first content rendering rate (FCP) in the transmission rate attributes; and packet loss rate, latency jitter, and HTTP status codes in the service robustness attributes. In the two-level indicator system of the streaming media, the first-level indicators include response time attributes, transmission rate attributes, and viewing attributes; the second-level indicators include: connection establishment time, DNS resolution time, total transmission time, and first packet time in the response time attributes; download speed, bit rate, and frame rate in the transmission rate attributes; and transmission packet loss rate, latency jitter, and frame rate in the viewing attributes. The two-level indicator system for instant messaging includes first-level indicators such as response time, interaction quality, and service robustness; and second-level indicators such as login time and dial-up connection time in response time, file transfer speed, message sending delay, image sending rate, and video sending rate in interaction quality, and disconnection rate and error codes in service robustness.

8. A method for evaluating the quality of service of multiple types of network applications, characterized in that, include: S1 Select the type of network application service to be evaluated, select the node to perform the monitoring and evaluation task, enter the URL of the network application service to be tested, select a custom monitoring module or a fixed monitoring template, and call the indicator API to collect initial information. S2 defines indicators for different types of network application evaluation templates based on the collected initial information; receives monitoring and evaluation instructions sent by the central control node, and monitors the indicators of the different types of network application evaluation templates by establishing corresponding monitoring models; S3 determines the scoring criteria for individual indicators; uses the analytic hierarchy process (AHP) to determine the weights between different indicators, develops scoring templates for different network applications, and obtains the overall score for the quality of network application services. S4 provides a visual output of the overall score for the quality of network application services.

Images