Multi-network export performance quality comparison evaluation method, device, equipment and medium

Abstract

The invention provides a multi-network export performance quality comparison evaluation method, which comprises the following steps of: determining a test object according to a network application category; selecting a source address of the test server, and configuring a strategy route for the source address, so that the test flow from the source address can pass through a test network outlet matched with the source address in the plurality of network outlets; based on an analog simulation method, respectively switching different test network exits to access network application resources, and testing network application performance; according to a routing tracking result, detecting BGP routing information passed by the test flow in the network, and determining network preference topology distribution of each network outlet; formulating an index system for evaluating the performance quality of each network application according to a test result; and comparing the performance quality of the plurality of test network exits according to the network preference topology distribution and the index system to obtain an evaluation result after comparison. The invention further provides a multi-network export performance quality comparison evaluation device, equipment and a medium.

Description

Technical Field

[0001] This disclosure relates to the field of Internet security technology, and more specifically, to a method, apparatus, equipment, and medium for comparing and evaluating the performance and quality of multiple network exits. Background Technology

[0002] With the rapid development of internet technology, internet users' demands for service quality continue to escalate. Modern applications and services, such as streaming media, online games, remote work, and cloud computing, are placing higher standards on the stability and response speed of network gateways, while the depth and breadth of various applications continue to expand.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following defects in the related technologies: Traditional network egress performance comparison and evaluation methods usually focus on measuring basic technical indicators such as network reachability, latency, jitter, and packet loss between the egress operator's peer line or its remote POP point. Although these indicators can provide preliminary information on network quality, they often cannot fully reflect the user's actual experience in a complex and ever-changing network environment through the egress.

[0004] Enriching network sensing capabilities helps to understand and effectively safeguard network operation quality and user experience in a timely manner, better adapt to future network technology and business development trends, and promote the high-quality development of the digital economy. Therefore, it is urgent to develop a more comprehensive and accurate method for comparing and evaluating the performance and quality of network exits. Summary of the Invention

[0005] In view of this, this disclosure provides a method, apparatus, device and medium for comparing and evaluating the performance quality of multiple network exits. By integrating network application simulation technology and BGP route awareness technology, it aims to achieve a comprehensive and accurate evaluation of the performance quality of different network exits.

[0006] One aspect of this disclosure provides a method for comparing and evaluating the performance quality of multiple network egress points, comprising: determining the test object based on the network application category; selecting the source address of the test server and configuring policy routing for the source address so that test traffic from the source address can pass through the test network egress points that match the source address among multiple network egress points, wherein different test network egress points can be switched by modifying the source address of the test server; for the test object, using a simulation method, switching different test network egress points to access network application resources, testing network application performance, and performing route tracing; based on the route tracing results, probing the BGP routing information traversed by the test traffic in the network, and determining the network preference topology distribution of each network egress point; formulating an indicator system for evaluating the performance quality of each network application based on the test results; and comparing the performance quality of multiple test network egress points based on the network preference topology distribution and the indicator system to obtain the evaluation results after comparison.

[0007] According to embodiments of this disclosure, selecting the source address of the test server and configuring policy routing for the source address includes: defining the network boundaries of each network exit; filtering out globally unique route prefixes from the route prefixes advertised by each network exit, wherein a globally unique route prefix indicates that it is advertised only at a specific network exit and there are no more detailed route prefixes advertised; selecting the globally unique route prefix as one of the source addresses of the test server; using policy routing to restrict the route of the source address to ultimately point to the route-advertising exit; traversing each network exit and setting a set of IPv4 / IPv6 generic source addresses for the test server, wherein the generic source addresses do not specify an access path to the Internet, but use the default optimal path to access the Internet, and the generic source addresses are advertised at each network exit; and selecting different network ports as test network exits by modifying the source address selected by the test server.

[0008] According to embodiments of this disclosure, accessing network application resources includes: DNS domain name resolution, website access, video playback, and file download.

[0009] According to embodiments of this disclosure, for the test object, based on simulation methods, different test network exits are switched to access network application resources to test network application performance, including: simulating DNS domain name resolution, using the DNS server in the test object to test the domain name resolution performance of different network exits; simulating user internet browsing behavior based on browser simulation, testing the access performance of key website homepages in the test object under different network exits; testing the video resource playback performance of the test object under different network exits based on streaming media testing tools; and testing the download performance of key file resources in the test object under different network exits based on wget file download tools.

[0010] According to embodiments of this disclosure, probing the BGP routing information traversed by test traffic in the network based on route tracing results and determining the network preference topology distribution of each network exit includes: analyzing route tracing results to identify and perceive the BGP router and node information traversed by test traffic in the network; identifying the autonomous system information of each network application resource in the access path, determining the autonomous system subject object, and fusing the access path information of each network application resource according to the partitioning information of each network exit; generating a topology diagram based on the BGP router and node information and the fused access path information; and assigning preference weights to each link and node in the topology diagram based on security risks to obtain the network preference topology distribution of each network exit.

[0011] According to embodiments of this disclosure, the indicator system for evaluating the performance quality of various network applications based on test results includes: generating a DNS resolution performance quality evaluation indicator system based on DNS domain name resolution, wherein the DNS resolution performance quality evaluation indicator system includes, but is not limited to: resolution latency indicator and resolution success rate indicator; generating a website access performance quality evaluation indicator system based on website access, wherein the website access performance quality evaluation indicator system includes, but is not limited to: TCP connection time indicator, first packet time indicator, first screen time indicator, homepage time indicator, and page element download speed indicator; generating a video playback performance quality evaluation indicator system based on video playback, wherein the video playback performance quality evaluation indicator system includes, but is not limited to: video successful playback rate indicator, video download speed indicator, and video stuttering rate indicator; and generating a file download performance quality evaluation indicator system based on file download, wherein the file download performance quality evaluation indicator system includes, but is not limited to: file download success rate indicator and file download speed indicator.

[0012] According to embodiments of this disclosure, the performance quality of multiple test network exits is compared based on network preference topology distribution and indicator system to obtain the evaluation results after comparison. This includes: extracting the indicator system corresponding to each test network exit among multiple network exits, wherein the indicator system includes a comprehensive access performance indicator of the network exit and a classified access performance indicator of the network exit; and comparing and evaluating the indicator system corresponding to each test network exit to obtain the evaluation results after comparison.

[0013] Another aspect of this disclosure provides a multi-network egress performance quality comparison and evaluation device, comprising: a first determination module, used to determine the test object according to the network application category; a configuration module, used to select the source address of the test server and configure policy routing for the source address, so that test traffic from the source address can pass through the test network egress that matches the source address among multiple network egresses, wherein different test network egresses can be switched by modifying the source address of the test server; a testing module, used to switch different test network egresses to access network application resources for the test object based on simulation methods, test the network application performance, and perform route tracing; a second determination module, used to detect the BGP routing information traversed by the test traffic in the network according to the route tracing results, and determine the network preference topology distribution of each network egress; a formulation module, used to formulate an index system for evaluating the performance quality of each network application based on the test results; and a comparison module, used to compare the performance quality of multiple test network egresses according to the network preference topology distribution and the index system, and obtain the evaluation results after comparison.

[0014] Another aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the methods described above.

[0015] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the methods described above.

[0016] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the methods described above.

[0017] Compared with existing technologies, the multi-network egress performance quality comparison and evaluation method, apparatus, equipment, and medium provided in this disclosure have at least the following beneficial effects:

[0018] The method, apparatus, equipment, and media for comparing and evaluating the performance and quality of multiple network exits disclosed herein, by integrating network application simulation technology and BGP route awareness technology, aim to achieve a comprehensive and accurate evaluation of the performance and quality of different network exits.

[0019] The method, apparatus, equipment, and media for comparing and evaluating the performance and quality of multiple network egress points disclosed herein utilize policy routing, simulation testing, BGP route awareness, and big data analysis technologies to focus on access testing of network application resources. This provides network service providers with a reliable tool / method for evaluating network egress performance, optimizing network service quality, and improving user experience. Attached Figure Description

[0020] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 A flowchart illustrating a method for comparing and evaluating the performance quality of multiple network egress points according to an embodiment of the present disclosure is shown schematically.

[0022] Figure 2 This illustration schematically shows a topology diagram of policy routing configured for the source address of a test server according to an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram illustrating the structure of a multi-network egress performance quality comparison and evaluation apparatus according to embodiments of the present disclosure is shown; and

[0024] Figure 4 The diagram illustrates a structural block diagram of an electronic device suitable for implementing a method for comparing and evaluating the performance quality of multiple network exits, according to an embodiment of the present disclosure. Detailed Implementation

[0025] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0029] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0030] With the rapid development of internet technology, internet users' demands for service quality continue to escalate. Modern applications and services, such as streaming media, online games, remote work, and cloud computing, are placing higher standards on the stability and response speed of network gateways, while the depth and breadth of various applications continue to expand.

[0031] In realizing the concept of this disclosure, the inventors discovered at least the following defects in the related technology:

[0032] Traditional methods for comparing and evaluating network egress performance typically focus on measuring basic technical indicators such as network reachability, latency, jitter, and packet loss between the egress operator's peer line or its remote POP (point of access). While these indicators can provide preliminary information about network quality, they often fail to fully reflect the actual user experience in complex and ever-changing network environments through that egress point.

[0033] Enriching network sensing capabilities helps to understand and effectively safeguard network operation quality and user experience in a timely manner, better adapt to future network technology and business development trends, and promote the high-quality development of the digital economy. Therefore, it is urgent to develop a more comprehensive and accurate method for comparing and evaluating the performance and quality of network exits.

[0034] Based on this, embodiments of this disclosure provide a method for comparing and evaluating the performance quality of multiple network egress points. The method includes: determining the test object based on the network application category; selecting the source address of the test server and configuring policy routing for the source address, so that test traffic from the source address can pass through a test network egress point that matches the source address among multiple network egress points, wherein different test network egress points can be switched by modifying the source address of the test server; for the test object, using a simulation method, switching different test network egress points to access network application resources, testing network application performance, and performing route tracing; based on the route tracing results, probing the BGP routing information traversed by the test traffic in the network, and determining the network preference topology distribution of each network egress point; formulating an indicator system for evaluating the performance quality of each network application based on the test results; and comparing the performance quality of multiple test network egress points based on the network preference topology distribution and the indicator system to obtain a comparative evaluation result.

[0035] The multi-network egress performance quality comparison and evaluation method disclosed herein, by integrating network application simulation technology and BGP route awareness technology, aims to achieve a comprehensive and accurate evaluation of the performance quality of different network egress points.

[0036] Figure 1 A flowchart illustrating a method for comparing and evaluating the performance quality of multiple network exits according to an embodiment of the present disclosure is shown.

[0037] like Figure 1 As shown, the multi-network egress performance quality comparison and evaluation method of this embodiment may include, for example, operations S1 to S6.

[0038] In operation S1, the test object is determined according to the network application category.

[0039] In this embodiment, key network applications are first determined based on user needs and network service functional requirements in this domain. Key network resources are statistically analyzed from multiple dimensions. For example, based on the total network egress traffic, the top network ports and top protocols accessed by users are statistically analyzed. User subjective application needs are statistically analyzed through surveys to supplement basic Internet functional applications. Then, the final test application categories are formed, and test objects are determined based on the application categories.

[0040] According to embodiments of this disclosure, key network resources are statistically analyzed from multiple dimensions based on network application type. Key networks and resources are categorized and statistically analyzed according to different fields, importance levels, regions, and frequency of access by users within the same domain. In embodiments of this disclosure, network resources are divided into university resources, international organization resources, research institution resources, cross-border e-commerce resources, and international journal resources, with priority given to network resources for which users of each application report frequent outages.

[0041] In this embodiment, basic test application categories may include, for example, DNS (Domain Name System) domain name resolution, website access, video playback, and file download.

[0042] For domain name resolution applications, statistics are compiled on domain name server resources, top-level domain name server resources, top domain name server resources with the highest user traffic in the domain, and authoritative domain name servers corresponding to domains with high user traffic in the domain, and their importance levels are marked respectively.

[0043] For website access applications, website resources are categorized and their importance is marked according to their level. Typical categorized internet websites published by authoritative institutions are selected, and the categorized website resources of highly recognized authoritative institutions are included in the network egress performance monitoring resources of this domain. Network resources with cooperative relationships with this domain are also statistically analyzed, including databases, CDN (Content Delivery Network) services, cloud resources, academic and research exchange resources, and key business resources. Popular websites within the region are also statistically analyzed, such as the top 100 international websites by organic traffic. Through surveys and other methods, key websites frequently visited by key users of this domain are widely statistically analyzed. Based on NetStream traffic sampling analysis and deep packet inspection technology, the top website domains with high access volume in network egress traffic are identified, and their importance is marked according to access frequency. Additionally, regional websites in different geographical locations can be statistically analyzed to ensure that there are no CDN mirrors in other locations, guaranteeing that the actual network location matches the region.

[0044] For video playback applications, key network video resources are statistically analyzed, including: identifying high-traffic video resources in network outbound traffic based on NetStream traffic sampling analysis and deep packet inspection technology; statistically analyzing typical video resources provided by streaming media service providers such as Tencent Video, Youku Video, and Amazon Prime Video; and further statistically analyzing video resources based on streaming media service provider classification, such as video length and whether it is a paid service.

[0045] For file download applications, key network file download resources are statistically analyzed, including: identifying file types with high download volumes in network outbound traffic based on NetStream traffic sampling analysis and deep packet inspection technology. This is supplemented with key network resources for file downloads based on statistics from academic research and industry reports, as well as results from surveys of key users. Statistics are categorized according to the different areas described in the files: by file type (software installation, data backup, resource tutorials, entertainment resource downloads, etc.) and by file format (images, documents, multimedia, compressed files, software applications, development resources, etc.).

[0046] The system can perform multi-dimensional statistics on key network resources. It can also classify and statistically analyze key networks and resources according to different fields, regions, importance levels, and access frequency of users in the local domain. The local domain refers to the network environment of the tester, including but not limited to network networks of various levels of operators, large government and enterprise networks with multiple exits, and any other network with multiple exits.

[0047] In this embodiment, network resources can be categorized into university resources, international organization resources, research institution resources, cross-border e-commerce resources, and international journal resources. Priority is given to network resources with the most reported faults from each application user, and the objects to be tested are determined from these network resources.

[0048] In operation S2, the source address of the test server is selected, and policy routing is configured for the source address so that test traffic from the source address can pass through the test network exits that match the source address from multiple network exits. Specifically, different test network exits can be switched by modifying the source address of the test server. See details... Figure 2 .

[0049] Figure 2 The diagram schematically illustrates a topology diagram of policy routing configured for the source address of a test server according to an embodiment of the present disclosure.

[0050] like Figure 2 As shown in the topology diagram of the embodiments of this disclosure, selecting the source address of the test server and configuring policy routing for the source address may specifically include operations S21~S24:

[0051] In operation S21, the network boundaries of each network exit are defined, and a globally unique route prefix is ​​selected from the route prefixes advertised by each network exit. The globally unique route prefix indicates that it is advertised only at a specific network exit and there are no more detailed route prefixes advertised.

[0052] In operation S22, a globally unique route prefix is ​​selected as one of the source addresses of the test server, and policy routing is used to restrict the route of the source address to ultimately point to the route advertisement exit.

[0053] In operation S23, traverse each network exit and set a set of IPv4 / IPv6 generic source addresses for the test server. The generic source addresses do not specify the path to access the Internet, but use the default optimal path to access the Internet, and the generic source addresses are announced at each network exit.

[0054] In operation S24, different network ports are selected as test network exits by modifying the source address selected by the test server.

[0055] In this embodiment, the network boundary is first defined. Among the routing prefixes announced by each network exit, globally unique routing prefixes are selected. The globally unique route is announced only to the upstream test operator at a certain network exit and no more detailed routing prefixes are announced to other exit operators. This restricts the traffic from the source address to access the global Internet to only return to the local network from the test operator.

[0056] Then, iterate through the network exits and configure the same number of IPv4 / IPv6 addresses for the test server. The test server switches the network exits by modifying the source address for accessing the Internet.

[0057] The test server selects a set of IPv4 / IPv6 generic source addresses, which do not specify an access path to the Internet. They use the default optimal path to access the Internet, and this address range is advertised at each network exit, i.e., the test default exit.

[0058] In S3, for the test object, based on the simulation method, different test network exits are switched to access network application resources to test network application performance and perform route tracing.

[0059] In this embodiment, accessing network application resources includes: DNS domain name resolution, website access, video playback, and file download.

[0060] According to embodiments of this disclosure, for the test object, based on a simulation method, different test network exits are switched to access network application resources, and the test network application performance can specifically include operations S31~S34:

[0061] In operation S31, simulate DNS domain name resolution and use the DNS server in the test object to test the domain name resolution performance of different network exits.

[0062] In this embodiment, the DNS resolution tool dig is used to simulate DNS domain name resolution. Based on the DNS server resources in the key network resources, the @server option of dig is used to specify the DNS server to be tested; the +stats option is set to obtain detailed statistical information, including performance metrics such as the total time spent on queries, and related response information is recorded; measurements are periodically performed on each network exit point, controlling the measurement frequency to avoid excessive measurement that could negatively impact DNS server performance.

[0063] By operating S32, and simulating user internet browsing behavior based on browser simulation, the performance of accessing the homepage of key websites in the test object under different network exits is tested.

[0064] In this embodiment, browser simulation frameworks such as Selenium and Puppeteer are used to simulate a user accessing a website using a browser. An automated script is written to access test website resources and load the target homepage resources based on the browser simulation framework. The simulation framework uses the Chrome DevTools Protocol (CDP) to directly access the browser's internal tools and functions, obtaining fine-grained events during the webpage loading process, such as resource loading, network request and response details, and DOM (Document Object Model) changes. JavaScript statements are executed within the simulation framework to directly obtain page loading-related performance metrics data. For example, "return performance.timing" is used to obtain timestamps for each stage of page loading, such as responseEnd (the time the browser receives the last byte) and domComplete (the time DOM processing is complete). "window.performance.getEntriesByType('resource')" is used to obtain detailed time information for all resources loaded on the page (such as CSS (Cascading Style Sheets), images, and JavaScript files), which can be used to analyze the loading time of each resource. Log data generated during the process of accessing the target website resources is recorded and stored.

[0065] Using S33, based on streaming media testing tools, we tested the video resource playback performance of the test object under different network exits.

[0066] In this embodiment, video playback performance testing tools such as VideoJS Benchmark are used to test the video playback performance indicators of different outputs. Automated scripts are written to simulate user operations such as initiating video playback, pausing, and fast-forwarding; based on the video resources in the key network resources, video playback performance is tested at different resolutions (720p, 1080p, 4K, etc.), bitrates, speeds, and playback positions; log data generated during the video playback process is recorded and stored.

[0067] When operating S34, the download performance of key files in the test object was tested using the wget file download tool under different network exits.

[0068] In this embodiment, file download tools such as wget and curl are used to simulate user file downloads. An automated script is written to periodically download test file resources using the file download tools, setting relevant options and parameters. These options and parameters include discarding downloaded data using the -o / dev / null option, specifying a format string using the -w option, and outputting detailed performance metrics, including total request completion time, TCP connection time, application protocol (such as HTTPS) handshake completion time, time from start to before file transfer, time from start to first byte transfer, number of bytes downloaded, average download speed, HTTP response code, etc. File download log data generated during the above testing process is categorized and recorded according to the test resource file and network egress point.

[0069] In this embodiment, the Looking Glass tool provided by major domestic and international operators is used to view the test source address route and perform route tracing on the test source address to verify whether its return routes all enter the test domain network from the test operator. If the condition is not met, the test source address is changed or the relevant routing configuration is checked to see if it meets the requirements.

[0070] In operation S4, based on the route tracing results, the BGP route information that the test traffic passes through in the network is detected, and the network preference topology distribution of each network exit is determined.

[0071] According to embodiments of this disclosure, based on route tracing results, probing the BGP routing information traversed by test traffic in the network and determining the network preference topology distribution of each network exit may specifically include operations S41-S44:

[0072] When operating S41, analyze the route tracing results to identify and perceive the BGP routers and nodes that the test traffic passes through in the network.

[0073] In operation S42, the autonomous system information of each network application resource in the access path is identified, the main object of the autonomous system is determined, and the access path information of each network application resource is merged according to the division information of each network exit.

[0074] When operating S43, a topology diagram is generated based on BGP router and node information, as well as the merged access path information.

[0075] In operation S44, based on security risks, each link and node in the topology diagram is assigned a preference weight to obtain the network preference topology distribution of each network exit.

[0076] In this embodiment, a route tracing tool is used to probe the path information from the test host to the test egress point, extract the routers at each node in the path, and configure policy-based routing (PBR) hop-by-hop on each node router using the selected source address. The next-hop route from the test source address to the test egress point is specified hop-by-hop until the test network egress point is reached. On the test server, the route tracing tool is used to obtain the path information of the test source address accessing the Internet address, and to verify whether it passes through the specified test network egress point. If the traffic from the test source address to the Internet does not pass through the test network egress point, the test source address is changed or the relevant routing policies in the path are checked to see if they are correct and effective.

[0077] In this embodiment, the topology of target resources accessed by each network egress is jointly analyzed. Visual analysis is performed based on test data, and preferential weights are assigned to each node / line in the access path based on fundamental principles such as security risks, resulting in a network preference topology distribution map. Specifically, autonomous system (AS) information in each resource access path is identified, and the main objects of each AS are determined. Based on network egress partitioning information, the information of each resource access path is merged to form a topology map of the test egress accessing the Internet. According to security risk principles and legal requirements, network preference scores are assigned to each link and node in the topology map of the test network egress accessing the Internet. The network preference topology distribution of each egress is formed by combining the topology map of the test egress accessing the Internet and the network preference indicators, and is used for weighting the network performance quality assessment of each resource egress.

[0078] When operating S5, an indicator system is developed based on the test results to evaluate the performance quality of various network applications.

[0079] According to embodiments of this disclosure, developing an indicator system for evaluating the performance quality of various network applications based on test results may specifically include operations S51-S54:

[0080] When operating S51, based on DNS domain name resolution, a DNS resolution performance quality evaluation index system is generated. The DNS resolution performance quality evaluation index system includes, but is not limited to, resolution latency index and resolution success rate index.

[0081] When operating S52, a website access performance quality evaluation index system is generated based on website access. This index system includes, but is not limited to: TCP connection time index, first packet time index, first screen time index, homepage time index, and page element download speed index.

[0082] When operating S53, a video playback performance quality evaluation index system is generated based on video playback. The video playback performance quality evaluation index system includes, but is not limited to: video successful playback rate index, video download speed index, and video stuttering rate index.

[0083] When operating S54, based on file download, a file download performance quality evaluation index system is generated. The file download performance quality evaluation index system includes, but is not limited to, the file download success rate index and the file download speed index.

[0084] In this embodiment, a performance quality evaluation index system is specified according to the network application type. For DNS domain name resolution, this includes resolution latency and success rate. For website access, it includes TCP connection time, first packet time, first screen time, homepage time, and page element download speed. For video playback, it includes video playback success rate, video download speed, and video stuttering rate. For file download, it includes file download success rate and file download speed. The overall performance of each resource access is calculated based on the aforementioned performance quality evaluation index system.

[0085] In this embodiment, a network performance quality evaluation index system is determined, including a DNS resolution performance quality evaluation index system, a website access performance quality evaluation index system, a video playback performance quality evaluation index system, and a file download performance quality evaluation index system.

[0086] Regarding the DNS resolution performance quality evaluation index system: Based on the log data output by the testing tool during the access process, performance quality-related fields are extracted / generated; based on the DNS domain name resolution process, records with network performance-related fields are selected as DNS resolution performance quality evaluation indicators. In this embodiment, domain name resolution latency and resolution success rate are selected as DNS resolution performance quality evaluation indicators, and weight values ​​and normalization methods are set for each indicator to form a DNS resolution performance quality evaluation index system.

[0087] Regarding the website access performance quality evaluation index system: Based on the browser log data output by the simulation framework during the access process, performance quality-related fields are extracted / generated; according to the client's website access process, records with network performance-related fields are selected as the basis for website access performance quality evaluation. In this embodiment, TCP connection time and first packet time are selected, and the time when 90% of the website homepage resources are loaded is taken as the first screen time, the time when all resources on the website homepage are loaded is taken as the homepage time, and the average download rate of larger page elements is taken as the download rate of website homepage resources. These are collectively used as website access performance quality evaluation indicators, and weight values ​​and normalization methods are set for each indicator to form the website access performance quality evaluation index system.

[0088] Regarding the video playback performance quality evaluation index system: Based on the browser log data output by the streaming media playback tool during the access process, performance quality-related fields are extracted / generated; according to the network video playback process, records with network performance-related fields are selected as the basis for website access performance quality evaluation. In this embodiment, the time from when the user triggers playback to the display of the first frame of the video is selected as the startup time, the number of buffering times, the average duration of each buffering as the buffer duration, the number of stutters and the duration of stutters during playback as playback smoothness, and the average data rate of video transmission as the video playback performance quality evaluation index system. Weight values ​​and normalization methods are assigned to each index to form the video playback performance quality evaluation index system.

[0089] Regarding the file download performance quality evaluation index system: Based on the log data output by the file download tool during the access process, performance quality-related fields are extracted / generated; based on the file download process, records with network performance-related fields are selected as file download performance quality evaluation indicators. In this embodiment, file download success rate and file download speed are selected as file download performance quality evaluation indicators, and weight values ​​and normalization methods are set for each indicator to form a file download performance quality evaluation index system.

[0090] When operating S6, the performance and quality of multiple test network exits are compared based on network preference topology distribution and indicator system to obtain the evaluation results after comparison.

[0091] According to embodiments of this disclosure, the performance quality of multiple test network exits is compared based on network preference topology distribution and indicator system to obtain the evaluation results after comparison, which may specifically include operations S61-S62:

[0092] In operation S61, the indicator system corresponding to each test network exit among multiple network exits is extracted. The indicator system includes the comprehensive access performance indicator of the network exit and the classified access performance indicator of the network exit.

[0093] In this embodiment, based on the network preference topology distribution, the access indicators of network application resources under each path are weighted to obtain a comprehensive network egress access performance indicator. Network application resources are then classified according to different domains to obtain classified network egress access performance indicators.

[0094] When operating S62, the indicator system corresponding to each test network exit is compared and evaluated to obtain the evaluation results after comparison.

[0095] In this embodiment, the access performance index data under the default network exit is used as the baseline level for access to each exit.

[0096] Based on the network application performance quality evaluation index system, the values ​​of each index are calculated and normalized. The access performance index of test resources is calculated according to the weight of each index, and the access performance of test resources under each path is weighted according to the network preference topology distribution map and the importance level of the test resources. According to the test resource classification, the average access performance index of each category of test resources is calculated to obtain the test resource classification performance index. Based on the resource field, resources can be divided into university resources, international organization resources, research institution resources, cross-border e-commerce resources, and international journal resources, etc. Resources are also classified by continent based on the location of their IP addresses and by different types of user-focused resources based on the source of resource statistics. The average access performance index of all resources under the same application is calculated to obtain the comprehensive network egress access performance index. The comprehensive egress access performance index of different network applications is then weighted and averaged to obtain the comprehensive egress access performance.

[0097] In this embodiment, a distribution map of network resource access performance of each test network egress is drawn by determining and comparing the network resource access performance of each test network egress.

[0098] Specifically, test resources are divided into different network locations based on IP addresses, such as mapping resources to provinces, cities, autonomous regions, and operators; test resources are aggregated based on their network locations, and the access performance indicators of resources in each network location are comprehensively calculated; based on the access performance indicators of resources in each network location, a graphing and analysis tool is used to visualize the performance quality distribution of internet access from each exit point.

[0099] The multi-network egress performance quality comparison and evaluation method disclosed herein utilizes policy routing, simulation testing, BGP route awareness, and big data analysis technologies to focus on access testing of network application resources. It provides network service providers with a reliable tool / method for evaluating network egress performance, optimizing network service quality, and improving user experience.

[0100] Figure 3 A schematic block diagram of a multi-network egress performance quality comparison and evaluation apparatus according to an embodiment of the present disclosure is shown.

[0101] like Figure 3 As shown, the multi-network egress performance quality comparison and evaluation device 300 of this disclosure includes: a first determination module 310, a configuration module 320, a test module 330, a second determination module 340, a formulation module 350, and a comparison module 360.

[0102] The first determination module 310 is used to determine the test object based on the network application category.

[0103] The configuration module 320 is used to select the source address of the test server and configure policy routing for the source address, so that test traffic from the source address can pass through the test network exit that matches the source address among multiple network exits. Different test network exits can be switched by modifying the source address of the test server.

[0104] The test module 330 is used to test the performance of network applications and perform route tracing by switching different test network exits to access network application resources based on simulation methods for the test object.

[0105] The second determining module 340 is used to detect the BGP routing information that the test traffic passes through in the network based on the route tracing results, and to determine the network preference topology distribution of each network exit.

[0106] Module 350 is used to develop an indicator system for evaluating the performance quality of various network applications based on the test results.

[0107] The comparison module 360 ​​is used to compare the performance and quality of multiple test network exits based on network preference topology distribution and indicator system, and obtain the evaluation results after comparison.

[0108] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0109] For example, any and more of the first determining module 310, configuration module 320, test module 330, second determining module 340, formulation module 350, and comparison module 360 ​​can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least some of the functionality of one or more of these modules / units / subunits can be combined with at least some of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the first determining module 310, configuration module 320, test module 330, second determining module 340, selection module 350, and comparison module 360 ​​can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), programmable logic array (PLA), system-on-a-chip, system-on-a-substrate, system-on-package, application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the first determining module 310, configuration module 320, test module 330, second determining module 340, selection module 350, and comparison module 360 ​​can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0110] It should be noted that the multi-network egress performance quality comparison and evaluation device part in the embodiments of this disclosure corresponds to the multi-network egress performance quality comparison and evaluation method part in the embodiments of this disclosure. For a detailed description of the multi-network egress performance quality comparison and evaluation device part, please refer to the multi-network egress performance quality comparison and evaluation method part, which will not be repeated here.

[0111] The multi-network egress performance quality comparison and evaluation device disclosed herein utilizes policy routing, simulation testing, BGP route awareness, and big data analysis technologies to focus on access testing of network application resources. It provides network service providers with a reliable tool / method for evaluating network egress performance, optimizing network service quality, and improving user experience.

[0112] Figure 4 The diagram illustrates a structural block diagram of an electronic device suitable for implementing a method for comparing and evaluating the performance quality of multiple network exits, according to an embodiment of the present disclosure. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0113] like Figure 4 As shown, an electronic device 400 according to an embodiment of the present disclosure includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0114] RAM 403 stores various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 402 and / or RAM 403. It should be noted that the programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0115] According to embodiments of this disclosure, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to a bus 404. The electronic device 400 may also include one or more of the following components connected to the input / output (I / O) interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0116] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0117] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0118] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0119] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 402 and / or RAM 403 described above and / or one or more memories other than ROM 402 and RAM 403.

[0120] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this disclosure.

[0121] When the computer program is executed by the processor 401, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0122] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via communication section 409, and / or installed from removable medium 411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0123] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0125] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for comparing and evaluating the performance and quality of multiple network exits, characterized in that, The method includes: Determine the test targets based on the type of network application; Select the source address of the test server and configure policy routing for the source address so that test traffic from the source address can pass through the test network exit that matches the source address among multiple network exits. Different test network exits can be switched by modifying the source address of the test server. For the test object, based on the simulation method, different test network exits were switched to access network application resources to test network application performance and perform route tracing. Based on the route tracing results, the BGP routing information of the test traffic in the network is detected, and the network preference topology distribution of each network exit is determined; Based on the test results, develop an indicator system to evaluate the performance quality of various network applications; Based on the network preference topology distribution and the indicator system, the performance quality of multiple test network exits is compared to obtain the evaluation results.

2. The method according to claim 1, characterized in that, The selection of the source address of the test server and the configuration of policy routing for the source address include: Define the network boundaries of each network exit, and select globally unique routing prefixes from the routing prefixes advertised by each network exit. The globally unique routing prefixes are characterized by being advertised only at a specific network exit and without any further detailed routing prefixes. Select one of the source addresses of the test server as the globally unique route prefix, and use policy routing to restrict the route of the source address to ultimately point to the route advertisement exit; Traverse each network exit and set a set of IPv4 / IPv6 generic source addresses for the test server. The generic source addresses do not specify an Internet access path, but use the default optimal path to access the Internet, and the generic source addresses are announced at each network exit. Different network ports can be selected as test network exits by modifying the source address selected by the test server.

3. The method according to claim 1, characterized in that, The access to network application resources includes: DNS domain name resolution, website access, video playback, and file download.

4. The method according to claim 3, characterized in that, For the test object, based on simulation methods, different test network exits are switched to access network application resources to test network application performance, including: Simulate the DNS domain name resolution and use the DNS server in the test object to test the domain name resolution performance at different network exits; Based on browser simulation, the performance of accessing the homepage of key websites in the test objects was tested under different network exits to simulate user internet browsing behavior. Based on streaming media testing tools, the video resource playback performance of the test objects under different network exits was tested; Based on the wget file download tool, the download performance of key file resources of the test objects under different network exits was tested.

5. The method according to claim 1, characterized in that, The step of probing the BGP routing information of the test traffic in the network based on the route tracing results and determining the network preference topology distribution of each network exit includes: Analyze route tracing results to identify and perceive the BGP routers and nodes that the test traffic passes through in the network; Identify the autonomous system information of each network application resource in the access path, determine the main object of the autonomous system, and merge the access path information of each network application resource according to the division information of each network exit. A topology diagram is generated based on the BGP router, the node information, and the merged access path information. Based on security risks, each link and node in the topology diagram is assigned a preference weight to obtain the network preference topology distribution of each network exit.

6. The method according to claim 5, characterized in that, The aforementioned indicator system for evaluating the performance quality of various network applications based on test results includes: Based on DNS domain name resolution, a DNS resolution performance quality evaluation index system is generated, wherein the DNS resolution performance quality evaluation index system includes, but is not limited to: resolution latency index and resolution success rate index; Based on website access, a website access performance quality evaluation index system is generated, wherein the website access performance quality evaluation index system includes, but is not limited to: TCP connection time index, first packet time index, first screen time index, homepage time index, and page element download speed index; Based on video playback, a video playback performance quality evaluation index system is generated, wherein the video playback performance quality evaluation index system includes, but is not limited to: video successful playback rate index, video download speed index, and video stuttering rate index. Based on file downloads, a file download performance quality evaluation index system is generated, wherein the file download performance quality evaluation index system includes, but is not limited to: file download success rate index and file download speed index.

7. The method according to claim 6, characterized in that, The comparison of the performance quality of multiple test network exits based on the network preference topology distribution and the indicator system yields the following evaluation results: An indicator system is extracted for each test network exit from multiple network exits. The indicator system includes a comprehensive access performance indicator for the network exit and a classified access performance indicator for the network exit. The indicator systems corresponding to each test network exit are compared and evaluated to obtain the evaluation results.

8. A multi-network egress performance quality comparison and evaluation device, characterized in that, The device includes: The first determination module is used to determine the test objects based on the network application category; The configuration module is used to select the source address of the test server and configure policy routing for the source address, so that test traffic from the source address can pass through the test network exit that matches the source address among multiple network exits. Different test network exits can be switched by modifying the source address of the test server. The testing module is used to test the network application performance and perform route tracing by switching different test network exits to access network application resources based on simulation methods for the test object. The second determining module is used to detect the BGP routing information that the test traffic passes through in the network based on the route tracing results, and to determine the network preference topology distribution of each network exit. A module is developed to create an indicator system for evaluating the performance quality of various network applications based on test results. The comparison module is used to compare the performance and quality of multiple test network exits based on the network preference topology distribution and the indicator system, and obtain the evaluation results after comparison.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, characterized in that, When executed by the processor, this instruction causes the processor to perform the method according to any one of claims 1 to 7.

Images