Intelligent blocking system and device for cross-provincial illegal pull flow IP

Abstract

The invention discloses an intelligent blocking system and device for a trans-provincial illegal pull flow IP, and relates to the technical field of network communication, and the system comprises a data collection and preprocessing module, a preliminary screening module, a filtering module, a scene fine screening module, a clustering enhancement module, a blocking execution module and an interpretability report generation module. Through multi-source network flow data fusion and rule and clustering collaborative analysis, accurate identification and hierarchical blocking of the cross-provincial illegal pull flow IP are realized, the method has a rule self-evolution capability while the misjudgment rate is reduced, and the intellectualization and refinement levels of cross-provincial flow management are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of network communication technology, and in particular to an intelligent blocking system and device for illegally intercepting cross-provincial IP traffic. Background Technology

[0002] With the explosive growth of OTT video, cloud storage, mirror sites, PCDN, and IoT services, content resources exhibit a typical characteristic of "centralized storage and cross-province access" in terms of geographical distribution. Inter-provincial bandwidth settlement policies stipulate that traffic between the province where content resources are stored and the province where the end user is located must be charged based on actual transmission traffic. In the existing network environment, there are technical behaviors such as adjusting caching strategies, constructing abnormal requests, changing port parameters, and high-concurrency, low-bandwidth access that cause an abnormal phenomenon where cross-provincial traffic shows a significant bias towards a single province. Technically, this abnormal traffic manifests as a concentration of content resource access traffic originally distributed across different regions to a specific area, thus causing deviations in cross-provincial traffic statistics. This not only affects the normal network traffic distribution characteristics and increases the transmission pressure on the backbone network, but also adversely impacts the fairness and accuracy of inter-provincial bandwidth settlement.

[0003] Typical streaming behaviors include IDC streaming to IDC, IDC streaming to metropolitan area network (MAN), MAN streaming to IDC, and MAN streaming to MAN. Each of these modes is further subdivided into various specific scenarios, such as CDN origin pull masquerading as streaming, PCDN resource push, BT protocol streaming, batch download of mirror services, and video push masquerading. These behaviors exhibit certain patterns in terms of traffic and other characteristics.

[0004] Existing stream-pull identification technologies have the following drawbacks: First, they rely on a single rule for judgment, making it difficult to cover diverse stream-pull scenarios and resulting in low identification accuracy; second, they do not incorporate machine learning algorithms to uncover potential stream-pull features, leading to a lag in response to new stream-pull methods; third, the scenario classification is vague, making it impossible to clearly define the specific methods of stream-pull, resulting in a lack of targeted blocking measures; fourth, the blocking process has a low degree of automation and a slow response speed; and fifth, they have poor interpretability, lacking in-depth analysis of the causes and scenarios of stream-pull behavior, and are unable to provide effective evidence.

[0005] Therefore, there is an urgent need for a system and device that integrates business experience with intelligent algorithms to accurately identify and efficiently block cross-provincial illegal IP traffic. Summary of the Invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] Therefore, this invention aims to solve the problems of low accuracy in traffic identification, ambiguous scene classification, and delayed blocking response in existing technologies, and to achieve high-precision identification, scene-based classification, interpretable analysis, and real-time dynamic blocking of cross-provincial illegal traffic-pulling IPs, thereby improving operators' ability to manage abnormal traffic.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0009] In a first aspect, embodiments of the present invention provide an intelligent blocking system for cross-provincial illegal IP traffic, comprising: a data acquisition and preprocessing module, used to acquire raw data, the raw data including NetFlow traffic logs, DNS logs and port scan logs, and extracting IP five-tuple information, traffic rate, AS number, geographic location tag and domain name resolution results based on the raw data, while cleaning, deduplicating, aggregating and deriving the raw data to form a structured traffic feature dataset;

[0010] The initial screening module is used to perform preliminary screening of IPs based on preset business rules and thresholds. The initial screening module includes an IDC streaming initial screening engine, a MAN streaming initial screening engine, and a special rule engine for executing the corresponding business rules and thresholds to generate a set of potential streaming IPs.

[0011] The filtering module is used to process the potential traffic-pulling IP set, and generate a traffic-pulling candidate IP set by eliminating falsely judged IPs, integrating blacklist and whitelist information and filtering small traffic data;

[0012] The scenario screening module is used to classify the candidate IP set for streaming based on the source and destination types, ports, domain names, and traffic distribution characteristics.

[0013] The clustering enhancement module is used to perform unsupervised clustering analysis on unknown or low-frequency pull-stream patterns not covered by the initial screening rules, including DBSCAN coarse clustering and KMeans fine clustering, and updates the mined feature parameters and thresholds to the initial screening rule library for the initial screening module to load and execute.

[0014] The blocking execution module is used to implement tiered blocking strategies on the set of IPs that are identified as illegally pulling traffic by the clustering enhancement module. These strategies include black hole routing, bandwidth limiting, and alarms. The module also supports customizing the blocking duration and recovery mechanism according to the scenario.

[0015] The interpretability report generation module is used to call the structured traffic feature dataset of the illegal traffic-pulling IP set within a preset time window after the blocking is executed, perform time series statistics on traffic changes, compare the statistical results with normal business benchmark indicators, and generate a traffic-pulling evidence analysis report based on the report template.

[0016] As a preferred embodiment of the intelligent blocking system for cross-provincial illegal IP traffic as described in this invention, the special rule engine in the initial screening module is configured to: jointly determine IPs whose feature indicators are within the neighborhood of a preset threshold, based on the identification results of the IPs over at least N consecutive statistical periods, where N is a positive integer greater than or equal to 2; and perform correlation analysis on the screening results within the same network segment that are at least a preset threshold number, and when the proportion of associated IPs reaching the preset threshold within the same network segment meets a preset proportion condition, supplementarily identify IPs that do not reach the threshold but are associated with them.

[0017] As a preferred embodiment of the intelligent blocking system for cross-provincial illegal IP traffic as described in this invention, the filtering module includes:

[0018] Whitelist filtering is used to remove IPs that have been confirmed as normal business transactions; normal business misjudgment filtering is used to exclude IPs for normal video streaming and cloud storage upload services; low traffic filtering is used to remove IPs corresponding to provinces with traffic volumes below the preset traffic threshold, IPs within network segments with traffic volumes below the preset traffic threshold, and IP interaction records with traffic volumes below the preset traffic threshold.

[0019] As a preferred embodiment of the intelligent blocking system for cross-provincial illegal IP traffic as described in this invention, the scenario screening module divides the potential IP traffic-pulling set into four scenarios based on the combination of service types of the source and destination ends: IDC pulling IDC, IDC pulling metropolitan area network, metropolitan area network pulling IDC, and metropolitan area network pulling metropolitan area network.

[0020] As a preferred embodiment of the intelligent blocking system for cross-provincial illegal IP traffic as described in this invention, the clustering enhancement module is implemented through the following steps:

[0021] Extract multidimensional features of IP, construct a feature matrix and perform standardization; use DBSCAN for coarse clustering to identify candidate streaming clusters; use KMeans for fine clustering of clusters with high feature dispersion in the coarse clustering results, output candidate streaming subclusters and mark potential new streaming scenarios.

[0022] As a preferred embodiment of the intelligent blocking system for cross-provincial illegal IP traffic as described in this invention, the blocking execution module supports tiered blocking based on the traffic traffic scenario, including:

[0023] Block PCDN push-type IPs;

[0024] Implement bandwidth limiting for MAN mirrored IPs;

[0025] Allow CDN streaming IPs to run but mark them for auditing;

[0026] It also supports timed activation, automatic deactivation, and manual review for bypass.

[0027] As a preferred embodiment of the intelligent blocking system for cross-provincial illegal IP traffic as described in this invention, the system includes an interpretability report generation module for generating a traffic traffic evidence analysis report. The traffic traffic evidence analysis report includes IP characteristic mean, scene label, cluster affiliation, and contour coefficient, which is used to support internal auditing by operators, cross-provincial dispute arbitration, and customer complaint handling.

[0028] Secondly, embodiments of the present invention provide an intelligent blocking device for cross-provincial unauthorized IP traffic, comprising:

[0029] processor;

[0030] Memory, used to store program instructions and structured traffic feature datasets;

[0031] The network interface is used to collect NetFlow traffic logs, DNS logs, and port scan logs, and to communicate with carrier firewalls, router ACLs, SDN controllers, or NetConf interfaces.

[0032] The execution module is executed by the processor to execute stored program instructions to implement the function of an intelligent blocking system for cross-provincial illegal IP traffic as described in the first aspect of the present invention;

[0033] The output interface is used to output blocking execution information and analysis reports.

[0034] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, they implement the steps of an intelligent blocking system for cross-provincial illegal IP traffic as described in the first aspect of the present invention.

[0035] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, they implement the steps of an intelligent blocking system for cross-provincial illegal IP traffic as described in the first aspect of the present invention.

[0036] The beneficial effects of this invention are as follows: By integrating multi-source raw network data such as NetFlow traffic logs, DNS resolution logs, and port scan logs, this invention constructs a structured traffic feature dataset. Combined with a collaborative analysis mechanism of rule-based initial screening and unsupervised clustering, it achieves accurate identification of illegal IPs in complex traffic-pull scenarios. It can effectively reduce the false positive rate while ensuring identification accuracy, and dynamically supplements and self-evolves the rule model through clustering results, improving the ability to discover unknown traffic-pull patterns. Simultaneously, based on a scenario-based hierarchical blocking strategy, it achieves precise intervention and controllable handling of illegal traffic-pull behaviors, avoiding impact on normal business operations and significantly improving the intelligence level of cross-provincial traffic governance and network resource utilization efficiency. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of the system structure of the present invention; Figure 2 This is a schematic diagram of the clustering process of the present invention; Figure 3 This is a schematic diagram of the consistency index calculation process of the present invention; Figure 4 This is a schematic diagram of the DBSCAN clustering results of the present invention; Figure 5 This is a schematic diagram of the secondary clustering results of the present invention; Figure 6 This is a statistical diagram showing the distribution of pull flow characteristics under different clusters according to the present invention. Detailed Implementation

[0039] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0040] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0041] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0042] Example 1

[0043] Reference Figures 1-6 This is the first embodiment of the present invention. This embodiment provides an intelligent blocking system for cross-provincial illegal IP traffic. The system consists of a data acquisition and preprocessing module, a preliminary screening module, a filtering module, a scenario fine screening module, a clustering enhancement module, a blocking execution module, and an interpretable report generation module.

[0044] Specifically, the data acquisition and preprocessing module is used to collect raw data, including NetFlow traffic logs, DNS logs, and port scan logs. Based on the raw data, it extracts IP 5-tuple information, traffic rate, AS number, geographic location tag, and domain name resolution results. At the same time, it cleans, deduplicates, aggregates, and derives the raw data to form a structured traffic feature dataset.

[0045] It should be noted that the data acquisition and preprocessing module is implemented through a distributed traffic acquisition and log parsing component, and its implementation process includes the following steps:

[0046] The data acquisition and preprocessing module obtains raw log data from network egress devices, DNS resolution servers, and security scanning devices. NetFlow traffic logs are collected periodically via traffic mirroring interfaces connected to routers or switches; DNS logs are obtained by monitoring query and response records of DNS resolution servers; and port scan logs are obtained via log interfaces connected to network security devices or scanning probes. The collected NetFlow traffic logs, DNS logs, and port scan logs are parsed to extract basic fields such as source IP address, destination IP address, source port, destination port, transport protocol, traffic volume, and timestamp. Fields from different log sources are mapped to a unified data structure format. Based on the parsed basic fields, IP 5-tuple information and traffic rate per unit time are further extracted or calculated. Simultaneously, by querying a pre-set AS number mapping table and geographic location database, the corresponding Autonomous System number and geographic location tag are associated with the IP address. Domain name query records in the DNS logs are associated to obtain the domain name resolution results corresponding to the IP. The parsed data undergoes integrity verification, removing records with missing fields or abnormal formats. A time window and IP 5-tuple combined deduplication method is used to eliminate redundant data caused by duplicate reporting or multi-source collection. Data from the same IP address or IP 5-tuple is aggregated and statistically analyzed according to a preset statistical period to generate derived features such as total traffic, average speed, number of sessions, and port distribution ratio.

[0047] The results of cleaning, deduplication, aggregation, and derivation are written into a unified data storage structure to form a structured traffic feature dataset, which can be called by subsequent modules such as the initial screening module, filtering module, scenario fine screening module, clustering enhancement module, blocking execution module, and interpretable report generation module.

[0048] Specifically, the initial screening module is used to perform preliminary screening of IPs based on preset business rules and thresholds. The initial screening module includes an IDC streaming initial screening engine, a MAN streaming initial screening engine, and a special rule engine for executing the corresponding business rules and thresholds to generate a set of potential streaming IPs.

[0049] Furthermore, the special rule engine in the initial screening module is configured to: jointly determine IPs whose feature indicators are within the neighborhood of a preset threshold, based on the identification results of the IPs over at least N consecutive statistical periods, where N is a positive integer greater than or equal to 2; and perform correlation analysis on the screening results within the same network segment that are at least a preset threshold number. When the proportion of associated IPs that reach the preset threshold within the same network segment meets the preset proportion condition, supplementary identification is performed on IPs that do not reach the threshold but are associated with them.

[0050] It should be noted that a special rule engine was designed as a supplementary mechanism to address issues such as blind spots in threshold judgment caused by data fluctuations, primarily solving the following three problems:

[0051] Data fluctuations cause threshold oscillations: Network traffic data has inherent volatility, and the characteristic indicators of some IPs may oscillate around the initial screening threshold (e.g., the out-of-province inflow / outflow ratio of a certain IP fluctuates between 29 and 31, close to the threshold of 30 for IDC traffic-pushing IPs). The same IP is judged differently at different times, resulting in unstable results and missed judgments. Design [If an IP has been identified as traffic-pushing in the past 3 days, and if it is not identified as traffic-pushing, and the relevant indicators deviate from the threshold within 10%, then the IP will still be regarded as traffic-pushing].

[0052] Differences between network segment and IP characteristics: In some network segments, there may be situations where "most IPs meet the characteristics of traffic pulling, while a few IPs do not reach the threshold due to slight fluctuations." These few IPs may actually be participating in traffic pulling and need to be identified uniformly through special rules. Design [If it is identified that the number of traffic pulling IPs in a certain network segment accounts for more than half of the total number of IPs in the network segment, and the relevant indicators of the remaining non-traffic pulling IPs deviate from the threshold by less than 10%, then these non-traffic pulling IPs are also considered as traffic pulling].

[0053] The covert nature of traffic-pushing behavior: Some illegal traffic-pushing behaviors deliberately control traffic characteristics to keep them at the threshold to avoid detection. Special rule processing can be used to identify such "borderline" behaviors.

[0054] The filtering rules for abnormal network segments and IPs in IDC streaming are as follows:

[0055] IDC traffic dumping typically manifests as multiple IPs within the same network segment simultaneously generating abnormal cross-province traffic. Therefore, the first step is to filter the abnormal network segments. In one specific implementation, the filtering rules include:

[0056] Rules for determining the scale of out-of-province inflows: For IPv4 addresses and IPv6 addresses, corresponding first and second preset thresholds are set respectively. When the out-of-province inflow traffic of the network segment within a preset statistical period is greater than or equal to the first preset threshold, or when the out-of-province inflow traffic of a single IP in the network segment is greater than or equal to the second preset threshold, the network segment is determined to be a candidate abnormal network segment.

[0057] Traffic distribution determination rules during busy and off-peak hours: A third preset threshold is set based on the statistical results of the ratio of out-of-province inflow traffic to that of the network segment during the busy and off-peak hours in historical normal business periods. The ratio of out-of-province inflow traffic to that of the candidate abnormal network segment during the busy and off-peak hours is calculated. When the ratio is less than or equal to the third preset threshold, the traffic time distribution of the network segment is determined to be inconsistent with normal business characteristics. The off-peak hours are set to 1:00 to 5:00 every day, and the busy hours are set to 19:00 to 23:00 every day.

[0058] Exclusion rule for coverage ratio within the province: For network segments marked as normal business, the distribution of the ratio of outbound traffic from the province to inbound traffic from other provinces is statistically analyzed over multiple statistical periods. The upper quantile or the mean-weighted standard deviation of this ratio distribution is selected as the fourth preset threshold. The ratio of outbound traffic from the province to inbound traffic from other provinces for the network segment is calculated. When the ratio is less than the fourth preset threshold, the network segment is determined not to be a normal business network segment with synchronous coverage outside the province and coverage within the province.

[0059] Cross-provincial traffic asymmetry determination rule: Among them, the distribution range of the ratio of out-of-province inflow traffic to out-of-province outflow traffic in normal business network segments is statistically analyzed, and the ratio exceeding the upper limit of the range is set as the fifth preset threshold. The ratio of out-of-province inflow traffic to out-of-province outflow traffic in the network segment is calculated. When the ratio is greater than or equal to the fifth preset threshold, it is determined that the network segment has obvious cross-provincial traffic asymmetry characteristics.

[0060] Inter-network traffic restriction rules: Based on normal business samples across operators, the range of the ratio of inter-network outflow traffic to inflow traffic from other provinces is statistically analyzed, and the ratio below the lower limit of this range is set as the sixth preset threshold. The ratio of inter-network outflow traffic to inflow traffic from other provinces is calculated. When the ratio is less than the sixth preset threshold, the influence of normal business traffic between different operators is excluded.

[0061] Refined IP filtering rules within the network segment: Specifically, the concentration range of abnormal traffic-pulling IPs in the ratio of out-of-province inflow to out-of-province outflow is statistically analyzed, and the minimum value or statistical quantile of this range is used as the seventh preset threshold. For normal business IPs, the ratio of out-of-province inflow to local inflow is statistically analyzed, and the ratio that can distinguish the characteristics of traffic dependent on out-of-province traffic is selected as the eighth preset threshold. After completing the abnormal network segment filtering, IPs within the network segment are further filtered. An IP is determined to be an abnormal traffic-pulling IP when at least one of the following conditions is met: the ratio of out-of-province inflow to out-of-province outflow is greater than or equal to the seventh preset threshold; or the ratio of out-of-province inflow to local inflow is greater than or equal to the eighth preset threshold.

[0062] MAN streaming IP filtering rules: For MAN streaming scenarios, in one specific implementation, the filtering rules include:

[0063] Rules for determining the scale of out-of-province inflows: Based on normal business traffic samples of MAN network segments within a historical statistical period, the distribution range of out-of-province inflows for each network segment is statistically analyzed. Combined with network capacity planning and business load baselines, the scale of out-of-province inflows that distinguishes between normal business and abnormal traffic surges is selected as the ninth preset threshold. Based on historical traffic samples of normal MAN user IPs and abnormal traffic surge IPs, the distribution of out-of-province inflows for a single IP within a preset statistical period is statistically analyzed, and the scale of out-of-province inflows exceeding the normal user traffic level is selected as the tenth preset threshold. When the out-of-province inflow of a network segment is greater than or equal to the ninth preset threshold, or when the out-of-province inflow of an IP is greater than or equal to the tenth preset threshold, it is determined to be a candidate abnormal object.

[0064] Exclusion rules for coverage ratio within the province: Statistical analysis is performed on the ratio of outbound traffic from the province to inbound traffic from other provinces for normal business IPs, and the ratio that effectively excludes this type of business is selected as the eleventh preset threshold; The ratio of outbound traffic from the province to inbound traffic from other provinces for the IP is calculated, and when the ratio is less than the eleventh preset threshold, normal business that is synchronized outside the province but covered within the province is excluded.

[0065] Cross-provincial traffic ratio determination rules: The distribution of out-of-province inflow to out-of-province outflow traffic ratios between abnormal traffic-pulling IPs and normal MAN user IPs is compared and analyzed. The ratio distinguishing between one-way traffic-pulling behavior and two-way or symmetrical communication behavior is set as the twelfth preset threshold. Based on the characteristics of the local traffic ratio of normal business IPs, the normal distribution range of the ratio of out-of-province inflow to local inflow traffic is statistically analyzed, and ratios exceeding this range are selected as the thirteenth preset threshold. When the ratio of out-of-province inflow to out-of-province outflow traffic of an IP is greater than or equal to the twelfth preset threshold, and the ratio of out-of-province inflow to local inflow traffic of an IP is greater than or equal to the thirteenth preset threshold, the IP is determined to be an abnormal traffic-pulling IP.

[0066] Specifically, the filtering module is used to process the potential traffic-pulling IP set, and generates a traffic-pulling candidate IP set by eliminating falsely judged IPs, integrating blacklist and whitelist information and filtering small traffic data.

[0067] Furthermore, the filtering module includes: whitelist filtering, used to remove IPs that have been confirmed as normal business; normal business misjudgment filtering, used to exclude IPs for normal video streaming and cloud storage upload services; and low traffic filtering, used to remove IPs corresponding to provinces with traffic volumes below a preset traffic threshold, IPs within network segments below a preset traffic threshold, and IP interaction records with traffic volumes below a preset traffic threshold.

[0068] It should be noted that normal business IPs include: Normal video streaming: For streaming IPs where both the source and destination are IDCs, if the following situations exist:

[0069] 1. Both the destination and top-ranked source domains are for video services;

[0070] 2. The proportion of out-of-province inflow rate at the destination IP's 1935 port is greater than θ1 (the threshold for the proportion of the average out-of-province inflow rate at the destination IP's 1935 port to the total average out-of-province inflow rate of that IP within a preset statistical period).

[0071] If it is determined to be a normal business where the source actively pushes video resources to the destination, it will be excluded.

[0072] For normal cloud storage services: For pull IPs where the source is a MAN and the destination is an IDC, if the following situations exist:

[0073] 1. The destination IP domain belongs to the cloud storage IDC service;

[0074] 2. The source MAN traffic accounts for more than m1% of the total out-of-province inflow (within the preset statistical period, the proportion of out-of-province inflow traffic from the MAN to the total out-of-province inflow traffic of the destination IP);

[0075] 3. The percentage of MAN users whose interaction time between the source and destination ends is less than 2 hours > θ2 (the threshold for the percentage of MAN users whose single interaction duration is less than 2 hours among all MAN users who interact with the destination IP within the preset statistical period).

[0076] If the user uploads resources to the cloud drive normally, the MAN user will be removed from the list.

[0077] Specifically, the scenario screening module is used to classify the candidate IP set for streaming based on the source and destination types, ports, domain names, and traffic distribution characteristics.

[0078] Furthermore, the scenario screening module divides the potential traffic-driving IP set into four scenarios based on the combination of service types at the source and destination ends: IDC-to-IDC, IDC-to-Metropolitan Area Network, Metropolitan Area Network-to-IDC, and Metropolitan Area Network-to-Metropolitan Area Network.

[0079] It should be noted that in the IDC-to-IDC scenario: the IDC pulling the stream has high bandwidth access capabilities, and the IDC being pulled has large-scale content resource storage capabilities. This makes the cross-provincial streaming traffic concentrated and of a high overall level in this type of scenario. Specific scenarios include CDN-to-CDN, IDC-to-CDN, and other methods of IDC-to-IDC.

[0080] CDN pull scenario: This mainly relies on CDN's origin pull method to pull traffic. Normal CDN origin pull occurs when a user requests an edge node, and the edge node actively requests the central node to pull the data after discovering data loss or cache expiration. CDN streaming, on the other hand, shortens or disables the caching time of an edge node in a certain location, and then triggers the origin pull operation through a request. The content data obtained from the origin pull synchronization is not covered by the end user's access network.

[0081] IDC pull CDN scenario: Data is pulled by leveraging the large bandwidth of the IDC and the ample resources of the CDN. If the IDC actively requests the CDN, the main traffic will return from the nearest edge node. Therefore, this method mainly relies on the CDN resources in other provinces to actively push.

[0082] Typical IDC-to-IDC scenario: Both the origin and destination are IDCs, and the origin is not a CDN; the specific screening process includes:

[0083] 1. From the potential streaming IP set output by the initial screening module, select streaming IPs whose source and target business types are both IDC, forming the first candidate IP set;

[0084] 2. Based on the domain name resolution results, select IPs from all IPs whose domain names contain preset CDN identifier keywords to form a CDN IP set;

[0085] 3. Perform matching analysis between the first candidate IP set and the CDN IP set to determine whether the source IP and the destination IP in the first candidate IP set belong to the CDN;

[0086] 4. When both the source and destination ends are identified as CDNs, and the ratio of inbound traffic from outside the province to outbound traffic from the province is greater than the first preset threshold, it is identified as a CDN pull CDN scenario;

[0087] 5. When the source end is determined to be a CDN, the target end is not determined to be a CDN, and no domain name resolution result is matched on the target end, and the traffic proportion of the top 1 inflow port on the target end is greater than the second preset threshold, it is determined to be a CDN push to IDC scenario;

[0088] 6. When the source is not identified as a CDN, it is identified as a regular IDC pulling IDC scenario.

[0089] IDC (Internet Data Center) data retrieval from metropolitan area networks (MANs): IDC data retrieval from MANs primarily utilizes the high bandwidth of the IDC to retrieve small amounts of traffic from distributed MANs. This mainly includes PCDN (Public Data Center) proactively pushing data to the IDC, IDC retrieving BT (BitTorrent) data, and other scenarios. The specific fine-tuning process is as follows:

[0090] 1. Filter out streaming IPs whose source is a metropolitan area network and whose destination is an IDC;

[0091] 2. Statistically analyze the average daily outflow traffic of the source metropolitan area network (MAN) IP within a preset statistical period. When the average daily outflow traffic exceeds a preset traffic threshold θ3 (which is the threshold for the average daily outflow traffic from other provinces of the source MAN IP within the preset statistical period), the source MAN IP is determined to have high-frequency content output characteristics. Simultaneously, statistically analyze the number of different MAN IPs or different MAN segments that the source MAN IP interacts with within the statistical period. When the number exceeds a preset coverage scale threshold N1, the source MAN IP is determined to have multi-node distribution characteristics. When the source MAN IP simultaneously satisfies both the traffic threshold θ3 and the coverage scale threshold N1, the source MAN IP is determined to be a PCDN node. The coverage scale threshold N1 is a configurable parameter used to characterize the scale of MAN nodes covered by a single MAN IP within a unit of time.

[0092] 3. If the top 10 outgoing IPs of the streaming IP contain commonly used Tracker addresses, it indicates that the streaming is being done through a BitTorrent seed.

[0093] Metropolitan Area Network (MAN) IDC (Internet Data Center) Pull Scenarios: Specific scenarios for MAN pulling IDC resources include MAN proactively pulling video resources, MAN proactively pulling mirror services, CDN proactively pushing, and other scenarios. Specifically: MAN proactively pulling video resources: The source IP domain belongs to the video service, mainly flowing in from port 1935 and then out from ports 80 and 443 to the destination. Combined with the preset technical thresholds used to determine whether the traffic scale or rate is abnormal when identifying potential abnormal traffic pulling behavior from MANs to IDCs, there is a pattern of MAN users proactively and continuously pulling video resources from other provinces. Proactively pulling mirror services: The source IP domain belongs to the mirror service, and the source outflow ports are concentrated, while the destination ports are dispersed. CDN proactively pushing: The source IP belongs to the CDN service and is proactively pushed to the MAN.

[0094] The specific screening process is as follows:

[0095] 1. Identify the domain name corresponding to the top-ranked source IP;

[0096] 2. If the video service characters are included, it will determine whether the traffic share of the Top 1 source port 1935 as the inflow port reaches m2% of the total inflow (the proportion of the traffic flowing into the Top 1 source port 1935 to the total inflow traffic of the source port), and then check whether the outflow ports 80 and 443 reach m2% of the total outflow traffic. If they meet the criteria, it means that the source ports are concentrated and the destination ports are dispersed, and the metropolitan area network is actively pulling video resources.

[0097] 3. If the source IP contains domains such as mirror / mirrors / repo / repos, and the top 3 outgoing ports of the source account for m3% (the proportion of the top 3 outgoing ports of the source to the total outgoing traffic of the source), it indicates that the metropolitan area network is actively pulling mirror services.

[0098] 4. If the source IP is determined to be the domain name corresponding to the CDN, and the source ports are scattered while the destination ports are concentrated, it is determined to be CDN active push;

[0099] 5. If no relevant domain name is identified, classify it as another domain within the metropolitan area network.

[0100] Metropolitan Area Network (MAN) Data Retrieval Scenarios: Specific scenarios for MAN data retrieval include PCDN push to the MAN and MAN data retrieval from the BT platform. PCDN push to the MAN: PCDN resources distributed across the country are actively pushed to the distributed IPs of the MAN. MAN data retrieval from the BT platform: The MAN accesses the tracker address, obtains the information corresponding to the resource, and then directly connects with other nodes to retrieve data.

[0101] The specific screening process is as follows:

[0102] 1. PCDN proactive push to metropolitan area networks: Determine whether the number of metropolitan area networks at the source of the destination IP exceeds 100, which is consistent with the scale of the PCDN super node cluster. If the number of metropolitan area networks served by the TOP1 source IP is greater than 500, which is consistent with the typical characteristics of PCDN's widespread distribution and the characteristics of the PCDN protocol, then it is a PCDN proactive push scenario.

[0103] 2. BT pulling from metropolitan area network: Determine if the top 10 IPs of the pulling IP include the Tracker address IP. If so, it means that the destination IP obtained the information corresponding to the resource by accessing the Tracker's address and pulled the corresponding resource accordingly.

[0104] Specifically, the clustering enhancement module is used to perform unsupervised clustering analysis on unknown or low-frequency pull-up patterns not covered by the initial screening rules, including DBSCAN coarse clustering and KMeans fine clustering, and updates the mined feature parameters and thresholds to the initial screening rule base for the initial screening module to load and execute.

[0105] Furthermore, the clustering enhancement module is implemented through the following steps:

[0106] Extract multidimensional features of IP, construct a feature matrix and perform standardization; use DBSCAN for coarse clustering to identify candidate streaming clusters; use KMeans for fine clustering of clusters with high feature dispersion in the coarse clustering results, output candidate streaming subclusters and mark potential new streaming scenarios.

[0107] It should be noted that the initial screening rules refer to a set of rules consisting of business rules, threshold parameters and their combined logic, which are stored in the initial screening rule library and support dynamic updates.

[0108] It should be noted that a feature matrix is ​​constructed by extracting multidimensional IP features from the data acquisition and preprocessing module, including traffic features, time-series features, and port features. The feature matrix is ​​then standardized, converting each feature value into standardized data with a mean of 0 and a variance of 1, eliminating the influence of differences in feature dimensions on the clustering results. The standardization formula is as follows:

[0109] ;

[0110] in, express The mean, express variance This represents the standardized eigenvalues;

[0111] Furthermore, such as Figure 2 , Figure 4 As shown, DBSCAN coarse clustering is used, and the clustering radius and minimum sample size are set to cluster the standardized feature matrix. IPs with high feature similarity are clustered into the same cluster, and multiple streaming candidate clusters (denoted as C1, C2...Cn) are output. Each cluster contains a different number of streaming IPs.

[0112] like Figure 3 As shown, a scene consistency index is calculated for each candidate cluster's IPs. This index quantifies the dispersion of scene distribution within the cluster to determine whether further fine-grained clustering is needed, including:

[0113] Select some core features with high differentiation in certain scenarios from the characteristics of the pulling IP, such as whether it is a CDN domain or a mirror service domain;

[0114] Calculate the n-dimensional feature mean vector of all IPs within the cluster. ;

[0115] Calculate the feature vector of each IP and European distance :

[0116] ;

[0117] in, This represents the j-th eigenvector of a single IP address. Let represent the mean vector of the j-th dimension feature of all IPs;

[0118] Statistical distance mean and standard deviation ;

[0119] Calculate the dispersion index:

[0120] ;

[0121] Where D represents a dimensionless index, σ represents the standard deviation of the cluster or feature dimension, and μ represents the mean of the cluster or feature dimension. A larger D value indicates more significant differences in features within the cluster. When D ≥ a threshold (e.g., 0.8), it indicates high feature dispersion and the existence of obvious sub-patterns; or when the distance... If the percentage of IPs is greater than or equal to a threshold (e.g., 20%), it indicates that there is a subset of IPs that are significantly deviated from the cluster center.

[0122] Furthermore, such as Figure 5 As shown, KMeans fine clustering is used to cluster the clusters that need fine clustering, and IPs with high feature similarity are clustered into the same cluster, outputting multiple streaming candidate sub-clusters (denoted as C1_0, C1_1...C2_1, etc.), each cluster containing a different number of streaming IPs.

[0123] Furthermore, such as Figure 6 As shown, for all abnormal clusters obtained by clustering, their core features are associated with the known scene feature library. If the feature combination of a sub-cluster does not match any known scene, it is marked as a potential new streaming scene, and its behavioral features that distinguish it from existing streaming scenes are analyzed.

[0124] Furthermore, to verify the effectiveness of the system of the present invention, the following key performance indicators were improved by comparing the system performance before and after the introduction of the clustering enhancement module:

[0125] Table 1. Comparison of key system performance indicators before and after introducing the clustering enhancement module.

[0126]

[0127] Table 1 above shows that by introducing the clustering enhancement module, the system not only achieves significant optimization in terms of recognition accuracy and false judgment rate, but also greatly shortens the response time to new trending behaviors. At the same time, the dynamic updating of the rule base is achieved through the clustering feature write-back mechanism, which improves the system's adaptability.

[0128] Specifically, the blocking execution module is used to implement tiered blocking strategies on the set of IPs that are identified as illegally pulling traffic by the clustering enhancement module. These strategies include black hole routing, bandwidth limiting, and alarms, and support customizing the blocking duration and recovery mechanism according to the scenario.

[0129] Furthermore, the blocking execution module supports tiered blocking based on the pull streaming scenario, including:

[0130] Block PCDN push-type IPs;

[0131] Implement bandwidth limiting for MAN mirrored IPs;

[0132] Allow CDN streaming IPs to run but mark them for auditing;

[0133] It also supports timed activation, automatic deactivation, and manual review for bypass.

[0134] It should be noted that the blocking execution module is used to implement a tiered blocking strategy on the identified set of IPs that are illegally pulling traffic. The implementation method includes the following steps: For IPs that are identified as illegally pulling traffic in the PCDN push category, their routing table entries are modified to point to the black hole route, so that the traffic is directly dropped, thereby achieving the purpose of blocking; the configuration of the black hole route can be automatically applied by the network controller to each access or edge router. For IPs identified as violating MAN (Man-Agent Traffic Flow) restrictions, their peak network bandwidth usage is calculated, and the rate limiting policy is dynamically adjusted to keep traffic within a preset threshold. Rate limiting can be implemented through QoS policies or a traffic scheduler, supporting fine-grained control based on traffic flow scenarios or IP groups. The dynamic adjustment of the rate limiting policy includes the following process: Within a preset time window, the actual bandwidth usage, peak bandwidth, and duration of continuous usage of the violating IP are periodically collected to form real-time traffic monitoring results; the real-time traffic monitoring results are compared with a preset bandwidth threshold, and when the actual bandwidth usage exceeds the preset bandwidth threshold, the rate limiting policy is adjusted; based on the traffic flow scenario type, historical traffic change trend, and rate limiting execution effect of the violating IP, corresponding rate limiting parameters are dynamically generated and sent to the QoS policy module or traffic scheduler; after the rate limiting policy takes effect, the bandwidth usage of the violating IP continues to be monitored, and when its traffic remains below the preset bandwidth threshold for a duration that meets the preset recovery conditions, the rate limiting is automatically relaxed or lifted.

[0135] For IPs identified as violating CDN push streaming rules, their traffic is not blocked, but the traffic records of these IPs are marked on network devices or in the traffic monitoring system. The marking information includes the source IP, destination IP, traffic volume, port, and timestamp, for subsequent operator auditing, cross-provincial dispute arbitration, and customer complaint handling. Preset start and end times for blocking ensure the policy takes effect and is lifted according to plan. The blocking policy is automatically revoked after a preset time window or rule condition is met, restoring normal traffic. Administrators can review blocking policies and implement temporary bypasses for specific IPs or scenarios to ensure business continuity. Various blocking policies can be flexibly configured according to different streaming scenarios; for example, PCDN push streams can be forcibly blocked, MAN pull streams can be rate-limited, and CDN push streams can be marked for auditing. Blocking policies can be dynamically adjusted based on traffic monitoring data to achieve refined and hierarchical management. The dynamic adjustment of the blocking policies is achieved through the following method: during the effective period of the blocking policy, traffic characteristic data of the marked or restricted violating streaming IPs is continuously collected according to a preset sampling period. The traffic characteristic data includes at least bandwidth usage, traffic change trends, duration, and the number of associated IPs.

[0136] The traffic characteristic data is compared with the policy thresholds under the corresponding traffic pulling scenario. When the traffic characteristics are detected to meet the policy upgrade conditions, the blocking policy of the corresponding IP is upgraded from marking and auditing to rate limiting, or from rate limiting to forced blocking. When the traffic characteristics are detected to meet the policy downgrade or cancellation conditions, the blocking level is automatically reduced or the blocking policy is cancelled, and normal forwarding is restored. The policy adjustment results are recorded in the audit log, and the adjustment effect is continuously monitored in subsequent statistical periods to form a closed-loop control for the execution of the blocking policy.

[0137] Specifically, the interpretability report generation module is used to call the structured traffic feature dataset of the illegal traffic-pulling IP set within a preset time window after the blocking is executed, perform time series statistics on traffic changes, compare the statistical results with normal business benchmark indicators, and generate a traffic-pulling evidence analysis report based on the report template.

[0138] Furthermore, the interpretability report generation module is used to generate a traffic pull evidence analysis report, which includes IP feature mean, scene tags, cluster affiliation and profile coefficient, to support internal audits of operators, cross-provincial dispute arbitration and customer complaint handling.

[0139] It should be noted that the interpretability report generation module is used to analyze the set of IPs illegally pulling traffic after the blocking is executed. Its implementation includes the following steps: The module calls the structured traffic feature dataset of the set of IPs illegally pulling traffic within a preset time window; the data includes the traffic rate, port usage, geographical location, AS number, and time series characteristics of each IP, for subsequent analysis. Statistical analysis is performed on the traffic data of each IP in chronological order, including traffic peak, mean, variance, and traffic trend; statistics are supported at different time granularities such as hourly, daily, and weekly, automatically generating a time series data matrix. The statistical results are compared with normal business benchmark indicators, including historical average traffic, normal business port ratio, and traffic distribution in typical scenarios; IPs or time periods with abnormal deviations are marked to identify potential illegal traffic pulling behavior. Based on a preset report template, the analysis results generate a traffic pulling evidence analysis report; the report content includes: IP feature mean, scenario label, cluster affiliation, and profile coefficient; the report can be exported as an electronic document or chart, and can be directly used for internal operator audits, cross-provincial dispute arbitration, and customer complaint handling. The module supports generating reports in different formats, such as PDF, Excel, or visual charts, based on custom templates; it also supports generating sub-reports by IP address, scenario, or time window for more refined analysis.

[0140] This embodiment also provides a computer device applicable to an intelligent blocking system for cross-provincial illegal IP traffic, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement an intelligent blocking system for cross-provincial illegal IP traffic as proposed in the above embodiment.

[0141] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0142] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements an intelligent blocking system for cross-provincial unauthorized IP traffic as proposed in the above embodiment.

[0143] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An intelligent blocking system for cross-provincial unauthorized IP traffic, characterized in that, include: The data acquisition and preprocessing module is used to collect raw data, including NetFlow traffic logs, DNS logs and port scan logs. Based on the raw data, it extracts IP 5-tuple information, traffic rate, AS number, geographic location tag and domain name resolution results. At the same time, it cleans, deduplicates, aggregates and derives the raw data to form a structured traffic feature dataset. The initial screening module is used to perform preliminary screening of IPs based on preset business rules and thresholds. The initial screening module includes an IDC streaming initial screening engine, a MAN streaming initial screening engine, and a special rule engine for executing the corresponding business rules and thresholds to generate a set of potential streaming IPs. The filtering module is used to process the potential traffic-pulling IP set, and generate a traffic-pulling candidate IP set by eliminating falsely judged IPs, integrating blacklist and whitelist information and filtering small traffic data; The scenario screening module is used to classify the candidate IP set for streaming based on the source and destination types, ports, domain names, and traffic distribution characteristics. The clustering enhancement module is used to perform unsupervised clustering analysis on unknown or low-frequency pull-stream patterns not covered by the initial screening rules, including DBSCAN coarse clustering and KMeans fine clustering, and updates the mined feature parameters and thresholds to the initial screening rule library for the initial screening module to load and execute. The blocking execution module is used to implement tiered blocking strategies on the set of IPs that are identified as illegally pulling traffic by the clustering enhancement module. These strategies include black hole routing, bandwidth limiting, and alarms. The module also supports customizing the blocking duration and recovery mechanism according to the scenario. The interpretability report generation module is used to call the structured traffic feature dataset of the illegal traffic-pulling IP set within a preset time window after the blocking is executed, perform time series statistics on traffic changes, compare the statistical results with normal business benchmark indicators, and generate a traffic-pulling evidence analysis report based on the report template.

2. The intelligent blocking system for cross-provincial illegal IP traffic as described in claim 1, characterized in that, The special rule engine in the initial screening module is configured to: jointly determine IPs whose feature indicators are within the neighborhood of a preset threshold, based on the identification results of the IPs over at least N consecutive statistical periods, where N is a positive integer greater than or equal to 2; and perform correlation analysis on the screening results within the same network segment that are at least a preset threshold number. When the proportion of associated IPs that reach the preset threshold within the same network segment meets the preset proportion condition, supplementary identification is performed on IPs that do not reach the threshold but are associated with them.

3. The intelligent blocking system for cross-provincial illegal IP traffic as described in claim 1, characterized in that, The filtering module includes: Whitelist filtering is used to remove IPs that have been confirmed as normal business transactions; normal business misjudgment filtering is used to exclude IPs for normal video streaming and cloud storage upload services; low traffic filtering is used to remove IPs corresponding to provinces with traffic volumes below the preset traffic threshold, IPs within network segments with traffic volumes below the preset traffic threshold, and IP interaction records with traffic volumes below the preset traffic threshold.

4. The intelligent blocking system for cross-provincial illegal IP traffic as described in claim 1, characterized in that, The scenario screening module divides the potential traffic-pulling IP set into four scenarios based on the combination of service types at the source and destination ends: IDC-to-IDC, IDC-to-Metropolitan Area Network, Metropolitan Area Network-to-IDC, and Metropolitan Area Network-to-Metropolitan Area Network.

5. The intelligent blocking system for cross-provincial illegal IP traffic as described in claim 1, characterized in that, The clustering enhancement module is implemented through the following steps: Extract multidimensional features of IP, construct a feature matrix and perform standardization; use DBSCAN for coarse clustering to identify candidate streaming clusters; use KMeans for fine clustering of clusters with high feature dispersion in the coarse clustering results, output candidate streaming subclusters and mark potential new streaming scenarios.

6. The intelligent blocking system for cross-provincial illegal IP traffic as described in claim 1, characterized in that, The blocking execution module supports tiered blocking based on the streaming scenario, including: Block PCDN push-type IPs; Implement bandwidth limiting for MAN mirrored IPs; Allow CDN streaming IPs to run but mark them for auditing; It also supports timed activation, automatic deactivation, and manual review for bypass.

7. The intelligent blocking system for cross-provincial illegal IP traffic as described in claim 1, characterized in that, The interpretability report generation module is used to generate a traffic pull evidence analysis report, which includes IP feature mean, scene label, cluster affiliation and profile coefficient, to support internal auditing of operators, cross-provincial dispute arbitration and customer complaint handling.

8. A smart blocking device for cross-provincial illegal IP traffic, characterized in that, include: processor; Memory, used to store program instructions and structured traffic feature datasets; The network interface is used to collect NetFlow traffic logs, DNS logs, and port scan logs, and to communicate with carrier firewalls, router ACLs, SDN controllers, or NetConf interfaces. The execution module is executed by the processor to execute the stored program instructions to implement the intelligent blocking system function of cross-provincial illegal IP traffic as described in any one of claims 1 to 7; The output interface is used to output blocking execution information and analysis reports.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the intelligent blocking system for cross-provincial illegal IP traffic as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent blocking system for cross-provincial illegal IP traffic as described in any one of claims 1 to 7.

Images