Proxy IP scheduling method and system based on dynamic quality evaluation, and storage medium

Abstract

The invention relates to the technical field of network data acquisition, and discloses a proxy IP scheduling method and system based on dynamic quality assessment and a storage medium, and the method comprises the steps: calculating the real-time health degree of an IP through multi-index fusion; creating a spatial index based on the real-time health degree of the IP and the multi-dimensional features; judging whether an abnormal IP exists or not according to the real-time health degree of the IP, and triggering an isolation strategy when the abnormal IP exists; a task queue containing state data is input, and optimal resources are allocated by combining the real-time health degree dynamic evaluation model and the multi-constraint resource allocation model of the IP; outputting an optimal candidate IP set; through real-time health degree monitoring, multi-constraint resource allocation and anomaly isolation, the resource utilization rate and the scheduling efficiency are improved; the stability and disaster tolerance of an acquisition system are enhanced, and the task completion rate in a high-concurrency scene is improved; the task success rate and privacy protection are optimized, the operation and maintenance cost is reduced, and the flexibility is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of network data collection, and in particular to a proxy IP scheduling method, system and storage medium based on dynamic quality assessment. Background Art

[0002] The traditional proxy IP scheduling method has the following defects:

[0003] Static allocation: Using fixed round-robin or random allocation, it fails to consider real-time conditions such as IP expiration and bandwidth fluctuations, leading to resource waste. Single-constraint optimization: This focuses solely on IP availability, ignoring the coordinated control of multiple constraints such as bandwidth and the number of concurrent tasks. Lag in exception response: IP invalidation is determined solely through timeouts, resulting in high detection latency and inability to isolate exceptions within seconds. Resource conflict and waste: When the same IP is preempted by multiple tasks, traffic control is lacking, leading to service overload. IPs are not dynamically allocated based on task type, and high-quality IPs may be wasted in low-demand scenarios. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a proxy IP scheduling method based on dynamic quality assessment to improve IP utilization and task success rate, solve the problems of inefficiency and resource waste caused by unreasonable proxy IP allocation, and is suitable for large-scale distributed network data collection scenarios.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] The proxy IP scheduling method based on dynamic quality assessment includes:

[0007] Calculate the real-time health of IP through multi-indicator fusion;

[0008] Create spatial indexes based on the real-time health and multi-dimensional features of IP;

[0009] Determine whether there are abnormal IPs based on the real-time health of the IPs, and trigger isolation policies when abnormal IPs are found;

[0010] Input the task queue containing status data, and allocate optimal resources by combining the real-time dynamic health assessment model of IP and the multi-constraint resource allocation model;

[0011] Output the optimal candidate IP set.

[0012] In the present invention, preferably, the calculation of the real-time health of the IP through multi-index fusion includes:

[0013] Create a dynamic IP health assessment model to calculate the health of IP by integrating multiple indicators;

[0014] Adjust the weight coefficient corresponding to each indicator according to task characteristics and system status;

[0015] Recalculate the real-time health of the IP based on the weight coefficient.

[0016] In the present invention, preferably, the multiple indicators include response time, request success rate, bandwidth weight, concurrent task health and validity period decay, and the formula for calculating the real-time health of IP is:

[0017]

[0018] Where w1 is the weight coefficient of response time, w2 is the weight coefficient representing the request success rate (Sqs), w3 is the weight coefficient of bandwidth, w4 is the weight coefficient of concurrent task health, w5 is the weight coefficient of validity period decay, Tcurr is the average response time (ms), Tmax is the response time threshold (default is 5000ms), Sqs is the request success rate within the specified window time (the window time defaults to 5 minutes) = number of successes / total number of requests, Bus is the bandwidth utilization score, C_score is the concurrent task health, Δt is the IP survival time and activation time (seconds), and λ is the validity period decay factor (default is 0.1);

[0019]

[0020] Where B_used is the currently used bandwidth, B_avg is the expected bandwidth allocation value of a single IP in the current task scenario, and B_max is the maximum bandwidth of a single IP.

[0021]

[0022] Through real-time calculations, bandwidth resources are managed in a refined manner, adapting to changes in task load (such as sudden traffic spikes). Overload protection: When B_used approaches B_max, the IP score is lowered, reducing the number of new tasks assigned. Idle recovery: When B_used is significantly lower than B_avg (e.g., ≤0.5Mbps), the IP is marked as inefficient, triggering a resource recovery mechanism.

[0023] C_score=max(0,1-N_current / N_max),

[0024] N_current is the current number of concurrent connections, and N_max is the maximum number of concurrent connections, which defaults to 20.

[0025] In the present invention, preferably, the adjusting the weight coefficient corresponding to each indicator according to the task characteristics and the system status specifically includes:

[0026] When the task feature is high-frequency call, increase w1;

[0027] When performing streaming download tasks, increase w3;

[0028] When the system load is driven, when the overall bandwidth approaches the threshold, increase w3;

[0029] When the proportion of abnormal IPs increases, increase w2 and w5;

[0030] When the task is batch interactive or the system is under load, increase the w4 weight.

[0031] In the present invention, preferably, the creation of a spatial index based on the real-time health of IP and multi-dimensional features specifically includes:

[0032] Prioritize tasks and determine the corresponding weights of different tasks;

[0033] Use R-tree to organize and index resources based on multidimensional space;

[0034] A hierarchical greedy allocation mechanism is set up. High-weight tasks are fully allocated first. When resources are insufficient, resource preemption is triggered. Medium and low-weight tasks are allocated greedily according to the standard allocation method.

[0035] Task requirements are mapped into three-dimensional space points according to their IP health, bandwidth, and concurrency. Starting from the root node, all resources that meet the conditions are quickly retrieved in the corresponding area of ​​the R-tree.

[0036] In the present invention, preferably, it also includes updating the R-tree according to resource status changes, sharding according to resource survival time, and reducing the single tree size of the R-tree.

[0037] In the present invention, preferably, the judgment of whether there is an abnormal IP adopts an abnormality detection rule. When the real-time health of the IP is continuously lower than the threshold for a certain period of time and the request failure rate is higher than the threshold, the IP status is deemed to be abnormal.

[0038] In the present invention, preferably, the isolation strategy includes:

[0039] When the real-time health of the IP is between 0.4 and 0.6, the allocation of new tasks is suspended;

[0040] When the real-time health of an IP is less than 0.4, the IP is removed from the resource pool;

[0041] When the real-time health of the IP continues to recover for a certain period of time, the IP will be added back to the resource pool.

[0042] A proxy IP scheduling system based on dynamic quality assessment includes: an IP health dynamic assessment model, which calculates the real-time health of the IP by fusing multiple indicators; a multi-constraint resource allocation model, which is used to create a spatial index based on the real-time health of the IP and multi-dimensional characteristics, and assign proxy IPs to tasks, under the premise of meeting multiple constraints; an exception handling module, which determines whether there are abnormal IPs based on the real-time health of the IPs and handles the abnormal IPs according to internal isolation strategies when there are abnormal IPs; and a dynamic resource management model, which is used to dynamically update the resource pool.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The method of the present invention improves resource utilization and scheduling efficiency through real-time health monitoring, multi-constraint resource allocation, and exception isolation; enhances the stability and disaster recovery capabilities of the acquisition system, and improves the task completion rate in high-concurrency scenarios; optimizes task success rate and privacy protection, reduces operation and maintenance costs, and improves flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of the proxy IP scheduling method based on dynamic quality assessment of the present invention.

[0046] Figure 2 Schematic diagram of the process of calculating the real-time health of IP according to the present invention.

[0047] Figure 3 The figure is a flow chart of creating a spatial index according to the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0050] Please also see Figures 1 to 3A preferred embodiment of the present invention provides a proxy IP scheduling method based on dynamic quality assessment. By integrating an IP health assessment model and a multi-constraint resource allocation model, it solves the problems of inefficiency and resource waste caused by unreasonable proxy IP allocation. It is suitable for large-scale distributed network data collection scenarios, achieves efficient resource scheduling in high-concurrency scenarios, and improves IP utilization and task success rate. Specifically, it includes:

[0051] Calculate the real-time health of IP through multi-indicator fusion;

[0052] Create spatial indexes based on the real-time health and multi-dimensional features of IP;

[0053] Determine whether there are abnormal IPs based on the real-time health of the IPs, and trigger isolation policies when abnormal IPs are found;

[0054] Input the task queue containing status data, and allocate optimal resources by combining the real-time dynamic health assessment model of IP and the multi-constraint resource allocation model;

[0055] Output the optimal candidate IP set.

[0056] In this embodiment, the calculation of the real-time health of the IP through multi-index fusion includes:

[0057] Create a dynamic IP health assessment model to calculate the health of IP by integrating multiple indicators;

[0058] Adjust the weight coefficient corresponding to each indicator according to task characteristics and system status;

[0059] Recalculate the real-time health of the IP based on the weight coefficient.

[0060] In this embodiment, the multiple indicators include response time, request success rate, bandwidth weight, concurrent task health and validity period decay. The formula for calculating the real-time health of the IP is:

[0061] HS(t)=w1*(1-Tcurr / Tmax)+w2*Sqs+w3*Bus+w4*C score +w5*e -λΔt ,

[0062] Where w1 is the weight coefficient of response time, w2 is the weight coefficient of request success rate (Sqs), w3 is the weight coefficient of bandwidth, w4 is the weight coefficient of concurrent task health, w5 is the weight coefficient of validity period decay, Tcurr is the average response time (ms), Tmax is the response time threshold (default is 5000ms), Sqs is the request success rate within the specified window time (the window time defaults to 5 minutes) = number of successes / total number of requests, Bus is the bandwidth utilization score, C scoreis the concurrent task health, and Δt is the IP survival time and activation time (seconds).

[0063] λ is the validity period decay factor (default is 0.1)

[0064]

[0065] Where B_used is the currently used bandwidth, B_avg is the expected bandwidth allocation value of a single IP in the current task scenario, and B_max is the maximum bandwidth of a single IP.

[0066]

[0067] Through real-time calculations, bandwidth resources are managed in a refined manner, adapting to changes in task load (such as sudden traffic spikes). Overload protection: When B_used approaches B_max, the IP score is lowered, reducing the number of new tasks assigned. Idle recovery: When B_used is significantly lower than B_avg (e.g., ≤0.5Mbps), the IP is marked as inefficient, triggering a resource recovery mechanism.

[0068] C score =max(0,1-N_current / N_max)

[0069] N_current is the current number of concurrent connections, and N_max is the maximum number of concurrent connections, which defaults to 20.

[0070] In this embodiment, adjusting the weight coefficient corresponding to each indicator according to the task characteristics and system status specifically includes:

[0071] When the task feature is high-frequency call, increase w1;

[0072] When performing streaming download tasks, increase w3;

[0073] When the system load is driven, when the overall bandwidth approaches the threshold, increase w3;

[0074] When the proportion of abnormal IPs increases, increase w2 and w5;

[0075] When the task is batch interactive or the system is under load, increase the w4 weight.

[0076] By dynamically adjusting w2, the system can flexibly balance the success rate and other resource indicators in different scenarios, avoiding bandwidth overload or premature IP failure due to excessive pursuit of success rate.

[0077] By dynamically adjusting weights, service stability is prioritized in critical tasks, the risk of business interruption is reduced, and multi-objective collaborative optimization is achieved.

[0078] In a specific embodiment, the weight coefficients (w1-w5) are shown in Table 1 below.

[0079] Table 1

[0080]

[0081] In this implementation, the real-time health of IP addresses is recalculated based on the weight coefficient. The adjusted weight coefficient is then applied to the aforementioned formula for calculating the real-time health of IP addresses. The IP status is then reported after task execution, and an IP that fails three times in a row is immediately marked as failed. IP addresses are also stratified into hot and cold zones. The hot pool stores IP addresses with ample remaining time, supporting real-time, high-priority tasks; the cold pool stores IP addresses with reduced remaining time, used for fault-tolerant retries or low-priority tasks.

[0082] In this implementation, the multi-constraint resource allocation model refers to optimally allocating resources (proxy IPs) to tasks (tasks) under the premise of satisfying multiple constraints (such as resource capacity, time window, priority, etc.) to maximize benefits (utilization) or minimize costs (such as time and energy consumption). The multi-constraint resource allocation model creates a spatial index based on the real-time health of IP and multidimensional features. The specific steps include:

[0083] Assess the priority of tasks, determine the corresponding weights of different tasks, and give priority to meeting the needs of high-weight tasks when allocating them;

[0084] An R-tree is used to organize and index resources in a multidimensional space. IP health, remaining bandwidth, and remaining concurrency are converted into normalized values ​​to ensure balanced weighting across dimensions. Each normalized resource is represented as a point in three-dimensional space. These points are inserted into the R-tree and dynamically updated. For example, when a resource status change exceeds a threshold (e.g., remaining bandwidth decreases), its normalized value is recalculated.

[0085] A hierarchical greedy allocation mechanism is set up. High-weight tasks are fully allocated first. When resources are insufficient, resource preemption is triggered. Medium and low-weight tasks are allocated greedily according to the standard allocation method.

[0086] Task requirements are mapped into three-dimensional space points according to their IP health, bandwidth, and concurrency. Starting from the root node, all resources that meet the conditions are quickly retrieved in the corresponding area of ​​the R-tree.

[0087] In this embodiment, the standard greedy allocation only considers the optimal match between the current task and the currently available resources at each allocation, and maximizes resource utilization or minimizes task waiting time under the premise of satisfying resource constraints (bandwidth, concurrency). The standard greedy allocation steps for medium and low weight tasks are: input the set of tasks to be allocated, and index the dynamic resource pool through the R-tree; output the matching results between tasks and resources, and directly allocate them when they are satisfied. The full allocation of high priority tasks can avoid the local defects of the greedy algorithm. If the high priority task needs resources, the system will reclaim the resources allocated to the low priority task (triggering preemption), and the preempted low priority task will re-enter the queue to be allocated and be reprocessed by the standard greedy allocation. The standard greedy allocation is used for medium and low priority tasks, which can quickly match the remaining resources and balance efficiency and fairness. Through this design, the system can not only guarantee the resource requirements of critical tasks, but also make efficient use of idle resources.

[0088] This implementation also includes updating the R-tree based on resource status changes; sharding based on resource lifetime to reduce the size of individual R-trees; and sharding by lifetime (e.g., short-term versus long-term resources) to reduce the size of individual R-trees. Auxiliary indexes are established for high-query frequency dimensions (e.g., remaining bandwidth); and high-frequency query results are cached to reduce the number of R-tree traversals. By mapping IP health, remaining bandwidth, and concurrency counts into three-dimensional spatial points and utilizing R-tree indexes for efficient multi-dimensional range queries, optimal resources can be quickly matched while satisfying multiple constraints.

[0089] In this implementation, the abnormal IP status is determined using anomaly detection rules. If the real-time health of an IP remains below a threshold for a certain period of time and the request failure rate exceeds a threshold, the IP status is considered abnormal. An isolation level is set, updated every 10 seconds, to isolate the abnormal IP.

[0090] In this embodiment, the isolation levels include soft isolation, hard isolation, and automatic recovery. The isolation strategy includes:

[0091] When the real-time health of the IP is between 0.4 and 0.6, the allocation of new tasks is suspended;

[0092] When the real-time health of an IP is less than 0.4, the IP is removed from the resource pool;

[0093] When the real-time health of the IP continues to recover for a certain period of time, the IP is added back to the resource pool, as shown in Table 2 below.

[0094] Table 2

[0095] Isolation Level Trigger Conditions Execute an action Soft Isolation Health 0.4~0.6 Pause new task assignments Hard isolation Health < 0.4 Remove from resource pool Automatic recovery Health continues to recover for a period of time Rejoin the resource pool

[0096] In this embodiment, the dynamic resource pool is also updated. After isolating the abnormal IP, the resource range in the R-tree is updated, that is, the abnormal node is deleted or marked.

[0097] Through the closed-loop linkage of anomaly detection models, hierarchical isolation strategies, and dynamic resource allocation, abnormal IP addresses can be efficiently identified and processed. The advantages of this solution are:

[0098] Real-time: The delay from detection to isolation can be controlled within seconds.

[0099] Self-healing ability: automatic unblocking mechanism combined with health recovery.

[0100] Resource efficiency: Resources released by abnormal IP addresses can be quickly reallocated to avoid service interruptions.

[0101] Another preferred embodiment of the present invention provides a proxy IP scheduling system based on dynamic quality assessment, comprising:

[0102] IP health dynamic assessment model, which calculates the real-time health of IP by integrating multiple indicators;

[0103] The multi-constraint resource allocation model is used to create a spatial index based on the real-time health and multi-dimensional characteristics of the IP, and assign proxy IPs to tasks under the premise of meeting multiple constraints.

[0104] The exception handling module determines whether there are abnormal IPs based on the real-time health of the IPs. If there are abnormal IPs, the module handles them according to the internal isolation strategy.

[0105] Dynamic resource management model for dynamically updating resource pools.

[0106] In some other preferred embodiments of the present invention, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method described in the above embodiment.

[0107] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0108] The above description is a detailed description of the preferred embodiments of the present invention, but the embodiments are not intended to limit the scope of the patent application of the present invention. Any equivalent changes or modifications completed under the technical spirit suggested by the present invention should fall within the patent scope covered by the present invention.

Claims

1. A proxy IP scheduling method based on dynamic quality assessment, characterized in that: include: Calculate the real-time health of IP through multi-indicator fusion; Create spatial indexes based on the real-time health and multi-dimensional features of IP; Determine whether there are abnormal IPs based on the real-time health of the IPs, and trigger isolation policies when abnormal IPs are found; Input the task queue containing status data, and allocate optimal resources by combining the real-time dynamic health assessment model of IP and the multi-constraint resource allocation model; Output the optimal candidate IP set.

2. The proxy IP scheduling method based on dynamic quality assessment according to claim 1, characterized in that: The real-time health of IP is calculated by integrating multiple indicators, including: Create a dynamic IP health assessment model to calculate the health of IP by integrating multiple indicators; Adjust the weight coefficient corresponding to each indicator according to task characteristics and system status; Recalculate the real-time health of the IP based on the weight coefficient.

3. The proxy IP scheduling method based on dynamic quality assessment according to claim 2, characterized in that: The multiple indicators include response time, request success rate, bandwidth weight, concurrent task health and validity period decay. The formula for calculating the real-time health of IP is: HS(t)=w1*(1-Tcurr / Tmax)+w2*Sqs+w3*Bus+w4*C score +w5*e -λΔt , Where w1 is the weight coefficient of response time, w2 is the weight coefficient of request success rate, w3 is the weight coefficient of bandwidth, w4 is the weight coefficient of concurrent task health, w5 is the weight coefficient of validity period decay, Tcurr is the average response time, Tmax is the response time threshold, Sqs is the request success rate within the specified window time, Bus is the bandwidth utilization score, and C score is the concurrent task health, Δt is the IP survival time and activation time, λ is the validity period decay factor, Where B_used is the currently used bandwidth, B_avg is the expected bandwidth allocation value of a single IP in the current task scenario, and B_max is the maximum bandwidth of a single IP.

4. The proxy IP scheduling method based on dynamic quality assessment according to claim 2, characterized in that: Adjust the weight coefficient of each indicator according to the task characteristics and system status, including: When the task feature is high-frequency call, increase w1; When performing streaming download tasks, increase w3; When the system load is driven, when the overall bandwidth approaches the threshold, increase w3; When the proportion of abnormal IPs increases, increase w2 and w5; When the task is batch interactive or the system is under load, increase the w4 weight.

5. The proxy IP scheduling method based on dynamic quality assessment according to claim 4, characterized in that: The creation of a spatial index based on the real-time health of IP and multi-dimensional features specifically includes: Prioritize tasks and determine the corresponding weights of different tasks; Use R-tree to organize and index resources based on multidimensional space; A hierarchical greedy allocation mechanism is set up. High-weight tasks are fully allocated first. When resources are insufficient, resource preemption is triggered. Medium and low-weight tasks are allocated greedily according to the standard allocation method. Task requirements are mapped into three-dimensional space points according to their IP health, bandwidth, and concurrency. Starting from the root node, all resources that meet the conditions are quickly retrieved in the corresponding area of ​​the R-tree.

6. The proxy IP scheduling method based on dynamic quality assessment according to claim 1, characterized in that: It also includes updating the R-tree according to resource status changes, sharding according to resource survival time, and reducing the single tree size of the R-tree.

7. The proxy IP scheduling method based on dynamic quality assessment according to claim 1, characterized in that: The judgment of whether there is an abnormal IP adopts an abnormal detection rule. When the real-time health of the IP is continuously lower than the threshold for a certain period of time and the request failure rate is higher than the threshold, the IP status is deemed to be abnormal.

8. The proxy IP scheduling method based on dynamic quality assessment according to claim 1, characterized in that: The isolation strategy includes: When the real-time health of the IP is between 0.4 and 0.6, the allocation of new tasks is suspended; When the real-time health of an IP is less than 0.4, the IP is removed from the resource pool; When the real-time health of the IP continues to recover for a certain period of time, the IP will be added back to the resource pool.

9. A proxy IP scheduling system based on dynamic quality assessment, used to implement the proxy IP scheduling method based on dynamic quality assessment according to any one of claims 1 to 8, characterized in that: include: IP health dynamic assessment model, which calculates the real-time health of IP by integrating multiple indicators; The multi-constraint resource allocation model is used to create a spatial index based on the real-time health and multi-dimensional characteristics of the IP, and assign proxy IPs to tasks under the premise of meeting multiple constraints. The exception handling module determines whether there are abnormal IPs based on the real-time health of the IPs. If there are abnormal IPs, the module handles them according to the internal isolation strategy. Dynamic resource management model for dynamically updating resource pools.

10. A storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to perform the steps of the proxy IP scheduling method based on dynamic quality assessment as described in any one of claims 1 to 8.

Images