An efficient proxy pool management method based on dynamic selection and intelligent optimization

Abstract

The application belongs to the technical field of computer networks, and discloses a kind of efficient proxy pool management method based on dynamic selection and intelligent optimization, comprising: obtaining the multidimensional performance index data of each proxy node in proxy pool;Obtain the activity measure value of each proxy node;Obtain the quality floating index of each proxy node;Multi-level division is carried out to proxy pool, hierarchical proxy resource pool is formed, and dynamic expansion and contraction is carried out;Select the optimal proxy node from hierarchical proxy resource pool to execute task;Real-time monitoring of proxy node execution efficiency, update activity measure value and quality floating index;The application realizes efficient management and task allocation of proxy resources through dynamic selection mechanism and intelligent optimization strategy, can be adjusted adaptively according to real-time performance change, improve the overall operation efficiency and stability of system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer networks, and more particularly, to an efficient proxy pool management method based on dynamic selection and intelligent optimization. BACKGROUND

[0002] In the era of rapid development of the Internet, proxy pool technology, as an important tool for network data collection, privacy protection and load balancing, is widely used in many fields. For example, when enterprises conduct web crawling, they can bypass access restrictions of target websites through proxy pools to achieve more efficient data collection; in network security, proxy pools can help users hide real IP to prevent malicious attacks; in load balancing scenarios, proxy pools can distribute traffic to different proxy servers to optimize network performance.

[0003] Traditional proxy pool management methods usually rely on fixed proxy selection strategies and simple resource allocation mechanisms; however, these methods have the following defects: uncontrollable proxy quality: the availability, delay and stability of proxies will fluctuate significantly due to changes in network environment, but traditional methods cannot evaluate proxy quality in real time, resulting in decreased overall performance; low resource utilization: fixed proxy selection strategies may cause high-quality proxies to be overused while low-quality proxy resources are idle, resulting in resource waste; lack of dynamic optimization capability: when facing increasing proxy pool size or dynamic changes in proxy usage demand, traditional methods cannot effectively adjust and optimize resource allocation, making it difficult to meet the needs of high-concurrency and high-performance application scenarios.

[0004] In view of this, the present application proposes an efficient proxy pool management method based on dynamic selection and intelligent optimization to solve the above problems. SUMMARY

[0005] In order to overcome the above-mentioned defects of the prior art, in order to achieve the above-mentioned purpose, the present application provides the following technical solution: an efficient proxy pool management method based on dynamic selection and intelligent optimization, comprising:

[0006] Step S1: obtaining multi-dimensional performance index data of each proxy node in the proxy pool;

[0007] Step S2: obtaining an activity metric value of each proxy node according to the multi-dimensional performance index data and historical performance records of each proxy node;

[0008] Step S3: obtaining a quality fluctuation index of each proxy node based on the difference between the activity metric value of each proxy node and the activity metric value of the neighboring proxy node;

[0009] Step S4: According to the quality floating index, the agent pool is divided into multiple levels to form a hierarchical agent resource pool, and the load level of the hierarchical agent resource pool is monitored in real time and the hierarchical agent resource pool is dynamically expanded and shrunk;

[0010] Step S5: According to the matching degree of task demand characteristics and agent nodes, the optimal agent node is selected from the hierarchical agent resource pool to execute the task;

[0011] Step S6: Real-time monitoring of the performance of the agent node executing the task, updating the activity metric value and the quality floating index of the agent node.

[0012] Further, the multi-dimensional performance index data of each agent node in the agent pool is obtained, including:

[0013] Sending a probe signal to each agent node at a preset time interval, recording the response state and response time of the probe signal;

[0014] Assigning test tasks to each agent node, recording task completion and completion quality;

[0015] Collecting the bandwidth utilization, connection stability and historical success rate of each agent node;

[0016] Combining the response state, response time, task completion, completion quality, bandwidth utilization, connection stability and historical success rate into multi-dimensional performance index data of the agent node.

[0017] Further, the activity metric value of each agent node is obtained, including:

[0018] Setting an evaluation weight for each performance indicator, and dynamically adjusting the evaluation weight according to the task type;

[0019] Calculating the timing fluctuation amplitude of each agent node on each performance indicator to obtain the response agility;

[0020] Combining the current value, historical average value, response agility and corresponding evaluation weight of each performance indicator to generate the activity metric value of the agent node.

[0021] Further, the evaluation weight is dynamically adjusted according to the task type, including:

[0022] Identifying agent usage patterns in different network environments to establish a scenario profile;

[0023] Adjusting the evaluation weight of the activity metric value according to the scenario profile; in a high-concurrency scenario, increasing the evaluation weight of bandwidth utilization and connection stability; in a data collection scenario, increasing the evaluation weight of response time and completion quality.

[0024] Further, the quality floating index of each agent node is obtained, comprising:

[0025] A near neighbor agent set is established for each agent node, which contains the agent nodes with the most similar activity metric values to the agent node, i.e. near neighbor agent nodes;

[0026] The deviation of the activity metric value of each agent node from the activity metric values of the agent nodes in the near neighbor agent set of the agent node is calculated;

[0027] The quality floating index of the agent node is determined according to the deviation and the historical performance stability.

[0028] Further, the agent pool is divided into multiple levels according to the quality floating index, comprising:

[0029] The level division threshold is determined according to the quality floating index of the agent node;

[0030] The agent nodes with the quality floating index higher than the first threshold are classified into a preferred resource layer;

[0031] The agent nodes with the quality floating index between the first threshold and the second threshold are classified into a regular resource layer;

[0032] The agent nodes with the quality floating index lower than the second threshold are classified into a standby resource layer;

[0033] The quality floating index of the agent nodes in each layer is re-evaluated regularly, and the layer to which the agent node belongs is dynamically adjusted according to the evaluation result.

[0034] Further, the load level of the hierarchical agent resource pool is monitored in real time, and the hierarchical agent resource pool is dynamically expanded and contracted, comprising:

[0035] The overall load level of the agent pool and the distribution of the agent nodes in each resource layer are continuously monitored;

[0036] When the number of agent nodes in the preferred resource layer is lower than a preset threshold, M agent nodes with the best performance in the regular resource layer are promoted to the preferred resource layer;

[0037] When the overall load is continuously higher than a preset threshold, new agent resources are introduced from the outside;

[0038] When the overall load is continuously lower than a preset threshold, N agent nodes with the worst performance in the standby resource layer are removed.

[0039] Further, the optimal agent node is selected from the hierarchical agent resource pool to perform a task, comprising:

[0040] The characteristics of the task demand are analyzed, and the key demand of the task on the performance of the agent node is extracted;

[0041] The matching degree of each agent node and the task demand is calculated to generate a task affinity score;

[0042] The agent node with the highest task affinity score is selected from the preferred resource layer as a priority;

[0043] When there is no suitable agent node in the preferred resource layer, the agent node with the highest task affinity score is selected from the general resource layer;

[0044] When there is no suitable agent node in the general resource layer, the agent node with the highest task affinity score is selected from the backup resource layer.

[0045] Further, the activity metric value and the quality floating index of the agent node are updated, comprising:

[0046] The success rate, response time, data transmission rate and connection interruption times of the agent node in executing the task are recorded;

[0047] The deviation value of the current performance and the historical performance is calculated;

[0048] Based on the deviation value, the activity metric value of the agent node is updated;

[0049] When the deviation value exceeds a preset threshold value, the re-calculation and hierarchical adjustment of the quality floating index of the agent node are triggered, and the agent node abnormality evaluation is performed.

[0050] Further, the agent node abnormality evaluation comprises:

[0051] A multi-level abnormality judgment standard is set, including slight abnormality, moderate abnormality and serious abnormality;

[0052] When the agent node is detected to have slight abnormality, the priority of the agent node in the current resource layer is reduced;

[0053] When the agent node is detected to have moderate abnormality, the agent node is degraded to the next resource layer;

[0054] When the agent node is detected to have serious abnormality, the agent node is temporarily removed from the agent pool and a cooling recovery period is set;

[0055] After the cooling recovery period ends, the performance of the agent node is re-evaluated, and when the performance of the agent node reaches a preset performance indicator, the agent node is recovered for use.

[0056] The technical effect and advantages of the high-efficiency agent pool management method based on dynamic selection and intelligent optimization are as follows:

[0057] The application realizes comprehensive evaluation of the proxy nodes by multi-dimensional performance index collection and historical performance record analysis, can accurately quantify the real-time state and performance level of each proxy node, thereby enhancing the reliability of the proxy pool. Through the calculation of the activity metric value and the quality floating index, the performance fluctuation and stability characteristics of the proxy nodes can be captured, and high-quality proxy resources can be effectively identified and screened. Based on the multi-level division mechanism of the quality floating index, a three-level resource pool structure of preferred, regular and standby is established, which significantly improves the accuracy of resource allocation and avoids resource waste and unreasonable task allocation problems. Through task demand and proxy node matching degree analysis, the most suitable proxy node can be selected for different types of tasks, improving the task execution success rate and efficiency. The real-time monitoring mechanism can dynamically adjust the evaluation index and resource level of the proxy nodes, and according to the load level, the resource expansion and contraction is carried out, so that the proxy pool is always in the optimal state. The multi-level abnormality judgment standard and the cooling recovery mechanism effectively prevent the problem proxy from continuously affecting the system performance, and improve the overall stability. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A high-efficiency proxy pool management method based on dynamic selection and intelligent optimization according to the present application;

[0059] Figure 2 A high-efficiency proxy pool management system based on dynamic selection and intelligent optimization according to the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0061] Embodiment 1

[0062] Please refer to Figure 1 The high-efficiency proxy pool management method based on dynamic selection and intelligent optimization according to the present embodiment includes:

[0063] Step S1: Obtain multi-dimensional performance index data of each proxy node in the proxy pool.

[0064] In the proxy pool environment, the performance of the proxy nodes has a decisive influence on the overall efficiency of the system. By collecting the multi-dimensional performance index data of the proxy nodes in real time, the current state and working capacity of the proxy nodes can be comprehensively evaluated. The multi-dimensional performance index data is the basis for evaluating the performance of the proxy nodes, and by comprehensively collecting various indicators, a complete performance portrait of the proxy nodes can be constructed, providing data support for subsequent intelligent allocation and optimization.

[0065] In an implementation manner of the embodiment of the application, a probe signal is sent to each agent node in the agent pool at an interval of every 30 seconds, and the response state and response duration of the probe signal are recorded; at the same time, standardized test tasks are periodically distributed to each agent node, and the completion condition and completion quality of the tasks are recorded; in addition, the bandwidth utilization, connection stability and historical success rate of each agent node are continuously collected.

[0066] It should be noted that the frequency of receiving the probe signal by all agent nodes in the agent pool is the same, but the distribution of the test tasks can be differentially adjusted according to the historical performance of the agent nodes.

[0067] Step S2: Obtain the activity metric value of each agent node according to the multi-dimensional performance index data and historical performance record of each agent node.

[0068] The performance of the agent node is not static but dynamically changes with time and load. The activity metric value is a comprehensive index for measuring the current working capacity and stability of the agent node, which not only reflects the current state of the agent node, but also considers the stability and trend of its historical performance. By calculating the activity metric value, the actual availability and performance potential of the agent node can be more accurately evaluated.

[0069] The calculation of the activity metric value needs to consider the current value, historical average value and time sequence fluctuation of the multi-dimensional performance index, and dynamically adjust the evaluation weight of each index according to the demand characteristics of different task types, so as to obtain a more realistic scene activity evaluation result.

[0070] Step S3: Obtain the quality floating index of each agent node based on the difference between the activity metric value of each agent node and the activity metric value of the near neighbor agent node.

[0071] In the agent pool, there is a limitation in simply relying on the absolute activity metric value to evaluate the performance of the agent node, because the network conditions faced by the agent nodes in different network environments and different geographical locations are different. The quality floating index can find the relative performance and stability of the agent node by comparing the difference between the activity metric value of the agent node and the activity metric value of the near neighbor agent node, and then more accurately evaluate the quality level of the agent node.

[0072] The quality floating index not only considers the performance of the agent node itself, but also considers the relative performance of the agent node with the same type of agent node. This relative evaluation method can effectively reduce the influence of environmental factors on the evaluation result, and improve the accuracy and fairness of the evaluation.

[0073] Step S4: Divide the agent pool into multiple levels according to the quality floating index, form a hierarchical agent resource pool, and monitor the load level of the hierarchical agent resource pool in real time and dynamically expand or shrink the hierarchical agent resource pool.

[0074] The performance of the agent nodes in the agent pool varies. By dividing the agent pool into multiple levels through the quality floating index, the agent resources can be more effectively managed and dispatched. The division of the hierarchical agent resource pool enables the system to reasonably allocate agent resources according to the importance and urgency of tasks, improving the overall system efficiency. At the same time, by monitoring the load level of each level in real time and dynamically adjusting the allocation and total amount of agent resources, the system can maintain high efficiency under various load conditions.

[0075] Through multi-level division and dynamic expansion and contraction, resource utilization can be maximized while ensuring service quality, reducing operating costs. When the system load is high, new agent resources can be introduced in time; when the system load is low, agent resources can be appropriately reduced to achieve efficient use of resources.

[0076] Step S5: Select the optimal agent node from the hierarchical agent resource pool according to the matching degree of task demand characteristics and agent nodes.

[0077] Different tasks have different performance requirements for agent nodes, such as data collection tasks requiring high response time and stability, and large file transfer tasks requiring higher bandwidth utilization. By analyzing the task demand characteristics and calculating the matching degree of agent nodes and tasks, the most suitable agent node can be selected for each task, improving task execution efficiency and success rate.

[0078] When selecting the optimal agent node, the agent node with the highest matching degree is preferred from the preferred resource layer. Only when there is no suitable agent node in the preferred resource layer, will the agent node be selected from the regular resource layer and the standby resource layer in turn. This hierarchical selection strategy not only guarantees the execution quality of important tasks, but also improves the resource utilization efficiency of the overall system.

[0079] Step S6: Real-time monitoring of the performance of the agent nodes executing tasks, updating the activity metric value and quality floating index of the agent nodes.

[0080] The performance state of the agent nodes is dynamic. By monitoring the performance of the agent nodes executing tasks in real time and updating the activity metric value and quality floating index of the agent nodes in a timely manner, the evaluation of the agent nodes can always be consistent with their actual state. This closed-loop feedback mechanism enables the system to dynamically adjust the resource level of the agent nodes, optimize the task allocation strategy, and improve the adaptability and stability of the overall system.

[0081] Through continuous monitoring and updating, performance abnormalities of the proxy nodes can be found in time, and corresponding adjustment measures can be taken, such as reducing priority, downgrading processing, or temporarily removing the proxy nodes from the proxy pool, to ensure the overall service quality of the proxy pool.

[0082] Preferably, in some possible implementation manners of the embodiments of the present application, the method for obtaining the multi-dimensional performance index data of the proxy nodes comprises: sending a probe signal to each proxy node at a preset time interval, recording a response state and a response time length of the probe signal; assigning a test task to each proxy node, recording a task completion condition and a completion quality; collecting a bandwidth utilization rate, a connection stability, and a historical success rate of each proxy node; and combining the response state, the response time length, the task completion condition, the completion quality, the bandwidth utilization rate, the connection stability, and the historical success rate into the multi-dimensional performance index data of the proxy node.

[0083] It should be noted that the probe signal can be a simple ICMP packet or an HTTP request, which is used to test the basic reachability and response speed of the proxy node; the test task is a standardized task simulating an actual business scenario, which is used to comprehensively evaluate the actual working capacity of the proxy node; the bandwidth utilization rate reflects the resource use efficiency of the proxy node, the connection stability reflects the reliability of the proxy node, and the historical success rate reflects the long-term performance of the proxy node. The combination of these multi-dimensional indexes can comprehensively reflect the performance state of the proxy node. The response state indicates whether the proxy node can successfully request, and includes 0 (failure) and 1 (success); the response time length represents the time length of the proxy response, reflecting the smoothness of the proxy, and the calculation formula is: , wherein represents the response time length, represents the delay; the task completion condition represents the probability of returning the expected content successfully, and is defined as the ratio of the number of successes to the total number of times; the completion quality represents the structural integrity of the returned content, 1 represents complete, 0 represents incomplete, and decreases in proportion; the bandwidth utilization rate represents the download rate per unit time, and the calculation formula is: , wherein represents the bandwidth utilization rate, represents the bandwidth; the connection stability: whether the proxy is frequently disconnected, which measures the resulting continuous availability, and the calculation formula is: ; wherein represents the connection stability, represents the disconnection rate.

[0084] In one implementation manner of the embodiments of the present application, the sending interval of the probe signal is set to 30 seconds, and the assignment frequency of the test task is set to once per hour.

[0085] Preferably, in some possible implementations of the embodiments of the present invention, the method for calculating the activity metric includes: setting an evaluation weight for each performance indicator, wherein the evaluation weight is dynamically adjusted according to the task type; calculating the temporal fluctuation amplitude of each proxy node on each performance indicator to obtain the response agility; and combining the current value, historical average value, response agility and corresponding evaluation weight of each performance indicator to generate the activity metric of the proxy node.

[0086] It should be noted that the evaluation weights of performance metrics determine the importance of each metric in the calculation of the activity metric, and these weights are dynamically adjusted according to different task types. Response agility reflects the agent node's ability to adapt to load changes and is an important indicator for evaluating the agent node's stability. A comprehensive consideration of the current value, historical average, and response agility can comprehensively evaluate the performance status and potential of the agent node. The activity metric is a comprehensive score that measures the overall availability and stability of an agent node, obtained by summing the weights of each performance metric and its corresponding evaluation weight. The formula for calculating the activity metric is: ;in, Represents an activity metric. Represents the total amount of indicators. Representing the The weights of each indicator can be dynamically adjusted according to the application scenario. Representing the The original values ​​of each indicator Representing the Normalization or scoring function for each indicator.

[0087] In one embodiment of the present invention, the dynamic adjustment mechanism for evaluation weights of different task types includes: identifying proxy usage patterns under different network environments and establishing scenario profiles; adjusting the evaluation weights of activity metrics based on the scenario profiles. In high-concurrency scenarios, the evaluation weights for bandwidth utilization and connection stability are increased; in data acquisition scenarios, the evaluation weights for response time and completion quality are increased.

[0088] Preferably, in some possible implementations of the embodiments of the present invention, the method for calculating the quality fluctuation index includes: establishing a nearest neighbor agent set for each agent node, wherein the nearest neighbor agent set contains the agent nodes whose activity metric value is closest to that of the agent node, i.e., the nearest neighbor agent nodes; calculating the degree of deviation between the activity metric value of each agent node and the activity metric values ​​of each agent node in its nearest neighbor agent set; and determining the quality fluctuation index of the agent node based on the degree of deviation and historical performance stability.

[0089] It should be noted that the nearest neighbor proxy set refers to a group of proxy nodes that are similar to the target proxy node in terms of performance characteristics, usually selected from those with the closest activity metrics. The number of proxy nodes; the deviation degree measures the performance difference between the proxy node and its near neighbor proxy nodes, which can be calculated using statistical methods such as standard deviation or mean absolute deviation; the historical performance stability reflects the fluctuation of the performance of the proxy node, and the higher the stability, the higher the quality fluctuation index.

[0090] In an implementation manner of the embodiment of the application, the size of the near neighbor proxy set is set to .

[0091] Obtain the total quota of available proxy IPs per day: ;

[0092] Obtain the average value of the proxy life cycle: (unit: minutes);

[0093] Set the total length of the scheduling period: (the default value is 1440 minutes);

[0094] Calculate the target proxy pool capacity according to the following formula: ; represents the floor function;

[0095] Verification case: private proxy (centralized extraction), IP quantity: 20000 / day, loan peak: 2.0 Mbps, IP validity duration: 1-5 minutes;

[0096] Capacity: ;

[0097] The specific calculation formula of the quality fluctuation index is: ; wherein, represents the quality fluctuation index, represents the evaluation value of the historical performance stability, represents the size of the near neighbor proxy set, represents the near neighbor proxy set, represents the activity metric value of the th proxy node in the near neighbor proxy set, represents the activity metric value of the currently evaluated proxy node.

[0098] Preferably, in some possible implementation manners of the embodiment of the application, the multi-level division method comprises: determining a hierarchical division threshold according to the quality fluctuation index of the proxy node; dividing the proxy node with a quality fluctuation index higher than a first threshold into a preferred resource layer; dividing the proxy node with a quality fluctuation index between the first threshold and a second threshold into a regular resource layer; dividing the proxy node with a quality fluctuation index lower than the second threshold into a backup resource layer; periodically re-evaluating the quality fluctuation index of the proxy nodes in each layer, and dynamically adjusting the layer to which the proxy node belongs according to the evaluation result.

[0099] It should be noted that the hierarchical division threshold can be dynamically adjusted according to the overall performance distribution of the agent pool, so as to ensure that the number of agent nodes in each resource layer is reasonable; the preferred resource layer contains the agent nodes with the best performance, and is mainly used to execute important or high-priority tasks; the regular resource layer contains the agent nodes with medium performance, and is mainly used to execute regular tasks; and the standby resource layer contains the agent nodes with weak performance, and is mainly used to provide supplementary resources in a high-load situation.

[0100] In an implementation manner of the embodiment of the application, the first threshold is set to 0.85, the second threshold is set to 0.6, and the agent node hierarchical reevaluation period is set to 12 hours.

[0101] Preferably, in some possible implementation manners of the embodiment of the application, the dynamic expansion and contraction method of the hierarchical agent resource pool comprises: continuously monitoring the overall load level of the agent pool and the agent node distribution of each resource layer; when the number of agent nodes in the preferred resource layer is lower than a preset threshold, promoting M agent nodes with the best performance in the regular resource layer to the preferred resource layer; when the overall load is continuously higher than a preset threshold, introducing new agent resources from outside; and when the overall load is continuously lower than a preset threshold, removing N agent nodes with the worst performance in the standby resource layer.

[0102] It should be noted that the overall load level of the agent pool can be measured by indicators such as the average task execution rate or the queue length of the agent nodes; the number of agent nodes in the preferred resource layer should be kept within a reasonable range to meet the execution demand of important tasks; the overall load continuously higher than the preset threshold indicates that the system resources are insufficient, and the agent resources need to be expanded; and the overall load continuously lower than the preset threshold indicates that the system resources are excessive, and the agent resources can be appropriately reduced.

[0103] In an implementation manner of the embodiment of the application, M is set to 5, N is set to 10, the preset threshold of the number of agent nodes in the preferred resource layer is set to 20% of the total number of agent nodes, the high preset threshold of the overall load is set to 80%, and the low preset threshold is set to 30%.

[0104] Preferably, in some possible implementation manners of the embodiment of the application, the agent node exception evaluation method comprises: setting multi-level exception judgment standards, including slight exception, moderate exception and serious exception; when a slight exception of an agent node is detected, the priority of the agent node in the current resource layer is reduced; when a moderate exception of the agent node is detected, the agent node is downgraded to the next resource layer; when a serious exception of the agent node is detected, the agent node is temporarily removed from the agent pool, and a cooling recovery period is set; after the cooling recovery period ends, the performance of the agent node is reevaluated, and the agent node is recovered when the performance of the agent node reaches a preset performance indicator.

[0105] It should be noted that slight abnormalities refer to short-term fluctuations in agent performance but still complete tasks; moderate abnormalities refer to significant performance degradation or reduced task completion rate; severe abnormalities refer to agents being unable to complete tasks normally or posing security risks; the cooling recovery period setting can give agents enough time to recover performance and avoid permanent removal of valuable agent resources due to short-term abnormalities.

[0106] In one implementation of an embodiment of the present application, the cooling recovery period is set to 24 hours and the performance evaluation period is set to 4 hours.

[0107] Through the above-mentioned efficient proxy pool management method based on dynamic selection and intelligent optimization, precise allocation and efficient utilization of proxy resources can be achieved, significantly improving the overall performance and stability of the system, while reducing operating costs, and providing high-quality proxy service support for various network applications.

[0108] This embodiment achieves comprehensive evaluation of agent nodes through multi-dimensional performance index collection and historical performance record analysis, accurately quantifying the real-time state and performance level of each agent node, thereby enhancing the reliability of the proxy pool. Through the calculation of activity metric and quality floating index, the performance fluctuations and stability characteristics of agent nodes can be captured, effectively identifying and selecting high-quality proxy resources. Based on the multi-level division mechanism of the quality floating index, a three-level resource pool structure of preferred, regular, and standby is established, significantly improving the accuracy of resource allocation and avoiding resource waste and unreasonable task allocation. Through task demand and agent node matching degree analysis, the most suitable agent node can be selected for different types of tasks, improving task execution success rate and efficiency. Real-time monitoring mechanism can dynamically adjust the evaluation indicators and resource levels of agent nodes, and expand or shrink resources according to load levels, ensuring that the proxy pool is always in the best state. The multi-level abnormality judgment standard and cooling recovery mechanism effectively prevent problem agents from continuously affecting system performance, improving overall stability.

[0109] Embodiment 2

[0110] Please refer to Figure 2 The embodiment does not describe some parts in detail, see the description of embodiment 1, and provides an efficient proxy pool management system based on dynamic selection and intelligent optimization, which includes:

[0111] Data acquisition module: acquire multi-dimensional performance index data of each agent node in the proxy pool;

[0112] Active evaluation module: according to the multi-dimensional performance index data and historical performance records of each agent node, obtain the activity metric value of each agent node;

[0113] The quality evaluation module obtains a quality floating index of each agent node based on the difference between the activity metric value of each agent node and the activity metric value of the near neighbor agent node;

[0114] The hierarchical division module divides the agent pool into multiple levels according to the quality floating index, forms a hierarchical agent resource pool, and monitors the load level of the hierarchical agent resource pool in real time and dynamically expands or shrinks the hierarchical agent resource pool;

[0115] The agent selection module selects the optimal agent node from the hierarchical agent resource pool to execute the task according to the matching degree between the task demand characteristics and the agent node;

[0116] The node monitoring module monitors the performance of the agent node in executing the task in real time, and updates the activity metric value and the quality floating index of the agent node;

[0117] Each module is connected through wired and / or wireless means to realize data transmission between modules.

[0118] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0119] It should be noted that in this paper, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0120] In the description of the present application, it should be understood that the terms "first", "second" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0121] In the description of the present application, unless otherwise specified, the meaning of "multiple" is two or more.

[0122] In the description of the present application, the meaning of "several" is one or more, and the meaning of "a large number" is two or more.

[0123] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are contained in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any suitable manner in any one or more embodiments or examples.

[0124] The formula in the specification is a value calculated by de-dimensioning, the formula is obtained by collecting a large amount of data to simulate a formula of the most recent real situation, and the preset parameters and threshold values in the formula are set by a person skilled in the art according to the actual situation.

[0125] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirit of the present application.

Claims

1. A highly efficient agent pool management method based on dynamic selection and intelligent optimization, characterized in that, include: Step S1: Obtain multi-dimensional performance index data for each proxy node in the proxy pool; Step S2: Based on the multi-dimensional performance index data and historical performance records of each agent node, obtain the activity metric value of each agent node; Step S3: Based on the difference between the activity metric of each proxy node and the activity metric of its nearest neighboring proxy nodes, obtain the quality fluctuation index of each proxy node; Step S4: Divide the proxy pool into multiple levels according to the quality fluctuation index to form a hierarchical proxy resource pool, monitor the load level of the hierarchical proxy resource pool in real time, and dynamically expand or shrink the hierarchical proxy resource pool. Step S5: Based on the matching degree between the task requirements and the proxy node, select the optimal proxy node from the hierarchical proxy resource pool to execute the task; Step S6: Monitor the performance of the agent nodes in executing tasks in real time, and update the activity metric and quality fluctuation index of the agent nodes; The method of dividing the agent pool into multiple levels based on the quality fluctuation index includes: The hierarchical division threshold is determined based on the quality fluctuation index of the proxy nodes; Agent nodes whose quality fluctuation index is higher than the first threshold are classified into the preferred resource layer; Agent nodes whose quality fluctuation index falls between the first and second thresholds are classified into the regular resource layer. Agent nodes whose quality fluctuation index is below the second threshold will be reassigned to the standby resource layer. Regularly reassess the quality fluctuation index of each proxy node and dynamically adjust the level to which the proxy node belongs based on the assessment results; The real-time monitoring of the load level of the hierarchical proxy resource pool and the dynamic scaling of the hierarchical proxy resource pool include: Continuously monitor the overall load level of the proxy pool and the distribution of proxy nodes in each resource layer; When the number of proxy nodes in the preferred resource layer is lower than a preset threshold, the best-performing proxy node among the M regular resource layers is promoted to the preferred resource layer. When the overall load continues to exceed a preset threshold, new proxy resources are introduced from outside. When the overall load remains below a preset threshold, remove the worst-performing proxy node from the N backup resource layers.

2. The efficient agent pool management method based on dynamic selection and intelligent optimization according to claim 1, characterized in that, The acquisition of multi-dimensional performance index data for each proxy node in the proxy pool includes: Send probe signals to each agent node at preset time intervals and record the response status and response duration of the probe signals; Assign test tasks to each agent node and record the task completion status and quality. Collect bandwidth utilization, connection stability, and historical success rate for each proxy node; The response status, response time, task completion status, completion quality, bandwidth utilization, connection stability, and historical success rate are combined to form multidimensional performance index data for the proxy node.

3. The efficient agent pool management method based on dynamic selection and intelligent optimization according to claim 2, characterized in that, The process of obtaining the activity metric for each proxy node includes: Each performance metric is assigned an evaluation weight, which is dynamically adjusted based on the task type. Calculate the time-series fluctuation of each agent node across various performance metrics to obtain the response agility; By combining the current value, historical average value, response agility, and corresponding evaluation weight of each performance indicator, an activity metric for the agent node is generated.

4. The efficient agent pool management method based on dynamic selection and intelligent optimization according to claim 3, characterized in that, The evaluation weights are dynamically adjusted according to the task type, including: Identify proxy usage patterns under different network environments and create scenario profiles; Adjust the evaluation weight of activity metrics based on scenario profiles; in high-concurrency scenarios, increase the evaluation weight of bandwidth utilization and connection stability; in data acquisition scenarios, increase the evaluation weight of response time and completion quality.

5. The efficient agent pool management method based on dynamic selection and intelligent optimization according to claim 4, characterized in that, The process of obtaining the quality fluctuation index for each proxy node includes: For each agent node, a nearest agent set is established, which includes the agent nodes whose activity metric is closest to that of the agent node, i.e., the nearest agent nodes. Calculate the degree of deviation between the activity metric of each agent node and the activity metric of each agent node in its nearest agent set; The quality fluctuation index of the proxy node is determined based on the degree of deviation and historical performance stability.

6. The efficient agent pool management method based on dynamic selection and intelligent optimization according to claim 5, characterized in that, The step of selecting the optimal proxy node from the hierarchical proxy resource pool to execute the task includes: Analyze the characteristics of task requirements and extract the key performance requirements of the proxy node. Calculate the matching degree between each agent node and the task requirements, and generate a task affinity score; Prioritize selecting the agent node with the highest task affinity score from the preferred resource layer; When there is no suitable proxy node in the preferred resource layer, select the proxy node with the highest task affinity score from the regular resource layer; When there are no suitable proxy nodes in the regular resource layer, select the proxy node with the highest task affinity score from the backup resource layer.

7. The efficient agent pool management method based on dynamic selection and intelligent optimization according to claim 6, characterized in that, The updated agent node's activity metric and quality fluctuation index include: Record the success rate, response time, data transmission rate, and number of connection interruptions of the agent node in executing tasks; Calculate the deviation between current performance and historical performance; Based on the deviation value, update the activity metric of the agent node; When the deviation value exceeds the preset threshold, the agent node quality fluctuation index is recalculated and the hierarchy is adjusted, and an agent node anomaly assessment is performed.

8. The efficient agent pool management method based on dynamic selection and intelligent optimization according to claim 7, characterized in that, The process of performing anomaly assessment on proxy nodes includes: Establish multi-level abnormality judgment criteria, including minor abnormality, moderate abnormality, and severe abnormality; When a minor anomaly is detected in a proxy node, its priority within the current resource layer is reduced. When a moderate anomaly is detected in a proxy node, it is downgraded to the next resource layer; When a serious anomaly is detected in a proxy node, it is temporarily removed from the proxy pool, and a cooldown recovery period is set. After the cooling-off period, the performance of the proxy nodes is reassessed, and they are put back into use when their performance reaches the preset performance indicators.

Images