A method and system for implementing a proxy pool for dynamically controlling the source of proxy IPs

Abstract

The present invention relates to a method and system for implementing a proxy pool for dynamically controlling the source of proxy IPs, belonging to the field of computers. The method includes the following steps: S1: Obtain proxy IPs: Dynamically control the source according to the configuration; S2: Detect proxy IPs: Cyclically detect the origin IP proxy pool at intervals; S3: Polling detection and alarm for the app proxy pool: Start the polling detection service. When the quantity is insufficient, turn on the switch for obtaining paid proxies and give an alarm at the same time; S4: Provide proxies: Provide an interface externally for each caller spider to use the application proxy IPs of the corresponding app to crawl target data. The present invention adopts a scheme combining a public proxy IP source and a paid proxy IP source and realizes dynamic control. For the obtained proxy IPs, through a detection algorithm, the validity detection of the proxy IPs is realized and application proxy pools for each app that do not affect each other are generated, making the proxy IPs have high validity and stability.

Description

Technical Field

[0001] The present invention belongs to the field of computers and relates to a method and system for implementing a proxy pool for dynamically controlling the source of proxy IPs. Background Art

[0002] With the continuous development of technologies such as big data, Internet of Things, artificial intelligence, and cloud computing, the core element is data. Data is becoming a breakthrough in technological innovation and is even metaphorically called "new oil". Big data analysis and data mining rely on massive amounts of data. In addition to the data accumulated over time, a large amount of data needs to be collected, gathered, and extracted from the Internet using web crawler technology to extract structured data. Among them, an important problem that web crawler technology needs to solve is proxy IP. Currently, most websites have certain restrictions on web crawlers, such as blocking the same IP that frequently accesses a website. At present, there are mainly two solutions to this problem: The first solution is to actively reduce the access frequency to the same website for the same IP; the second solution is to build a proxy pool to dynamically provide proxy IP services. Among them, the proxy pool technology solution meets the actual needs of actual production web crawler projects.

[0003] Traditional proxy pool solutions usually use proxy IPs from a single source, either public proxy IPs or paid proxy IPs. Among them, the number of sources of single public proxy IPs is unstable, and the validity and timeliness of proxy IPs are also short, affecting the web crawler efficiency. For a single source of paid proxy IPs, in some cases, it cannot proxy all app applications and there is no customized isolation, affecting the use of web crawlers. At the same time, paid proxy IPs are charged, which cannot take into account the enterprise benefits.

[0004] In addition, traditional proxy pool data storage solutions mostly use a single database. If the database service fails or the server crashes, the proxy pool cannot be used.

[0005] In view of the current problems, the present invention provides a method and system for implementing a highly available proxy pool for dynamically controlling the source of proxy IPs. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a method and system for implementing a proxy pool for dynamically controlling the source of proxy IPs.

[0007] To achieve the above purpose, the present invention provides the following technical solutions:

[0008] A method for implementing a proxy pool for dynamically controlling the source of proxy IPs, the method comprising the following steps:

[0009] S1: Obtain proxy IPs: Configure the public proxy website and the paid proxy website, turn on the public proxy IP acquisition switch, regularly obtain the proxy IPs of each public resource at cyclic intervals, assign an initial weight coefficient, and save them to the origin IP proxy pool; when obtaining proxy IPs, filter the origin IP proxy pool and the unavailable IP proxy pool to avoid the same IPs and invalid detections.

[0010] S2: Detect proxy IPs: Regularly detect the origin IP proxy pool at cyclic intervals, including two parts:

[0011] The first part: Use the test address to perform anonymous proxy detection on all proxy IPs in the origin IP proxy pool with a weight coefficient less than 1. If it is a valid anonymous proxy, set the weight coefficient to 1; otherwise, reduce the weight coefficient by 0.1. For the proxy IPs with a weight coefficient of 0 in the origin IP proxy pool, delete them from the origin IP proxy pool and put them into the unavailable IP proxy pool.

[0012] The second part: Use the test address of the crawler target object app application to detect the proxy IPs with a weight coefficient of 1 in the origin IP proxy pool. If the detection of the corresponding app application test address passes, that is, the detection result is available and the response time meets the set threshold, then put the proxy IP into the corresponding app application proxy pool, and the proxy IP in the original origin IP proxy pool remains. If the detection of the corresponding app application test address fails, then delete the proxy IP from the corresponding app application proxy pool.

[0013] If all application test addresses fail to detect this proxy IP, then set the weight coefficient of this proxy IP in the origin IP proxy pool to 0.5.

[0014] S3: Polling and alarming: After obtaining the public proxy IP resources for a period of time, turn on the application proxy polling detection service, scan the number of available proxies in each app application proxy pool. If the minimum proxy IP quantity requirement is not met, then turn on the paid proxy IP acquisition switch, call the proxy acquisition service and the proxy detection service. After the detection is completed, store it in the proxy pool of the corresponding app, and at the same time set the weight coefficient of the proxy IP to 1 and put it into the origin IP proxy pool; when the number of proxy IPs reaches the maximum threshold, turn off the paid proxy IP acquisition switch.

[0015] S4: Provide proxies: Provide an interface externally for each caller spider to use the application proxy IPs of the corresponding app to crawl the target data.

[0016] Optionally, in S3, whether the paid proxy IP acquisition switch is turned on or off, the system maintenance personnel are notified by email.

[0017] Optionally, between S3 and S4, it also includes: regularly cleaning up the useless proxy IPs in the unavailable IP proxy pool, including cleaning up the useless proxy IPs three days ago.

[0018] Optionally, in S4, the Flask framework is used to implement the interface for providing available proxy IPs externally. After starting the service service, the proxy IPs in the corresponding app application proxy pool are provided according to the calling party spider; the random value-taking method is used to provide proxy IPs to ensure that each IP in the corresponding app application proxy pool is evenly used.

[0019] A proxy pool implementation system for dynamically controlling the source of proxy IPs based on the above method, which includes a control module, a proxy acquisition module, a proxy detection module, a storage module, a proxy pool polling module, a proxy cleaning module, and a proxy providing module;

[0020] Control module: Initialize the configuration file, and cyclically call the proxy acquisition module every 20 minutes to obtain public proxy IP resources or paid proxy IP resources; after 10 minutes, cyclically run the proxy detection module; after 1 hour, call the proxy pool polling module; regularly run the proxy cleaning module at midnight every day; start the interface service program to provide the proxy IPs in the corresponding app application proxy pool;

[0021] Proxy acquisition module: Obtain public proxy IP resources and paid proxies through the configuration switch and save them to redis; the proxy detection module performs anonymous proxy validity verification on the proxy IPs obtained by the proxy acquisition module, and then performs target app application test verification. After passing the detection, they are saved to the corresponding app application proxy pools;

[0022] Proxy detection module: Use the test address to perform anonymous validity detection on the proxy addresses in the origin IP proxy pool. For those that pass the verification, set the weight coefficient to 1; then, according to the target app application test address configured in the configuration file, perform availability verification on all proxy IPs with a weight coefficient of 1. If the test passes, save the proxy IP to the corresponding app application proxy pool, otherwise delete it from the corresponding app application proxy pool; if the weight coefficient is less than the set threshold, remove the proxy IP from the origin IP proxy pool and store it in the unavailable IP proxy pool;

[0023] Storage module: Adopt the redis master-slave database architecture, deployed on different servers, to store the IPs of each proxy pool;

[0024] Proxy pool polling module, which scans the number of IPs in each application proxy pool to ensure the effectiveness and usability of the later target crawler spider system and sends email notifications.

[0025] Proxy cleaning module: Deletes data in the unavailable IP proxy pool that is three days old.

[0026] Proxy providing module: Obtains public proxy IPs and paid proxy IPs, filters the origin IP proxy pool and the unavailable IP proxy pool, saves the newly added proxy IPs to the origin IP proxy pool, and sets the initial weight coefficient.

[0027] A computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the method described above is implemented.

[0028] The beneficial effects of the present invention are as follows:

[0029] (1) Adopts a solution that combines public proxy IP sources and paid proxy IP sources and realizes dynamic control.

[0030] (2) Achieves high availability by building a primary and standby Redis database and deploying multiple servers to store the obtained proxy IPs.

[0031] (3) For the obtained proxy IPs, through a detection algorithm, realizes the detection of the effectiveness of proxy IPs and generates individual non-interfering app application proxy pools, making the proxy IPs have high effectiveness and stability.

[0032] Other advantages, objectives, and features of the present invention will be described to some extent in the subsequent specification, and to some extent, will be obvious to those skilled in the art based on the study of the following text, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be achieved and obtained through the following specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in detail preferably with reference to the accompanying drawings, where:

[0034] Figure 1 Is the flowchart of the method of the present invention;

[0035] Figure 2 Is the system diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the drawings provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0037] Among them, the drawings are only for illustrative purposes, showing only schematic diagrams, not physical diagrams, and should not be construed as a limitation to the present invention; in order to better illustrate the embodiments of the present invention, some components in the drawings will be omitted, enlarged or reduced, which does not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0038] In the drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "rear", etc. indicating the orientation or positional relationship, they are based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the positional relationship in the drawings are only for illustrative purposes and should not be construed as a limitation to the present invention. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0039] Such as Figure 1 This embodiment discloses a method for implementing a highly available proxy pool for dynamically controlling the source of proxy IPs, including:

[0040] S1: Obtain proxy IPs: Obtain the proxy source switch according to the configuration file. The initial setting is that the switch for obtaining public proxy IP resources is true, and the switch for paid proxy IPs is false. According to the configured public resource parameter information, use beautifulsoup, PyQuery, and regular expression techniques to parse the web page to obtain proxy IPs in the last three days. Then set the initial permission coefficient to 0.5 and save them to the origin IP proxy pool in the redis database. The data type uses Redis sorted set (sortedset). Execute the proxy acquisition module every 20 minutes. Each time, filter the data in the origin IP proxy pool and the unavailableip proxy pool, and only obtain the incremental part of the data;

[0041] S2: Detect proxy IP: Perform cyclic interval detection on the origin IP proxy pool, which is divided into two parts:

[0042] In the first part of the detection, use the test address (https: / / ident.me) to detect all proxy IPs with a weight coefficient less than 1. If the detection result is an anonymous proxy, set the weight coefficient of this proxy IP to 1; otherwise, reduce the weight coefficient by 0.1. At the same time, judge that if the weight coefficient is 0, delete this proxy IP from the origin IP proxy pool and put it into the unavailable IP proxy pool.

[0043] In the second part of the detection, according to the target app application test address configured in the configuration file, detect all proxy IPs with a weight coefficient equal to 1. Each app application establishes its own available proxy pool. If the detection result of the corresponding app application test address is available and the response time meets the set threshold, put this proxy IP into the corresponding app application proxy pool (such as the app1 IP proxy pool, app2 IP proxy pool,...); otherwise, delete this proxy IP from the corresponding app application proxy pool (if it exists). If all application test addresses fail to pass the detection of this proxy IP, set the weight coefficient of this proxy IP in the origin IP proxy pool to 0.5.

[0044] S3: Polling and alarming: Perform timed polling scans on each app application proxy pool. If it is detected that the number of IPs in a certain app application proxy pool is less than the set minimum threshold, dynamically set the switch to turn on the acquisition of paid proxy IPs, call the proxy service, set the weight coefficient of the obtained paid proxy IPs to 1, and then perform the detection in the second part of the proxy service; when the number of proxy IPs in the proxy pool reaches the set maximum threshold, dynamically turn off the switch to acquire paid proxy IPs; all operations of dynamically turning on or off the switch will be notified by email;

[0045] S4: Provide proxy IP: After the above steps, each valid and stable app application proxy pool IP will be obtained. Use the Flask framework to implement an interface for providing available proxy IPs externally. After starting the service, provide the proxy IP in the corresponding app application proxy pool according to the parameters passed by the caller (spider); use the random selection method to provide proxy IPs, ensuring that each IP in the corresponding app application proxy pool is evenly used while maintaining the disorder of being called;

[0046] Such as Figure 2As shown in the figure, the construction of a highly available proxy pool system for dynamically controlling the source of proxy IPs in this implementation is based on the implementation method of a highly available proxy pool for dynamically controlling the source of proxy IPs. This system consists of a control module, a proxy acquisition module, a proxy detection module, a storage module, a proxy pool polling module, a proxy cleaning module, a proxy providing module, etc. It adopts a data storage solution using a Redis master-slave database, uses PyQuery and regular expression technologies to parse web pages to obtain proxy IPs, uses an asynchronous method to detect the anonymity and validity of proxies, uses a polling method to detect the order of magnitude of each application proxy pool, and adopts a Flask framework to provide an interface for available proxy IPs externally. The details are as follows:

[0047] Control module: After the system starts, it initializes the configuration file, and calls the proxy acquisition module in a loop every 20 minutes to obtain public proxy IP resources or paid proxy IP resources (specifically according to the configuration). After 10 minutes, it runs the proxy detection module in a loop. After 1 hour, it calls the proxy pool polling module. At midnight every day, it runs the proxy cleaning module regularly to delete the data in the unavailable IP proxy pool that is three days old; it starts the interface service program to provide the proxy IPs in the corresponding app application proxy pool;

[0048] Storage module: Adopts a Redis master-slave database architecture, deployed on different servers, to store the IPs of each proxy pool to ensure the high availability of the system;

[0049] Proxy acquisition module: According to the switch in the configuration file, it obtains public proxy IPs and paid proxy IPs, filters the origin IP proxy pool and the unavailable IP proxy pool, saves the newly added proxy IPs to the origin IP proxy pool, and sets the initial weight coefficient;

[0050] Proxy detection module: Uses the test address (https: / / ident.me) to detect the anonymity and validity of the proxy addresses in the origin IP proxy pool. For those that pass the verification, the weight coefficient is set to 1. Then, according to the test addresses of the target app applications configured in the configuration file, it verifies the availability of all proxy IPs with a weight coefficient of 1. If the test passes, it saves the proxy IP to the corresponding app application proxy pool; otherwise, it deletes it from the corresponding app application proxy pool. If the weight coefficient is less than the set threshold, the proxy IP is removed from the origin IP proxy pool and stored in the unavailable IP proxy pool;

[0051] Proxy Pool Polling Module: Periodically poll and scan the IPs in each application proxy pool. If the number of IPs in the application proxy pool is less than the set minimum threshold, turn on the switch to obtain paid proxy IPs, call the proxy acquisition module and the proxy detection module, and then save the proxy IPs to the origin IP proxy pool and the corresponding app application proxy pool;

[0052] Proxy Cleaning Module: Every day at midnight, periodically clean the data in the unavailable IP proxy pool that is three days old;

[0053] Proxy Provision Module: Use the Flask framework to externally provide an available proxy IP interface, start the service, and provide the IPs in the corresponding app application proxy pool according to the parameters passed by the caller (spider);

[0054] 1. At the beginning, the control module obtains the public proxy IP resources according to the configuration file. After a period of time, it polls and detects the number of each app application proxy pool. If it meets the set minimum threshold, do not turn on the switch to obtain paid proxy IPs; otherwise, turn on the switch to obtain paid proxy IP resources to achieve dynamic control of the proxy IP source.

[0055] 2. By building a primary and standby Redis database and deploying multiple servers, the system accesses the primary Redis. Assuming that the primary Redis service cannot be accessed or the server where the primary Redis is located crashes, the database will automatically switch to the standby Redis database to provide normal services, realizing the high availability of the proxy pool.

[0056] 3. Through a loop of periodic detection algorithms, for the obtained proxy IP resources, first conduct the first round of anonymous validity detection, and assign the highest weight coefficient to the valid proxy IPs. Then conduct the second round of target crawler object app application test address detection. Use the valid anonymous proxy IP to access the target test address. If the access is successful, put it into the corresponding app application proxy pool. If the access fails, delete it from the corresponding app proxy pool, thus realizing the isolation of each app application proxy pool from each other and not affecting each other.

[0057] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The methods can be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - in accordance with the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Additionally, for this purpose the program is capable of running on a programmed application-specific integrated circuit.

[0058] Furthermore, the operations of the processes described herein can be performed in any suitable order, unless otherwise indicated herein or otherwise clearly contradicted by the context. The processes described herein (or variations and / or combinations thereof) can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or by a combination thereof. The computer programs include a plurality of instructions executable by one or more processors.

[0059] Further, the methods can be implemented in any type of computing platform operably connected, including but not limited to personal computers, minicomputers, mainframes, workstations, network or distributed computing environments, standalone or integrated computer platforms, or communicating with charged particle tools or other imaging devices, etc. Aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into the computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer and, when read by the computer, can be used to configure and operate the computer to perform the processes described herein. Additionally, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media include instructions or programs that implement the above-described steps in conjunction with a microprocessor or other data processor, the inventions described herein include these and other different types of non-transitory computer-readable storage media. When programmed according to the method and technology for implementing a proxy pool for dynamically controlling the source of proxy IPs according to the present invention, the present invention also includes the computer itself.

[0060] A computer program can be applied to input data to perform the functions described herein, thereby transforming the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the transformed data represents physical and tangible objects, including a specific visual depiction of the physical and tangible objects generated on the display.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the present technical solution, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A method for implementing a proxy pool for dynamically controlling the source of proxy IPs, characterized in that: The method includes the following steps: S1: Obtain proxy IPs: Configure the public proxy website and the paid proxy website, turn on the public proxy IP acquisition switch, periodically obtain the proxy IPs of each public resource at regular intervals in a loop, assign an initial weight coefficient, and save them to the origin IP proxy pool; when obtaining proxy IPs, the origin IP proxy pool and the unavailable IP proxy pool will be filtered to avoid the same IP and invalid detection; S2: Detect proxy IPs: Periodically detect the origin IP proxy pool in a loop, which includes two parts: The first part: Use the test address to perform anonymous proxy detection on all proxy IPs in the origin IP proxy pool with a weight coefficient less than 1. If it is a valid anonymous proxy, the weight coefficient is set to 1, otherwise the weight coefficient is reduced by 0.1; for the proxy IPs with a weight coefficient of 0 in the origin IP proxy pool, they are deleted from the origin IP proxy pool and put into the unavailable IP proxy pool; The second part: Use the crawler target object app application test address to detect the proxy IPs with a weight coefficient of 1 in the origin IP proxy pool. If the corresponding app application test address passes the detection, that is, the detection result is available and the response time meets the set threshold, then the proxy IP is put into the corresponding app application proxy pool, and the proxy IP in the original origin IP proxy pool is retained. If the corresponding app application test address fails the detection, then the proxy IP is deleted from the corresponding app application proxy pool; If all application test addresses fail to pass the detection of this proxy IP, then the weight coefficient of this proxy IP in the origin IP proxy pool is set to 0.5; S3: Polling and alarming: After obtaining the public proxy IP resources for a period of time, turn on the application proxy polling detection service, scan the number of available proxies in each app application proxy pool. If the minimum proxy IP number requirement is not met, turn on the paid proxy IP acquisition switch, call the proxy acquisition service and the proxy detection service, and after the detection is completed, store them in the proxy pool of the corresponding app, and at the same time set the weight coefficient of the proxy IP to 1 and put it into the origin IP proxy pool; when the number of proxy IPs reaches the maximum threshold, turn off the paid proxy IP acquisition switch; S4: Provide proxies: Provide an interface externally for each caller spider to use the application proxy IP of the corresponding app to crawl the target data.

2. The proxy pool implementation method for dynamically controlling the source of proxy IPs according to claim 1, characterized in that: In S3, whether to turn on or off the paid proxy IP acquisition switch is notified to the system maintainer by email.

3. A proxy pool implementation method for dynamically controlling the source of proxy IPs according to claim 1, characterized in that: Between S3 and S4, it also includes: periodically cleaning up the useless proxy IPs in the unavailable IP proxy pool, including cleaning up the useless proxy IPs three days ago.

4. The proxy pool implementation method for dynamically controlling the source of proxy IPs according to claim 1, characterized in that: In S4, the Flask framework is used to implement an interface for providing available proxy IPs externally. After starting the service service, the proxy IPs in the corresponding app application proxy pool are provided according to the calling party spider; the random selection method is used to provide proxy IPs to ensure that each IP in the corresponding app application proxy pool is evenly used.

5. A proxy pool implementation system for dynamically controlling the source of proxy IPs based on the method according to any one of claims 1 to 4, characterized in that: The system includes a control module, a proxy acquisition module, a proxy detection module, a storage module, a proxy pool polling module, a proxy cleaning module, and a proxy providing module; Control module: Initialize the configuration file, cyclically call the proxy acquisition module every 20 minutes to obtain public proxy IP resources or paid proxy IP resources; after 10 minutes, cyclically run the proxy detection module; after 1 hour, call the proxy pool polling module; run the proxy cleaning module regularly at midnight every day; start the interface service program to provide the proxy IPs in the corresponding app application proxy pool; Proxy acquisition module: Obtain public proxy IP resources and paid proxies through the configuration switch and save them in redis; the proxy detection module performs anonymity proxy validity verification on the proxy IPs obtained by the proxy acquisition module, and after passing, perform target app application test verification, and save them in the corresponding app application proxy pools after passing the detection; Proxy detection module: Use the test address to perform anonymity validity detection on the proxy addresses in the origin ip proxy pool. For those that pass the verification, set the weight coefficient to 1; then, according to the target app application test address configured in the configuration file, perform availability verification on all proxy IPs with a weight coefficient of 1. If the test passes, save the proxy IP in the corresponding app application proxy pool, otherwise delete it from the corresponding app application proxy pool; if the weight coefficient is less than the set threshold, remove the proxy IP from the origin ip proxy pool and store it in the unavailable ip proxy pool; Storage module: Adopt a redis master-slave database architecture, deployed on different servers, to store the IPs of each proxy pool; Proxy pool polling module, scan the number of IPs in each application proxy pool to ensure the effectiveness and use of the later target crawler spider system, and send email notifications; Proxy cleaning module: Delete the data in the unavailable ip proxy pool three days ago; Proxy providing module: Obtain public proxy IPs and paid proxy IPs, filter the origin ip proxy pool and the unavailable ip proxy pool, save the newly added proxy IPs in the origin ip proxy pool, and set the initial weight coefficient.

6. A computer-readable storage medium storing a computer program thereon, characterized in that: When the computer program is executed by the processor, it implements the method described in any one of claims 1-4.

Images