Management scheduling method and device for proxy IP pool, and storage medium
By building a scoring matrix of proxy IP and target address and using alternating least squares method optimization, the comprehensive scoring of proxy IP is dynamically evaluated, which solves the problem of unstable maintenance of proxy IP pools and improves service quality and access efficiency.
Patent Information
- Application Number
- CN202510844669.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-23
- Publication Date
- 2025-08-12
AI Technical Summary
The existing proxy IP pool maintenance lacks stability and efficiency, and user requests are difficult to be processed through the fastest proxy IP, resulting in insufficient user experience and service efficiency.
By constructing a scoring matrix between the proxy IP and the target address, the optimization objective function is solved by using the alternating least squares method, the comprehensive score of the proxy IP is dynamically evaluated, and scheduled based on the score.
Improve the service quality and reliability of the proxy IP pool, improve access efficiency and ability to prevent IP bans.
Smart Images
Figure CN120475015A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of proxy IP management, and in particular to a management and scheduling method, device, and storage medium for a proxy IP pool. Background Art
[0002] Proxy IP technology is an important research area in internet applications. A proxy IP pool collects and maintains a large collection of available proxy IP addresses for user use. This pool forwards user requests through an intermediary server to hide the true IP address, thereby achieving privacy protection, network acceleration, and access control. Unlike directly accessing the target website, using a proxy IP disperses access traffic, preventing single IP addresses from being blocked, and improving access success rates and efficiency. By collecting multiple free proxy IP addresses and accessing and evaluating target websites based on these IP addresses, proxy IP technology can screen high-quality proxy IPs and store them in a proxy IP pool. When using the proxy IP pool for data crawling or access requests, applications can select the appropriate proxy IP based on the target website's requirements, improving access efficiency and reducing the risk of being blocked.
[0003] Compared to traditional direct access methods, using a proxy IP pool can effectively reduce request failure rates and increase data retrieval speed. Automated management of the proxy IP pool reduces manual maintenance costs, and through regular updates and evaluations, it ensures the high quality and availability of proxy IPs. With the growing demand for internet applications, proxy IP technology holds broad application prospects in improving service quality and user satisfaction. Furthermore, proxy IP technology can also be used to enhance network security and privacy protection. By hiding the user's true IP address, proxy IPs can prevent hacker attacks and privacy leaks. In areas such as cross-border e-commerce and video streaming, proxy IP technology helps users bypass geographical restrictions and access more content. In short, proxy IP technology offers significant advantages in data scraping, network security, privacy protection, and content unlocking. With the continuous development and improvement of technology, proxy IP pool management methods will become more efficient and their application scope will further expand, becoming a key supporting technology in internet applications.
[0004] However, the existing proxy IPs still have the following defects: the maintenance of traditional proxy IP pools lacks stability and efficiency, user requests are difficult to be processed through the fastest proxy IP, and user experience and service efficiency are obviously insufficient. Summary of the Invention
[0005] The present application provides a method, device and storage medium for managing and scheduling a proxy IP pool to at least solve one technical problem existing in the prior art.
[0006] According to a first aspect of the present application, a method for managing and scheduling a proxy IP pool is provided, comprising the following steps: S1, constructs a proxy IP to target address scoring matrix based on the historical response time between the proxy IP and the target address; S2, approximating the scoring matrix as the product of the proxy IP feature matrix and the target address feature matrix; S3, introduces a regularization term based on the square loss function and defines the optimization objective function; S4, solving the optimization objective function using alternating least squares method; S5, accumulate and normalize the scores of each proxy IP for all target addresses to obtain the comprehensive score of the corresponding proxy IP; S6, scheduling the proxy IPs in descending order of the comprehensive scores.
[0007] In certain embodiments of the first aspect of the present application, the specific content of S1 is as follows: Construct the proxy IP set as , the target address set is ; Statistics on the historical response time when the proxy IP requests the target address within the specified time; All historical response times are divided into several time periods from shortest to longest, and a corresponding score is set for each time period. If the response time of the proxy IP requesting the target address falls into a certain time period, the score of that time period is assigned to the proxy IP's score for the target address. Constructing a scoring matrix , the scoring matrix Elements in Representative Proxy IP To the target address 's rating.
[0008] In certain embodiments of the first aspect of the present application, the specific content of S2 is as follows: Assume that the latent factor dimension is , the proxy IP feature matrix is , the target address feature matrix is ; Rating Matrix Approximate expression is:
[0009] Among them, the proxy IP feature matrix No. OK Indicates proxy IP Potential features of target address feature matrix No. OK Indicates the target address potential characteristics.
[0010] In certain embodiments of the first aspect of the present application, the specific content of S3 is as follows:
[0011] in, represents the optimization goal, represents the set of indices with known ratings, is the regularization parameter.
[0012] In certain embodiments of the first aspect of the present application, the specific content of S4 is as follows: S41, first fix the target address feature matrix , the proxy IP feature matrix is As a variable, for each Find the partial derivative and we get:
[0013]
[0014] Gradient descent iteration:
[0015] S42, then, the fixed proxy IP feature matrix is , the target address feature matrix As a variable, for each Find the partial derivative and perform gradient descent iteration to get:
[0016] S43, repeat S41 and S42 until the iterative convergence condition is reached.
[0017] In certain embodiments of the first aspect of the present application, the iterative convergence condition is to satisfy one of the following conditions: and No longer changes, or the maximum number of iterations is reached.
[0018] In certain embodiments of the first aspect of the present application, the specific content of S5 is as follows:
[0019] in, Representative Proxy IP The comprehensive rating of represents the latent factor dimension, Represents the maximum value of the rating.
[0020] According to a second aspect of the present application, an electronic device is provided, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in this application.
[0021] According to a third aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the present application.
[0022] Compared with the prior art, this application has the following beneficial effects: This application dynamically scores proxy IPs based on the historical response time between the proxy IP and the target address to measure the availability of the current proxy IP in the IP pool resources, thereby determining the scheduling priority of the proxy IP to improve the overall service quality and reliability.
[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The above and other objects, features and advantages of the exemplary embodiments of the present application will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, in which: In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.
[0025] Figure 1 A flow chart of the management and scheduling method of embodiment 1 of the present application is shown.
[0026] Figure 2 The figure shows the scoring diagram of the proxy IP of this application corresponding to different target addresses or heartbeats.
[0027] Figure 3 A schematic diagram of the structure of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0028] In order to make the purpose, features, and advantages of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.
[0029] Example 1: This embodiment provides a method for managing and scheduling a proxy IP pool. From the perspective of overall proxy forwarding requests, selecting a proxy IP pool with fast response time to improve access efficiency is crucial. Therefore, the response time of requests forwarded by different proxy IPs can be described as minimizing the request time.
[0030] For details, please refer to Figure 1 , the management and scheduling method of the proxy IP pool includes the following steps: S1, constructs a proxy IP to target address scoring matrix based on the historical response time between the proxy IP and the target address.
[0031] Specifically, the proxy IP set is constructed as , the target address set is ; Devices connected to the platform report heartbeat data every minute. The system simultaneously collects this data and the response time when interacting with the device. Devices with stable heartbeat reports in the recent period are selected and their corresponding IP addresses are added to the candidate proxy IP pool.
[0032] Statistics on the historical response time when the proxy IP requests the target address within the specified time; All historical response times are divided into several time periods from small to large, and a corresponding score is set for each time period; if the response time of the proxy IP requesting the target address falls into a certain time period, the score of that time period is assigned to the proxy IP's score for the target address.
[0033] Constructing a scoring matrix , the scoring matrix Elements in Representative Proxy IP To the target address 's rating.
[0034] like Figure 2 As shown, as an embodiment, the response time is divided into five equal segments, with a maximum score of 5 points. Each time segment is assigned a score from high to low and a matrix is established. Finally, a score matrix between the proxy IP and the target address is constructed to represent the request performance score of each proxy IP for different target addresses.
[0035] S2. Approximately express the scoring matrix as the product of the proxy IP feature matrix and the target address feature matrix.
[0036] If the response score of an IP to an unrequested target address can be inferred based on the IP's historical response score, then a suitable proxy IP can be assigned to the target request address based on the prediction result.
[0037] Assume that the latent factor dimension is , the proxy IP feature matrix is , the target address feature matrix is , hereinafter referred to as P and Q respectively; Rating Matrix Approximate expression is:
[0038] Among them, the proxy IP feature matrix No. OK Indicates proxy IP Potential features of target address feature matrix No. OK Indicates the target address potential characteristics.
[0039] S3, introduces a regularization term based on the square loss function and defines the optimization objective function.
[0040] In order to more accurately fit the response score of the proxy IP to the target address and reduce the error from the user experience perspective, we use the square loss function as the optimization target and introduce the regularization term To prevent overfitting, the optimization objective function is as follows:
[0041] in, represents the optimization goal, represents the set of indices with known ratings, is the regularization parameter.
[0042] S4, solving the optimization objective function using alternating least squares method; The matrix factorization problem is converted into a standard optimization problem, the goal is to solve and In order to minimize the optimization objective function, the alternating least squares method is used to solve this problem. The basic idea of the alternating least squares method is: since the matrix and are all unknown and coupled to each other through matrix multiplication. To decouple them, first fix ,Will As a variable, we can find the value by minimizing the loss function. , which is equivalent to a classic least squares problem. Then fix the obtained ,Will Solving for variables .
[0043] The specific steps include: S41, first fix the target address feature matrix , the proxy IP feature matrix is As a variable, for each Find the partial derivative and we get:
[0044]
[0045] Gradient descent iteration:
[0046] S42, then, the fixed proxy IP feature matrix is , the target address feature matrix As a variable, for each Find the partial derivative and perform gradient descent iteration to get:
[0047] S43, repeat S41 and S42 until the iterative convergence condition is reached.
[0048] Preferably, the iterative convergence condition is to satisfy one of the following conditions: and It becomes stable and no longer changes, or reaches the preset maximum number of iterations.
[0049] S5, accumulate and normalize the scores of each proxy IP for all target addresses to obtain the comprehensive score of the corresponding proxy IP.
[0050]
[0051] in, Representative Proxy IP The comprehensive rating of represents the latent factor dimension, Represents the maximum value of the rating.
[0052] S6, scheduling the proxy IPs in descending order of the comprehensive scores.
[0053] Proxy IP The higher the comprehensive score, the shorter the request time and the higher the access efficiency, so it will be scheduled first.
[0054] This solution obtains the score of the proxy IP and records the target address of the proxy IP request in the business system. For target addresses frequently requested by the same IP, the proxy IP issuing the request is promptly replaced. For target addresses frequently accessed by different IP addresses with the same user information, the proxy IP in the history is dynamically matched and requests from the same proxy IP are made to effectively prevent the target address from being blocked.
[0055] Example 2: According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.
[0056] Figure 3 A schematic block diagram of an example electronic device that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0057] like Figure 3 As shown, the device includes a computing unit that can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). The RAM can also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. An input / output (I / O) interface is also connected to the bus.
[0058] Many components in a device are connected to the I / O interface, including: input units, such as a keyboard and mouse; output units, such as various types of displays and speakers; storage units, such as magnetic disks and optical disks; and communication units, such as network cards, modems, and wireless communication transceivers. The communication unit allows the device to exchange information / data with other devices via computer networks such as the Internet and / or various telecommunication networks.
[0059] The computing unit can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of computing units include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processors, controllers, microcontrollers, etc. The computing unit executes the various methods and processes described above, such as the proxy IP pool management and scheduling method described in Example 1. For example, in some embodiments, the proxy IP pool management and scheduling method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed onto a device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the computing unit, one or more steps of the proxy IP pool management and scheduling method described above can be performed. Alternatively, in other embodiments, the computing unit can be configured to execute the proxy IP pool management and scheduling method via any other suitable means (e.g., via firmware).
[0060] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0061] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0062] In the context of this application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0063] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0064] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.
[0065] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.
[0066] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of this application can be achieved. This is not limited herein.
[0067] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.
[0068] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.
Claims
1. A method for managing and scheduling a proxy IP pool, characterized in that: The following steps are involved: S1, constructs a proxy IP to target address scoring matrix based on the historical response time between the proxy IP and the target address; S2, approximating the scoring matrix as the product of the proxy IP feature matrix and the target address feature matrix; S3, introduces a regularization term based on the square loss function and defines the optimization objective function; S4, solving the optimization objective function using alternating least squares method; S5, accumulate and normalize the scores of each proxy IP for all target addresses to obtain the comprehensive score of the corresponding proxy IP; S6, scheduling the proxy IPs in descending order of the comprehensive scores.
2. A method for managing and scheduling a proxy IP pool according to claim 1, characterized in that: The specific content of S1 is as follows: Construct the proxy IP set as , the target address set is ; Statistics on the historical response time when the proxy IP requests the target address within the specified time; All historical response times are divided into several time periods from shortest to longest, and a corresponding score is set for each time period. If the response time of the proxy IP requesting the target address falls into a certain time period, the score of that time period is assigned to the proxy IP's score for the target address. Constructing a scoring matrix , the scoring matrix Elements in Representative Proxy IP To the target address 's rating.
3. A method for managing and scheduling a proxy IP pool according to claim 2, characterized in that: The specific content of S2 is as follows: Assume that the latent factor dimension is , the proxy IP feature matrix is , the target address feature matrix is ; Rating Matrix Approximate expression is: Among them, the proxy IP feature matrix No. OK Indicates proxy IP Potential features of target address feature matrix No. OK Indicates the target address potential characteristics.
4. A method for managing and scheduling a proxy IP pool according to claim 3, characterized in that: The specific content of S3 is as follows: in, represents the optimization goal, represents the set of indices with known ratings, is the regularization parameter.
5. A method for managing and scheduling a proxy IP pool according to claim 4, characterized in that: The specific content of S4 is as follows: S41, first fix the target address feature matrix , the proxy IP feature matrix is As a variable, for each Find the partial derivative and we get: Gradient descent iteration: S42, then, the fixed proxy IP feature matrix is , the target address feature matrix As a variable, for each Find the partial derivative and perform gradient descent iteration to get: S43, repeat S41 and S42 until the iterative convergence condition is reached.
6. A method for managing and scheduling a proxy IP pool according to claim 5, characterized in that: The iterative convergence condition is to meet one of the following conditions: and No longer changes, or the maximum number of iterations is reached.
7. A method for managing and scheduling a proxy IP pool according to claim 6, characterized in that: The specific content of S5 is as follows: in, Representative Proxy IP The comprehensive rating of represents the latent factor dimension, Represents the maximum value of the rating.
8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.
9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.