Proxy IP identification and management method and system based on URL resource detection, and non-transitory computer readable storage medium

Abstract

The application discloses a proxy IP identification and management method and system based on URL resource detection and a non-transient computer readable storage medium, and comprises the following steps: S1, acquiring URL resource connections provided by each proxy manufacturer, and adding the URL resource connections to a to-be-detected queue; S2, taking out a URL resource connection from the to-be-detected queue, and initiating a connection request to a preset target detection server; S3, identifying and extracting a final egress IP address; S4, performing availability detection on a proxy resource corresponding to a URL resource of which the final egress IP address has been identified; S5, performing URL resource state management according to the availability determination result; S6, performing IP re-detection and availability re-detection on inventory URL resources; and S7, providing a proxy IP resource query service based on a resource query interface. The application has the advantages that proxy IP transparency can be realized, resource controllability can be improved, business continuity can be ensured, automatic full-life-cycle management can be realized, and data real-time performance can be maintained.

Description

Technical Field

[0001] This invention belongs to the field of network proxy and IP resource management technology, specifically relating to a method, system and non-transitory computer-readable storage medium for proxy IP identification and management based on URL resource detection. Background Technology

[0002] With the distributed development of internet services, enterprises are increasingly demanding proxy IP resources in scenarios such as cross-regional business monitoring, public information collection, multi-node API calls, and network performance testing. Currently, proxy service providers in the market generally deliver proxy services to users in the form of URL resource links, allowing users to utilize proxy forwarding capabilities by calling these URL resources.

[0003] However, existing proxy service models have the following technical problems in enterprise applications: IP address not visible: The proxy provider only provides the URL resource entry point, and users cannot directly know the actual exit IP address behind the URL. The black box delivery model makes it impossible for users to perform fine-grained management and scheduling of proxy IPs.

[0004] Dynamic IP address changes: The exit IP behind the URL resource is dynamically adjusted by the vendor, and users cannot predict the change nodes and the scope of the change, which can easily lead to problems such as the failure of IP whitelists and the confusion of routing policies in the business link.

[0005] Unknown resource availability: Users cannot verify the connectivity and validity of URL resources in advance. They can only discover faults such as authentication failure, connection timeout, and service unavailability when making business calls, which affects business continuity.

[0006] Low resource management efficiency: URL resources from multiple vendors and in multiple batches are managed in a decentralized manner, lacking a unified IP dimension identifier and status ledger, making it difficult to achieve resource classification, filtering, statistics and full lifecycle management.

[0007] Insufficient precision in business scheduling: For business scenarios that require policy routing and node allocation based on IP addresses, the inability to know the specific IP address will result in insufficient scheduling granularity, making it difficult to achieve resource pooling and elastic scheduling based on IP addresses.

[0008] Therefore, there is an urgent need for a technical solution that can automatically detect the actual exit IP behind a URL resource, simultaneously complete availability verification, and achieve unified resource management to solve the above-mentioned problems of existing technologies. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a proxy IP identification and management method, system, and non-transitory computer-readable storage medium based on URL resource detection. By automatically detecting the final exit IP address corresponding to the proxy vendor's URL resource, combined with multi-dimensional availability detection and full lifecycle status management, a unified proxy IP resource database is constructed and standardized query services are provided externally. Furthermore, transparent management of proprietary proxy resources is achieved within the scope of user authorization, and compliance detection principles are followed throughout the process. This solves the technical problems of invisible proxy IPs, uncontrollable dynamics, inability to pre-verify availability, and low resource management efficiency in existing technologies.

[0010] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A method for identifying and managing proxy IPs based on URL resource detection includes the following steps: S1: Obtain the URL resource links and corresponding agent information provided by each agent, and add the URL resource links to the queue to be probed; S2: Take a URL resource connection from the queue to be probed, and send a connection request to the preset target probe server that is used to provide feedback on the source IP of the connection through the URL resource connection; S3: Based on the network connection information fed back by the target detection server, identify and extract the final exit IP address corresponding to the URL resource connection; S4: Perform availability testing on the proxy resources corresponding to the URL resources whose final exit IP addresses have been identified, and obtain the resource availability judgment results and performance index data; S5: Perform URL resource status management based on the availability determination result: If the detection passes, store the URL resource connection, the corresponding final exit IP address, the proxy vendor, the URL resource status and performance index data into the proxy IP resource database, mark it as available and include it in the available resource pool; if the detection fails, accumulate the number of consecutive failures, and when the number of consecutive failures reaches a preset threshold, mark the URL resource as invalid and remove it from the available resource pool. S6: Perform IP re-probing and availability re-detection on the existing URL resources in the proxy IP resource database according to a preset period, and dynamically update the final exit IP address and URL resource status corresponding to the URL resource; S7: Construct a resource query interface based on the proxy IP resource database, and provide proxy IP resource query services to external applications through the resource query interface.

[0011] In one embodiment, identifying the final exit IP address in step S3 specifically includes: The network connection information at the corresponding layer is obtained by using three methods in sequence: HTTP request probing, TCP connection probing, and DNS resolution probing. Among them, HTTP request probing is used to obtain the application layer response and proxy chain field, TCP connection probing is used to obtain the actual peer IP of the transport layer, and DNS resolution probing is used to obtain the IP of the entry domain name resolution. The results are fused according to the rule that the actual observed source IP of the target end has a higher priority than the HTTP proxy chain field, and the HTTP proxy chain field has a higher priority than the DNS resolution entry IP. If two or more valid detection results point to the same public IP address within the same detection period, that IP address is determined to be the final exit IP address corresponding to the current URL resource connection.

[0012] In one embodiment, the availability detection in step S4 includes connectivity detection, authentication detection, stability detection, and performance detection; The detection process uses either a token bucket or leaky bucket algorithm, and performs rate limiting control based on the dimensions of the agent, the certified account, the URL resource connection, and the target detection server. When a detection fails, the next detection interval is extended according to the exponential backoff strategy. When the number of authentication failure errors reaches a preset number, the process is transferred to manual verification. For network jitter errors, a low-frequency retest strategy is adopted.

[0013] In one embodiment, the final exit IP address identified in step S3 is, in a multi-layered cascaded proxy scenario, the nearest one-hop proxy exit IP observed by the target probing server. In a cascaded link, upstream agent nodes, vendor scheduling nodes, and ingress IPs are stored only as supplementary information and do not participate in the primary key deduplication and whitelisting of available egress IPs.

[0014] In one embodiment, the proxy authentication information corresponding to the URL resource used to initiate the connection request and availability detection is stored using an envelope encryption method: a single proxy authentication information is encrypted with authentication using a data encryption key, and then the data encryption key is encrypted using a master key; The proxy authentication information is only briefly decrypted in memory when probing requests and availability checks are initiated, and the credential information is anonymized in log records and interface responses.

[0015] This invention also discloses a proxy IP identification and management system based on URL resource detection, comprising: The URL resource acquisition module is used to acquire URL resource links and corresponding agent information provided by each agent, and add the URL resource links to the probe queue. The IP address detection module is used to retrieve a URL resource connection from the queue to be detected, initiate a connection request to a preset target detection server that provides feedback on the source IP of the connection through the URL resource connection, and identify and extract the final exit IP address corresponding to the URL resource connection based on the network connection information fed back by the target detection server. The resource availability detection module is used to perform availability detection on the proxy resources corresponding to the URL resources whose final exit IP address has been identified, and to obtain the resource availability judgment results and performance index data. The resource status management module is used to perform URL resource status management based on the availability determination result: If the detection passes, the URL resource connection, the corresponding final exit IP address, the proxy vendor, the URL resource status and performance index data are stored in the proxy IP resource database, and marked as available and included in the available resource pool; if the detection fails, the number of consecutive failures is accumulated, and when the number of consecutive failures reaches a preset threshold, the URL resource is marked as invalid and removed from the available resource pool. The periodic inspection module is used to perform IP re-probing and availability re-detection on the existing URL resources in the proxy IP resource database according to a preset cycle, and dynamically update the final exit IP address and URL resource status corresponding to the URL resource. The resource query interface module is used to build a resource query interface based on the proxy IP resource database, and provide proxy IP resource query services to external applications through the resource query interface.

[0016] In one embodiment, the IP address detection module integrates an HTTP request detection unit, a TCP connection detection unit, and a DNS resolution detection unit. It performs detection in the order of HTTP request detection priority, TCP connection detection verification, and DNS resolution detection reference, and performs multi-source result fusion and confidence marking according to the rule of prioritizing the actual IP of the target end.

[0017] In one embodiment, the resource availability detection module integrates a detection execution unit, a rate limiting scheduling unit, and an anomaly avoidance unit; The detection execution unit is used to perform connectivity detection, authentication detection, stability detection, and performance detection. The rate limiting scheduling unit uses the token bucket algorithm to control the probe request rate in multiple dimensions. The abnormal backoff unit performs hierarchical index backoff for different failure types and adopts different retry strategies to distinguish between authentication failure and network jitter scenarios.

[0018] In one embodiment, the system also includes an authentication information encryption storage module that uses envelope encryption to store proxy authentication credentials, supports key rotation and audit log recording, and performs on-demand decryption and desensitization output when probed.

[0019] The present invention also discloses a non-transitory computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described proxy IP identification and management method based on URL resource detection.

[0020] By adopting the above technical solution, the present invention has the following beneficial effects: (1) This invention identifies the final exit IP address corresponding to the URL resource through the actual test of the target detection server, breaks the black box mode of the existing proxy service, and enables users to clearly grasp the actual exit IP corresponding to each URL resource, which can provide a data foundation for fine-grained scheduling. (2) By constructing a unified proxy IP resource database, users can classify, filter and manage resources according to dimensions such as IP address, vendor, and availability status, thereby achieving precise control and pooled scheduling of proxy IP resources. (3) This invention can detect and remove failed resources in advance through a multi-dimensional availability pre-detection and automatic failure cleanup mechanism, ensuring that the resources in the available resource pool remain effective and reducing the probability of failure in business operations; (4) The resource acquisition, IP detection, availability detection, status update and periodic inspection of the present invention are all executed automatically without manual intervention, which can significantly reduce operation and maintenance management costs and improve resource management efficiency; (5) By providing accurate exit IP address information, this invention enables business systems to implement advanced strategies such as IP whitelist matching, regional scheduling, and IP deduplication, thereby meeting the needs of distributed services for precise IP control. (6) This invention supports simultaneous access to resources from multiple agent vendors, with unified format, unified storage, and unified query, which can provide data support for vendor selection, resource price comparison, and flexible scheduling. (7) The present invention dynamically updates IP addresses and resource status through a regular inspection mechanism, which can promptly detect IP changes and service fluctuations on the vendor side, and ensure the accuracy and timeliness of resource database information; In summary, this invention has the advantages of enabling transparent proxy IPs, improving resource controllability, ensuring business continuity, achieving automated full lifecycle management, supporting fine-grained business scheduling, being compatible with multiple vendors for unified management, and maintaining data real-time performance. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the proxy IP identification and management method based on URL resource detection of the present invention; Figure 2 This is a structural block diagram of the proxy IP identification and management system based on URL resource detection of the present invention; Among them, there are URL resource acquisition module 101, IP address detection module 102, resource availability detection module 103, resource status management module 104, periodic inspection module 105, resource query interface module 106, and authentication information encrypted storage module 107. Detailed Implementation

[0022] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0023] The components of the embodiments of the invention described and shown in the accompanying drawings can typically be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention.

[0024] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0025] Example 1 In this embodiment, a method, system, and non-transitory computer-readable storage medium for proxy IP identification and management based on URL resource detection are proposed. This method can be applied to scenarios such as compliant public data collection, business availability monitoring, distributed API calls, network security testing, and network performance optimization.

[0026] It is understood that this invention manages proxy URL resources that users have legally authorized. All probing operations are conducted within the scope permitted by the service agreement, aiming to improve the operational efficiency of proprietary resources. It does not involve any unauthorized resource acquisition or system intrusion. This invention achieves transparent management of proprietary proxy resources within the scope of user authorization, and follows the principle of compliant probing throughout the process. Moreover, all probing targets in this embodiment are proxy URL resources that users have the right to use. The probing behavior complies with the vendor's service terms and network security specifications. In this invention, the final exit IP refers to the public IP address of the closest hop to the target server in the proxy link. Primary key deduplication refers to resource deduplication management based on the exit IP address as the primary key. Whitelist determination refers to matching the access whitelist rules of the business system based on the exit IP address.

[0027] like Figure 1As shown, in one embodiment of the present invention, the specific detailed steps of the proxy IP identification and management method based on URL resource detection are as follows: S1: URL Resource Acquisition and Queuing This invention obtains a list of URL resource connections from multiple proxy vendors through API interface integration and configuration file import, while synchronizing the proxy vendor information corresponding to each URL resource. Then, this invention adds the obtained URL resource connections to the probe queue for subsequent probe processing. Moreover, this invention supports scheduled tasks to automatically pull new resources from vendors and continuously replenish the probe queue.

[0028] S2: Initiate a connection probe request The scheduler of this invention retrieves URL resource connections sequentially from the queue to be probed, and initiates a connection request to a preset (i.e., pre-deployed) target probe server through the URL resource. The target probe server is a pre-deployed service node used to receive proxy connections and return the source IP of the connection. It is the infrastructure for IP probe. In this invention, the target probe server is deployed on an independent public network node with a fixed public IP address, avoiding deployment behind NAT or proxy networks, to ensure that the observed source IP is a real public network exit IP. Multiple regional nodes can be deployed according to the business coverage area. During probe, the target probe server that matches the expected exit region of the proxy resource is selected first to reduce the impact of network links on the probe results.

[0029] S3: Final exit IP address identification Upon receiving a request, the target detection server of this invention records the source IP address of the connection and sends it back. Based on the feedback network connection information, it identifies and extracts the final exit IP address corresponding to the URL resource.

[0030] In S3, a combination of three detection methods is used: HTTP Request Probing: An HTTP request is sent to the target probing server via a proxy URL. The source IP and proxy chain fields such as X-Forwarded-For and X-Real-IP in the response are analyzed. This can simultaneously verify authentication status and application layer connectivity. Specifically, the client sends a GET request to the target probing server's api / ip-detect path via the proxy URL. The target server obtains the remote address from the TCP connection layer and reads the X-Forwarded-For and X-Real-IP fields in the request header. It then encapsulates the source IP, proxy chain fields, and timestamp into JSON format and returns it. The client parses the response to obtain the IP information.

[0031] TCP Connection Probing: A TCP connection is established with the target probe server. The server records and returns the peer IP address at the transport layer, which is used to verify the HTTP probe results or to adapt to non-pure HTTP scenarios such as SOCKS and tunnel proxies. Specifically, the client establishes a TCP connection with the specified port of the target probe server through the proxy URL. After the connection is established, the target server directly writes the peer IP address into the connection as a string and closes the connection. The client reads the returned string, which is the transport layer exit IP address.

[0032] DNS resolution detection: The domain name of the proxy URL is resolved to obtain the entry IP, which is used to identify domain name migration and changes in entry nodes. Specifically, the domain name part of the proxy URL is extracted, an A record query is initiated to the public DNS server, and the entry IP address corresponding to the domain name is obtained, which is used to verify changes in entry nodes.

[0033] The execution priority of the three methods is as follows: HTTP request probing first, TCP connection probing second, and DNS resolution probing last. The result acceptance priority is as follows: the connection layer source IP actually observed by the target probing server has higher priority than the HTTP proxy chain field, and the HTTP proxy chain field has higher priority than the DNS resolution entry IP. When two or more valid probing results point to the same public IP within the same detection period, it is determined to be the final exit IP of that URL. For multi-layer cascading proxies, this step only confirms the final exit IP visible to the target probing server. Upstream relay nodes are only stored as supplementary information and are not included in the management of available exit IPs. The boundary for determining the final exit IP is based on the peer IP directly observed by the target probing server at the connection layer, corresponding to the exit address of the closest hop to the target in the proxy link. Upstream node information is extracted from the proxy chain IP list in the X-Forwarded-For field of the HTTP request header and recorded sequentially as upstream node information. Intermediate node information is only supplemented when the vendor interface provides official cascading data. Upstream node information is only used for link analysis data storage and does not participate in core business logic such as IP deduplication, whitelist matching, and resource scheduling. The same detection period is defined as: all detection operations completed within a 1-minute time window for the same URL resource are considered as the same detection period; The determination of a valid probe result is as follows: a successful request return, a normal status code, and a valid IP format are considered valid; a connection timeout, authentication failure, or an abnormal return result are considered invalid and will not be included in the fusion. The specific rules for handling inconsistent results are as follows: When the actual measured IP of the target end of the HTTP probe and the TCP probe are inconsistent, they are recorded and marked as "multiple exit IP resources" and three consecutive retests are triggered; if they are still inconsistent after multiple retests, they are recorded and stored separately according to different probe protocols; when the entry IP obtained by DNS resolution is inconsistent with the exit IP obtained by connection probe, the DNS resolution result is only saved as entry node information and must not overwrite the exit IP obtained by probe.

[0034] S4: Agent Resource Availability Detection Perform comprehensive availability testing on the proxy resources corresponding to the URL resources with the identified final egress IP, and output availability judgment results and performance indicator data. The testing items include connectivity testing, authentication testing, stability testing, and performance testing, specifically: Connectivity testing verifies whether the proxy can establish a network connection normally; authentication testing verifies whether authentication information such as account passwords and tokens are valid; stability testing performs multiple consecutive probes to evaluate the stability of resources; and performance testing records performance indicators such as response time and transmission rate.

[0035] The detection process employs the token bucket algorithm, implementing rate limiting across four dimensions: proxy vendor, authentication account, URL resource, and target probe server. This avoids abnormal pressure on vendor services. When a detection fails, an exponential backoff strategy is implemented, progressively extending the detection interval to 1 minute, 5 minutes, 15 minutes, and 30 minutes. The token bucket algorithm configures independent token bucket capacity and token generation rate for each dimension: for example, a single URL resource has a token bucket capacity of 5 and a token generation rate of 1 every 5 minutes, ensuring a minimum detection interval of 5 minutes; a single vendor account has a token bucket capacity of 20 and a token generation rate of 2 per minute, controlling concurrent requests. Authentication failure errors are not retried frequently; once a threshold is reached, the process is transferred to manual verification. Network jitter errors employ a low-frequency retest strategy. More specifically, rate limiting is implemented by creating independent token buckets for each of the four dimensions: vendor, account, URL, and target node. By default, the minimum detection interval for a single URL resource is no less than 5 minutes, the number of concurrent detection requests under a single vendor account does not exceed 10, and the total detection request rate handled by a single target probe server does not exceed 50 requests per second. The index backoff rules are as follows: after the first failure, retry after 1 minute; after the second failure, retry after 5 minutes; after the third failure, retry after 15 minutes; and after the fourth or subsequent failures, retry after 30 minutes. A single URL resource can be retried a maximum of 8 times within 24 hours. The manual verification process is as follows: when authentication failures accumulate to 3, the automatic detection task corresponding to that account is automatically suspended, and an alarm ticket is generated. Staff manually verify whether the account password is valid and whether the account has been banned by the proxy provider. After verification, the authentication credentials are updated or the corresponding resource is marked as invalid.

[0036] S5: Resource Status Management and Inventory Management Perform URL resource status management based on availability test results: For those that pass the test, information such as the URL resource connection, final exit IP address, affiliated proxy vendor, resource status, and performance indicators are stored in a proxy IP resource database for unified management, marked as available, and included in the available resource pool. The core table fields of the proxy IP resource database in this invention include: resource ID, URL address, exit IP address, and vendor name as primary keys; four resource statuses: available, invalid, under testing, and pending verification; number of consecutive failures, last test time, next probe time, and backoff level; JSON format performance data, authentication ciphertext, and key version number.

[0037] The status transition follows these rules: newly added resources are initially marked as "under testing"; after the first successful test, they are changed to "available"; after consecutive failures reach a threshold, they are changed to "invalid"; after the invalid resource becomes available again, it re-enters the "under testing" state; after authentication failures reach a threshold, it is changed to "pending verification". The index design establishes separate indexes for egress IP, vendor name, resource status, and next probe time to support fast querying and inspection scheduling.

[0038] If a URL fails the availability test, the number of consecutive failures is accumulated. When the number of consecutive failures reaches a preset threshold (e.g., 3 times), the URL resource is marked as invalid, removed from the available resource pool, and the reason and time of failure are recorded. When the availability test of the URL resource passes, the number of consecutive failures and the backoff level corresponding to the URL resource are automatically cleared, and the default normal testing cycle is restored. If it is a URL resource that has been recovered after failure, it first enters the testing state, and after passing 3 consecutive tests, it is officially converted to the available state.

[0039] S6: Regular Inspections and Dynamic Updates This invention re-probes and re-detects the existing URL resources in the proxy IP resource database according to a preset period (e.g., hourly), and synchronously updates the final exit IP address and resource status, so as to promptly detect IP changes, service fluctuations and recovery status on the vendor side, and ensure the real-time and accuracy of the database data.

[0040] S7: Resource query service output A RESTful resource query interface is built based on a proxy IP resource database, providing proxy IP resource query services. External applications can filter resources by vendor name, IP address, availability status, and other criteria to obtain a list of proxy IP resources that meet their needs. The resource query interface adopts a RESTful architecture, supports HTTP GET calls, and supports query parameters including: vendor name, outgoing IP address range, resource status, and region tag. The returned result fields include: resource ID, URL address, outgoing IP address, vendor, resource status, last response time, and availability statistics. Paginated queries and batch export in JSON format are supported.

[0041] like Figure 2 As shown, the proxy IP identification and management system based on URL resource detection in this embodiment includes a URL resource acquisition module 101, an IP address detection module 102, a resource availability detection module 103, a resource status management module 104, a periodic inspection module 105, a resource query interface module 106, and an authentication information encryption storage module 107.

[0042] The URL resource acquisition module 101 is responsible for interfacing with multiple proxy vendors, obtaining URL resource connections and vendor information through APIs, configuration files, etc., and adding the URL resources to the probe queue. It supports automatic periodic retrieval of new resources. The IP address detection module 102 has built-in HTTP request detection, TCP connection detection, and DNS resolution detection units. It retrieves URL resources from the probe queue according to preset priorities, initiates connection requests to the target probe server, identifies the final exit IP address based on the returned connection information, and performs multi-source result fusion and confidence marking. During multi-source detection result fusion, confidence marking is performed simultaneously: for example, the confidence of the target probe server's transport layer measured IP is 95%, the confidence of the IP extracted from the HTTP proxy chain field is 70%, and the confidence of the DNS resolution entry IP is 50%. Multiple probe results within the same detection period point to the same IP. When the confidence level is highest, it is increased by 5%; when the results are inconsistent, the result with the highest confidence level is used, and the corresponding confidence level value is marked. The resource availability detection module 103 has a built-in detection execution unit, a rate limiting scheduling unit, and an anomaly backoff unit. The detection execution unit completes four tests: connectivity, authentication, stability, and performance. The rate limiting scheduling unit controls the detection rate according to multiple dimensions. The anomaly backoff unit implements graded backoff and differentiated retry strategies for different failure types. The resource status management module 104 performs resource status management based on the availability detection results: qualified resources are written into the proxy IP resource database and included in the available resource pool. The cumulative number of failures of unqualified resources reaches a threshold, at which point they are marked as invalid and removed from the availability pool. The resource status management module 104 has a built-in resource cleanup subunit, which is responsible for marking invalid resources, removing them from the availability pool, archiving invalid resources, and cleaning up data. The periodic inspection module 105 performs IP re-probing and availability re-testing on the existing resources in the database according to a preset cycle, dynamically updates IP addresses and resource status, and maintains data timeliness. The resource query interface module 106 builds a standardized query interface based on the proxy IP resource database, supports multi-condition filtering queries, and provides stable proxy IP resource query services to external applications.

[0043] The authentication information encryption storage module 107 of this invention uses envelope encryption to store sensitive credentials such as agent accounts, passwords, and tokens. It supports key rotation and audit logs, decrypts on demand during probes, and automatically desensitizes output. The envelope encryption process involves encrypting each agent authentication information (account, password, token) using a randomly generated data encryption key via the AES-256-GCM algorithm to generate ciphertext and an authentication tag. The data encryption key is then protected by the system master key. Key management specifically involves the master key being uniformly stored and managed by the Key Management Service (KMS), with different vendors using different data encryption keys. The key is automatically rotated every 90 days. After rotation, the old key version is retained for decrypting historical ciphertext. Background asynchronous tasks gradually re-encrypt historical authentication ciphertext using the new key. Decryption and desensitization refer to obtaining the master key from the KMS to decrypt the data encryption key only when a probe request is initiated, then decrypting the authentication information in memory, and immediately overwriting and clearing the decrypted plaintext data in memory after use. In log records, interface return results, and data export files, sensitive fields such as account and password are desensitized using a mask, displaying only the first two fields. The first character and the last two characters.

[0044] In this invention, the target detection server is the core infrastructure for IP identification. This embodiment provides two implementation schemes: Option 1: HTTP Service Mode Specifically, an HTTP server is deployed to listen on a specified port and receive HTTP requests forwarded through the proxy. The server obtains the real source IP from the TCP connection peer and reads the proxy chain field from the request header, returning a JSON response containing the source IP, timestamp, and proxy chain information. This solution is suitable for most HTTP / HTTPS proxy scenarios and can verify authentication and connectivity simultaneously.

[0045] Option 2: TCP Service Mode Specifically, a TCP server is deployed, bound to a specified port, to listen for client connection requests. Upon accepting a connection, the peer's IP address is obtained, the IP information is returned to the client, the connection is closed, and the server continues listening for the next request. This solution operates at the transport layer, is unaffected by HTTP header forgery, and yields more reliable results, making it suitable for verification and non-HTTP proxy scenarios.

[0046] Furthermore, the proxy IP identification and management system based on URL resource detection of this invention supports the simultaneous management of resources from multiple proxy vendors. It stores the basic information of each vendor through configuration files or databases, and independently configures the API address, authentication credentials, resource format, rate limiting parameters, and other operating parameters for each vendor. It unifies the heterogeneous resource formats returned by different vendors into a system standard format containing fields such as URL address, authentication information, vendor identifier, and resource type. In the database, resources from different vendors are stored in isolation through the vendor identifier field, and operations such as querying, statistics, and inspection are supported to be executed independently at the vendor level. At the same time, it automatically calculates the total resource volume, availability, average response time, failure rate, and other operating indicators of each vendor, providing data support for vendor selection and resource scheduling and allocation.

[0047] To ensure the operational stability of the detection process and the security of sensitive data storage, this embodiment adopts a control mechanism that combines hierarchical scheduling, compliant requests, and encrypted storage: At the detection scheduling level, hierarchical queue management is implemented for resources in different states. Newly added resources enter the fast confirmation queue, and their initial availability is verified through multiple short-cycle detections; stable and available resources enter the regular inspection queue, and detection is performed at fixed intervals with added random time jitter; unstable resources have their service call priority reduced, while the frequency of retesting is increased; failed resources enter the cooling queue, and detection requests will not be re-initiated during the cooling period. At the request execution level, probe requests use standard and stable request header fields to maintain consistent protocol behavior and avoid behaviors that could trigger vendor risk control, such as sudden requests or abnormal concurrency. Probes are executed in strict accordance with the vendor's terms of service, and the frequency is automatically reduced or probes are paused when a rate limiting warning is received. At the data security level, proxy authentication credentials are stored using envelope encryption. Each credential is encrypted using the AES-256-GCM algorithm, and the data encryption key is encrypted with the master key. The master key is managed uniformly by the key management service and supports periodic rotation. Authentication information is only briefly decrypted and used in memory during probe execution. Logs, interface outputs, and exported data are all anonymized, and audit logs are recorded for the entire process.

[0048] The present invention also discloses a non-transitory computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described proxy IP identification and management method based on URL resource detection.

[0049] This embodiment does not impose any limitation on the shape, material, structure, etc. of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for identifying and managing proxy IPs based on URL resource detection, characterized in that, Includes the following steps: S1: Obtain the URL resource links and corresponding agent information provided by each agent, and add the URL resource links to the queue to be probed; S2: Take a URL resource connection from the queue to be probed, and send a connection request to the preset target probe server that is used to provide feedback on the source IP of the connection through the URL resource connection; S3: Based on the network connection information fed back by the target detection server, identify and extract the final exit IP address corresponding to the URL resource connection; S4: Perform availability testing on the proxy resources corresponding to the URL resources whose final exit IP addresses have been identified, and obtain the resource availability judgment results and performance index data; S5: Perform URL resource status management based on the availability determination result: If the detection passes, store the URL resource connection, the corresponding final exit IP address, the proxy vendor, the URL resource status and performance index data into the proxy IP resource database, mark it as available and include it in the available resource pool; if the detection fails, accumulate the number of consecutive failures, and when the number of consecutive failures reaches a preset threshold, mark the URL resource as invalid and remove it from the available resource pool. S6: Perform IP re-probing and availability re-detection on the existing URL resources in the proxy IP resource database according to a preset period, and dynamically update the final exit IP address and URL resource status corresponding to the URL resource; S7: Construct a resource query interface based on the proxy IP resource database, and provide proxy IP resource query services to external applications through the resource query interface.

2. The method for identifying and managing proxy IPs based on URL resource detection according to claim 1, characterized in that: Step S3, identifying the final exit IP address, specifically includes: The network connection information at the corresponding layer is obtained by using three methods in sequence: HTTP request probing, TCP connection probing, and DNS resolution probing. Among them, HTTP request probing is used to obtain the application layer response and proxy chain field, TCP connection probing is used to obtain the actual peer IP of the transport layer, and DNS resolution probing is used to obtain the IP of the entry domain name resolution. The results are fused according to the rule that the actual observed source IP of the target end has a higher priority than the HTTP proxy chain field, and the HTTP proxy chain field has a higher priority than the DNS resolution entry IP. If two or more valid detection results point to the same public IP address within the same detection period, that IP address is determined to be the final exit IP address corresponding to the current URL resource connection.

3. The method for identifying and managing proxy IPs based on URL resource detection according to claim 1, characterized in that: The availability testing in step S4 includes connectivity testing, authentication testing, stability testing, and performance testing; The detection process uses either a token bucket or leaky bucket algorithm, and performs rate limiting control based on the dimensions of the agent, the certified account, the URL resource connection, and the target detection server. When a detection fails, the next detection interval is extended according to the exponential backoff strategy. When the number of authentication failure errors reaches a preset number, the process is transferred to manual verification. For network jitter errors, a low-frequency retest strategy is adopted.

4. The method for identifying and managing proxy IPs based on URL resource detection according to claim 1, characterized in that: The final exit IP address identified in step S3 is, in a multi-layered cascading proxy scenario, the nearest hop proxy exit IP observed by the target probe server. In a cascaded link, upstream agent nodes, vendor scheduling nodes, and ingress IPs are stored only as supplementary information and do not participate in the primary key deduplication and whitelisting of available egress IPs.

5. The method for identifying and managing proxy IPs based on URL resource detection according to claim 1, characterized in that: The proxy authentication information corresponding to the URL resource used to initiate connection requests and availability checks is stored using an envelope encryption method: a single proxy authentication information is encrypted with authentication using a data encryption key, and then the data encryption key is encrypted using a master key; The proxy authentication information is only briefly decrypted in memory when probing requests and availability checks are initiated, and the credential information is anonymized in log records and interface responses.

6. A proxy IP identification and management system based on URL resource detection, characterized in that, include: The URL resource acquisition module is used to acquire URL resource links and corresponding agent information provided by each agent, and add the URL resource links to the probe queue. The IP address detection module is used to retrieve a URL resource connection from the queue to be detected, initiate a connection request to a preset target detection server that provides feedback on the source IP of the connection through the URL resource connection, and identify and extract the final exit IP address corresponding to the URL resource connection based on the network connection information fed back by the target detection server. The resource availability detection module is used to perform availability detection on the proxy resources corresponding to the URL resources whose final exit IP address has been identified, and to obtain the resource availability judgment results and performance index data. The resource status management module is used to perform URL resource status management based on the availability determination result: If the detection passes, the URL resource connection, the corresponding final exit IP address, the proxy vendor, the URL resource status and performance index data are stored in the proxy IP resource database, and marked as available and included in the available resource pool; if the detection fails, the number of consecutive failures is accumulated, and when the number of consecutive failures reaches a preset threshold, the URL resource is marked as invalid and removed from the available resource pool. The periodic inspection module is used to perform IP re-probing and availability re-detection on the existing URL resources in the proxy IP resource database according to a preset cycle, and dynamically update the final exit IP address and URL resource status corresponding to the URL resource. The resource query interface module is used to build a resource query interface based on the proxy IP resource database, and provide proxy IP resource query services to external applications through the resource query interface.

7. The proxy IP identification and management system based on URL resource detection according to claim 6, characterized in that: The IP address detection module integrates an HTTP request detection unit, a TCP connection detection unit, and a DNS resolution detection unit. It performs detection in the order of HTTP request detection priority, TCP connection detection verification, and DNS resolution detection reference, and performs multi-source result fusion and confidence marking according to the rule of prioritizing the actual IP of the target end.

8. The proxy IP identification and management system based on URL resource detection according to claim 6, characterized in that: The resource availability detection module has a built-in detection execution unit, a rate limiting scheduling unit, and an anomaly avoidance unit. The detection execution unit is used to perform connectivity detection, authentication detection, stability detection, and performance detection. The rate limiting scheduling unit uses the token bucket algorithm to control the probe request rate in multiple dimensions. The abnormal backoff unit performs hierarchical index backoff for different failure types and adopts different retry strategies to distinguish between authentication failure and network jitter scenarios.

9. The proxy IP identification and management system based on URL resource detection according to claim 6, characterized in that: It also includes an authentication information encryption storage module that uses envelope encryption to store proxy authentication credentials, supports key rotation and audit log recording, and performs on-demand decryption and desensitization output when probed.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the proxy IP identification and management method based on URL resource detection as described in any one of claims 1 to 5.

Images