A method and device for implementing a two-level forward proxy
By implementing a secondary forward proxy in OpenResty and using public areas and timer mechanisms to manage proxy resources, the problems of cumbersome and inefficient maintenance of forward proxy static routing are solved, efficient proxy resource allocation and effectiveness detection are achieved, and improved Proxy performance and scalability.
Patent Information
- Application Number
- CN202210030378.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-12
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-01-12
AI Technical Summary
The existing forward proxy static routing method is cumbersome and inefficient. The frequent updates and restarts of Squid proxy affect performance, and cannot effectively solve the problem of secondary forward proxy.
By implementing a secondary forward proxy in OpenResty, dividing public areas, using the proxy resource extraction module and management module to obtain and detect proxy resources, the timer mechanism circulates and detects resource validity, clears invalid resources, and handles modules through proxy requirements Allocate valid resources to the client.
It improves the efficiency of proxy resources, reduces the burden of maintaining static routing tables, ensures the effectiveness of proxy IP, and improves the forwarding performance and scalability of proxy software.
Smart Images

Figure CN114390094B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of Internet communications, and in particular to a method and device for implementing a two-level forward proxy. Background Art
[0002] In the scenario of Internet access, sometimes it is necessary to hide the real IP and break through the IP access restrictions of the visited pages. In this case, a forward proxy server mechanism needs to be adopted. When the IP access restrictions of the forward proxy server are broken, a technical means needs to be introduced to solve this problem.
[0003] At present, static routing and squid proxy are two common technical solutions to solve this problem. However, the core of static routing is to add and maintain static routing tables. Because the client program needs to access a variety of pages, the client needs to add a large number of static routing tables, which leads to worse and worse network performance. Squid's cache_peer instruction can specify the parent proxy IP address. If the parent proxy IP address fails and needs to be replaced, the cache_peer configuration also needs to be updated and restarted to take effect. This will cause squid to update, restart and load frequently, which will also affect the forwarding performance of the squid proxy software.
[0004] Therefore, a more convenient, scalable and efficient technical solution is needed to solve the problem of secondary forward proxy. Summary of the invention
[0005] In order to solve the problems of low access efficiency and cumbersome maintenance of the forward proxy static routing table, the present invention provides a method and device for implementing a secondary forward proxy, establishes a public area, a proxy resource extraction module obtains resources that need to be proxied and places them in the public area, a proxy resource management module obtains proxy resources from the public area and detects whether they are valid, and removes invalid proxy resources from the public area. When a client needs a proxy demand, the proxy demand processing module obtains valid proxy resources from the public area and allocates them to the client for use.
[0006] In a first aspect, to achieve the above-mentioned purpose, the present application provides a method for implementing a two-level forward proxy, including:
[0007] Initialize proxy services, activate proxy resource extraction services and proxy resource management services; divide public areas; encapsulate proxy demand processing services;
[0008] Preferably, the proxy resource extraction service accesses the proxy resource pool to obtain and store the proxy resources in the shared area; the proxy resource management service detects the validity of the proxy resources in the public area and removes invalid resources.
[0009] Preferably, the proxy demand processing service obtains the proxy demand from the client, extracts idle proxy resources from the public area and allocates them to the proxy demand.
[0010] Preferably, the proxy resource extraction service includes a first cyclic timer, which sets an access frequency f1 for cyclically accessing the proxy resource pool.
[0011] Preferably, the proxy resource management service includes a second cyclic timer, which sets a polling frequency f2 and is used to cyclically detect the validity of the proxy resources in the public area;
[0012] The proxy resource management service cyclically uses the proxy resources according to the frequency f2, creates a new forward proxy request, and obtains the response result of the request.
[0013] Preferably, the method for the proxy resource management service to detect the validity of the proxy resources in the public area includes determining the response time of the proxy resource to establish a forward proxy request, and if the response time times out, invalidating the proxy resource; the invalidation includes: marking the proxy resource invalid in the public area and removing the proxy resource from the public area.
[0014] On the other hand, the present invention provides a device for implementing a two-level forward proxy, comprising:
[0015] Agent resource extraction module, agent resource management module, public area and agent demand processing module;
[0016] Preferably, the proxy resource extraction module is used to access the proxy resource pool, obtain and store the proxy resources in the shared area;
[0017] The proxy resource extraction module includes a first cyclic timer for setting an access frequency f1 and controlling cyclic access to the proxy resource pool.
[0018] On the other hand, the proxy resource extraction module is used to access the proxy resource pool, obtain and store the proxy resources in the shared area, and determine whether there is the same proxy IP in the shared area. If there is the same proxy IP, storage in the shared area is abandoned.
[0019] Preferably, the proxy resource management module is used to detect the validity of proxy resources in the public area and clear invalid resources;
[0020] The proxy resource management module includes a second cycle timer for setting a polling frequency f2 and controlling a cycle detection of the validity of the proxy resources in the public area.
[0021] Preferably, the proxy resource management module determines the response time of each forward proxy request, and if the response time exceeds, the proxy resource is invalidated; the invalidation process includes: marking the proxy resource invalid in the public area and removing the proxy resource from the public area.
[0022] Preferably, the proxy demand processing module is used to obtain proxy demands from the client, extract idle proxy resources from the public area and allocate them to the client proxy demands;
[0023] When the proxy demand processing module obtains the proxy demand from the client, it first establishes a random algorithm, randomly obtains the proxy IP from the public area, and allocates the valid proxy IP to the client.
[0024] The resource management service provided by the present invention detects the validity of the proxy resources in the public area, solving the problem of connection timeout when the client accesses the proxy resources. The method for obtaining and storing the proxy resources in the shared area solves the problem of static maintenance of the routing table. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a flow chart of a method for implementing a two-level forward proxy according to an embodiment of the present invention;
[0026] Figure 2 It is a schematic diagram of a method for implementing a two-level forward proxy according to an embodiment of the present invention;
[0027] Figure 3 It is a structural diagram of an implementation device of a two-level forward proxy provided according to an embodiment of the present invention. DETAILED DESCRIPTION
[0028] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings.
[0029] The implementation method of the secondary forward proxy provided by the present invention is implemented based on OpenResty, which is a high-performance Web platform based on Nginx and Lua, which integrates a large number of sophisticated Lua libraries, third-party modules and most dependencies. It is used to easily build dynamic Web applications, Web services and dynamic gateways that can handle ultra-high concurrency and high scalability. The method of the present application is implemented on the basis of OpenResty, with low maintenance cost, flexibility and scalability.
[0030] Figure 1 is a flow chart of a method for implementing a two-level forward proxy in an embodiment of the present invention. Figure 2 A schematic diagram of the implementation method of the secondary forward proxy is provided in Figure 1 and Figure 2, the specific implementation methods of this application are described:
[0031] In the present invention, a forward proxy port is started to obtain the Internet access address that the client program needs to access. The client program will access the Internet through this forward proxy port. The specific implementation method is shown in Figure 1 :
[0032] Step S110: Initialize the proxy service, activate the proxy resource extraction service and the proxy resource management service, encapsulate the proxy demand processing service, and divide the public area.
[0033] In the specific implementation, two child threads ngx.worker.0 and ngx.worker.1 are started simultaneously in the init_worker_by_lua phase of OpenResty, and the address acquisition algorithm is encapsulated in the preread_by_lua phase.
[0034] Proxy resource extraction service, that is, ngx.worker.0 child thread, ngx.worker.0 is responsible for obtaining the proxy IP address from the third-party proxy pool;
[0035] The proxy resource management service, namely the ngx.worker.1 child thread, is responsible for detecting the validity of the obtained proxy IP;
[0036] The public area is a new shared memory lua_shared_dict created by OpenResty, named proxy_ups, which is used to read and write information to ngx.worker.0 and ngx.worker.1, such as Figure 2 As shown, ngx.worker.0 writes to proxy_ups, and ngx.worker.1 reads and writes to proxy_ups.
[0037] Step S120: the proxy resource extraction service accesses the proxy resource pool, obtains and stores the proxy resources in the shared area; the proxy resource management service detects the validity of the proxy resources in the public area and removes invalid resources;
[0038] In this step, ngx.worker.0 includes the first loop timer ngx.timer.every, for example, determining the frequency f1=10 seconds, that is, obtaining the proxy IP from the third-party proxy pool every 10 seconds and adding it to the shared memory proxy_ups; Figure 2 As shown, ngx.worker.0 obtains the IP address from the proxy pool through http request access.
[0039] ngx.worker.1 includes the second loop timer ngx.timer.every. For example, the frequency f2 is set to 60 seconds, that is, the proxy IP is polled from proxy_ups every minute, a new forward proxy request is created through lua-resty-http, and the validity of the proxy IP is detected according to the response of the request. The proxy IP that responds normally is retained, and the proxy IP that responds timeout or does not respond is invalidated. For invalid processing, it can be marked invalid in shared memory or removed from shared memory proxy_ups. Invalid marking in shared memory can avoid taking the resource out of the third-party proxy pool again and performing invalid operations.
[0040] Step S130: The proxy demand processing service obtains the proxy demand from the client, extracts idle proxy resources from the public area and allocates them to the corresponding proxy demand; the proxy demand processing service obtains the detected valid proxy IP from the shared memory proxy_ups according to the address acquisition algorithm, and assigns it to the upstream of OpenResty. When the client requests, it can access external network resources through this proxy IP. This algorithm can be a random algorithm, or it can be an algorithm combined with the network segment of the access address, the amount of visits, and other environments. Figure 2 As shown in the figure, preread_by_lua uses the method of randomly obtaining the proxy IP, obtains the IP address from proxy_ups, assigns it to the upstream, and provides it to the client in need, and accesses it in the form of http connect.
[0041] Figure 3 The structural diagram of the implementation device of the two-level forward proxy provided by the present invention is shown in the figure. The implementation device of the two-level forward proxy of the present application includes:
[0042] P310: Initialization agent module, P311: Agent resource extraction module, P312: Agent resource management module, P320: Public area and P313: Agent demand processing module;
[0043] P310: Initialize the proxy module, which is used to start two submodules at the same time in the init_worker_by_lua phase of OpenResty, one is P311: proxy resource extraction module, and the other is P312: proxy resource management module.
[0044] P311: The proxy resource extraction module is used to access the proxy resource pool, obtain and store proxy resources in the shared area;
[0045] The first cycle timer is included, which is used to set the access frequency f1 and control the cyclic access to the proxy resource pool.
[0046] P311: The proxy resource extraction module is used to access the proxy resource pool, obtain and store the proxy resources in the shared area, and determine whether there is the same proxy IP in the shared area. If there is the same proxy IP, the IP is abandoned and stored in the shared area.
[0047] P312: The proxy resource management module is used to detect the validity of proxy resources in the public area and clear invalid resources;
[0048] The second cycle timer is included, which is used to set the polling frequency f2 and control the cycle detection of the validity of the proxy resources in the public area.
[0049] P312: When the proxy resource management module determines the response time of each forward proxy request, if the response time times out, the proxy resource is invalidated; the invalidation process includes: marking the proxy resource as invalid in the public area and clearing the proxy resource from the public area.
[0050] P320: The public area is shared and accessed by P311: agent resource extraction module, P312: agent resource management module, and P313: agent demand processing module.
[0051] Among them, P320: Public area synchronous access adds a mutex lock to prevent data synchronization problems in P320: public area access.
[0052] P313: The proxy demand processing module is used to obtain proxy demands from the client and extract idle proxy resources from the public area and allocate them to the proxy demands.
[0053] When obtaining proxy requirements from clients, a random algorithm is first established to randomly obtain proxy IPs from the public area, and then the valid proxy IPs are allocated to the clients to realize the forward proxy requirements.
[0054] Through the embodiments provided by the present invention, clients with access requirements are supported to implement forward proxy access from the IP address in the upstream, and the method can ensure that the provided proxy IP is valid, and separate the verification of the proxy IP address from the actual use by the demander, thereby ensuring the performance of the proxy; when the client initiates proxy access with different requirements, since the present invention can also expand the algorithm and expand the port to meet the needs of the client, the present invention is more scalable.
[0055] The above disclosures are only several specific embodiments of the present invention; however, the present invention is not limited thereto, and any changes that can be conceived by those skilled in the art should fall within the protection scope of the present invention.
Claims
1. A method for implementing a two-level forward proxy, characterized in that: The following steps are involved: Initialize proxy services, activate proxy resource extraction services and proxy resource management services; Divide the public area; encapsulate the proxy demand processing service; including: simultaneously start two sub-threads ngx.worker.0 and ngx.worker.1 in the init_worker_by_lua phase of OpenResty, and encapsulate the address acquisition algorithm in the preread_by_lua phase; wherein the public area includes the shared memory created by OpenResty's lua_shared_dict; the proxy resource extraction service accesses the proxy resource pool, obtains and stores the proxy resources to the shared memory; including the ngx.worker.0 sub-thread obtaining the proxy IP address from the third-party proxy pool; The proxy resource management service detects the validity of the proxy resources in the public area and clears invalid resources; includes the ngx.worker.1 child thread responsible for detecting the validity of the acquired proxy IP; the proxy resource extraction service includes a first cyclic timer, sets the access frequency f1, and is used to cyclically access the proxy resource pool, including the first cyclic timer ngx.timer.every in the ngx.worker.0, which is used to obtain the proxy IP from the third-party proxy pool and add it to the shared area; and also includes ngx.worker.0 obtaining the proxy IP from the proxy IP pool through the access method of http request; The proxy resource management service includes a second cyclic timer, which sets a polling frequency f2 and is used to cyclically detect the validity of proxy resources in the public area; wherein the proxy resource management service cyclically uses the proxy resources according to the frequency f2, creates a new forward proxy request, and obtains the response result of the request; wherein the second cyclic timer includes the second cyclic timer ngx.timer.every in the ngx.worker.1; the second cyclic timer creates a forward proxy request through lua-resty-http and detects the validity of the proxy IP according to the response of the request; The proxy demand processing service obtains a proxy demand from a client, and extracts idle proxy resources from the public area and allocates them to the proxy demand.
2. The method for implementing a secondary forward proxy according to claim 1, characterized in that: The method for detecting the validity of the proxy resource in the public area includes determining the response time of the proxy resource to establish a forward proxy request, and if the response time times out, invalidating the proxy resource; the invalidation includes marking the proxy resource invalid in the public area and removing the proxy resource from the public area.
3. A device for implementing a two-level forward proxy, characterized in that: include: Initialization agent module, agent resource extraction module, agent resource management module, public area and agent demand processing module; The initialization proxy module includes: starting two sub-threads ngx.worker.0 and ngx.worker.1 at the same time in the init_worker_by_lua phase of OpenResty, and encapsulating the address acquisition algorithm in the preread_by_lua phase; wherein the common area includes the shared memory created by lua_shared_dict of OpenResty; The proxy resource extraction module is used to access the proxy resource pool, obtain and store proxy resources in the shared area; including the ngx.worker.0 child thread obtaining the proxy IP address from the third-party proxy pool; also includes a first cyclic timer, set the access frequency f1, for cyclically accessing the proxy resource pool, including the first cyclic timer ngx.timer.every in the ngx.worker.0, for obtaining the proxy IP from the third-party proxy pool and adding it to the shared area; also includes ngx.worker.0 obtaining the proxy IP from the proxy IP pool through the access method of http request; The proxy resource management module is used to detect the validity of the proxy resources in the public area and clear invalid resources; it includes the ngx.worker.1 subthread responsible for detecting the validity of the acquired proxy IP; it also includes a second loop timer, which sets a polling frequency f2 for cyclically detecting the validity of the proxy resources in the public area; and cyclically uses the proxy resources according to the frequency f2, creates a new forward proxy request, and obtains the response result of the request; wherein the second loop timer includes the second loop timer ngx.timer.every in the ngx.worker.1; the second loop timer creates a forward proxy request through lua-resty-http and detects the validity of the proxy IP according to the response of the request; The proxy demand processing service is used to obtain proxy demands from the client, and extract idle proxy resources from the public area and allocate them to the proxy demands.
4. The device for implementing a secondary forward proxy according to claim 3, characterized in that: The proxy resource extraction module includes a first cyclic timer for setting an access frequency f1 and controlling cyclic access to the proxy resource pool.
5. The device for implementing a secondary forward proxy according to claim 3, characterized in that: The proxy resource management module includes a second cycle timer for setting a polling frequency f2 and controlling a cycle detection of the validity of the proxy resources in the public area.
6. The device for implementing a secondary forward proxy according to claim 5, characterized in that: The proxy resource management module determines the response time of each forward proxy request, and if the response time exceeds the time limit, the proxy resource is invalidated; The invalidation processing includes: marking the proxy resource invalid in the public area, and removing the proxy resource from the public area.
Citation Information
Patent Citations
Distributed access system based on public agent
CN110557462A
Method and device for processing proxy environment, terminal and storage medium
CN113923260A
