IP acquisition system based on dynamic IP agent pool
By using a dynamic IP proxy pool system for verification and prioritization, the problem of low efficiency in screening high-quality IPs in existing technologies has been solved, enabling efficient collection of high-quality IPs from policy information data networks.
Patent Information
- Application Number
- CN202410929861.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-15
AI Technical Summary
In existing technologies, static IP proxy services cannot meet the needs of rapidly changing network environments, resulting in the inability to effectively filter high-quality IPs from general IPs in policy information data networks, leading to low efficiency in high-quality IP filtering.
An IP acquisition system based on a dynamic IP proxy pool is adopted. By verifying the validity of IP addresses, setting a priority mechanism, distinguishing between high-quality and general IPs, and avoiding invalid IPs, high-priority IPs are acquired first.
This improves the efficiency of screening high-quality IPs, ensuring timely and effective collection of high-quality IPs from the policy information data network and avoiding the reuse of invalid IPs.
Smart Images

Figure CN118972369A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of intelligent acquisition systems, and in particular relates to an IP acquisition system based on a dynamic IP proxy pool. Background Art
[0002] Data has become a production factor as important as land, capital, and labor; developing the big data industry is an era requirement to give full play to my country's advantages in massive data scale and rich application scenarios and activate the potential of data factors. It is also an inevitable choice to accelerate economic development and reform and build a modern industrial system; policy information is mainly sourced from the websites of nearly 1,000 construction authorities in 23 provinces across the country (from national construction authorities to provincial construction authorities, and then to local county-level construction authorities and associations), and collects their published notices, announcements, circulars, service guidelines, policies and regulations, etc., and publishes them simultaneously; the three main functions and features of big data policy information are: 1. Daily express email subscription to get the national construction policy dynamics; 2. Ministry policy information national construction 1. Grasp the important dynamics of construction policy information; 2. Localize regional policy information to understand the dynamics of local construction policy; The website collects policy information to facilitate people to view various effective policy information; Big data major will systematically help enterprises master the solutions to various typical problems in big data applications from the three main aspects of big data application (i.e. data management, system development, massive data analysis and mining), including the implementation and analysis of collaborative filtering algorithms, the operation and learning of classification algorithms, the construction and benchmarking of distributed Hadoop clusters, the construction and benchmarking of distributed Hbase clusters, the implementation of a parallel algorithm based on Mapreduce, the deployment of Hive and the implementation of a data operation, etc., to actually improve the ability of enterprises to solve practical problems;
[0003] In the existing technology, with the increase of network applications, the limitation and switching of IP addresses have become common problems. The existing static IP proxy service cannot meet the needs of the rapidly changing network environment. For the policy information data network, there is a difference between high-quality IPs and general IPs. When high-quality IPs and general IPs are collected at the same time, the existing collection system cannot screen different IPs, which easily leads to low efficiency in high-quality IP screening. Summary of the invention
[0004] The purpose of the present invention is to provide an IP acquisition system based on a dynamic IP proxy pool, aiming to solve the problem in the prior art that there is a distinction between high-quality IPs and general IPs for policy information data networks. When high-quality IPs and general IPs are collected at the same time, the existing collection system cannot screen different IPs, which easily leads to low efficiency in screening high-quality IPs.
[0005] To achieve the above object, the present invention provides the following technical solutions:
[0006] An IP acquisition system based on a dynamic IP proxy pool, comprising:
[0007] Collection source;
[0008] IP proxy collection, the IP proxy collection collects the IP address of the collection source;
[0009] A database is provided, wherein the database stores invalid IP addresses, and an invalid IP address in the database is obtained to verify the validity of the IP address. If the IP address is valid, the IP priority is increased by 1, and if the IP address is invalid, the IP priority is decreased by 1.
[0010] As a preferred solution of the present invention, the collection source is input into the database after being processed by scrapy.
[0011] As a preferred solution of the present invention, if the source IP address collected by the IP proxy is valid, it will not be stored in the database.
[0012] As a preferred solution of the present invention, the random acquisition of IP interfaces is completed by executing an SQL query in the database.
[0013] As a preferred solution of the present invention, the random acquisition IP interface calls an interface to acquire the IP.
[0014] As a preferred solution of the present invention, when the IP address with IP priority -1 has an IP priority of 0, the IP data is deleted.
[0015] As a preferred solution of the present invention, the random acquisition IP interface gives priority to the IP with a high priority.
[0016] As a preferred solution of the present invention, IP addresses are divided into free IPs and paid IPs, and IPs are obtained randomly.
[0017] As a preferred solution of the present invention, the IP acquisition location is a website address.
[0018] As a preferred solution of the present invention, the deleted IP data is recorded and is not used as a collection target when collecting data.
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] 1. In the present invention, by using this solution, for the policy information data network, high-quality IPs and general IPs are distinguished, and deleted IP data is avoided, high-quality IPs are collected in a timely and effective manner, and the screening efficiency of high-quality IPs is improved.
[0021] 2. In the present invention, the randomly acquired IP interface gives priority to the IP with high priority, and the deleted IP address data is recorded in the system, and the deleted IP address is avoided during the next collection. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0023] Figure 1 It is a system block diagram of the present invention. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] Example 1
[0026] See also Figure 1 , the present invention provides the following technical solutions:
[0027] An IP acquisition system based on a dynamic IP proxy pool, comprising:
[0028] Collection source;
[0029] IP proxy collection, IP proxy collection collects the IP address of the collection source;
[0030] Database, the database stores invalid IP addresses, takes the invalid IP address in the database, verifies the validity of the IP address, if it is a valid IP address, the IP priority is +1, if it is an invalid IP address, the IP priority is -1.
[0031] In a specific embodiment of the present invention, the id address of the collection source is collected through an ip proxy. During the process, the validity of the ip is verified. If it is an invalid ip, it is input into the database. The validity of the ip is further verified in the database, and the ip address is taken. If it is a valid ip, the priority of the ip is +1, if it is not a valid ip address, the priority of the ip is -1; the proxy ip is mainly in the dialogue layer of the open system interconnection (OSI) model, thereby playing the role of a firewall. The IP is a unique identity address and identity certificate required for surfing the Internet, and the proxy IP is an intermediate platform in our surfing process. Your computer first accesses the proxy IP, and then the proxy IP accesses the page you clicked. Therefore, what is left in the access record of this page is the address of the proxy IP, not your computer local IP; by using this solution, for the policy information data network, high-quality ip and general ip are distinguished, and deleted ip data is avoided, high-quality ip is collected in a timely and effective manner, and the screening efficiency of high-quality ip is improved.
[0032] For details, please refer to Figure 1 , the collection source is processed by scrapy and then input into the database.
[0033] In this embodiment: input to scrapy for processing through the open debugging interface; after the scrapyp processing is completed, determine whether the request or the general analysis list page is abnormal, if there is an abnormality, update the collection status of the collection source; if there is no abnormality, determine whether the general analysis is abnormal, if there is an abnormality, update the collection status of the collection source, if there is no abnormality, process or clean the data; the detailed page is the title, author, source, matched with common rules, and the optimal solution is screened and evaluated; the release time is matched with the common rules and the text content, and the time format is returned after processing and verification; the text content, through the main body extraction method of readability, returns the labeled and formatted text content; finally, the processed data is input into the database, if there is an abnormality in the general analysis, the collection status of the collection source is updated, and the collection source is input into the database after being processed by scrapyp.
[0034] For details, please refer to Figure 1 If the source IP address collected by the IP proxy is valid, it will not be stored.
[0035] In this embodiment: in the procedure for verifying whether the IP is valid, if the IP is a valid IP, it will not be entered into the database.
[0036] For details, please refer to Figure 1 , the database executes an SQL query to complete the random acquisition of the IP interface.
[0037] In this embodiment: determine the number of IP addresses that need to be randomly acquired. If there is no database table for storing IP addresses, you need to design and create one, which usually includes fields such as IP address, type, region, etc., build an SQL query according to the needs, and use the database-specific random function to randomly acquire IP interfaces for the IP addresses entered in the database.
[0038] For details, please refer to Figure 1 , randomly obtain IP interface and call interface to obtain IP.
[0039] In this embodiment: the randomly obtained IP interface calls the interface to obtain the IP.
[0040] For details, please refer to Figure 1 , when the IP address with IP priority -1 has an IP priority of 0, delete the IP data.
[0041] In this embodiment: each time the IP is verified to be valid, if the IP priority is 0, the IP data is deleted and the IP data is no longer in use.
[0042] For details, please refer to Figure 1 , the random IP interface gives priority to the IP with high priority.
[0043] In this embodiment: among the randomly obtained IP interfaces, the IP with the highest priority is obtained first.
[0044] For details, please refer to Figure 1 , IP addresses are divided into free IPs and paid IPs, and IPs are obtained randomly.
[0045] In this embodiment: IP addresses are divided into free IPs and paid IPs.
[0046] For details, please refer to Figure 1 , the IP location is the website address.
[0047] In this embodiment: the website's URL part obtains the IP address.
[0048] For details, please refer to Figure 1 The deleted IP data is recorded and is not used as a collection target when collecting data.
[0049] In this embodiment: the deleted IP address data is recorded in the system, and the deleted IP address is avoided during the next collection.
[0050] The working principle and use process of the present invention: When this solution is used, for the IP address of the intelligent collection of big data policy information, the ID address of the collection source is collected through the IP proxy. In this process, it is verified whether the IP is valid. If it is an invalid IP, it is entered into the database, and the IP is further verified in the database. The IP address is taken, if it is a valid IP, the priority of the IP is +1, if it is not a valid IP address, the priority of the IP is -1; by using this solution, for the policy information data network, high-quality IPs and general IPs are distinguished, and deleted IP data is avoided, high-quality IPs are collected in a timely and effective manner, and the screening efficiency of high-quality IPs is improved.
[0051] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. An IP acquisition system based on a dynamic IP proxy pool, characterized in that: include: Collection source; IP proxy collection, the IP proxy collection collects the IP address of the collection source; A database is provided, wherein the database stores invalid IP addresses, and an invalid IP address in the database is obtained to verify the validity of the IP address. If the IP address is valid, the IP priority is increased by 1, and if the IP address is invalid, the IP priority is decreased by 1.
2. According to claim 1, an IP acquisition system based on a dynamic IP proxy pool is characterized in that: The collection source is processed by scrapy and then input into the database.
3. The IP acquisition system based on a dynamic IP proxy pool according to claim 2, characterized in that: If the source IP address collected by the IP proxy is valid, it will not be stored.
4. The IP acquisition system based on a dynamic IP proxy pool according to claim 3, characterized in that: The random acquisition of IP interfaces is accomplished by executing an SQL query in the database.
5. The IP acquisition system based on dynamic IP proxy pool according to claim 4 is characterized in that: The random acquisition IP interface calls an interface to acquire IP.
6. The IP acquisition system based on a dynamic IP proxy pool according to claim 5, characterized in that: When the IP address with the IP priority of -1 has an IP priority of 0, the IP data is deleted.
7. The IP acquisition system based on dynamic IP proxy pool according to claim 6, characterized in that: The random IP interface gives priority to the IP with higher priority.
8. The IP acquisition system based on dynamic IP proxy pool according to claim 7, characterized in that: IP addresses are divided into free IPs and paid IPs, and IPs are obtained randomly.
9. The IP acquisition system based on dynamic IP proxy pool according to claim 8, characterized in that: The IP acquisition location is a website address.
10. The IP acquisition system based on dynamic IP proxy pool according to claim 9, characterized in that: The deleted IP data is recorded and is not used as a collection target when collecting data.
