Internet spider system based on efficient data extraction and processing

Abstract

The invention discloses an internet spider system based on efficient data extraction and processing. Comprising a bare computer software unit, a working load unit, a page processing unit, a user-defined capturing unit, a flow control unit, an agent support unit, a mode matching unit, an automatic analysis unit, a map support unit, an intelligent cache unit, a data acquisition unit and a data processing unit. The bare computer software unit is used for compiling in a Rust programming language and providing predictable performance, memory security, concurrency and cross-platform compatibility; and the workload unit is used for distributing workloads on a plurality of nodes. The invention belongs to the technical field of Internet. Information data acquired by the data acquisition unit is analyzed and judged, a distributed workload and an asynchronous page processing mechanism are utilized, the processing efficiency of Internet data is effectively improved, and furthermore, expandability and flexibility are enhanced through customizable network climbing and flow control.

Description

Technical Field

[0001] The present invention relates to the field of Internet technologies, and more specifically discloses an Internet spider system based on efficient data extraction and processing. Background Art

[0002] The Internet has become an indispensable important tool for information collection and processing in modern society. However, current data extraction and processing tools still have significant deficiencies in terms of efficiency and effectiveness. Specifically, when dealing with large amounts of data, these tools often exhibit low efficiency, resulting in slow processing times and being unable to meet the need for quickly obtaining information. Internet big data is usually generated at high speed, meaning it must be processed in a near-real-time manner. However, traditional Internet data extraction and processing processes are usually batch-processing oriented and are difficult to keep up with this speed. Summary of the Invention

[0003] Object of the Invention: The object of the present invention is to solve the problem that Internet big data is usually generated at high speed, meaning it must be processed in a near-real-time manner. However, traditional Internet data extraction and processing processes are usually batch-processing oriented and are difficult to keep up with this speed.

[0004] To solve the above technical problems, according to one aspect of the present invention, more specifically, an Internet spider system based on efficient data extraction and processing, includes: a bare-metal software unit, a workload unit, a page processing unit, a custom crawling unit, a traffic control unit, a proxy support unit, a pattern matching unit, an automatic parsing unit, a map support unit, an intelligent caching unit, a data collection unit, and a data processing unit;

[0005] Bare-metal software unit: Used to be written in the Rust programming language, providing predictable performance, memory safety, concurrency, and cross-platform compatibility;

[0006] Workload unit: Used to distribute the workload across multiple nodes to improve processing efficiency and scalability;

[0007] Page processing unit: Used to enable the system to process pages in parallel;

[0008] Custom crawling unit: Used for users to customize the crawling settings for each path;

[0009] Traffic control unit: Used to allow users to manually control the crawling operation;

[0010] Proxy support unit: Used to support proxy servers and support anonymous browsing;

[0011] Pattern matching unit: Used for users to specify URL patterns using Glob syntax to simplify data collection for specific domains or directories;

[0012] Automatic parsing unit: used to parse the linked JavaScript file through a dedicated JavaScript engine, allowing the extraction of dynamic content;

[0013] Map support unit: used to generate a website map, implement link tracking, and ensure comprehensive data coverage;

[0014] Intelligent cache unit: used for the system to adopt a caching mechanism to improve processing efficiency and reduce memory usage;

[0015] Data acquisition unit: used to collect information data generated during the process of extracting and analyzing Internet information data in real time;

[0016] Data processing unit: used to process and analyze the information data collected in real time, obtain the processing and analysis results, and at the same time, forward the obtained processing and analysis results to the remote monitoring unit.

[0017] Furthermore, the system further includes: a data transmission unit, a data storage unit, and a remote monitoring unit;

[0018] Data transmission unit: used to transmit the information data collected by the data acquisition unit in real time;

[0019] Data storage unit: used to store the information data transmitted in real time by the data transmission unit;

[0020] Remote monitoring unit: used to display the information data stored in the data storage unit and the analysis results of the data processing unit in real time on the background terminal, and at the same time, when the result obtained by the data processing unit analysis is an abnormal result, an alarm is sent to the background terminal.

[0021] Furthermore, the data acquisition unit includes: a connection number acquisition module, an information volume acquisition module, a utilization rate acquisition module, a page acquisition module, a broadband acquisition module, and a task acquisition module;

[0022] Connection number acquisition module: used to collect the connection number information data of the load node in real time;

[0023] Information volume acquisition module: used to collect the information volume data of node load, processed pages, and crawled tasks in real time;

[0024] Utilization rate acquisition module: used to collect the utilization rate information data of the CPU and running memory in real time;

[0025] Page acquisition module: used to collect the number of processed pages and information processing speed information data in real time;

[0026] Broadband acquisition module: used to collect network broadband information data in real time;

[0027] Task acquisition module: used to collect the quantity information data of crawling tasks in real time.

[0028] Furthermore, the data processing unit includes: a data acquisition module, an analysis and judgment module, and a result forwarding module;

[0029] Data acquisition module: used to obtain the information data stored in the data storage unit in real time;

[0030] Analysis and judgment module: used to analyze and judge the information data obtained by the data acquisition module, so as to obtain the result of analysis and judgment;

[0031] Result forwarding module: used to forward the result analyzed and judged by the analysis and judgment module to the remote monitoring unit.

[0032] Furthermore, the remote monitoring unit includes: a data receiving module, a data display module, and an abnormal alarm module;

[0033] Data receiving module: used to receive the information data stored in the data storage unit and the analysis result information of the data processing unit in real time;

[0034] Data display module: used to display the information data received by the data receiving module on the background terminal display screen;

[0035] Abnormal alarm module: used to send an alarm prompt to the background terminal when the result obtained by the analysis and judgment module is an abnormal result.

[0036] Furthermore, the analysis and judgment module can obtain the connection number of the load node, the information volume of the node load, and the usage rate information data of the CPU and the running memory from the data acquisition module, and obtain the load value of the load node through the data analysis obtained above:

[0037]

[0038] Then, by comparing the load values of each node, the best node for data information loading is selected:

[0039] Node=min{node1,node2,…,node n}

[0040] where node j is the load value of the jth load node, and C j is the current connection number of the jth load node, and C maxis the maximum number of connections among all load nodes, and time is the total duration of data acquisition. is the information data volume of the j-th load node at the i-th second. is the CPU usage rate of the j-th load node. is the operating memory usage rate of the j-th load node, and Node is the best node for loading data information.

[0041] Furthermore, the analysis and judgment module can obtain the information volume of the processing page, the number of processing pages, and the information processing speed information data from the data acquisition module, and analyze the page processing index through the above-obtained data. If:

[0042]

[0043] it indicates that the page information processing is stable. Among them, E is the page processing index, Q is the total number of processed pages, and H k is the information data volume of the k-th page, and V k is the information data processing speed of the k-th page, B is the network bandwidth, is the CPU usage rate of the k-th page, is the operating memory usage rate of the k-th page, α is a preset conversion coefficient, and δ is a preset threshold parameter for page processing.

[0044] Furthermore, the analysis and judgment module can obtain the information volume of the crawling task, the network bandwidth, and the number information data of the crawling task from the data acquisition module, and analyze the traffic control index through the above-obtained data. If:

[0045]

[0046] it indicates that the crawling information traffic control is stable. Among them, T is the traffic control index, B is the network bandwidth, b is the bandwidth required for each crawling task, N is the total number of crawling tasks, β is a preset redundancy coefficient for the network fluctuation and server response delay affecting the task crawling, time is the total duration of data acquisition, and M i is the information data volume crawled at the i-th second, and m i is the information data volume processed at the i-th second, and M max is the maximum information data volume crawled within the time of time, and m min is the minimum information data volume processed within the time of time. is a preset threshold parameter for traffic control.

[0047] The beneficial effects of an Internet spider system based on efficient data extraction and processing according to the present invention are as follows: By analyzing and judging the information data collected by the data collection unit, using the distributed workload and asynchronous page processing mechanism, the processing efficiency of Internet data is effectively improved. Further, the scalability and flexibility are enhanced through customizable crawling and traffic control, the data quality is improved through JavaScript parsing and website map support, and the resource usage is reduced through the caching mechanism. Brief Description of the Drawings

[0048] The present invention will be further described in detail below with reference to the drawings and specific implementation methods.

[0049] Figure 1 It is a schematic diagram of the system principle. Specific Embodiments

[0050] The present invention will be described in detail below with reference to the drawings and embodiments. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other.

[0051] According to one aspect of the present invention, as Figure 1As shown, an Internet spider system based on efficient data extraction and processing is provided. First, the bare-metal software unit is written in the Rust programming language, ensuring the system's predictable performance, memory safety, high concurrency ability, and cross-platform compatibility, providing a solid foundation for the stable operation of the entire system. Next, the distributed workload mechanism of the workload unit distributes tasks to multiple nodes, improving processing efficiency and system scalability through parallel processing. The asynchronous page processing unit function allows the system to process multiple pages simultaneously, significantly reducing the overall processing time and improving the crawling speed. Users can customize the crawling settings through the customizable crawling unit, configure the crawling parameters for each path, achieve optimal resource allocation, and ensure that critical data is processed first. The built-in traffic control unit in the system gives users the ability to manually control the crawling operation, including pausing, restarting, or stopping the process to meet the requirements in different scenarios. In addition, the proxy support unit allows the system to browse anonymously through a proxy server, effectively reducing the risk of IP being blocked. Using the Glob pattern matching unit, users can conveniently specify the URL pattern, simplifying the data collection process for specific domains or directories. The built-in JavaScript automatic parsing unit in the system can parse the JavaScript files in the links, thereby extracting dynamic content and ensuring the integrity and accuracy of the data. The sitemap support unit enables the system to generate a sitemap, achieve comprehensive tracking of links, and ensure no data coverage is missed. Finally, the application of the caching mechanism in the intelligent caching unit improves the system's processing efficiency, reduces memory usage, and further optimizes the overall performance. Through the close cooperation of these components, the ultra-high-speed Internet spider system can efficiently and accurately complete the tasks of Internet data crawling and processing.

[0052] Then, during the processes of working node load, page processing, and traffic control, the connection number acquisition module, information volume acquisition module, usage rate acquisition module, page acquisition module, broadband acquisition module, and task acquisition module are used to respectively collect the real-time information data of the connection number of the load node, node load, processed pages, information volume of the crawling task, CPU and operating memory usage rate, number of processed pages and information processing speed, network broadband, and number of crawling tasks.

[0053] Then, the collected data is transmitted in real time to the data storage unit through the data transmission unit. Then, through the data acquisition module in the data analysis unit, the information data is obtained from the data storage unit. Further, the obtained information data is processed and analyzed by the analysis and judgment module.

[0054] After that, when the result analyzed and judged by the analysis and judgment module shows an abnormal situation, the abnormal alarm module will be triggered to alarm the background terminal.

[0055] The data receiving module in the remote monitoring unit can receive in real time the information data stored in the data storage unit and the analysis results of the data processing unit, and display the received information data on the background terminal display screen through the data display module.

[0056] The analysis and judgment module can obtain from the data acquisition module the number of connections of the load nodes, the amount of information of the node load, and the usage rate information data of the CPU and the operating memory, and obtain the load value of the load nodes through the analysis of the above-obtained data:

[0057]

[0058] Then, by comparing the load values of each node, the best node for data information loading is selected:

[0059] Node=min{node1,node2,…,node n}

[0060] Among them, node j is the load value of the jth load node, C j is the current number of connections of the jth load node, C max is the maximum number of connections among all load nodes, time is the total duration of data collection, is the amount of information data loaded by the jth load node at the ith second, is the CPU usage rate of the jth load node, is the operating memory usage rate of the jth load node, and Node is the best node for data information loading.

[0061] The analysis and judgment module can obtain from the data acquisition module the amount of information of the processing page, the number of processing pages, and the information processing speed information data, and obtain the page processing index through the analysis of the above-obtained data. If:

[0062]

[0063] It indicates that the page information processing is stable. If It indicates that the page information processing is unstable, the page information processing speed is slow, and it is necessary to provide improved broadband or increase the operating memory to optimize the page information processing efficiency. At the same time, the abnormal alarm module will be triggered to alarm the background terminal. Among them, E is the page processing index, Q is the total number of processing pages, H k is the amount of information data of the kth page, V k is the information data processing speed of the kth page, B is the network bandwidth, is the CPU usage rate of the kth page, is the running memory usage rate of the k-th page, α is a preset conversion coefficient, and δ is a preset threshold parameter for page processing.

[0064] The analysis and judgment module can obtain the information volume of the crawling task, network bandwidth, and the quantity information data of the crawling task from the data acquisition module, and analyze and obtain the traffic control index through the above-obtained data analysis. If:

[0065]

[0066] It indicates that the crawling information traffic control is stable. If It indicates that the crawling information traffic control is unstable, the crawling information traffic is too fast, pause, restart or stop the process as needed. At the same time, the exception alarm module will be triggered to alarm the background terminal. Wherein, T is the traffic control index, B is the network bandwidth, b is the bandwidth required for each task to be crawled, N is the total number of crawling tasks, β is a preset redundancy coefficient for the impact of network fluctuations and server response delays on task crawling, time is the total duration of data collection, M i is the information data volume crawled at the i-th second, m i is the information data volume processed at the i-th second, M max is the maximum information data volume crawled within the time of time, m min is the minimum information data volume processed within the time of time, is a preset threshold parameter for traffic control. [[ID=?]] [[ID=?]]

[0067] The above-described embodiments merely represent one implementation manner of the present invention, and the description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention should be subject to the appended claims.

Claims

1. An Internet spider system based on efficient data extraction and processing, characterized in that, Including: Bare-metal software unit, workload unit, page processing unit, custom scraping unit, traffic control unit, proxy support unit, pattern matching unit, automatic parsing unit, map support unit, intelligent caching unit, data collection unit, and data processing unit; Bare-metal software unit: Written in the Rust programming language to provide predictable performance, memory safety, concurrency, and cross-platform compatibility; Workload unit: Used to distribute the workload across multiple nodes to improve processing efficiency and scalability; Page processing unit: Used to enable the system to process pages in parallel; Custom scraping unit: Used for users to customize the scraping settings for each path; Traffic control unit: Used to allow users to manually control the crawling operation; Proxy support unit: Used to support proxy servers and enable anonymous browsing; Pattern matching unit: Used for users to specify URL patterns using the Glob syntax to simplify data collection for specific domains or directories; Automatic parsing unit: Used to parse linked JavaScript files through a dedicated JavaScript engine to allow extraction of dynamic content; Map support unit: Used to generate a sitemap, enable link tracking, and ensure comprehensive data coverage; Intelligent caching unit: Used for the system to adopt a caching mechanism to improve processing efficiency and reduce memory usage; Data collection unit: Used to collect real-time the information data generated during the process of extracting and analyzing Internet information data; Data processing unit: Used to process and analyze the real-time collected information data to obtain processing and analysis results. At the same time, forward the obtained processing and analysis results to the remote monitoring unit.

2. The Internet spider system based on efficient data extraction and processing according to claim 1, characterized in that: The present system further includes: a data transmission unit, a data storage unit, and a remote monitoring unit; Data transmission unit: Used to transmit in real-time the information data collected by the data collection unit; Data storage unit: Used to store the information data transmitted in real-time by the data transmission unit; Remote monitoring unit: Used to display in real-time at the background terminal the information data stored by the data storage unit and the analysis results of the data processing unit. At the same time, when the result analyzed by the data processing unit is an abnormal result, issue an alarm to the background terminal.

3. An Internet spider system based on efficient data extraction and processing according to claim 2, characterized in that: The data collection unit includes: a connection number collection module, an information volume collection module, a usage rate collection module, a page collection module, a broadband collection module, and a task collection module; Connection number collection module: Used to collect in real-time the connection number information data of the load nodes; Information volume collection module: Used to collect in real-time the information volume data of node load, processed pages, and crawling tasks; Usage rate collection module: Used to collect in real-time the usage rate information data of the CPU and the running memory; Page collection module: Used to collect in real-time the number of processed pages and the information processing speed information data; Broadband collection module: Used to collect in real-time the network broadband information data; Task collection module: Used to collect in real-time the number information data of crawling tasks.

4. An Internet spider system based on efficient data extraction and processing according to claim 3, characterized in that: The data processing unit includes: a data acquisition module, an analysis and judgment module, and a result forwarding module; Data acquisition module: used to acquire the information data stored in the data storage unit in real time; Analysis and judgment module: used to analyze and judge the information data obtained by the data acquisition module, so as to obtain the result of the analysis and judgment; Result forwarding module: used to forward the result analyzed and judged by the analysis and judgment module to the remote monitoring unit.

5. An Internet spider system based on efficient data extraction and processing according to claim 4, characterized in that: The remote monitoring unit includes: a data receiving module, a data display module, and an abnormal alarm module; Data receiving module: used to receive in real time the information data stored in the data storage unit and the analysis result information of the data processing unit; Data display module: used to display the information data received by the data receiving module on the background terminal display screen; Abnormal alarm module: used to send an alarm prompt to the background terminal when the result obtained by the analysis and judgment module is an abnormal result.

6. An Internet spider system based on efficient data extraction and processing according to claim 5, characterized in that: The analysis and judgment module can obtain from the data acquisition module the connection number of the load node, the amount of information of the node load, and the usage rate information data of the CPU and the running memory, and obtain the load value of the load node through the analysis of the data obtained above: Then, by comparing the load values of each node, select the best node for data information loading: Node = min{node1, node2, …, node n} where node j is the load value of the j-th load node, C j is the current connection count of the j-th load node, C max is the maximum connection count among all load nodes, time is the total duration of data collection, is the amount of information data of the j-th load node at the i-th second, is the CPU usage rate of the j-th load node, is the running memory usage rate of the j-th load node, Node is the best node for loading data information.

7. An Internet spider system based on efficient data extraction and processing according to claim 6, characterized in that: The analysis and judgment module can obtain from the data acquisition module the amount of information of the processing page, the number of processing pages, and the information processing speed information data, and obtain the page processing index through the analysis of the data obtained above. If: It indicates that the page information processing is stable, where E is the page processing index, Q is the total number of processed pages, H k is the amount of information data of the k-th page, V k is the information data processing speed of the k-th page, B is the network bandwidth, is the CPU usage rate of the k-th page, is the running memory usage rate of the k-th page, α is a preset conversion coefficient, and δ is a preset threshold parameter for page processing.

8. An Internet spider system based on efficient data extraction and processing according to claim 7, characterized in that: The analysis and judgment module can obtain from the data acquisition module the amount of information of the crawling task, the network bandwidth, and the number of crawling tasks information data, and obtain the traffic control index through the analysis of the data obtained above. If: It indicates that the crawling information traffic control is stable, where T is the traffic control index, B is the network bandwidth, b is the bandwidth required for each crawling task, N is the total number of crawling tasks, β is the redundancy coefficient for the preset network fluctuation and server response delay affecting task crawling, time is the total duration of data collection, M i is the amount of information data crawled at the i-th second, m i is the amount of information data processed at the i-th second, M max is the maximum amount of information data crawled within the time of time, m min is the minimum amount of information data processed within the time of time, is the threshold parameter for the preset traffic control.

Images