Internet data intelligent acquisition method and system

Abstract

The invention discloses an intelligent Internet data acquisition method and system, and relates to the technical field of Internet data acquisition, and the method comprises the steps that a user defines parameters of an acquisition task through a graphical configuration tool, and generates a task description file; based on the task description file, performing webpage structure analysis on the target website by using a pre-trained machine learning model, and generating a webpage classification report; starting a multi-thread concurrent processing mechanism according to an acquisition rule, executing cross-website parallel data acquisition, and performing load balancing and fault-tolerant processing of tasks through a distributed architecture; and filtering and cleaning the processed data, outputting high-quality data after standardization processing, and generating a task completion report. According to the method, the user operation is simplified through graphical configuration, the analysis capability on a complex webpage structure is enhanced by utilizing a pre-trained machine learning model and a dynamic content detection algorithm, the performance is improved by adopting a multi-thread and distributed architecture, and the data quality is improved through refined processing.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet data collection, and in particular to an Internet data intelligent collection method and system. Background Art

[0002] With the rapid development of Internet technology, network data has become an indispensable source of information for many fields such as corporate decision-making, market analysis, and product development. As an important bridge connecting online information and offline applications, Internet data collection technology has evolved from simple web crawlers to complex intelligent data mining systems in recent years. Early data collection mainly relied on manually written scripts to capture data for specific websites. This method is not only inefficient, but also lacks flexibility when facing changes in web page structure. With the continuous advancement of big data and artificial intelligence technologies, modern Internet data collection technology has begun to integrate advanced algorithms such as machine learning and deep learning to achieve intelligent analysis and dynamic adaptation of web page content. In particular, for websites with a large amount of dynamically loaded content, traditional methods are often difficult to deal with effectively, while intelligent collection technology can accurately capture the complete information of the page by simulating user interaction behavior, greatly improving the quality and efficiency of data collection.

[0003] However, the existing Internet data collection technology still has many shortcomings. First, most solutions fail to fully consider the user experience. Users often need to have a certain programming foundation and technical knowledge when configuring collection tasks, which increases the usage threshold for ordinary users. Secondly, when dealing with large-scale data collection tasks, the performance of existing technologies is not satisfactory, especially in terms of concurrent processing capabilities and system stability. Many systems are difficult to meet the requirements of high concurrency and high availability. In addition, the ability to parse complex web page structures is also a major challenge, especially when the target website is frequently updated or adopts advanced anti-crawler strategies, traditional collection methods often seem to be unable to cope with it. Finally, data quality control is also a major shortcoming of existing technologies. How to filter out effective information from massive data and reduce the interference of noise data is the key to improving the value of data. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an Internet data intelligent collection method and system to solve the problems of high user configuration difficulty and poor data quality control in the prior art.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In the first aspect, an embodiment of the present invention provides an intelligent Internet data collection method, which includes: a user defines the parameters of the collection task through a graphical configuration tool and generates a task description file; based on the task description file, a pre-trained machine learning model is used to parse the web page structure of the target website and generate a web page classification report; according to the web page classification report, collection rules that adapt to different web page structures and dynamic loading modes are automatically generated through a dynamic content detection algorithm; according to the collection rules, a multi-threaded concurrent processing mechanism is started to execute parallel data collection across websites, and load balancing and fault-tolerant processing of tasks are performed through a distributed architecture; the processed data is filtered and cleaned, and high-quality data after normalization is output to generate a task completion report.

[0008] As a preferred solution of the method for intelligently collecting Internet data of the present invention, wherein: the task description file includes collection parameters and target website information;

[0009] The task completion report includes the collection status, data volume and abnormal conditions.

[0010] As a preferred solution of the method for intelligently collecting Internet data of the present invention, the step of generating a task description file includes:

[0011] React.js combined with Ant Design is used to build the user interface. When the page is loaded, the useEffect hook is used to initialize the user interface and receive the URL of the target website entered by the user.

[0012] After the user selects the page type, common fields are automatically recommended based on the K nearest neighbor algorithm;

[0013] Users add new fields by selecting recommended fields through the interface, and associate fields with corresponding DOM nodes by clicking specific elements on the page;

[0014] After the field configuration is completed, the bound fields and DOM nodes are saved in a structured manner;

[0015] Automatically detect whether the page is dynamically loaded based on the target URL entered by the user;

[0016] If dynamic loading is detected, the user is prompted to select the appropriate handling method;

[0017] If no dynamic loading is detected, the static page acquisition mode is automatically adopted to directly grab the HTML content of the page;

[0018] Integrate user-selected fields, DOM nodes, and dynamic loading modes to generate collection rule files;

[0019] The user selects the task scheduling frequency through the interface, and the user's selection is converted into a Cron expression to form scheduling information;

[0020] Merge the collection rule file with the scheduling information to generate the final task description file.

[0021] As a preferred solution of the method for intelligently collecting Internet data of the present invention, the step of generating a web page classification report includes:

[0022] Read the task description file generated by the graphical configuration tool, extract the target website URL, the fields to be collected and their corresponding DOM nodes, and key information of the dynamic loading mode;

[0023] Use Python's requests library to initiate an HTTP request to the target website URL and obtain the HTML source code;

[0024] For websites that support dynamic loading, use Selenium tools to simulate user behavior, trigger JavaScript events, fully load the HTML source code, and obtain the HTML source code;

[0025] The pre-trained BERT model is combined with the TF-IDF algorithm, and the dynamic semantic feature fusion algorithm is used to parse the obtained HTML source code. The dynamic semantic feature fusion algorithm is:

[0026]

[0027] Among them, F represents the final dynamic semantic feature fusion score, i represents the index variable of the semantic feature, j represents the index variable of the visual feature, and w i represents the importance weight of the i-th feature, S i represents the score of the i-th semantic feature, N is the number of semantic features, and v j represents the strength value of the jth visual feature, M is the number of visual features, λ represents the parameter for adjusting the degree of difference, D represents the difference between the page content and the model training data set, and e -λ·D An exponential decay term representing the degree of difference;

[0028] Based on the analysis results, the webpage content is divided into functional areas, and the elements in each area are classified and labeled;

[0029] Generate detailed web page classification report based on classification annotation and preset business logic.

[0030] As a preferred solution of the method for intelligently collecting Internet data of the present invention, the generation of collection rules adapted to different web page structures and dynamic loading modes includes:

[0031] Read the functional areas of the divided web pages and the element classification annotation information from the web page classification report;

[0032] Use the dynamic content detection algorithm to analyze the dynamically loaded parts of the web page and identify the dynamically loaded content and loading mechanism. The dynamic content detection algorithm is:

[0033]

[0034] Where C represents the dynamic content detection score, t1 and t0 are the initial time and the end time respectively, P(t) represents the total number of bytes of the web page content at time t, Indicates the rate of change of the web page content size over time, d k represents the content increment of the kth dynamic loading, β represents the normalization constant, γ represents the nonlinear weight adjustment parameter of the loading times and content increment, and α represents the weight control parameter;

[0035] For static content, directly specify the DOM node path and field name to be collected;

[0036] For dynamically loaded content, generate corresponding collection rules based on the dynamic content detection score C and loading mechanism;

[0037] According to the functional area division and element classification labeling in the web page classification report, the collection rules are optimized and the generated collection rules are stored in the database.

[0038] As a preferred solution of the Internet data intelligent collection method of the present invention, wherein: according to the collection rules, the multi-threaded concurrent processing mechanism is started to execute parallel data collection across websites, and the load balancing and fault tolerance processing of tasks are performed through the distributed architecture, including:

[0039] Use Python's json library to read the collection rule file in the database;

[0040] Parse the file content to extract the target website URL, the fields to be collected and their corresponding DOM nodes, and key information of the dynamic loading mode;

[0041] Evaluate the complexity of each task using a predefined dynamic content detection algorithm;

[0042] Monitor system resources in real time and record resource fluctuation values;

[0043] Based on the complexity and resource fluctuation value of each task, the optimal number of threads is predicted through a linear regression model;

[0044] Dynamically adjust the number of threads in the thread pool based on the calculated number of threads;

[0045] Each thread uses Python's requests library to initiate HTTP requests according to the collection rules assigned to it to obtain the HTML source code of the target web page;

[0046] After each working node receives a task, if the current load exceeds the preset threshold, the node will reject the new task;

[0047] Introduce a heartbeat detection mechanism and a data redundancy backup mechanism to regularly check the status of each working node and simultaneously collect data between different nodes.

[0048] As a preferred solution of the method for intelligently collecting Internet data of the present invention, the step of generating a task completion report includes:

[0049] Deploy the data collection task in a Docker container, with each node running a container;

[0050] When the heartbeat detection mechanism confirms a node failure, it automatically stops the container on the node and uses the container orchestration tool to automatically restart the container on the failed node to resume the data collection task.

[0051] Collect all collected data from the temporary cache of each node, and perform data filtering, data verification and exception processing on the data;

[0052] Summarize the overall status of the task, the amount of data collected, and abnormal situations to generate a task completion report.

[0053] In a second aspect, the present invention provides an Internet data intelligent collection system, comprising: a task configuration module, a web page structure analysis module, a collection rule generation module, a data collection execution module and a report generation module; the task configuration module is used for the user to define the parameters of the collection task through a graphical configuration tool and generate a task description file; the web page structure analysis module is used for, based on the task description file, using a pre-trained machine learning model to perform web page structure analysis on the target website and generate a web page classification report; the collection rule generation module is used for, based on the web page classification report, automatically generating collection rules that adapt to different web page structures and dynamic loading modes through a dynamic content detection algorithm; the data collection execution module is used for, based on the collection rules, starting a multi-threaded concurrent processing mechanism, executing parallel data collection across websites, and performing load balancing and fault-tolerant processing of tasks through a distributed architecture; the report generation module is used for filtering and cleaning the processed data, outputting high-quality data after normalization processing, and generating a task completion report.

[0054] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the method for intelligent collection of Internet data as described in the first aspect of the present invention is implemented.

[0055] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the method for intelligent collection of Internet data as described in the first aspect of the present invention is implemented.

[0056] The beneficial effects of the present invention are as follows: the present invention greatly reduces the difficulty of user operation by providing a graphical configuration tool, so that even users without professional skills can easily define complex collection tasks; at the same time, the present invention uses a pre-trained machine learning model to deeply analyze the structure of a web page, and combines a dynamic content detection algorithm to automatically generate collection rules that adapt to different web page structures and dynamic loading modes, which significantly improves the intelligence level of data collection; in addition, the system also adopts a multi-threaded concurrent processing mechanism and a distributed architecture to ensure high efficiency and reliability when executing large-scale data collection tasks; finally, by performing fine processing on the collected data, the data quality is effectively improved, ensuring the high availability of the output data. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0058] Figure 1 This is a flow chart of the method for intelligent collection of Internet data in Example 1.

[0059] Figure 2 This is a module diagram of the Internet data intelligent collection method in Example 1. DETAILED DESCRIPTION

[0060] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0061] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0062] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0063] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides an Internet data intelligent collection method, comprising the following steps:

[0064] S1. The user defines the parameters of the acquisition task through the graphical configuration tool and generates a task description file.

[0065] Furthermore, React.js combined with Ant Design is used to build the user interface. When the page is loaded, the useEffect hook is used to initialize the user interface and receive the URL of the target website entered by the user;

[0066] Preferably, the core advantage of React.js lies in its componentized architecture, which allows developers to divide the interface into independent, reusable components. This makes the code easy to maintain, expand, and refactor; Ant Design provides a wealth of ready-made UI components, such as forms, buttons, tables, cards, navigation bars, etc. These components are beautifully designed and powerful, and developers can use them directly, greatly improving development efficiency. Using React.js combined with AntDesign to build a user interface can significantly improve development efficiency, ensure code maintainability, and provide a consistent user experience.

[0067] After the user selects the page type (such as an e-commerce page), common fields are automatically recommended based on the K nearest neighbor algorithm. The expression is:

[0068]

[0069] in, represents the set of recommended fields, k represents the number of most similar tasks, y(x i ) represents the field configuration of similar tasks, x i represents the i-th task similar to the current task x, where i represents the index variable of the task;

[0070] Users add new fields by selecting recommended fields through the interface, and associate fields with corresponding DOM nodes by clicking specific elements on the page;

[0071] After the field configuration is completed, the bound fields and DOM nodes are saved in a structured manner;

[0072] Automatically detect whether the page contains dynamic loading based on the target URL entered by the user, such as using a crawler tool (such as Selenium) to simulate user behavior, trigger JavaScript events, and detect whether the page contains dynamically loaded content;

[0073] If dynamic loading is detected, the user is prompted to select an appropriate handling method, such as waiting for the page to be fully rendered or listening for specific AJAX requests;

[0074] If no dynamic loading is detected, the static page acquisition mode is automatically adopted to directly grab the HTML content of the page, skipping unnecessary dynamic loading processing configuration, simplifying user operations and improving task execution efficiency;

[0075] Integrate user-selected fields, DOM nodes, and dynamic loading modes to generate collection rule files;

[0076] The user selects the task scheduling frequency (such as daily, hourly, etc.) through the interface, and the user's selection is converted into a Cron expression to form scheduling information;

[0077] Preferably, a Cron expression is a string format used to represent time scheduling, which can accurately define the execution of a task at a specific time point or period. Usually, the user selects the frequency of task execution (such as daily, hourly, etc.) in the front-end interface, and the system converts these selections into expressions that conform to the Cron syntax to accurately control the time of task execution.

[0078] Merge the collection rule file with the scheduling information to generate the final task description file.

[0079] S2. Based on the task description file, use the pre-trained machine learning model to parse the web page structure of the target website and generate a web page classification report.

[0080] Furthermore, the task description file generated by the graphical configuration tool is read to extract the target website URL, the fields to be collected and their corresponding DOM nodes, and key information of the dynamic loading mode.

[0081] Use Python's requests library to initiate HTTP requests to the target website and obtain the HTML source code;

[0082] For websites that support dynamic loading, use Selenium tools to simulate user behavior, trigger JavaScript events, fully load the HTML source code, and obtain the HTML source code;

[0083] The pre-trained BERT model is combined with the TF-IDF algorithm, and the dynamic semantic feature fusion algorithm is used to parse the obtained HTML source code to identify the main components of the page;

[0084] The dynamic semantic feature fusion algorithm is:

[0085]

[0086] Among them, F represents the final dynamic semantic feature fusion score, which is used to reflect the similarity between the web page content and the model training data set, i represents the index variable of the semantic feature, j represents the index variable of the visual feature, and w i represents the importance weight of the i-th feature, which is used to reflect the importance of different features in parsing. i represents the score of the i-th semantic feature, which is used to measure the significance of the feature in the text. N is the number of semantic features, that is, the total number of semantic features considered during the parsing process. j represents the strength value of the jth visual feature, which is used to evaluate the visual appeal of the page elements. M is the number of visual features, that is, the total number of visual features considered during the parsing process. λ represents the parameter for adjusting the degree of difference, which is used to control the impact of page changes on the parsing effect. D represents the difference between the page content and the model training data set, which is used to reflect the similarity between the current page and the training sample. -λ·D An exponential decay term representing the degree of difference, used to represent the negative impact of page content changes on parsing results;

[0087] Based on the analysis results, the web page content is divided into functional areas, and the elements in each area are classified and labeled. Through the detailed classification of web page content, clear guidance can be provided for subsequent data collection, ensuring that the collected data is both comprehensive and accurate.

[0088] Based on classification annotations and preset business logic, a detailed web page classification report is generated, including an overview of each part of the content, keyword analysis, link distribution, etc.

[0089] It should be noted that "preset business logic" refers to the logic for further processing and organizing the parsing and classification results according to specific business needs and rules. These logics ensure that the generated report can meet the needs of actual applications and improve the accuracy and practicality of the report. The specific steps for generating a detailed web page classification report according to the preset business logic are as follows:

[0090] First, the classification annotation results are obtained from the parsing and classification annotation process in the previous step, and the classification annotation results of each functional area (such as the head, body, and bottom) are extracted from the parsed HTML document;

[0091] Specifically, use the BeautifulSoup library to parse the HTML document, traverse all the header, body, and footer tags, extract the child elements (such as titles, paragraphs, images, links, etc.) under each tag, and store this information in a list;

[0092] Next, remove irrelevant or redundant information to ensure the neatness and accuracy of the report, and remove advertisement-related elements and duplicate paragraphs or images.

[0093] Specifically, define a function to determine whether an element is an advertisement. If the class or id of the element contains the word "ad", it is considered an advertisement and filtered out. At the same time, define another function to determine whether an element is duplicate. If the text content and tag type of the new element are the same as those of the existing element, it is considered duplicate and filtered out;

[0094] After data cleaning is completed, organize the cleaned data in a preset format for subsequent processing and display. Organize the element classification information of each functional area into a dictionary, including information such as the tag, text content, and attributes of each element.

[0095] Specifically, traverse the cleaned data, organize the tag, text content, and element list of each functional area into a dictionary, and store these dictionaries in a list;

[0096] After data organization is completed, generate a content quality score according to the dynamic semantic feature fusion score F. This step requires defining a scoring criterion and generating a corresponding score based on the value of F

[0097] Specifically, if F > 1.0, the score is "high-quality content";

[0098] If 0.8 < F ≤ 1.0, the score is "medium-quality content";

[0099] If F ≤ 0.8, the score is "low-quality content";

[0100] Finally, generate a detailed web page classification report based on the organized data and content quality score. This step requires organizing information such as the classification annotation results and content quality score into a JSON-format report and saving it to a file. Specifically, create a dictionary containing the classification annotation results of each functional area, the fields to be collected, and their content quality scores, and then save this dictionary as a.JSON-format file.

[0101] S3. According to the web page classification report, automatically generate collection rules that adapt to different web page structures and dynamic loading modes through a dynamic content detection algorithm.

[0102] Furthermore, the functional areas of the web pages divided from the web page classification report, as well as the element classification annotation information;

[0103] Use the dynamic content detection algorithm to analyze the dynamically loaded parts of the web page and identify the dynamically loaded content and loading mechanism. The dynamic content detection algorithm is:

[0104]

[0105] Among them, C represents the dynamic content detection score, which reflects the activity level of dynamically loaded content in the web page. t1 and t0 are the initial time and end time, respectively, which represent the time window of dynamic loading. P(t) represents the total number of bytes of web page content at time t, which reflects the size change of web page content. Indicates the rate of change of the size of the web page content over time, indicating the speed of content update, d k represents the content increment of the kth dynamic loading, represents the size of the newly added content each time it is loaded, β represents the normalization constant, which is used to adjust the relative influence of the content increment, γ represents the nonlinear weight adjustment parameter of the number of loading times and the content increment, and α represents the weight control parameter, which is used to adjust the influence intensity of the dynamic loading content increment (the second term) in the whole formula;

[0106] For static content, directly specify the DOM node path and field name to be collected;

[0107] For dynamically loaded content, generate corresponding collection rules based on the dynamic content detection score C and loading mechanism;

[0108] According to the functional area division and element classification labeling in the web page classification report, the collection rules are optimized, redundant collection is reduced and the collection efficiency is improved, and the generated collection rules are stored in the database.

[0109] S4. Based on the collection rules, start the multi-threaded concurrent processing mechanism, execute parallel data collection across websites, and perform load balancing and fault tolerance of tasks through a distributed architecture.

[0110] Furthermore, Python's json library is used to read the collection rule file in the database;

[0111] Parse the file content to extract the target website URL, the fields to be collected and their corresponding DOM nodes, and key information of the dynamic loading mode;

[0112] Evaluate the complexity of each task using a pre-defined dynamic content detection algorithm (such as the C value in the previous step);

[0113] Monitor system resources in real time and record resource fluctuation value V;

[0114] It should be noted that real-time monitoring of system resources refers to determining the system resource indicators that need to be monitored, such as CPU usage, memory usage, disk I / O, network bandwidth, etc. The specific implementation steps are as follows:

[0115] Set the monitoring frequency, for example, monitor once every second or every 5 seconds. The selection of monitoring frequency needs to balance real-time performance and system overhead;

[0116] Use the tools or libraries provided by the system to obtain the current system resource information. For example, the psutil library can be used to easily obtain CPU usage and memory usage;

[0117] Record the resource information monitored each time to form a historical record of resource fluctuation values. These records can be stored in files, databases, or memory. For example, the CPU usage and memory usage monitored each time can be recorded in a CSV file;

[0118] Analyze the recorded resource fluctuation values ​​to identify the changing trends and abnormalities of resource usage. For example, you can calculate statistical indicators such as the average, maximum, and minimum values ​​of resource usage, or draw a time series graph of resource usage to better understand the system's resource usage.

[0119] It should also be noted that through the above steps, real-time monitoring of system resources can be achieved and resource fluctuation values ​​can be recorded, including determining monitoring indicators, setting monitoring frequency, obtaining system resource information, recording resource fluctuation values ​​and analyzing resource fluctuation values, ensuring that the monitoring data of system resources can be obtained and analyzed in a timely manner, providing strong support for subsequent resource management and task scheduling, helping to optimize system performance, and being able to promptly discover and resolve resource bottleneck problems.

[0120] Based on the complexity and resource fluctuation value of each task, the optimal number of threads T is predicted by a linear regression model;

[0121] Furthermore, the specific steps for building the linear regression model are as follows: First, the complexity of historical tasks, resource fluctuation values, and the number of threads actually used need to be collected. These data can be extracted from previous task records, including the complexity (C), CPU usage, memory usage, and the number of threads actually used for each task. For example, the following data can be recorded:

[0122] Task 1: complexity 0.8, CPU usage 70%, memory usage 60%, actual number of threads used 4; Task 2: complexity 0.5, CPU usage 50%, memory usage 40%, actual number of threads used 3; Task 3: complexity 1.2, CPU usage 80%, memory usage 70%, actual number of threads used 5; More data...

[0123] After collecting the data, we need to convert it into a format suitable for the linear regression model. Specifically, extract features (complexity, CPU usage, memory usage) and labels (the actual number of threads used). Organize this data into two lists, one for features and the other for labels.

[0124] After preparing the training data, use the linear regression model for training. The goal of the linear regression model is to find the linear relationship between the input features (complexity, CPU usage, memory usage) and the output label (the actual number of threads used). The process of training the model includes fitting the model parameters so that the model can predict a value close to the actual number of threads based on the input features.

[0125] The trained linear regression model can be used to predict the optimal number of threads for a new task. The specific steps are to extract the complexity and resource fluctuation values ​​of the current task, input these values ​​into the trained model, and the model will output the predicted number of threads. These predicted numbers of threads will be used to dynamically adjust the number of threads in the thread pool to optimize the execution efficiency of the task.

[0126] Dynamically adjust the number of threads in the thread pool based on the calculated number of threads;

[0127] If the calculated optimal number of threads is greater than the current maximum number of threads in the thread pool, increase the number of threads in the thread pool; if the calculated optimal number of threads is less than the current maximum number of threads in the thread pool, reduce the number of threads in the thread pool;

[0128] Submit tasks to the adjusted thread pool for execution, process the task results returned by the thread pool, and ensure that all tasks are completed successfully;

[0129] Each thread uses Python's requests library to initiate HTTP requests according to the collection rules assigned to it to obtain the HTML source code of the target web page;

[0130] After each working node receives a task, if the current load exceeds the preset threshold, the node can choose to reject the new task, thereby achieving dynamic load balancing;

[0131] The specific method of setting preset thresholds is as follows: First, analyze the system's hardware configuration, including the number of CPU cores, memory size, disk I / O speed, etc., to understand the system's maximum processing capacity and resource limitations. Check the number of cores of the system's CPU to determine the system's maximum processing capacity; check the memory size to ensure that there is enough memory to process the task; evaluate the disk I / O speed to ensure that data reading and writing will not become a bottleneck. Next, refer to historical data to understand the system's resource usage when processing similar tasks. Collect resource usage data for historical tasks, including CPU usage, memory usage, disk I / O, and network bandwidth, and analyze these data to find out the average and peak resource usage of the system when processing tasks. Based on the system's hardware configuration and historical data, set the initial preset threshold. Generally, the threshold for CPU usage can be set to around 80%, which means that when the CPU usage reaches 80%, the node will reject new tasks; the threshold for memory usage can also be set to around 80% to prevent insufficient memory from causing the system to crash. Adjust these thresholds according to actual conditions. For example, if the system has more idle time, the threshold can be appropriately lowered to improve resource utilization. Then, perform a stress test to verify whether the set thresholds are appropriate. Simulate high-load conditions, observe the system's performance when it reaches the preset threshold, record the system's performance indicators under different loads, including response time, throughput, etc., adjust the preset thresholds based on the test results to ensure that the system can still run stably under high load. Finally, dynamically adjust the preset thresholds based on the system's operating conditions. Monitor the system's resource usage in actual operation, and regularly evaluate the rationality of the preset thresholds. If it is found that the system performs poorly in certain situations, the thresholds can be adjusted appropriately to optimize system performance. Machine learning methods can be used to automatically adjust the thresholds based on historical data to adapt to different workloads.

[0132] Introduce a heartbeat detection mechanism and a data redundancy backup mechanism to regularly check the status of each working node and simultaneously collect data between different nodes;

[0133] Preferably, once the heartbeat detection mechanism finds that a node is offline or faulty, it immediately reallocates the tasks on the node. This mechanism improves the fault tolerance and stability of the system.

[0134] Preferably, the data redundancy backup mechanism can restore data through other nodes even if some nodes fail, which increases the reliability of the system and the security of the data.

[0135] S5. Filter and clean the processed data, output the standardized high-quality data, and generate a task completion report.

[0136] Going a step further, the data collection tasks are deployed in Docker containers, with each node running a container;

[0137] When the heartbeat detection mechanism confirms a node failure, it automatically stops the container on the node and uses a container orchestration tool (such as Kubernetes, Docker Swarm, etc.) to automatically restart the container on the failed node and resume the data collection task. The specific operations include:

[0138] In Kubernetes, you can use Deployment and ReplicaSet to manage the lifecycle of containers.

[0139] In Docker Swarm, you can use services to manage the life cycle of containers.

[0140] Preferably, a heartbeat detection mechanism is implemented to regularly check the status of each working node. A timed task can be used to send a heartbeat signal. If a node does not respond within a predetermined time, the node is considered to be faulty.

[0141] All collected data is collected from the temporary cache of each node and temporarily stored in the local cache. The central node regularly collects data from the temporary cache of each node and performs data filtering, data verification and exception processing on the data;

[0142] Data filtering to remove invalid or duplicate data;

[0143] Data validation to ensure data integrity and accuracy. For example, using regular expressions, data validation algorithms, etc.

[0144] Exception handling, marking or deleting data that does not meet the requirements, for example, recording logs of abnormal data for subsequent analysis;

[0145] Summarize the overall status of the task, the amount of data collected, and abnormal situations to generate a task completion report.

[0146] The present embodiment also provides an Internet data intelligent collection system, including: a task configuration module, a web page structure analysis module, a collection rule generation module, a data collection execution module and a report generation module; the task configuration module is used for the user to define the parameters of the collection task through a graphical configuration tool and generate a task description file; the web page structure analysis module is used to use a pre-trained machine learning model to perform web page structure analysis on the target website based on the task description file, and generate a web page classification report; the collection rule generation module is used to automatically generate collection rules that adapt to different web page structures and dynamic loading modes through a dynamic content detection algorithm based on the web page classification report; the data collection execution module is used to start a multi-threaded concurrent processing mechanism based on the collection rules, execute parallel data collection across websites, and perform load balancing and fault-tolerant processing of tasks through a distributed architecture; the report generation module is used to filter and clean the processed data, output high-quality data after normalization processing, and generate a task completion report.

[0147] This embodiment also provides a computer device, which is suitable for the case of the Internet data intelligent collection method, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the Internet data intelligent collection method proposed in the above embodiment.

[0148] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0149] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for realizing intelligent collection of Internet data as proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0150] In summary, the present invention greatly reduces the difficulty of user operation by providing a graphical configuration tool, so that even users without professional skills can easily define complex collection tasks; at the same time, the present invention uses a pre-trained machine learning model to deeply analyze the structure of a web page, and combines it with a dynamic content detection algorithm to automatically generate collection rules that adapt to different web page structures and dynamic loading modes, which significantly improves the intelligence level of data collection; in addition, the system also adopts a multi-threaded concurrent processing mechanism and a distributed architecture to ensure efficiency and reliability when executing large-scale data collection tasks; finally, by fine-tuning the collected data, the data quality is effectively improved, ensuring the high availability of the output data.

[0151] Example 2, referring to Table 1, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of the Internet data intelligent collection method are provided.

[0152] Two representative websites were selected as the objects of data collection: a large e-commerce platform (Site A) and a news information website (Site B). These two websites represent web pages with complex dynamic loading mechanisms and relatively static content websites, respectively. This embodiment aims to prove through actual operation that the Internet data intelligent collection method provided by the present invention has significant advantages in data collection efficiency, data quality, and dynamic content processing capabilities compared to traditional data collection methods.

[0153] First, through the graphical configuration tool, the parameters of the collection task were defined according to the characteristics of the target website, including but not limited to the collection frequency, target website URL, specific fields to be collected, etc. For example, for Station A, the main focus is on information such as product prices, inventory status, and user reviews; while for Station B, the focus is on article titles, publishing time, author information, etc. At this stage, React.js combined with Ant Design was used to build a user-friendly interface to ensure the convenience and accuracy of parameter setting.

[0154] Next, based on the generated task description file, the pre-trained BERT model combined with the TF-IDF algorithm was used to analyze the webpage structure of the target website. In particular, in view of the complex dynamic loading mechanism of Site A, the Selenium tool was used to simulate real user behavior to ensure the complete loading of the HTML source code. On this basis, the obtained HTML source code was deeply analyzed through the dynamic semantic feature fusion algorithm to achieve accurate classification of webpage content.

[0155] Subsequently, based on the web page classification report, the collection rules adapted to different web page structures and dynamic loading modes were automatically generated. In this process, the dynamic content detection algorithm played a key role. By evaluating the change rate of web page content size over time, the number of loads, and content increments, it accurately identified the dynamically loaded content and loading mechanism, thus laying a solid foundation for subsequent data collection.

[0156] After the above preparations were completed, the multi-threaded concurrent processing mechanism was started to perform parallel data collection across websites. The distributed architecture not only achieved load balancing of tasks, but also effectively improved the fault tolerance of the system. In addition, the heartbeat detection mechanism and data redundancy backup mechanism were introduced to ensure the stability and reliability of the data collection process.

[0157] Finally, the collected data was strictly filtered and cleaned to remove invalid or erroneous information and generate a high-quality data set. During the entire process, the system automatically generated a task completion report, which recorded in detail the collection status, data volume, abnormal conditions and other information, providing a reliable basis for subsequent data analysis.

[0158] The details are shown in Table 1 below:

[0159] Table 1 Experimental data comparison table

[0160]

[0161] By comparing the data in the table, it can be seen that the Internet data intelligent collection method of the present invention has shown obvious advantages in many aspects. First, in terms of data volume, whether it is station A or station B, the amount of data collected using the method of the present invention is significantly higher than that of the traditional method. Specifically, the amount of data at station A increased by 38%, while that at station B increased by 17.6%. This shows that the present invention can more efficiently mine and obtain valuable information, especially when faced with complex web page structures, its advantages are more prominent.

[0162] Secondly, from the perspective of the number of abnormal situations, the method of the present invention only had 2 abnormalities at station A, while no abnormalities occurred at station B. In contrast, the traditional method had 5 and 3 abnormalities at the two stations, respectively. This result not only reflects the stability of the method of the present invention during the data collection process, but also reflects its powerful exception handling capabilities. By introducing the heartbeat detection mechanism and the data redundancy backup mechanism, the present invention can promptly discover and solve possible problems, ensuring the smooth progress of the data collection task.

[0163] Furthermore, the data of average response time further verified the high efficiency of the method of the present invention. At station A, the average response time of the method of the present invention was 350ms, which was 46.1% faster than the traditional method; at station B, the figure was 120ms, which was also better than the 300ms of the traditional method. This is due to the fact that the present invention adopts a multi-threaded concurrent processing mechanism and a distributed architecture, which not only achieves load balancing of tasks, but also greatly shortens the time of data collection.

[0164] In summary, this embodiment fully demonstrates the significant advantages of the Internet data intelligent collection method of the present invention compared with traditional methods in terms of improving data collection efficiency, ensuring data quality and enhancing system stability. These improvements not only meet the needs of modern Internet data collection, but also provide strong technical support for applications in related fields.

[0165] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for intelligent collection of Internet data, characterized in that: include: The user defines the parameters of the acquisition task through the graphical configuration tool and generates a task description file; Based on the task description file, use the pre-trained machine learning model to analyze the web page structure of the target website and generate a web page classification report; According to the web page classification report, the dynamic content detection algorithm automatically generates collection rules that adapt to different web page structures and dynamic loading modes; According to the collection rules, start the multi-threaded concurrent processing mechanism, execute parallel data collection across websites, and perform load balancing and fault tolerance of tasks through the distributed architecture; Filter and clean the processed data, output high-quality data after normalization, and generate a task completion report.

2. The method for intelligently collecting Internet data according to claim 1, characterized in that: The task description file includes acquisition parameters and target website information; The task completion report includes the collection status, data volume and abnormal conditions.

3. The method for intelligently collecting Internet data according to claim 2, characterized in that: The generating task description file comprises: React.js combined with Ant Design is used to build the user interface. When the page is loaded, the useEffect hook is used to initialize the user interface and receive the URL of the target website entered by the user. After the user selects the page type, common fields are automatically recommended based on the K nearest neighbor algorithm; Users add new fields by selecting recommended fields through the interface, and associate fields with corresponding DOM nodes by clicking specific elements on the page; After the field configuration is completed, the bound fields and DOM nodes are saved in a structured manner; Automatically detect whether the page is dynamically loaded based on the target URL entered by the user; If dynamic loading is detected, the user is prompted to select the appropriate handling method; If no dynamic loading is detected, the static page acquisition mode is automatically adopted to directly grab the HTML content of the page; Integrate user-selected fields, DOM nodes, and dynamic loading modes to generate collection rule files; The user selects the task scheduling frequency through the interface, and the user's selection is converted into a Cron expression to form scheduling information; Merge the collection rule file with the scheduling information to generate the final task description file.

4. The method for intelligently collecting Internet data as claimed in claim 3, characterized in that: The generating of the web page classification report comprises: Read the task description file generated by the graphical configuration tool, extract the target website URL, the fields to be collected and their corresponding DOM nodes, and key information of the dynamic loading mode; Use Python's requests library to initiate an HTTP request to the target website URL and obtain the HTML source code; For websites that support dynamic loading, use Selenium tools to simulate user behavior, trigger JavaScript events, fully load the HTML source code, and obtain the HTML source code; The pre-trained BERT model is combined with the TF-IDF algorithm, and the dynamic semantic feature fusion algorithm is used to parse the obtained HTML source code. The dynamic semantic feature fusion algorithm is: Among them, F represents the final dynamic semantic feature fusion score, i represents the index variable of the semantic feature, j represents the index variable of the visual feature, and w i represents the importance weight of the i-th feature, S i represents the score of the i-th semantic feature, N is the number of semantic features, and v j represents the strength value of the jth visual feature, M is the number of visual features, λ represents the parameter for adjusting the degree of difference, D represents the difference between the page content and the model training data set, and e -λ·D An exponential decay term representing the degree of difference; Based on the analysis results, the webpage content is divided into functional areas, and the elements in each area are classified and labeled; Generate detailed web page classification report based on classification annotation and preset business logic.

5. The method for intelligently collecting Internet data as claimed in claim 4, characterized in that: The generation of collection rules adapted to different web page structures and dynamic loading modes includes: Read the functional areas of the divided web pages and the element classification annotation information from the web page classification report; Use the dynamic content detection algorithm to analyze the dynamically loaded parts of the web page and identify the dynamically loaded content and loading mechanism. The dynamic content detection algorithm is: Where C represents the dynamic content detection score, t1 and t0 are the initial time and the end time respectively, P(t) represents the total number of bytes of the web page content at time t, Indicates the rate of change of the web page content size over time, d k represents the content increment of the kth dynamic loading, β represents the normalization constant, γ represents the nonlinear weight adjustment parameter of the loading times and content increment, and α represents the weight control parameter; For static content, directly specify the DOM node path and field name to be collected; For dynamically loaded content, generate corresponding collection rules based on the dynamic content detection score C and loading mechanism; According to the functional area division and element classification labeling in the web page classification report, the collection rules are optimized and the generated collection rules are stored in the database.

6. The method for intelligently collecting Internet data according to claim 5, characterized in that: According to the collection rules, a multi-threaded concurrent processing mechanism is started to perform parallel data collection across websites, and load balancing and fault tolerance processing of tasks are performed through a distributed architecture, including: Use Python's json library to read the collection rule file in the database; Parse the file content to extract the target website URL, the fields to be collected and their corresponding DOM nodes, and key information of the dynamic loading mode; Evaluate the complexity of each task using a predefined dynamic content detection algorithm; Monitor system resources in real time and record resource fluctuation values; Based on the complexity and resource fluctuation value of each task, the optimal number of threads is predicted through a linear regression model; Dynamically adjust the number of threads in the thread pool based on the calculated number of threads; Each thread uses Python's requests library to initiate HTTP requests according to the collection rules assigned to it to obtain the HTML source code of the target web page; After each working node receives a task, if the current load exceeds the preset threshold, the node will reject the new task; Introduce a heartbeat detection mechanism and a data redundancy backup mechanism to regularly check the status of each working node and simultaneously collect data between different nodes.

7. The method for intelligently collecting Internet data according to claim 6, characterized in that: The generating of the task completion report comprises: Deploy the data collection task in a Docker container, with each node running a container; When the heartbeat detection mechanism confirms a node failure, it automatically stops the container on the node and uses the container orchestration tool to automatically restart the container on the failed node to resume the data collection task. Collect all collected data from the temporary cache of each node, and perform data filtering, data verification and exception processing on the data; Summarize the overall status of the task, the amount of data collected, and abnormal situations to generate a task completion report.

8. An Internet data intelligent collection system, based on the Internet data intelligent collection method according to any one of claims 1 to 7, characterized in that: include: Task configuration module, web page structure analysis module, collection rule generation module, data collection execution module and report generation module; The task configuration module is used for the user to define the parameters of the acquisition task through a graphical configuration tool and generate a task description file; The webpage structure analysis module is used to analyze the webpage structure of the target website using a pre-trained machine learning model based on the task description file and generate a webpage classification report; The collection rule generation module is used to automatically generate collection rules that adapt to different web page structures and dynamic loading modes through a dynamic content detection algorithm according to the web page classification report; The data collection execution module is used to start a multi-threaded concurrent processing mechanism according to the collection rules, execute parallel data collection across websites, and perform load balancing and fault tolerance processing of tasks through a distributed architecture; The report generation module is used to filter and clean the processed data, output high-quality data after normalization, and generate a task completion report.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for intelligent collection of Internet data as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for intelligently collecting Internet data as described in any one of claims 1 to 7 are implemented.

Images