Distributed webpage information automatic acquisition and scheduling method and device based on multi-source hierarchical analysis, processor and storage medium thereof

Abstract

The invention relates to a distributed webpage information automatic collecting and scheduling method based on multi-source hierarchical analysis, which comprises the following steps: receiving to-be-processed data through a link analysis layer, and classifying the obtained to-be-processed data according to different dimensions; a page capture layer executes a hierarchical capture task according to an input result sent by the link analysis layer, respectively collects data according to link types, and schedules in combination with task priorities; carrying out element extraction on the capturing result of the page capturing layer by an element extraction layer according to key contents, secondary elements and associated elements, and filtering out irrelevant elements contained in the capturing result; the quality evaluation layer performs crawling quality judgment on the structured data, expands a valuable link according to a search engine rule, and finally stores and submits an evaluation result to the decision optimization layer; and the decision optimization layer automatically adjusts parameters of a capture strategy and an extraction algorithm, and constructs an adaptive optimization mechanism to realize continuous improvement of the system in a large-scale and diversified scene.

Description

Technical Field

[0001] This invention relates to the field of Internet data acquisition and processing technology, and in particular to distributed automated hierarchical parsing and scheduling technology for data. Specifically, it refers to a distributed automated acquisition and scheduling method, device, processor, and computer-readable storage medium for web page information based on multi-source hierarchical parsing. Background Technology

[0002] Existing web scraping technologies mainly rely on two types of methods: one is a rule-based scraping method, which typically uses techniques such as regular expressions, XPath, and CSS selectors to parse the webpage structure into a template. This method can be implemented quickly in scenarios where the page structure is stable and the update frequency is low, but its portability and robustness are poor. Once the page structure of the target site changes, manual intervention and rule rewriting are required, making it difficult to adapt to large-scale, dynamically changing scraping needs.

[0003] Another approach is based on distributed crawler frameworks such as Scrapy and Heritrix. These systems typically improve crawling efficiency through task distribution and parallel scheduling, but lack intelligent parsing and monitoring capabilities for the crawling targets. In practical applications, problems such as incorrect page content extraction, duplicate crawling, and a large number of broken links can easily occur, leading to a waste of storage and bandwidth resources.

[0004] In recent years, with the development of artificial intelligence, some data collection methods based on machine learning or large language models have emerged, such as using AI script generators to automatically write parsing logic. These methods improve webpage adaptation capabilities to some extent, but due to the need for a large number of labeled samples and computing resources, the training and maintenance costs are extremely high. Furthermore, these methods often focus on single-source, single-page data collection, making it difficult to achieve structured extraction from cross-site and cross-level pages, resulting in insufficient practicality in large-scale, general data collection scenarios.

[0005] In summary, existing web scraping technologies have the following prominent problems: (1) Severe rule dependence: The data collection task is highly dependent on fixed templates and rules and lacks dynamic adaptability.

[0006] (2) Low resource utilization: The lack of effective monitoring and scheduling of the collection targets can easily lead to duplicate collection and waste of resources.

[0007] (3) Insufficient intelligence: Although AI-driven parsing methods have made progress, they suffer from high costs and poor generalization.

[0008] (4) Lack of universality: It is difficult to achieve stable and accurate content extraction in complex web page environments with multiple sources and multiple levels.

[0009] Therefore, there is an urgent need for a new data acquisition framework that can integrate multi-source hierarchical analysis, intelligent scheduling, and large-model-driven optimization to improve the stability, versatility, and automation of data acquisition, thereby meeting the needs of large-scale data acquisition and processing. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the prior art and provide a distributed automated web page information collection and scheduling method, device, processor and computer-readable storage medium based on multi-source hierarchical parsing.

[0011] To achieve the above objectives, the present invention provides a distributed automated web page information collection and scheduling method, apparatus, processor, and computer-readable storage medium based on multi-source hierarchical parsing as follows: The main feature of this distributed automated web page information collection and scheduling method based on multi-source hierarchical parsing is that the method includes the following steps: (1) Receive the data to be processed through the link resolution layer, and classify the obtained data to be processed according to different dimensions and complete the link resolution; (2) The page crawling layer performs hierarchical crawling tasks based on the input results sent by the link parsing layer, collects data according to the link type, and schedules the tasks according to the priority to achieve efficient coverage; (3) The element extraction layer extracts elements from the page crawling layer according to the key content, secondary elements and related elements, filters out irrelevant elements contained therein, saves all extraction results and generates structured data for quality assessment. (4) The quality assessment layer performs crawling quality judgment on the structured data, identifies the links, articles, advertising content, risk control prompts contained therein, expands valuable links according to search engine rules, and finally saves the assessment results and submits them to the decision optimization layer. (5) The decision optimization layer receives the judgment and extraction results and automatically adjusts the capture strategy and extraction algorithm parameters to build an adaptive optimization mechanism so as to realize the continuous improvement of the system in large-scale and diversified scenarios.

[0012] Preferably, step (1) specifically comprises: Users import target task links into the system, and the link resolution layer performs hierarchical parsing processing, including domain name analysis, extraction, and classification, to complete the classification and parsing of the data to be processed.

[0013] Preferably, step (2) specifically includes: After the link resolution process is completed, the page crawling layer initiates crawling tasks according to the three levels of domain name, navigation bar, and article details page. If the crawling fails, the system automatically triggers a retry mechanism and reports the crawling results to the decision optimization layer based on the crawling situation. Specifically, according to the default Internet crawling rules of the Robots Exclusion Protocol, the site's robots.txt file is extracted, the sitemaps page is extracted according to the rules, and the links contained in the sitemaps are sent to the element extraction layer.

[0014] Preferably, after the complete HTML page is collected, step (3) proceeds to the page parsing stage, and links are extracted in the following manner: The system first traverses the current page's DOM tree structure and extracts all... Tags and other elements that may have hyperlinks are identified by using URL pattern regular expressions, deep path structure calculations, and semantic feature recognition of link text and surrounding tags to comprehensively determine the key content and directly remove functional links.

[0015] Preferably, step (3) further includes filtering irrelevant elements in the following manner: (3.1) First, based on the feature identifiers of common advertising and recommendation components, nodes that clearly belong to advertising or promotional content are initially filtered; (3.2) Combining the DOM structure and positional characteristics of the page, calculate the relative position of the relevant elements in the page hierarchy, and mark the nodes in the typical functional areas of the header, sidebar, and footer as having low relevance to the main text; (3.3) When text appears on the page that is extremely short, has low text density, contains a large number of repeated symbols, or has no semantic fragments, it is judged as non-text noise and automatically excluded; (3.4) Through the comprehensive judgment of the above structural features, semantic features and layout features, stable filtering of irrelevant content can be achieved.

[0016] Preferably, step (3) further includes locating text elements in the following manner: Priority from <h1> , <title>The title is retrieved from the tag; if it does not exist, the most likely title is inferred from the candidate nodes based on the text block length, font size, and its hierarchical position in the DOM structure.< / title> Time and author information are extracted through regular expression matching of various date types and authorship patterns, as well as contextual clues. The text region is located based on paragraph density, text distribution continuity, and text ratio characteristics. or The node also identifies images and attachment links related to the main text.

[0017] Preferably, step (3) further includes generating the key content XPath in the following manner: The system records successfully parsed XPaths and their structural features for direct reuse in pages with similar structures. When changes in page structure cause existing XPaths to fail to match, the system automatically generates new candidate XPaths based on node structure similarity and contextual text features, and verifies their extraction performance. Subsequently, the system dynamically updates the credibility of each XPath based on its success rate and stability, reduces the weight of failed rules, and increases the priority of stable rules, thereby continuously maintaining the accuracy, stability, and adaptability of page parsing.

[0018] Preferably, step (4) specifically includes: (4.1) Plain text region recognition in the main text area: Based on the XPath rules generated in the previous step, a text extraction tool is introduced as a supplement for the main text area to ensure compatibility with complex pages; (4.2) Ad and Redundant Content Filtering: Remove non-core information containing scripts, style sheets, and ads, and establish regular expression filtering rules for common ad templates; (4.3) Text cleaning and standardization: Remove HTML tags and keep only plain text. Process special symbols and escape characters and replace them with standard characters. At the same time, standardize the time format to ensure consistency in subsequent analysis. (4.4) Extraction of related elements: Extract the img and attatchments tags from the page, extract the links from them, download the corresponding files, perform MD5 calculation and save them, and finally form JSON format.

[0019] Preferably, step (5) specifically includes: (5.1) Rule optimization and automatic correction: Based on the field completeness of the key element extraction results, abnormal records are automatically identified and abnormal text is marked as parsing failure samples. The system sends the parsing failure sample and its original HTML together into the structure comparison process to identify whether there are template changes on the page. (5.2) The system generates candidate XPaths based on DOM features using the parsing model, and performs independent parsing and field coverage evaluation for each XPath. When the parsing effect of a candidate XPath is better than the original strategy, the system writes it into the backup strategy pool and verifies it in parallel on subsequent pages of the same domain or similar structure. If the strategy maintains a high parsing success rate in continuous samples, the system automatically replaces it with the main parsing strategy. (5.3) Template learning and adaptation: The system performs cluster analysis on the text length, similarity features and structural features of page content to identify and classify page types that may belong to invalid content; for structural templates that are determined to be stable and reusable, the system caches them in the form of template fingerprints. When the same domain name or a page matching the structural template appears later, the system will skip the parsing or remove the corresponding link; at the same time, the system will automatically mark pages with excessively short text, highly repetitive content and abnormal advertising ratios, in order to continuously correct the collection strategy and improve the overall parsing efficiency and quality. (5.4) Unified language storage and translation: The system first identifies the language of the page and performs unified translation processing on texts in different languages. At the same time, it generates cross-language alignable semantic vectors to realize the storage and comparison of multilingual content in the same semantic space. (5.5) Continuous improvement and monitoring: The system compares the historical and current parsing results within the set time window, and automatically identifies the decline in the parsing ability of a site by monitoring the changes in the parsing success rate; once the success rate of a site is lower than the threshold, the system will trigger an early warning and execute strategy adjustments, including enabling alternative parsing paths or regenerating parsing rules; relevant failed samples will enter the model learning process to update the parsing model parameters.

[0020] The main feature of this distributed web page information automated collection and scheduling device based on multi-source hierarchical parsing is that the device includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing described above.

[0021] The main feature of this distributed web page information automated collection and scheduling processor based on multi-source hierarchical parsing is that the processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing described above are implemented.

[0022] The main feature of this computer-readable storage medium is that it stores a computer program that can be executed by a processor to implement the steps of the above-described distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing.

[0023] The distributed web page information automated collection and scheduling method, device, processor, and computer-readable storage medium based on multi-source hierarchical parsing of the present invention have the following technical advantages compared with the prior art: (1) Improved crawling efficiency: Through the layered input and hierarchical crawling mechanism, tasks can be flexibly scheduled according to domain name, column and other levels, avoiding resource waste and greatly improving the efficiency and coverage of large-scale data crawling.

[0024] (2) Data quality assurance: By integrating search engine optimization rules and quality assessment algorithms, redundant and low-quality information is effectively eliminated, ensuring that the data entering subsequent processing has higher accuracy and reliability.

[0025] (3) Improved element extraction accuracy: By continuously optimizing the element recognition model using positive and negative samples, the accuracy of extracting key page information such as title, body text, tags, and time is significantly improved.

[0026] (4) Adaptive optimization capability: Through the feedback mechanism of the optimization layer, the system can automatically adjust the crawling and extraction strategy under different data environments and task requirements to achieve continuous learning and performance improvement.

[0027] (5) Stable and continuous data acquisition: This invention ensures efficient data acquisition while taking into account data quality and element extraction accuracy, ultimately forming an efficient, stable and scalable data acquisition and parsing solution. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the processing flow of the distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing of the present invention.

[0029] Figure 2 This is a flowchart of a specific embodiment of the present invention. Detailed Implementation

[0030] To more clearly describe the technical content of the present invention, the following description is provided in conjunction with specific embodiments.

[0031] Before describing the embodiments of the present invention in detail, it should be noted that, in the following, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0032] This distributed automated webpage information collection and scheduling method based on multi-source hierarchical parsing employs the following technical solution: First, the collection objects are classified at the link parsing layer. Then, the page crawling layer performs hierarchical crawling at the domain, section, and article levels, and the quality of the crawled content is judged based on search engine optimization rules. Subsequently, the collection and parsing process is dynamically optimized based on the judgment results to achieve continuous crawling of deep data. During the page parsing stage, webpage elements are structurally extracted and identified, and the extraction results are used to construct positive and negative samples. The element extraction model is continuously trained and optimized to improve the overall parsing accuracy and the system's adaptability. The specific steps of this method are as follows: Step 1: Receive the data to be processed through the link resolution layer, and classify it according to different dimensions such as domain name, column or section to ensure that the subsequent processing is targeted and hierarchical.

[0033] Step 2: The crawling layer executes hierarchical crawling tasks based on the input results, collecting data according to domain level and column structure, and scheduling based on task priority to achieve efficient coverage.

[0034] Step 3: Based on the past and present analysis of the crawled results, and based on the relationship analysis of sibling elements and superior and subordinate elements, extract the XPath of key element positions, filter out low-value elements, save all the filtered results, generate structured data, and distribute it for quality assessment.

[0035] Step 4: Assess the quality of the crawled structured data, identify links, articles, advertising content, risk control prompts, and other information, expand valuable links according to search engine rules, and finally save and submit the evaluation results to the optimization decision layer.

[0036] Step 5: The optimization layer receives the judgment and extraction results, automatically adjusts the crawling strategy and extraction algorithm parameters, builds an adaptive optimization mechanism, and realizes continuous improvement of the system in large-scale and diversified scenarios.

[0037] In a specific embodiment of the present invention, taking the input of a website domain name as an example, the overall data collection method of the present invention includes the following steps: 1. Input and Parsing Users import target task links into the system, for example: The system splits the link https: / / example.com / politics / china / 20250917 / article123.html into: Extract the domain name portion (e.g., https: / / example.com); Extract the section path (e.g., / politics / china); Extract the detailed page path (e.g., / politics / china / 20250917 / article123.html).

[0038] Through hierarchical parsing, the system can clearly define the hierarchical structure of domain names, sections, and specific articles, providing a basis for subsequent data collection tasks.

[0039] 2. Hierarchical crawling The system initiates data collection tasks at three levels: domain name, navigation bar, and article details page. If collection fails, the system automatically triggers a retry mechanism and reports the collection results to the decision optimization layer based on the collection situation. (1) Collect the domain page html, and after collection, send the page to the element extraction layer.

[0040] (2) Collect the navigation bar page, and after collection, send the page to the element extraction layer.

[0041] (3) Collect the article details page, remove useless tags after collection, and send the page to the element extraction layer.

[0042] Based on the Robots Exclusion Protocol (REP) default web crawler rules, the robots.txt file of the site is extracted, the sitemaps page is extracted according to the rules, and the links in the sitemaps are sent to the element extraction layer.

[0043] 3. Page processing and element parsing After acquiring the complete HTML page, the system enters the page parsing phase. The goal of this step is to transform the raw webpage into structured data that can be further processed. (1) Link extraction: The system first traverses the page's DOM tree structure and extracts all links. Tags and other elements that may have hyperlink attributes (such as buttons and jump links in scripts) are comprehensively judged as potential article detail pages through URL pattern regular expressions (identifying dates, IDs, content page suffixes, etc.), deep path structure calculation (determining list pages / detail pages based on hierarchy), and semantic feature recognition of link text and surrounding tags (such as the appearance of keywords like article, news, detail, read more, etc.); functional links such as ad jumps, empty links, login and registration links are directly eliminated.

[0044] (2) Irrelevant Element Filtering: The system identifies and removes non-text modules on the page through multi-dimensional features. First, based on the feature identifiers of common advertising and recommendation components (such as ID / class containing keywords such as ad, adsense, sponsor, promo, banner, etc.), nodes that are obviously advertising or promotional content are initially filtered. Second, combined with DOM structure and position features, the relative position of elements in the page hierarchy is calculated, and nodes in typical functional areas such as headers, sidebars, and footers are marked as having low relevance to the main text. Furthermore, the system evaluates the text density and content features of nodes. When there are extremely short text lengths, excessively low text density, a large number of repeated symbols, or semantically meaningless fragments (such as "—", "···", "more>>"), they are judged as non-text noise and automatically excluded. Through the comprehensive judgment of the above structural features, semantic features, and layout features, stable filtering of irrelevant content is achieved.

[0045] (3) Text element location and XPath generation: After removing invalid elements, the system identifies the key structures of the page text. Titles are prioritized from... <h1> 、 <title> The title is retrieved from the source; if it doesn't exist, the most likely title is inferred from candidate nodes based on the text block length, font size, and its position in the DOM. Time and author information are extracted using regular expression matching of various date types and authorship patterns, as well as contextual clues. The main text area is located based on features such as paragraph density, text distribution continuity, and text-to-tag ratio.< / title> or The system identifies nodes and links to images and attachments related to the main text. XPath generation and maintenance utilize a "rule reuse and automatic correction" approach. Successfully parsed XPaths and their structural features are recorded for direct reuse on structurally similar pages. When changes in page structure cause existing XPath matches to fail, the system automatically generates new candidate XPaths based on node structural similarity and contextual text features, and verifies their extraction performance. Subsequently, the system dynamically updates the credibility of each XPath based on its success rate and stability, reducing the weight of failed rules and increasing the priority of stable rules, thereby continuously maintaining the accuracy, stability, and adaptability of page parsing.

[0046] 4. Content cleaning and quality assessment (1) Plain text region recognition in the main text area: Based on the XPath rules generated in the previous step, text extraction tools such as goose3 / trafilatura are introduced as supplements for the main text area to ensure compatibility of complex pages.

[0047] (2) Filtering of advertising and redundant content: Remove non-core information such as scripts, style sheets, and advertisements, and establish regular expression filtering rules for common advertising templates (such as "Continue reading" and "Sponsored links").

[0048] (3) Text cleaning and standardization: Remove HTML tags and keep only plain text. Process special symbols and escape characters (such as , &) and replace them with standard characters. At the same time, standardize the time format (convert to UTC+8) to ensure consistency in subsequent analysis. (4) Extraction of related elements: Extract the img and attributions tags from the page, extract the links from them, download the corresponding files, perform MD5 hashing on all elements, and save them to form a JSON format. { "title": "Example Article Title", author: "Zhang San", "publish_time": "2025-09-17T10:30:00+08:00", "content": "This is the main content..." "images": [ {"url": "http: / / example.com / img1.jpg","md5": "d81qra3kb3kasjskg3kdk2kd","ext":"jpg"} ], "attatchments": [ {"url":"http: / / example.com / name.docx","md5":d81qra3kb3kasjskg3kdk2kd","ext":"docx"} ], } 5. Use the results for further optimization. After content extraction and cleaning are completed, the system feeds the results back to the optimization stage. This optimization module consists of a rule adaptive engine, a template clustering subsystem, a multilingual semantic normalization subsystem, and a parsing quality monitor. By continuously learning the mapping relationship between "failed samples - HTML structure - successfully parsed samples," it achieves more stable and accurate data collection. (1) Rule optimization and automatic correction: Based on the completeness of the fields in the extracted results (such as missing titles, empty body text, etc.), abnormal records are automatically identified and marked as "parsing failure samples". The system sends the sample and its original HTML together into the structure comparison process to identify whether there are template changes on the page. Subsequently, the system generates candidate XPaths based on DOM features from the parsing model, and performs independent parsing and field coverage evaluation on each XPath. When the parsing effect of the candidate XPath is better than the original strategy, the system writes it into the backup strategy pool and verifies it in parallel on subsequent pages of the same domain or similar structure. If the strategy maintains a high parsing success rate in continuous samples, the system automatically replaces it with the main parsing strategy to achieve adaptive correction to changes in page structure, thereby reducing manual maintenance and improving parsing stability.

[0049] (2) Template Learning and Adaptation: The system performs cluster analysis on the text length, similarity features, and structural features of page content to identify and classify page types that may belong to invalid content. For structural templates that are determined to be stable and reusable, the system caches them as template fingerprints. When a page with the same domain name or matching the structural template appears later, the system can directly skip parsing or remove the corresponding link to reduce invalid parsing overhead. At the same time, the system will automatically mark pages with excessively short text, highly repetitive content, and abnormal advertising ratios to continuously correct the collection strategy and improve the overall parsing efficiency and quality.

[0050] (3) Unified Language Storage and Translation: The collected content may involve multiple languages. The system first identifies the language of the page and performs unified translation processing on the text in different languages. At the same time, it generates cross-language aligned semantic vectors, realizing the storage and comparison of multilingual content in the same semantic space. In this way, the system can achieve cross-language consistency in subsequent retrieval, similarity calculation and content filtering, thereby improving the adaptability and parsing stability in international site collection scenarios.

[0051] (4) Continuous Improvement and Monitoring: Within a set time window, the system compares historical and current parsing results and automatically identifies a decline in a site's parsing capability by monitoring changes in the parsing success rate. Once the success rate of a site falls below a threshold, the system will trigger an alert and execute strategy adjustments, including enabling alternative parsing paths or regenerating parsing rules. Relevant failed samples will enter the model learning process to update the parsing model parameters, enabling the system to have adaptive and self-evolving capabilities during continuous operation, thereby maintaining parsing stability and reducing manual intervention.

[0052] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0053] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution device.

[0054] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0055] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0056] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "embodiment," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0057] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

[0058] In this specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations can be made without departing from the spirit and scope of the invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive. < / h1> < / h1>

Claims

1. A distributed automated web page information collection and scheduling method based on multi-source hierarchical parsing, characterized in that, The method includes the following steps: (1) Receive the data to be processed through the link resolution layer, and classify the obtained data to be processed according to different dimensions and complete the link resolution; (2) The page crawling layer performs hierarchical crawling tasks based on the input results sent by the link parsing layer, collects data according to the link type, and schedules the tasks according to the priority to achieve efficient coverage; (3) The element extraction layer extracts elements from the page crawling layer according to the key content, secondary elements and related elements, filters out irrelevant elements contained therein, saves all extraction results and generates structured data for quality assessment. (4) The quality assessment layer performs crawling quality judgment on the structured data, identifies the links, articles, advertising content, risk control prompts contained therein, expands valuable links according to search engine rules, and finally saves the assessment results and submits them to the decision optimization layer. (5) The decision optimization layer receives the judgment and extraction results and automatically adjusts the capture strategy and extraction algorithm parameters to build an adaptive optimization mechanism so as to realize the continuous improvement of the system in large-scale and diversified scenarios.

2. The distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing according to claim 1, characterized in that, The specific steps (1) are as follows: Users import target task links into the system, and the link resolution layer performs hierarchical parsing processing, including domain name analysis, extraction, and classification, to complete the classification and parsing of the data to be processed.

3. The distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing according to claim 2, characterized in that, The specific steps (2) are as follows: After the link resolution process is completed, the page crawling layer initiates crawling tasks according to the three levels of domain name, navigation bar, and article details page. If the crawling fails, the system automatically triggers a retry mechanism and reports the crawling results to the decision optimization layer based on the crawling situation. Specifically, according to the default Internet crawling rules of the Robots Exclusion Protocol, the site's robots.txt file is extracted, the sitemaps page is extracted according to the rules, and the links contained in the sitemaps are sent to the element extraction layer.

4. The distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing according to claim 3, characterized in that, After the complete HTML page is collected, step (3) proceeds to the page parsing stage, and links are extracted in the following manner: The system first traverses the current page's DOM tree structure and extracts all... Tags and other elements that may have hyperlinks are identified by using URL pattern regular expressions, deep path structure calculations, and semantic feature recognition of link text and surrounding tags to comprehensively determine the key content and directly remove functional links.

5. The distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing according to claim 4, characterized in that, Step (3) further includes filtering irrelevant elements in the following manner: (3.1) First, based on the feature identifiers of common advertising and recommendation components, nodes that clearly belong to advertising or promotional content are initially filtered; (3.2) Combining the DOM structure and positional characteristics of the page, calculate the relative position of the relevant elements in the page hierarchy, and mark the nodes in the typical functional areas of the header, sidebar, and footer as having low relevance to the main text; (3.3) When text appears on the page that is extremely short, has low text density, contains a large number of repeated symbols, or has no semantic fragments, it is judged as non-text noise and automatically excluded; (3.4) Through the comprehensive judgment of the above structural features, semantic features and layout features, stable filtering of irrelevant content can be achieved.

6. The distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing according to claim 4, characterized in that, Step (3) further includes locating text elements in the following manner: Priority from <h1>、 <title> The title is retrieved from the tag; if it does not exist, the most likely title is inferred from the candidate nodes based on the text block length, font size, and its hierarchical position in the DOM structure.< / title> Time and author information are extracted through regular expression matching of various date types and authorship patterns, as well as contextual clues. The text region is located based on paragraph density, text distribution continuity, and text ratio characteristics. or The node also identifies images and attachment links related to the main text.

7. The distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing according to claim 4, characterized in that, Step (3) also includes generating the key content XPath in the following manner: The system records successfully parsed XPaths and their structural features for direct reuse in pages with similar structures. When changes in page structure cause existing XPaths to fail to match, the system automatically generates new candidate XPaths based on node structure similarity and contextual text features, and verifies their extraction performance. Subsequently, the system dynamically updates the credibility of each XPath based on its success rate and stability, reduces the weight of failed rules, and increases the priority of stable rules, thereby continuously maintaining the accuracy, stability, and adaptability of page parsing.

8. The distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing according to claim 7, characterized in that, Step (4) specifically includes: (4.1) Plain text region recognition in the main text area: Based on the XPath rules generated in the previous step, a text extraction tool is introduced as a supplement for the main text area to ensure compatibility with complex pages; (4.2) Ad and Redundant Content Filtering: Remove non-core information containing scripts, style sheets, and ads, and establish regular expression filtering rules for common ad templates; (4.3) Text cleaning and standardization: Remove HTML tags and keep only plain text. Process special symbols and escape characters and replace them with standard characters. At the same time, standardize the time format to ensure consistency in subsequent analysis. (4.4) Extraction of related elements: Extract the img and attatchments tags from the page, extract the links from them, download the corresponding files, perform MD5 calculation and save them, and finally form JSON format.

9. The distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing according to claim 8, characterized in that, Step (5) specifically includes: (5.1) Rule optimization and automatic correction: Based on the field completeness of the key element extraction results, abnormal records are automatically identified and abnormal text is marked as parsing failure samples. The system sends the parsing failure sample and its original HTML together into the structure comparison process to identify whether there are template changes on the page. (5.2) The system generates candidate XPaths based on DOM features using the parsing model, and performs independent parsing and field coverage evaluation for each XPath. When the parsing effect of a candidate XPath is better than the original strategy, the system writes it into the backup strategy pool and verifies it in parallel on subsequent pages of the same domain or similar structure. If the strategy maintains a high parsing success rate in continuous samples, the system automatically replaces it with the main parsing strategy. (5.3) Template learning and adaptation: The system performs cluster analysis on the text length, similarity features and structural features of page content to identify and classify page types that may belong to invalid content; for structural templates that are determined to be stable and reusable, the system caches them in the form of template fingerprints. When the same domain name or a page matching the structural template appears later, the system will skip the parsing or remove the corresponding link; at the same time, the system will automatically mark pages with excessively short text, highly repetitive content and abnormal advertising ratios, in order to continuously correct the collection strategy and improve the overall parsing efficiency and quality. (5.4) Unified language storage and translation: The system first identifies the language of the page and performs unified translation processing on texts in different languages. At the same time, it generates cross-language alignable semantic vectors to realize the storage and comparison of multilingual content in the same semantic space. (5.5) Continuous improvement and monitoring: The system compares the historical and current parsing results within the set time window, and automatically identifies the decline in the parsing ability of a site by monitoring the changes in the parsing success rate; once the success rate of a site is lower than the threshold, the system will trigger an early warning and execute strategy adjustments, including enabling alternative parsing paths or regenerating parsing rules; relevant failed samples will enter the model learning process to update the parsing model parameters.

10. A distributed web page information automated collection and scheduling device based on multi-source hierarchical parsing, characterized in that, The device includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing as described in any one of claims 1 to 9.

11. A distributed web page information automated collection and scheduling processor based on multi-source hierarchical parsing, characterized in that, The processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the distributed web page information automated collection and scheduling method based on multi-source hierarchical parsing as described in any one of claims 1 to 9. < / h1>

Images