Webpage information extraction method, device and equipment

Abstract

The application provides a webpage information extraction method, device and equipment, including: clustering uniform resource locators (URLs) of multiple webpages, generating at least one URL pattern, the similarity of the URL corresponding to the URL pattern is greater than or equal to a first similarity threshold and / or the webpage structure similarity is greater than or equal to a second similarity threshold; performing information extraction on a first webpage corresponding to a first URL pattern in the at least one URL pattern, and generating, according to the extracted rules, at least one of the following: a positioning rule of an element in the webpage, a data cleaning rule, and an element extraction pattern of an extraction result format specification; in response to a URL of a second webpage matching a second URL pattern in the at least one URL pattern, performing information extraction on the second webpage according to the element extraction pattern of the second URL pattern; or, in response to the URL of the second webpage not matching any URL pattern in the at least one URL pattern, performing information extraction on the second webpage, and generating an element extraction pattern of the second webpage, to improve information extraction efficiency.

Description

Technical Field

[0001] This application relates to the field of Internet information processing technology, and more specifically, to a method, apparatus, and device for extracting web page information. Background Technology

[0002] In the field of internet information retrieval and processing, automatically extracting structured information from web pages—a fundamental and crucial task—is web page information extraction. For example, through web page information extraction, key elements such as the title, abstract, author, date, and main text of a web page can be obtained. These key elements can then be applied to various scenarios, including internet content governance, search index construction, public opinion monitoring, knowledge graph production, intelligence analysis, and large-scale model corpus processing.

[0003] Currently, it is often necessary to write customized extraction rules for each webpage, and extract information from each webpage separately using these individual rules. This approach requires significant manual intervention and continuous investment in algorithms and maintenance personnel, thus resulting in high information extraction costs and low efficiency. Summary of the Invention

[0004] This application provides a method, apparatus, and device for extracting web page information. It can obtain a Uniform Resource Locator (URL) pattern by clustering the URLs of multiple web pages, and generate an element extraction pattern based on the URL pattern. This enables automated batch extraction of web pages with the same URL pattern. Therefore, it can solve the problems of high cost and low efficiency caused by writing customized extraction rules for each web page and extracting web page information from each web page through these extraction rules, thereby reducing the cost of web page information extraction and improving the efficiency of web page information extraction.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] According to one aspect of this application, a webpage information extraction method is provided, comprising: clustering the URLs of multiple webpages to generate at least one URL pattern, wherein the URL similarity of the webpages corresponding to each URL pattern is greater than or equal to a first similarity threshold and / or the webpage structure similarity of the corresponding webpages is greater than or equal to a second similarity threshold; for any first URL pattern in the at least one URL pattern, extracting information from any first webpage corresponding to the first URL pattern, generating an element extraction pattern corresponding to the first URL pattern according to the extraction rules of the information extraction, wherein the element extraction pattern includes at least one of the following: location rules for each element in the corresponding webpage, data cleaning rules, and format specifications for the extraction results; in response to a second webpage URL matching any second URL pattern in the at least one URL pattern, extracting information from the second webpage according to the element extraction pattern of the second URL pattern; or, in response to a second webpage URL not matching any of the at least one URL patterns, extracting information from the second webpage and generating an element extraction pattern corresponding to the second webpage.

[0007] According to one aspect of this application, a webpage information extraction apparatus is provided, comprising: a first generation module, configured to: cluster the URLs of multiple webpages to generate at least one URL pattern, wherein the URL similarity of the webpage corresponding to each URL pattern is greater than or equal to a first similarity threshold and / or the webpage structure similarity of the corresponding webpage is greater than or equal to a second similarity threshold; a second generation module, configured to: extract information from any first webpage corresponding to any first URL pattern in the at least one URL pattern, and generate an element extraction pattern corresponding to the first URL pattern according to the extraction rules of the information extraction, wherein the element extraction pattern includes at least one of the following: location rules for each element in the corresponding webpage, data cleaning rules, and format specifications for the extraction results; and an information extraction module, configured to: extract information from the second webpage according to the element extraction pattern of the second URL pattern in response to the URL of the second webpage matching any second URL pattern in the at least one URL pattern; or, extract information from the second webpage in response to the URL of the second webpage not matching any URL pattern in the at least one URL pattern, and generate an element extraction pattern corresponding to the second webpage.

[0008] According to one aspect of this application, an electronic device is provided, comprising: a processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the steps of the web page information extraction method described above.

[0009] According to one aspect of this application, a chip is provided, comprising: a processor for calling and running a computer program from a memory, such that the processor performs the steps of the web page information extraction method described above.

[0010] According to one aspect of this application, a computer-readable storage medium is provided for storing a computer program that causes a computer to perform the steps of the above-described web page information extraction method.

[0011] Based on the above technical solution, the Uniform Resource Locator (URL) values ​​of multiple web pages can be clustered to generate at least one URL pattern. The URL similarity of the corresponding web pages for each URL pattern is greater than or equal to a first similarity threshold, and / or the web page structure similarity of the corresponding web pages is greater than or equal to a second similarity threshold. For any first URL pattern among the at least one URL pattern, information extraction is performed on any first web page corresponding to the first URL pattern. An element extraction pattern corresponding to the first URL pattern is generated according to the information extraction rules. The element extraction pattern includes at least one of the following: location rules for each element in the corresponding web page, data cleaning rules, and format specifications for the extraction results. In response to a second web page's URL matching any second URL pattern among the at least one URL pattern, information extraction is performed on the second web page according to the element extraction pattern of the second URL pattern. Alternatively, in response to a second web page's URL not matching any of the at least one URL pattern, information extraction is performed on the second web page, generating an element extraction pattern corresponding to the second web page. In this process, URL patterns can be obtained by clustering the URLs of multiple web pages, and feature extraction patterns can be generated based on these patterns. This enables automated batch extraction of web pages with the same URL pattern. Therefore, it solves the problems of high cost and low efficiency caused by writing customized extraction rules for each web page and extracting information from each page using these rules, thereby reducing the cost and improving the efficiency of web page information extraction. Furthermore, this application considers the similarity of web pages with the same URL pattern in URL structure and / or web page structure. It reuses the feature extraction patterns (including location rules, cleaning rules, format specifications, etc.) under the same URL pattern for extraction. Therefore, it can also solve the limitations of unclear rule reuse boundaries and insufficient cross-template generalization ability caused by diverse web page structures and mixed template versions, improving the comprehensiveness and accuracy of extracting information from massive heterogeneous web pages.

[0012] Other features and advantages of the embodiments of this application will become apparent from the following detailed description, or may be learned in part by practice of this application.

[0013] It should be understood that the above general description and the following detailed description are merely exemplary and do not constitute a limitation on this application. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0015] Figure 1 This diagram illustrates an application scenario provided according to an embodiment of the present application. Figure 2 A flowchart illustrating a web page information extraction method according to an embodiment of this application is shown; Figure 3 A schematic diagram illustrating a web page information extraction method according to an embodiment of this application is shown; Figure 4 A block diagram of a web page information extraction apparatus according to an embodiment of this application is shown; Figure 5 A schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application is shown. Detailed Implementation

[0016] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make the description of this application more complete and to fully convey the concept of the exemplary embodiments to those skilled in the art. The accompanying drawings are schematic illustrations of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0017] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced with one or more specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0018] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different networks, processor devices, or microcontroller devices.

[0019] Figure 1 This illustration shows an application scenario diagram according to an embodiment of the present application, such as... Figure 1 As shown, this application scenario includes terminal 110 and server 120.

[0020] In some embodiments, the technical solution of this application is mainly applied to scenarios involving the extraction of web page information, such as the extraction of key elements such as title, body, author, and publication time from news and information web pages, the extraction of structured information such as product name, price, reviews, and specifications from e-commerce web pages, or the extraction of information such as user nickname, post content, publication time, and interaction data from forums and social web pages, but not limited to these.

[0021] In some embodiments, the technical solution of this application can be embedded in corresponding systems or devices such as search engine index building, public opinion monitoring system, knowledge graph production, large model corpus processing, intelligence analysis system, and information aggregation platform, for intelligent information extraction from web pages in corresponding scenarios.

[0022] For example, the technical solution of this application can be integrated into a search engine index building system to enhance the ability to extract structured information from massive web pages in batches, thereby improving index quality and update efficiency; it can be integrated into a public opinion monitoring platform for real-time information extraction and sentiment analysis of diverse web pages such as news, forums, and social media; it can be integrated into a knowledge graph production tool as a structured data extraction module to automatically extract entities and relationships from heterogeneous web pages; and it can be integrated into a large model corpus processing chain as a web page text cleaning and structured preprocessing module to output high-quality training corpus.

[0023] In some embodiments, the server 120 can obtain the URLs of multiple web pages from the terminal 110. The server 120 can cluster the URLs of the multiple web pages to generate at least one URL pattern. The URL similarity of the web pages corresponding to each URL pattern is greater than or equal to a first similarity threshold and / or the web page structure similarity of the corresponding web pages is greater than or equal to a second similarity threshold. For any first URL pattern in the at least one URL pattern, information extraction is performed on any first web page corresponding to the first URL pattern. According to the extraction rules of information extraction, an element extraction pattern corresponding to the first URL pattern is generated. The element extraction pattern includes at least one of the following: The items include: the location rules, data cleaning rules, and format specifications of the extraction results for each element in the corresponding webpage; in response to the URL of the second webpage matching any of the at least one URL patterns, information extraction is performed on the second webpage according to the element extraction pattern of the second URL pattern; or, in response to the URL of the second webpage not matching any of the at least one URL patterns, information extraction is performed on the second webpage to generate the element extraction pattern corresponding to the second webpage. In addition, the server 120 can also send the information extraction results, extraction rules, element extraction patterns, etc. to the terminal 110, which can display them.

[0024] It is understood that the above application scenario is merely an example and does not constitute a limitation on the web page information extraction method provided in the embodiments of this application. Furthermore, when the data, information, and instructions involved in the technical solution of this application are applied to specific products or technologies, user permission, consent, or authorization is required, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards.

[0025] The server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 110 can be a smartphone, tablet, laptop, desktop computer, smart speaker, or smartwatch, but is not limited to these. The terminal 110 and server 120 can be directly or indirectly connected via wired or wireless communication, and this application does not limit this connection.

[0026] For example, the technical solution of this application can also be executed through a web page information intelligent extraction system. This system can run on the aforementioned server 120 or on other devices with computing capabilities. For instance, the web page information intelligent extraction system can be deployed independently on a single physical server, distributed across a cluster of multiple server nodes, or deployed as a cloud service on a public cloud, private cloud, or hybrid cloud platform, providing web page information extraction services externally through an application programming interface (API). Alternatively, the system can be deployed locally on terminal 110, running as a software application on a smartphone, tablet, or laptop, utilizing the terminal's local computing resources to extract a small number of web pages in real time. This application does not limit the specific deployment method of the web page information intelligent extraction system.

[0027] It should be noted that the technical solutions in this application involve URLs, URL patterns, web pages, extraction rules, element extraction patterns, page features of web pages, and extraction results, as well as related data, information, requests, or instructions, all of which have been permitted, agreed to, or authorized by the user, or have been fully permitted, agreed to, or authorized by all parties. Furthermore, the collection, use, and processing of such information, requests, or instructions must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the multiple web pages, their URLs, and the extraction results involved in this application were all obtained with full authorization.

[0028] The specific implementation process of the embodiments of this application will be described in detail below.

[0029] Figure 2 A flowchart illustrating a webpage information extraction method according to an embodiment of this application is shown. This webpage information extraction method can be executed by a device with computing power or the aforementioned intelligent webpage information extraction system, for example, by the aforementioned terminal 110, server 120, or a combination of terminal 110 and server 120. (Refer to...) Figure 2 As shown, this webpage information extraction method may include at least steps S210 to S240, which are detailed below: In S210, the URLs of multiple web pages are clustered to generate at least one URL pattern. The URL similarity of the corresponding web page to each URL pattern is greater than or equal to a first similarity threshold and / or the web page structure similarity of the corresponding web page is greater than or equal to a second similarity threshold.

[0030] Clustering the URLs of multiple web pages can refer to dividing the URLs of multiple web pages (e.g., URL strings) into different categories based on the similarity of their structures and / or the similarity of the web page structures of the corresponding web pages.

[0031] URL patterns can refer to the general expression corresponding to each category formed after clustering (i.e., the general expression of URLs belonging to the same category), which can summarize the common structure of all URLs in that category. The different parts of all URLs in that category (such as dynamic parameters in the URL) can be identified by placeholders (such as date, id, etc.).

[0032] URL similarity of corresponding web pages can refer to the degree of similarity between two or more web pages in at least one aspect such as URL path structure, file name conventions, and parameter composition.

[0033] The structural similarity of corresponding web pages can refer to the degree of similarity between two or more web pages in at least one of the following aspects: Document Object Model (DOM) tree topology, visual layout features (e.g., the position (coordinates), size (width-to-height ratio), layout (reading order, multi-column / single-column), font size, color contrast, paragraph spacing, etc.), and HyperText Markup Language (HTML) structure.

[0034] The first similarity threshold and the second similarity threshold can be preset thresholds.

[0035] Specifically, the above-mentioned clustering of URLs of multiple web pages includes the following steps: by comparing at least one of the following in the preset hierarchical address of the URLs of multiple web pages (such as fixed string, variable parameter), address length (such as 8-digit date, variable length ID), address type (pure number, pure letter, mixed, special format), the URLs of multiple web pages are clustered (similar URLs are grouped into the same category, which can be that the similarity is greater than a threshold), and each category corresponds to at least one URL pattern.

[0036] Furthermore, the clustering results of URLs of multiple web pages obtained based on at least one of the following: address content, address length, and address type, can be updated / verified based on the web page structure similarity of the corresponding web pages. This prevents erroneous clustering due to coincidentally similar URL structures that result in different actual page structures. For example, the same website might return two completely different page templates under the same URL pattern (e.g., different test scenarios). In such cases, URL similarity alone cannot distinguish them, but by comparing web page structure similarity, they can be accurately separated into two different URL pattern categories.

[0037] The preset level address can be the first preset number of levels in the URL path hierarchy.

[0038] For example, the URLs of multiple web pages can be clustered using frequent itemset mining methods based on greedy algorithms or frequent pattern trees (FP-Tree).

[0039] For example, based on a delimiter, each URL in the URLs of multiple web pages can be split into multi-dimensional feature items; based on at least one of the following: the repetition frequency of each multi-dimensional feature item in the URLs of multiple web pages (which can reflect the number of times the feature item appears globally, i.e., in the URLs of multiple web pages; a high frequency indicates that it may be a fixed path segment), the path level sequence number (which can reflect the position of the feature item in the URL (level 1, level 2, etc.); earlier levels are usually more stable), and the path level priority (which can be a configurable weight parameter), a first-dimensional feature item is determined from each multi-dimensional feature item; based on the first-dimensional feature item, a set of multi-dimensional feature items is generated; and based on the similarity between the multi-dimensional feature items of each URL and the set of multi-dimensional feature items, the URLs of multiple web pages are clustered.

[0040] For example, the URL path can be split into multi-dimensional feature items according to separators (such as forward slashes " / "). For instance, the URL path " / api / user / list" can be split into the first dimension "api", the second dimension "user", and the third dimension "list". Alternatively, each dimension feature item can correspond to one or more path levels, such as "api+user". Then, invalid characters in the multi-dimensional feature items can be filtered out, retaining the first N core features (n is a positive integer). Feature items with higher path levels are selected first, thereby generating a set of preorder feature items for each URL (i.e., the set corresponding to the multi-dimensional feature items).

[0041] Next, the repetition frequency of each feature item in the multi-dimensional feature items can be counted, and a minimum support threshold can be pre-set (for example, a feature combination (corresponding to a feature item of one dimension or a combination of multiple feature items) must appear at least a certain number of times to be considered a meaningful pattern). Then, high-frequency features that meet the repetition frequency standard (e.g., greater than or equal to the minimum support threshold) in the preceding dimensions (e.g., some or all of the dimensions determined according to the corresponding path level priority and / or path level sequence number) are selected as the first dimension feature items, and an item header table is constructed accordingly. Next, each URL is scanned, and the feature items of each URL are sorted according to "preceding dimension priority (priority determined according to the corresponding path level priority and / or path level sequence number)" and "frequency (i.e., repetition frequency)". Then, through recursive mining, the multi-element frequent itemsets composed of the first dimension feature items are extracted, i.e., the dimension feature itemsets, such as the combination of the first two dimensions "api+user" or "api+order".

[0042] Next, the similarity between the dimensional features of each URL and the mined frequent itemsets can be calculated. If the similarity between the dimensional features and the mined frequent itemsets reaches a preset standard (e.g., greater than or equal to a threshold), it is assigned to the corresponding cluster. URLs that fail to match any frequent itemsets can be assigned to the default cluster. Finally, the aggregation results can be output, with each cluster corresponding to a URL pattern. For example, for multiple URLs belonging to the same cluster, the common fixed parts of the multiple URLs can be identified first, while the variable parts (i.e., different parts) can be identified and replaced with variable placeholders. Specifically, for each URL in the cluster, the protocol (http / https) and domain name can be ignored (or the domain name and protocol can be retained as part of the URL pattern), and the remaining path can be split into multiple levels by the separator " / ". The split results of multiple URLs are aligned hierarchically, and the values ​​at each position are compared layer by layer. For positions that change, the type characteristics of their values ​​can be analyzed (e.g., analysis dimensions include: length, format, character type (pure numbers, pure letters, mixed, special formats), and value range) to replace generalized wildcards with semantic variable names. The fixed parts (i.e., the parts common to multiple URLs) are preserved as is, and the changing parts are replaced with corresponding variable placeholders, and then concatenated to obtain the URL pattern.

[0043] In addition, URL patterns can be stored in two forms: variable placeholders (such as / news / {year} / {month} / {day} / {id}.html, which is easy for users to understand and manage intuitively) and regular expressions (which can facilitate fast matching by machines).

[0044] For example, the URLs of multiple web pages are as follows: https: / / news.shili / socialgd / 10000169 / 20260211 / 49244566.html; https: / / news.shili / socialgd / 10000169 / 20260211 / 49244979.html; https: / / news.shili / socialgd / 10000169 / 20251227 / 49113107.html; https: / / news.shili / socialgd / 10000169 / 20250826 / 48761234.html; If we ignore the HTTP layer and only compare the first four layers of the path, then the above URLs can be clustered into three categories: Category 1: [https: / / news.shili / socialgd / 10000169 / 20260211 / 49244566.html, https: / / news.shili / socialgd / 10000169 / 20260211 / 49244979.html] Category 2: [https: / / news.shili / socialgd / 10000169 / 20251227 / 49113107.html] Category 3: [https: / / news.shili / socialgd / 10000169 / 20250826 / 48761234.html] The fourth level of the URL above actually represents the date (in yyyyMMdd format), and the fifth level represents the article ID (a numeric string). Therefore, the above URLs can actually be grouped into one category and represented using a unified URL pattern as follows: http[s]: / / news.shili / socialgd / 10000169 / yyyyMMdd / \d+.html.

[0045] Before executing S210, you can collect the URLs of multiple web pages.

[0046] Specifically, before clustering the URLs of multiple web pages, a large number of valid URLs (i.e., URLs of multiple web pages) can be collected first. A valid URL is one whose corresponding web page contains actual content information, rather than an error page displaying a 404 error, an empty page, or a placeholder page indicating "no content available." To efficiently collect valid URLs, an asynchronous concurrent architecture can be adopted, based on asynchronous I / O (asyncio) and an asynchronous HTTP client (aiohttp) to achieve tens of thousands of concurrent requests. A single machine can achieve a processing capacity of over 1000 queries per second (QPS). Simultaneously, an intelligent scheduling mechanism can be equipped to prioritize URLs based on website weight, update frequency, and depth. In terms of anti-scraping measures, a comprehensive approach can be adopted, including rotating Internet Protocol (IP) proxy pools, continuously evaluating proxy quality, automatically removing invalid proxies, randomizing request headers (including diverse combinations of User-Agent (UA), Accept-Language, and Referer), and employing a progressive request strategy (using an exponential backoff algorithm and possessing honeypot detection capabilities). Furthermore, incremental updates are supported; that is, changes to webpage content can be detected by parsing the Last-Modified or Entity Tag (ETag) fields in the Hypertext Transfer Protocol (HTTP) response headers, thereby avoiding the repeated crawling of outdated pages.

[0047] In some embodiments, before collecting the URLs of multiple web pages, a process of obtaining permission, consent, or authorization from users or other parties (such as the content provider of the web page or its legally authorized third party) can be executed first. Specifically, in response to a request to collect the URLs of at least one web page and / or a request to extract information from at least one web page, an authorization interface can be displayed. The authorization interface can include at least: the purpose of data collection (e.g., to provide URL pattern clustering and information extraction functions, it is necessary to analyze the current web page's structure, URL, and content), the type of URL to be collected and the scope of data (e.g., specific fields to be extracted, such as: product title, price, review text, author, publication time; the type of URL to be collected can refer to the business or content category of the web page pointed to by the URL, such as product details page, list page, article page, or forum post page), and the method of data usage (e.g., what will happen to the extracted data). This process is used to generate feature extraction patterns, cluster analysis, and large model corpus processing; storage details (e.g., specifying whether data is stored, the storage period, and whether it is shared with third parties; data to be collected and extraction results saved until the user withdraws consent); and a user rights statement (e.g., methods for withdrawing consent, methods for requesting data deletion, and appeal channels). Then, in response to received consent authorization actions from users or other parties (e.g., clicking "Permit," "Agree," or "Authorize" on the authorization interface, or checking "I have read and agree"), it generates and stores corresponding authorization credentials (e.g., digital signatures or timestamped authorization records). Afterward, it can collect URLs from multiple web pages and execute S210. URL collection and subsequent information extraction operations are prohibited if the aforementioned consent authorization actions are not obtained or the authorization credentials are invalid.

[0048] In some embodiments, the web page information intelligent extraction system may include an element crawling module and a pattern learning module. The element crawling module can be used to collect the URLs of multiple web pages, obtain the HTML content of the web pages, and save them. The pattern learning module can be used to generate URL patterns, such as by executing S210.

[0049] In the above embodiments, by performing fine-grained analysis on the preset hierarchy of URLs, such as comparing the address content (fixed words or variables), address length (e.g., 8-digit date), and address type (numbers, letters, mixed, dynamic parameters) of each level, URLs with similar webpage structures or URL structures are grouped into the same URL pattern. This can distinguish webpages with different URL structures but the same domain name generated by different backend versions (old and new versions), different terminals (PC and mobile H5 sites), and different languages ​​(Chinese and English sites) under the same domain. This solves the limitations of unclear rule reuse boundaries and insufficient cross-template generalization ability caused by diverse webpage structures and mixed template versions, and improves the comprehensiveness and accuracy of extracting massive heterogeneous webpage information.

[0050] In S220, for any first URL pattern in at least one URL pattern, information extraction is performed on any first webpage corresponding to the first URL pattern. Based on the extraction rules of information extraction, an element extraction pattern (Pattern) corresponding to the first URL pattern is generated. The element extraction pattern includes at least one of the following: the positioning rules of each element (e.g., title, body, author, etc.) in the corresponding webpage (which may refer to path expressions or selectors used to locate each element in the webpage), data cleaning rules (which may refer to rules used to process the extracted raw data, including but not limited to: removing leading and trailing spaces, removing specific prefixes and suffixes, replacing invalid characters, converting relative time to absolute time, converting HTML entities to ordinary characters, etc.), and the format specification of the extraction results (which may refer to the requirements for the organization of the output data, including but not limited to: the data type of the field (string, integer, date), the format pattern of date and time, the separator of multi-value fields, the representation of null values, etc.).

[0051] The first webpage can be one or more webpages. For example, it can be the webpage corresponding to the first URL in the category corresponding to the first URL pattern, the webpage corresponding to a random URL, or the webpage corresponding to a URL selected based on a certain strategy (such as page quality or loading speed) as the first webpage.

[0052] Extraction rules can refer to specific instructions used to correctly locate and extract a certain field. For example, an XPath expression can be a location rule; a regular expression can be an extraction or cleaning rule.

[0053] Specifically, the above-mentioned information extraction for any first webpage corresponding to the first URL pattern includes the following steps: determining an extraction strategy based on the page characteristics of the first webpage, wherein the extraction strategy corresponds to at least one general information extraction tool; extracting information from the first webpage according to the extraction strategy; if the extraction result of the first webpage is verified to be an error, then the information is re-extracted from the first webpage using a preset visual model or a large language model.

[0054] For example, one can obtain the HTML source code of the first webpage, analyze it, and extract page features. Page features may include, but are not limited to: text density (the ratio of text characters to HTML tags), link density (the ratio of hyperlinks to total text), the average depth of the DOM tree, and specific tags (such as...). <article> 、 <h1>、 <time>The presence of ) and the proportion of Chinese characters, etc.

[0055] For example, different extraction strategies correspond to different general information extraction tools or combinations of tools. For instance, for text with high density, low link density, and existing... <article>For news-related web pages tagged with certain keywords, the General News Extractor (GNE) or Trafilatura can be selected; for well-structured multilingual web pages, Newspaper3k can be chosen; and for web pages with complex visual layouts, a visual model can be invoked. This "adaptive model routing" mechanism avoids the problem of unstable performance of a single tool on different types of web pages.

[0056] For example, information extraction can be performed according to a predetermined extraction strategy. First, the corresponding general information extraction tool is called to perform the extraction. After extraction, a quick verification can be performed, such as checking whether the title field is empty, whether the length of the body field exceeds a preset threshold (e.g., 100 characters), and whether the time field can be parsed normally. If the quick verification passes, the extraction result is confirmed to be correct. If the verification finds that the result is incorrect (e.g., all general tools return empty or obviously incorrect results), a preset visual model (e.g., a deep learning model based on computer vision technology, which takes the rendered web page image as input, identifies semantic blocks such as the title region, body region, and sidebar region in the image through convolutional neural network architecture, and then combines optical character recognition (OCR) technology to extract the text content in these regions. The advantage of the visual model is that it does not depend on the HTML structure, so it has strong robustness to frequent changes in the front-end framework) or a large language model can be used for fallback extraction. Visual models render web pages as images and use computer vision techniques to identify visual semantic blocks such as title areas (typically large font, centered, and high contrast) and body text areas (continuous paragraph blocks, left-aligned, and occupying a large area). Large language models take HTML code or a simplified DOM tree as input and perform semantic understanding and information extraction using natural language instructions (such as "Please extract the title, body text, author, and publication time from the following HTML"). A fallback mechanism ensures that even with extremely complex pages (such as deeply nested React applications or dynamically loaded SPA sites), usable extraction results can still be produced, thereby improving the overall extraction success rate.

[0057] The visual model can be based on an object detection architecture (such as You Only Look Once version 8, YOLOv8) or a multimodal visual language model (such as Generative Pre-trained Transformer 4 with Vision, GPT-4V, Contrastive Language-Image Pre-training, CLIP). The Large Language Model (LLM) can be a model using a Transformer-based generative architecture. For example, the visual model may include: a feature extraction backbone network, used to extract visual feature maps from rendered webpage screenshots and identify semantic blocks such as title areas, body text areas, and sidebar areas on the page; an object detection head, used to locate the position coordinates of key elements (such as product images, price tags, and title text) on the page through bounding box regression; and a multimodal fusion layer, used to align visual features with textual semantic features using a cross-modal attention mechanism to achieve joint understanding of text and images. For example, the large language model may include: an embedding layer, used to convert input HTML text fragments or Markdown content into vector representations, capturing the semantic information and contextual relationships of words. A multi-layer Transformer decoder processes the input sequence layer by layer using self-attention and feedforward neural networks to model long-range dependencies in text and understand the semantic meaning of HTML tags. A function call module supports function calling, allowing the model to invoke predefined function signatures when generating structured output, mapping natural language inference results to a specified JSON schema. An output projection layer maps the decoder's hidden states to outputs in a specified format (such as JSON objects), enabling the transformation from unstructured text to structured data.

[0058] For example, a headless browser (such as Headless Chrome or WebKit-based browsers) can be used to load the target webpage. After JavaScript execution, asynchronous API backfilling, and lazy loading are complete, the page rendering result is converted into image data. Then, computer vision technology is used to analyze the page's visual layout, including element position coordinates, size, color contrast, font size differences, block area proportions, reading order, and multi-column layout relationships. The system uses these visual signals to infer the semantic boundaries of the page, dividing it into functional modules such as title areas, body text areas, sidebar areas, and recommendation areas. Then, OCR processing is performed on the identified target blocks (such as title areas and body text areas) to extract the text content. For complex visual elements such as tables and charts, OCR can capture their structured arrangement information. Then, the text extracted by OCR is combined with block metadata to form a structured output. For example, the output may be a JSON object containing fields such as title text, coordinates of the title area, body paragraphs, image position, and size.

[0059] Specifically, the above-mentioned generation of the element extraction pattern corresponding to the first URL pattern based on the extraction rules of information extraction includes the following steps: integrating the extraction rules of information extraction and the extracted key elements (such as performing correlation encoding on the extraction rules and key elements) into the element extraction pattern corresponding to the first URL pattern.

[0060] For example, after successfully extracting information, the extracted structured data can be retained, and all the rules used to successfully achieve this extraction can be recorded: during the extraction process, the internal mechanisms of general extraction tools or large models will generate corresponding node positioning information (for example, GNE can output the XPath of the main text area); based on the successful extraction results, relatively stable positioning paths can be derived in reverse (for example, by comparing the DOM trees of multiple similar pages, the common parent path of the title node can be found). Subsequently, the positioning rules, data cleaning rules, and format specifications can be integrated into a complete JSON object or other structured format, which is the "feature extraction mode".

[0061] Furthermore, a mapping relationship can be established between the feature extraction pattern and the first URL pattern, and stored in a storage unit, such as a database, using an inverted index. The inverted index method refers to using the domain name and path prefix of the URL pattern as keywords and the corresponding feature extraction pattern as the index value, thereby achieving millisecond-level fast matching in subsequent steps.

[0062] In some embodiments, such as Figure 2 As shown, during the crawling phase, a target URL can be entered to check the URL pool. If the URL already exists, an incremental update check is performed; if it is a new URL, the distributed crawler scheduling begins. The distributed crawler scheduling executes proxy rotation and then sends an HTTP request, simultaneously handling anti-crawling strategies. If the request succeeds, the HTML source code is obtained; if the request fails, proxy rotation is repeated and the HTTP request is sent again.

[0063] After URL clustering, the webpage crawling and parsing phase can begin. For each URL pattern, a representative URL (the URL corresponding to the first webpage) can be selected, and its corresponding HTML source code can be obtained using the feature crawling module. Subsequently, the URL is parsed using a general information extraction tool to extract key elements such as title, abstract, author, publication time, and body content. For general information extraction tools, the General News Extractor (GNE) can be based on Document Object Model (DOM) tree density calculation and text block merging algorithms, supporting incremental training and noise filtering; Trafilatura can use a variant of the readability algorithm combined with coordinate extraction, and has built-in sentence segmentation and keyword extraction functions; Newspaper3k can use Natural Language Toolkit (NLTK) word segmentation and text ranking algorithms, supporting multilingualism and keyword extraction; JinaReader can be based on visual attention mechanisms and a Large Language Model (LLM), supporting zero-shot sampling and Application Programming Interface (API) calls.

[0064] After extraction is complete, simple checks can be performed to determine if the results are correct. For example, check if the title is empty or the body text is too short. If extraction results from multiple general tools are incorrect, a fallback extraction can be performed using a visual model or a large language model from deep learning to ensure usable extraction results even for extremely complex pages.

[0065] After successful extraction, the specific location path of each element in the HTML document (such as XML Path Language (XPath) or Cascading Style Sheets (CSS) selectors) and the extraction rules used can be recorded. These location rules, data cleaning rules, and formatting specifications can be integrated into a complete "feature extraction pattern". Simultaneously, the patterns corresponding to the same type of URL pattern and their quantity can be recorded and stored using an inverted index. That is, the key features of the URL pattern (such as domain name or path prefix) are used as the index key, and the corresponding pattern is used as the index value, thereby achieving fast retrieval and matching.

[0066] In some embodiments, the intelligent web page information extraction system may include a pattern learning module, which can be used to generate feature extraction patterns, such as in S220. Furthermore, the intelligent web page information extraction system may also include a database module for storing URL patterns, feature extraction patterns, and historical extraction results.

[0067] In the above embodiments, a step-by-step extraction strategy of "prioritizing lightweight general-purpose extraction tools and using visual models or large language models as a fallback" can be adopted. The high-cost model is only activated for supplementary extraction when the extraction result verification of the general extraction tool fails. This can ensure a high extraction success rate while keeping the overall computing power consumption within the cost range of lightweight tools. It solves the problems of high computing power consumption and difficulty in maintaining high throughput and low cost for massive tasks caused by complex scenarios such as dynamic page rendering and asynchronous loading, and achieves the best balance between efficiency and effectiveness in large-scale web page information extraction scenarios.

[0068] Furthermore, by integrating the positioning rules (XPath or CSS selectors), data cleaning rules, and output format specifications used in successful extraction into a feature extraction pattern and storing it in association with the corresponding URL pattern, this pattern can be directly reused for rapid extraction when encountering web pages with similar URL patterns in the future. This solves the problem of repeated parsing and recalculation of similar pages caused by the lack of a knowledge accumulation mechanism, achieving a leap in batch processing efficiency of "learn once, reuse ten thousand times" and improving the throughput and resource utilization of massive isomorphic web page information extraction.

[0069] Before executing S230 or S240, you can first execute S250: determine whether the URL of the second webpage matches at least one URL pattern.

[0070] Specifically, if it is determined that the URL of the second webpage matches any of the at least one URL patterns, then S230 can be executed; if it is determined that the URL of the second webpage does not match any of the at least one URL patterns, then S240 can be executed.

[0071] The second URL pattern can be one or more URL patterns.

[0072] Specifically, determining whether the URL of a second webpage matches at least one URL pattern can include the following steps: extracting the feature information of the second webpage's URL. This feature information can include the complete URL string, domain name, path portion, query parameters, path hierarchy split by delimiters, and feature analysis results for each hierarchy (such as whether the hierarchy is a fixed string or a number, or a dynamic parameter like a date format). Using the extracted feature information (e.g., domain name and path prefix) as search keywords, candidate URL patterns are quickly located in a storage unit, such as an inverted index in a database. The structure of the inverted index can be a mapping of domain name and path prefix to a Pattern identifier list, i.e., a URL pattern identifier list. For example, searching for the keyword "news.shili / socialgd" can find all URL patterns starting with that prefix. For each URL pattern in the candidate URL patterns, the URL of the second webpage is compared with each URL pattern de (which can be a pattern expression of the URL pattern). During the comparison, the matching rule for dynamic parameters can be: if a certain level in the URL pattern is marked as a dynamic parameter (such as "{date}" or "{id}"), then that level can match any specific value that conforms to the rule for that parameter type (for example, "{date}" matches an 8-digit number, and "{id}" matches any numeric string). This matching rule can also be applied similarly to the URL pattern generation process in S210.

[0073] If at least one URL pattern completely matches the URL of the second webpage (i.e., the fixed part of the URL matches the fixed part of the pattern, and the dynamic part (or changing part) conforms to the parameter rules (i.e., the rules involved in the dynamic part of the URL pattern)), then output "Match Successful" and the identifier of the matched URL pattern (which may contain one or more), and then execute S230. If no URL pattern completely matches the URL of the second webpage, then output "Match Failed", and then execute S240.

[0074] In the above process, a matching judgment step can be added before the extraction operation. The URL of the second webpage is compared with at least one URL pattern stored in the database. Based on the matching result, two different processing paths are executed: reuse extraction (S230) or learning extraction (S240). When the match is successful, the existing feature extraction pattern can be reused for fast extraction. When the match fails, the URL is used as a new cluster category and a new URL pattern is generated. This solves the problem of incomplete rule coverage and the need for continuous manual intervention in configuration caused by the continuous emergence of new page types. It realizes automatic discovery and adaptive expansion of new websites and new page structures, and improves the continuous coverage and automation level of the webpage information extraction system in the dynamic Internet environment.

[0075] Furthermore, by constructing an inverted index using domain name and path prefix as search keys, the matching range of a large number of URL patterns is narrowed down from full traversal to a candidate set before precise comparison at the path level. This maintains a high matching response speed, solves the problem of decreased matching efficiency and low real-time performance caused by too many URL patterns, and improves the performance of web page information extraction.

[0076] In S230, in response to the URL of the second webpage matching any second URL pattern in at least one URL pattern, information is extracted from the second webpage according to the element extraction pattern of the second URL pattern.

[0077] The second URL pattern can be any URL pattern that successfully matches the URL of the second webpage during the matching process. There can be one or more second URL patterns. When multiple URL patterns match the second webpage (for example, a URL matches multiple URL patterns with different wildcard strengths), the element extraction pattern of the URL pattern that best meets the priority rules can be selected to extract information from the second webpage according to preset priority rules.

[0078] Specifically, the above-mentioned information extraction of the second webpage based on the element extraction mode of the second URL pattern includes the following steps: extracting information from the second webpage according to the preset order of each URL pattern in the second URL pattern, based on the element extraction mode of the URL patterns in the second URL pattern. The preset order is determined based on at least one of the following: priority of the corresponding URL pattern (which can be a configurable weight value that reflects the importance or reliability of the URL pattern; higher priority URL patterns can be assigned an earlier preset order), number of uses (which can refer to the cumulative number of times the URL pattern has been successfully used; URL patterns with a higher number of uses usually have higher reliability and can be assigned an earlier preset order), and extraction success rate; if the extraction result of the information extraction of the second webpage based on the second URL pattern is incorrect, the information extraction of the second webpage is re-performed to generate a new element extraction mode corresponding to the second webpage.

[0079] For example, each URL pattern can be associated with at least one element extraction pattern in the database. The extraction can be performed sequentially using each element extraction pattern according to a preset order (e.g., sorting by version number, creation time, extraction success rate, usage frequency, and priority). This continues until the extraction result of a certain element extraction pattern passes verification (i.e., is deemed correct), or all element extraction patterns have been attempted but all extractions have failed. During the element extraction process, elements such as title, body text, author, and publication time can be located in the second webpage according to pre-stored positioning rules, and the text content can be extracted. Then, the text content can be processed according to recorded data cleaning rules (e.g., removing leading and trailing spaces, removing specific prefixes, and formatting dates). Finally, structured extraction results are output according to output format specifications (e.g., JSON structure, date format). The verification process for the extraction results can refer to the verification process for the information extraction results of the first webpage described above, or it can also refer to the verification process in the following embodiments.

[0080] If the information extraction result of the second webpage based on the second URL pattern is incorrect, the information extraction of the second webpage can be performed again (at this time, the URL of the second webpage can be used as a new cluster category, that is, a category other than the cluster category in S210, and a new URL pattern can also be generated based on the URL of the second webpage. The specific extraction method is similar to the extraction method for the first webpage, or a preset visual model or large model can be used directly for extraction) to generate a new feature extraction pattern corresponding to the second webpage.

[0081] In the above process, multiple element extraction patterns can be associated with the same URL pattern and extracted sequentially according to priority. This achieves smooth degradation and dynamic sorting of extraction rules. Moreover, when all element extraction patterns fail, a new element extraction pattern can be directly generated, completing the online repair and update of failed element extraction patterns. This solves the problems of rules suddenly becoming invalid due to website redesigns requiring manual repair, and the low extraction efficiency and accuracy caused by the mixing of old and new rules, achieving information extraction with high efficiency and accuracy.

[0082] In S240, in response to the fact that the URL of the second webpage does not match any of the URL patterns in at least one URL pattern, information extraction is performed on the second webpage to generate the element extraction pattern corresponding to the second webpage.

[0083] In step S240, the second webpage can be treated as a new cluster category, separate from the clustering results in S210 (such as the clustering categories obtained in S210), and information can be extracted from the second webpage. For the specific information extraction process, please refer to the above embodiments, such as step S230.

[0084] In the above process, the learning extraction process is triggered when the URL of the second webpage does not match any of the existing URL patterns. The second webpage is then used as a new clustering category for information extraction and feature extraction pattern generation. This allows for the automatic discovery and recording of previously uncovered webpage types during online operation, solving the problem of new types of pages not being extracted correctly. It also enables dynamic incremental expansion of URL patterns and feature extraction patterns, improving the comprehensiveness of information extraction.

[0085] In some embodiments, the web page information intelligent extraction system may include a pattern matching module, which can be used to retrieve matching URL patterns and perform information extraction, such as executing S250 and S230 or S240.

[0086] For example, for the URL of the second webpage that has not undergone information extraction, it can be determined whether it belongs to a cluster category corresponding to a certain URL pattern already existing in the database, and different processing branches are executed according to the determination result. Specifically, if the URL belongs to a cluster category corresponding to an existing URL pattern, then the feature extraction patterns of that URL pattern are extracted in the order they correspond (for example, the same URL pattern may correspond to multiple feature extraction patterns, which can be sorted by priority, version number, or extraction success rate. When a match is successful, each feature extraction pattern can be tried in this order until the correct result is obtained). If the extraction result is correct, the number of feature extraction patterns used corresponding to the correct extraction result is incremented by one; if the extraction results are all incorrect, the second webpage is re-extracted using a method similar to S210, and the newly generated feature extraction patterns are recorded in the database. This corresponds to S240 above.

[0087] If a URL does not belong to any existing URL pattern cluster, it is stored separately in the database. Information is extracted from the second webpage using a method similar to S210, and recorded in the database, awaiting matching with subsequent new URLs. In other words, when a new URL cannot match any existing URL pattern, it is not discarded or flagged as an error; instead, it can be treated as a new type and its information extracted. This ensures the database can be continuously expanded to accommodate the emergence of new page types.

[0088] In addition, the above process can be executed in a database, for example, using a remote dictionary server (Redis) or a structured query language (SQL) database (such as MySQL) to perform matching judgments, URL pattern retrieval, and other operations. Specifically, the database's indexing and query capabilities can be utilized to improve processing efficiency and maintainability.

[0089] In the above process, by associating multiple versions of element extraction patterns with the same URL pattern and attempting extraction sequentially according to a preset order, the old version of the element extraction pattern can be retained while a new version of the element extraction pattern is added when the webpage structure is redesigned, thereby improving the robustness and maintainability of information extraction in scenarios with dynamic changes in webpage structure.

[0090] In some embodiments, the technical method of this application may further include the following steps: verifying the extraction results of the second webpage in at least one of the following dimensions: core field completeness dimension (e.g., determining whether the core fields are complete. For information pages, at least three types of fields should be included: title, body text, and publication time; if any core field is empty, or the body text is only a very short text (the text quantity is less than a threshold), the result can be determined to be unreliable), content rationality dimension (e.g., determining whether the body text has continuous semantic text features. If a large number of navigation words, column words, recommendation words, and comment section words appear in the result, the result is considered unreliable). Copyright endnotes with very few valid paragraphs usually indicate that noise has not been properly cleaned up. If the main text is almost entirely links, button text, or repetitive sentences, it can also be considered an extraction bias. For example, check whether the main text contains a large amount (which could mean the number exceeds a threshold) of navigation words, recommended words, copyright endnotes, or other noise, or whether the main text is a continuous natural language paragraph. Field consistency dimension (for example, check whether the logic between different fields is consistent, such as whether the title is related to the main text topic, whether the publication time matches the time context of the main text content, and whether the author field is mistakenly filled with a column name or source name). The evaluation criteria include: time validity (e.g., verifying that the publication time is a reasonable date value, not empty, a default placeholder (e.g., 1970-01-01), a future time, or a time that clearly contradicts the crawling time); and structural readability (e.g., verifying that the paragraph order of the main text is correct, that the table structure is not misaligned, and that there are large sections of templated noise unrelated to the main text; if the paragraph order is disordered, the table content is severely misaligned, or there are large sections of templated noise in the main text, it indicates that the usability of the parsed content is insufficient); based on the results of at least one of these criteria, the second network is judged. The extraction result of the second webpage is classified as a first-class result, a second-class result, or a third-class result. If the extraction result of the second webpage is determined to be a first-class result, then the extraction result of the second webpage is stored or used. Alternatively, if the extraction result of the second webpage is determined to be a second-class result, then the extraction result of the second webpage is reviewed or the extraction rules corresponding to the second webpage are adjusted (for example, by inputting it into a lightweight review or rule fine-tuning queue). Alternatively, if the extraction result of the second webpage is determined to be a third-class result, then the information of the second webpage is re-extracted through a preset visual model, a large language model, or a manual method.

[0091] The results are categorized into three types: the first type indicates that the extraction results of the second webpage meet the following criteria: the corresponding core fields are complete, the main text is continuously readable, and there are no logical conflicts between fields; the second type indicates that the extraction results of the second webpage meet the following criteria: the number of corresponding missing core fields is less than the quantity threshold, the noise level is less than the noise threshold, and there are unstable fields; the third type indicates that the extraction results of the second webpage meet the following criteria: the number of corresponding missing core fields is greater than or equal to the quantity threshold, the fields are misaligned, the main text is unreadable, and the noise level is greater than or equal to the noise threshold.

[0092] In some embodiments, the intelligent web page information extraction system may include an element verification module for verifying the accuracy of the extracted elements, i.e., the extraction results, as described in the above embodiments. This reduces the cost of manual screening in large-scale information extraction scenarios and forms a closed loop of "extraction-evaluation-correction," thereby improving the overall parsing stability and maintainability.

[0093] In some embodiments, the technical method of this application may further include the following steps: statistically analyzing the usage frequency of URL patterns (e.g., the number of uses can be the number of times a URL pattern is matched and used for extraction within a specific time period (e.g., the last 30 days, the last 10,000 extractions), and the ratio of the number of uses to the total number of extractions can be the usage frequency) and the extraction result accuracy; removing URL patterns whose corresponding usage frequency is lower than a frequency threshold (e.g., 1%) from the storage unit (e.g., the database mentioned above) (e.g., physical deletion or logical deletion); and reducing at least one of the priority, usage frequency, and confidence of URL patterns whose corresponding extraction result accuracy is lower than an accuracy threshold.

[0094] In some embodiments, the web page information intelligent extraction system may include an optimization and update module for evaluating the accuracy of URL patterns and / or feature extraction patterns and triggering pattern optimization, as described in the above embodiments.

[0095] In the above process, by statistically analyzing the usage frequency of each URL pattern and removing patterns with a usage frequency below a preset threshold from the storage unit, redundant patterns that have not been used for a long time or have been rarely matched can be automatically cleaned up, improving matching speed and resource utilization. Furthermore, by reducing at least one of the priority, usage frequency, or confidence level of URL patterns with extraction accuracy below a preset threshold, URL patterns that consistently fail or degrade in quality are relegated to a lower priority in subsequent information extraction without directly deleting them. This ensures that more reliable URL patterns are prioritized for information extraction, improving information extraction efficiency.

[0096] In some embodiments, URL patterns, feature extraction patterns, historical extraction results, and quality verification records can be uniformly associated and stored in a database. Based on verification feedback, the accuracy of the patterns can be automatically monitored, failure modes can be triggered for relearning, and low-frequency or failure modes can be cleaned up. This can form a fully automated closed-loop optimization process of "extraction-evaluation-correction-re-extraction", which solves the problems of weak strategy adaptability and high long-term maintenance costs caused by the dispersion of modules and the coarse quality closed loop. It realizes intelligent extraction without human intervention throughout the entire process and improves the long-term operational stability and maintainability of the system in scenarios with dynamic changes in web page structure.

[0097] Understandably, in S230, if the information extraction result of the second webpage based on the second URL pattern is incorrect, the information extraction of the second webpage is re-performed to generate a new feature extraction pattern corresponding to the second webpage. This process requires re-extracting information from the second webpage, and if the extraction process is similar to S220, it can easily lead to high costs. Therefore, the information extraction of the second webpage can be re-performed through the following steps to generate a new feature extraction pattern corresponding to the second webpage: classify the reasons for the extraction result error (refer to the embodiment corresponding to the verification of at least one dimension mentioned above), select different processing paths according to the error type, such as correcting the rules corresponding to the erroneous part, and updating the feature extraction pattern according to the modified rules, instead of directly re-extracting. For example, for the case where only a few fields fail to be extracted, instead of re-executing the complete extraction process, the following lightweight operation is performed: extract the correct value of the field (which can be inferred from historical versions, other positions on the page, or adjacent pages), search for nodes containing the correct value of the field in the DOM tree, and generate new positioning rules; replace the corresponding invalid rules in the current feature extraction pattern with the newly generated positioning rules to form a corrected pattern. Use a second webpage to quickly verify the corrected pattern (only perform this extraction). If the extraction is correct, the update is complete; if it is still incorrect, mark the field as "awaiting manual review". Store the corrected pattern in the database, associate it with the original URL pattern, and record the version change log (including invalid fields, old rules, new rules, and correction time).

[0098] In some embodiments, before performing information extraction and element extraction modes on web pages (such as a first web page or a second web page), a process of obtaining permission, consent, or authorization from the user or other parties is first executed. The specific process is similar to the permission, consent, or authorization process performed before collecting the URLs of multiple web pages described above.

[0099] Figure 4 A block diagram of a web page information extraction apparatus according to an embodiment of this application is shown. This web page information extraction apparatus may be a software unit or a hardware unit, or a combination of both, as part of a computer device. Figure 4 As shown, the web page information extraction device 400 provided in this application embodiment may specifically include: The first generation module 410 is used to: cluster the Uniform Resource Locator (URL) of multiple web pages to generate at least one URL pattern, wherein the URL similarity of the web page corresponding to each URL pattern is greater than or equal to a first similarity threshold and / or the web page structure similarity of the corresponding web page is greater than or equal to a second similarity threshold. The second generation module 420 is used to: extract information from any first webpage corresponding to any first URL pattern in at least one URL pattern, and generate an element extraction pattern corresponding to the first URL pattern according to the extraction rules of the information extraction. The element extraction pattern includes at least one of the following: the positioning rules of each element in the corresponding webpage, the data cleaning rules, and the format specifications of the extraction results. Information extraction module 430 is configured to: extract information from the second webpage in response to a URL of the second webpage matching any second URL pattern from at least one URL pattern; or, If the URL of the second webpage does not match any of the URL patterns in at least one URL pattern, information extraction is performed on the second webpage to generate the element extraction pattern corresponding to the second webpage.

[0100] In some embodiments, the first generation module 410 is specifically used to: cluster the URLs of multiple web pages by comparing at least one of the address content, address length, and address type of the preset hierarchical address of each web page's URL, with each category corresponding to at least one URL pattern.

[0101] In some embodiments, the first generation module 410 is specifically used for: Based on the delimiter, each URL in the URLs of multiple web pages is split into multi-dimensional feature items; Based on at least one of the following: the repetition frequency, path level number, and path level priority of each dimension feature item in the URLs of multiple web pages, determine the first dimension feature item from each dimension feature item; Generate a set of feature terms for each dimension based on the first dimension feature terms; The URLs of multiple web pages are clustered based on the similarity between the dimensional feature terms of each URL and the set of dimensional feature terms.

[0102] In some embodiments, the second generation module 420 is specifically used for: Based on the page structure of the first webpage, determine the extraction strategy, and the extraction strategy corresponds to at least one general information extraction tool; Information is extracted from the first webpage according to the extraction strategy; If the information extraction result of the first webpage is incorrect, the information extraction of the first webpage will be performed again using the preset visual model or large language model.

[0103] In some embodiments, the second generation module 420 is specifically used for: The extraction rules and key elements extracted are integrated into the element extraction mode corresponding to the first URL mode.

[0104] In some embodiments, the information extraction module 430 is specifically used for: According to the preset order of each URL pattern in the second URL pattern, information is extracted from the second webpage according to the element extraction mode of the URL pattern in the second URL pattern. The preset order is determined according to at least one of the priority and number of times the corresponding URL pattern is used. If the information extraction result of the second webpage based on the second URL pattern is incorrect, then the information extraction of the second webpage is re-performed to generate a new element extraction pattern corresponding to the second webpage.

[0105] In some embodiments, the web page information extraction device 400 may include a verification module, used for: The extraction results of the second webpage shall be validated in at least one of the following dimensions: core field completeness, content rationality, field consistency, time validity, and structural readability. Based on the verification results of at least one dimension, determine whether the extraction result of the second webpage is a first-class result, a second-class result, or a third-class result; If the extraction result of the second webpage is determined to be a first-type result, then the extraction result of the second webpage is stored or used; or, If the extraction result of the second webpage is determined to be a second type of result, then the extraction result of the second webpage is reviewed or the extraction rules corresponding to the second webpage are adjusted; or... If the extraction result of the second webpage is determined to be a third type of result, then the information of the second webpage will be extracted again using a preset visual model or a large language model. Among them, the first type of result indicates that the extraction result of the second webpage satisfies the following conditions: the corresponding core fields are complete, the main text is continuously readable, and there are no logical conflicts between fields; The second type of result indicates that the extraction result of the second webpage satisfies the following conditions: the number of missing core fields is less than the number threshold, the noise level is less than the noise threshold, and there are unstable fields. The third type of result indicates that the extraction result of the second webpage satisfies the following conditions: the number of corresponding missing core fields is greater than or equal to the number threshold, the fields are misaligned, the main text is unreadable, and the noise level is greater than or equal to the noise threshold.

[0106] In some embodiments, the web page information extraction device 400 may include an update module for: Analyze the frequency of URL pattern usage and the accuracy of extraction results; URL patterns whose usage frequency is below the frequency threshold will be removed from the storage unit; Reduce at least one of the following factors for URL patterns whose extraction accuracy is below the accuracy threshold: priority, frequency of use, and confidence level.

[0107] The specific implementation of each module in the web page information extraction device provided in this application embodiment can refer to the content of the web page information extraction method described above, and will not be repeated here.

[0108] Each module in the aforementioned web page information extraction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0109] Figure 5 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 5 The computer system 500 of the electronic device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0110] like Figure 5 As shown, the computer system 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 502 or programs loaded from storage section 508 into Random Access Memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An Input / Output (I / O) interface 505 is also connected to the bus 504.

[0111] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a local area network (LAN) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.

[0112] Specifically, according to embodiments of this application, the processes described in the flowcharts above can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts above. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the various functions defined in the apparatus of this application.

[0113] In one embodiment, an electronic device is also provided, comprising: Processor; and Memory is used to store the processor's executable instructions; The processor is configured to execute the steps in the above method embodiments by executing executable instructions.

[0114] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0115] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0116] It should be noted that the computer-readable storage medium of this application can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, disk storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, the computer-readable signal medium may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0117] This embodiment is only used to illustrate this application. The selection of software and hardware platform architecture, development environment, development language, message acquisition source, etc. in this embodiment can be varied. Based on the technical solution of this application, any improvement or equivalent transformation made to a certain part according to the principle of this application should not be excluded from the protection scope of this application.

[0118] It should be noted that the terminology used in the embodiments of this application and the appended claims is for the purpose of describing specific embodiments only, and is not intended to limit the embodiments of this application.

[0119] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0120] If implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application embodiment, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method of this application embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0121] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices, apparatuses and methods can be implemented in other ways.

[0123] For example, the division of units, modules, or components in the device embodiments described above is merely a logical functional division. In actual implementation, there may be other division methods. For example, multiple units, modules, or components may be combined or integrated into another system, or some units, modules, or components may be ignored or not executed.

[0124] For example, the units / modules / components described above as separate / display components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the units / modules / components can be selected to achieve the objectives of the embodiments of this application, depending on actual needs.

[0125] Finally, it should be noted that the mutual coupling or direct coupling or communication connection shown or discussed above can be an indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0126] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be included within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.< / article> < / time> < / h1> < / article>

Claims

1. A method for extracting web page information, characterized in that, include: Clustering the Uniform Resource Locator (URL) data of multiple web pages to generate at least one URL pattern, wherein the URL similarity of the corresponding web page for each URL pattern is greater than or equal to a first similarity threshold and / or the web page structure similarity of the corresponding web page is greater than or equal to a second similarity threshold. For any first URL pattern in the at least one URL pattern, information is extracted from any first webpage corresponding to the first URL pattern, and an element extraction pattern corresponding to the first URL pattern is generated according to the extraction rules of the information extraction. The element extraction pattern includes at least one of the following: positioning rules for each element in the corresponding webpage, data cleaning rules, and format specifications for the extraction results. In response to the URL of the second webpage matching any of the at least one URL patterns, information is extracted from the second webpage according to the element extraction pattern of the second URL pattern. or, In response to the fact that the URL of the second webpage does not match any of the at least one URL patterns, information is extracted from the second webpage to generate an element extraction pattern corresponding to the second webpage.

2. The method according to claim 1, characterized in that, The clustering of Uniform Resource Locator (URL) values ​​for multiple web pages includes: By comparing at least one of the address content, address length, and address type of the preset hierarchical address of the URLs of the multiple web pages, the URLs of the multiple web pages are clustered, and each cluster corresponds to at least one URL pattern.

3. The method according to claim 2, characterized in that, The step of clustering the URLs of the multiple web pages by comparing at least one of the following: address content, address length, and address type of the preset hierarchical address of each web page's URL, includes: Based on the delimiter, each URL in the URLs of the multiple web pages is split into multi-dimensional feature items; Based on at least one of the repetition frequency, path level number, and path level priority of each dimension feature item in the URL of the plurality of web pages, a first dimension feature item is determined from each dimension feature item; Generate a set of dimensional feature terms based on the first dimensional feature terms; The URLs of the multiple web pages are clustered based on the similarity between the dimension feature terms of each URL and the set of dimension feature terms.

4. The method according to claim 1, characterized in that, The step of extracting information from any first webpage corresponding to the first URL pattern includes: Based on the page features of the first webpage, an extraction strategy is determined, and the extraction strategy corresponds to at least one general information extraction tool. Information is extracted from the first webpage according to the extraction strategy; If the information extraction result of the first webpage is incorrect, the information extraction of the first webpage will be performed again using a preset visual model or a large language model.

5. The method according to claim 1, characterized in that, The step of generating the element extraction pattern corresponding to the first URL pattern according to the extraction rules of the information extraction includes: The extraction rules and key elements extracted are integrated into the element extraction mode corresponding to the first URL mode.

6. The method according to claim 1, characterized in that, The step of extracting information from the second webpage according to the element extraction pattern of the second URL pattern includes: According to the preset order of each URL pattern in the second URL pattern, information is extracted from the second webpage according to the element extraction mode of the URL pattern in the second URL pattern. The preset order is determined according to at least one of the priority, number of times used, and extraction success rate of the corresponding URL pattern. If the information extraction result of the second webpage according to the second URL pattern is incorrect, the information extraction of the second webpage shall be performed again to generate a new element extraction pattern corresponding to the second webpage.

7. The method according to any one of claims 1-5, characterized in that, The method further includes: The extraction results of the second webpage are validated in at least one of the following dimensions: core field completeness, content rationality, field consistency, time validity, and structural readability. Based on the verification results of at least one dimension, determine whether the extraction result of the second webpage is a first type of result, a second type of result, or a third type of result; If the extraction result of the second webpage is determined to be the first type of result, then the extraction result of the second webpage is stored or used; or, If the extraction result of the second webpage is determined to be the second type of result, then the extraction result of the second webpage is reviewed or the extraction rules corresponding to the second webpage are adjusted; or, If the extraction result of the second webpage is determined to be the third type of result, then the information of the second webpage is extracted again through a preset visual model or a large language model. The first type of result indicates that the extraction result of the second webpage satisfies the following conditions: the corresponding core fields are complete, the main text is continuously readable, and there are no logical conflicts between fields. The second type of result indicates that the extraction result of the second webpage satisfies the following conditions: the number of missing core fields is less than the number threshold, the noise level is less than the noise threshold, and there are unstable fields. The third type of result indicates that the extraction result of the second webpage satisfies the following conditions: the number of corresponding missing core fields is greater than or equal to the number threshold, the fields are misaligned, the text is unreadable, and the noise level is greater than or equal to the noise threshold.

8. The method according to any one of claims 1-5, characterized in that, The method further includes: The frequency of use of the URL pattern and the accuracy of the extraction results were statistically analyzed. URL patterns whose usage frequency is below the frequency threshold will be removed from the storage unit; Reduce at least one of the following factors for URL patterns whose extraction accuracy is below the accuracy threshold: priority, frequency of use, and confidence level.

9. A web page information extraction device, characterized in that, include: The first generation module is used to: cluster the URLs of multiple web pages to generate at least one URL pattern, wherein the URL similarity of the web page corresponding to each URL pattern is greater than or equal to a first similarity threshold and / or the web page structure similarity of the corresponding web page is greater than or equal to a second similarity threshold. The second generation module is used to: extract information from any first webpage corresponding to any first URL pattern in the at least one URL pattern, and generate an element extraction pattern corresponding to the first URL pattern according to the extraction rules of the information extraction. The element extraction pattern includes at least one of the following: positioning rules for each element in the corresponding webpage, data cleaning rules, and format specifications for the extraction results. The information extraction module is used to: extract information from the second webpage in response to the URL of the second webpage matching any second URL pattern in the at least one URL pattern; according to the element extraction pattern of the second URL pattern. or, In response to the fact that the URL of the second webpage does not match any of the at least one URL patterns, information is extracted from the second webpage to generate an element extraction pattern corresponding to the second webpage.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images