Cyclic automatic data acquisition method and system

Abstract

The invention provides a cyclic automatic data acquisition method and system. The method comprises the following steps: forming an entry URL set; according to the entry URL set, based on DOM structure feature analysis and semantic association degree evaluation, and link value scores of TF-I DF and Word2Vec, forming a high-value link queue; according to the high-value link queue, obtaining page contents, and forming a page queue containing effective telephone numbers; according to the page queue, performing time sequence interpolation by using a self-attention diffusion model to form a merchant data set; performing field information extraction and telephone number grouping identification by utilizing a field extraction neural network model and a telephone number grouping identification model according to the merchant data set, and generating a structured data set; and according to the structured data set, executing multi-dimensional data fingerprint generation to perform data deduplication. According to the method and the device, the technical problems of the traditional automatic data acquisition technology in the aspects of complex webpage structure identification, data timeliness maintenance and data quality assurance are solved.

Description

Technical Field

[0001] This application relates to the technical field of Internet automated data collection, and particularly to a cyclic automated data collection method and system. Background Art

[0002] Internet automated data collection technology is a technical means for automatically collecting network information, which is widely used in fields such as search engines, data analysis, and market research. With the explosive growth of Internet information, the structures of various websites are becoming increasingly complex, and data formats are diverse. Traditional automated data collection technologies are facing huge challenges.

[0003] Currently, the common automated data collection technologies on the market are mainly divided into two categories: rule-based automated data collection and distributed automated data collection. Rule-based automated data collection locates and extracts specific content in web pages through preset rules such as XPath and CSS selectors. It is simple to operate but has poor adaptability. Distributed automated data collection improves the collection efficiency through multi-node parallel processing, but there are still problems with low extraction accuracy under complex web page structures.

[0004] Existing automated data collection technologies usually adopt a single data extraction method, mainly relying on DOM parsing and regular expression matching, and have limited recognition capabilities for complex web page structures. When dealing with dynamically loaded content, unstructured text, and diverse layouts, these technologies often need to customize rules for specific websites, lacking generality and flexibility, and cannot achieve real-time update and efficient cyclic collection.

[0005] In addition, existing automated data collection technologies have obvious deficiencies in data timeliness and accuracy. On the one hand, Internet information is updated frequently, and traditional automated data collection cannot capture content changes in a timely manner. On the other hand, there is a relatively high error rate in the entity extraction process, and the crawling speed is relatively slow, making it difficult to meet the requirements of large-scale and high-quality data collection. These problems seriously restrict the application of automated data collection technologies in scenarios with high timeliness requirements such as business intelligence and risk monitoring. Summary of the Invention

[0006] In view of this, this application provides a cyclic automated data collection method and system, which solves the problems of limited recognition capabilities for complex web page structures and insufficient data timeliness and accuracy in the prior art.

[0007] The embodiment of the present application provides a cyclic automated data collection method, including: constructing a multi-source data entry set according to the distributed entry discovery mechanism of the knowledge graph, and forming an entry URL set by using dynamic priority queue management, timeliness evaluation model and semi-parametric batch processing global decision mechanism; forming a high-value link queue according to the entry URL set based on DOM structure feature analysis and semantic association degree evaluation, as well as link value scoring of TF-IDF and Word2Vec; obtaining page content according to the high-value link queue, extracting digital sequences by using regular expressions, and performing matching judgment by using a telephone number pattern knowledge base to form a page queue containing valid telephone numbers; performing time series interpolation on the page queue by using a self-attention diffusion model, and associating merchant information by means of a domain name extraction algorithm and a domain name merchant mapping knowledge base to form a merchant data set; extracting field information and grouping and identifying telephone numbers by using a field extraction neural network model based on supervised learning and a telephone number grouping and identification model based on sequence annotation to generate a structured data set according to the merchant data set; performing multi-dimensional data fingerprint generation on the structured data set for data deduplication, and adopting differential privacy technology for data protection to form a high-quality data set after deduplication and desensitization.

[0008] The construction of a multi-source data entry set according to the distributed entry discovery mechanism of the knowledge graph and the formation of an entry URL set by using dynamic priority queue management and timeliness evaluation model include: obtaining an initial URL set according to a search engine API, a social media API and an industry directory database; constructing a knowledge graph according to the initial URL set and combining domain knowledge, and identifying high-value entry nodes through graph analysis to generate a preliminarily screened URL subset; designing a dynamic priority queue according to the URL subset, and calculating priority scores based on the PageRank value, content update frequency and historical collection results of each URL to generate a URL queue with priority sorting; performing timeliness evaluation on the URL queue, dynamically adjusting the access frequency and priority, and forming the entry URL set.

[0009] Based on the set of entry URLs, through DOM structure feature analysis, semantic relevance evaluation, and link value scoring using TF-IDF and Word2Vec, a high-value link queue is formed, including: obtaining web page content according to the set of entry URLs, performing HTML parsing and cleaning to obtain a standardized DOM tree structure; based on the DOM tree structure, using regular expressions, XPath positioning, and CSS selectors to extract links to form an initial link set; according to the initial link set, through the TF-IDF algorithm, analyzing the relationship between link text and context, and combining with the Word2Vec model to calculate semantic similarity to obtain link value scoring results; based on the link value scoring results, setting a dynamic threshold to filter advertisements and low-value links, and using a heuristic algorithm to remove automated data collection traps to form a high-value link queue.

[0010] Based on the high-value link queue, obtaining page content, using regular expressions to extract digital sequences, and using a telephone number pattern knowledge base for matching and judgment to form a page queue containing valid telephone numbers, including: obtaining page content according to the high-value link queue, using DOM parsing to extract visible text content; according to the visible text content, using regular expressions to extract candidate digital sequences and record the context information of each candidate digital sequence in the original text; based on the candidate digital sequences, using a telephone number pattern knowledge base for various combinations and formatting processes to generate a preliminary list of identified telephone numbers; according to the list of telephone numbers and combining with context information, performing verification and functional type identification to form a page queue containing valid telephone numbers.

[0011] Based on the page queue, using a domain name extraction algorithm and a domain name merchant mapping knowledge base for merchant information association to form a merchant data set, including: based on the page queue, extracting the second-level domain name and the top-level domain name, and using a domain name clustering algorithm for grouping to generate a domain name grouping result; establishing a domain name merchant mapping table according to a commercial database and historical collection data; based on the domain name grouping result and the domain name merchant mapping table, adding merchant basic information to each URL to generate a URL data set associated with merchants; according to the URL data set, using a text clustering algorithm to perform content segmentation on page titles and content summaries to form the merchant data set.

[0012] The methods for obtaining the supervised learning-based field extraction neural network model and the sequence annotation-based telephone number grouping recognition model include: collecting historical web page data containing various website formats and field types; establishing a training data set through expert annotation, and annotating the keyword fields including merchant name, address, business scope, and the functional classification of telephone numbers in the data; performing feature engineering processing on the training data set, extracting DOM structure features, text semantic features, and positional relationship features, and constructing a training model based on the DOM structure features, text semantic features, and positional relationship features; adopting a transfer learning method to construct a field extraction neural network model based on a pre-trained language model and fine-tuning it for web page structured information extraction; designing a sequence annotation network structure and training a telephone number grouping recognition model using the BiLSTM-CRF architecture; optimizing the hyperparameters of the field extraction network and the telephone number grouping recognition model through cross-validation to improve the generalization ability of the field extraction neural network model and the sequence annotation-based telephone number grouping recognition model under different website formats; using the supervised learning-based field extraction neural network model and the sequence annotation-based telephone number grouping recognition model to perform field information extraction and telephone number grouping recognition on the merchant data set to generate a structured data set, including: automatically analyzing each merchant group's page according to the field extraction neural network model to identify and extract keyword field information; automatically grouping and functionally identifying multiple telephone numbers of the same merchant according to the telephone number grouping recognition model to identify and extract telephone number data; performing a correlation analysis on the keyword field information and the telephone number data to confirm the main merchant name of the data grouping and obtain the structured data set.

[0013] The self-attention diffusion model performs time series imputation, including: collecting the raw data generated by automated data collection activities, and forming a standardized time series data set through normalization and feature extraction processing; based on the standardized time series data set, using a conditional diffusion model to perform forward diffusion and backward diffusion processes to generate time series imputation data; performing data analysis using a diversity sampling algorithm according to the time series imputation data to obtain a prediction uncertainty index; based on the prediction uncertainty index, implementing an adaptive adjustment strategy to update the acquisition frequency parameter to form data with better time continuity and integrity.

[0014] The steps of designing a dynamic priority queue according to the URL subset include: executing a semi-parametric batch global decision-making mechanism with covariates, specifically including: collecting the feature data of the target website, generating a feature vector including domain name age, update frequency, and content richness through feature engineering processing; training using a parametric and non-parametric hybrid model based on the feature vector to form a semi-parametric reward prediction model; according to the semi-parametric reward prediction model, executing the Thompson sampling algorithm to calculate the expected data revenue value of each URL target, generating a priority decision plan for URL batch acquisition; according to the priority decision plan for URL batch acquisition, dynamically adjusting the weight parameters of the URL exploration and exploitation strategy, calculating the priority score of each URL, and updating the URL queue with priority sorting to form an optimized URL access order and frequency.

[0015] The steps of executing the Thompson sampling algorithm according to the semi-parametric reward prediction model to calculate the expected data revenue value of each URL target and generating a priority decision plan for URL batch acquisition include: obtaining the expected reward data of the URL target set, calculating the optimal combination through a combinatorial optimization algorithm to form an initial batch plan; according to the initial batch plan, performing diversity constraint calculation using the submodular function maximization framework to generate a diverse batch combination; according to the diverse batch combination, coordinating multi-node task allocation using the federated learning framework to form a distributed decision plan; according to the distributed decision plan, executing a multi-objective optimization algorithm to fuse compliance constraint conditions to generate a final URL batch acquisition execution plan for updating the URL queue with priority sorting.

[0016] An embodiment of the present application further provides a computer device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above-mentioned cyclic automated data collection method.

[0017] An embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions for causing a computer to execute the above-mentioned cyclic automated data collection method. An embodiment of the present application further provides a computer program product, including computer instructions, which implement the steps of the above-mentioned cyclic automated data collection method when executed by a processor.

[0018] The present application has the following technical effects: Through the knowledge graph distributed entry discovery mechanism and dynamic priority queue management, more efficient resource allocation and collection scheduling are achieved, improving the collection efficiency.

[0019] The link discovery algorithm based on DOM structure features and semantic relevance improves the accuracy of link value evaluation and reduces the invalid collection rate.

[0020] The telephone number recognition mechanism using a combination of multiple algorithms greatly improves the accuracy and adaptability of telephone number extraction.

[0021] The self-attention diffusion model is used for time series imputation, effectively solving the data continuity problem in scenarios of interruption recovery and incremental update.

[0022] The AI-based data extraction model can automatically adapt to different website structures, improving the accuracy and coverage of data extraction. The application of multi-dimensional data fingerprints and differential privacy technology ensures the uniqueness and compliance of data and improves data quality. Brief Description of the Drawings

[0023] To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. The drawings herein are incorporated into the specification and constitute a part of this specification. These drawings show embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0024] Figure 1 It is a schematic flowchart of the cyclic automated data collection method provided by the embodiment of the present application; Figure 2 It is a schematic flowchart of the intelligent entry discovery method provided by the embodiment of the present application; Figure 3 It is a schematic flowchart of the intelligent link discovery method provided by the embodiment of the present application; Figure 4 It is a schematic flowchart of the telephone number extraction method provided by the embodiment of the present application. Detailed Embodiments

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are only some of the embodiments of the present disclosure, rather than all of them. The components of the embodiments of the present disclosure usually described and illustrated in the accompanying drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed present disclosure, but merely represents selected embodiments of the present disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative efforts fall within the scope of protection of the present disclosure.

[0026] It should be noted that like reference numerals and letters denote like items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0027] As used herein, the term "and / or" merely describes an association relationship and indicates that three relationships may exist. For example, A and / or B may represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. Additionally, the term "at least one" as used herein represents any one of a plurality or any combination of at least two of a plurality. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set composed of A, B, and C. <----

[0028] As Figure 1 shown, an embodiment of the present application provides a cyclic automated data acquisition method, including: S1: According to the distributed entry discovery mechanism of the knowledge graph, construct a multi-source data entry set, and use a dynamic priority queue management, timeliness evaluation model, and semi-parametric batch processing global decision mechanism to form an entry URL set.

[0029] In practical applications, the system first obtains an initial URL set through a search engine API, a social media API, and an industry directory database.

[0030] In the embodiments of the present application, before automated data acquisition, permission or consent for the data acquisition path has been obtained. For example, when acquiring data, a browser is used to access the target page, and after taking a screenshot of the target page, page information is obtained in the form of ocr or through the function of viewing the source code of the browser itself.

[0031] For example, when it is necessary to collect telephone number data of the catering industry, the system can search for "contact information of XX restaurant" through the API, obtain merchant information under catering-related topics through the Weibo API, and simultaneously extract the URL of the restaurant list page from the industry directory.

[0032] Based on these initial URLs, the system constructs a knowledge graph, taking information such as restaurant names, regions, and cuisines as nodes and establishing the association relationships between them. Through graph analysis, the system identifies those aggregated pages that contain contact information of multiple restaurants, and these pages usually have higher data value. Then, the system uses a semi-parametric batch processing global decision-making mechanism for optimal resource allocation.

[0033] Specifically, for each food and beverage website, the system analyzes its domain name age (such as a certain review website having a relatively long domain name history), update frequency (such as the updates of a food blog may be relatively irregular), and content richness (such as the information on an official website may be more comprehensive) to construct a feature vector.

[0034] Based on these features, the system trains a semi-parametric reward prediction model and uses Thompson sampling to calculate the expected data benefits of each URL. Finally, the system generates an optimized URL batch acquisition plan, such as preferentially accessing the latest restaurant list page of a certain review website with a high update frequency, while reducing the access frequency to the official website with slow updates, thereby forming a set of entry URLs.

[0035] S2: According to the set of entry URLs, based on DOM structure feature analysis, semantic association degree evaluation, and link value scoring of TF-IDF and Word2Vec, a high-value link queue is formed.

[0036] Taking the data collection of the food and beverage industry as an example, the system starts from the entry URLs obtained in S1 (such as the XX restaurant page of a certain review website), first obtains the web page content and performs preprocessing. The system parses the HTML structure, removes irrelevant elements such as advertisements and navigation, and constructs a standardized DOM tree. Then, the system uses a variety of technical strategies to extract links in parallel: uses regular expressions to match all http(s) links, uses XPath to locate the links of all restaurant detail pages (such as / / div[@class="shop-list"] / a), and at the same time uses CSS selectors to identify clickable elements (such as.shop-card.shop-titlea).

[0037] For some content that is dynamically loaded using JavaScript (such as loading more restaurants by scrolling down), the system can also capture these dynamically generated links. Then, the system uses the TF-IDF algorithm and Word2Vec model to analyze the value of these links.

[0038] For example, when the link text contains keywords such as "Contact Us", "Reservation Phone Number", or the text around the link is highly relevant to contact information, the system will give a higher score. The system also considers the position of the link in the DOM tree. Usually, the links in the main content of the page are more valuable than those in the navigation bar or footer.

[0039] Finally, the system sets dynamic thresholds to filter advertisements and low-value links, such as automatically removing obvious advertisement links (such as URLs containing words like "ad", "sponsor") and typical automated data collection traps (such as infinite links caused by calendar page flipping), and finally forms a high-value link queue. These links mainly point to high-value pages such as restaurant detail pages and contact pages that contain phone numbers.

[0040] S3: According to the high-value link queue, obtain the page content, use regular expressions to extract digital sequences, and use a phone number pattern knowledge base for matching and judgment to form a page queue containing valid phone numbers.

[0041] When the system accesses a high-value link generated by S2 (such as the detail page of a certain restaurant), it first extracts the visible text content of the page through DOM parsing. The system processes various encoding formats and identifies hidden text to ensure comprehensive coverage of areas that may contain phone numbers. Then, the system uses regular expressions to extract all potential digital sequences, including consecutive digits (such as "13812345678"), digits with separators (such as "010-87654321"), and special formats (such as "+86(10)87654321").

[0042] For each extracted digital sequence, the system records its context information, such as prefix texts like "Ordering phone number:", "Contact us:". Then, the system uses a global phone number format rule knowledge base to verify these candidate numbers.

[0043] For example, Chinese mobile phone numbers are usually 11 digits and start with 1, and XX landlines usually start with 010 followed by 8 digits. The system can handle various format changes, such as phone numbers with parentheses, international area codes, or separators.

[0044] Finally, the system performs function recognition in combination with context information, such as identifying a phone number with the prefix "Ordering" as a restaurant ordering phone number and a phone number containing "Complaint" as a customer service phone number, thereby forming a page queue containing the valid phone numbers of each restaurant and their functional attributes.

[0045] S4: According to the page queue, use a self-attention diffusion model for time series interpolation, and use a domain name extraction algorithm and a domain name merchant mapping knowledge base for merchant information association to form a merchant dataset.

[0046] In the data collection of the catering industry, the system designs a reasonable resource allocation strategy, such as limiting the maximum number of requests for a single catering website to 1000 times per hour and the total crawling depth not exceeding 5 layers.

[0047] By mixing breadth-first and depth-first strategies, the system arranges the request order: first widely obtain the basic information pages of multiple restaurants, and then deeply obtain the detailed contact information of each restaurant. The system continuously monitors the collection progress and success rate. When it finds that the anti-crawling mechanism of a certain food delivery platform causes the success rate to decline, it will dynamically reduce the request frequency for that platform.

[0048] Meanwhile, the system records the processed status and supports resume from breakpoint. For example, it can continue the collection from the list of restaurants where the process was aborted last time after a network interruption. For the collected data, the system extracts domain names for grouping. For example, all URLs from "dianping.com" are grouped into one group, and those from "meituan.com" are grouped into another group.

[0049] The system also establishes a domain name-merchant mapping using historical data. For example, it identifies that "xms.dianping.com" and "www.xiaomaoshu.com" both belong to "Xiaomaoshu Restaurant". Additionally, the system uses a self-attention diffusion model for time series interpolation, especially for restaurant data that is updated regularly (such as new menus released every Friday). The system records its historical update pattern and can intelligently predict possible data changes during network interruptions. Through multiple sampling analyses, the system increases the actual collection frequency for data points with high uncertainty (such as special menus on holidays), while relying more on model predictions for highly predictable data (such as fixed phone numbers), thereby optimizing resource allocation.

[0050] S5: According to the merchant dataset, use a field extraction neural network model based on supervised learning and a phone number grouping recognition model based on sequence labeling to perform field information extraction and phone number grouping recognition, generating a structured dataset.

[0051] To train a high-performance data extraction model, the system first collects a large amount of historical web page data containing various formats of food delivery websites, such as pages with different styles from certain review platforms, certain group buying platforms, and official websites of various restaurants. The expert team annotates this data, clearly marking the positions of keyword fields such as merchant names, addresses, business scopes, and the functional attributes of each phone number (such as ordering phone numbers, delivery phone numbers, etc.).

[0052] Then, the system performs feature engineering on the training data, extracting DOM structure features (such as the type and depth of HTML tags where the fields are located), text semantic features (such as text content, context relationship), and positional relationship features (such as relative position on the page). Based on these features, the system constructs a training model.

[0053] Through transfer learning methods, the system uses a pre-trained BERT language model to establish a field extraction neural network and fine-tunes it for specific terms and structures in the food delivery industry.

[0054] At the same time, the system uses the BiLSTM-CRF architecture to train a telephone number grouping recognition model, which can accurately identify the functional types of multiple telephone numbers of the same restaurant.

[0055] After model training, the system uses 10-fold cross-validation to optimize model hyperparameters, ensuring good generalization across diverse restaurant website formats. Using these models, the system can accurately extract standardized merchant information from cluttered webpage content. For example, from text like "Xiaomaoshu (Chaoyang Branch) Business Hours: 10:00 AM - 10:00 PM, Order Hotline: 010-12345678," it can accurately identify the merchant name "Xiaomaoshu (Chaoyang Branch)" and the phone number "010-12345678" as having the functional attribute "order hotline."

[0056] S6: Based on the structured data set, perform multi-dimensional data fingerprint generation to deduplicate the data, and use differential privacy technology to protect the data, forming a high-quality data set after deduplication and desensitization.

[0057] When processing large amounts of restaurant data, the system uses a multi-dimensional data fingerprint algorithm to remove duplicates. This algorithm not only considers the restaurant name and phone number, but also combines information such as address and business category to generate a unique data fingerprint.

[0058] For example, although the names of "Xiaomaoshu (Chaoyang Branch)" and "Xiaomaoshu Chaoyang Branch" are slightly different, the system can identify them as the same restaurant based on the similarity of their phone numbers and addresses.

[0059] The system uses a multi-level deduplication strategy: first, exact matching is performed to remove complete duplicates (such as identical restaurant records), then similarity calculations are used to identify near-duplicates (such as records with slightly different names but the same phone number and address), and finally, intelligent merging is used to retain the most complete information.

[0060] For example, if one record contains a detailed address but an incomplete phone number, and another record has a complete phone number but a brief address, the system will merge them into one record containing a complete phone number and detailed address.

[0061] In terms of privacy protection, the system automatically identifies personal privacy information, such as the restaurant manager's private mobile phone number, ID number, etc., and processes these sensitive fields.

[0062] For example, a personal mobile phone number is partially masked (such as "138****5678") and the detailed address is blurred (retained only to the street level).

[0063] The system also adopts differential privacy technology, adding appropriate noise during data analysis to ensure that even when aggregated statistical data is released, individual information will not be leaked. Through these technologies, the system finally forms a high-quality dataset after deduplication and desensitization, which not only guarantees the uniqueness and integrity of the data, but also ensures compliance and privacy protection.

[0064] Among them, as Figure 2 shown, S1 specifically includes: S1.1: Obtain the initial URL set according to the search engine API, social media API, and industry directory database.

[0065] During the acquisition process of multi-source data entry, the system makes full use of various data sources for the collection and screening of initial URLs.

[0066] Taking the collection of business phone numbers as an example, the system first uses search engine APIs such as Baidu and Google, and uses keyword combinations (such as "contact information of a certain e-commerce company", "phone number of XX catering enterprise", etc.) to obtain the URL list of relevant web pages.

[0067] At the same time, the system accesses social media APIs such as Weibo and LinkedIn to crawl the contact information and relevant links published by enterprise official accounts. In addition, the system also accesses industry directory databases such as Huangye.com, Qichacha, and Tianyancha to directly extract the official website addresses and contact page links of enterprises.

[0068] These operations generate a large number of original URLs, forming a wide but unfiltered initial URL set. Through this multi-source data fusion method, the system can maximize the coverage of data collection and avoid the limitations brought by relying on a single data source. Especially for some small enterprises or newly established enterprises that are difficult to discover through conventional channels, through the latest indexes of social media APIs and search engines, the system can timely obtain their information to ensure the comprehensiveness and timeliness of the data.

[0069] S1.2: Based on the initial URL set and combined with domain knowledge, construct a knowledge graph, and generate a preliminary screened URL subset through graph analysis.

[0070] In this link, the system combines the initial URL set obtained in S1.1 with the pre-established domain knowledge to construct a simple knowledge graph.

[0071] Taking the catering industry as an example, the system first extracts basic information such as domain names, path structures, and page titles from the initial URLs as the basic nodes of the graph. Then, the system associates these nodes with concepts in domain knowledge (such as restaurant classification, geographical location, cuisine type, etc.).

[0072] For example, nodes containing "huoguo" in the URL or "hot pot" in the title will be associated with the concept of "hot pot restaurant." The system also establishes relationships between nodes, such as linking different restaurant websites under the same company or clustering restaurants in the same region.

[0073] Based on this knowledge graph, the system conducts in-depth analysis to identify the most valuable entry points. For example, the system prioritizes aggregated pages containing information about multiple restaurants (such as recommendation posts on food forums), official contact pages, and detailed pages with rich user reviews, as these pages often contain more valid phone numbers. Through graph analysis, the system generates a subset of URLs that have undergone preliminary screening and value assessment, providing the basis for subsequent prioritization.

[0074] S1.3: Based on the URL subset, a dynamic priority queue is designed, and a priority score is calculated based on the PageRank value, content update frequency and historical collection results of each URL to generate a priority-sorted URL queue.

[0075] In the dynamic priority queue management link, the system further evaluates and sorts the URL subsets filtered by S1.2.

[0076] The system has designed a dynamic priority queue that assigns a weight value to each URL by comprehensively considering multiple factors.

[0077] First, the system calculates the PageRank value of each URL to evaluate its importance and authority in the entire network; corporate websites with higher PageRank values are usually given higher initial weights.

[0078] Secondly, the system analyzes the content update frequency of the website corresponding to the URL, and assigns a higher timeliness weight to frequently updated websites (such as catering websites that update promotional information daily) to ensure that the latest information can be captured in a timely manner.

[0079] In addition, the system also considers historical collection records and gives higher experience weight to websites that have provided high-quality phone number data in the past. Based on the comprehensive calculation of these factors, the system generates a comprehensive priority score for each URL and establishes a prioritized URL queue accordingly.

[0080] This approach not only considers the static importance of URLs but also incorporates dynamic factors and historical experience, making resource allocation more rational and efficient. It's worth noting that this queue isn't static; it continuously adjusts based on real-time collection results and system resource status, embodying its "dynamic" nature.

[0081] Among them, a semi-parametric batch global decision-making mechanism with covariates is executed, specifically including: A1: Collect the feature data of the target website, and through feature engineering processing, generate a feature vector including domain name age, update frequency, and content richness.

[0082] In the first step of the semi-parametric batch global decision-making mechanism, the system conducts comprehensive collection and processing of the feature data of the target website.

[0083] For each target website, the system collects multi-dimensional feature data, including domain name age (obtained by querying WHOIS for the domain name registration time), website scale (evaluated by site map or page count), technical architecture (identifying the used CMS system, front-end framework, etc.), update frequency (analyzed by comparing historical snapshots), content richness (evaluating text content density, number of multimedia elements, etc.), external citation degree (analyzing the quantity and quality of backlinks), and historical collection success rate, etc.

[0084] After collecting these original features, the system conducts complex feature engineering processing. First, standardize the numerical features to make features with different dimensions comparable; then handle missing values, and fill in the features that cannot be obtained (such as some websites where the domain name age cannot be queried) with industry averages or data of similar websites; then conduct feature selection, and use correlation analysis and importance evaluation to retain the most predictive features; finally, construct combined features, such as combining the update frequency and content richness to form a "information value density" feature. After this series of processing, the system generates a high-quality feature vector containing key dimensions such as domain name age, update frequency, and content richness, providing a solid foundation for subsequent model training.

[0085] A2: According to the feature vector, use a parametric and non-parametric hybrid model for training to form a semi-parametric reward prediction model.

[0086] In this link, the system uses the feature vector generated by A1 to construct a semi-parametric reward prediction model. This model adopts a hybrid architecture that combines parametric and non-parametric methods, giving full play to the advantages of both methods.

[0087] In the parametric part, the system uses a generalized linear model to capture the linear relationship between features and the collected rewards (such as the number of valid phone numbers obtained). This part has a simple and clear structure and high computational efficiency, and is suitable for handling explicit feature associations.

[0088] For example, there is usually a positive correlation between the website update frequency and data freshness, which can be effectively expressed by a linear model. In the non-parametric part, the system adopts complex models such as random forests or gradient boosting trees to capture the non-linear interactions and complex patterns between features.

[0089] This part is particularly suitable for dealing with feature combinations that have complex relationships or are difficult to express with simple formulas, such as the complex relationship between domain name age, content structure, and crawling efficiency.

[0090] The system weights and fuses the prediction results of these two parts of the model, and continuously adjusts the weights through the Bayesian optimization method to finally form a semi-parametric reward prediction model. This hybrid architecture not only maintains the interpretability and computational efficiency of the parametric model but also has the ability of the non-parametric model to handle complex relationships, making it particularly suitable for automated data crawling scenarios with diverse website characteristics and complex relationships.

[0091] A3: According to the semi-parametric reward prediction model, execute the Thompson sampling algorithm to calculate the expected data revenue value of each URL target, and generate a priority decision plan for batch URL acquisition.

[0092] Based on the semi-parametric reward prediction model trained by A2, the system uses the Thompson sampling algorithm to calculate the expected data revenue value of each URL target and generates an optimized batch acquisition decision plan. The core idea of the Thompson sampling algorithm is to achieve a balance between exploration and exploitation through probabilistic sampling.

[0093] In specific implementation, the system first maintains a posterior belief of the reward distribution for each URL, which can be a Gaussian distribution based on the prediction model initially. At each decision point, the system randomly samples a value from the posterior distribution of each URL, representing the potential revenue of that URL, and then selects a batch of URLs with the highest sample values for access.

[0094] This method naturally balances "exploration" and "exploitation": For URLs with a wider posterior distribution (i.e., options with high uncertainty), there is a chance to draw high values and be selected, reflecting the spirit of exploration; while URLs with a high mean and small variance in the posterior distribution (i.e., options with high certain revenue) are frequently selected, reflecting the exploitation strategy.

[0095] As the system continuously collects new data, the posterior distribution will gradually converge, and the decision-making will become more and more accurate. Compared with the traditional ε-greedy algorithm, Thompson sampling does not require manual setting of the exploration rate parameter but adjusts according to uncertainty naturally, making it more flexible and adaptive. Through this method, the system generates a priority decision plan for batch URL acquisition that can both maximize the expected revenue and maintain sufficient exploration.

[0096] A3.1: Obtain the expected reward data of the URL target set, calculate the optimal combination through the combinatorial optimization algorithm, and form an initial batch plan.

[0097] In the detailed implementation of Thompson sampling, the system first needs to obtain the expected reward data of the URL target set and calculate the optimal combination.

[0098] For each candidate URL, the system samples multiple values (usually 100 - 1000) from its posterior reward distribution, and calculates the expected reward value and its confidence interval. These reward data are usually measured by "the effective information amount obtained per unit of resource", such as the number of valid phone numbers obtained per request, the amount of useful data crawled per second, etc.

[0099] After obtaining the expected reward data, the system faces a combinatorial optimization problem: how to select a set of URLs to maximize the overall expected reward under limited resources. This is essentially a knapsack problem with constraints.

[0100] The system uses an improved greedy algorithm or dynamic programming method to solve the problem, taking into account the dependencies and complementary effects between URLs. For example, after accessing the list page of a certain food and beverage platform, the value of its detail page will increase accordingly; while the value of accessing multiple similar websites simultaneously may decrease due to information redundancy. By solving this optimization problem, the system forms an initial batch plan, determining which URLs should be accessed in the current batch, as well as their access order and resource allocation ratio.

[0101] A3.2: According to the initial batch plan, use the submodular function maximization framework to calculate the diversity constraint and generate a diverse batch combination.

[0102] After forming the initial batch plan, the system introduces diversity constraints through the submodular function maximization framework to further optimize the batch combination.

[0103] Submodular functions are a class of functions with the property of "diminishing marginal returns", which are very suitable for modeling selection problems with diversity requirements. In an automated data collection system, continuously selecting URLs of the same type usually leads to information redundancy and reduces overall efficiency. The system defines a submodular function such that the marginal benefit of adding a new URL to the batch is inversely proportional to the similarity of the already selected URLs.

[0104] In specific implementation, the system first constructs a similarity matrix between URLs, calculating the similarity degree between each pair of URLs based on dimensions such as domain name, content type, target audience, etc. Then, it uses a greedy algorithm to gradually construct the batch under the condition of satisfying the submodular constraint: each time it selects the URL that can maximize the marginal benefit, while ensuring that the average similarity with the already selected URLs does not exceed a preset threshold. This method can ensure the overall expected benefit while ensuring that the batch contains URLs of different types and sources, improving information coverage and diversity.

[0105] For example, the system will mix and select food and beverage review websites, restaurant official websites, and local business directories in the same batch instead of concentrating on a single source, thus forming a diverse batch combination.

[0106] A3.3: According to the diverse batch combinations, use the federated learning framework to coordinate multi-node task allocation and form a distributed decision-making scheme.

[0107] After the diverse batch combinations are formed, the system uses the federated learning framework to coordinate multi-node task allocation and generate a distributed decision-making scheme.

[0108] In a large-scale automated data collection system, multiple collection nodes are usually deployed in different geographical locations or network environments. The federated learning framework enables these nodes to share knowledge and coordinate actions while maintaining a certain degree of autonomy.

[0109] First, each node maintains a local model and collects local observations, such as the response time and success rate of a specific website. Then, the nodes regularly synchronize model parameters to update the global knowledge base, but the original data remains local. This design improves the overall learning efficiency of the system and reduces communication overhead.

[0110] In terms of task allocation, the system adopts a decentralized coordination mechanism, such as a weighted voting or auction mechanism. For example, when multiple nodes are suitable for accessing a certain high-value URL, the system will comprehensively consider the current load, historical success rate, and network conditions of the nodes and allocate the task to the most suitable node. The system also implements knowledge distillation technology to compress and distribute the globally learned strategy to each node, enabling each node to quickly adapt to environmental changes. Through this coordination mechanism, the system forms an efficient distributed decision-making scheme, fully leveraging the advantages of the distributed architecture and improving the overall collection efficiency and robustness.

[0111] A3.4: According to the distributed decision-making scheme, execute a multi-objective optimization algorithm to integrate compliance constraints and generate a final URL batch acquisition execution plan for updating the prioritized URL queue.

[0112] After the distributed decision-making scheme is formed, the system integrates compliance constraints through a multi-objective optimization algorithm to generate a final URL batch acquisition execution plan.

[0113] The system regards data collection as a multi-objective optimization problem. The main objectives include: maximizing data value (such as the number of phone numbers obtained), minimizing resource consumption (such as the number of requests and bandwidth usage), and minimizing compliance risks (such as complying with robots.txt rules and avoiding overloading the target website).

[0114] The system uses the Pareto optimization method to find the best balance among these objectives.

[0115] In terms of specific implementation, the system first converts compliance constraints into hard conditions and soft penalty terms. For hard constraints such as complying with the robots.txt prohibition rules, the system will directly exclude URLs that violate these rules; for soft constraints such as request frequency limits, they are converted into penalty terms and added to the optimization objective function.

[0116] The system also establishes an adaptive rate limit mechanism to dynamically adjust the request frequency according to the response time and error rate of the target website, avoiding triggering the anti-crawling mechanism. Through this multi-objective optimization method, the system generates a final URL batch acquisition execution plan that balances data value, resource efficiency, and compliance risks. This plan clearly specifies the access time, request parameters, and processing priorities of each URL, and is used to guide the system to update the URL queue with priority sorting, ensuring that the acquisition process is both efficient and compliant and can operate stably in the long term.

[0117] A4: According to the priority decision-making scheme for URL batch acquisition, dynamically adjust the weight parameters of the URL exploration and exploitation strategies, calculate the priority scores of each URL, update the URL queue with priority sorting, and form an optimized URL access order and frequency.

[0118] In this link, the system further realizes the adjustment of the dynamic exploration and exploitation strategy and the priority update according to the URL batch acquisition priority decision-making scheme generated by A3.

[0119] First of all, the system dynamically adjusts the weight parameters of the exploration and exploitation strategies according to the current task progress and resource status.

[0120] For example, in the initial stage of acquisition, the system tends to increase the exploration weight, try various types of URLs, and accumulate experience; while in the later stage of acquisition, it increases the exploitation weight and concentrates resources on known high-return URL types to ensure results.

[0121] Secondly, the system dynamically adjusts the specific parameters of each URL according to real-time feedback. When it is found that the data quality of a certain type of URL has been significantly improved recently, its priority will be correspondingly increased.

[0122] In addition, the system also considers time and environmental factors, such as reducing requests during the peak website traffic period to avoid triggering the protection mechanism, or increasing access to e-commerce websites at certain specific time points (such as holiday promotion periods). Through these multi-dimensional dynamic adjustments, the system continuously updates the priority scores of each item in the URL queue and re-sorts them to form an optimized URL access order and frequency. This adaptive mechanism enables the system to operate efficiently in a complex and changing network environment and flexibly respond to various challenges.

[0123] S1.4: Conduct timeliness evaluation based on the said URL queue, dynamically adjust the access frequency and priority, and form the said set of entry URLs.

[0124] In the timeliness evaluation and adaptive adjustment phase, the system conducts dynamic timeliness analysis and adjustment on the priority queue established in S1.3.

[0125] The system first establishes an update cycle model for website content, and analyzes the content change patterns of different websites through historical crawling records.

[0126] For example, the promotional information on an e-commerce platform may be updated daily, while the basic contact information of a company may change monthly or quarterly. Based on these analyses, the system establishes a timeliness evaluation model to set appropriate access cycles for different types of URLs. For high-timeliness content (such as the contact information of new positions on a recruitment website), the system will increase its priority in the queue and the access frequency; while for low-timeliness content (such as the fixed phone number of a government agency), the access frequency will be correspondingly reduced, and resources will be allocated to URLs that need to be updated in a timely manner.

[0127] The system also implements an adaptive mechanism that can adjust the estimated update cycle according to the actual collection results. For example, if no content changes are found after consecutive visits to a certain catering website, the system will automatically extend the access interval; once it is found that the content starts to change frequently, the access interval will be correspondingly shortened. Through this dynamic adjustment, the system forms a set of entry URLs that takes into account both comprehensive coverage and key focus, which can not only ensure the timeliness of data but also optimize the resource usage efficiency.

[0128] As Figure 3 shown, S2 specifically includes: S2.1: Obtain the web page content according to the set of entry URLs, and use HTML parsing and cleaning to obtain a standardized DOM tree structure.

[0129] In the page acquisition and preprocessing phase, the system starts the actual data collection work based on the optimized set of entry URLs.

[0130] First, the system sends an HTTP request to obtain the original content of the target web page. This process needs to handle various network conditions and server responses, including setting a reasonable timeout mechanism, handling redirects, maintaining the Cookie status, etc.

[0131] After obtaining the original content, the system first performs an encoding detection to automatically identify the character encoding of the page (such as UTF-8, GB2312, GBK, etc.) to ensure that multi-language content such as Chinese can be correctly parsed. Then, the system performs HTML parsing to convert the text content into a structured DOM tree. This process uses professional HTML parsers (such as lxml, BeautifulSoup, etc.) that can handle non-standard HTML formats and fix common tag errors. After parsing, the system performs content cleaning to remove non-content elements such as JavaScript code, CSS styles, comments, etc., and at the same time identifies and filters out parts unrelated to the target data such as advertising content, navigation bars, and footers. This step is crucial for improving the accuracy of subsequent analysis and avoiding interference from noisy data on the analysis results. Finally, the system normalizes the cleaned content into a standard DOM tree structure for unified processing by subsequent algorithms.

[0132] The entire preprocessing process takes into account the handling of various abnormal situations, such as automatic repair of missing tags, normalization of special characters, etc., to ensure the stability and accuracy of subsequent analysis. Through these processes, the system converts the original chaotic web page content into a DOM tree with a clear structure and easy to analyze, laying a solid foundation for intelligent link discovery.

[0133] S2.2: According to the DOM tree structure, use regular expressions, XPath positioning, and CSS selectors to extract links to form an initial link set.

[0134] In the multi-strategy link extraction session, the system uses a variety of technical means to process the DOM tree structure in parallel to extract valuable links to the greatest extent.

[0135] First, the system uses regular expressions to match all possible URL patterns to identify http(s) links in the text content. This method can capture links that are not in the standard a tag, such as plain text URLs or links in JavaScript code.

[0136] At the same time, the system uses XPath technology to accurately locate links in specific structures, such as / / div[@class="content"] / a / @href, which can locate all links in the content area.

[0137] In addition, the system also uses CSS selector technology, such as.product-list.itema, to select all item links in the product list.

[0138] In addition to these static extraction methods, the system also integrates a JavaScript execution environment that can capture dynamically generated links, addressing the challenges posed by the widespread use of AJAX technology for dynamic content loading on modern websites. For example, when the page scrolls to load more content or the "Show More" button is clicked, the system can simulate these actions and extract newly emerged links.

[0139] The system also specifically handles some special cases, such as the automatic completion of relative paths (converting " / products / 1" to a full URL), URL decoding (processing encoded characters such as %20), and removing session identifiers from URLs, ensuring that the extracted links are in a unified and valid format.

[0140] Through the parallel application of these multiple strategies, the system can comprehensively collect link resources on the page, forming an initial set containing links from various sources, providing rich candidates for subsequent value evaluation. This multi-strategy parallel approach significantly improves the coverage and adaptability of link discovery and can handle websites with various different structures and technical implementations.

[0141] S2.3: According to the initial link set, through the TF-IDF algorithm, analyze the relationship between the link text and the context, and calculate the semantic similarity in combination with the Word2Vec model to obtain the link value scoring result; In the link value evaluation stage, the system deeply analyzes the extracted initial link set and calculates a value score for each link.

[0142] First, the system uses the TF-IDF algorithm to analyze the relevance between the link text and the context. The system regards the link text and its surrounding context as a document, calculates the term frequency (TF) and inverse document frequency (IDF), and identifies highly discriminative keywords. For example, links containing words such as "Contact Us", "Phone", "Customer Service", etc. in the link text or surrounding text will receive a higher relevance score.

[0143] At the same time, the system combines the Word2Vec model for in-depth semantic analysis. This model can understand the semantic relationships between words through pre-trained word vectors and can identify relevant content even if there are no exactly matching keywords.

[0144] For example, even if the link does not directly contain the word "Phone", but has semantically related words such as "Dial", "Consult", etc., the system can identify its potential value.

[0145] In addition to text semantics, the system also evaluates the structural features of links, such as the depth of DOM tree nodes (usually, links in the main content are more valuable than those in the navigation bar or footer), the density of sub-links (the more links pointing to a page, the more important that page is usually), and the position of the link on the page (links in the central area of the page are usually more important than those on the edge).

[0146] The system also considers historical data, such as the historical output rate of specific path patterns under this domain name. All these features are integrated through a random forest model to calculate a comprehensive value score for each link. The score range is usually 0 - 100, and the higher the score, the more likely the link is to contain the target information. This multi-dimensional scoring mechanism can comprehensively evaluate the potential value of links and provide a scientific basis for subsequent link screening.

[0147] S2.4: According to the link value scoring results, set a dynamic threshold to filter ads and low-value links, and use a heuristic algorithm to remove automated data collection traps to form a high-value link queue.

[0148] In the link filtering and optimization process, the system intelligently screens and processes the links scored in S2.3 to form a final high-value link queue.

[0149] First, the system sets a dynamic threshold to filter low-value links. This threshold is not fixed but is automatically adjusted according to the overall quality distribution of the current batch of links.

[0150] For example, if the current batch of links is generally of high quality, the system will increase the threshold to retain only the highest-quality links; otherwise, it will appropriately lower the threshold to ensure sufficient collection volume. The system pays special attention to and filters ad links, automatically removing such interference items by identifying common ad features (such as containing keywords like "ad", "sponsor", "promotion", etc., or pointing to ad network domain names).

[0151] At the same time, the system uses a heuristic algorithm to identify and avoid automated data collection traps, such as infinite calendar flipping, tag loops, parameter traps, etc., which can cause automated data collection to fall into a loop or an exponential URL explosion pattern.

[0152] The system will also detect and handle URL duplication problems, standardize URLs that have different forms but actually point to the same content (such as URLs with different session identifiers or sorting parameters) to avoid repeated access.

[0153] In addition, the system optimizes the retained high-value links, including completing relative paths, removing URL anchors, standardizing parameter order, etc., to ensure that the link format is unified and standardized.

[0154] Finally, the system sorts the links according to the value scores and sets a reasonable collection interval to avoid sending too many requests to the same domain name within a short period of time.

[0155] Through this series of filtering and optimization processes, the system extracts a high-quality and high-value link queue from the original mixed link set, providing an efficient set of target pages for the subsequent phone number extraction process, and significantly improving the overall collection efficiency and data quality.

[0156] As Figure 4 shown, S3 specifically includes: S3.1: Obtain the page content based on the high-value link queue, and use DOM parsing to extract the visible text content.

[0157] In the page text extraction process, the system accesses the target pages one by one based on the high-value link queue and extracts the text content that may contain phone numbers.

[0158] First, the system sends an HTTP request to obtain the page content and dynamically adjusts the request strategy according to the actual situation, such as setting different User-Agents, maintaining the Cookie status, or handling JavaScript redirects, etc., to cope with the access restrictions of various websites.

[0159] After obtaining the content, the system extracts the visible text through DOM parsing. This process not only includes obvious text elements such as regular paragraphs and headings, but also pays special attention to special areas that may contain contact information, such as the footer, sidebar, contact page, etc.

[0160] The system can intelligently handle various complex situations, such as handling special encodings (converting HTML entities such as, ') to normal characters), identifying the text alternative descriptions (alt attributes) in pictures, and even trying to extract the content of pseudo-elements in CSS styles (such as the content added through, ).

[0161] It is particularly worth mentioning that the system has also realized the detection and extraction of hidden text. Some websites will use various technologies to hide phone numbers, such as using CSS's display, visibility, or setting the text color to the same as the background. The system can identify and extract this hidden content by analyzing CSS properties.

[0162] In addition, the system also pays attention to the dynamically loaded content, and obtains the complete information by simulating user interactions (such as clicking the "Show More" button or triggering specific events). For complex layouts, the system will analyze the spatial relationship of elements to correctly associate the phone number with its description text.

[0163] Through these comprehensive and refined extraction techniques, the system ensures complete coverage of all areas on the page that may contain phone numbers, and outputs cleaned and normalized plain text content, laying a foundation for subsequent digital sequence recognition.

[0164] S3.2: According to the visible text content, use regular expressions to extract candidate digital sequences and record the context information of each candidate digital sequence in the original text.

[0165] In the digital sequence recognition stage, the system conducts a refined analysis of the extracted plain text content to identify all digital sequences that may form phone numbers.

[0166] First, the system uses a series of specially designed regular expression patterns to identify various forms of digital combinations. These patterns include: consecutive digits (such as 13800138000), digits with separators (such as 010-88888888, 0755.83744944), digits with parentheses (such as (010)88888888), digits with international prefixes (such as +8613800138000), and various mixed forms (such as +86(10)6552-9988).

[0167] The system's regular expressions are carefully designed to handle the phone number format habits of different countries and regions, such as the 11-digit format of Chinese mobile phone numbers, the 10-digit format with area codes in the United States, etc.

[0168] At the same time, the system will handle some special cases, such as the digits in the text may be separated by spaces, tab characters or line breaks, and may even be represented by full-width digits or Chinese characters.

[0169] For each identified digital sequence, the system not only records the sequence itself, but also records its complete context information in the original text, usually including 50-100 characters before and after. These context information are crucial for subsequent phone number verification and function recognition, and can provide key hints such as "Customer service phone:", "Order hotline:", "For service:", etc.

[0170] The system will also record the position information of the digital sequence on the page, such as the type of HTML element it is in (whether it is in 、 、 in tags such as , CSS class names (such as class="tel" or class="contact", etc.), and the spatial position relative to the page. This information helps to judge the importance and function of the digital sequence.

[0171] Through this series of delicate recognition and information association processing, the system outputs a set of candidate digital sequences containing rich metadata, providing a comprehensive analysis basis for the next step of telephone number pattern matching.

[0172] S3.3: According to the candidate digital sequence, use the telephone number pattern knowledge base to perform various combinations and formatting processes to generate a list of initially recognized telephone numbers.

[0173] In the telephone number pattern matching link, the system uses the global telephone number format knowledge base to verify and format the candidate digital sequence.

[0174] The telephone number knowledge base of the system covers the number rules of more than 200 countries and regions around the world, including the international area codes of each country / region, the length rules of domestic area codes, the total length requirements of numbers, the prefix rules of mobile phone numbers, etc.

[0175] For example, Chinese mobile phone numbers must be 11 digits and start with 1, and fixed telephones usually consist of an area code (such as 010, 0755) plus a 7-8 digit local number; the United States and Canada adopt the North American Numbering Plan (NANP), using a 3-digit area code plus a 7-digit local number, etc.

[0176] The system performs various combinations and formatting processes on each candidate digital sequence. For example, for "01088888888", the system will try various splitting methods such as "010-8888-8888", "0108-888-888", etc., and then verify which splitting conforms to the valid telephone number format according to the knowledge base.

[0177] The system can also handle some special situations, such as the short representation where locals are used to omitting the area code, the complex format using extension numbers, or the situation where multiple telephone numbers are included in the same text block. For multinational enterprise websites, the system can identify the telephone number formats of multiple countries / regions and correctly classify them.

[0178] During the verification process, the system not only considers the compliance of the number format, but also combines some heuristic rules to enhance the judgment accuracy, such as excluding digital sequences that are obviously product models, prices, dates, etc.

[0179] To improve efficiency, the system adopts a multi-level filtering strategy: First, it quickly screens out sequences that clearly do not conform to the characteristics of telephone numbers using simple rules, and then conducts more detailed format verification on potentially valid sequences. Through this series of complex matching and verification processes, the system outputs a list of initially identified telephone numbers, each number marked with its possible country / region attribution and format validity score, providing a basis for subsequent context verification and classification.

[0180] S3.4: According to the said telephone number list and combined with context information, conduct verification and functional type identification to form a page queue containing valid telephone numbers.

[0181] In the context verification and classification stage, the system conducts in-depth context analysis on the initially identified telephone numbers to further confirm their validity and identify their functional types.

[0182] First, the system analyzes the context vocabulary around each telephone number to find clear identifiers that can confirm it as a telephone number, such as keywords like "phone", "Tel", "contact", "dial", etc.

[0183] The system uses a sliding window method to analyze the text in different ranges before and after the number, and assigns weights according to the distance between the keyword and the number. The closer the keyword is, the greater its influence.

[0184] This context-based verification can effectively exclude digital sequences that, although conforming to the format, are not actually telephone numbers, such as product numbers, order numbers, etc.

[0185] After confirming the validity, the system further analyzes the context to identify the functional type of the telephone number. The system uses a pre-defined functional type dictionary, which contains various common telephone function classifications and their corresponding characteristic words, such as customer service (customer service, service, consultation, support), sales (sales, order, purchase, sales), technical support (technology, failure, repair, technical), etc.

[0186] By matching the characteristic words in the context, the system can assign the most likely functional label to each telephone number.

[0187] For cases where the context information is insufficient, the system will analyze the position of the number on the page and the DOM structure characteristics. For example, a number located on the "Contact Us" page is more likely to be a customer service phone number, and a number located on the product details page is more likely to be a sales phone number.

[0188] In addition, the system will also identify the usage time limit of the number, such as time period markings like "9:00 - 18:00 on weekdays".

[0189] After completing these analyses, the system stores each confirmed valid phone number, along with the URL and title of the current page, in queue B in the form of key-value pairs. Each entry in this queue contains complete metadata, such as the number text, standardized format, country / region attribution, function type, credibility score, etc., providing rich structured data for subsequent merchant information association. Through this in-depth context analysis and function recognition, the system significantly improves the accuracy and practical value of phone number extraction.

[0190] Among them, S4 specifically includes: S4.1: According to the page queue, extract the second-level domain name and the top-level domain name, and use the domain name clustering algorithm for grouping to generate the domain name grouping result.

[0191] In the domain name extraction and analysis phase, the system systematically processes the page queue containing valid phone numbers to achieve a preliminary classification of merchant information.

[0192] First, the system parses each URL to extract its second-level domain name and top-level domain name. For example, from a URL like "https: / / beijing.shop.example.com / contact", the system identifies the second-level domain name "shop.example" and the top-level domain name "com", while recording the subdomain "beijing" as possible geographical information. The system also identifies some special cases, such as websites using country code top-level domains (such as.cn,.jp) may represent business entities in specific countries, and websites using special domain names such as.edu,.gov may be educational institutions or government departments. After extracting the domain names, the system uses the domain name clustering algorithm for grouping. This clustering is not only based on exact matches but also considers the similarity of domain names, and can identify cases such as "shop.example.com" and "mobile.example.com" that originate from the same organization but use different subdomains. The system uses algorithms such as edit distance and longest common substring to calculate the domain name similarity and combines known common domain name patterns (such as common subdomain prefixes like www, m, shop used by enterprises) for intelligent matching. In addition, the system also analyzes the path structure pattern of the URL to identify different merchants that may come from the same content management system or e-commerce platform. For example, "platform.com / shop / A" and "platform.com / shop / B" may be different merchants on the same platform. Through these multi-dimensional domain name analysis and clustering techniques, the system generates the domain name grouping result, effectively grouping the pages that may belong to the same merchant or the same organization into one group, laying the foundation for subsequent merchant information association. This preliminary grouping based on domain names greatly improves the efficiency of data processing and avoids the redundant work of independently processing each URL.

[0193] S4.2: Establish a domain name - merchant mapping table based on commercial databases and historical collection data.

[0194] In the link building process of the domain name - merchant mapping table, the system integrates multiple data sources to establish the corresponding relationship between domain names and actual merchant entities.

[0195] First, the system utilizes existing commercial database resources, such as enterprise industrial and commercial registration databases, commercial information service platforms (such as Qichacha and Tianyancha), and industry directory databases, to obtain known domain name - merchant corresponding relationships. These official or professional data sources provide a large amount of verified basic mapping information.

[0196] Second, the system analyzes historical collection data to extract the association patterns between domain names and merchant names. Through statistical analysis of a large amount of historical data, the system can identify those high - frequency and stable corresponding relationships, such as a specific domain name almost always appearing simultaneously with a certain merchant name.

[0197] The system also adopts machine learning technology to train a specialized mapping prediction model. This model comprehensively considers domain name text features (such as brand keywords contained in the domain name), web page content features (such as the company name in the website logo and copyright statement), and link relationship features (such as the citation patterns of other known merchant websites) to predict the merchant entities that an unknown domain name may correspond to.

[0198] This model continuously improves the accuracy through continuous learning and can handle new domain names without direct matching records.

[0199] In addition, the system also implements a manual verification and feedback mechanism, allowing experts to review and correct the automatically generated mapping relationships. These feedbacks are used to further improve the model performance. Through these multi - source data fusion and intelligent learning technologies, the system constructs a comprehensive and accurate domain name - merchant mapping table. This table not only contains direct mapping relationships but also records the credibility scores of the mappings and data sources, providing reliable knowledge base support for subsequent merchant information association. The construction of such a mapping table is a dynamic and continuous process, and the system will regularly update and expand the mapping data to ensure its synchronization with the constantly changing Internet business environment.

[0200] S4.3: According to the domain name grouping results and the domain name - merchant mapping table, add merchant basic information to each URL to generate a URL dataset associated with merchants.

[0201] In the initial link of merchant information association, the system combines the previously generated domain name grouping results with the domain name - merchant mapping table to add the corresponding merchant basic information to each URL.

[0202] First, the system queries and matches each domain name group, and finds the corresponding merchant records from the domain name-merchant mapping table. For the case of direct matching, the system directly associates the basic information such as the corresponding merchant ID, name, and industry classification.

[0203] For domain names without direct matching records, the system will attempt approximate matching based on similarity calculation and rule inference, such as handling cases of domain name variants (example-shop.com and exampleshop.com) or sub-domain name changes (shop.example.com and m.example.com).

[0204] After determining the merchant association, the system marks each URL with a set of core merchant attributes, including unique identifiers (such as merchant IDs), merchant names (which may include formal names and common abbreviations), industry classifications (such as major categories like catering, retail, and service industries, as well as more detailed sub-categories), business scales (such as large chains, small and medium-sized enterprises, individual merchants, etc.), establishment time, geographical information, etc.

[0205] For some special cases, such as a domain name may correspond to multiple merchants (such as a commercial platform website) or a merchant may use multiple domain names, the system will establish a many-to-many association relationship and record the credibility scores of each association.

[0206] In addition, the system also marks each association with data sources and timestamps, facilitating subsequent data updates and conflict resolution. Through this systematic information association processing, the URL data is transformed from simple web address links into structured data with rich merchant context, forming a URL data set associated with merchants. This association not only provides valuable business background information but also provides important classification dimensions for subsequent data grouping and content analysis, greatly enhancing the business value and application potential of the data.

[0207] S4.4: According to the said URL data set, use text clustering algorithms to conduct content segmentation on the page titles and content summaries to form the said merchant data set.

[0208] In the content-based grouping optimization phase, the system conducts more refined content analysis and grouping on the URL data set with associated merchant information.

[0209] First, the system analyzes the titles and content summaries of the pages corresponding to each URL, extracts keywords and theme information. Through natural language processing technology, the system identifies the main content types of the pages, such as product introduction pages, contact information pages, company profile pages, etc.

[0210] The system uses text clustering algorithms (such as K-means, hierarchical clustering, or topic models) to conduct fine-grained grouping on different content pages of the same merchant.

[0211] For example, the pages of all branches of a certain restaurant may form a sub-cluster, the menu pages form another sub-cluster, and the order contact pages form a third sub-cluster. This clustering not only considers text similarity but also combines URL path patterns and page structure features, enabling more accurate identification of the functional categories of content. The system pays particular attention to pages containing contact information and marks them as high-value data sources.

[0212] For large merchant websites, the system can also identify page groupings for different departments or business lines, such as the sales department, customer service center, technical support, etc. This is of great guiding significance for the subsequent functional classification of phone numbers. In addition, the system will also analyze the time attributes of pages to identify regularly updated content (such as promotional information) and relatively stable content (such as basic contact information) for differential processing.

[0213] Through this fine-grained grouping based on content, the system organizes the originally potentially mixed URL data into a merchant data set with a clear structure and well-defined functions. Each grouping has specific content characteristics and business functions. This optimized data structure not only improves the accuracy of subsequent AI model extraction but also provides a more reasonable organizational framework for data presentation and application, making the final data product more in line with actual business needs and user usage habits.

[0214] S4.5: Collect the raw data generated by automated data collection activities, and through normalization and feature extraction processing, form a standardized time series data set.

[0215] In the data processing and standardization link of automated data collection activity data, the system systematically processes the raw data generated during the collection process to prepare for subsequent time series analysis.

[0216] First, the system collects all the raw data generated by automated data collection activities, including the timestamp, URL, response status code, response time, data volume size of each request, and the extracted information content, etc. These raw data usually have inconsistent formats, large scales, and contain noise, and need to be standardized for processing.

[0217] The system first cleans the data, dealing with missing values (such as cases where some requests do not get responses), outliers (such as extreme response times), and conflicting data (such as different contents obtained for the same URL in a short period of time). Then, the system normalizes the data, uniformly converting features with different dimensions (such as response time in milliseconds and data volume in KB) into the standard range, usually using Z-score normalization or Min-Max scaling methods. Next, the system performs feature extraction, mining valuable time pattern features from the raw data, such as the access frequency of a specific URL, the content update cycle, the time variation of data acquisition success rate, etc.

[0218] The system will also construct derivative features, such as second-order features like calculating data change rate, request density, content similarity, etc., to enhance the expressive ability of the data. Finally, the system aligns the data according to the time granularity based on business requirements, and may aggregate the original second-level data into minute-level, hour-level, or day-level time series for subsequent analysis.

[0219] Through this series of processes, the system converts the chaotic original automated data collection activity data into a standardized time series data set with unified structure, rich features, and time alignment. These data not only reflect the content change rules of the target website but also record the performance characteristics of the automated data collection system itself, providing a high-quality training and inference basis for the subsequent diffusion model.

[0220] S4.6: According to the standardized time series data set, use the conditional diffusion model to perform the forward diffusion and reverse diffusion processes to generate time series interpolation data.

[0221] In the conditional diffusion model and time series interpolation link, the system uses the advanced diffusion probability model technology to handle the missing value problem in time series data.

[0222] The core idea of the conditional diffusion model is to regard the data generation process as a process of gradually denoising, and this technology is particularly suitable for processing website content change data with complex time dependencies.

[0223] First, the system defines the forward diffusion process, gradually adding Gaussian noise to the complete time series data through multiple steps until it is completely randomized.

[0224] Then, the system trains a neural network to learn the reverse diffusion process, that is, gradually recovering the original signal from the noise. This network usually adopts the U-Net or Transformer architecture and can effectively capture the long-term and short-term dependencies of time series data.

[0225] After training, the system uses this model for conditional generation: when encountering a missing segment in the time series, the known part of the time series is used as the conditional input to guide the model to generate the missing part consistent with the known data.

[0226] Specifically, the system will keep the known data points unchanged and only apply the reverse diffusion process to the missing part to gradually recover the possible data values from the random noise. This conditional generation method ensures the natural continuity between the interpolation result and the known data. Compared with traditional interpolation methods, the diffusion model can generate results that more conform to the internal distribution characteristics of the data, especially for website content update data with complex patterns (such as periodicity, trend, and sudden changes).

[0227] For example, when an e-commerce website is temporarily inaccessible due to technical maintenance, the system can predict the possible content changes during this period based on historical access patterns and automatically adjust the prediction results after access is restored. Through this advanced time series imputation technology, the system can effectively handle data missing problems caused by various reasons, ensuring the continuity and integrity of time series data and providing a reliable basis for subsequent analysis.

[0228] S4.7: According to the time series imputation data, use the diversity sampling algorithm for data analysis to obtain the prediction uncertainty index.

[0229] In the diversity sampling and uncertainty assessment phase, the system deeply analyzes the time series imputation data generated by the diffusion model to quantify the reliability of the prediction.

[0230] First, the system adopts a diversity sampling strategy. Instead of simply generating a single prediction result, it runs the diffusion model multiple times, each time using a different random seed, to generate multiple sets of possible imputation schemes (usually 50 - 100 sets). These different schemes together form a prediction distribution, reflecting the uncertainty of the model's predicted values at different time points.

[0231] Then, for each time point, the system calculates the statistical characteristics of all sampling results, such as the mean (as the final prediction value), the standard deviation (as a measure of uncertainty), and quantiles (used to construct the prediction interval), etc. The system pays special attention to those time points with a large variance between samples, which usually indicates a high degree of uncertainty in the model's prediction at these points and may require more real data for verification.

[0232] Based on these statistical analyses, the system generates an uncertainty index for each predicted time point, usually represented as a confidence score between 0 - 1 or the width of the prediction interval.

[0233] The system also identifies different types of uncertainty sources, such as aleatoric uncertainty (the randomness of the data itself) and epistemic uncertainty (the limitations of the model's knowledge). For abnormal changes that may be caused by special events (such as a sharp increase in traffic on the e-commerce website during a promotion day), the system will mark them as high-uncertainty regions, reminding that special attention may be required.

[0234] In addition, the system continuously updates these uncertainty assessments over time. When more observed data is obtained, the prediction interval of the model usually narrows and the uncertainty decreases. Through this comprehensive diversity sampling and uncertainty assessment, the system not only provides specific prediction values but also quantifies the reliability of these predictions, enabling decision-makers to reasonably allocate resources according to the credibility of the predictions and more scientifically plan the data collection strategy.

[0235] S4.8: According to the predicted uncertainty index, execute the adaptive adjustment strategy to update the acquisition frequency parameter, and form data with better time continuity and integrity.

[0236] In the implementation link of the adaptive adjustment strategy, the system intelligently optimizes the acquisition strategy according to the uncertainty evaluation result, forming a closed-loop adaptive system.

[0237] First, the system establishes an uncertainty threshold rule, and divides the predicted time points into multiple levels according to the uncertainty level, such as highly certain (confidence level > 0.9), moderately certain (confidence level 0.6 - 0.9), and highly uncertain (confidence level < 0.6).

[0238] Then, the system designs a differentiated resource allocation strategy for each level: for highly certain time points, the system may reduce the actual acquisition frequency and rely more on model prediction; for moderately certain points, maintain the normal acquisition frequency; while for highly uncertain points, significantly increase the acquisition frequency to obtain more real data to reduce uncertainty. The system also considers the business value of the data. For data with low business value even if the prediction is uncertain (such as secondary information on non-core pages), a more conservative acquisition strategy may be adopted; while for high-value data (such as updates of contact information of important customers), even if the prediction is relatively certain, a certain acquisition frequency will be maintained to ensure security.

[0239] In addition, the system implements a dynamic feedback mechanism. When there is a significant difference between the actually acquired data and the prediction, it will trigger the retraining of the model or parameter adjustment to adapt to the change of data distribution.

[0240] The system also designs a resource balancing algorithm to ensure that under the overall resource constraints, each target website and time point can obtain a reasonable acquisition resource allocation. This intelligent resource scheduling not only considers the prediction uncertainty, but also takes into account practical factors such as network conditions, server load, and anti-crawling mechanisms. Through this series of adaptive adjustment strategies, the system can maximize the resource utilization efficiency on the premise of ensuring data quality, and achieve the optimal time continuity and integrity of the data. As the system runs for a longer time and the data accumulation becomes richer, this adaptive mechanism will become more and more accurate, forming an intelligent acquisition system that continuously self-optimizes.

[0241] S5 specifically includes: S5.1: Collect historical web page data containing various website formats and field types.

[0242] In the link of collecting historical web page data, the system establishes a comprehensive and diverse training data resource library, providing a solid foundation for the subsequent training of the AI model.

[0243] First, the system systematically collects historical web page data covering various industries and types of websites, with particular attention to pages containing rich structured information (such as merchant names, addresses, phone numbers, etc.). The collection sources are diverse, including public web archives (such as the Wayback Machine of Internet Archive), historical snapshots provided by commercial data providers, and collection records accumulated during the long-term operation of the system itself, etc.

[0244] The system classifies the collected pages in multiple dimensions, marking them by industry (such as catering, retail, service industry, etc.), website type (such as corporate official websites, e-commerce platforms, social media, etc.), content structure (such as tabular, list, paragraph, etc.), and technical implementation (such as static HTML, JavaScript rendering, responsive design, etc.), ensuring the diversity and representativeness of the training data.

[0245] In particular, the system will focus on collecting those web page types with special challenges, such as abnormally complex layouts, non-standard field representation methods, multi-language mixed content, pages with a large amount of image-based information display, etc., to enhance the generalization ability of the model. In addition, the system also collects data of the same website at different times to capture the evolution trends of website design and content organization, enabling the model to adapt to the ever-changing web design styles.

[0246] For scarce but important web page types, the system will also adopt synthetic data technology to create more training samples through template mutation or content recombination. All the collected data undergoes preliminary quality screening to eliminate pages that are significantly damaged, have incomplete content, or are overly repetitive, and version management is carried out to record the data sources, collection times, and basic statistical characteristics.

[0247] Through this systematic historical data collection work, the system constructs a rich training resource library containing various website formats and field types, laying a solid foundation for subsequent expert annotation and model training.

[0248] S5.2: Establish a training data set through expert annotation, and annotate the keyword fields in the data including merchant names, addresses, business scopes, and the functional classification of phone numbers.

[0249] In the link of expert annotation and establishment of the training data set, the system organizes a professional team to perform high-quality manual annotation on the historical web page data to generate the standard answers required for supervised learning.

[0250] First, the system formulates detailed annotation guidelines, clearly defining various fields that need to be identified (such as merchant names, addresses, business scopes, business hours, etc.) and their standard formats, as well as the judgment criteria for the functional classification of phone numbers (such as switchboards, customer service, sales, technical support, etc.).

[0251] To ensure the annotation quality, the system adopts a multi-level review mechanism: after the primary annotators complete the basic annotation, the senior reviewers conduct a review. For complex or controversial cases, the domain experts make the final decision.

[0252] The system also implements an annotator performance evaluation and training mechanism. Through regular consistency tests and case discussions, the professional level and standard uniformity of the annotation team are continuously improved.

[0253] At the technical level, the system develops a dedicated annotation tool, which supports efficient field selection, attribute marking, and function classification. At the same time, it records the uncertainty and difficulty ratings during the annotation process, providing important references for subsequent model training. Considering the diverse data requirements, the system ensures the balance of annotation samples in dimensions such as industry distribution, website type, and field complexity, avoiding bias in the training data.

[0254] For rare but important cases (such as unconventional contact information formats), the system will specifically increase the annotation ratio of corresponding samples to ensure that the model can handle various edge cases.

[0255] In addition, the system also implements a data splitting strategy, dividing the annotated data into a training set, a validation set, and a test set according to a ratio of 8:1:1, ensuring the distribution similarity of the three sets in each dimension and avoiding data leakage (such as pages from the same website being scattered into different sets).

[0256] Through this professional and rigorous annotation process, the system establishes a high-quality training dataset, which contains rich merchant field information and telephone number function classification, providing a reliable supervision signal for the next feature engineering and model training.

[0257] S5.3: Perform feature engineering processing on the training dataset, extract DOM structure features, text semantic features, and positional relationship features, and construct a training model based on the DOM structure features, text semantic features, and positional relationship features.

[0258] In the feature engineering and training model construction link, the system deeply analyzes and extracts features from the annotated dataset, preparing rich input signals for subsequent model training.

[0259] First, the system extracts DOM structure features, including HTML tag types, tag nesting depth, element positions, CSS class names and IDs, element sizes, and visibility, etc. These features can reflect the structured information and visual layout of web pages.

[0260] For example, the system will identify that elements located within a specific container (such as class="contact-info") are more likely to contain contact information.

[0261] Secondly, the system extracts text semantic features, including the bag-of-words representation of the text content, TF-IDF features, word embedding vectors, named entity recognition results, etc. These features can capture the semantic information and entity types of the text.

[0262] For example, the system will learn to recognize the association patterns between keywords such as "contact phone number" and "dial" and phone numbers.

[0263] Thirdly, the system extracts positional relationship features, including the relative positions between elements, proximity relationships, possible text-number pairing patterns, etc. These features can express the spatial relationships between page elements.

[0264] For example, the system will learn to recognize the common layout patterns of label texts and their corresponding values, such as the pattern where the label is on the left and the value is on the right in "Phone: 12345678".

[0265] Based on these rich features, the system constructs the infrastructure of the training model. For the field extraction task, the system adopts an encoder-decoder structure, where the encoder is responsible for converting page features into high-dimensional representations, and the decoder is responsible for identifying specific fields from these representations.

[0266] For the phone number grouping and function recognition task, the system adopts a sequence labeling framework, regarding the phone number and its context as a sequence, and learning to predict the function labels of each phone number.

[0267] The system also implements feature selection and dimensionality reduction techniques, removing redundant features through methods such as correlation analysis and principal component analysis to improve the model training efficiency.

[0268] In addition, the system designs a feature fusion mechanism that can adaptively adjust the weights of different types of features and optimize the feature combination for different website structures. Through these systematic feature engineering and model construction efforts, the system provides rich and refined input representations for the subsequent deep learning model training, greatly improving the learning efficiency and performance level of the model.

[0269] S5.4: Adopt transfer learning methods to construct a field extraction neural network model based on a pre-trained language model and fine-tune it for web page structured information extraction.

[0270] In the transfer learning and field extraction model construction phase, the system utilizes the powerful semantic understanding ability of the pre-trained language model to develop a neural network model specifically for web page structured information extraction.

[0271] First of all, the system selects a suitable pre-trained language model as the basis, such as BERT, RoBERTa or their Chinese versions. These models have mastered rich language knowledge and semantic understanding ability through self-supervised learning on a large amount of text.

[0272] Then, the system performs domain adaptation fine-tuning on these pre-trained models, further training the models with a large amount of industry-related texts (such as business descriptions, product introductions, etc.) to enable them to better understand the terms and expressions in a specific domain.

[0273] Next, the system designs a dedicated task fine-tuning stage, combines the pre-trained model with a task-specific output layer, and constructs a complete field extraction network.

[0274] In terms of the specific architecture, the system adopts an encoder-decoder framework. The encoder is based on the pre-trained model and is responsible for converting the web page text and its structural features into a context-aware vector representation. The decoder adopts structures such as a conditional random field (CRF) or a pointer network and is responsible for accurately locating and extracting the target fields from these vector representations. The system also innovatively combines text and structural information, converts the HTML structure into special tokens and inserts them into the text sequence, enabling the model to simultaneously understand the semantics of the text content and the page structure.

[0275] For example, convert Phone: 12345678 Convert to a mixed input of special token sequences and text content. During the training process, the system adopts a multi-task learning method to optimize multiple related objectives (such as field boundary recognition, field type classification, entity relationship extraction, etc.) simultaneously, enabling the model to learn data features from different perspectives.

[0276] The system also implements adversarial training and data augmentation techniques. By generating adversarial samples and transforming existing samples, the generalization ability and robustness of the model are improved. Through this transfer learning-based method, the system effectively utilizes the language knowledge contained in the pre-trained model, greatly reducing the dependence on labeled data while improving the field extraction accuracy of the model under complex web page structures, especially for those field information with variable formats or non-standard expressions.

[0277] S5.5: Design the sequence labeling network structure and train the telephone number grouping recognition model using the BiLSTM-CRF architecture.

[0278] In the sequence labeling model and the telephone number grouping recognition section, the system designs a dedicated neural network architecture to identify the functional types and grouping relationships of multiple telephone numbers of the same merchant.

[0279] First, the system uses a bidirectional long short-term memory network (BiLSTM) as the basic structure. This network can effectively capture the forward and backward dependencies of sequence data and is particularly suitable for processing text and number sequences.

[0280] The BiLSTM network receives the input features processed by the embedding layer, including telephone number text, surrounding context vocabulary, location information, etc., and learns sequence patterns from both the forward and backward directions. Then, the system adds a conditional random field (CRF) layer on the output layer of the BiLSTM to form the BiLSTM-CRF architecture. The CRF layer can learn the transition probabilities between labels and consider the overall rationality of the label sequence. For example, it is unlikely that the same merchant has multiple "switchboard numbers" but may have multiple "branch numbers".

[0281] This structural design enables the model to not only focus on the local features of individual telephone numbers but also consider the overall label consistency constraints.

[0282] In terms of feature engineering, the system constructs rich feature vectors for each telephone number, including format features of the number itself (such as length, whether it contains an area code, etc.). In the sequence labeling model and the telephone number grouping recognition section, the system designs a dedicated neural network architecture to identify the functional types and grouping relationships of multiple telephone numbers of the same merchant.

[0283] First, the system uses a bidirectional long short-term memory network (BiLSTM) as the basic structure. This network can effectively capture the forward and backward dependencies of sequence data and is particularly suitable for processing text and number sequences.

[0284] The BiLSTM network receives input features processed by the embedding layer, including telephone number text, surrounding context words, location information, etc., and learns sequence patterns from both forward and backward directions. Then, the system adds a Conditional Random Field (CRF) layer on the output layer of the BiLSTM to form the BiLSTM-CRF architecture. The CRF layer can learn the transition probabilities between labels, taking into account the overall rationality of the label sequence. For example, it is unlikely for the same merchant to have multiple "switchboard numbers" but may have multiple "branch numbers".

[0285] This structural design enables the model to not only focus on the local features of a single telephone number but also consider the overall label consistency constraints.

[0286] In terms of feature engineering, the system constructs rich feature vectors for each telephone number, including format features of the number itself (such as length, whether it contains an area code, whether it is a mobile number), context semantic features (such as function indicator words "customer service", "sales", etc. that appear around), location features (such as relative position on the page, distance from other numbers), etc.

[0287] The system also introduces an attention mechanism, enabling the model to better focus on context information related to the function judgment of the telephone number, such as department names or service type descriptions that may appear before and after.

[0288] In addition, the system also implements a multi-head self-attention structure, enabling the model to simultaneously focus on different types of relevant information and integrate them. Through this professionally designed BiLSTM-CRF architecture, the system can accurately identify the respective function types (such as switchboard, customer service, sales, technical support, etc.) of multiple telephone numbers under the same merchant, and reasonably group telephone numbers with similar functions, providing an important basis for subsequent data integration and display.

[0289] S5.6: Optimize the hyperparameters of the field extraction network and the telephone number grouping recognition model through cross-validation to improve the generalization ability of the field extraction neural network model and the sequence-labeling-based telephone number grouping recognition model under different website formats.

[0290] In the cross-validation and hyperparameter optimization phase, the system uses a scientific and rigorous method to evaluate and optimize the model performance to ensure its good generalization ability under various website formats.

[0291] First, the system implements K-fold cross-validation, usually using a 10-fold scheme. The training dataset is divided into 10 subsets of similar size. Each time, 9 subsets are used to train the model, and the remaining 1 is used for validation. This is done 10 times in turn, and finally, the average performance is taken as the evaluation metric.

[0292] This method can comprehensively evaluate the performance of the model under different data distributions, avoiding the bias that a single test set may bring.

[0293] Based on the cross-validation results, the system conducts comprehensive hyperparameter optimization. The parameters to be adjusted include: network architecture parameters (such as the hidden layer size, number of layers, number of attention heads of LSTM, etc.), optimizer parameters (such as learning rate, momentum, weight decay, etc.), regularization parameters (such as dropout rate, L1 / L2 regularization strength, etc.), and training strategy parameters (such as batch size, learning rate scheduling strategy, early stopping conditions, etc.).

[0294] The system uses the Bayesian optimization method for efficient parameter search. Compared with traditional grid search or random search, this method can explore the parameter space more intelligently and quickly find a parameter combination close to the optimal one.

[0295] During the optimization process, the system separately evaluates the model performance for different types of website formats, especially focusing on the performance under those special formats (such as highly dynamic pages, pages with non-traditional layouts), ensuring that the model does not significantly deteriorate on certain specific types.

[0296] The system also adopts comprehensive multi-metric evaluation, considering accuracy, recall, F1-score, and custom metrics related to specific tasks simultaneously, such as the precision of field boundary recognition, the consistency of telephone number function classification, etc., to ensure that the model achieves a balanced performance in all aspects.

[0297] In addition, the system also implements an error analysis mechanism, detailedly recording and classifying the error cases on the validation set, identifying the shortcoming of the model and adjusting the architecture or parameters accordingly. Through this systematic cross-validation and hyperparameter optimization process, the system finally obtains a field extraction network and a telephone number grouping recognition model with stable performance and strong generalization ability. These models can maintain a high level of performance under various website formats and data distributions, providing reliable technical support for practical applications.

[0298] S5.7: According to the field extraction neural network model, perform automatic analysis on the pages of each merchant group, and identify and extract keyword field information.

[0299] In the application link of the field extraction neural network, the system deploys the optimized model to the actual production environment, and performs automated analysis and information extraction on the pages of each merchant group.

[0300] First, the system preprocesses the web page to be processed, including HTML parsing, text extraction, feature calculation, etc., and converts it into a model input with the same format as the training data. Then, the system activates the field extraction neural network, inputs the preprocessed page data into the model, and the neural network automatically identifies and locates the target field through multiple layers of calculations.

[0301] During the recognition process, the model calculates the probability scores for each token in the text belonging to various field types, such as "merchant name", "address", "business scope", "establishment time", etc., and then uses the forward-backward algorithm combined with the CRF layer to find the optimal field boundaries and type assignments.

[0302] This process makes full use of the pre-trained language model's ability to understand context and the sequence labeling model's ability to model label dependencies, and can accurately identify field information with complex formats and expressions.

[0303] For each identified field, the system also calculates a confidence score to reflect the certainty of the model about the extraction result. The system will pay special attention to those extraction results with low confidence, and may initiate a backup processing flow, such as using a rule engine for secondary verification or marking for manual review.

[0304] In addition, the system has also implemented an adaptive processing mechanism that automatically adjusts the processing strategy for different types of web pages. For example, it increases the depth of feature extraction for pages with complex structures, and may need to render and then analyze highly dynamic pages, etc.

[0305] The system also records the intermediate states and attention weight distributions during the processing, which is convenient for subsequent result interpretation and model improvement. Through this intelligent field extraction process, the system can accurately identify and extract key merchant information from the mixed web page content, such as merchant name, detailed address, business scope, establishment time, registered capital, etc. These structured field information provide high-quality basic data for subsequent merchant portrait construction and data application.

[0306] S5.8: According to the telephone number grouping recognition model, automatically group and identify multiple telephone numbers of the same merchant, and identify and extract telephone number data.

[0307] In the application link of the telephone number grouping recognition model, the system uses the trained BiLSTM-CRF model to perform automated function recognition and grouping processing on multiple telephone numbers of the same merchant.

[0308] First, the system collects all the extracted phone numbers under the same merchant, along with their context information (such as the descriptive text around the number, the title and URL of the page where it is located, etc.), as input. Then, the system preprocesses this input data, including standardizing the phone number format, tokenizing and tagging the context text, calculating location features, etc., and converts it into a sequence input form acceptable to the model. Next, the system activates the phone number grouping recognition model. The model converts the input sequence into a vector representation through the embedding layer, then extracts sequence features through the BiLSTM layer, and finally outputs the most likely functional label for each phone number through the CRF layer.

[0309] This process not only considers the characteristics of each number itself and its direct context, but also takes into account the relationships between numbers and the rationality of the overall label sequence.

[0310] The system attaches the functional labels recognized by the model (such as switchboard, customer service, sales, technical support, complaints, reservations, etc.) to the corresponding phone numbers and initially groups the numbers according to functional similarity. In addition, the system analyzes the geographical characteristics of the numbers (such as the region to which the area code belongs) and usage scenarios (such as dedicated line, 24-hour service, etc.) to further refine the grouping information.

[0311] For some complex situations, such as a number that may serve multiple functions simultaneously, the system calculates the probability distribution of multiple labels and selects the most primary function as the main label, and other possible functions as auxiliary labels.

[0312] The system also records the confidence index of the grouping recognition and may initiate a manual review process for results with low confidence.

[0313] Through this intelligent phone number function recognition and grouping process, the system can convert the originally isolated phone number data into address book information with clear functional attributes and organizational structures, greatly improving the practicality and commercial value of the data. Users can quickly find the contact information suitable for specific purposes according to their needs. For example, when they need to consult product information, they can directly contact the sales department, and when they encounter technical problems, they can directly contact the technical support, significantly improving the information query and usage efficiency.

[0314] S5.9: Conduct a correlation analysis on the keyword field information and the phone number data, confirm the main merchant name of the data grouping, and obtain the structured data set.

[0315] In the correlation analysis and main merchant confirmation phase, the system conducts an in-depth correlation analysis on the extracted field information and phone number data to confirm the main merchant name and core information of the data grouping.

[0316] First, the system statistically analyzes the variants of merchant names that appear in each merchant group (such as "ABC Company", "ABC Group", "ABC Co., Ltd.", etc.), calculates the occurrence frequency, positional importance (such as appearing in the title or prominent position), and association strength (co-occurrence relationship with other keyword fields) of each variant.

[0317] Based on these statistical features, the system uses a weighted voting mechanism to determine the most likely main merchant name, which will be used as the primary key identifier for the entire dataset.

[0318] In addition, the system analyzes the association pattern between the merchant name and the phone number, identifies the most core contact information and its corresponding business entity. In complex cases, the system also analyzes the page hierarchy and content organization, differentiates relationships such as main business and subsidiary business, parent company and subsidiary company, etc., to ensure the accuracy and hierarchy of data grouping. The system pays special attention to possible multiple relationships, such as different brands or business lines under the same group, and represents these complex associations by constructing an entity relationship graph.

[0319] At the same time, the system also analyzes the consistency and complementarity between fields, such as the geographical consistency between the address and the area code of the phone number, the corresponding relationship between the business scope and the specific business department, etc., and improves the overall data credibility through these cross-verifications. For information with conflicts or inconsistencies, the system will perform intelligent reconciliation based on the credibility score and business importance, and preferentially retain more reliable and core information.

[0320] Finally, the system organizes all the information that has undergone association analysis and integration into a structured dataset centered on the main merchant, clearly displaying key data such as the basic information of the merchant, multi-level contact information, business scope, etc., and maintaining the association relationship and hierarchical structure between each piece of information. This structured dataset that has undergone in-depth association analysis and main entity confirmation not only solves the problems of data dispersion and identity confusion, but also provides a rich merchant portrait and relationship network, greatly enhancing the commercial application value and user experience of the data.

[0321] The embodiment of the present application also provides a cyclic automated data acquisition system, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above method is implemented.

[0322] The embodiment of the present application also provides a computer device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor can execute the method of the above cyclic automated data acquisition method.

[0323] The embodiments of the present application also provide a computer-readable storage medium storing computer instructions for causing a computer to execute the above-mentioned loop automated data collection method. The embodiments of the present application also provide a computer program product including computer instructions that, when executed by a processor, implement the steps of the above-mentioned loop automated data collection method.

[0324] The present application has the following technical effects: Through the knowledge graph distributed entry discovery mechanism and dynamic priority queue management, more efficient resource allocation and collection scheduling are achieved, improving the collection efficiency. The link discovery algorithm based on DOM structure features and semantic association degree improves the accuracy of link value evaluation and reduces the invalid collection rate. The telephone number recognition mechanism using a combination of multiple algorithms greatly improves the accuracy and adaptability of telephone number extraction. The self-attention diffusion model is used for time series interpolation, effectively solving the data continuity problem in the scenarios of interruption recovery and incremental update. The AI-based data extraction model can automatically adapt to different website structures, improving the accuracy and coverage of data extraction. The application of multi-dimensional data fingerprints and differential privacy technology ensures the uniqueness and compliance of data, improving the data quality.

[0325] Through the above technical solutions, the present application can achieve more efficient and accurate data collection in the context of the explosive growth of Internet information and the increasing complexity of website structures, meeting the application scenario requirements of business intelligence, risk monitoring, etc. with high requirements for data quality and timeliness. At the same time, the application of the multi-dimensional data fingerprint and differential privacy technology in the present application also ensures the uniqueness and compliance of data, providing an important guarantee for data security and privacy protection.

Claims

1. A cyclic automated data acquisition method, characterized in that, Including: Based on the distributed entry discovery mechanism of the knowledge graph, construct a multi-source data entry set, and use dynamic priority queue management, timeliness evaluation model and semi-parametric batch processing global decision-making mechanism to form an entry URL set; Based on the entry URL set, based on DOM structure feature analysis and semantic relevance evaluation, as well as link value scoring of TF-IDF and Word2Vec, form a high-value link queue; Based on the high-value link queue, obtain page content, use regular expressions to extract digital sequences, and use a telephone number pattern knowledge base for matching and judgment to form a page queue containing valid telephone numbers; Based on the page queue, use a self-attention diffusion model for time series interpolation, and use a domain name extraction algorithm and a domain name merchant mapping knowledge base for merchant information association to form a merchant data set; According to the merchant data set, use a field extraction neural network model based on supervised learning and a telephone number grouping recognition model based on sequence annotation for field information extraction and telephone number grouping recognition to generate a structured data set; Based on the structured data set, perform multi-dimensional data fingerprint generation for data deduplication, and use differential privacy technology for data protection to form a high-quality data set after deduplication and desensitization.

2. The method according to claim 1, wherein The construction of the multi-source data entry set based on the distributed entry discovery mechanism of the knowledge graph and the formation of the entry URL set by using dynamic priority queue management and timeliness evaluation model include: Obtain an initial URL set according to the search engine API, social media API and industry directory database; Based on the initial URL set and combined with domain knowledge, construct a knowledge graph, identify high-value entry nodes through graph analysis, and generate a preliminary screened URL subset; According to the URL subset, design a dynamic priority queue, and calculate the priority score based on the PageRank value, content update frequency and historical collection results of each URL to generate a URL queue sorted by priority; Conduct timeliness evaluation based on the URL queue, dynamically adjust the access frequency and priority, and form the entry URL set.

3. The method according to claim 1, wherein The formation of the high-value link queue based on the entry URL set, based on DOM structure feature analysis and semantic relevance evaluation, as well as link value scoring of TF-IDF and Word2Vec, includes: Obtain the web page content according to the entry URL set, and use HTML parsing and cleaning to obtain a standardized DOM tree structure; Based on the DOM tree structure, use regular expressions, XPath positioning and CSS selectors to extract links to form an initial link set; According to the initial link set, use the TF-IDF algorithm to analyze the relationship between the link text and the context, and combine with the Word2Vec model to calculate the semantic similarity to obtain the link value scoring result; Based on the link value scoring result, set a dynamic threshold to filter advertisements and low-value links, and use a heuristic algorithm to remove automated data collection traps to form a high-value link queue.

4. The method according to claim 1, wherein Obtaining page content based on the high-value link queue, using regular expressions to extract digital sequences, and performing matching judgments with the help of a telephone number pattern knowledge base to form a page queue containing valid telephone numbers, including: Obtaining page content based on the high-value link queue, and using DOM parsing to extract visible text content; According to the visible text content, using regular expressions to extract candidate digital sequences, and recording the context information of each candidate digital sequence in the original text; Based on the candidate digital sequences, using the telephone number pattern knowledge base to perform various combinations and formatting processes to generate a list of initially recognized telephone numbers; According to the telephone number list and combined with the context information, perform verification and functional type recognition to form a page queue containing valid telephone numbers.

5. The method according to claim 1, characterized in that, Based on the page queue, using a domain name extraction algorithm and a domain name merchant mapping knowledge base to perform merchant information association to form a merchant data set, including: Based on the page queue, extract the second-level domain name and the top-level domain name, and use a domain name clustering algorithm for grouping to generate a domain name grouping result; Establish a domain name merchant mapping table based on the commercial database and historical collection data; Based on the domain name grouping result and the domain name merchant mapping table, add merchant basic information to each URL to generate a URL data set associated with the merchant; According to the URL data set, use a text clustering algorithm to perform content segmentation on the page title and content summary to form the merchant data set.

6. The method according to claim 1, wherein The obtaining methods of the field extraction neural network model based on supervised learning and the telephone number grouping recognition model based on sequence labeling include: Collect historical web page data containing various website formats and field types; Establish a training data set through expert annotation, and annotate the keyword fields including merchant name, address, business scope and the functional classification of telephone numbers in the data; Perform feature engineering processing on the training data set, extract DOM structure features, text semantic features and positional relationship features, and construct a training model based on the DOM structure features, text semantic features and positional relationship features; Adopt a transfer learning method to construct a field extraction neural network model based on a pre-trained language model, and perform fine-tuning for web page structured information extraction; Design a sequence labeling network structure, and use the BiLSTM-CRF architecture to train the telephone number grouping recognition model; Optimize the hyperparameters of the field extraction network and the telephone number grouping recognition model through cross-validation, and improve the generalization ability of the field extraction neural network model and the telephone number grouping recognition model based on sequence labeling under different website formats; Based on the merchant data set, using the field extraction neural network model based on supervised learning and the telephone number grouping recognition model based on sequence labeling to perform field information extraction and telephone number grouping recognition to generate a structured data set, including: Based on the field extraction neural network model, perform automatic analysis on the pages of each merchant group, and identify and extract keyword field information; According to the telephone number grouping recognition model, perform automatic grouping and functional recognition on multiple telephone numbers of the same merchant, and identify and extract telephone number data; Perform a correlation analysis on the keyword field information and the telephone number data to confirm the main merchant name of the data grouping, and obtain the structured data set.

7. The method according to claim 1, characterized in that, The self-attention diffusion model performs time series imputation, including: Collect the raw data generated by the automated data collection activity, and through normalization and feature extraction processing, form a standardized time series data set; According to the standardized time series data set, use the conditional diffusion model to perform the forward diffusion and reverse diffusion processes to generate time series imputation data; According to the time series imputation data, use the diversity sampling algorithm to perform data analysis to obtain the prediction uncertainty index; According to the prediction uncertainty index, execute the adaptive adjustment strategy to update the collection frequency parameter to form data with better time continuity and integrity.

8. The method according to claim 2, characterized in that The steps of designing the dynamic priority queue according to the URL subset include: Execute a semi-parametric batch global decision mechanism with covariates, specifically including: Collect the feature data of the target website, and through feature engineering processing, generate a feature vector including domain name age, update frequency, and content richness; According to the feature vector, use a hybrid model of parametric and non-parametric to train and form a semi-parametric reward prediction model; According to the semi-parametric reward prediction model, execute the Thompson sampling algorithm to calculate the expected data revenue value of each URL target, and generate a priority decision plan for batch URL acquisition; According to the priority decision plan for batch URL acquisition, dynamically adjust the weight parameters of the URL exploration and exploitation strategy, calculate the priority score of each URL, and update the URL queue with priority sorting to form an optimized URL access order and frequency.

9. The method according to claim 8, wherein The steps of executing the Thompson sampling algorithm according to the semi-parametric reward prediction model to calculate the expected data revenue value of each URL target and generate a priority decision plan for batch URL acquisition include: Obtain the expected reward data of the URL target set, calculate the optimal combination through the combinatorial optimization algorithm, and form an initial batch plan; According to the initial batch plan, use the submodular function maximization framework to perform diversity constraint calculation to generate a diverse batch combination; According to the diverse batch combination, use the federated learning framework to coordinate multi-node task allocation to form a distributed decision plan; According to the distributed decision plan, execute the multi-objective optimization algorithm to fuse the compliance constraint conditions to generate the final URL batch acquisition execution plan for updating the URL queue with priority sorting.

10. A cyclic automated data acquisition system, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the method according to any one of claims 1 to 9.

Images