A method and system for intelligently collecting and analyzing web merchant information

Abstract

The present invention discloses a method and system for intelligent collection and analysis of merchant information on web pages, which realizes the automatic collection of merchant information through five main steps: intelligent web page crawling, intelligent field recognition, precise analysis of address information, intelligent merchant grouping and automatic page discovery. The present invention abandons the traditional rule-based extraction method, and uses contextual semantic analysis and pattern recognition technology to automatically identify information such as merchant names, telephone numbers and addresses in web pages. It realizes accurate association grouping of fields through a dynamic weight multi-dimensional similarity calculation method, and utilizes an automatic page discovery mechanism to continuously discover and collect merchant information without manually specifying the collection path. The present invention breaks through the technical bottlenecks of traditional methods such as the need to write specific rules for different websites, difficulty in coping with changes in web page structure, and inability to handle multiple merchant information groups. It adapts to changes in various web page structures and greatly improves the automation, accuracy and efficiency of merchant information collection.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet data collection and information processing, and in particular to a method and system for intelligently collecting and analyzing web merchant information. Background Art

[0002] Web data collection and information extraction are important technologies in the internet sector, widely used in search engines, business intelligence gathering, market analysis, and other scenarios. In the area of ​​merchant information collection, key information such as store names, phone numbers, and addresses needs to be extracted from various websites to support subsequent data analysis and applications.

[0003] Traditional web data scraping techniques rely primarily on rule-based extraction methods, such as regular expressions and XPath selectors. For example, a common technique involves manually analyzing web page structures and writing fixed XPath rules for specific websites to locate and extract target elements.

[0004] The most relevant existing techniques typically involve manually writing scripts for web pages. Technicians analyze the DOM structure of each target site and write specific rules to match target elements. This approach relies on the fixed structure of web pages and specific HTML tag attributes, traversing HTML elements, extracting node content, and identifying and classifying them using pre-set rules.

[0005] However, this traditional method has obvious technical flaws: first, different parsing rules need to be written for each new website, which is a lot of work; second, changes in web page structure (such as website redesign) will cause the written rules to become invalid, requiring continuous maintenance and updates; third, it is difficult to handle the situation where there are multiple merchant information on a page (such as a branch list), and different information fields of the same merchant cannot be automatically associated; finally, manual operations are required to find the page containing the target information, and the lack of automated discovery capabilities leads to low collection efficiency.

[0006] Therefore, there is an urgent need for an intelligent collection and parsing method for web merchant information that can automatically adapt to various web page structures, does not require preset rules, can intelligently identify and associate merchant information, and automatically discover target pages. Summary of the Invention

[0007] The purpose of the present invention is to solve the problems in the prior art such as the need to write specific rules for each website, difficulty in coping with changes in web page structure, inability to process multiple merchant information, and lack of automated discovery capabilities, and to provide a method and system for intelligent collection and analysis of web merchant information.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A method for intelligently collecting and parsing merchant information on a webpage, comprising:

[0010] Based on the merchant brand name, query the search engine and parse the links in the search results page to build an initial URL crawling queue, use browser automation tools to access the links in the initial URL crawling queue, and obtain the web page DOM structure data stream;

[0011] Using the webpage DOM structure data stream, combined with the semantic association between key identifiers and text paragraphs, the merchant name, telephone number and address of the webpage content are identified to generate a candidate field set;

[0012] Performing administrative division identification and address component analysis on the address field in the candidate field set, and outputting structured address data;

[0013] For the candidate field set, the path similarity, visual proximity and semantic relevance between the fields are calculated, the correlation relationship between the fields is determined through nonlinear fusion, and structured merchant information is generated;

[0014] When the webpage DOM structure data stream of the new webpage is obtained, the text content and URL feature analysis of the links in the webpage DOM structure data stream of the new webpage is performed, the target link containing the merchant information is identified, and the target link is added to the initial URL crawling queue to realize automatic information collection.

[0015] Preferably, the constructing of the initial URL crawling queue includes:

[0016] Based on the links in the search results page, domain name features, TLD type features, and URL path features are extracted to calculate a multi-dimensional score;

[0017] Based on the multi-dimensional scoring, the links are filtered and prioritized to obtain an optimized initial URL crawling queue.

[0018] Preferably, the webpage DOM structure data stream is used to identify the merchant name, phone number, and address of the webpage content in combination with the semantic association between key identifiers and text paragraphs to generate a set of candidate fields, including:

[0019] Based on the predefined keyword library, identify the keyword words in the text nodes and mark the text area containing the target information;

[0020] Based on the text area marked as containing target information, performing feature matching of the merchant name, telephone number and address to generate candidate fields;

[0021] Verification is performed on the candidate fields, and confidence scores are calculated based on the verification results, and the candidate field set is output.

[0022] Preferably, the step of performing administrative division identification and address component parsing on the address field in the candidate field set and outputting structured address data includes:

[0023] Performing text cleaning and traditional / simplified conversion on the address field in the candidate field set to generate pre-processed address text;

[0024] Based on the preset administrative division library, the provincial, municipal and district / county address information of the pre-processed address text is identified, and based on the trained deep learning model, the street and house number recognition is performed on the remaining text to output the structured address data.

[0025] Preferably, the step of calculating path similarity, visual proximity, and semantic relevance between fields for the candidate field set, determining the relationship between fields through nonlinear fusion, and generating structured merchant information includes:

[0026] Based on the candidate field set, calculating the XPath path depth difference of the candidate fields and generating a path similarity feature;

[0027] Based on the candidate field set, calculating the DOM element coordinate distance and generating a visual proximity feature;

[0028] Based on the candidate field set, calculating the relevance of different text contents and generating semantic relevance;

[0029] Based on the path similarity feature, the visual proximity feature and the semantic association, radial basis function fusion is performed to output the association relationship between fields.

[0030] Preferably, the step of identifying the merchant name, telephone number and address of the webpage content is enhanced using a random sequence model and includes:

[0031] Based on the webpage DOM structure data stream, a node sequence relationship graph is constructed to generate a random sequence model;

[0032] Based on the pre-trained feature extraction model, the multi-dimensional feature vector of the node is extracted and the generalized assignment space is constructed;

[0033] Based on the generalized value space and utilizing the inter-node sequence relationship determined by the random sequence model, semi-supervised learning training is performed to generate a probability distribution of merchant information types.

[0034] Preferably, it also includes:

[0035] Based on the node multidimensional feature vector and the probability distribution of the merchant information type, spectral clustering analysis is performed to achieve automatic grouping of multiple merchant information;

[0036] Based on the feedback information of the collection results, perform online learning optimization to update the model parameters of the random sequence model;

[0037] Based on the random order model, graph convolutional network technology is used to integrate cross-page merchant information and output a complete merchant information set.

[0038] Preferably, the step of determining the association relationship between fields by nonlinear fusion is enhanced by a binary decision diagram and includes:

[0039] Based on the candidate field set, construct a binary decision graph structure of node relationships;

[0040] Based on the multi-threshold quantization technology, the similarity feature space is discretized;

[0041] Based on Boolean logic expression optimization, binary decision graph minimization is performed to generate a similarity reasoning model.

[0042] Preferably, the binary decision graph structure of building node relationships includes:

[0043] Based on the principle of maximizing information entropy, for different types of similarity feature spaces, determining a set of segmentation points for feature quantization;

[0044] Based on the segmentation point set, a shared subgraph structure optimization strategy is adopted to perform shared optimization on the subgraph structure in the binary decision graph to reduce the graph complexity;

[0045] When the structure and content of a web page change, based on a dynamic update strategy, the subgraph structure in the binary decision graph is updated through local reconstruction technology to achieve incremental graph structure adjustment and incorporate new feature relationships.

[0046] Preferably, the step of analyzing the text content and URL features of the links in the webpage DOM structure data stream of the new webpage is enhanced by using integrated tensor representation, and includes:

[0047] Acquire link features through the webpage DOM structure data stream of the new webpage, and perform tensor decomposition operations based on the link features to extract potential pattern representations;

[0048] performing self-organizing map clustering based on the latent pattern representation to construct a local reversible map;

[0049] Based on the local reversible mapping, a compact decision graph is constructed and a link priority evaluation result is output.

[0050] A device for intelligently collecting and analyzing merchant information on a webpage, comprising:

[0051] The webpage crawling module is used to query the search engine based on the merchant brand name and parse the links in the search results page, build an initial URL crawling queue, use the browser automation tool to access the links in the initial URL crawling queue, and obtain the webpage DOM structure data stream;

[0052] A field identification module is used to use the webpage DOM structure data stream and combine the semantic association between key identifiers and text paragraphs to identify the merchant name, telephone number and address of the webpage content and generate a set of candidate fields;

[0053] An address parsing module, configured to perform administrative division identification and address component parsing on the address fields in the candidate field set, and output structured address data;

[0054] A merchant grouping module is used to calculate the path similarity, visual proximity and semantic relevance between the candidate field sets, determine the relationship between the fields through nonlinear fusion, and generate structured merchant information;

[0055] The page discovery module is used to obtain the web page DOM structure data stream of a new web page, analyze the text content and URL characteristics of the links in the web page DOM structure data stream, identify the target link containing merchant information, and add the target link to the initial URL crawling queue to realize automatic information collection.

[0056] The beneficial effects of the present invention are:

[0057] 1. No need to write specific rules for different websites: Through contextual semantic analysis and pattern recognition technology, merchant information on web pages is automatically identified and adapted to various web page structure changes. There is no need to write specific XPath rules or regular expressions for each website, which greatly reduces manual workload.

[0058] 2. Automatically respond to changes in web page structure: By analyzing text semantics and content patterns to identify merchant information, rather than relying on a fixed HTML tag structure, the system can continue to function properly even when the DOM structure changes due to website redesign, improving system stability and ongoing effectiveness.

[0059] 3. Accurately handle scenarios with multiple merchant information: Using dynamic weighted multi-dimensional similarity calculation and nonlinear fusion technology, it can automatically correctly group and associate different merchant names, addresses, phone numbers, and other information on the same page, solving the problem that traditional methods cannot handle the mixing of multiple merchant information.

[0060] 4. Automatic page discovery capability: Through link text semantic analysis and URL feature analysis, it automatically identifies target pages that may contain merchant information and adds them to the crawling queue, eliminating the need to manually specify the collection path, significantly improving collection coverage and efficiency.

[0061] 5. High-precision address parsing: Through multi-level administrative division identification and deep learning model processing, accurate parsing of various non-standard address expressions is achieved, improving the structured quality of address information. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 This is a flow chart of the method for intelligently collecting and analyzing web merchant information of the present invention;

[0064] Figure 2 A flowchart of the intelligent field recognition step of the present invention;

[0065] Figure 3 This is a flow chart of the steps for grouping smart merchants of the present invention;

[0066] Figure 4 A flowchart of the automatic page discovery steps of the present invention;

[0067] Figure 5 The flowchart of intelligent field recognition enhanced by the random sequence model of the present invention;

[0068] Figure 6 This is a flowchart of the nonlinear similarity fusion enhanced by the binary decision diagram of the present invention;

[0069] Figure 7 This is a structural diagram of the web merchant information intelligent collection and analysis system of the present invention. DETAILED DESCRIPTION

[0070] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0071] The present invention provides a method and system for intelligently collecting and parsing merchant information on web pages, which can automatically adapt to various changes in web page structures without the need to write specific rules for different websites, can accurately handle multiple merchant information scenarios, and has automatic page discovery capabilities.

[0072] Reference Figure 1The method for intelligently collecting and analyzing web merchant information of the present invention comprises the following steps:

[0073] In step S1, a search engine is queried based on the merchant's brand name and links in the search results page are parsed to construct an initial URL crawl queue. Browser automation tools are then used to access the links in the initial URL crawl queue, obtaining a webpage DOM structure data stream. This step first accesses a mainstream search engine, enters the merchant's brand name as a search criterion, and retrieves the search results page. All links are then extracted from the search results page. The links are analyzed for domain name characteristics (e.g., whether they contain the brand name), TLD type (e.g., ".com", ".cn", etc.), and URL path characteristics (e.g., whether they contain keywords such as "official" and "index") to assess whether the link is the target merchant's official website. This constructs the initial URL crawl queue. The system then uses browser automation tools such as Selenium or Puppeteer to access the links in the queue. These tools can simulate real user behavior and support complex page processing such as JavaScript rendering, cookie processing, and dynamic content loading, thus preventing interception by the target website's anti-crawler system. Finally, the obtained HTML page content is cleaned to remove interference content such as comments and irrelevant scripts, build a complete DOM tree structure, extract all visible text nodes and their XPath paths, coordinate positions and other information, form a web page DOM structure data stream, and provide basic data for subsequent field recognition.

[0074] Step S2 utilizes the webpage DOM structure data stream and combines the semantic association between key identifiers and text paragraphs to identify the merchant name, telephone number, and address of the webpage content and generate a set of candidate fields. This step first traverses all text nodes in the webpage DOM tree, identifies key identifiers such as "Phone:", "Contact information:", "Address:", etc., and combines the text content before and after the node to construct a semantic association graph of the text paragraph and mark the text area that may contain the target information. Then, in the marked text area, the system uses regular expressions to match various telephone number patterns (such as "123-4567-8901", "12345678901", "(123)45678901", etc.), taking into account variants such as international number prefixes and extension numbers, extracts candidate telephone numbers, and records their XPath paths. At the same time, based on a Chinese address feature vocabulary (including administrative divisions such as province / city / district / county / road / street and common address terms), the system identifies areas of text with high concentrations of address feature terms. Using a sliding window method, it extracts possible complete address text and records its XPath path. To identify business names, the system analyzes features such as the page title, H1-H3 tag content, text near the logo, and key highlight styles (such as large font, bold, and special colors), combined with search keyword matching, to identify possible business names and record their XPath paths. Finally, the extracted candidate fields are verified: phone numbers are verified using rules such as number format and length; addresses are verified using administrative division matching; and business names are verified using search term similarity. Based on the verification results, a confidence score is calculated for each field to form a set of candidate fields.

[0075] Step S3 identifies administrative divisions and parses address components for the address fields in the candidate field set, outputting structured address data. This step first cleans and converts the address fields in the candidate field set into simplified and traditional Chinese characters, removing extra spaces and special characters, unifying the text, detecting repeated substrings (e.g., "B City B City"), correcting common input errors, and generating preprocessed address text. The system then uses a preloaded library of provincial administrative divisions to identify provincial-level addresses, applying the longest substring first principle to avoid mismatches (e.g., misidentifying "H Province" in "H Province Road" as a province), and outputs provincial-level address information. Based on the provincial-level identification results, the system then identifies city-level and district-level administrative divisions in the remaining text, employing a long-name first strategy to resolve duplicate districts and counties (e.g., "XX District" exists in both City B and City C), ensuring logical administrative divisions. After identifying the provinces, districts, and cities, the system uses a BiLSTM+CRF deep learning model to identify unstructured address components such as streets and house numbers in the remaining text, processing various non-standard expressions and extracting key address components. Finally, by calling the map API service, the parsed address is geocoded and verified to determine whether the address exists, and the standardized address expression is obtained, possible errors are corrected, and finally high-quality structured address data is output.

[0076] Step S4 calculates path similarity, visual proximity, and semantic relevance between the candidate field sets. Inter-field relationships are determined through nonlinear fusion to generate structured merchant information. This step first analyzes the DOM structural features of all extracted fields on the page, calculating structural metrics such as XPath depth difference, element Y coordinate distribution, and text noise ratio. These metrics serve as basic parameters for dynamic weight calculation, enabling the system to adapt to different page structures. Based on these page structural features, the system then calculates weight coefficients for multidimensional features such as path similarity, visual proximity, semantic relevance, and temporal matching, achieving adaptive weight adjustment. Next, the system calculates path similarity (achieved through edit distance and tree structure comparison), visual proximity (calculated based on element coordinates), and semantic relevance (based on text content relevance analysis) between all fields on the same page, generating a similarity matrix between the fields. To overcome the limitations of linear weighting models, the system uses the radial basis function (RBF) kernel technique to nonlinearly fuse multidimensional similarity features, calculate a comprehensive similarity score, and form a more accurate field relationship. Finally, based on the comprehensive similarity score, the system applies a hierarchical clustering algorithm to automatically group and associate the name, address, telephone number and other fields of the same merchant, handle the mixed information of multiple merchants, and output a structured merchant information set.

[0077] Step S5, when the webpage DOM structure data stream of the new webpage is obtained, the text content and URL feature analysis of the links in the webpage DOM structure data stream of the new webpage is performed, the target link containing the merchant information is identified, and the target link is added to the initial URL crawling queue to realize automatic information collection. This step first extracts all the The system identifies target links likely to contain merchant information by matching keywords (e.g., "Contact Us," "Branch Information," "National Stores") and calculating semantic similarity within the link text of the tag. The system also analyzes the URL path characteristics of the link, identifying common path patterns containing merchant information, such as " / contact," " / about," " / stores," and " / branch," to assess the likelihood that the link contains the target information. Combining the link text semantic score with the URL feature score, the system calculates a comprehensive score to determine link crawling priority, prioritizing pages most likely to contain merchant information and improving crawling efficiency. Based on the information acquisition performance of crawled pages, the system dynamically adjusts the link priority calculation model, forming a self-learning page discovery mechanism that continuously optimizes crawling strategies and enhances the system's adaptability. Finally, the system adds identified high-value links to the URL crawling queue, looping back to step S1 to automate the merchant information collection process, eliminating the need for manual crawling path specification and significantly improving crawling coverage and efficiency.

[0078] As a specific embodiment of the present invention, step S1 further includes:

[0079] Step S11: Build an initial URL queue based on search engines. This involves accessing mainstream search engines, entering merchant brand names or keywords as search criteria, obtaining search result pages, parsing links in the result pages, extracting possible merchant official website URLs, and building an initial URL crawling queue.

[0080] Step S12, URL priority analysis and filtering, includes:

[0081] Based on the links in the search results page, domain name features (such as brand name), TLD type features (.com / .cn, etc.), and URL path features (such as keywords such as "official" / "index") are extracted to calculate a multi-dimensional score;

[0082] Filtering and prioritizing links based on the multi-dimensional scores to obtain an optimized initial URL crawling queue;

[0083] Step S13: Browser simulation request implementation, using browser automation tools such as Selenium / Puppeteer to simulate real user access behavior, supporting complex page processing such as JavaScript rendering, cookie processing, and dynamic content loading, to avoid being intercepted by anti-automatic data collection systems and ensure that the complete page content is obtained;

[0084] Step S14: web page content extraction and DOM structure analysis. The obtained HTML page content is cleaned to remove comments, irrelevant scripts and other interfering content, build a complete DOM tree structure, extract all visible text nodes and their XPath paths, coordinate positions and other information, form a web page DOM structure data stream, and provide basic data for subsequent field recognition.

[0085] Reference Figure 2 As a specific embodiment of the present invention, step S2 further includes:

[0086] Step S21, context semantic analysis, includes:

[0087] Based on the predefined keyword library, identify the keyword words in the text nodes (such as "Phone:", "Contact:", "Address:", etc.), and mark the text area containing the target information;

[0088] Combine the text content before and after the node to build a semantic association graph of the text paragraph and mark the text area that may contain the target information;

[0089] Step S22, telephone number pattern recognition, in the marked text area, using regular expressions to match telephone number patterns (such as "123-4567-8901", "12345678901", "(123)45678901", and other formats), taking into account variants such as international number prefixes and extension numbers, extracting candidate telephone numbers and recording their XPath paths;

[0090] Step S23: Address feature recognition. Based on the Chinese address feature word library (including administrative divisions such as province / city / district / county / road / street and common address words), the address feature word-dense areas are identified in the text. The possible complete address text is extracted using a sliding window method, and its XPath path is recorded.

[0091] Step S24, merchant name recognition, by analyzing features such as the page title, H1-H3 tag content, text near the logo, and key highlight styles (such as large font, bold, special color), combined with the search keyword matching degree, to identify possible merchant names and record their XPath paths;

[0092] Step S25, candidate field verification and confidence assessment, includes:

[0093] Based on the text area marked as containing target information, performing feature matching of the merchant name, telephone number and address to generate candidate fields;

[0094] Perform validation on the candidate fields and calculate a confidence score based on the validation results, and output the set of candidate fields. Among them, the phone number is verified by rules such as digital format and length; the address is verified by administrative division matching; the merchant name is verified by search term similarity.

[0095] As a specific embodiment of the present invention, step S3 further includes:

[0096] Step S31, address text preprocessing, including:

[0097] Perform text cleaning and traditional-simplified conversion on the address fields in the set of candidate fields, remove extra spaces, special characters, perform unified traditional-simplified conversion, detect duplicate substrings (such as "B City B City"), and correct common input errors to generate preprocessed address text;

[0098] Specifically, this step performs comprehensive text cleaning and standardization processing on the address fields in the set of candidate fields. The system first removes white space characters such as extra spaces, tab characters, and line break characters in the address text, and at the same time clears special characters such as parentheses and quotation marks that may interfere with parsing. For Chinese addresses, the system performs unified traditional-simplified conversion to ensure that all traditional Chinese characters are converted to simplified Chinese characters for matching with the administrative division library. In addition, the system will also identify and process common error patterns in the text, such as detecting duplicate substrings such as "B City B City" and "C City C City", and automatically correcting such duplicate errors caused by data input or OCR recognition. The system will also identify and standardize common address abbreviations and aliases, such as converting "North Third Ring Road" to "North Third Ring Road". Through this series of preprocessing operations, the system generates standardized address text, laying a foundation for subsequent accurate parsing and significantly improving the parsing success rate.

[0099] Step S32, provincial administrative division recognition, use the pre-loaded provincial administrative division library, implement provincial address recognition through the provincial address recognition function, adopt the principle of the longest substring first to avoid incorrect matching (such as misidentifying "H Province Road" as a province), and output provincial address information;

[0100] Specifically, provincial administrative division identification. This step uses a pre-loaded comprehensive provincial administrative division library and a specially designed provincial address identification function to achieve accurate provincial address identification. The system adopts the longest substring first matching principle and tries to match the longest possible provincial administrative division name from the beginning of the address text. This method can effectively avoid the problem of incorrect matching, such as preventing the "H Province" in "No. 10, Jianshe Street, H Province Road" from being mistakenly identified as an independent provincial identifier, and correctly understanding it as part of the road name. At the same time, it can also handle a variety of expressions of provincial administrative divisions, such as "XX", "XX Province", "abbreviation", etc. can all be correctly identified as the same province. For successfully matched provincial information, the system separates it from the original text and outputs standardized provincial address information to provide accurate context constraints for subsequent city-level identification and improve the accuracy of the overall parsing.

[0101] Step S33: Identify city-level administrative divisions. Based on the provincial-level identification results, use the district-level address recognition function to identify city-level and district-level administrative divisions in the remaining text. A long name priority strategy is adopted to resolve issues with districts and counties with the same name (e.g., "XX District" exists in both City B and City C), ensuring logical and reasonable administrative divisions.

[0102] Specifically, the system identifies city-level administrative divisions. Based on the successful provincial-level identification results from the previous step, the system uses the district-level address recognition function to continue identifying city-level and district-level administrative divisions in the remaining text. The system first searches for matches in the city-level administrative division database for the specific province to determine the city-level information. Then, based on the identified province-level information, it matches the corresponding district-level administrative division database to determine the district-level information. The entire process utilizes a long name priority strategy, prioritizing complete administrative division names, which helps resolve duplicate district and county names. For example, if "XX District" is encountered and the identified provincial-level information is "City B," it will be correctly associated with XX District of City B, rather than XX District of City C. It also handles special cases such as abbreviations and aliases of administrative divisions, such as "City D" (a straight-tube city with no divisions). By maintaining the hierarchical constraints of province, city, and district / county, the identified administrative divisions remain logically consistent, significantly improving the accuracy of address resolution.

[0103] Step S34: Street and detailed address parsing. Using the trained deep learning model, the remaining text is used to identify streets and house numbers. After identifying the province, city, and district, the remaining text is then analyzed using a BiLSTM+CRF deep learning model to identify unstructured address components such as streets and house numbers, processing various non-standard expressions and extracting key address components.

[0104] Specifically, after successfully identifying high-level administrative divisions such as provinces, cities, districts, and counties, the next step is to identify unstructured address components such as streets, road names, house numbers, and neighborhood names contained in the remaining text. This step is handled by a BiLSTM+CRF deep learning model, trained with extensive labeled data and capable of recognizing various non-standard expressions. The model treats the remaining portion of the address as a sequence labeling problem, assigning specific labels to each character (e.g., B-Street and I-Street indicate the beginning and interior of a street, B-Number and I-Number indicate the beginning and interior of a house number, etc.). The system also integrates a dictionary of road names and a dictionary of common neighborhood names to enhance the model's recognition capabilities. For house numbers with unusual formats (e.g., "Building 2, Unit 3, Room 501"), the system uses specially designed rules for identification and structuring. This combined approach enables the system to accurately extract and classify key address components, handle various non-standard expressions, and generate structured, detailed address information.

[0105] The BiLSTM+CRF deep learning model used in this system is specifically optimized for Chinese address parsing. The model employs a three-layer architecture: an embedding layer, a bidirectional LSTM layer, and a conditional random field layer. In the embedding layer, each Chinese character is represented by a 300-dimensional character vector. These vectors are pre-trained on a corpus of 10 million Chinese addresses and contain rich address semantic information. The bidirectional LSTM layer consists of three stacked layers, each containing 128 hidden units. Tanh is used as the activation function, and dropout (with a ratio of 0.3) is incorporated into the LSTM units to prevent overfitting. A conditional random field layer is added on top of the bidirectional LSTM to model transition probabilities between labels, ensuring that predictions meet address structure constraints. The model training data is derived from a collection of one million manually annotated Chinese address samples, covering address information in various formats and expressions from across China. Each address is annotated at the character level with its corresponding component type (such as province, city, district, street, and house number). During training, a batch size of 64 and an initial learning rate of 0.001 were used. The Adam optimizer was used, and a learning rate decay strategy was implemented. The learning rate was halved when validation set performance stopped improving for three consecutive epochs. To prevent overfitting, in addition to dropout, an early stopping mechanism was introduced. Training was terminated when validation set performance stopped improving for five consecutive epochs. The model achieved an F1 score of 95.7% on an independent test set, demonstrating its excellent performance on the complex and diverse Chinese address parsing task.

[0106] Step S35, address validity verification, by calling the map API service, geocoding verification of the parsed address, determining whether the address exists, and obtaining standardized address expression, correcting possible errors, and finally outputting high-quality structured address data.

[0107] Specifically, to ensure the actual accuracy and usability of the parsing results, validity verification is performed after the address parsing is completed. This step verifies the geocoding of the parsed address by calling the map API service. The system passes the structured address information as a parameter to the map API to obtain the returned geocoding results, including latitude and longitude coordinates, standardized address expression, confidence level and other information. If the map API can successfully identify the address and return a high-confidence result, it means that the parsed address is valid; if it returns a low confidence level or cannot be identified, the system will try to adjust the combination of the address components and resubmit for verification. Through the standardized address expression returned by the map API, the system can also correct and improve the original parsing results, such as supplementing missing administrative division information, unifying the expression of street names, etc. Ultimately, the system outputs verified and optimized high-quality structured address data, providing a reliable foundation for subsequent merchant information processing and application.

[0108] Reference Figure 3 As a specific embodiment of the present invention, step S4 further includes:

[0109] Step S41: Page structure feature analysis, analyzing the DOM structure features of all extracted fields in the page, calculating structural indicators such as XPath depth difference, element Y coordinate distribution, and text noise ratio, as basic parameters for dynamic weight calculation, and adapting to different page structures;

[0110] Specifically, this step comprehensively analyzes the overall structure of the webpage, providing a foundation for subsequent dynamic weight calculation and field association. The system first extracts the DOM structural features of all candidate fields on the page, including the XPath path length and depth of each field's node, its hierarchical position in the DOM tree, and parent-child relationships. By comparing the common prefix lengths and branch points of the XPath paths of different fields, the system calculates the XPath depth difference index to assess the relative positional relationships of the fields within the DOM structure. Simultaneously, the system analyzes the visual layout of the page, calculating metrics such as element Y-coordinate distribution, horizontal alignment, and element spacing to identify sets of fields that may belong to the same information block. Furthermore, the system assesses the page's text noise ratio (such as the proportion of non-core information such as advertising content and navigation menus) to determine the page's information density and structural complexity. These structural metrics collectively constitute the page's structural feature vector, which serves as the basic parameter for dynamic weight calculation. This allows the system to adapt to the structural characteristics of different types of webpages and improve the accuracy of field association.

[0111] Step S42: Dynamic weight calculation, based on the page structure features, calculates the weight coefficients of multi-dimensional features such as path similarity, visual proximity, semantic relevance, and temporal matching to achieve adaptive adjustment of the weights;

[0112] Specifically, based on the page structural features analyzed in the previous step, a dynamic weight calculation is performed to adaptively determine the importance of different similarity features in the comprehensive evaluation. The system first evaluates the structural regularity of the page. For pages with regular structures (such as standard table layouts), path similarity should be weighted higher; for pages with loose structures (such as free-text paragraphs), semantic relevance should be weighted higher. The system then calls the calc_dynamic_weights() function to dynamically calculate weight coefficients for multi-dimensional features such as path similarity (α), visual proximity (β), semantic relevance (γ), and temporal matching (ε) based on metrics such as XPath depth difference and element coordinate distribution dispersion. For example, on pages with multi-column table layouts, visual proximity is automatically weighted higher; on pages with dense information and complex structures, path similarity is weighted accordingly. Through this dynamic weighting mechanism, the system can automatically adjust the similarity calculation strategy based on the characteristics of different web pages, significantly improving the adaptability and accuracy of merchant information grouping, and effectively handling a variety of page structures, from simple single-column layouts to complex multi-merchant mixed layouts.

[0113] Step S43, multi-dimensional similarity calculation, includes:

[0114] Based on the candidate field set, calculating the XPath path depth difference of the candidate fields and generating a path similarity feature;

[0115] Based on the candidate field set, calculating the DOM element coordinate distance and generating a visual proximity feature;

[0116] Based on the candidate field set, calculating the relevance of different text contents and generating semantic relevance;

[0117] For all fields on the same page, calculate the path similarity (through edit distance and tree structure comparison), visual proximity (based on element coordinates), and semantic relevance (based on text content relevance) between different fields to generate a similarity matrix between fields;

[0118] Specifically, after determining the dynamic weights, a multi-dimensional similarity calculation is performed on all candidate fields extracted from the page to quantify the degree of association between different fields. For path similarity calculation, the path_similarity() function performs XPath path edit distance and tree structure comparison analysis. Two XPath expressions are converted into a tree structure and their structural similarity is calculated. Fields under the same parent node receive a higher path similarity score. For visual proximity calculation, the Euclidean distance and relative position relationship between elements are calculated based on the rendering coordinates of DOM elements. Elements that are closely positioned and visually aligned receive a higher visual proximity score. For semantic relevance calculation, the semantic relevance of the field text content is analyzed. A word vector model or a pre-trained language model is used to calculate semantic similarity between texts. Content-related fields (such as keywords in a merchant name and related information in the address) receive a higher semantic relevance score. Through similarity calculations across these three dimensions, the system generates a comprehensive inter-field similarity matrix, laying the foundation for subsequent nonlinear fusion and merchant grouping.

[0119] Step S44, nonlinear similarity fusion, uses the radial basis function (RBF) kernel technique to nonlinearly fuse the multidimensional similarity features, calculate the comprehensive similarity score, perform radial basis function fusion based on the path similarity feature, the visual proximity feature and the semantic association, output the correlation relationship between the fields, form a more accurate field correlation relationship, and overcome the limitations of the linear weighted model;

[0120] Specifically, to overcome the limitations of the simple linear weighted model, the radial basis function (RBF) kernel technique is used for nonlinear similarity fusion, which can more accurately capture the complex association patterns between fields. The multidimensional features such as path similarity, visual proximity and semantic association calculated in the previous step are mapped to a high-dimensional feature space, and the nonlinear similarity between feature vectors is calculated using the RBF kernel function. This method can effectively handle the nonlinear interaction between features. For example, when the path similarity and visual proximity of two fields are high at the same time, their comprehensive similarity is not just a simple weighting of the two components, but presents a nonlinear enhancement effect. The system adjusts the parameter γ in the RBF kernel function according to the dynamic weight calculated in step S42 to control the intensity of the nonlinear mapping. The fused comprehensive similarity score can more accurately reflect the true association relationship between fields, effectively deal with field association judgment in various complex web page structures, and greatly improve the accuracy of merchant information association, especially when processing complex pages with mixed merchant information.

[0121] Step S45: Merchant information clustering and grouping. Based on the comprehensive similarity score, a hierarchical clustering algorithm is applied to automatically group and associate the name, address, telephone number and other fields of the same merchant, handle the mixed situation of multiple merchant information, and output a structured merchant information set.

[0122] Specifically, a hierarchical clustering algorithm (such as UPGMA or Ward's method) is used to convert the comprehensive similarity matrix into a distance matrix. Starting from the most similar field pairs, the fields are gradually merged to form a cluster tree. By analyzing the structure and distance distribution of the cluster tree, the optimal split point is determined, and the fields are divided into different merchant groups. For each group, the system further identifies the key field types such as name, address, and telephone number to build a complete merchant information structure. It can also handle fields that may be missing within the group, such as by supplementing information across pages or filling in based on inference of known fields. This step can effectively handle mixed situations where multiple merchant information is contained on a page, such as complex scenarios such as branch lists and franchisee information, automatically grouping and associating different information fields of the same merchant correctly, and outputting a structured merchant information set. This automatic grouping capability is a significant advantage of the present invention over traditional methods, greatly improving the accuracy and completeness of information collection.

[0123] The implementation of the spectral clustering algorithm in this system involves several sophisticated steps. First, the system constructs an affinity matrix W based on the comprehensive similarity scores calculated in the previous stage. Matrix elements Wij represent the similarity between fields i and j, with higher similarity values ​​indicating a greater likelihood that the two fields belong to the same merchant. The system uses a Gaussian similarity function to ensure the matrix's non-negativity and symmetry. Next, the system calculates the degree matrix D, a diagonal matrix whose diagonal element Dii is the sum of all elements in the i-th row of the affinity matrix W, representing the total connection strength between node i and all other nodes. The system then calculates the normalized Laplacian matrix L. This normalization process fixes the eigenvalue distribution to the interval [0, 2], facilitating subsequent analysis. The system then performs eigenvalue decomposition on the normalized Laplacian matrix L, calculating all eigenvalues ​​and corresponding eigenvectors. By analyzing the eigenvalue distribution, specifically identifying significant transition points in the eigenvalue sequence, the system automatically determines the optimal number of clusters k. This eigenvalue-based approach to determining the number of clusters is known as the "feature gap method" and can adapt to changes in the number of merchants on different pages. After determining k, the system selects the eigenvectors corresponding to the k smallest non-zero eigenvalues ​​to form an n×k-dimensional feature matrix (n is the number of fields). Finally, the system applies the K-means++ algorithm to cluster the row vectors of this feature matrix, with each cluster representing all relevant fields for a merchant. To improve clustering stability, the system performs multiple randomly initialized K-means runs, selecting the result with the highest silhouette coefficient as the final cluster. This spectral clustering method is more capable of handling non-convex data distributions than traditional hierarchical clustering, making it particularly suitable for grouping merchant information on complex web pages.

[0124] Reference Figure 4 As a specific embodiment of the present invention, step S5 further includes:

[0125] Step S51: semantic analysis of link text to extract all The link text of the tag is identified through keyword matching (such as "Contact Us," "Branch Information," and "National Stores") and semantic similarity calculation to identify target links that may contain merchant information;

[0126] Step S52: Link URL feature analysis: analyzing the link URL path features, identifying common path patterns containing merchant information, such as " / contact", " / about", " / stores", and " / branch", and evaluating the likelihood that the link contains the target information;

[0127] Step S53: Prioritize links, combine the link text semantic score and URL feature score, calculate a comprehensive score, determine the crawling priority of links, and give priority to crawling pages that are most likely to contain merchant information to improve collection efficiency;

[0128] Step S54: Dynamically construct a crawling strategy, dynamically adjust the link priority calculation model based on the information acquisition effect of the crawled pages, form a self-learning page discovery mechanism, continuously optimize the crawling strategy, and improve the system's adaptive ability;

[0129] In step S55, the crawling queue is updated and the collection is cyclical. The identified high-value links are added to the URL crawling queue, forming a loop with step S1 to realize an automated merchant information collection process without the need for manual specification of the collection path, thereby greatly improving the collection coverage and efficiency.

[0130] Reference Figure 5 As a specific embodiment of the present invention, step S2 may also be enhanced by using a random order model, including:

[0131] Step S201, establishing a random order model, includes:

[0132] Based on the webpage DOM structure data stream, a node sequence relationship graph is constructed to generate a random sequence model;

[0133] Perform depth-first traversal of the DOM tree to establish a sequential relationship graph between nodes. Use Markov random field theory to construct a node transition probability matrix to form a random sequence model that can describe the content structure of the web page.

[0134] Specifically, this step solves the problem that traditional methods have difficulty in accurately identifying relevant merchant information in unstructured web pages by modeling the hierarchical and sequential relationships of web page content. First, the input web page DOM structure data stream is deeply traversed and analyzed to establish a sequential relationship graph between nodes to capture the structural and sequential characteristics of the web page content. In the specific implementation, the web page is regarded as an ordered but random data stream, and a node transition probability matrix is ​​constructed using Markov random field theory. For each DOM node, its relationship with the adjacent nodes is analyzed, and the conditional probability P (node ​​j | node i) is calculated to represent the possibility of moving from node i to node j. This random sequence model can describe the dynamic flow characteristics of web page content and can better capture the potential correlation between different information blocks than the static DOM tree structure. It is particularly suitable for processing web pages with complex layouts and lack of clear structural identification. The random sequence model constructed in this way provides a probabilistic structural understanding framework for subsequent merchant information identification.

[0135] Step S202, generalized valued feature extraction, includes:

[0136] Based on the pre-trained feature extraction model, the multi-dimensional feature vector of the node is extracted and the generalized assignment space is constructed;

[0137] By combining Conditional Random Fields (CRF) with deep learning technology, we extract multi-dimensional feature vectors from web page nodes, including text semantic features (obtained through pre-trained models such as BERT), structural features (DOM path features of nodes), visual features (rendering position and style of nodes), and relational features (connectivity between nodes).

[0138] Specifically, this step expands and enhances the traditional feature extraction method. It not only focuses on the direct features of the node (such as text content, label type), but also considers the position features, transfer features and global context features of the node in the random sequence model. In the specific implementation, multi-dimensional feature vectors are extracted from web page nodes through conditional random fields (CRF) combined with deep learning technology. For text semantic features, pre-trained language models such as BERT are used to map node text into high-dimensional semantic vectors to capture the deep semantic information of the text; for structural features, the DOM path features of the node are extracted, including label sequence, attribute set, sibling node features, etc.; for visual features, the rendering position and style information of the node are analyzed, including coordinate position, font size, color, visibility, etc.; for relational features, the system extracts the connection relationship between nodes, such as hyperlinks, reference relationships, etc. These multi-dimensional features together constitute a generalized assignment space, which provides rich feature representation for subsequent merchant information identification and greatly improves the system's ability to identify merchant information in various non-standard formats.

[0139] In actual deployment, the system uses the Chinese BERT-Base model (12 layers, 768 hidden units, 12 attention heads, and 110 million parameters) as the foundation for text semantic feature extraction. This model was pre-trained on a Chinese corpus (including encyclopedias, news, and online text). To adapt to the merchant information collection scenario, the system fine-tuned the BERT model using 50,000 manually annotated merchant information samples (including various merchant names, addresses, phone numbers, and their context). A small learning rate (2e-5) was used during fine-tuning to preserve the pre-trained general semantic knowledge while adapting to the specific semantic expressions of merchant information. During feature extraction, the system uses a weighted average of the outputs of the last four BERT layers as the semantic representation of the text, rather than just the last layer. This allows it to capture semantic information at different levels. Specifically, each text node is first segmented into segments of no more than 512 tokens. Feature representations are then obtained using the BERT model. For texts longer than 512 tokens, a sliding window approach is used to process the resulting data and concatenate the resulting data. Ultimately, the system concatenates the resulting 768-dimensional BERT feature vector with traditional features (such as positional features and HTML tag features) to form a more comprehensive node representation. This approach, combined with pre-trained language models, significantly improves the system's ability to understand various non-standard expressions of merchant information, particularly when dealing with implicit references, informal expressions, and industry-specific terms.

[0140] Step S203, generalized assignment weight learning, uses a semi-supervised learning method to train a generalized assignment weight model using a small amount of labeled data, uses variational inference technology to estimate the initial model parameters, and then uses the expectation maximization (EM) algorithm to iteratively optimize the parameters, ultimately learning a generalized assignment weight matrix suitable for different web page structures;

[0141] Specifically, based on a randomized sequential model, a semi-supervised learning method is used to automatically learn the importance weights of different features in the recognition process. This step first uses a small amount of manually annotated merchant information samples as seed data, and then utilizes variational inference techniques to estimate the initial model parameters. In its implementation, the system models the merchant information recognition problem as a conditional probability field and iteratively optimizes the parameters using the variational EM algorithm (a variational form of the expectation-maximization algorithm). In the E-step, the system uses the current parameters to estimate the label distribution of the unlabeled data; in the M-step, the model parameters are updated based on all data (both annotated and unlabeled). Through multiple rounds of iteration, a generalized assignment weight matrix is ​​ultimately learned that is applicable to different web page structures. This matrix can adaptively assign weights to different features, enabling the system to accurately identify merchant information in a variety of web page environments. This semi-supervised learning method significantly reduces the reliance on annotated data and overcomes the limitations of traditional fixed-weight schemes when dealing with diverse web pages.

[0142] Step S204, merchant information probability inference, includes:

[0143] Based on the generalized value space, using the inter-node sequence relationship determined by the random sequence model, semi-supervised learning training is performed to generate a probability distribution of merchant information types;

[0144] By calculating the posterior probability distribution of each node belonging to various merchant information (name, address, phone number, etc.), the system can make more accurate inferences under high uncertainty conditions and implement approximate inference using the Gibbs sampling method.

[0145] Specifically, based on a generalized assignment model, probabilistic inference of merchant information types is performed for web page nodes. This step treats merchant information recognition as a sequence labeling problem, calculating the posterior probability distribution for each node belonging to each type of merchant information (name, address, phone number, etc.). In its implementation, approximate inference is performed using Gibbs sampling, a Markov Chain Monte Carlo (MCMC) technique suitable for high-dimensional and complex probabilistic models. All unknown labels are first randomly initialized. Each label is then iteratively sampled and updated from the conditional distribution P(label i | all other labels, observation data). After sufficient iterations, the sampling results converge to the true posterior distribution. This probabilistic inference method enables more accurate judgments under high uncertainty, effectively handling irregular layouts and complex nested structures on web pages, and significantly improving merchant information recognition accuracy. It is particularly effective when dealing with merchant information with blurred boundaries, partially visible content, or non-standard formats.

[0146] Step S205: Multi-entity association clustering. Spectral clustering analysis is performed based on the node multidimensional feature vectors and the probability distribution of the merchant information types to automatically group the merchant information. First, an affinity matrix between information entities is constructed (based on the similarity of generalized valued features). Then, the optimal number of clusters is determined through Laplace matrix eigenvalue decomposition. Finally, the K-means++ algorithm is applied to complete the clustering in the reduced dimensionality space.

[0147] Specifically, for web pages containing information about multiple merchants, the system automatically groups information about multiple merchants based on information entities in a generalized valued space. This step first constructs an affinity matrix between information entities. The matrix elements represent the similarity between the entities, using Euclidean distance or cosine similarity calculated based on generalized valued features. The system then applies a spectral clustering algorithm to this affinity matrix: First, a Laplacian matrix L = DW is constructed (where W is the affinity matrix and D is a diagonal matrix whose elements are the row sums of W). Next, the eigenvalues ​​and eigenvectors of L are calculated, and the eigenvalue distribution is analyzed to determine the optimal number of clusters k. Eigenvectors corresponding to the smallest k non-zero eigenvalues ​​are selected to form a reduced-dimensional representation. Finally, the K-means++ algorithm is applied to the reduced-dimensional space to perform clustering. This spectral-based clustering method effectively captures the nonlinear structure of the data and excels in handling complex data distributions. This enables the system to accurately separate information about different merchants while correctly grouping various types of information (name, address, phone number, etc.) belonging to the same merchant, addressing the multi-entity confusion problem encountered in traditional methods.

[0148] Step S206, adaptive model update, based on the feedback information of the collection results, performs online learning optimization and updates the model parameters of the random sequence model; by continuously collecting feedback information of the collection results, the system uses an online learning algorithm to continuously optimize the random sequence model and generalized assignment weights, so that the system can automatically adapt to changes in website structure and emerging information expression methods;

[0149] Specifically, an adaptive model update mechanism was designed to address the dynamic changes in web page structure. This step continuously optimizes the random sequence model and generalized assignment weights by continuously collecting feedback on the collection results. In its implementation, an online learning algorithm is used to update the model parameters each time a new web page is processed and feedback (such as manual confirmation or cross-validation) is obtained. To balance the impact of new and old data, the system uses an incremental learning method with weight decay, where newer samples receive higher learning weights. At the same time, the system maintains a sliding window that retains only the most recent N samples for online learning to prevent the model from being affected by outdated data. Furthermore, the system implements a model performance monitoring mechanism that automatically triggers the model retraining process when performance degradation exceeds a threshold. Through this adaptive update mechanism, the system can continuously learn emerging web page patterns and information expressions, continuously improving recognition accuracy as usage increases, and maintaining adaptability to changes in website structure.

[0150] Step S207: cross-page information integration. Based on the random order model, graph convolutional network technology is used to integrate cross-page merchant information and output a complete merchant information set. The page set of the website is regarded as a hypergraph structure. Through link analysis and content similarity evaluation, an association network between pages is established, and then graph convolutional network (GCN) technology is used to propagate and fuse information on the hypergraph.

[0151] Specifically, for situations where merchant information is scattered across multiple pages, a cross-page version of the random order model is used to effectively integrate this information. This step treats the website's page collection as a hypergraph structure, where nodes represent pages and edges represent links between pages. Through link analysis and content similarity assessment, a network of inter-page associations is established. Graph Convolutional Network (GCN) technology is used to propagate and fuse information on this hypergraph, enabling cross-page knowledge transfer.

[0152] Reference Figure 6 As a specific embodiment of the present invention, step S44 may also be enhanced using a binary decision diagram, including:

[0153] Step S441, constructing a binary decision diagram, includes:

[0154] Based on the candidate field set, construct a binary decision graph structure of node relationships;

[0155] Each node in the webpage DOM tree is abstracted into a set of Boolean variables, each variable indicating whether the node meets a specific condition (such as containing a phone number pattern, address characteristics, etc.). Then, a normalized binary decision graph is constructed to simplify the complex node relationship logic into an efficient graph structure representation.

[0156] Specifically, this step applies the Binary Decision Diagram (BDD), an efficient Boolean function representation method, to the merchant information collection scenario. First, the candidate field set is analyzed, and the relationships between the fields are abstracted into a series of Boolean variables and conditional judgments. In implementation, the system abstracts each node in the webpage's DOM tree into a set of Boolean variables, each variable indicating whether the node meets a specific condition (such as whether it contains a phone number pattern or address features). For any two candidate fields, the system defines a set of Boolean variables to represent various possible relationships between them, such as "whether the XPath path similarity exceeds a threshold T1," "whether the visual distance is less than a threshold D1," and "whether they contain relevant semantic content." The system then constructs a normalized binary decision diagram, simplifying these complex relationship judgment logic into an efficient graph structure. A BDD is a directed acyclic graph, where each non-terminal node represents a Boolean variable, with two outgoing edges corresponding to the variable taking values ​​of 0 and 1, respectively. The terminal node represents the function result (whether the fields are related). This structured representation enables the system to efficiently process the complex logical relationships between a large number of nodes on a webpage, providing a solid foundation for subsequent similarity calculations.

[0157] Step S442, multi-threshold feature quantization, includes:

[0158] Based on the principle of maximizing information entropy, for different types of similarity feature spaces, determining a set of segmentation points for feature quantization;

[0159] Based on the multi-threshold quantization technology, the similarity feature space is discretized;

[0160] The continuous similarity feature space is divided into multiple discrete intervals. The system designs adaptive threshold sets for different types of features (such as path similarity, visual distance, semantic relevance, etc.), determines the optimal segmentation point by maximizing entropy, and then converts the continuous features into multi-level discrete variables.

[0161] Specifically, traditional similarity calculations often use a single threshold for feature quantization, which can easily lead to information loss due to improper threshold selection. This step introduces a multi-threshold feature quantization technique, which partitions the continuous similarity feature space into multiple discrete intervals, preserving more fine-grained difference information. Based on the principle of maximizing information entropy, the system automatically determines the optimal set of feature quantization split points for different types of similarity feature spaces (such as path similarity, visual distance, and semantic relevance). In implementation, the system first analyzes the distribution of each feature in the historical data and then uses a dynamic programming algorithm to find the split point combination that maximizes information gain. For example, for the path similarity feature, the system might determine the split point set [0.3, 0.5, 0.7, 0.9] to divide the continuous value into five intervals; for the visual distance feature, it might determine the split points [50px, 100px, 200px, 500px]. This multi-threshold quantization method retains the fine-grained difference information of features, enabling subsequent BDD to capture more complex conditional combination patterns, significantly improving the accuracy and robustness of similarity judgment, especially when dealing with boundary cases and outliers.

[0162] Step S443: Boolean logic expression generation: Based on domain knowledge and historical data, a series of Boolean logic expressions for determining node relationships are constructed. These expressions contain multiple condition combinations and are standardized using Disjunctive Normal Form (DNF) or Conjunctive Normal Form (CNF).

[0163] Specifically, the problem of determining merchant information similarity is transformed into a Boolean logic problem, and precise judgment rules are constructed. Based on domain knowledge and historical data analysis, a series of Boolean logic expressions are generated to determine node relationships. These expressions use complex multi-condition combinations to describe various scenarios of field associations. For example, to determine whether two fields belong to the same merchant, the system might generate the expression "(XPath similarity > 0.7 AND visual distance < 100px) OR (XPath similarity > 0.5 AND visual distance < 50px AND contains the same merchant name keyword)." These expressions are automatically generated and optimized through symbolic reasoning, including rule induction, condition generalization, and redundancy elimination. All generated expressions are canonized using Disjunctive Normal Form (DNF) or Conjunctive Normal Form (CNF) to facilitate subsequent BDD construction. This Boolean logic-based representation enables the system to accurately model complex conditions for field associations, especially for handling complex rules such as "If feature A meets condition X and feature B meets condition Y, then the similarity is high," significantly improving the system's flexibility and accuracy in handling various web page structures.

[0164] Step S444, BDD simplification and optimization, includes:

[0165] Based on the segmentation point set, a shared subgraph structure optimization strategy is adopted to perform shared optimization on the subgraph structure in the binary decision graph to reduce the graph complexity;

[0166] Based on Boolean logic expression optimization, binary decision graph minimization is performed to generate a similarity reasoning model;

[0167] Apply BDD simplification algorithms (such as variable reordering and node merging) to convert complex Boolean expressions into minimized BDDs. Use dynamic variable sorting technology to automatically adjust the order of variables in the BDD based on their influence, and reduce redundant nodes by sharing subgraph structures.

[0168] Specifically, original Boolean expressions often contain redundant judgments and repeated calculations, which hinder system efficiency. This step applies a series of BDD optimization techniques to transform complex Boolean expressions into streamlined, computationally efficient, and minimized BDDs. First, based on the previously determined set of partition points, a shared subgraph structure optimization strategy is employed to identify common substructures within the Boolean expression and merge them into shared nodes, significantly reducing the number of nodes in the BDD. Next, a dynamic variable reordering technique is applied to determine the optimal variable order through heuristic algorithms (such as swapping adjacent variables and evaluating the change in BDD size), further reducing BDD complexity. Furthermore, Boolean algebraic rules such as idempotence and absorption laws are employed to simplify the structure of Boolean expressions, and path compression techniques are used to merge paths with the same decision outcome. These optimization techniques work together to simplify a BDD that may originally contain thousands of nodes into a streamlined structure of tens or hundreds of nodes, while fully preserving the original logical functionality. The optimized BDD is not only computationally efficient but also clearly captures the dependencies between variables, providing an efficient and reliable decision-making basis for similarity fusion.

[0169] The optimization of binary decision diagrams is a key link in improving system performance, which involves the coordinated application of multiple strategies. The first is the variable reordering technology. The system uses a heuristic algorithm to evaluate the size change of BDD under different variable orders. In the specific implementation, the system uses the method of exchanging adjacent variables and calculating the change in the number of BDD nodes to explore the optimal order. It also introduces a genetic algorithm to support a wider range of searches. For large BDDs, the system uses a window sliding method, which only reorders a fixed-size variable window at a time, and then moves the window position to gradually optimize the entire BDD structure. The second is the node merging technology. The system identifies functionally equivalent subgraph structures and merges them. During the merging process, a hash table is used to store the functional characteristics of the nodes, which reduces the search complexity of equivalent nodes to O(1). In addition, the system implements path compression technology. When two subnodes of a node point to the same target, the node can be skipped directly, reducing redundant judgments in the BDD. When processing large-scale BDDs, the system adopts a hierarchical processing strategy, decomposing the entire BDD into multiple smaller sub-BDDs for independent optimization and then recombining. For the identification of shared subgraph structures, the system uses an incremental hashing algorithm to maintain a global unique table to ensure that subgraphs with the same function are only stored once. This approach not only saves storage space but also accelerates the BDD construction and evaluation process. The system also introduces a variable grouping strategy, placing highly correlated variables adjacent to each other within the BDD, further improving the compactness of the graph structure. Through the combined application of these optimization techniques, the system can simplify BDDs that might originally contain thousands of nodes into a streamlined structure of tens or hundreds of nodes, significantly improving computational efficiency while maintaining logical functionality.

[0170] Step S445: BDD-based similarity reasoning first maps various similarity features to BDD input variables. Then, using the BDD path traversal algorithm, all possible paths from the root node to the terminal node and their probabilities are calculated. For a given feature input combination, the system can quickly locate the corresponding BDD path and obtain an accurate similarity assessment result.

[0171] Specifically, traditional linear or simple nonlinear fusion methods struggle to handle complex interactions between features. This step, based on an optimized BDD, achieves highly reliable similarity inference, accurately simulating the expert decision-making process. First, various similarity features (quantized using multiple thresholds) are mapped to BDD input variables. Then, using the BDD's path traversal algorithm, all possible paths from the root node to terminal node 1 (indicating correlation) and their probabilities are calculated. For a given feature input combination, the corresponding BDD path is quickly located, resulting in an accurate similarity assessment. Compared to traditional methods, this BDD-based reasoning offers the advantages of computational efficiency (time complexity is proportional to the BDD size, not exponentially with the number of variables) and decision consistency (the same input always produces the same output). More importantly, BDDs can handle complex logical relationships and conditional dependencies between features, such as complex rules such as "Feature C is meaningful only if both Feature A and Feature B meet a specific condition," significantly improving the accuracy and interpretability of similarity inference.

[0172] Step S446, uncertainty assessment and decision-making, introduces a BDD-based uncertainty assessment mechanism, quantifies the uncertainty of the decision by calculating the entropy of possible outcomes, analyzes the distribution of critical paths in the BDD, adds a confidence index to each similarity assessment result, and dynamically adjusts the decision threshold based on the confidence index;

[0173] Specifically, this step introduces a BDD-based uncertainty assessment mechanism to quantify the confidence of similarity judgment results. By analyzing the distribution of key paths in the BDD and calculating the entropy of possible outcomes, the uncertainty of the decision is quantified. In its implementation, the system assigns lower confidence to inputs close to the decision boundary (where a small change in the values ​​of multiple key variables could lead to a reversal of the outcome), and higher confidence to inputs located in the center of the decision region (where changes in the values ​​of multiple key variables maintain the same outcome). Based on this confidence metric, the decision threshold is dynamically adjusted: when the judgment certainty is high, the standard threshold is used; when the judgment certainty is low, the threshold requirement is increased or an alternative decision path is triggered. If the uncertainty exceeds the preset threshold, multiple possible grouping schemes can be retained, or additional feature information can be requested to improve judgment accuracy. This mechanism significantly improves the system's robustness in complex web page environments, effectively preventing incorrect groupings, and excels in handling edge cases and abnormal samples.

[0174] Step S447, incremental BDD update, includes:

[0175] When the structure and content of a web page change, based on a dynamic update strategy, the subgraph structure in the binary decision graph is updated through local reconstruction technology to achieve incremental graph structure adjustment and incorporate new feature relationships;

[0176] An incremental BDD update algorithm is designed to efficiently incorporate new feature variables and logical relationships without rebuilding the entire BDD. It uses local reconstruction techniques to update only the affected BDD subgraphs, and maintains the optimal BDD structure by reordering variables.

[0177] Specifically, this step designs an efficient incremental BDD update algorithm that enables the system to quickly adapt to changes without rebuilding the entire model. When a change in the web page structure or a new field association pattern is detected, the system first locates the affected BDD subgraph. In the specific implementation, when the web page structure and content change, based on the dynamic update strategy, the scope of the change on the existing BDD structure is analyzed, and then only the affected BDD subgraph structure is updated through local reconstruction technology. For new variables or conditions that need to be added, new variables are inserted through node splitting and edge reconnection operations while maintaining the order of existing variables; for logical relationships that need to be updated, conditional updates are achieved through local path reconstruction. After completing the local update, variable reordering and graph compression technology are applied again to maintain the optimal structure of the BDD. This incremental update mechanism enables the system to adapt to web page changes at a very low computational cost without retraining the entire model, significantly improving the timeliness and continuous adaptability of the system.

[0178] Step S448, multimodal BDD fusion, implements multimodal BDD fusion technology, builds dedicated BDDs for different information modalities, and then fuses them into a unified decision model through a hierarchical combination strategy, designs a collaborative enhancement mechanism between modalities, and when the judgment of a certain modality is uncertain, the information of other modalities can be dynamically used to improve the reliability of the decision.

[0179] Specifically, this step implements multimodal BDD fusion technology to integrate the judgment capabilities of different information sources. Dedicated BDDs are constructed for different information modalities (such as text semantic modality, DOM structure modality, and visual layout modality), each capturing the similarity judgment logic within that modality. These single-modal BDDs are then fused into a unified decision model through a hierarchical combination strategy. Specific implementation methods include sequential combination (where the output of modality A serves as the input condition for modality B), weighted voting (where the judgment results of each modality's BDD are weighted and summed), and meta-decision (where a high-level BDD is constructed to determine how to combine the results of each modality). A collaborative enhancement mechanism between modalities is also designed. When the judgment of a particular modality is uncertain, the weight of other more certain modalities is dynamically increased. For example, on pages with sparse text content, the influence of the visual layout modality is enhanced; on pages with complex structures, the weight of the text semantic modality is increased. This multimodal fusion approach fully utilizes all types of information on a webpage, maintains high accuracy in complex and changing web environments, and significantly improves the robustness and adaptability of the system.

[0180] As a specific embodiment of the present invention, step S5 may also be enhanced by using integrated tensor representation, including:

[0181] Step S501, constructing an integrated tensor representation, includes:

[0182] Acquire link features through the webpage DOM structure data stream of the new webpage, and perform tensor decomposition operations based on the link features to extract potential pattern representations;

[0183] Organize the multidimensional features of web page links into a high-order tensor structure, extract the text semantic features, URL structure features, contextual relationship features, and historical click features of all links from the web page, and build a fourth-order or higher-order feature tensor. Then, use tensor decomposition techniques such as Tucker decomposition or CP decomposition to extract potential feature patterns and association structures, generating a more compact and information-rich link representation.

[0184] Step S502, defining a local reversible mapping, includes:

[0185] performing self-organizing map clustering based on the latent pattern representation to construct a local reversible map;

[0186] The self-organizing map (SOM) technique is used to cluster the link features and determine the natural partitions of the feature space. Then, a reversible mapping function is constructed within each partition using the normalizing flow technique.

[0187] Specifically, the core of the local reversible mapping technique is to partition the high-dimensional feature space into multiple local regions and construct a dedicated reversible transformation within each region. In implementation, the system first applies a self-organizing map (SOM) to naturally partition the feature space. The SOM network adopts a 20×20 two-dimensional grid topology, with each grid point corresponding to a high-dimensional prototype vector. During training, the system randomly selects a feature sample and finds the closest prototype vector (called a best matching unit (BMU)). The BMU and the prototype vectors in its neighborhood are then updated to move toward the sample. A large neighborhood radius and learning rate are initially used, which are gradually reduced as training progresses, achieving a coarse to fine spatial partitioning. After SOM training is complete, the system partitions the feature space into multiple local regions, each containing samples mapped to the same grid point or adjacent grid points. Next, a normalized flow model is constructed within each local region to achieve reversible mapping. The normalized flow consists of a combination of multiple simple reversible transformations, primarily employing an architecture that alternates affine coupling layers and activation normalization layers. The affine coupling layer divides the input features into two parts, one of which remains unchanged, and the other is updated through the nonlinear transformation of the previous part to ensure that the overall transformation is reversible. The system independently trains the normalized flow model for each local region and maximizes the likelihood function using the samples within the region. To ensure that the local mapping transitions smoothly at the region boundary, the system implements a boundary sample sharing mechanism, that is, samples near the boundary are also used to train the mapping model of the adjacent region. In addition, the system also designs an adaptive partition adjustment strategy to dynamically subdivide or merge feature space regions according to the mapping quality. Compared with global mapping, this local reversible mapping method can more accurately capture the local structure of the feature space and retain key information through reversibility, providing a more reliable feature representation for subsequent decision-making.

[0188] Step S503: designing a compact decision graph structure. Based on the local reversible mapping, a compact decision graph is constructed, and the link priority evaluation result is output. An initial decision tree is constructed based on the information gain ratio principle, and then converted into a more compact graph structure through edge merging, path compression, and isomorphic subgraph identification.

[0189] Step S504: Multi-scale feature hierarchical fusion. This implements a multi-scale feature hierarchical fusion mechanism. Feature representations at different scales are connected via a compact decision graph to construct a feature pyramid comprising the micro-level (text vocabulary, URL patterns), the meso-level (page sections, link context), and the macro-level (website structure, domain name authority). A cross-level feature transfer channel is then designed to achieve bidirectional flow and fusion of features at different scales.

[0190] Step S505: Priority reasoning based on tensor networks, implementing a priority reasoning mechanism based on tensor networks, mapping the compact decision graph into a tensor network structure, designing a special tensor contraction operation, and efficiently calculating the priority score of the link through a series of tensor product and contraction steps;

[0191] Step S506: Locally reversible correction and optimization. This utilizes the characteristics of locally reversible mapping to implement a high-precision priority correction mechanism. This monitors the actual effectiveness of link collection (e.g., whether the target merchant information is included), maps this feedback back to the feature space, and then uses the inverse transformation of the locally reversible mapping to precisely locate the feature areas and decision paths that cause errors.

[0192] Step S507: Dynamic decision path adaptation. This introduces a dynamic decision path adaptation mechanism, enabling the compact decision graph to automatically adjust its decision logic based on environmental changes. This uses an online learning framework to continuously monitor the effectiveness of different decision paths. By using multi-armed bandit algorithms such as Thompson sampling, a balance is struck between exploration and exploitation, dynamically adjusting the activation probability of decision paths.

[0193] Step S508, explainability enhancement and visualization, designs an explainability enhancement and visualization mechanism to make the decision process of the compact decision graph transparent, converts complex tensor operations and decision paths into intuitive graphical representations, and displays the key factors and decision basis of link evaluation through an interactive interface.

[0194] Reference Figure 7 The present invention also provides a system for intelligently collecting and analyzing web merchant information, comprising:

[0195] The webpage crawling module 100 is used to query the search engine based on the merchant brand name and parse the links in the search results page, build an initial URL crawling queue, use the browser automation tool to access the links in the initial URL crawling queue, and obtain the webpage DOM structure data stream;

[0196] The field identification module 200 is used to use the webpage DOM structure data stream and combine the semantic association between key identifiers and text paragraphs to identify the merchant name, phone number and address in the webpage content and generate a set of candidate fields;

[0197] An address parsing module 300 is used to perform administrative division identification and address component parsing on the address field in the candidate field set, and output structured address data;

[0198] The merchant grouping module 400 is used to calculate the path similarity, visual proximity and semantic relevance between the candidate field sets, determine the relationship between the fields through nonlinear fusion, and generate structured merchant information;

[0199] The page discovery module 500 is used to obtain the web page DOM structure data stream of a new web page, perform text content and URL feature analysis on the links in the web page DOM structure data stream, identify target links containing merchant information, and add the target links to the initial URL crawling queue to realize automatic information collection.

[0200] The above description is merely an illustration of preferred embodiments of the present invention and the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by combining the above-mentioned features with (but not limited to) the technical features disclosed in this invention.

Claims

1. A method for intelligently collecting and analyzing merchant information on a webpage, characterized in that: include: Based on the merchant brand name, query the search engine and parse the links in the search results page to build an initial URL crawling queue, use browser automation tools to access the links in the initial URL crawling queue, and obtain the web page DOM structure data stream; Using the webpage DOM structure data stream, combined with the semantic association between key identifiers and text paragraphs, the merchant name, telephone number and address of the webpage content are identified to generate a candidate field set; Performing administrative division identification and address component analysis on the address field in the candidate field set, and outputting structured address data; For the candidate field set, the path similarity, visual proximity and semantic relevance between the fields are calculated, the correlation relationship between the fields is determined through nonlinear fusion, and structured merchant information is generated; When obtaining a webpage DOM structure data stream of a new webpage, performing text content and URL feature analysis on links in the webpage DOM structure data stream of the new webpage, identifying target links containing merchant information, and adding the target links to the initial URL crawling queue to achieve automated information collection; The step of calculating the path similarity, visual proximity, and semantic relevance between the candidate field sets, determining the relationship between the fields through nonlinear fusion, and generating structured merchant information includes: Based on the candidate field set, calculating the XPath path depth difference of the candidate fields and generating a path similarity feature; Based on the candidate field set, calculating the DOM element coordinate distance and generating a visual proximity feature; Based on the candidate field set, calculating the relevance of different text contents and generating semantic relevance; Based on the path similarity feature, the visual proximity feature and the semantic association, radial basis function fusion is performed to output the association relationship between fields.

2. The method according to claim 1, characterized in that The step of constructing the initial URL crawling queue includes: Based on the links in the search results page, domain name features, TLD type features, and URL path features are extracted to calculate a multi-dimensional score; Based on the multi-dimensional scoring, the links are filtered and prioritized to obtain an optimized initial URL crawling queue.

3. The method according to claim 1, characterized in that The method utilizes the webpage DOM structure data stream and combines the semantic association between key identifiers and text paragraphs to identify the merchant name, phone number, and address of the webpage content and generate a candidate field set, including: Based on the predefined keyword library, identify the keyword words in the text nodes and mark the text area containing the target information; Based on the text area marked as containing target information, performing feature matching of the merchant name, telephone number and address to generate candidate fields; Verification is performed on the candidate fields, and confidence scores are calculated based on the verification results, and the candidate field set is output.

4. The method according to claim 1, wherein The step of performing administrative division identification and address component parsing on the address field in the candidate field set and outputting structured address data includes: Performing text cleaning and traditional / simplified conversion on the address field in the candidate field set to generate pre-processed address text; Based on the preset administrative division library, the provincial, municipal and district / county address information of the pre-processed address text is identified, and based on the trained deep learning model, the street and house number recognition is performed on the remaining text to output the structured address data.

5. The method according to claim 1, wherein The step of identifying the merchant name, phone number, and address of the webpage content is enhanced using a random sequence model and includes: Based on the webpage DOM structure data stream, a node sequence relationship graph is constructed to generate a random sequence model; Based on the pre-trained feature extraction model, the multi-dimensional feature vector of the node is extracted and the generalized assignment space is constructed; Based on the generalized value space and utilizing the inter-node sequence relationship determined by the random sequence model, semi-supervised learning training is performed to generate a probability distribution of merchant information types.

6. The method according to claim 5, characterized in that Also includes: Based on the node multidimensional feature vector and the probability distribution of the merchant information type, spectral clustering analysis is performed to achieve automatic grouping of multiple merchant information; Based on the feedback information of the collection results, perform online learning optimization to update the model parameters of the random sequence model; Based on the random order model, graph convolutional network technology is used to integrate cross-page merchant information and output a complete merchant information set.

7. The method according to claim 1, characterized in that The step of determining the association relationship between fields through nonlinear fusion is enhanced by a binary decision diagram and includes: Based on the candidate field set, construct a binary decision graph structure of node relationships; Based on the multi-threshold quantization technology, the similarity feature space is discretized; Based on Boolean logic expression optimization, binary decision graph minimization is performed to generate a similarity reasoning model.

8. The method according to claim 7, characterized in that The binary decision graph structure for constructing node relationships includes: Based on the principle of maximizing information entropy, for different types of similarity feature spaces, determining a set of segmentation points for feature quantization; Based on the segmentation point set, a shared subgraph structure optimization strategy is adopted to perform shared optimization on the subgraph structure in the binary decision graph to reduce the graph complexity; When the structure and content of a web page change, based on a dynamic update strategy, the subgraph structure in the binary decision graph is updated through local reconstruction technology to achieve incremental graph structure adjustment and incorporate new feature relationships.

9. The method according to claim 1, characterized in that The step of analyzing the text content and URL features of the links in the webpage DOM structure data stream of the new webpage, which is enhanced by integrated tensor representation, includes: Acquire link features through the webpage DOM structure data stream of the new webpage, and perform tensor decomposition operations based on the link features to extract potential pattern representations; performing self-organizing map clustering based on the latent pattern representation to construct a local reversible map; Based on the local reversible mapping, a compact decision graph is constructed and a link priority evaluation result is output.

10. A system for intelligent collection and analysis of web merchant information, characterized in that: include: The webpage crawling module is used to query the search engine based on the merchant brand name and parse the links in the search results page, build an initial URL crawling queue, use the browser automation tool to access the links in the initial URL crawling queue, and obtain the webpage DOM structure data stream; A field identification module is used to use the webpage DOM structure data stream and combine the semantic association between key identifiers and text paragraphs to identify the merchant name, telephone number and address of the webpage content and generate a set of candidate fields; An address parsing module, configured to perform administrative division identification and address component parsing on the address fields in the candidate field set, and output structured address data; The merchant grouping module is used to calculate the path similarity, visual proximity and semantic relevance between the candidate field sets, determine the inter-field association relationship through nonlinear fusion, and generate structured merchant information, including: calculating the difference in XPath path depth of the candidate fields based on the candidate field set to generate a path similarity feature; calculating the DOM element coordinate distance based on the candidate field set to generate a visual proximity feature; calculating the relevance of different text contents based on the candidate field set to generate a semantic association; performing radial basis function fusion based on the path similarity feature, the visual proximity feature and the semantic association to output the inter-field association relationship; The page discovery module is used to obtain the web page DOM structure data stream of a new web page, analyze the text content and URL characteristics of the links in the web page DOM structure data stream, identify the target link containing merchant information, and add the target link to the initial URL crawling queue to realize automatic information collection.

Images