Method and system for intelligently collecting and analyzing webpage merchant information

Abstract

The invention discloses a method and a system for intelligently acquiring and analyzing webpage merchant information. Automatic acquisition of the merchant information is realized through five main steps of intelligent webpage capture, intelligent field identification, accurate address information analysis, intelligent merchant grouping and automatic page discovery. According to the method, a traditional rule-based extraction method is abandoned, information such as merchant names, phone numbers and addresses in webpages is automatically recognized by using context semantic analysis and mode recognition technologies, and accurate association grouping of fields is realized through a dynamic weight multi-dimensional similarity calculation method; and the merchant information can be continuously discovered and collected by using an automatic page discovery mechanism without manually specifying a collection path. The technical bottlenecks that specific rules need to be written for different websites, webpage structure changes are difficult to deal with, multi-merchant information grouping cannot be processed and the like in a traditional method are broken through, the method adapts to various webpage structure changes, and the automation degree, accuracy and efficiency of merchant information collection are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet data collection and information processing, and particularly to a method and system for intelligent collection and parsing of web merchant information. Background Art

[0002] Web data collection and information extraction are important technologies in the Internet field and are widely used in scenarios such as search engines, business intelligence collection, and market analysis. In the field of merchant information collection, it is necessary to extract key information such as store names, phone numbers, and addresses from various websites to support subsequent data analysis and applications.

[0003] Traditional web data collection technologies mainly rely on rule-based extraction methods such as Regular Expressions and XPath selectors. For example, a common technique is to manually analyze the web page structure and write fixed XPath rules for a specific website to locate and extract target elements.

[0004] The most relevant prior art usually adopts the method of manually writing scripts for web pages. Technical personnel need to analyze the DOM structure of each target site and write specific rules to match target elements. This method relies on the fixed structure of the web page and specific HTML tag attributes, traverses HTML elements, extracts node content, and identifies and classifies through preset rules.

[0005] However, this traditional method has obvious technical defects: First, different parsing rules need to be written for each new website, resulting in a large workload; Second, changes in the web page structure (such as website redesign) will cause the written rules to become invalid, requiring continuous maintenance and update; Third, it is difficult to handle the situation where there are multiple merchant information in a single page (such as a branch list), and it is impossible to automatically associate different information fields of the same merchant; Finally, manual operations are required to find the page containing the target information, lacking the ability of automatic discovery, resulting in low collection efficiency.

[0006] Therefore, there is an urgent need for an intelligent method and system for collecting and parsing web merchant information that can automatically adapt to various web page structures, do not require preset rules, can intelligently identify and associate merchant information, and automatically discover target pages. Summary of the Invention

[0007] The purpose of the present invention is to solve the problems in the prior art such as the need to write specific rules for each website, difficulty in coping with web page structure changes, inability to handle multiple merchant information, and lack of automatic discovery ability, and to provide a method and system for intelligent collection and parsing of web merchant information.

[0008] To achieve the above purpose, the present invention provides the following technical solutions: A method for intelligent collection and parsing of web merchant information, including: Query and access the search engine according to the merchant brand name, parse the links in the search result page, construct an initial URL crawling queue, and use a browser automation tool to access the web pages of the links in the initial URL crawling queue to obtain the web page DOM structure data stream; Utilize the web page DOM structure data stream, combine key identification words and semantic associations of text paragraphs, identify the merchant name, telephone number, and address in the web page content, and generate a candidate field set; Perform administrative division identification and address component parsing on the address fields in the candidate field set, and output structured address data; For the candidate field set, calculate the path similarity, visual proximity, and semantic association degree between fields, determine the association relationship between fields through non-linear fusion, and generate structured merchant information; When obtaining the web page DOM structure data stream of a new web page, perform text content and URL feature analysis on the links in the web page DOM structure data stream of the new web page, identify the target links containing merchant information, and add the target links to the initial URL crawling queue to achieve automated information collection.

[0009] Preferably, the construction of the initial URL crawling queue includes: Based on the links in the search result page, extract domain name features, TLD type features, and URL path features, and calculate a multi-dimensional score; According to the multi-dimensional score, filter and prioritize the links to obtain an optimized initial URL crawling queue.

[0010] Preferably, the utilization of the web page DOM structure data stream, combining key identification words and semantic associations of text paragraphs, to identify the merchant name, telephone number, and address in the web page content and generate a candidate field set includes: Based on a predefined key identification word library, identify the key identification words in the text nodes and mark the text areas containing target information; Based on the marked text areas containing target information, perform feature matching of the merchant name, telephone number, and address to generate candidate fields; Verify the candidate fields and calculate the confidence score according to the verification results, and output the candidate field set.

[0011] Preferably, the performing of administrative division identification and address component parsing on the address fields in the candidate field set and outputting structured address data includes: Perform text cleaning and traditional-simplified conversion on the address fields in the candidate field set to generate preprocessed address text; Based on a preset administrative division database, identify provincial, municipal, and district / county-level address information in the preprocessed address text, and based on the trained deep learning model, identify street and house number information in the remaining text, and output the structured address data.

[0012] Preferably, for the candidate field set, calculate the path similarity, visual proximity, and semantic association degree between fields, and determine the association relationship between fields through non-linear fusion to generate structured merchant information, including: Based on the candidate field set, calculate the difference in XPath path depth of candidate fields to generate a path similarity feature; Based on the candidate field set, calculate the DOM element coordinate distance to generate a visual proximity feature; Based on the candidate field set, calculate the relevance of different text contents to generate a semantic association degree; Based on the path similarity feature, the visual proximity feature, and the semantic association degree, perform radial basis function fusion and output the association relationship between fields.

[0013] Preferably, for the identification of merchant name, telephone number, and address in the web page content, this step is enhanced by a random order model, including: Based on the web page DOM structure data stream, construct a node order relationship graph to generate a random order model; Based on a pre-trained feature extraction model, extract node multi-dimensional feature vectors to construct a generalized assignment space; Based on the generalized assignment space, use the node order relationship determined by the random order model to perform semi-supervised learning training to generate a probability distribution of merchant information types.

[0014] Preferably, it further includes: Based on the node multi-dimensional feature vectors, combined with the probability distribution of merchant information types, perform spectral clustering analysis to realize automatic grouping of multi-merchant information; Based on the feedback information of the collection result, perform online learning optimization to update the model parameters of the random order model; Based on the random order model, use graph convolutional network technology to integrate cross-page merchant information and output a complete set of merchant information.

[0015] Preferably, for the determination of the association relationship between fields through non-linear fusion, this step is enhanced by a binary decision diagram, including: Based on the candidate field set, construct a binary decision diagram structure of node relationships; Based on multi-threshold quantization technology, perform discretization processing on the similarity feature space; Based on the optimization of Boolean logic expressions, perform binary decision diagram minimization to generate a similarity inference model.

[0016] Preferably, the binary decision diagram structure for constructing node relationships includes: Based on the principle of maximizing information entropy, determine a set of segmentation points for feature quantization for different types of the similarity feature space; Based on the set of segmentation points, adopt an optimization strategy of a shared subgraph structure to perform shared optimization on the subgraph structure in the binary decision diagram, reducing the graph complexity; When the web page structure and content change, based on a dynamic update strategy, update the subgraph structure in the binary decision diagram through local reconstruction technology to achieve incremental graph structure adjustment and incorporate new feature relationships.

[0017] Preferably, for the analysis of text content and URL features of links in the web page DOM structure data stream of the new web page, this step is enhanced by integrated tensor representation, including: Obtain link features through the web page DOM structure data stream of the new web page. Based on the link features, perform tensor decomposition operations to extract potential pattern representations; Based on the potential pattern representations, perform self-organizing mapping clustering to construct a local reversible mapping; Based on the local reversible mapping, construct a compact decision diagram and output the link priority evaluation result.

[0018] An apparatus for intelligent collection and parsing of web page merchant information includes: A web page scraping module, used to query and access a search engine according to the merchant brand name, parse the links in the search result page, construct an initial URL scraping queue, and use a browser automation tool to access the links in the initial URL scraping queue to obtain a web page DOM structure data stream; A field recognition module, used to utilize the web page DOM structure data stream, combine key identification words and the semantic association of text paragraphs to identify the merchant name, phone number, and address in the web page content, and generate a set of candidate fields; An address parsing module, used to perform administrative division recognition and address component parsing on the address fields in the set of candidate fields, and output structured address data; A merchant grouping module, used to calculate the path similarity, visual proximity, and semantic association degree between fields for the set of candidate fields, determine the inter-field association relationship through non-linear fusion, and generate structured merchant information; A page discovery module, which is used to analyze the text content and URL features of links in the web page DOM structure data stream when obtaining the web page DOM structure data stream of a new web page, identify target links containing merchant information, and add the target links to the initial URL crawling queue to achieve automated information collection.

[0019] The beneficial effects of the present invention are: 1. There is no need to write specific rules for different websites: By using context semantic analysis and pattern recognition technologies to automatically identify merchant information in web pages, adapting to various web page structure changes, there is no need to write specific XPath rules or regular expressions for each website, greatly reducing the manual workload. 2. It can automatically handle web page structure changes: By analyzing text semantics and content patterns to identify merchant information, rather than relying on a fixed HTML tag structure, it can still work properly when the website is revised and the DOM structure changes, improving the system stability and continuous effectiveness. 3. Accurately handle scenarios with multiple merchant information: By using dynamic weight multi-dimensional similarity calculation and non-linear fusion technologies, it can automatically group and associate different merchant names, addresses, phone numbers, etc. information on the same page correctly, solving the problem that traditional methods cannot handle the mixture of multiple merchant information.

[0020] 4. Automatic page discovery ability: By analyzing link text semantics and URL features, it automatically identifies target pages that may contain merchant information and adds them to the crawling queue, without the need for manual specification of the collection path, greatly improving the collection coverage rate and efficiency.

[0021] 5. High-precision address parsing: Through multi-level administrative division recognition and deep learning model processing, it realizes accurate parsing of various non-standard address expressions, improving the structured quality of address information. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0023] Figure 1 It is a flowchart of the intelligent collection and parsing method for web page merchant information of the present invention; Figure 2 It is a flowchart of the intelligent field recognition step of the present invention; Figure 3 It is a flowchart of the intelligent merchant grouping step of the present invention; Figure 4 It is a flowchart of the automatic page discovery step of the present invention; Figure 5 Flow chart of intelligent field recognition enhanced by the random order model of the present invention; Figure 6 Flow chart of non - linear similarity fusion enhanced by the binary decision diagram of the present invention; Figure 7 Structural diagram of the intelligent acquisition and parsing system for web merchant information of the present invention. Detailed implementation manners

[0024] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0025] The present invention provides a method and system for intelligent acquisition and parsing of web merchant information, which can automatically adapt to various web page structure changes, without the need to write specific rules for different websites, can accurately process multi - merchant information scenarios, and has the ability of automatic page discovery.

[0026] Referring to Figure 1 , the intelligent acquisition and parsing method for web merchant information of the present invention includes the following steps: Step S1, according to the merchant brand name, query and access the search engine, parse the links in the search result page, construct an initial URL crawling queue, and use a browser automation tool to access the links in the initial URL crawling queue to obtain the web page DOM structure data stream. In this step, first, the mainstream search engine is accessed, and the merchant brand name is input as the search condition to obtain the search result page. Then, all links are extracted from the search result page, and the links are evaluated to determine whether they are the official websites of the target merchants by analyzing the domain name features of the links (such as whether they contain the brand name), the TLD types (such as ".com", ".cn", etc.) and the URL path features (such as whether they contain keywords such as "official", "index", etc.), so as to construct the initial URL crawling queue. Next, the system uses browser automation tools such as Selenium or Puppeteer to access the links in the queue. These tools can simulate real user behaviors, support complex page processing such as JavaScript rendering, Cookie handling, and dynamic content loading, and avoid being intercepted by the anti - crawler systems of the target websites. Finally, the obtained HTML page content is cleaned, removing interference content such as comments and irrelevant scripts, constructing a complete DOM tree structure, extracting all visible text nodes and their XPath paths, coordinate positions, etc. information to form the web page DOM structure data stream, providing basic data for subsequent field recognition.

[0027] Step S2: Using the web page DOM structure data stream, combining key identifying words and the semantic associations of text paragraphs, identify the merchant name, phone number, and address in the web page content to generate a set of candidate fields. This step first traverses all text nodes of the web page DOM tree, identifies key identifying words such as "Phone:", "Contact Information:", "Address:", etc., and combines the text content before and after the nodes to construct a semantic association graph of the text paragraph, marking the text areas that may contain the target information. Then, in the marked text areas, the system uses regular expressions to match various phone number patterns (such as "123-4567-8901", "12345678901", "(123)45678901", etc.), considering variant cases such as international number prefixes and extension numbers, extracts candidate phone numbers and records their XPath paths. At the same time, based on a Chinese address feature word library (including administrative divisions such as provinces / cities / districts / counties / roads / streets and common address words), the system identifies areas with dense address feature words in the text, extracts possible complete address texts through a sliding window method, and records their XPath paths. For the identification of the merchant name, the system analyzes features such as the page title, content of H1-H3 tags, text near the logo, and prominent styles (such as large fonts, bold, special colors), combines the search keyword matching degree, identifies possible merchant names, and records their XPath paths. Finally, verify the extracted candidate fields. The phone number is verified through rules such as digital format and length; the address is verified through administrative division matching; the merchant name is verified through search term similarity. Calculate the confidence score for each field based on the verification results to form a set of candidate fields.

[0028] Step S3: Perform administrative division recognition and address component parsing on the address fields in the candidate field set, and output structured address data. In this step, the system first performs text cleaning and traditional-simplified conversion on the address fields in the candidate field set, removes redundant spaces and special characters, unifies traditional-simplified conversion, detects duplicate substrings (such as "B City B City"), corrects common input errors, and generates preprocessed address text. Then, the system uses a pre-loaded provincial administrative division library to achieve provincial address recognition, adopts the principle of the longest substring first to avoid incorrect matching (such as misidentifying "H Province Road" as a province), and outputs provincial address information. Based on the provincial recognition result, continue to identify the municipal and district / county-level administrative divisions in the remaining text, adopt the long name first strategy to solve the problem of same-name districts and counties (such as "XX District" existing in both B City and C City), and ensure reasonable administrative division logic. After identifying the province, city, and district, the system uses a BiLSTM+CRF deep learning model to identify unstructured address components such as streets and house numbers in the remaining text, processes various non-standard expressions, and extracts key address components. Finally, by calling the map API service, the system performs geocoding verification on the parsed address, determines whether the address exists, obtains the standardized address expression, corrects possible errors, and finally outputs high-quality structured address data.

[0029] Step S4: For the candidate field set, calculate the path similarity, visual proximity, and semantic association degree between fields, determine the association relationship between fields through non-linear fusion, and generate structured merchant information. In this step, the system first analyzes the DOM structure features of all extracted fields on the page, calculates structured metrics such as XPath depth difference, element Y coordinate distribution, and text noise ratio as the basic parameters for dynamic weight calculation, enabling the system to adapt to different page structures. Then, based on these page structure features, the system calculates the weight coefficients of multi-dimensional features such as path similarity, visual proximity, semantic association degree, and temporal matching degree to achieve adaptive adjustment of weights. Next, the system calculates the path similarity (implemented through edit distance and tree structure comparison), visual proximity (calculated based on element coordinates), and semantic association degree (analyzed based on text content relevance) between different fields for all fields on the same page, and generates a similarity matrix between fields. To overcome the limitations of the linear weighted model, the system uses the radial basis function (RBF) kernel trick to non-linearly fuse multi-dimensional similarity features, calculate the comprehensive similarity score, and form a more accurate field association relationship. Finally, based on the comprehensive similarity score, the system applies the hierarchical clustering algorithm to automatically group and associate fields such as the name, address, and phone number of the same merchant, process the mixed situation of multi-merchant information, and output a structured set of merchant information.

[0030] Step S5, when obtaining the web page DOM structure data stream of the new web page, analyze the text content and URL features of the links in the web page DOM structure data stream of the new web page, identify the target links containing merchant information, and add the target links to the initial URL crawling queue to achieve automated information collection. This step first extracts all The link text of the label is used to identify the target link that may contain merchant information through keyword matching (such as "Contact Us", "Branch Information", "National Stores", etc.) and semantic similarity calculation. At the same time, the system analyzes the URL path characteristics of the link, identifies common path patterns that contain merchant information such as " / contact", " / about", " / stores", " / branch", etc., and evaluates the possibility that the link contains the target information. Combining the semantic score of the link text and the URL feature score, the system calculates the comprehensive score, determines the crawling priority of the link, and preferentially crawls the page that is most likely to contain merchant information to improve the collection efficiency. According to the information acquisition effect of the crawled page, the system dynamically adjusts the link priority calculation model, forms a self-learning page discovery mechanism, continuously optimizes the crawling strategy, and improves the adaptive ability of the system. Finally, the system adds the identified high-value links to the URL crawling queue, forming a loop with step S1 to realize an automated merchant information collection process, without manual specification of the collection path, greatly improving the collection coverage and efficiency.

[0031] As a specific embodiment of the present invention, step S1 further includes: Step S11, constructing an initial URL queue based on a search engine. By accessing mainstream search engines, inputting the merchant brand name or keyword as the search condition, obtaining the search result page, parsing the links in the result page, extracting the possible merchant official website URLs, and constructing an initial URL crawling queue; Step S12, URL priority analysis and filtering, including: Based on the links in the search result page, extract domain name features (such as containing the brand name), TLD type features (such as.com / .cn, etc.), and URL path features (such as containing keywords such as "official" / "index", etc.), and calculate a multi-dimensional score; According to the multi-dimensional score, filter and prioritize the links to obtain an optimized initial URL crawling queue; Step S13, implementing browser simulation requests. Use browser automation tools such as Selenium / Puppeteer to simulate real user access behaviors, support complex page processing such as JavaScript rendering, cookie handling, and dynamically loaded content, avoid being intercepted by anti-automation data collection systems, and ensure the acquisition of complete page content; Step S14, web content extraction and DOM structure parsing. Clean the obtained HTML page content, remove interfering content such as comments and irrelevant scripts, construct a complete DOM tree structure, extract all visible text nodes and their XPath paths, coordinate positions, etc. information, and form a web DOM structure data stream to provide basic data for subsequent field recognition.

[0032] Reference Figure 2 , as a specific embodiment of the present invention, step S2 further includes: Step S21, context semantic analysis, including: Based on a predefined key identifier library, identify the key identifier words (such as "Telephone:", "Contact Information:", "Address:", etc.) in the text nodes, and mark the text areas containing the target information; Combine the text content before and after the nodes to construct a semantic association graph of the text paragraph, and mark the text areas that may contain the target information; Step S22, telephone number pattern recognition, in the marked text areas, use regular expressions to match telephone number patterns (such as "123-4567-8901", "12345678901", "(123)45678901", etc. in various formats), consider variants such as international number prefixes and extension numbers, extract the candidate telephone numbers and record their XPath paths; Step S23, address feature recognition, based on a Chinese address feature word library (including administrative divisions such as provinces / cities / districts / counties / roads / streets and common words for addresses), identify the areas with dense address feature words in the text, and extract the possible complete address texts by the sliding window method and record their XPath paths; Step S24, merchant name recognition, by analyzing features such as the page title, H1-H3 tag content, text near the logo, and prominent styles (such as large fonts, bold fonts, special colors), combined with the search keyword matching degree, identify the possible merchant names and record their XPath paths; Step S25, candidate field verification and confidence evaluation, including: Based on the text areas marked as containing the target information, perform feature matching of the merchant name, telephone number, and address to generate candidate fields; Verify the candidate fields and calculate the confidence scores according to the verification results, and output the set of candidate fields. Among them, the telephone number is verified by rules such as digital format and length; the address is verified by administrative division matching; the merchant name is verified by search term similarity.

[0033] , as a specific embodiment of the present invention, step S3 further includes: Step S31, address text preprocessing, including: Perform text cleaning and traditional / simplified Chinese conversion on the address fields in the candidate field set, remove redundant spaces, special characters, unify traditional and simplified Chinese, detect duplicate substrings (such as "B City B City"), correct common input errors, and generate preprocessed address text; Specifically, this step performs comprehensive text cleaning and standardization on the address fields in the candidate field set. The system first removes white space characters such as redundant spaces, tab characters, and line break characters from the address text, and at the same time clears special characters such as parentheses and quotation marks that may interfere with parsing. For Chinese addresses, the system performs unified traditional / simplified Chinese conversion to ensure that all traditional Chinese characters are converted to simplified Chinese characters for matching with the administrative division database. In addition, the system also identifies and processes common error patterns in the text, such as detecting duplicate substrings such as "B City B City" and "C City C City", and automatically correcting such duplicate errors caused by data input or OCR recognition. The system also identifies and standardizes common address abbreviations and aliases, such as converting "North Third Ring Road" to "North Third Ring Road". Through this series of preprocessing operations, the system generates standardized address text, laying a foundation for subsequent accurate parsing and significantly improving the parsing success rate.

[0034] Step S32, provincial administrative division recognition. Use the pre-loaded provincial administrative division database to implement provincial address recognition through the provincial address recognition function. Adopt the principle of the longest substring first to avoid incorrect matching (such as incorrectly identifying "H Province" in "H Province Road" as a province), and output provincial address information; Specifically, provincial administrative division recognition. This step uses the pre-loaded comprehensive provincial administrative division database to implement accurate provincial address recognition through a specially designed provincial address recognition function. The system adopts the principle of the longest substring first for matching, and tries to match the longest possible provincial administrative division name from the start position of the address text. This method can effectively avoid incorrect matching problems. For example, it can prevent incorrectly identifying "H Province" in "No. 10, Jianshe Street, H Province Road" as an independent province identifier, but correctly understand it as part of the road name. At the same time, it can also handle various expressions of provincial administrative divisions, such as "XX", "XX Province", and abbreviations, etc., which can all be correctly recognized as the same province. For the successfully matched provincial information, the system separates it from the original text and outputs the standardized provincial address information, providing accurate context constraints for subsequent city and district level recognition and improving the overall parsing accuracy.

[0035] Step S33, city and district level administrative division recognition. Based on the provincial recognition result, use the district and county address recognition function to identify the city and district level administrative divisions in the remaining text. Adopt the long name first strategy to solve the problem of districts and counties with the same name (such as "XX District" existing in both B City and C City) to ensure reasonable administrative division logic; Specifically, the identification of urban and rural administrative divisions. Based on the provincial results successfully identified in the previous step, the system uses the district and county address identification function to continue identifying the municipal and district / county-level administrative divisions in the remaining text. The system first searches for matches in the municipal administrative division database of a specific province to determine the municipal information; then, based on the identified provincial and municipal information, it performs a match in the corresponding district and county administrative division database to determine the district and county information. The entire process adopts the long name priority strategy, giving priority to the complete administrative division names, which helps to solve the problem of districts and counties with the same name. For example, when encountering "XX District", if the identified provincial information is "City B", it will be correctly associated with XX District of City B, rather than XX District of City C. It can also handle the abbreviations and aliases of administrative divisions, such as special cases like "City D" (a straight-tube city without administrative divisions). By maintaining the hierarchical constraint relationship of province-city-district / county, it ensures that the identified administrative divisions are logically reasonable and consistent, greatly improving the accuracy of address parsing.

[0036] Step S34, street and detailed address parsing. Based on the trained deep learning model, the remaining text is used to identify the street and house number. That is, after identifying the province, city, and district, the remaining text is used with the BiLSTM+CRF deep learning model to identify unstructured address components such as streets and house numbers, handle various non-standard expressions, and extract the key address components. Specifically, after successfully identifying the high-level administrative divisions such as provinces, cities, districts, and counties, the task is to identify the unstructured address components in the remaining text, including streets, road names, house numbers, and community names. This step is processed using the BiLSTM+CRF deep learning model, which is trained with a large amount of labeled data and can identify various non-standard expressions. The model regards the remaining part of the address as a sequence labeling problem, assigning specific labels to each character (such as B-Street and I-Street representing the start and inside of the street, B-Number and I-Number representing the start and inside of the house number, etc.). The system also integrates a road name dictionary and a common community name dictionary to enhance the model's recognition ability. For special formats of house numbers (such as "Room 501, Unit 3, Building 2"), the system uses specially designed rules for identification and structuring. Through this combined method, the system can accurately extract and classify each key component of the address, handle various non-standard expressions, and generate structured detailed address information.

[0037] The BiLSTM+CRF deep learning model used in this system is specifically optimized for Chinese address parsing tasks. The model adopts a three-layer structure: an embedding layer, a bidirectional LSTM layer, and a conditional random field layer. In the embedding layer, the system uses 300-dimensional word vectors to represent each Chinese character. The word vectors are obtained through pre-training on a 10-million Chinese address corpus, which contains rich address semantic information. The bidirectional LSTM layer consists of three stacked layers, with each layer containing 128 hidden units. The tanh function is used as the activation function, and a dropout mechanism (with a ratio of 0.3) is introduced in the LSTM unit to prevent overfitting. On top of the bidirectional LSTM, a conditional random field layer is added to model the transition probabilities between labels, ensuring that the prediction results meet the address structure constraints. The model training data comes from 1 million manually annotated Chinese address samples, covering address information in different formats and expressions across the country. Each address is labeled with the corresponding component types (such as province, city, district, street, house number, etc.) at the character level. During training, a batch size of 64 is used, the initial learning rate is 0.001, the Adam optimizer is used, and a learning rate decay strategy is implemented. When the performance on the validation set does not improve for 3 consecutive epochs, the learning rate is halved. To prevent overfitting, in addition to dropout, an early stopping mechanism is introduced. Training stops when the performance on the validation set does not improve for 5 consecutive epochs. The model achieved an F1 score of 95.7% on the independent test set, demonstrating its excellent performance in complex and diverse Chinese address parsing tasks.

[0038] Step S35, address validity verification. By calling the map API service, geocoding verification is performed on the parsed address to determine whether the address exists, obtain the standardized address expression, correct possible errors, and finally output high-quality structured address data.

[0039] Specifically, to ensure the actual accuracy and usability of the parsing results, validity verification is performed after address parsing. In this step, geocoding verification is performed on the parsed address by calling the map API service. The system passes the structured address information as a parameter to the map API and obtains the returned geocoding results, including latitude and longitude coordinates, standardized address expressions, confidence levels, and other information. If the map API can successfully identify the address and return a high-confidence result, it indicates that the parsed address is valid; if a low-confidence result or non-identification is returned, the system will try to adjust the combination of address components and resubmit for verification. Through the standardized address expression returned by the map API, the system can also correct and improve the original parsing results, such as supplementing missing administrative division information, unifying the expression of street names, etc. Finally, the system outputs high-quality structured address data that has been verified and optimized, providing a reliable basis for subsequent merchant information processing and applications.

[0040] Refer toFigure 3 , as a specific embodiment of the present invention, step S4 further includes: Step S41, page structure feature analysis, analyzing the DOM structure features of all extracted fields in the page, calculating structured indicators such as XPath depth difference, element Y coordinate distribution, and text noise ratio, as the basic parameters for dynamic weight calculation, and adapting to different page structures; Specifically, this step comprehensively analyzes the overall structure of the web page, providing a basis for subsequent dynamic weight calculation and field association. The system first extracts the DOM structure features of all candidate fields in the page, including the XPath path length and depth of the node where each field is located, the hierarchical position in the DOM tree, the parent-child relationship, etc. By comparing the common prefix length and branch points of the XPath paths of different fields, the system calculates the XPath depth difference indicator to evaluate the relative position relationship of the fields in the DOM structure. At the same time, the system analyzes the visual layout features of the page, calculates indicators such as element Y coordinate distribution, horizontal alignment degree, and element spacing, and identifies the set of fields that may belong to the same information block. In addition, the system also evaluates the text noise ratio of the page (such as the ratio of non-core information such as advertising content and navigation menus), and judges the information density and structural complexity of the page. These structured indicators together constitute the structure feature vector of the page, as the basic parameters for dynamic weight calculation, enabling the system to adapt to the structural characteristics of different types of web pages and improve the accuracy of field association.

[0041] Step S42, dynamic weight calculation, calculating the weight coefficients of multi-dimensional features such as path similarity, visual proximity, semantic association, and temporal matching based on the page structure features, and realizing the adaptive adjustment of weights; Specifically, based on the page structure features analyzed in the previous step, dynamic weight calculation is performed to adaptively determine the importance of different similarity features in the comprehensive evaluation. The system first evaluates the structural regularity of the page - for a page with a regular structure (such as a standard table layout), the path similarity weight should be higher; for a page with a loose structure (such as a free text paragraph), the semantic association weight should be higher. The system calls the calc_dynamic_weights() function to dynamically calculate the weight coefficients of multi-dimensional features such as path similarity (α), visual proximity (β), semantic association (γ), and temporal matching (ε) according to indicators such as XPath depth difference and element coordinate distribution dispersion. For example, in a page with a multi-list table layout, the visual proximity weight will automatically increase; in a page with dense information and complex structure, the path similarity weight will increase accordingly. Through this dynamic weight mechanism, the system can automatically adjust the similarity calculation strategy according to the characteristics of different web pages, greatly improving the adaptability and accuracy of merchant information grouping, and effectively processing various page structures from simple single-column layouts to complex multi-merchant mixed layouts.

[0042] Step S43, multi-dimensional similarity calculation, including: Based on the candidate field set, calculate the XPath path depth difference of candidate fields to generate a path similarity feature; Based on the candidate field set, calculate the DOM element coordinate distance to generate a visual proximity feature; Based on the candidate field set, calculate the relevance of different text contents to generate a semantic association degree; For all fields on the same page, calculate the path similarity (by edit distance and tree structure comparison), visual proximity (calculated based on element coordinates), and semantic association degree (based on text content relevance) between different fields to generate a similarity matrix between fields; Specifically, after determining the dynamic weights, perform multi-dimensional similarity calculation on all candidate fields extracted from the page to quantify the association degree between different fields. For path similarity calculation, the edit distance and tree structure comparison analysis of XPath paths are implemented through the path_similarity() function. Convert two XPath expressions into tree structures, calculate their structural similarity, and fields under the same parent node obtain higher path similarity scores. For visual proximity calculation, based on the rendering coordinate information of DOM elements, calculate the Euclidean distance and relative position relationship between elements. Elements that are close in position and visually aligned obtain higher visual proximity scores. For semantic association degree calculation, analyze the semantic relevance of field text contents, and use word vector models or pre-trained language models to calculate the semantic similarity between texts. Fields with relevant contents (such as keywords contained in the merchant name and relevant information in the address) obtain higher semantic association degree scores. Through the similarity calculation in these three dimensions, the system generates a comprehensive similarity matrix between fields, laying a foundation for subsequent non-linear fusion and merchant grouping.

[0043] Step S44, non-linear similarity fusion, using the radial basis function (RBF) kernel trick to non-linearly fuse multi-dimensional similarity features, calculate the comprehensive similarity score, and based on the path similarity feature, the visual proximity feature, and the semantic association degree, perform radial basis function fusion to output the association relationship between fields, forming a more accurate field association relationship and overcoming the limitations of the linear weighted model; Specifically, to overcome the limitations of the simple linear weighted model, the radial basis function (RBF) kernel technique is adopted for non-linear similarity fusion, which can more accurately capture the complex association patterns between fields. The multi-dimensional features such as the path similarity, visual proximity, and semantic association degree calculated in the previous step are mapped to a high-dimensional feature space, and the non-linear similarity between feature vectors is calculated through the RBF kernel function. This method can effectively handle the non-linear interaction relationship between features. For example, when the path similarity and visual proximity of two fields are both high, their comprehensive similarity is not simply the weighted sum of the two components, but shows a non-linear enhancement effect. The system adjusts the parameter γ in the RBF kernel function according to the dynamic weight calculated in step S42 to control the intensity of the non-linear mapping. The fused comprehensive similarity score can more accurately reflect the true association relationship between fields, effectively handle the field association judgment in various complex web page structures, and greatly improve the accuracy of merchant information association, especially showing superiority when dealing with complex pages with mixed multi-merchant information.

[0044] Step S45, merchant information clustering and grouping. Based on the comprehensive similarity score, apply the hierarchical clustering algorithm to automatically group and associate fields such as the name, address, and phone number of the same merchant, handle the mixed situation of multi-merchant information, and output a structured set of merchant information.

[0045] Specifically, adopt the hierarchical clustering algorithm (such as UPGMA or Ward's method), convert the comprehensive similarity matrix into a distance matrix, start from the most similar field pairs, and gradually merge fields to form a clustering tree. By analyzing the structure and distance distribution of the clustering tree, determine the optimal cut-off point, and divide the fields into different merchant groups. For each group, the system further identifies the keyword field types such as name, address, and phone number, and constructs a complete merchant information structure. It can also handle the fields that may be missing within the group, such as supplementing information across pages or filling in based on inferences from known fields. This step can effectively handle the mixed situation where a page contains multiple merchant information, such as complex scenarios like branch lists and franchisee information, automatically group and correctly associate different information fields of the same merchant, and output a structured set of merchant information. This automatic grouping ability is a significant advantage of the present invention compared with traditional methods, greatly improving the accuracy and integrity of information collection.

[0046] The implementation of the spectral clustering algorithm in this system involves multiple delicate steps. First, the system constructs an affinity matrix W based on the comprehensive similarity scores calculated in the previous stage. The matrix element Wij represents the similarity between field i and field j. The higher the similarity value, the more likely the two fields belong to the same merchant. The system adopts a Gaussian similarity function to ensure the non-negativity and symmetry of the matrix. Then, the degree matrix D is calculated. This is a diagonal matrix, and its diagonal element Dii is the sum of all elements in the i-th row of the affinity matrix W, representing the total connection strength between node i and all other nodes. Next, the system calculates the normalized Laplacian matrix L. This normalization process makes the eigenvalue distribution range fixed in the interval [0, 2], facilitating subsequent analysis. The system performs eigenvalue decomposition on the normalized Laplacian matrix L, calculates all its eigenvalues and corresponding eigenvectors. By analyzing the eigenvalue distribution, especially by finding the significant jump points in the eigenvalue sequence, the system can automatically determine the optimal number of clusters k. This method of determining the number of clusters based on eigenvalues is called the "eigenvalue gap method" and can adapt to the changes in the number of merchants on different pages. After determining k, the system selects the eigenvectors corresponding to the smallest k non-zero eigenvalues to form an n×k-dimensional eigenmatrix (n is the number of fields). Finally, the system applies the K-means++ algorithm to the row vectors of this eigenmatrix for clustering, and each cluster represents all relevant fields of a merchant. To improve the clustering stability, the system performs multiple runs of K-means with random initialization and selects the result with the highest silhouette coefficient as the final grouping. This spectral clustering method can better handle non-convex data distributions compared to traditional hierarchical clustering and is particularly suitable for the grouping task of merchant information in complex web pages.

[0047] Referring to Figure 4 , as a specific embodiment of the present invention, step S5 further includes: Step S51, link text semantic analysis, extract all The link text of the label, through keyword matching ("Contact Us", "Branch Information", "National Stores", etc.) and semantic similarity calculation, identifies target links that may contain merchant information; Step S52, Link URL Feature Analysis. Analyze the URL path features of the link, identify common path patterns that contain merchant information such as " / contact", " / about", " / stores", " / branch", etc., and evaluate the possibility that the link contains the target information; Step S53, Link Priority Ranking. Combine the semantic score of the link text and the URL feature score, calculate the comprehensive score, determine the crawling priority of the link, and preferentially crawl the page that is most likely to contain merchant information to improve the collection efficiency; Step S54, Dynamically Construct Crawling Strategy. According to the information acquisition effect of the crawled pages, dynamically adjust the link priority calculation model, form a self-learning page discovery mechanism, continuously optimize the crawling strategy, and improve the adaptive ability of the system; Step S55, Crawling Queue Update and Loop Collection. Add the identified high-value links to the URL crawling queue, form a loop with Step S1, implement an automated merchant information collection process, without the need for manual specification of the collection path, and greatly improve the collection coverage and efficiency.

[0048] Refer to Figure 5 , as a specific embodiment of the present invention, step S2 can also be enhanced by a random order model, including: Step S201, random order model establishment, including: Based on the web page DOM structure data stream, construct a node order relationship graph to generate a random order model; Perform a depth-first traversal of the DOM tree to establish an order relationship graph between nodes, construct a node transition probability matrix through the Markov random field theory, and form a random order model that can describe the content structure of the web page; Specifically, this step addresses the problem that traditional methods have difficulty accurately identifying relevant merchant information in unstructured web pages by modeling the hierarchical and sequential relationships of web page content. First, perform a deep traversal analysis on the input web page DOM structure data stream to establish an order relationship graph between nodes, capturing the structural and sequential characteristics of the web page content. In specific implementation, the web page is regarded as an ordered but random data stream, and a node transition probability matrix is constructed through the Markov random field theory. For each DOM node, analyze its relationship with adjacent nodes and calculate the conditional probability P(node j|node i), representing the possibility of moving from node i to node j. This random order model can describe the dynamic flow characteristics of web page content and is more capable of capturing the potential correlations between different information blocks than the static DOM tree structure, especially suitable for processing web pages with complex layouts and lacking clear structure identifiers. The random order model constructed in this way provides a probabilistic structural understanding framework for subsequent merchant information identification.

[0049] Step S202, generalized assignment feature extraction, includes: Based on a pre-trained feature extraction model, extract the multi-dimensional feature vectors of nodes and construct a generalized assignment space; Through conditional random field (CRF) combined with deep learning techniques, extract multi-dimensional feature vectors from web page nodes, including: text semantic features (obtained through pre-trained models such as BERT), structural features (DOM path features of nodes), visual features (rendering positions and styles of nodes), and relationship features (connection relationships between nodes); Specifically, this step realizes the extension and enhancement of traditional feature extraction methods. It not only focuses on the direct features of nodes (such as text content, label types), but also considers the position features, transition features, and global context features of nodes in the random order model. In specific implementation, through conditional random field (CRF) combined with deep learning techniques, extract multi-dimensional feature vectors from web page nodes. For text semantic features, use pre-trained language models such as BERT to map node text into high-dimensional semantic vectors to capture the deep semantic information of the text; for structural features, extract the DOM path features of nodes, including label sequences, attribute sets, sibling node features, etc.; for visual features, analyze the rendering positions and style information of nodes, including coordinate positions, font sizes, colors, visibility, etc.; for relationship features, the system extracts the connection relationships between nodes, such as hyperlinks, reference relationships, etc. These multi-dimensional features together constitute the generalized assignment space, providing rich feature representations for subsequent merchant information identification and greatly improving the system's ability to identify various non-standard format merchant information.

[0050] In actual deployment, the system uses the Chinese BERT-Base model (12 layers, 768 hidden units, 12 attention heads, 110 million parameters) as the basis for text semantic feature extraction. This model has been pre-trained on a Chinese corpus (including encyclopedias, news, web texts, etc.). To adapt to the merchant information collection scenario, the system performs targeted fine-tuning on the BERT model, using 50,000 manually annotated merchant information samples (including various merchant names, addresses, phone numbers, etc. and their contexts). A relatively small learning rate (2e-5) is used during the fine-tuning process to retain the general semantic knowledge of the pre-training while adapting to the specific semantic expressions of merchant information. When extracting features, the system uses the weighted average of the last four layers of BERT outputs as the semantic representation of the text, rather than just using the last layer, which can capture semantic information at different levels simultaneously. Specifically, for each text node, the system first segments it into segments of no more than 512 tokens, obtains the feature representation through the BERT model, and then for texts longer than 512, it is processed using a sliding window approach and the results are concatenated. Finally, the system concatenates the obtained 768-dimensional BERT feature vector with traditional features (such as position features, HTML tag features, etc.) to form a more comprehensive node representation. This method combining pre-trained language models significantly improves the system's ability to understand various non-standard merchant information expressions, especially performing excellently in dealing with implicit references, informal expressions, and industry-specific terms.

[0051] Step S203, Generalized Assignment Weight Learning. Through semi-supervised learning methods, use a small amount of labeled data to train the generalized assignment weight model, use variational inference techniques to estimate the initial parameters of the model, and then iteratively optimize the parameters through the Expectation-Maximization (EM) algorithm. Finally, learn the generalized assignment weight matrix applicable to different web page structures; Specifically, based on the random order model, through semi-supervised learning methods, automatically learn the importance weights of different features during the recognition process. This step first uses a small amount of manually annotated merchant information samples as seed data, and then uses variational inference techniques to estimate the initial parameters of the model. In the specific implementation, the system models the merchant information recognition problem as a conditional probability field and iteratively optimizes the parameters through the variational EM algorithm (the variational form of the Expectation-Maximization algorithm). In the E step, the system uses the current parameters to estimate the label distribution of the unlabeled data; in the M step, update the model parameters based on all data (including labeled and unlabeled data). Through multiple rounds of iteration, finally learn the generalized assignment weight matrix applicable to different web page structures. This matrix can adaptively assign weights to different features, enabling the system to accurately identify merchant information in various web page environments. This semi-supervised learning method significantly reduces the dependence on labeled data and overcomes the limitations of traditional fixed weight schemes when facing diverse web pages.

[0052] Step S204, Merchant Information Probability Inference, including: Based on the generalized assignment space and using the node order relationship determined by the stochastic order model, perform semi-supervised learning training to generate the probability distribution of merchant information types; By calculating the posterior probability distribution of nodes belonging to various types of merchant information (such as name, address, phone number, etc.), the system can make more accurate inferences in cases of high uncertainty, and implement approximate inference using the Gibbs Sampling method; Specifically, based on the generalized assignment model, perform probability inference on web page nodes for merchant information types. This step regards merchant information recognition as a sequence labeling problem, and calculates the posterior probability distribution of each node belonging to various types of merchant information (such as name, address, phone number, etc.). In specific implementation, the Gibbs Sampling method is used for approximate inference, which is a Markov Chain Monte Carlo (MCMC) technique suitable for dealing with high-dimensional and high-complexity probability models. First, randomly initialize all unknown labels, and then iteratively sample and update each label from the conditional distribution P(label i | all other labels, observed data). After sufficient iterations, the sampling results converge to the true posterior distribution. This probabilistic inference method can make more accurate judgments in cases of high uncertainty, effectively handle various irregular layouts and complex nested structures on the web page, and significantly improve the accuracy of merchant information recognition, especially performing well when dealing with fuzzy boundaries, partially visible, or non-standard format merchant information.

[0053] Step S205, Multi-entity Association Clustering. Based on the node multi-dimensional feature vector and combined with the probability distribution of merchant information types, perform spectral clustering analysis to achieve automatic grouping of multiple merchant information; First, construct an affinity matrix between information entities (similarity based on generalized assignment features), then determine the optimal number of clusters through eigenvalue decomposition of the Laplacian matrix, and finally apply the K-means++ algorithm to complete clustering in the dimensionality-reduced space; Specifically, for a web page containing multiple merchant information, based on the information entities in the generalized assignment space, automatic grouping of multiple merchant information is realized. This step first constructs an affinity matrix between information entities, where the matrix elements represent the similarity between entities, calculated based on the Euclidean distance or cosine similarity of generalized assignment features. Then, the system applies the spectral clustering algorithm to process this affinity matrix: first, construct the Laplacian matrix L = D - W (where W is the affinity matrix and D is a diagonal matrix with elements being the row sums of W); second, calculate the eigenvalues and eigenvectors of L, analyze the eigenvalue distribution to determine the optimal number of clusters k; then select the eigenvectors corresponding to the smallest k non-zero eigenvalues to form a reduced-dimensional representation; finally, apply the K-means++ algorithm in the reduced-dimensional space to complete the clustering. This clustering method based on spectral theory can effectively capture the non-linear structure of the data, perform excellently when dealing with data distributions of complex shapes, enable the system to accurately separate the information of different merchants, and at the same time correctly group various types of information (such as name, address, phone, etc.) of the same merchant, solving the problem of multi-entity confusion in traditional methods.

[0054] Step S206, adaptive model update. Based on the feedback information of the acquisition result, perform online learning optimization to update the model parameters of the random order model; by continuously collecting the feedback information of the acquisition result, the system uses the online learning algorithm to continuously optimize the random order model and the generalized assignment weights, enabling the system to automatically adapt to the website structure changes and newly emerging information expression methods. Specifically, to cope with the dynamic changes of the web page structure, an adaptive model update mechanism is designed. This step continuously optimizes the random order model and the generalized assignment weights by continuously collecting the feedback information of the acquisition result. In the specific implementation, the online learning algorithm is adopted. Whenever a new web page is processed and feedback (such as manual confirmation or cross-validation) is obtained, the model parameters are updated. To balance the influence of new and old data, the system uses an incremental learning method with weight decay, and newer samples obtain higher learning weights. At the same time, the system maintains a sliding window, only retaining the most recent N samples for online learning to prevent the model from being affected by outdated data. In addition, the system also implements a model performance monitoring mechanism. When it is detected that the performance drops by more than the threshold, the model retraining process is automatically triggered. Through this adaptive update mechanism, the system can continuously learn newly emerging web page patterns and information expression methods, and continuously improve the recognition accuracy over time, maintaining adaptability to website structure changes.

[0055] Step S207, cross-page information integration. Based on the random order model, use the graph convolutional network technology to integrate the cross-page merchant information and output a complete set of merchant information; regard the set of web pages of the website as a hypergraph structure, establish an association network between pages through link analysis and content similarity evaluation, and then use the graph convolutional network (GCN) technology to perform information propagation and fusion on the hypergraph.

[0056] Specifically, for the case where merchant information is scattered across multiple pages, based on the cross-page version of the random order model, effective integration of information is achieved. In this step, the set of web pages of the website is regarded as a hypergraph structure, where nodes represent pages and edges represent the link relationships between pages. Through link analysis and content similarity evaluation, an association network between pages is established. The graph convolutional network (GCN) technology is used to perform information propagation and fusion on this hypergraph to achieve cross-page knowledge transfer.

[0057] Referring to Figure 6 , as a specific embodiment of the present invention, step S44 can also be enhanced by a binary decision diagram, including: Step S441, binary decision diagram construction, including: Based on the set of candidate fields, construct a binary decision diagram structure of node relationships; Abstract each node in the web page DOM tree into a set of Boolean variables, where each variable represents whether the node meets a specific condition (such as containing a phone number pattern, address feature, etc.), and then construct a normalized binary decision diagram to simplify the complex node relationship logic into an efficient graph structure representation; Specifically, this step applies the efficient Boolean function representation method of the binary decision diagram (BDD) to the merchant information collection scenario. First, analyze the set of candidate fields, and abstract the relationships between fields into a series of Boolean variables and conditional judgments. In specific implementation, the system abstracts each node in the web page DOM tree into a set of Boolean variables, where each variable represents whether the node meets a specific condition (such as containing a phone number pattern, address feature, etc.). For any two candidate fields, the system defines a set of Boolean variables to represent various possible relationships between them, such as "whether the XPath path similarity exceeds the threshold T1", "whether the visual distance is less than the threshold D1", "whether it contains relevant semantic content", etc. Then, the system constructs a normalized binary decision diagram to simplify these complex relationship judgment logics into an efficient graph structure representation. BDD is a directed acyclic graph, where each non-terminal node represents a Boolean variable, and there are two out-edges corresponding to the cases where the variable takes values of 0 and 1 respectively, and the terminal node represents the function result (whether the fields are relevant). Through this structured representation, the system can efficiently process the complex logical relationships between a large number of nodes in the web page, providing a solid foundation for subsequent similarity calculation.

[0058] Step S442, multi-threshold feature quantization, including: Based on the principle of maximizing information entropy, determine the set of segmentation points for feature quantization for different types of the similarity feature space; Based on multi-threshold quantization technology, perform discretization processing on the similarity feature space; Divide the continuous similarity feature space into multiple discrete intervals. The system designs an adaptive set of thresholds for different types of features (such as path similarity, visual distance, semantic relevance, etc.), determines the optimal segmentation points through the principle of entropy maximization, and then converts the continuous features into multi-level discrete variables; Specifically, traditional similarity calculations often use a single threshold for feature quantization, which is prone to information loss due to improper threshold selection. This step introduces multi-threshold feature quantization technology to divide the continuous similarity feature space into multiple discrete intervals, retaining more fine-grained difference information. Based on the principle of information entropy maximization, for different types of similarity feature spaces (such as path similarity, visual distance, semantic relevance, etc.), automatically determine the optimal set of feature quantization segmentation points. In specific implementation, the system first analyzes the distribution of each feature in the historical data, and then uses the dynamic programming algorithm to find the segmentation point combination that maximizes the information gain. For example, for the path similarity feature, the system may determine [0.3, 0.5, 0.7, 0.9] as the segmentation point set, dividing the continuous values into five intervals; for the visual distance feature, it may determine [50px, 100px, 200px, 500px] as the segmentation points. This multi-threshold quantization method retains the fine-grained difference information of the features, enabling the subsequent BDD to capture more complex conditional combination patterns, significantly improving the accuracy and robustness of similarity judgment, especially in dealing with boundary cases and outliers.

[0059] Step S443, generation of Boolean logic expressions. Based on domain knowledge and historical data, construct a series of Boolean logic expressions for judging node relationships. These expressions contain multi-condition combinations and are normalized using disjunctive normal form (DNF) or conjunctive normal form (CNF); Specifically, the problem of judging the similarity of merchant information is transformed into a Boolean logic solving problem, and an accurate judgment rule is constructed. Based on domain knowledge and historical data analysis, a series of Boolean logic expressions for judging node relationships are generated. These expressions adopt complex multi-condition combination forms and can describe various scenarios of the association between fields. For example, for judging whether two fields belong to the same merchant, the system may generate the expression "(XPath similarity > 0.7 AND visual distance < 100px) OR (XPath similarity > 0.5 AND visual distance < 50px AND contains the same merchant name keyword)". These expressions are automatically generated and optimized through symbolic reasoning methods, including steps such as rule induction, condition generalization, and redundancy elimination. All generated expressions are normalized using disjunctive normal form (DNF) or conjunctive normal form (CNF) for subsequent BDD construction. This Boolean logic-based representation method enables the system to accurately model complex conditions of field associations, especially suitable for handling complex rules such as "if feature A meets condition X and feature B meets condition Y, then the similarity is high", greatly improving the flexibility and accuracy of the system in processing various web page structures.

[0060] Step S444, BDD simplification and optimization, including: Based on the set of splitting points, adopt an optimization strategy of a shared subgraph structure to share and optimize the subgraph structure in the binary decision diagram, reducing the graph complexity; Based on the optimization of Boolean logic expressions, perform the minimization of the binary decision diagram to generate a similarity inference model; Apply BDD simplification algorithms (such as variable reordering, node merging, etc.) to convert complex Boolean expressions into minimized BDDs, adopt dynamic variable ordering techniques to automatically adjust the variable order in the BDD according to the influence degree of variables, and reduce redundant nodes through a shared subgraph structure; Specifically, the original Boolean expressions often contain redundant judgments and repeated calculations, which affect the system efficiency. This step applies a series of BDD optimization techniques to convert complex Boolean expressions into minimized BDDs with a streamlined structure and high computational efficiency. First, based on the previously determined set of splitting points, an optimization strategy using a shared subgraph structure is adopted to identify common substructures in the Boolean expressions and merge them into shared nodes, significantly reducing the number of nodes in the BDD. Then, the dynamic variable reordering technique is applied. Through a heuristic algorithm (such as swapping adjacent variables and evaluating the change in BDD size), the optimal variable order is determined to further reduce the BDD complexity. In addition, Boolean algebra rules such as idempotency and absorption law are used to simplify the structure of the Boolean expressions; the path compression technique is used to merge paths with the same decision results. The combined effect of these optimization techniques simplifies the BDD that might originally contain thousands of nodes into a streamlined structure with dozens or hundreds of nodes, while completely retaining the original logical function. The optimized BDD not only has high computational efficiency but also can clearly capture the dependency relationships between variables, providing an efficient and reliable decision-making basis for similarity fusion.

[0061] The optimization of binary decision diagrams is a key link in improving system performance, involving the collaborative application of multiple strategies. First is the variable reordering technique. The system evaluates the change in BDD size under different variable orders based on a heuristic algorithm. In specific implementation, the system uses the method of swapping adjacent variables and calculating the change in the number of BDD nodes to explore the optimal order, and a genetic algorithm is also introduced to support a wider range of searches. For large BDDs, the system adopts the window sliding method, where only a variable window of a fixed size is reordered each time, and then the window position is moved to gradually optimize the entire BDD structure. Second is the node merging technique. The system identifies functionally equivalent subgraph structures and merges them. During the merging process, a hash table is used to store the functional characteristics of the nodes, reducing the lookup complexity of equivalent nodes to O(1). In addition, the system implements the path compression technique. When the two child nodes of a node point to the same target, the node can be directly skipped, reducing redundant judgments in the BDD. When dealing with large-scale BDDs, the system adopts a hierarchical processing strategy, decomposing the entire BDD into multiple smaller sub-BDDs for independent optimization and then recombining them. For the identification of shared subgraph structures, the system uses an incremental hash algorithm to maintain a globally unique table, ensuring that the same functional subgraph is stored only once. This method not only saves storage space but also accelerates the BDD construction and evaluation processes. The system also introduces a variable grouping strategy, placing variables with strong correlations in adjacent positions in the BDD, further improving the compactness of the graph structure. Through the comprehensive application of these optimization techniques, the system can simplify the BDD that might originally contain thousands of nodes into a streamlined structure with dozens or hundreds of nodes, greatly improving the computational efficiency while maintaining the integrity of the logical function.

[0062] Step S445: BDD-based similarity reasoning. First, map various similarity features to the input variables of the BDD. Then, through the path traversal algorithm of the BDD, calculate all possible paths from the root node to the terminal nodes and their probabilities. For a given combination of feature inputs, the system can quickly locate the corresponding BDD path and obtain an accurate similarity evaluation result. Specifically, traditional linear or simple non-linear fusion methods are difficult to handle the complex interaction relationships between features. This step realizes highly reliable similarity reasoning based on the optimized BDD, which can accurately simulate the expert decision-making process. First, map various similarity features (after multi-threshold quantization) to the input variables of the BDD. Then, through the path traversal algorithm of the BDD, calculate all possible paths from the root node to the terminal node 1 (indicating relevance) and their probabilities. For a given combination of feature inputs, it can quickly locate the corresponding BDD path and obtain an accurate similarity evaluation result. Compared with traditional methods, this BDD-based reasoning has the advantages of computational efficiency (the time complexity is proportional to the size of the BDD, rather than being exponential with the number of variables) and decision consistency (the same input always produces the same output). More importantly, the BDD can handle the complex logical relationships and conditional dependencies between features, such as complex rules like "feature C has judgment significance only when features A and B both meet specific conditions", significantly improving the accuracy and interpretability of similarity reasoning.

[0063] Step S446: Uncertainty evaluation and decision-making. Introduce a BDD-based uncertainty evaluation mechanism, quantify the uncertainty of the decision by calculating the entropy value of possible results, analyze the key path distribution in the BDD, attach a confidence index to each similarity evaluation result, and dynamically adjust the decision threshold according to the confidence. Specifically, this step introduces a BDD-based uncertainty evaluation mechanism to quantify the confidence of the similarity judgment results. By analyzing the key path distribution in the BDD, calculate the entropy value of possible results to quantify the uncertainty of the decision. In specific implementation, for inputs close to the decision boundary (a small change in the values of multiple key variables may cause the result to reverse), the system will assign a lower confidence; for inputs located in the center of the decision region (the result remains the same when the values of multiple key variables change), a higher confidence will be assigned. Based on this confidence index, dynamically adjust the decision threshold - when the judgment certainty is high, use the conventional threshold; when the judgment certainty is low, increase the threshold requirement or trigger an alternative decision path. If the uncertainty exceeds the preset threshold, multiple possible grouping schemes can be selected, or additional feature information can be requested to improve the judgment accuracy. This mechanism significantly improves the robustness of the system in a complex web environment, effectively preventing incorrect grouping, especially performing excellently when dealing with boundary cases and abnormal samples.

[0064] Step S447, incremental BDD update, including: When the web page structure and content change, based on the dynamic update strategy, update the sub-graph structure in the binary decision diagram through local reconstruction technology, realize incremental graph structure adjustment, and incorporate new feature relationships; Design an incremental BDD update algorithm that can efficiently incorporate new feature variables and logical relationships without reconstructing the entire BDD. Through local reconstruction technology, only update the affected BDD sub-graph, and maintain the optimal structure of the BDD through variable reordering; Specifically, this step designs an efficient incremental BDD update algorithm, enabling the system to quickly adapt to changes without reconstructing the entire model. When a web page structure change or a new field association pattern appears, the system first locates the affected part of the BDD sub-graph. In the specific implementation, when the web page structure and content change, based on the dynamic update strategy, analyze the scope of the impact of the change on the existing BDD structure, and then through local reconstruction technology, only update the structure of the affected BDD sub-graph. For new variables or conditions that need to be added, insert the new variables through node splitting and edge reconnection operations while maintaining the existing variable order; for the logical relationships that need to be updated, implement conditional updates through local path reconstruction. After completing the local update, apply variable reordering and graph compression technology again to maintain the optimal structure of the BDD. This incremental update mechanism enables the system to adapt to web page changes at extremely low computational costs without retraining the entire model, significantly improving the timeliness and continuous adaptation ability of the system.

[0065] Step S448, multi-modal BDD fusion, implement multi-modal BDD fusion technology, construct dedicated BDDs for different information modalities, and then fuse them into a unified decision model through a hierarchical combination strategy. Design a collaborative enhancement mechanism between modalities to dynamically borrow information from other modalities to improve decision reliability when the judgment of a certain modality is uncertain.

[0066] Specifically, this step implements the multi-modal BDD fusion technology to integrate the judgment capabilities of different information sources. Special BDDs are constructed for different information modalities (such as text semantic modality, DOM structure modality, visual layout modality, etc.), each capturing the similarity judgment logic in that modality. Then, these single-modal BDDs are fused into a unified decision model through a hierarchical combination strategy. The specific implementation methods include: sequential combination (the output of modality A is used as the input condition for modality B), weighted voting (the judgment results of each modality BDD are weighted and summed), meta-decision (constructing a high-level BDD to determine how to combine the results of each modality), etc. A collaborative enhancement mechanism between modalities is also designed. When the judgment of a certain modality is uncertain, the weights of other high-certainty modalities can be dynamically increased. For example, in a page with scarce text content, the influence of the visual layout modality will be enhanced; in a page with a chaotic structure, the weight of the text semantic modality will be increased. This multi-modal fusion method can make full use of various types of information on the web page, maintain a high accuracy rate in a complex and changing web environment, and significantly improve the robustness and adaptability of the system.

[0067] As a specific embodiment of the present invention, step S5 can also adopt integrated tensor representation enhancement, including: Step S501, integrated tensor representation construction, including: Obtain link features through the web page DOM structure data stream of the new web page. Based on the link features, perform tensor decomposition operations to extract potential pattern representations; Organize the multi-dimensional features of web page links into a high-order tensor structure. Extract the text semantic features, URL structure features, context relationship features, historical click features, etc. of all links from the web page, construct a fourth-order or higher-order feature tensor, and then through tensor decomposition techniques such as Tucker decomposition or CP decomposition, extract potential feature patterns and association structures to generate a more compact and information-rich link representation; Step S502, local invertible mapping definition, including: Based on the potential pattern representation, perform self-organizing mapping clustering to construct a local invertible mapping; Adopt self-organizing mapping (SOM) technology to cluster link features, determine the natural partition of the feature space, and then construct an invertible mapping function within each partition through normalizing flow technology; Specifically, the core of the local invertible mapping technique is to divide the high-dimensional feature space into multiple local regions and construct a dedicated invertible transformation within each region. In implementation, the system first applies self-organizing mapping (SOM) for natural partitioning of the feature space. The SOM network adopts a two-dimensional grid topology of 20×20, where each grid point corresponds to a high-dimensional prototype vector. During the training process, the system randomly selects feature samples, finds the prototype vector closest to them (referred to as the best matching unit, BMU), and then updates the BMU and the prototype vectors within its neighborhood, moving them towards the sample direction. A larger neighborhood radius and learning rate are used in the initial stage and gradually reduced as the training progresses, achieving a rough-to-fine spatial partitioning. After the SOM training is completed, the system divides the feature space into multiple local regions, and each region contains samples mapped to the same grid point or adjacent grid points. Then, the system constructs a normalizing flow model within each local region to achieve invertible mapping. The normalizing flow consists of a combination of multiple simple invertible transformations, mainly adopting an architecture of alternating stacking of affine coupling layers and activation normalization layers. The affine coupling layer divides the input features into two parts, one part remains unchanged, and the other part is updated through the non-linear transformation of the previous part, ensuring the invertibility of the overall transformation. The system independently trains the normalizing flow model for each local region, maximizing the likelihood function using the samples within that region. To ensure smooth transition of the local mapping at the region boundaries, the system implements a boundary sample sharing mechanism, that is, the samples near the boundary are simultaneously used to train the mapping models of adjacent regions. In addition, the system also designs an adaptive partitioning adjustment strategy to dynamically subdivide or merge the feature space regions according to the mapping quality. This local invertible mapping method can capture the local structure of the feature space more accurately than global mapping and retain key information through invertibility, providing a more reliable feature representation for subsequent decision-making.

[0068] Step S503, Compact decision graph structure design, based on the local invertible mapping, construct a compact decision graph and output the link priority evaluation result; construct an initial decision tree based on the information gain ratio principle, and then convert it into a more compact graph structure through techniques such as edge merging, path compression, and isomorphic subgraph recognition; Step S504, Multi-scale feature hierarchical fusion, implement a multi-scale feature hierarchical fusion mechanism, connect feature representations of different scales through a compact decision graph, construct a feature pyramid including a micro layer (text vocabulary, URL pattern), a meso layer (page block, link context), and a macro layer (website structure, domain name weight), and then design a cross-layer feature transfer channel to achieve two-way flow and fusion of features at different scales; Step S505, Priority inference based on tensor network, implement a priority inference mechanism based on tensor network, map the compact decision graph to a tensor network structure, design special tensor contraction operations, and efficiently calculate the priority score of the link through a series of tensor product and contraction steps; Step S506, Local Reversible Calibration and Optimization: Utilize the characteristics of local reversible mapping to implement a high-precision priority calibration mechanism, monitor the actual effect of link collection (such as whether it contains target merchant information), map this feedback back to the feature space, and then accurately locate the feature regions and decision paths that cause errors through the inverse transformation of local reversible mapping. Step S507, Dynamic Decision Path Adaptation: Introduce a dynamic decision path adaptation mechanism to enable the compact decision graph to automatically adjust the decision logic according to environmental changes. Adopt an online learning framework to continuously monitor the effectiveness of different decision paths, and achieve a balance between exploration and exploitation through multi-armed bandit algorithms such as Thompson sampling, and dynamically adjust the activation probability of decision paths. Step S508, Interpretability Enhancement and Visualization: Design an interpretability enhancement and visualization mechanism to make the decision-making process of the compact decision graph transparent, convert complex tensor operations and decision paths into intuitive graphical representations, and display the key factors and decision bases for link evaluation through an interactive interface.

[0069] Referring to Figure 7 , the present invention also provides a system for intelligent collection and parsing of web page merchant information, including: A web page scraping module 100, configured to query and access a search engine according to a merchant brand name, parse the links in the search result page, construct an initial URL scraping queue, and use a browser automation tool to access the links in the initial URL scraping queue to obtain a web page DOM structure data stream. A field recognition module 200, configured to utilize the web page DOM structure data stream, combine key identification words and semantic associations of text paragraphs to identify merchant names, phone numbers, and addresses in the web page content, and generate a candidate field set. An address parsing module 300, configured to perform administrative division recognition and address component parsing on the address fields in the candidate field set, and output structured address data. A merchant grouping module 400, configured to calculate the path similarity, visual proximity, and semantic association degree between fields for the candidate field set, determine the inter-field association relationship through non-linear fusion, and generate structured merchant information. A page discovery module 500, configured to, when obtaining the web page DOM structure data stream of a new web page, perform text content and URL feature analysis on the links in the web page DOM structure data stream, identify target links containing merchant information, and add the target links to the initial URL scraping queue to achieve automated information collection.

[0070] The above description is only a preferred embodiment of the present invention and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the present invention is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by the mutual combination of the above features and the technical features (but not limited to) disclosed in the present invention.

Claims

1. A method for intelligently collecting and analyzing merchant information on a web page, characterized in that: include: According to the brand name of the merchant, query and access the search engine and parse the links in the search result page, build an initial URL crawling queue, use a browser automation tool to access the links in the initial URL crawling queue, and obtain the web page DOM structure data stream; Using the webpage DOM structure data stream, combined with the semantic association between key identifiers and text paragraphs, the webpage content is identified with the merchant name, telephone number and address, and a candidate field set is generated; Performing administrative division identification and address component analysis on the address field in the candidate field set, and outputting structured address data; For the candidate field set, the path similarity, visual proximity and semantic relevance between the fields are calculated, the correlation relationship between the fields is determined through nonlinear fusion, and structured merchant information is generated; When the webpage DOM structure data stream of the new webpage is obtained, the text content and URL feature analysis of the links in the webpage DOM structure data stream of the new webpage is performed, the target link containing the merchant information is identified, and the target link is added to the initial URL crawling queue to realize automatic information collection.

2. The method according to claim 1, characterized in that: The step of constructing the initial URL crawling queue includes: Based on the links in the search results page, domain name features, TLD type features and URL path features are extracted to calculate a multi-dimensional score; Based on the multi-dimensional scores, the links are filtered and prioritized to obtain an optimized initial URL crawling queue.

3. The method according to claim 1, characterized in that The method utilizes the webpage DOM structure data stream, combines the semantic association between key identifiers and text paragraphs, identifies the merchant name, telephone number and address of the webpage content, and generates a candidate field set, including: Based on the predefined keyword word library, identify the keyword words in the text nodes and mark the text area containing the target information; Based on the text area marked as containing target information, feature matching of the merchant name, telephone number and address is performed to generate candidate fields; Verification is performed on the candidate fields, and confidence scores are calculated according to the verification results, and the candidate field set is output.

4. The method according to claim 1, characterized in that: The step of performing administrative division identification and address component parsing on the address field in the candidate field set and outputting structured address data includes: Performing text cleaning and traditional-simplified conversion on the address field in the candidate field set to generate preprocessed address text; Based on the preset administrative division library, the provincial, municipal and district / county address information of the preprocessed address text is identified, and based on the trained deep learning model, the street and house number recognition is performed on the remaining text to output the structured address data.

5. The method according to claim 1, characterized in that: The step of calculating the path similarity, visual proximity and semantic association between the fields for the candidate field set, determining the association relationship between the fields through nonlinear fusion, and generating structured merchant information includes: Based on the candidate field set, calculate the XPath path depth difference of the candidate field and generate a path similarity feature; Based on the candidate field set, calculating the DOM element coordinate distance and generating a visual proximity feature; Based on the candidate field set, calculating the relevance of different text contents and generating semantic relevance; Based on the path similarity feature, the visual proximity feature and the semantic association, radial basis function fusion is performed to output the association relationship between fields.

6. The method according to claim 1, characterized in that The step of identifying the merchant name, telephone number and address of the webpage content is enhanced by a random order model, and includes: Based on the webpage DOM structure data stream, a node sequence relationship graph is constructed to generate a random sequence model; Based on the pre-trained feature extraction model, the multi-dimensional feature vector of the node is extracted to construct the generalized assignment space; Based on the generalized value space and utilizing the order relationship between nodes determined by the random order model, semi-supervised learning training is performed to generate a probability distribution of merchant information types.

7. The method according to claim 6, characterized in that Also includes: Based on the node multi-dimensional feature vector and the probability distribution of the merchant information type, spectral clustering analysis is performed to realize automatic grouping of multiple merchant information; Based on the feedback information of the collection results, online learning optimization is performed to update the model parameters of the random sequence model; Based on the random order model, graph convolutional network technology is used to integrate cross-page merchant information and output a complete merchant information set.

8. The method according to claim 1, characterized in that The step of determining the association relationship between fields by nonlinear fusion is enhanced by a binary decision diagram and includes: Based on the candidate field set, construct a binary decision graph structure of node relationships; Based on multi-threshold quantization technology, the similarity feature space is discretized; Based on Boolean logic expression optimization, binary decision graph minimization is performed to generate a similarity reasoning model.

9. The method according to claim 8, characterized in that The binary decision graph structure for constructing node relationships includes: Based on the principle of maximizing information entropy, for different types of similarity feature spaces, a set of segmentation points for feature quantization is determined; Based on the segmentation point set, an optimization strategy of a shared subgraph structure is adopted to perform shared optimization on the subgraph structure in the binary decision graph to reduce the graph complexity; When the structure and content of a web page change, based on a dynamic update strategy, the subgraph structure in the binary decision graph is updated through local reconstruction technology to achieve incremental graph structure adjustment and incorporate new feature relationships.

10. The method according to claim 1, characterized in that The step of analyzing the text content and URL features of the links in the webpage DOM structure data stream of the new webpage, which is enhanced by integrated tensor representation, includes: Acquire link features through the webpage DOM structure data stream of the new webpage, and perform tensor decomposition operations based on the link features to extract potential pattern representations; Based on the latent pattern representation, performing self-organizing map clustering to construct a local reversible map; Based on the local reversible mapping, a compact decision graph is constructed and a link priority evaluation result is output.

11. A system for intelligent collection and analysis of web merchant information, characterized in that: include: The webpage crawling module is used to query and access the search engine according to the merchant brand name and parse the links in the search result page, build an initial URL crawling queue, use the browser automation tool to access the links in the initial URL crawling queue, and obtain the webpage DOM structure data stream; A field identification module is used to identify the merchant name, telephone number and address of the web page content by using the web page DOM structure data stream and combining the semantic association between the key identifier and the text paragraph to generate a candidate field set; An address parsing module, used to perform administrative division identification and address component parsing on the address fields in the candidate field set, and output structured address data; A merchant grouping module, for calculating the path similarity, visual proximity and semantic association between the fields for the candidate field set, determining the association relationship between the fields through nonlinear fusion, and generating structured merchant information; The page discovery module is used to obtain the web page DOM structure data stream of a new web page, perform text content and URL feature analysis on the links in the web page DOM structure data stream, identify the target link containing merchant information, add the target link to the initial URL crawling queue, and realize automatic information collection.

Images