Webpage data analysis method, analysis device, electronic equipment and storage medium

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for parsing web page data, belonging to the field of data processing technology. It extracts structural information from unified web page files from multiple sources and adaptively calculates the target cluster number. Using structural similarity, it clusters massive amounts of web pages into web page clusters and records the correspondence between web pages, web page clusters, and parsing scripts in a front-end routing table. This overcomes the data omission problem caused by forcibly parsing heterogeneous web pages using the same set of rules in traditional methods, and also avoids the high cost of manually writing rules page by page. By using the routing table as a central scheduling mechanism, it can accurately retrieve parsing scripts specifically adapted to a certain structural feature within a cluster to perform batch parsing of web pages within the same cluster. It reduces the complex task of parsing heterogeneous web pages to multiple highly cohesive homogeneous web page concurrent parsing tasks, thereby greatly improving the accuracy, execution efficiency, and system scalability of web page parsing in large-scale data cleaning scenarios.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for parsing web page data. Background Technology

[0002] In the pre-training process of large-scale models, data cleaning is crucial, as its quality directly affects the completeness of the model's knowledge representation and the performance of downstream tasks. Web page data accounts for approximately 60% of the training data for large models, and the quality of web page parsing determines the model's cognitive and generalization abilities. Faced with massive, multi-source, and heterogeneous web page corpora, how to efficiently and accurately parse web pages into high-quality training data is a pressing problem that needs to be solved.

[0003] The webpage data parsing methods in the relevant technical solutions mainly include the following two types: The first method uses open-source rule-based tools for parsing, extracting the main content of web pages in batches using XPath, CSS selectors, and readability algorithms. This type of method is suitable for highly templated web pages and has high processing efficiency.

[0004] The second method involves manual page-by-page parsing. This involves manually analyzing the Document Object Model (DOM) nodes of the webpage and designing customized parsing rules for each website or specific page. This method offers higher parsing accuracy and can adapt well to complex webpage structures.

[0005] However, the above two analytical methods have the following shortcomings: Existing automated rule base tools are prone to missing key content when processing complex or heterogeneous web pages, making it impossible to guarantee data integrity. Furthermore, once the web page structure is updated or changed, the original static parsing rules become invalid, making it difficult to adapt to changes in the web page structure. This results in difficulties in updating rules and makes it impossible to support continuous cleaning of multi-source heterogeneous web page data.

[0006] The method of manually analyzing page by page and writing customized parsing rules is extremely labor-intensive and time-consuming, making it difficult to meet the high-concurrency processing requirements of the massive web page data needed for large-scale model pre-training, and thus not feasible for large-scale promotion. Summary of the Invention

[0007] This invention provides a method, apparatus, electronic device, and storage medium for parsing web page data. It addresses the shortcomings of existing technologies where static parsing rules become ineffective when the web page structure is updated or changed, making it difficult to adapt to changes in the web page structure, leading to difficulties in rule updates, inability to support continuous cleaning of multi-source heterogeneous web page data, and low efficiency of manual cleaning. The invention reduces the complex heterogeneous web page parsing task to multiple sets of highly cohesive homogeneous web page concurrent parsing tasks, thereby significantly improving the accuracy, execution efficiency, and system scalability of web page parsing in large-scale data cleaning scenarios.

[0008] This invention provides a method for parsing web page data, comprising the following steps.

[0009] Obtain multiple webpage files with a uniform format, wherein the multiple webpage files with a uniform format are determined based on multiple webpage data from different sources; Structural information is extracted and cluster analysis is performed on multiple web page files with the same format to obtain the target cluster number; Multiple web page files are divided into multiple web page clusters with a cluster number equal to the target cluster number, and a front-end routing table is established to describe the mapping relationship between the web page files, the web page clusters, and the parsing scripts within the clusters. Based on the front-end routing table, determine the parsing script within the target webpage cluster corresponding to the target webpage cluster and the target webpage file corresponding to the target webpage cluster; The target webpage files are parsed in batches based on the parsing script within the target cluster to obtain the parsing results; The target webpage cluster is any one of the multiple webpage clusters.

[0010] According to a webpage data parsing method provided by the present invention, the step of extracting structural information and performing cluster analysis on multiple webpage files with uniform format to obtain the target cluster number includes: Fingerprint extraction and vectorization representation are performed on each webpage file to obtain the webpage feature vector corresponding to each webpage file; Under different preset candidate cluster numbers, a preset clustering objective function is used to perform clustering analysis on the webpage feature vector corresponding to each webpage file to obtain clustering results under different preset candidate cluster numbers; Based on the clustering results, the average contour coefficient of all web page files is determined for each preset number of candidate clusters; The number of candidate clusters corresponding to the maximum average profile coefficient is taken as the number of target clusters.

[0011] According to a webpage data parsing method provided by the present invention, the step of extracting fingerprints and vectorizing each webpage file to obtain a webpage feature vector corresponding to each webpage file includes: For each tag in each of the web page files, a document object model path calculation is performed to obtain the path fingerprint corresponding to each tag; The path fingerprints corresponding to each tag are statistically analyzed to obtain the path fingerprint set corresponding to each web page file. Perform the following operations on each of the multiple webpage files to obtain the webpage feature vector corresponding to each webpage file: The sum of the frequencies of each path fingerprint appearing in the first web page file in the first path fingerprint set is used as the first quantization value of the first web page file. The first path fingerprint set is the path fingerprint set corresponding to the first web page file. The weighting value is determined based on the total number of web pages in the web page file and the number of web pages containing the target path fingerprint, wherein the target path fingerprint is the fingerprint of each path appearing in the first web page file; The product of the first quantized value and the weighted value is normalized to obtain the webpage feature vector corresponding to the first webpage file; The first webpage file is a selected webpage file from the plurality of webpage files.

[0012] According to a webpage data parsing method provided by the present invention, the step of determining the average silhouette coefficient of all webpage files for each preset number of candidate clusters based on the clustering results includes: Based on the clustering results, the contour coefficient of each webpage file is determined for each preset number of candidate clusters. The average contour coefficient of all web pages is calculated by averaging the total number of web pages and the contour coefficient of all web pages. The contour coefficient of each webpage file is determined based on the average distance between the first webpage file and the second webpage file, and the average distance between the first webpage file and the third webpage file. The second webpage file is a webpage file that is in the same webpage cluster as the first webpage file, and the third webpage file is a webpage file in the webpage cluster closest to the first webpage file.

[0013] According to a webpage data parsing method provided by the present invention, the step of determining the parsing script within the target webpage cluster corresponding to the target webpage cluster and the target webpage file corresponding to the target webpage cluster based on the front-end routing table further includes: The webpage files in the target webpage cluster are randomly shuffled and divided to obtain a development set and a verification set; Load one of the web page files in the development set as the selected web page file; The selected webpage file and the predefined prompt words are input into the large model that is called, and a temporary parsing script generated by the large model is obtained; Load one of the webpage files in the verification set as a test webpage file; Determine the first parsing result obtained by executing the temporary parsing script to parse the test webpage file; If the similarity between the first parsing result and the first annotation result is less than a preset similarity threshold, the first parsing script is updated based on the parsing logic determined by the temporary parsing script to obtain the updated first parsing script. The first parsing script is the parsing script that was successfully developed last time, and the first annotation result is the annotation result corresponding to the test webpage file. The cluster-level parsing script in the front-end routing table is updated based on the updated first parsing script.

[0014] According to a web page data parsing method provided by the present invention, if the temporary parsing script fails to execute or the cumulative number of updates where the similarity between the first parsing result and the first annotation result is less than a preset similarity threshold is greater than a preset number, the script is rolled back to the first parsing script.

[0015] According to a webpage data parsing method provided by the present invention, the step of obtaining multiple webpage files with a uniform format includes: Retrieve data from multiple web pages from different sources; Each of the webpage data is subjected to webpage identification and webpage management to obtain multiple initial webpage files; Perform one or more of the following processes on each of the initial webpage files to obtain multiple webpage files with a uniform format: Noise reduction, invalid webpage text removal, character encoding standardization, and file integrity checking and repair.

[0016] According to the present invention, a method for parsing webpage data is provided, wherein the language of the webpage data includes at least one of Chinese, English, and minority languages; and / or The web page data includes at least one of the following: news pages, blog pages, forum pages, Q&A pages, and product pages.

[0017] The present invention also provides a web page data parsing device, comprising the following modules: The acquisition module is used to acquire multiple web page files with a uniform format, wherein the multiple web page files with a uniform format are determined based on multiple web page data from different sources; The information extraction and cluster analysis module is used to extract structural information and perform cluster analysis on multiple web page files with the same format to obtain the target number of clusters; The processing module is used to divide multiple web page files into multiple web page clusters with a cluster number equal to the target cluster number, and to establish a front-end routing table describing the mapping relationship between the web page files, the web page clusters, and the parsing scripts within the clusters. The determination module is used to determine, based on the front-end routing table, the parsing script within the target webpage cluster corresponding to the target webpage cluster and the target webpage file corresponding to the target webpage cluster; The parsing module is used to perform batch parsing of the target webpage files based on the parsing script within the target cluster, and obtain the parsing results; The target webpage cluster is any one of the multiple webpage clusters.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the web page data parsing method described above.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the web page data parsing method described above.

[0020] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the web page data parsing method described above.

[0021] This invention provides a method, apparatus, electronic device, and storage medium for parsing web page data. It extracts structural information from unified web page files from multiple sources and adaptively calculates the target cluster number. Using structural similarity, it clusters massive amounts of web pages into web page clusters and records the correspondence between web pages, web page clusters, and parsing scripts in a front-end routing table. This data processing mechanism overcomes the data omission problem caused by forcibly parsing heterogeneous web pages using the same set of rules in traditional methods, and also avoids the high cost of manually writing rules page by page. By using the routing table as a central scheduling mechanism, it can accurately retrieve parsing scripts specifically adapted to a certain structural feature within a cluster to perform batch parsing of web pages within the same cluster. This reduces the complexity of heterogeneous web page parsing tasks to multiple sets of highly cohesive homogeneous web page concurrent parsing tasks, thereby greatly improving the accuracy, execution efficiency, and system scalability of web page parsing in large-scale data cleaning scenarios. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is one of the flowcharts illustrating the webpage data parsing method provided by the present invention.

[0024] Figure 2 This is a flowchart illustrating the process of extracting structural information and performing cluster analysis on multiple web page files with the same format to obtain the target number of clusters, as provided by the present invention.

[0025] Figure 3 This is a schematic diagram of the process provided by the present invention for extracting fingerprints and vectorizing each web page file to obtain the web page feature vector corresponding to each web page file.

[0026] Figure 4 This is a flowchart illustrating the process of determining the average contour coefficient of all web page files based on clustering results for each preset number of candidate clusters, as provided by the present invention.

[0027] Figure 5 This is a flowchart illustrating the process of determining the parsing script within the target webpage cluster and the target webpage file corresponding to the target webpage cluster based on the front-end routing table provided by the present invention.

[0028] Figure 6 This is a flowchart illustrating the process of obtaining multiple web page files with the same format provided by the present invention.

[0029] Figure 7 This is the second flowchart illustrating the webpage data parsing method provided by the present invention.

[0030] Figure 8 This is a schematic block diagram of the web page data parsing device provided by the present invention.

[0031] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0033] This invention provides a method for parsing web page data. The execution subject of this method can be an electronic device with data processing capabilities, such as a server, a cloud computing platform, or a data cleaning cluster for large model training, to realize an automated process from raw multi-source web page collection and structured clustering to high-quality data batch output.

[0034] The following is combined Figures 1 to 9 The present invention describes a method, apparatus, electronic device, and storage medium for parsing web page data.

[0035] Figure 1 This is one of the flowcharts illustrating the webpage data parsing method provided by the present invention, such as... Figure 1 As shown, the method includes the following: Step 101: Obtain multiple webpage files with the same format. These multiple webpage files with the same format are determined based on data from multiple webpages from different sources.

[0036] In this embodiment, multiple web page data from different sources are first acquired. These sources may include archived web page data sets acquired by various crawling tools, data files pulled by interfaces, and raw Hypertext Markup Language pages.

[0037] In some embodiments, the language of the web page data includes at least one of Chinese, English, and minority languages; and / or the web page data includes at least one of news pages, blog pages, forum pages, Q&A pages, and product pages.

[0038] In some embodiments, one or more webpages from the same source contain the same language. For example, the first and second webpages from the same source contain Chinese and English. Alternatively, the first and second webpages from the same source may contain the same minority language.

[0039] In some embodiments, one or more webpage data from different sources contain different languages. For example, the first and second webpage data from a first source contain Chinese and English. The third and fourth webpage data from a second source contain the same minority language.

[0040] Because the initial webpage data differs in format, encoding, and structure, it is necessary to standardize these webpage data from different sources, converting them into a consistent data format suitable for subsequent analysis, thus obtaining multiple uniformly formatted webpage files. These uniformly formatted webpage files provide a reliable and consistent input foundation for subsequent unified structured feature extraction.

[0041] For example, minority languages ​​are the languages ​​of one or more of the following ethnic groups: Mongolian, Tibetan, Uyghur, Kazakh, Kyrgyz, Yi, Dai, Lahu, Jingpo, Xibe, Russian, and Korean.

[0042] In some embodiments, the web page file is a HyperText Markup Language (HTML) file.

[0043] Step 102: Extract structural information and perform cluster analysis on multiple web page files with the same format to obtain the target cluster number.

[0044] After obtaining webpage files with a uniform format, structural information extraction is performed on each file to achieve automated and accurate parsing, given the significant differences in layout and node arrangement between different types of webpages. This extraction aims to extract information reflecting the layout patterns of the webpage, such as node hierarchy and tag composition, transforming the originally unstructured webpage layout into a data representation that can be calculated and compared. Based on this, cluster analysis is performed on the extracted structural information of all webpage files. By evaluating the similarity of structural information among different webpage files, the optimal number of categories for dividing this set of webpage files is determined; this optimal number is the target cluster number. This target cluster number objectively reflects the true types of structural templates contained in the multiple webpage files in the current batch, avoiding bias caused by manually pre-setting the number of categories.

[0045] Step 103: Divide multiple web page files into multiple web page clusters with the target number of clusters, and establish a front-end routing table that describes the mapping relationship between web page files, web page clusters, and parsing scripts within clusters.

[0046] Based on a defined number of target clusters, multiple web page files are categorized according to structural similarity, forming multiple web page clusters. This ensures that all web page files within the same cluster have a high degree of consistency in layout structure. To enable unified scheduling and management of the categorized web page files and subsequent parsing scripts, a front-end routing table is established. This front-end routing table is a structured relational mapping table that records the identifier of each web page file, the identifier of the web page cluster to which the web page file belongs, and the dedicated parsing script assigned to that web page cluster. In this way, the front-end routing table constructs a two-level scheduling mechanism from web page files to web page clusters, and then from web page clusters to specific parsing logic, ensuring that each web page can find the parsing rule most suitable for its own structure in subsequent processing.

[0047] Step 104: Based on the front-end routing table, determine the parsing script within the target webpage cluster corresponding to the target webpage cluster and the target webpage file corresponding to the target webpage cluster. The target webpage cluster is any one of multiple webpage clusters.

[0048] In the specific parsing and scheduling phase, for any one of the multiple webpage clusters, i.e., the target webpage cluster, the aforementioned front-end routing table is queried to retrieve all webpage files belonging to that target webpage cluster, which are then used as the target webpage files. Simultaneously, the parsing script bound to that target webpage cluster is retrieved and used as the parsing script within that target cluster. This parsing script within the target cluster is specifically generated based on the uniform structural characteristics within that target webpage cluster. Through the query action of the front-end routing table, the electronic device transforms the chaotic webpage parsing task into a structured processing task based on clusters.

[0049] Step 105: Perform batch parsing of the target webpage files based on the parsing script within the target cluster to obtain the parsing results.

[0050] After identifying the matching scripts and webpage files, the parsing script within the target cluster is invoked. Since target webpage files within the same cluster share isomorphic structural features, the parsing script within the target cluster can perfectly accommodate the internal node patterns of these files. This script is used in a high-concurrency manner to batch extract and parse the target webpage files, extracting key content such as the main text and titles, thus obtaining standardized parsing results. The processed parsing results can be directly delivered as high-quality corpus for downstream tasks such as large-scale model pre-training.

[0051] In this embodiment, structural information is extracted from unified webpage files from multiple sources, and the target cluster number is adaptively calculated. Structural similarity is used to cluster massive amounts of webpages into webpage clusters, and the correspondence between webpages, webpage clusters, and parsing scripts is recorded in the front-end routing table. This data processing mechanism overcomes the data omission problem caused by forcibly parsing heterogeneous webpages using the same set of rules in traditional methods, and also avoids the high cost of manually writing rules page by page. Using the routing table as a central scheduling mechanism, it can accurately retrieve parsing scripts specifically adapted to a certain structural feature within a cluster to perform batch parsing of webpages within the same cluster. The above feature processing steps are interconnected, reducing the complex heterogeneous webpage parsing task to multiple highly cohesive homogeneous webpage concurrent parsing tasks, thereby greatly improving the accuracy, execution efficiency, and system scalability of webpage parsing in large-scale data cleaning scenarios.

[0052] In some embodiments, such as Figure 2 As shown, structural information extraction and cluster analysis were performed on multiple web page files with uniform format to obtain the target cluster number, including: Step 201: Extract fingerprints and vectorize each webpage file to obtain the webpage feature vector corresponding to each webpage file.

[0053] Specifically, each webpage file, after being formatted and standardized, is parsed to extract identifying information reflecting its layout and hierarchical structure—a process known as fingerprint extraction. Fingerprint extraction aims to transform the complex nested node relationships within a webpage into a structured set of features. Since the extracted fingerprints are typically in text or categorical attribute form and cannot be directly used for mathematical distance calculations, they are further vectorized. By quantifying the distribution of each fingerprint within the webpage file, they are mapped to a continuous numerical multidimensional array, thus obtaining the webpage feature vector for each file. This step successfully transforms uncomputable, unstructured webpage documents into mathematical vectors that can be directly processed and compared by a computer.

[0054] Step 202: Under different preset candidate cluster numbers, use a preset clustering objective function to perform clustering analysis on the webpage feature vector corresponding to each webpage file to obtain the clustering results under different preset candidate cluster numbers.

[0055] To avoid bias caused by manually specifying the number of categories, a series of possible categories, i.e., different preset candidate cluster numbers, are pre-defined. For each preset candidate cluster number, a preset clustering objective function (e.g., an objective function aimed at minimizing the distance between data points within a cluster) is used to iteratively optimize and partition all webpage feature vectors obtained in the previous step, thereby obtaining the clustering result under the current preset candidate cluster number. By traversing all preset candidate cluster numbers, multiple different webpage grouping schemes can be generated, providing basic data for subsequent search for the optimal solution.

[0056] Step 203: Based on the clustering results, determine the average silhouette coefficient of all web page files for each preset number of candidate clusters.

[0057] After obtaining multiple clustering results, it is necessary to objectively evaluate the rationality of each clustering scheme. For each preset number of candidate clusters, the silhouette coefficient of each webpage file under the current partition is calculated. This silhouette coefficient is used to comprehensively measure the degree of cohesion between the webpage file and other webpages within the same cluster, as well as its degree of separation from webpages in different clusters. Subsequently, the average silhouette coefficient of all webpage files is calculated to obtain the average silhouette coefficient corresponding to the preset number of candidate clusters. This average silhouette coefficient, as a quantitative indicator, intuitively reflects the quality of the overall clustering structure under the current number of candidate clusters.

[0058] Step 204: Use the preset number of candidate clusters corresponding to the maximum average profile coefficient as the target number of clusters.

[0059] The average silhouette coefficients corresponding to the calculated number of preset candidate clusters are compared. Since a larger average silhouette coefficient means a higher structural similarity within each webpage cluster in the current clustering scheme and a clearer boundary between different clusters, the maximum average silhouette coefficient is selected, and the preset candidate cluster number corresponding to this maximum value is determined as the final target cluster number.

[0060] In this embodiment, by extracting fingerprints and vectorizing web page files, the unstructured web page layout is transformed into web page feature vectors that can be used for distance measurement. Based on this, clustering is performed under multiple preset candidate cluster numbers, and the intra-cluster compactness and inter-cluster separation of the clustering results are quantitatively evaluated using the average silhouette coefficient. Finally, the candidate cluster number corresponding to the largest average silhouette coefficient is taken as the target cluster number. In this process, the optimal number of classifications for the current batch of massive web pages can be adaptively searched entirely based on the mathematical distribution law of the data itself, completely eliminating the subjective interference of manually preset cluster numbers. This ensures that the divided web page clusters have extremely high structural isomorphism, providing an extremely reliable data classification foundation for subsequent accurate routing and efficient execution of automated scripts.

[0061] In some embodiments, such as Figure 3 As shown, fingerprint extraction and vectorization are performed on each webpage file to obtain the webpage feature vector corresponding to each webpage file, including: Step 301: Perform document object model path calculation on each tag in each web page file to obtain the path fingerprint corresponding to each tag.

[0062] The Document Object Model (DOM) is the programming interface for HTML documents, representing web pages as a tree structure. Here, it iterates through all node tags in each web page file, calculating the hierarchical nesting path from the root node (e.g., the body tag) to the current specific tag (e.g., a paragraph, heading, or list tag), and extracting the sequence of all ancestor tags traversed along that path.

[0063] For example, calculating the document object model path of a certain tag can be represented as the path depth starting from the body tag, passing through the first-level ancestor tags, the second-level ancestor tags, and finally down to the target tag. This specific hierarchical path sequence is the path fingerprint corresponding to that tag, which can accurately characterize the macroscopic position and layout hierarchy of the node in the entire webpage structure tree.

[0064] Step 302: Statistically analyze the path fingerprints corresponding to each tag to obtain the set of path fingerprints corresponding to each web page file.

[0065] After calculating the path fingerprints of all tags within the webpage file, these path fingerprints are summarized and deduplicated. The fingerprints containing path information for all text nodes and structural nodes of the webpage are combined to form the path fingerprint set corresponding to the webpage file. This path fingerprint set comprehensively characterizes the skeletal features of the webpage file, providing a complete data space for subsequent construction of quantitative features.

[0066] Step 303: Perform the following operations on multiple webpage files respectively to obtain the webpage feature vector corresponding to each webpage file: Take the sum of the frequencies of each path fingerprint appearing in the first webpage file in the first path fingerprint set as the first quantization value of the first webpage file, where the first path fingerprint set is the path fingerprint set corresponding to the first webpage file; Determine the weighting value based on the total number of webpages in the webpage file and the number of webpages containing the target path fingerprint, where the target path fingerprint is each path fingerprint appearing in the first webpage file; Normalize the product of the first quantization value and the weighting value to obtain the webpage feature vector corresponding to the first webpage file; wherein, the first webpage file is a selected webpage file among multiple webpage files.

[0067] Specifically, for any selected first webpage file among multiple webpage files, the total number of times each target path fingerprint appears within that first webpage file in the corresponding first path fingerprint set is counted (i.e., term frequency feature). This frequency data reflects the density of a specific structure (such as a particular list or text paragraph nesting pattern) in that single webpage, and is used as the first quantification value of the first webpage file. This first quantification value can initially highlight the local structural preferences of the webpage.

[0068] To eliminate the interference of common structures (such as simple nested divs) prevalent across all web pages on feature distinguishability, the concept of inverse document frequency (IVF) is introduced. Specifically, the total number of web pages currently being processed is obtained, and the number of web pages containing the aforementioned target path fingerprint is counted. A weighted value is calculated by determining the ratio between the total number of web pages and the number of web pages containing the target path fingerprint (e.g., through logarithmic operations). If a target path fingerprint appears in very few web pages, it indicates that it is a unique and distinctive structure of this type of web page, and the weighted value will be higher; conversely, the weighted value will be lower.

[0069] Subsequently, the first quantized value reflecting local structural features is multiplied by the weighted value reflecting global discriminative power to obtain the comprehensive weight value of the target path fingerprint in the first webpage file. To eliminate the calculation bias caused by the varying lengths of content in different webpage files (leading to large differences in the absolute number of structural tags), the calculated comprehensive weight value is further normalized (e.g., L2 norm normalization) to ensure that the feature representations of each webpage are on the same numerical order of magnitude, ultimately generating the high-dimensional webpage feature vector corresponding to the first webpage file.

[0070] In this embodiment, a path fingerprint is obtained by calculating the document object model path for each tag, transforming the abstract webpage layout into a clear path sequence. Then, a first quantified value is obtained by statistically analyzing the fingerprint frequency, and a weighted value is calculated by combining the total number of global webpages and the number of webpages containing a specific fingerprint. This applies the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm from natural language processing to the extraction of structured features from webpages. This assigns higher weight to rare structures that truly reflect the unique layout patterns of a particular type of webpage, while weakening commonly existing interfering structures. Finally, normalization processing eliminates the feature scale differences caused by webpage length. In this process, the originally unstructured webpage skeleton, which could not be directly compared, is accurately transformed into a computable and robust structural feature vector in a high-dimensional space, thereby greatly improving the accuracy and robustness of subsequent webpage clustering analysis.

[0071] In some embodiments, such as Figure 4 As shown, based on the clustering results, the average silhouette coefficient of all web page files is determined for each preset number of candidate clusters, including: Step 401: Based on the clustering results, determine the contour coefficient of each web page file for each preset number of candidate clusters.

[0072] The contour coefficient of each webpage file is determined based on the average distance between the first webpage file and the second webpage file, and the average distance between the first webpage file and the third webpage file. The second webpage file is the webpage file that is in the same webpage cluster as the first webpage file, and the third webpage file is the webpage file in the webpage cluster closest to the first webpage file.

[0073] In this process, the silhouette coefficient is a data metric used to evaluate the intra-cluster tightness and inter-cluster separation of a single webpage sample in the current clustering result. For any selected first webpage file, the electronic device first calculates the distance between the first webpage file and all second webpage files (i.e., all other webpage files that are classified in the same webpage cluster as the first webpage file) in the feature vector space under the current preset number of candidate clusters. These distances are then averaged to obtain the first average distance, which reflects the intra-cluster tightness. Subsequently, the electronic device calculates the distance between the first webpage file and all third webpage files (i.e., webpage files in different clusters that are closest to the first webpage file), and these distances are averaged to obtain the second average distance, which reflects the inter-cluster separation. Based on the first and second average distances, the electronic device calculates the difference between them and uses the maximum of the two as the denominator for normalization, thereby obtaining the silhouette coefficient of the first webpage file under the current preset number of candidate clusters. Through this method, the electronic device can accurately quantify the rationality of each webpage file under the current clustering division.

[0074] Step 402: Calculate the average profile coefficient of all web pages by averaging the total number of web pages and the profile coefficient of all web pages.

[0075] After obtaining the silhouette coefficient of each webpage file, they are summarized and statistically analyzed. Specifically, the silhouette coefficients of all webpage files under the current preset number of candidate clusters are summed and divided by the total number of webpages to calculate the average silhouette coefficient of all webpage files. This average silhouette coefficient is a macro-evaluation indicator that can reflect the overall clustering quality and structural partitioning of the entire webpage set under the current preset number of candidate clusters.

[0076] In this embodiment, by introducing the calculation logic of the silhouette coefficient, the intra-cluster density is characterized by the average distance between the first webpage file and the second webpage file in the same cluster, while the inter-cluster separation is characterized by the average distance to the nearest third webpage file in a different cluster. The individual silhouette coefficients are then summed and averaged to obtain the overall average silhouette coefficient. This process transforms the previously difficult-to-measure webpage structure clustering effect into a quantifiable and comparable numerical indicator. It allows for adaptive evaluation of the clustering quality at each candidate cluster number from a data perspective through a mathematical evaluation mechanism, completely eliminating the subjectivity and limitations of manually pre-setting the cluster number. This provides the most reliable clustering basis for subsequently generating parsing scripts adapted to specific webpage clusters.

[0077] In some embodiments, such as Figure 5 As shown, based on the front-end routing table, the parsing script within the target webpage cluster corresponding to the target webpage cluster and the target webpage file corresponding to the target webpage cluster are determined, and the process also includes: Step 501: Randomly shuffle and divide the webpage files in the target webpage cluster to obtain the development set and the verification set.

[0078] Because all webpage files within the target webpage cluster share a high degree of similarity in their macroscopic structure, to achieve automatic generation and objective evaluation of the parsing script, all webpage files within the target webpage cluster are first acquired, and their order is randomly shuffled to eliminate potential feature distribution biases caused by the initial data arrangement. After shuffling, these webpage files are divided into a development set and a validation set according to a preset proportion or quantity requirement. The development set serves as the input sample set for generating the parsing logic, guiding the coding of the large model; the validation set serves as an independent test sample set, used to objectively evaluate the accuracy of the generated parsing logic. For example, 200 webpage files within the target webpage cluster can be randomly shuffled, with 40 selected as the development set and the remaining files used as the validation set.

[0079] Step 502: Load a webpage file from the development set as the selected webpage file.

[0080] When entering the closed-loop iterative processing flow for script generation, the data content of any webpage file from the development set obtained above is first read and loaded into memory. This currently loaded webpage file for code generation is the selected webpage file. During the loading process, the selected webpage file can also be compressed and cleaned, for example, by removing whitespace characters, redundant invisible tags, or noise elements from the webpage data, thereby extracting more concise webpage text data as input for subsequent large model processing.

[0081] Step 503: Input the selected web page file and the predefined prompt words into the called large model to obtain the temporary parsing script generated by the large model.

[0082] The system invokes a large model with code generation and natural language understanding capabilities, either locally or in the cloud. To accurately guide the model to output extraction code that meets business requirements, pre-configured prompts are used. These prompts contain specific instructions for the parsing task, descriptions of the expected extracted fields, and constraints such as the code output format. The content of the selected webpage file is concatenated with these predefined prompts and sent as complete contextual input to the invoked large model. The large model performs inference based on the structural features of the input and generates code to extract key information from the currently selected webpage file. The code output by the large model in this current round is the temporary parsing script. This temporary parsing script represents the initial parsing logic defined for the current structure of the selected webpage file.

[0083] Step 504: Load a webpage file from the verification set as a test webpage file.

[0084] To verify the accuracy and generalization ability of the temporary parsing script just generated by the large model, another webpage file was read from the validation set and entered into the execution environment, serving as the test webpage file. This test webpage file has a corresponding standard parsing reference text pre-annotated by the large model or manually, which is the first annotation result in the following text, used to provide a benchmark for subsequent comparative evaluation.

[0085] Step 505: Determine the first parsing result obtained by executing the temporary parsing script to parse the test webpage file.

[0086] A temporary parsing script runs in a controlled code execution environment, processing the test webpage file as its input data. After execution, the script extracts relevant fields from the test webpage file according to its internal logic, such as body paragraphs and specific title text. The set of data actually extracted by the temporary parsing script constitutes the first parsing result.

[0087] Step 506: If the similarity between the first parsing result and the first annotation result is less than the preset similarity threshold, the first parsing script is updated based on the parsing logic determined by the temporary parsing script to obtain the updated first parsing script. The first parsing script is the parsing script that was successfully developed last time, and the first annotation result is the annotation result corresponding to the test web page file.

[0088] The system retrieves the first annotation result corresponding to the test webpage file and calculates the similarity between this first annotation result and the first parsing result output by the aforementioned script. This similarity calculation can employ algorithms such as the longest common subsequence to quantify the degree of consistency between the two text contents. A preset similarity threshold is a numerical limit set by the system to determine whether the parsing logic meets the acceptable standard. When the calculated similarity is less than this preset similarity threshold, it indicates that the current temporary parsing logic still has feature omissions or extraction errors when processing webpages of the same cluster, thus triggering an incremental code update mechanism. At this time, the newly added valid parsing logic in the temporary parsing script is extracted and superimposed onto the first parsing script. The first parsing script refers to the historical parsing script that has been successfully verified and recorded in the state machine in processing the previous sample or previous iteration. Through this incremental update method of retaining historical logic and integrating newly added temporary parsing logic, a more complete and updated first parsing script is obtained.

[0089] Step 507: Update the cluster-specific parsing script in the front-end routing table based on the updated first parsing script.

[0090] After completing the iterative and logical evaluation of webpage files in the development and validation sets, and forming the final stable and high-precision parsing code, the first parsing script, which has been iteratively updated and continuously updated, is registered and overwritten into the previously established front-end routing table. That is, this script is identified as the parsing script within the target cluster corresponding to the target webpage cluster, for unified use during subsequent large-scale webpage parsing.

[0091] In this embodiment, a mechanism is introduced during the process of determining the parsing script to shuffle the web pages within a cluster and divide them into a development set and a validation set. A large model combined with prompt words is used to generate a temporary parsing script for the development set file. The similarity between the first parsing result and the first annotation result output by this script on the validation test file is compared. If the similarity is insufficient, new logic is extracted and incrementally updated to update the previously successful first parsing script. In this process, the traditional static rule writing is transformed into a closed-loop iterative process based on similarity data feedback. Numerical comparison triggers the fusion of historical successful logic and new logic, ensuring that the generated script can adapt to subtle structural variations within the same cluster of web pages while avoiding the problem of directly rewriting code and destroying historical success experience. This results in the cluster parsing script ultimately written into the front-end routing table having extremely high extraction accuracy and fault tolerance.

[0092] In some embodiments, if the temporary parsing script fails to execute or the cumulative number of updates where the similarity between the first parsing result and the first annotation result is less than a preset similarity threshold exceeds a preset number, the script is rolled back to the first parsing script.

[0093] During the process of generating parsing scripts using a development set-driven closed-loop iteration, various abnormal situations may occur during the execution of the temporary parsing scripts generated from the large model due to the inherent uncertainty of these scripts. Specifically, the status of each execution process will be monitored. If the temporary parsing script fails to execute, i.e., if a code-level error or abnormal crash occurs during feature extraction or code execution of the test webpage file, resulting in the inability to output parsing results correctly; or, although the script executes successfully and outputs the first parsing result, the similarity between the first parsing result and the corresponding first annotation result (i.e., the reference standard text) fails to reach the preset similarity threshold after comparison calculation.

[0094] When any of the above situations occur, an incremental code update operation is typically triggered to attempt to fix or regenerate the temporary parsing script. However, to prevent large models from getting stuck in an endless modification loop due to complex webpage structures, or from breaking the already effective parsing logic due to excessive modifications, the update behavior for the current test webpage file is counted to obtain a cumulative update count. The electronic device compares this cumulative update count with a preset count (for example, based on the actual business scenario, this preset count can be set to 6 times).

[0095] Once the cumulative update count exceeds the preset number, it indicates that for the current specific test webpage file, the large model has failed to generate valid and compliant incremental parsing logic after multiple attempts. At this point, a script rollback mechanism is triggered and executed, revoking all unqualified temporary parsing scripts and their update logic generated in the current round, and safely rolling back the parsing script state to the first parsing script. This first parsing script is the verified, stable version of the successful script saved when processing the previous successful webpage file. After the rollback, the current failure state can be recorded, and the next webpage file can be loaded from the breakpoint for subsequent processing.

[0096] In this embodiment, by introducing a cumulative update count judgment logic triggered by both execution status and similarity evaluation, and executing script rollback when the count limit is exceeded, the system can promptly block the continued evolution of low-quality temporary parsing logic when encountering extremely abnormal or difficult-to-parse webpage structures, and safely restore the first parsing script of the previous stable version. This processing mechanism effectively ensures that existing and historical parsing logic is not damaged or contaminated, avoids the collapse of the entire cluster parsing script due to parsing anomalies in a single webpage file, and significantly improves the data processing stability and incremental generation fault tolerance of automated parsing scripts when handling massive heterogeneous webpages.

[0097] In some embodiments, such as Figure 6 As shown, multiple webpage files with a uniform format are obtained, including: Step 601: Obtain data from multiple web pages from different sources.

[0098] In some embodiments, web page data from different sources is accessed and obtained through methods such as web crawling, API calls, or local reading. This diverse web page data may include WARC files extracted from web archives, JSON data crawled via APIs, and directly crawled raw HTML files. The aforementioned web page data covers various page types, including news, blogs, forums, Q&A pages, and product pages, and supports multilingual data, including Chinese, English, and minority languages.

[0099] Step 602: Perform webpage identification and management on each webpage data to obtain multiple initial webpage files.

[0100] Because the web page data comes from diverse sources and has varying formats, it is necessary to automatically identify the data type of each web page and incorporate it into the system for unified management. During the identification and management process, data of various formats (such as text fields in JSON files or page records in WARC files) are extracted and uniformly stored and converted into basic initial web page files (such as HTML files that have not been deeply cleaned), providing a unified operation object for subsequent standardized cleaning.

[0101] Step 603: Perform one or more of the following processes on each initial webpage file to obtain multiple webpage files with uniform format: noise reduction, invalid webpage text removal, character encoding standardization, and file integrity check and repair.

[0102] To ensure the accuracy of subsequent document object model structure clustering, each initial webpage file undergoes deep cleaning and normalization. Specifically, the following data processing operations can be performed: First, noise reduction processing is performed. The code structure of the initial webpage file is parsed, and invalid tag elements unrelated to the core content of the webpage are identified and removed. For example, noisy data such as script (script language tag), style (style sheet tag), and iframe (inline frame tag) are removed, thereby extracting the clean structural content.

[0103] Secondly, invalid webpage text is removed. The length of the valid text on the webpage after the above denoising is calculated. In practice, the total length of the characters within the tags of each text paragraph on the webpage can be used as the total length of the valid text. Subsequently, the electronic device compares this total length of the valid text with a preset length threshold (e.g., 50 characters). If the total length is less than the preset length threshold, the initial webpage file is determined to be an invalid page with too short content and is removed, thus avoiding the introduction of data without substantial content into the subsequent large model training stage.

[0104] Third, character encoding standardization is performed. To address the encoding issues caused by multiple sources and languages, the content of the initial webpage files is uniformly converted and forcibly encoded in UTF-8 format. At the same time, HTML entities and special characters in the page are escaped or processed to ensure that the content of multi-source webpages is consistent and free of garbled characters in subsequent processing.

[0105] Fourth, perform file integrity checks and repairs. Check the tag closures and HTML structure integrity of the initial webpage file, automatically identify missing tags (such as missing closing tags) or corrupted nodes, and perform necessary repairs and closures according to webpage syntax rules.

[0106] By performing one or more of the above processing operations, a high-quality and standardized web page file with a uniform format is finally generated, which serves as the input basis for subsequent extraction of DOM fingerprints.

[0107] In this embodiment, by introducing a multi-source webpage data acquisition and identification management mechanism, combined with denoising, invalid webpage text removal, character encoding standardization, and file integrity checks and repairs, this multi-dimensional data processing mechanism filters out non-textual noise nodes and meaningless short text pages in the early data input stage, and solves the problems of inconsistent encoding and incomplete nodes in multi-source webpages. The aforementioned denoising, repair, and format standardization actions work together to transform the originally messy webpages, containing a large amount of dirty data and incomplete structures, into standardized webpage files with unified encoding, no garbled characters, and complete tag structures. This significantly reduces the interference of useless nodes on subsequent structural feature extraction, directly improving the accuracy and reliability of subsequent DOM structure extraction and cluster analysis steps.

[0108] In some embodiments, the web page data parsing method mentioned in this invention is explained by dividing it into a preprocessing module, a front-end routing module, an agent automated parsing module, and a parsing result batch processing and output module.

[0109] like Figure 7 As shown, the preprocessing module includes: 1.1 Obtaining webpage data source; 1.2 Web page data preprocessing.

[0110] By employing a unified cleaning and standardization strategy for multi-source web pages, high-quality preprocessing is performed on web pages from different sources and in different formats, providing reliable input for subsequent DOM analysis and clustering.

[0111] Specifically, in the webpage data source acquisition section, data from multiple sources are accessed, including WARC files, JSON scraping, and raw HTML, covering news, blogs, forums, Q&A pages, and product pages. Multilingual support is provided, including Chinese, English, and minority languages. After automatic identification and unified management, all data sources are stored as standardized HTML files, providing the foundational data for subsequent processing.

[0112] The web page data preprocessing involves uniformly cleaning and standardizing each HTML file. The process includes:

[0113] (1) Noise reduction and text length determination: After removing invalid tags such as script, style, and iframe, calculate the length of the effective text on the webpage. The formula is as follows: (Formula 1); In the formula: The total length of valid text on the webpage (number of characters). It is then determined to be an invalid page with content that is too short. This is the preset length threshold, which is usually set to 50 characters. For the first Each text paragraph label; Total number of paragraphs; For paragraph The character length.

[0114] (2) Character encoding standardization: The web page content is uniformly encoded as UTF-8, and HTML entities and special characters are processed to ensure that the content of multi-source web pages is consistent and free of garbled characters.

[0115] (3) HTML integrity check and repair: The HTML structure is checked for integrity, missing or corrupted tags are identified, and necessary repairs are performed. Upon completion, a standardized HTML file is generated, providing high-quality input for subsequent DOM fingerprint extraction, vectorization, and cluster analysis.

[0116] like Figure 7 As shown, the front-end routing module includes: 2.1 DOM fingerprint extraction; 2.2 TF-IDF Vectorization; 2.3 Adaptive Optimal Cluster Number Search; 2.4. Front-end routing table generation.

[0117] By employing DOM structure fingerprinting and adaptive clustering strategies, webpage structural information is encoded into computable features, enabling automatic clustering and front-end routing table generation for webpages. This provides high-precision input for subsequent intra-cluster self-evolutionary parsing. This method can uniformly represent the structural features of different webpages and quantify the structural similarity between them.

[0118] 2.1 DOM fingerprint extraction.

[0119] For the standardized HTML files after the webpage data preprocessing in section 1.2, the system calculates the DOM path for each webpage and generates a corresponding DOM fingerprint, using the following formula: (Formula 2); (Formula 3); In the formula: For tags DOM path fingerprint; For HTML < Label; For the first to the second on the path Layer ancestor tag; For path depth; This represents the upper limit of the path depth. For webpages A collection of DOM fingerprints, containing the paths of all text nodes on the webpage; This refers to all tags on the preprocessed webpage.

[0120] The above methods can be used to extract structural information such as paragraphs, titles, and lists from web pages, providing basic input for subsequent vectorization and clustering analysis.

[0121] 2.2 TF-IDF vectorization.

[0122] Based on the webpage obtained in section 2.1 DOM fingerprint set First, the frequency of each path fingerprint in the set is counted. Then, TF-IDF weighting is performed based on the distribution of fingerprints across all webpage samples to construct the structural feature vector of the webpage, providing a unified input for subsequent clustering analysis. The specific formula is as follows: (Formula 4); (Formula 5); (Formula 6); In the formula: For path On the webpage Frequency of occurrence in; For path Number of occurrences; Total number of web pages; For included paths The number of web pages; For path The weighted TF-IDF value; This is the normalized webpage feature vector (L2 norm). That is, the one mentioned above .

[0123] 2.3 Adaptive Optimal Cluster Number Search.

[0124] In section 2.2, we obtain the webpage structure feature vector, i.e., the webpage feature vector. Subsequently, cluster analysis is performed on all webpage samples. To avoid the subjectivity of manually pre-setting the number of clusters, this invention introduces a silhouette coefficient evaluation mechanism to adaptively search the clustering results under different candidate cluster numbers, thereby determining the optimal number of clusters. The specific formula is as follows: (Formula 7); In the formula: The silhouette coefficient is the clustering objective function. The number of candidate clusters; For the first A cluster; This is the normalized webpage feature vector obtained from Formula 6; For the first The center vector of each cluster; For webpages Webpage feature vector With the Cluster center vectors The square of the Euclidean distance between them.

[0125] (Formula 8); (Formula 9); In the formula: For webpages The profile coefficient; For webpages The average distance to other web pages in the same cluster; For webpages The average distance to the nearest heterogeneous webpage; This represents the optimal number of clusters, which is also the target number of clusters. The number of candidate clusters; This represents the total number of web pages.

[0126] Formula 7 is used to characterize the clustering objective given a certain number of clusters; Formula 8 is used to evaluate the intra-cluster compactness and inter-cluster separation of a single webpage sample in the current clustering result; Formula 9 adaptively determines the optimal number of clusters by maximizing the average silhouette coefficient of all webpage samples. This provides a reliable basis for the subsequent generation of the front-end routing table.

[0127] 2.4. Front-end routing table generation.

[0128] After determining the optimal number of clusters in section 2.3, each webpage is assigned to a corresponding cluster based on the clustering results, and a front-end routing table is generated to establish the mapping relationship between webpages, clusters, and parsing scripts. Specifically, the system first records the cluster identifier of each webpage, then aggregates webpage samples according to the cluster level to form a corresponding structured page set; subsequently, it pre-registers an empty parsing script for each cluster, enabling webpages within the same cluster to call a unified script for parsing processing.

[0129] The above method enables two-level routing mapping from web pages to clusters and from clusters to scripts, providing a unified scheduling entry point for subsequent intra-cluster self-evolutionary parsing and batch data processing, while improving the consistency and processing efficiency of isomorphic web page parsing.

[0130] The agent automated parsing module includes: 3.1 Generation of development and test set tags; 3.2 LangGraph State Machine Management; 3.3 Similarity Feedback and Incremental Optimization; 3.4 Failure rollback and breakpoint resume.

[0131] This module achieves automatic generation, incremental optimization, and reliable execution of parsing scripts through in-cluster self-evolution of the Agent. It receives the parsing scripts generated by the front-end routing module. The cluster of web pages is guided by the development set to generate parsing scripts. Combined with similarity feedback, incremental updates and state management, it can achieve high-precision automated parsing of web pages within the cluster, while also supporting failure rollback and breakpoint resume.

[0132] 3.1 Generation of development set and test set tags.

[0133] This invention provides a high-quality reference standard for incremental agent parsing by uniformly annotating the development and test sets. The specific operation is as follows: The HTML data set within the cluster is randomly shuffled and then divided into a development set and a validation set: (Formula 10); (Formula 11); In the formula: To develop an HTML collection for generating parsing scripts; This is a collection of HTML files used to evaluate the accuracy of script parsing. For the number of development sets (e.g., 40); This represents the total number of samples within the cluster (e.g., 200).

[0134] For each HTML file in the development and validation sets, the large model is called in conjunction with the designed prompt to generate parsed labels: (Formula 12); In the formula: For webpages The analysis reference results; , , This is the parsing result for the corresponding tag field.

[0135] 3.2 LangGraph State Machine Management.

[0136] This invention is achieved through The driver uses a closed-loop iteration to automatically generate incremental parsing scripts. The state machine iteratively processes `dev`, with the following loop logic: Load→Coder→Executor→Save→Load (Formula 13); It includes four stages: (1) Load: Loading A single HTML file is processed and compressed (whitespace, redundant tags, and noisy elements are removed) to generate... As input for script generation.

[0137] (2) Coder: Combined with compression And a predefined prompt, calling the large model to generate a temporary parsing script. The new script is based on the previous successful script. Incremental development: (Formula 14); In the formula: This is the updated script; This is the previous successful script version; This is the newly added parsing logic extracted from the temporary script. It only adds new functionality and does not modify existing logic.

[0138] (3) Executor: Executes temporary scripts and extracts parsed results. For use in similarity assessment.

[0139] (4) Save (Status Record): Records the current parsing script and similarity.

[0140] 3.3 Similarity feedback and incremental optimization.

[0141] In each round of the HTML development loop, a temporary parsing script is executed. Obtain the analysis results References to the development set's dev annotations Text-level similarity is calculated. The longest common subsequence (LCS) method is used for similarity. (Formula 15); In the formula: For webpages The parsing similarity (values ​​0-1); This is the result of parsing a temporary script; Reference label text for development set annotation; Let be the length of the longest common subsequence of the two texts. If (Threshold) triggers incremental code updates, generating new parsing logic. And perform incremental updates based on Formula 14.

[0142] 3.4 Failure rollback and breakpoint resume.

[0143] To ensure the stability of incremental parsing driven by the development set, this invention introduces rollback and breakpoint resumption mechanisms. When a temporary parsing script... Execution failed or webpage parsing similarity Below the threshold If the number of updates exceeds n (e.g., 6 times), the system will roll back to the last successful script. This ensures that the historical parsing logic is not disrupted, while also recording the number of HTML sets that have been developed and the number of failures.

[0144] After the rollback, the parser resumes execution from the breakpoint, iterating through each HTML file in the development set to ensure that the incrementally generated new scripts are executed correctly. While maintaining compatibility, the parsing capabilities were gradually optimized, eventually generating the in-cluster parsing script html_parse.py, which can parse all HTML within the cluster in batches.

[0145] The batch processing and output module for parsing results includes: 4.1 Routing table script registration; 4.2 Batch data processing; 4.3 Delivery of finished product data.

[0146] Through a batch processing strategy, this module is tightly integrated with the front-end routing module and the Agent automated parsing module to achieve a closed loop of routing table → in-cluster script → batch parsing → standardized output, ensuring that each cluster of web pages can call the corresponding script, thereby achieving efficient, scalable and stable batch parsing.

[0147] 4.1. Routing table script registration.

[0148] The system first registers each parsing script, html_parse.py, in the front-end routing table, establishing a mapping relationship between clusters, HTML pages, and scripts. Through this mapping, web pages within each cluster can call the corresponding parsing script, providing a unified scheduling entry point for batch processing and ensuring consistency in parsing web pages within the same cluster.

[0149] 4.2 Batch data processing.

[0150] For each cluster of The system calls the corresponding parsing scripts in batches to perform high-concurrency parsing and generate parsing results.

[0151] 4.3 Delivery of finished product data.

[0152] After parsing, the system performs deduplication, field standardization, and format unification on the results, including but not limited to... , , This ensures high-quality and traceable output data. The processed data can be directly used as pre-training corpus for large models, achieving efficient and stable data delivery.

[0153] The web page data parsing apparatus provided by the present invention is described below. The web page data parsing apparatus described below and the web page data parsing method described above can be referred to in correspondence.

[0154] In some embodiments, such as Figure 8 As shown, a webpage data parsing device is proposed, comprising: The acquisition module 801 is used to acquire multiple web page files with the same format. These multiple web page files with the same format are determined based on multiple web page data from different sources.

[0155] The information extraction and cluster analysis module 802 is used to extract structural information and perform cluster analysis on multiple web page files with the same format to obtain the target number of clusters.

[0156] The processing module 803 is used to divide multiple web page files into multiple web page clusters with a cluster number equal to the target cluster number, and to establish a front-end routing table that describes the mapping relationship between web page files, web page clusters, and parsing scripts within clusters.

[0157] The determination module 804 is used to determine the parsing script within the target webpage cluster and the target webpage file corresponding to the target webpage cluster based on the front-end routing table.

[0158] The parsing module 805 is used to perform batch parsing of target web page files based on the parsing script within the target cluster to obtain the parsing results; wherein, the target web page cluster is any one of multiple web page clusters.

[0159] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a webpage data parsing method. This method includes: acquiring multiple webpage files with a uniform format, the multiple uniform format webpage files being determined based on multiple webpage data from different sources; extracting structural information and performing cluster analysis on the multiple uniform format webpage files to obtain a target cluster number; dividing the multiple webpage files into multiple webpage clusters with a cluster number equal to the target cluster number, and establishing a front-end routing table describing the mapping relationship between webpage files, webpage clusters, and parsing scripts within clusters; based on the front-end routing table, determining the parsing script within the target webpage cluster corresponding to the target webpage cluster and the target webpage file corresponding to the target webpage cluster; performing batch parsing of the target webpage files based on the parsing script within the target cluster to obtain parsing results; wherein the target webpage cluster is any one of the multiple webpage clusters.

[0160] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the web page data parsing method provided by the above methods. The method includes: acquiring multiple web page files with uniform format, the multiple web page files with uniform format being determined based on multiple web page data from different sources; extracting structural information and performing cluster analysis on the multiple web page files with uniform format to obtain a target cluster number; dividing the multiple web page files into multiple web page clusters with a cluster number equal to the target cluster number, and establishing a front-end routing table describing the mapping relationship between web page files, web page clusters, and parsing scripts within clusters; determining the parsing scripts within the target clusters corresponding to the target web page clusters and the target web page files corresponding to the target web page clusters based on the front-end routing table; performing batch parsing of the target web page files based on the parsing scripts within the target clusters to obtain parsing results; wherein, the target web page cluster is any one of the multiple web page clusters.

[0162] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for parsing web page data provided by the methods described above. This method includes: acquiring multiple web page files with uniform format, the multiple uniform format web page files being determined based on multiple web page data from different sources; extracting structural information and performing cluster analysis on the multiple uniform format web page files to obtain a target cluster number; dividing the multiple web page files into multiple web page clusters with a cluster number equal to the target cluster number, and establishing a front-end routing table describing the mapping relationship between web page files, web page clusters, and parsing scripts within clusters; based on the front-end routing table, determining the parsing script within the target web page cluster corresponding to the target web page cluster and the target web page file corresponding to the target web page cluster; performing batch parsing of the target web page files based on the parsing script within the target cluster to obtain parsing results; wherein, the target web page cluster is any one of the multiple web page clusters.

[0163] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0164] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for parsing webpage data, characterized in that, include: Obtain multiple webpage files with a uniform format, wherein the multiple webpage files with a uniform format are determined based on multiple webpage data from different sources; Structural information is extracted and cluster analysis is performed on multiple web page files with the same format to obtain the target cluster number; Multiple web page files are divided into multiple web page clusters with a cluster number equal to the target cluster number, and a front-end routing table is established to describe the mapping relationship between the web page files, the web page clusters, and the parsing scripts within the clusters. Based on the front-end routing table, determine the parsing script within the target webpage cluster corresponding to the target webpage cluster and the target webpage file corresponding to the target webpage cluster; The target webpage files are parsed in batches based on the parsing script within the target cluster to obtain the parsing results; The target webpage cluster is any one of the multiple webpage clusters.

2. The method for parsing webpage data according to claim 1, characterized in that, The process of extracting structural information and performing cluster analysis on multiple webpage files with uniform format to obtain the target cluster number includes: Fingerprint extraction and vectorization representation are performed on each webpage file to obtain the webpage feature vector corresponding to each webpage file; Under different preset candidate cluster numbers, a preset clustering objective function is used to perform clustering analysis on the webpage feature vector corresponding to each webpage file to obtain clustering results under different preset candidate cluster numbers; Based on the clustering results, the average contour coefficient of all web page files is determined for each preset number of candidate clusters; The number of candidate clusters corresponding to the maximum average profile coefficient is taken as the number of target clusters.

3. The method for parsing webpage data according to claim 2, characterized in that, The step of extracting fingerprints and vectorizing each webpage file to obtain a webpage feature vector corresponding to each webpage file includes: For each tag in each of the web page files, a document object model path calculation is performed to obtain the path fingerprint corresponding to each tag; The path fingerprints corresponding to each tag are statistically analyzed to obtain the path fingerprint set corresponding to each web page file. Perform the following operations on each of the multiple webpage files to obtain the webpage feature vector corresponding to each webpage file: The sum of the frequencies of each path fingerprint appearing in the first web page file in the first path fingerprint set is used as the first quantization value of the first web page file. The first path fingerprint set is the path fingerprint set corresponding to the first web page file. The weighting value is determined based on the total number of web pages in the web page file and the number of web pages containing the target path fingerprint, wherein the target path fingerprint is the fingerprint of each path appearing in the first web page file; The product of the first quantized value and the weighted value is normalized to obtain the webpage feature vector corresponding to the first webpage file; The first webpage file is a selected webpage file from the plurality of webpage files.

4. The method for parsing webpage data according to claim 3, characterized in that, The step of determining the average contour coefficient of all web page files for each preset number of candidate clusters based on the clustering results includes: Based on the clustering results, the contour coefficient of each webpage file is determined for each preset number of candidate clusters. The average contour coefficient of all web pages is calculated by averaging the total number of web pages and the contour coefficient of all web pages. The contour coefficient of each webpage file is determined based on the average distance between the first webpage file and the second webpage file, and the average distance between the first webpage file and the third webpage file. The second webpage file is a webpage file that is in the same webpage cluster as the first webpage file, and the third webpage file is a webpage file in the webpage cluster closest to the first webpage file.

5. The method for parsing webpage data according to claim 1, characterized in that, The step of determining the parsing script within the target webpage cluster corresponding to the target webpage cluster and the target webpage file corresponding to the target webpage cluster based on the front-end routing table further includes: The webpage files in the target webpage cluster are randomly shuffled and divided to obtain a development set and a verification set; Load one of the web page files in the development set as the selected web page file; The selected webpage file and the predefined prompt words are input into the large model that is called, and a temporary parsing script generated by the large model is obtained; Load one of the webpage files in the verification set as a test webpage file; Determine the first parsing result obtained by executing the temporary parsing script to parse the test webpage file; If the similarity between the first parsing result and the first annotation result is less than a preset similarity threshold, the first parsing script is updated based on the parsing logic determined by the temporary parsing script to obtain the updated first parsing script. The first parsing script is the parsing script that was successfully developed last time, and the first annotation result is the annotation result corresponding to the test webpage file. The cluster-level parsing script in the front-end routing table is updated based on the updated first parsing script.

6. The method for parsing webpage data according to claim 5, characterized in that, If the temporary parsing script fails to execute or the cumulative number of updates where the similarity between the first parsing result and the first annotation result is less than a preset similarity threshold exceeds a preset number, the script is rolled back to the first parsing script.

7. The method for parsing webpage data according to any one of claims 1 to 6, characterized in that, The process of obtaining multiple webpage files with a uniform format includes: Retrieve data from multiple web pages from different sources; Each of the webpage data is subjected to webpage identification and webpage management to obtain multiple initial webpage files; Perform one or more of the following processes on each of the initial webpage files to obtain multiple webpage files with a uniform format: Noise reduction, invalid webpage text removal, character encoding standardization, and file integrity checking and repair.

8. The method for parsing webpage data according to any one of claims 1 to 6, characterized in that, The language of the webpage data includes at least one of Chinese, English, and minority languages; and / or The web page data includes at least one of the following: news pages, blog pages, forum pages, Q&A pages, and product pages.

9. A webpage data parsing device, characterized in that, include: The acquisition module is used to acquire multiple web page files with a uniform format, wherein the multiple web page files with a uniform format are determined based on multiple web page data from different sources; The information extraction and cluster analysis module is used to extract structural information and perform cluster analysis on multiple web page files with the same format to obtain the target number of clusters; The processing module is used to divide multiple web page files into multiple web page clusters with a cluster number equal to the target cluster number, and to establish a front-end routing table describing the mapping relationship between the web page files, the web page clusters, and the parsing scripts within the clusters. The determination module is used to determine, based on the front-end routing table, the parsing script within the target webpage cluster corresponding to the target webpage cluster and the target webpage file corresponding to the target webpage cluster; The parsing module is used to perform batch parsing of the target webpage files based on the parsing script within the target cluster, and obtain the parsing results; The target webpage cluster is any one of the multiple webpage clusters.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method for parsing web page data as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for parsing web page data as described in any one of claims 1 to 8.

Images