A webpage content intelligent crawling method and system based on data analysis

Abstract

This invention discloses a method and system for intelligent web page content crawling based on data analysis, belonging to the field of web page crawling technology. The method includes identifying candidate content blocks and calculating their Shannon entropy, generating an importance score by combining text density; constructing a global vocabulary based on the candidate content blocks and calculating inverse document frequencies of terms, defining a topic-specific factor, multiplying the importance score by the topic-specific factor to obtain a comprehensive priority, selecting candidate content blocks with a comprehensive priority higher than the average as target crawling blocks, and generating crawling rules for each target crawling block; and using the generated crawling rules to extract content from the initial HTML source code. This invention improves the content differentiation of web page crawling by combining text density analysis and Shannon entropy evaluation, and significantly enhances the accuracy of the crawling rules through in-depth analysis of web page structure and visual elements.

Description

Technical Field

[0001] This invention relates to the field of web scraping technology, and in particular to a method and system for intelligent web content scraping based on data analysis. Background Technology

[0002] With the rapid development of the Internet, the complexity and diversity of web page content are increasing day by day. Traditional web crawling technology can no longer effectively extract useful information from the huge and ever-changing web pages. Most existing web crawling technologies rely on simple HTML parsing and extract text information by statically analyzing the web page source code. However, this method is extremely weak when faced with dynamic web pages, complex interactive elements and multimodal content.

[0003] Existing web crawling technologies still have many shortcomings. Traditional web crawling methods often neglect the semantic structure of page content and the assessment of information importance, resulting in crawled content that is often redundant, low in information content, and difficult to effectively extract the core information from web pages. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a method and system for intelligent web page content crawling based on data analysis, which solves the problem that traditional web page crawling methods often neglect the semantic structure of page content and the assessment of information importance, resulting in crawled content that is often redundant, low in information content, and difficult to effectively extract the core information from web pages.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a method for intelligent webpage content crawling based on data analysis, which includes the following steps:

[0008] Collect initial HTML source code and multimodal web page data, define the region to be processed, convert it into the text density of the region, perform recursive termination condition judgment on the region to be processed, identify candidate content blocks, calculate the Shannon entropy of the candidate content blocks, and generate importance scores by combining the text density.

[0009] A global vocabulary is constructed based on candidate content blocks and the inverse document frequency of terms is calculated. A topic specificity factor is defined, and the importance score is multiplied by the topic specificity factor to obtain a comprehensive priority. Candidate content blocks with a comprehensive priority higher than the average are selected as target crawling blocks, and crawling rules are generated for each target crawling block.

[0010] The generated crawling rules are used to extract content from the initial HTML source code. If the extraction fails, a rollback process is initiated.

[0011] As a preferred embodiment of the intelligent webpage content crawling method based on data analysis described in this invention, the steps of defining the region to be processed, converting it into the text density of the region, performing recursive termination condition judgment on the region to be processed, identifying candidate content blocks, calculating the Shannon entropy of the candidate content blocks, and generating an importance score by combining the text density include:

[0012] Define the region corresponding to the root node of the DOM tree as the initial region to be processed. Traverse all text nodes in the region to be processed, accumulate the number of visible characters in the text content to obtain the total number of plain text characters in the region. Calculate the overall bounding rectangle based on all visual bounding boxes of the region to be processed to obtain the visual bounding box of the region, and convert it into the text density of the region.

[0013] The region to be processed is recursively terminated. If any recursive termination condition is met, the region to be processed is marked as a candidate content block.

[0014] The recursion termination condition includes that the DOM node corresponding to the area to be processed is a text node, and that all element nodes in the DOM subtree corresponding to the area to be processed are leaf nodes with no visible child elements.

[0015] If the region to be processed does not meet the recursion termination condition, then obtain all direct child nodes in the DOM tree, define each child node as a sub-region, calculate the text density of each sub-region, set the region threshold, traverse all sub-regions, and perform separation judgment on the sub-regions.

[0016] The separation judgment includes separating the sub-region from the parent region if the text density of the sub-region is higher than the region threshold, marking the sub-region as a candidate content block, and taking all direct sub-regions that are not marked as candidate content blocks as new regions to be processed, and returning to the step "judging the recursive termination condition of the region to be processed" to judge the recursive termination condition of the new region to be processed.

[0017] If none of the sub-regions meet the separation criteria, and the ratio of the text density of all sub-regions to the density of the current region is close to 1, then the entire region to be processed is marked as a candidate content block.

[0018] Extract the plain text string of each candidate content block, perform standard Chinese word segmentation on the plain text string, remove common stop words, obtain the term sequence of the candidate content block, count the occurrence of each non-repeating term in the sequence, convert it into term probability, and calculate the Shannon entropy of the candidate content block.

[0019] The importance index is obtained by multiplying the text density by the Shannon entropy. All candidate content blocks are traversed to find the maximum value of the importance index. Normalization is then performed on each candidate content block to obtain the final importance score.

[0020] As a preferred embodiment of the intelligent webpage content crawling method based on data analysis described in this invention, the step of constructing a global vocabulary based on candidate content blocks and calculating the inverse document frequency of terms, and defining a topic-specific factor, includes:

[0021] Treat each candidate content block as an independent document, and merge the term lists of all candidate content blocks to form a global vocabulary;

[0022] Based on the global vocabulary, a TF-IDF vector is constructed for each candidate content block. The TF-IDF vectors of all candidate content blocks are summed and averaged to obtain the average topic vector. The cosine similarity between the TF-IDF vector of each candidate content block and the average topic vector is calculated, and a topic specificity factor is defined.

[0023] As a preferred embodiment of the intelligent webpage content crawling method based on data analysis described in this invention, the method involves multiplying the importance score by a topic specificity factor to obtain a comprehensive priority, selecting candidate content blocks with a comprehensive priority higher than the average as target crawling blocks, and generating crawling rules for each target crawling block, including:

[0024] The importance score is multiplied by the topic specificity factor to obtain the overall priority. The arithmetic mean of the overall priorities of all candidate content blocks is calculated. Candidate content blocks with an overall priority higher than the average priority are selected to form the target crawling block set.

[0025] Sort the target blocks in the target grabbing block set according to the order in which the corresponding DOM nodes are accessed during the preorder traversal of the DOM tree, to obtain an ordered list. For each target block in the ordered list, start from the DOM root node and traverse upwards along the parent node chain until the first ancestor element node with an id attribute is found. Use this node as the positioning anchor point and generate a relative XPath path from this anchor point to the root node of the target block, which is used as the preferred positioning path for the target block.

[0026] If no element with an id attribute is found after traversing to the document root node, a complete absolute XPath path from the document root to the DOM root node is generated as a backup positioning path.

[0027] The generated two paths, combined priority, and term list are encapsulated into crawling rules, and arranged according to the order of the corresponding target blocks in the DOM tree to form the final ordered crawling rule list.

[0028] As a preferred embodiment of the intelligent webpage content crawling method based on data analysis described in this invention, the step of extracting content from the initial HTML source code using the generated crawling rules, and then proceeding to a rollback process if the extraction fails, includes:

[0029] For each rule in the rule list, the initial HTML source code is used as the parsing basis. An XPath query engine is used, and the preferred location path in the rule is used as the input query expression. The XPath query is executed. If the query successfully matches a DOM node, the text content of all text child nodes of the first matching node is extracted, concatenated and basically cleaned to obtain the text content of this crawl.

[0030] If no node is matched in the query, the execution of the main rule is determined to have failed, the fetched text is set to an empty string, and the rollback process is immediately initiated.

[0031] When a rollback is triggered, the reconstruction strategy is executed sequentially.

[0032] As a preferred embodiment of the intelligent webpage content crawling method based on data analysis described in this invention, the reconstruction strategy includes:

[0033] First-level fallback: Use the backup positioning path stored in the same rule to re-execute XPath query and text extraction on the initial HTML;

[0034] Second-level rollback: If the first level fails, abandon the current block, record that the block is not crawlable in this round of crawling, and do not produce any text content.

[0035] As a preferred embodiment of the intelligent webpage content crawling method based on data analysis described in this invention, the step of collecting initial HTML source code and multimodal webpage data includes:

[0036] Access the target URL through a headless browser, capture the initial HTML source code returned by the server during page loading, and collect multimodal web page data through the headless browser's API.

[0037] The multimodal web page data includes a DOM tree and a visual bounding box corresponding to each visible element.

[0038] Secondly, the present invention provides a web page content intelligent crawling system based on data analysis, comprising:

[0039] The multimodal data acquisition module is used to load the target webpage through the headless browser and determine that rendering is complete, while simultaneously acquiring multimodal webpage data;

[0040] The segmentation and importance assessment module is used to calculate text density by combining the number of plain text characters and visual area with DOM subtrees as the analysis unit. It also performs adaptive candidate content block segmentation of web page regions based on recursion and statistical thresholds, performs word segmentation and entropy calculation on the segmented candidate content blocks, and integrates Shannon entropy with text density to obtain a content importance score that reflects visual compactness and semantic information content.

[0041] The analysis and target content filtering module is used to construct the topic vector of candidate content blocks, calculate the difference between them and the overall topic of the webpage, and filter out high-priority target crawling candidate content blocks by integrating importance scores.

[0042] The crawling rules and content extraction module is used to generate XPath crawling rules based on the DOM structure and perform rule matching and text extraction on the initial HTML source code.

[0043] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the intelligent web page content crawling method based on data analysis as described in the first aspect of the present invention.

[0044] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the intelligent web page content crawling method based on data analysis as described in the first aspect of the present invention.

[0045] The beneficial effects of this invention are as follows: By combining text density analysis and Shannon entropy assessment, this invention improves the content differentiation of web page crawling, and significantly enhances the accuracy of crawling rules through in-depth analysis of web page structure and visual elements. Our method not only considers the text density of content blocks, but also combines the calculation of topic specificity factors, and conducts a comprehensive evaluation from both semantic and visual dimensions, thereby improving the accuracy of core content extraction and avoiding interference from redundant information and low-value content in the crawling process. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the operation of the intelligent webpage content crawling method based on data analysis in Example 1.

[0048] Figure 2 This is a schematic diagram of the intelligent web page content crawling system based on data analysis in Example 1. Detailed Implementation

[0049] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0050] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0051] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0052] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a method for intelligent web page content crawling based on data analysis, including the following steps:

[0053] S1. Collect initial HTML source code and multimodal web page data, define the region to be processed, convert it into the text density of the region, perform recursive termination condition judgment on the region to be processed, identify candidate content blocks, calculate the Shannon entropy of the candidate content blocks, and generate importance scores by combining the text density.

[0054] Specifically, this involves collecting initial HTML source code and multimodal webpage data, including:

[0055] By accessing the target URL through a headless browser, the initial HTML source code returned by the server is captured during the page loading process. After waiting for the DOM structure to remain unchanged for several consecutive time slices, the page rendering is determined to be complete. Multimodal web page data is collected through the API of the headless browser.

[0056] The multimodal web page data includes a DOM tree (the rendered document object model tree) and a visual bounding box corresponding to each visible element (traversing DOM tree nodes, querying their rendering status and position information, filtering out invisible elements (such as elements with a bounding box area of ​​0), and finally collecting the visual bounding box corresponding to each visible element, including the coordinates of the top left corner, width, and height).

[0057] DOM node, short for Document Object Model node, is the basic unit that makes up the DOM tree. After an HTML or XML document is parsed by a browser, each component of the document (such as elements, attributes, text, comments, etc.) is converted into an object with a hierarchical relationship.

[0058] The root node refers to the top-level node in a tree data structure that has no parent node. It is the starting point for accessing the entire tree. In this invention, it specifically refers to the document root node, which is the DOM node corresponding to the tag in the entire HTML document.

[0059] The DOM root node, in this invention, has the same meaning as "root node," specifically referring to the root node of the DOM tree. The prefix "DOM" is used to more precisely define its data structure and avoid confusion with the root nodes of other tree structures (such as the rendering tree). Its application in this invention is as follows: For example, when describing content blocks, "each content block corresponds to a subtree of the DOM tree, and its DOM root node is...", meaning the top-level node of the substructure to which the content block belongs in the DOM tree.

[0060] A leaf node, in a tree, refers to a node that has no child nodes. In this invention, its specific meaning and importance are as follows: In the context of the webpage DOM tree, the most important type of leaf node is the text node, which contains the plain text content that is ultimately presented to the user and has no child nodes. Its application in this invention is as follows: In the recursive termination condition of the steps, "if the DOM node corresponding to the region is a text node", the "text node" here is a type of leaf node.

[0061] In the DOM tree, all direct child nodes refer to all nodes that are directly subordinate to a particular node at the next level in the DOM tree hierarchy. These nodes are its "children," not its "grandchildren" or deeper descendants.

[0062] Furthermore, the region to be processed is defined, converted into the text density of the region, a recursive termination condition is applied to the region to be processed, candidate content blocks are identified, and the Shannon entropy of the candidate content blocks is calculated. An importance score is generated by combining the text density, including:

[0063] The region corresponding to the root node of the DOM tree is defined as the initial region to be processed. For any node in the DOM tree (Document Object Model tree), the corresponding processing region consists of all descendant nodes of that node in the DOM tree. The visual bounding box of this region is calculated by the union of the bounding boxes of all its visible descendant elements. The visual area is width multiplied by height. The total number of plain text characters within the region is obtained by traversing all text nodes in the corresponding DOM subtree and counting the characters. Traverse all text nodes in the region to be processed and accumulate the number of visible characters in the text content (excluding leading and trailing whitespace characters and...). <script>、<style>标签内的不可见文本），得到区域总纯文本字符数，根据待处理区域所有（可视子孙元素的边界框）视觉边界框，计算整体外接矩形，得到该区域的视觉边界框，转换为区域的文本密度，公式为：

[0064] ，

[0065] 其中为当前待处理区域R的文本密度，R为当前正在处理的区域，对应DOM树中的一个节点及其所辖的子树，为当前待处理区域R的内的纯文本字符总数，为当前待处理区域R的视觉面积，通过获取区域内所有可视元素节点的视觉边界框，计算这些边界框的并集，得到一个能完全包围它们的最小外接矩形，该矩形的面积即为视觉面积，外接矩形的宽度和高度相乘；

[0066] 对待处理区域进行递归终止条件判断，若满足以下任一递归终止条件，则将待处理区域标记为一个最终的候选内容块（并终止对待处理区域的进一步分割）；

[0067] 所述递归终止条件，包括待处理区域对应的DOM节点为文本节点，待处理区域对应的DOM子树中，所有元素节点均为无子可视元素的叶节点（即该子树中不存在一个可视元素，其自身还包含其他子可视元素）；

[0068] 若待处理区域不满足上述递归终止条件，则获取在DOM树中的所有直接子节点，每个子节点定义为一个子区域，计算每个子区域的文本密度；

[0069] 使用统计分析分位数法，计算将当前待处理区域中所有直接子区域的文本密度值的70%，设定区域阈值，遍历所有子区域，对子区域进行分离判断；

[0070] 所述分离判断，包括若子区域文本密度高于区域阈值，则判定子区域内容高度紧凑、独立性强，将子区域从父区域中分离，标记为一个独立的候选内容块，所有未被标记为独立候选内容块的直接子区域，作为新的待处理区域，并返回步骤"对待处理区域进行递归终止条件判断”，对新的待处理区域进行递归终止条件判断；

[0071] 若所有子区域均不满足分离判断，且满足所有子区域的文本密度与当前区域密度的比值均处于一个接近1的范围内，即满足：

[0072] ，

[0073] 其中为对于所有的j，这里的j是索引变量，用于遍历当前区域的所有直接子区域，为当前区域的第j个直接子区域，它对应DOM树中R节点的第j个子节点所构成的子树区域，为均匀性偏差阈值，使用网格搜索优化设定，通过在预设范围（如）内进行网格搜索，以在验证数据集上使候选内容块分割结果与人工标注的核心内容区域重合度（如IoU）最大化为目标，优化确定的一个经验值；

[0074] 则判定当前待处理区域内文本分布均匀，视觉布局连贯，无需进一步分割，将整个待处理区域标记为一个候选内容块；

[0075] 提取每个候选内容块的纯文本字符串，对纯文本字符串执行标准的中文分词处理，并移除通用的停用词（如"的”、"了”、"在”等），得到候选内容块的词项序列，统计序列中每个不重复词项的出现次数，转换为词项概率（出现次数除以序列的总词项数），计算候选内容块的香农熵，公式为：

[0076] ，

[0077] 其中为第k个候选内容块的香农熵，为分割后得到的第k个候选内容块，为候选内容块经过分词和去停用词后，得到的不重复词项的集合，为词项t在候选内容块中出现的概率；

[0078] 香农熵是信息论中用于度量随机变量不确定性的经典指标（参见C. E. Shannon,"A Mathematical Theory of Communication,” The Bell System Technical Journal,1948），在此上下文中，它将一个候选内容块的文本视为一个信息源，词项视为随机事件，熵值越高，表示该块内词项分布越均匀，信息含量越大，内容越不可预测（即冗余度低、信息量大）；反之，熵值越低，则表明内容可能重复、单一或信息量小，该指标能有效区分主体正文与重复性低信息区域（如导航栏、广告），为后续重要性排序提供可靠的语义维度。

[0079] 将文本密度与香农熵相乘，得到重要性指标；

[0080] 此乘法运算实现视觉紧凑度与语义信息量的协同放大，只有同时具备高文本密度（视觉上紧凑独立）和高香农熵（语义上信息丰富）的区域才能获得显著更高的原始评分，从而突出真正的核心候选内容块（如文章主体），同时抑制低密度或低信息量的区域（如空白、图标、重复性导航）。

[0081] 遍历所有候选内容块，找出重要性指标的最大值，对每个候选内容块执行归一化处理，得到最终的重要性评分。

[0082] 定义初始待处理区域与视觉边界框计算，它能精确地识别网页中每个区域的视觉范围，为后续的文本密度计算和内容块识别提供了清晰、准确的基础，优化了网页内容的结构化识别，即对于文本密度高的区域，能够更加准确地识别其为核心内容区域，避免抓取冗余或无关信息，如广告、导航栏等，减少了冗余内容的抓取，并确保了抓取的内容块具有较高的信息密度与独立性，在网页抓取中，通过将香农熵与文本密度相乘得到重要性指标，能够有效突出具有高信息含量且视觉紧凑的内容区域，能够区分出冗余信息和核心信息块，例如，能够自动忽略广告、导航栏等低信息量区域，而抓取如文章主体、核心数据等重要内容区域，有效识别和提取出对用户具有高度价值的网页内容，并自动过滤掉低价值信息，极大地提高了抓取的准确性和相关性。

[0083] S2、基于候选内容块构建全局词汇表并计算词项逆文档频率，定义主题特异性因子，将重要性评分与主题特异性因子相乘，得到综合优先级，筛选出综合优先级高于平均值的候选内容块作为目标抓取块，并为每个目标抓取块生成抓取规则；

[0084] 具体的，基于候选内容块构建全局词汇表并计算词项逆文档频率，定义主题特异性因子，包括：

[0085] 将每个候选内容块视为一个独立的文档，合并所有候选内容块的词项列表（提取其中所有不重复的词项），形成一个包含所有不重复词项的全局词汇表；

[0086] 基于全局词汇表，采用TF-IDF模型为每个候选内容块构建TF-IDF向量，计算全局词汇表中每个词项的文档频率和逆文档频率，转换为TF-IDF向量，公式为：

[0087] ，

[0088] 其中为词项t对于候选内容块的TF-IDF权重，将TF-IDF权重定义为TF-IDF向量的分量，构建TF-IDF向量，为词项在候选内容块中的词频，为词项t的逆文档频率；

[0089] 将所有候选内容块的TF-IDF向量相加后求平均，得到代表整个网页平均主题向量，计算每个候选内容块的TF-IDF向量与平均主题向量的余弦相似度，并定义主题特异性因子，公式为：

[0090] ，

[0091] 其中为候选内容块的主题特异性因子，为候选内容块的TF-IDF向量，为整个网页平均主题向量。

[0092] 逆文档频率（IDF）的计算则可以进一步强调在整个网页中相对稀缺的词项，进而提高抓取内容的区分度和信息量，通过引入IDF的概念，能够自动识别出页面中独特且具代表性的词汇，为后续内容块的精确筛选奠定基础，提升了抓取规则的语义准确性，使得抓取系统能够根据内容块的主题相关性，筛选出具有较高独特性和信息量的目标内容，避免抓取与整个页面主题无关的冗余信息，如通用导航栏或广告内容。

[0093] 进一步的，将重要性评分与主题特异性因子相乘，得到综合优先级，筛选出综合优先级高于平均值的候选内容块作为目标抓取块，并为每个目标抓取块生成抓取规则，包括：

[0094] 将重要性评分与主题特异性因子相乘，得到综合优先级；

[0095] 乘法融合策略使得仅当候选内容块在视觉重要性和主题特异性两个维度上均具有较高值时，才能获得高综合优先级，从而有效过滤掉高视觉重要性但低主题特异性（如通用导航栏）或高主题特异性但低视觉重要性（如特殊格式的广告）的噪声候选内容块；

[0096] 计算所有候选内容块综合优先级的算术平均值，筛选综合优先级高于平均优先级的候选内容块，构成目标抓取块集合；

[0097] 将目标抓取块集合中的目标块，按照对应DOM节点在DOM树中进行前序遍历时被访问的顺序进行排序，这种排序方式保证了生成的规则列表顺序与网页内容的视觉呈现和逻辑阅读顺序基本一致；

[0098] 得到有序列表，对有序列表中每个目标块，从DOM根节点开始，沿父节点链向上遍历，直至找到第一个具有id属性的祖先元素节点，以该节点为定位锚点，生成一条从该锚点出发，到目标块根节点的相对XPath路径，作为该目标块的首选定位路径；

[0099] 若遍历至文档根节点（无论是否找到）仍未找到具有id属性的元素，则生成从文档根到DOM根节点的完整绝对XPath路径，作为保底定位路径；

[0100] 将生成的两种路径、综合优先级以及词项列表封装成抓取规则，并按照对应目标块在DOM树中前序位置的顺序进行排列，形成最终的有序抓取规则列表。

[0101] 双重加权策略，能够有效过滤掉低相关性的内容，如在视觉上显眼但信息量低的广告，或者语义上独特但视觉布局混乱的内容区域，自动识别并优先抓取最具语义价值的区域，从而提高网页抓取结果的准确性和信息量，并确保抓取的内容能够较好地反映网页的核心主题和结构，通过优先级排序和逻辑顺序调整，本发明能够更加智能地处理网页内容，避免抓取顺序混乱或重要信息遗漏，在复杂的动态网页中，排序策略能够保证抓取结果与用户阅读习惯相一致，增强了抓取内容的可用性和实际价值，它提供了准确且稳定的抓取路径，确保即使网页结构发生变化，系统也能基于最优路径定位目标内容，确保了抓取规则能够紧密匹配页面的逻辑结构和视觉布局，提升了网页抓取的精准度和效率，通过综合优先级的加权筛选与动态生成XPath路径，不仅提升了抓取内容的相关性与准确性，还极大增强了抓取系统的适应性和鲁棒性。

[0102] S3、使用生成的抓取规则对初始HTML源码进行内容提取，若提取失败转入回退流程；

[0103] 具体的，使用生成的抓取规则对初始HTML源码进行内容提取，若提取失败转入回退流程，包括：

[0104] 对于规则列表中的每条规则，以初始HTML源码作为解析基础，使用一个XPath查询引擎，将规则中的首选定位路径作为查询表达式输入，执行XPath查询，若查询成功匹配到一个或多个DOM节点，则提取第一个匹配节点的所有文本子节点的文本内容，进行拼接与基本清洗（如去除首尾空白字符），得到本次抓取的文本内容；

[0105] 若查询未匹配到任何节点，则直接判定本次主规则执行失败，将抓取文本置为空字符串，并立即转入回退流程；

[0106] 当触发回退时，依序执行重构策略；

[0107] 所述重构策略，包括：

[0108] 第一级回退：使用同一规则中预存的保底定位路径，重新对初始HTML执行XPath查询与文本提取；

[0109] 第二级回退：若第一级失败，则放弃抓取当前块，并记录该块在本轮抓取中不可抓取，且不产出文本内容；

[0110] 所述基本清洗包括但不限于：去除文本首尾的空白字符（包括空格、制表符、换行符）、将多个连续的空白字符压缩为单个空格、移除不可见的控制字符等。

[0111] 使用XPath查询引擎能够大大减少抓取过程中的误差和错误，尤其是在面对结构复杂、元素多变的网页时，能高效、可靠地确保抓取到精确的内容，本发明通过综合应用XPath查询引擎、回退机制和基本清洗，在网页内容抓取过程中，提供了一种高效、稳定且可靠的解决方案，通过结合抓取规则执行、回退策略、基本清洗以及定位路径的多层次保障，本发明确保了抓取结果的高质量、稳定性和准确性。

[0112] 实施例2，参照图2为本发明第二个实施例，一种基于数据分析的网页内容智能抓取系统，包括：

[0113] 采集多模态数据模块，用于通过无头浏览器加载目标网页并判定渲染完成，同步采集多模态网页数据；

[0114] 分割与重要性评估模块，用于以DOM子树为分析单元，结合纯文本字符数与视觉面积计算文本密度，并基于递归与统计阈值对网页区域进行自适应候选内容块分割，对分割得到的候选内容块进行分词与熵值计算，并将香农熵与文本密度融合，得到反映视觉紧凑性与语义信息量的内容重要性评分；

[0115] 分析与目标内容筛选模块，用于构建候选内容块主题向量，计算其与网页整体主题的差异度，并融合重要性评分筛选出高优先级的目标抓取候选内容块；

[0116] 抓取规则与内容提取模块，用于生成基于DOM结构的XPath抓取规则，并在初始HTML源码上执行规则匹配与文本提取。

[0117] 本实施例还提供一种计算机设备，适用于基于数据分析的网页内容智能抓取方法的情况，包括：存储器和处理器；存储器用于存储计算机可执行指令，处理器用于执行计算机可执行指令，实现如上述实施例提出的基于数据分析的网页内容智能抓取方法。

[0118] 该计算机设备可以是终端，该计算机设备包括通过系统总线连接的处理器、存储器、通信接口、显示屏和输入装置。其中，该计算机设备的处理器用于提供计算和控制能力。该计算机设备的存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统和计算机程序。该内存储器为非易失性存储介质中的操作系统和计算机程序的运行提供环境。该计算机设备的通信接口用于与外部的终端进行有线或无线方式的通信，无线方式可通过WIFI、运营商网络、NFC（近场通信）或其他技术实现。该计算机设备的显示屏可以是液晶显示屏或者电子墨水显示屏，该计算机设备的输入装置可以是显示屏上覆盖的触摸层，也可以是计算机设备外壳上设置的按键、轨迹球或触控板，还可以是外接的键盘、触控板或鼠标等。

[0119] 本实施例还提供一种存储介质，其上存储有计算机程序，该程序被处理器执行时实现如上述实施例提出的实现基于数据分析的网页内容智能抓取方法；存储介质可以由任何类型的易失性或非易失性存储设备或者它们的组合实现，如静态随机存取存储器（Static Random Access Memory，简称SRAM），电可擦除可编程只读存储器（ElectricallyErasable Programmable Read－Only Memory，简称EEPROM），可擦除可编程只读存储器（Erasable Programmable Read Only Memory，简称EPROM），可编程只读存储器（Programmable Red－Only Memory，简称PROM），只读存储器（Read－Only Memory，简称ROM），磁存储器，快闪存储器，磁盘或光盘。

[0120] 应说明的是，以上实施例仅用以说明本发明的技术方案而非限制，尽管参照较佳实施例对本发明进行了详细说明，本领域的普通技术人员应当理解，可以对本发明的技术方案进行修改或者等同替换，而不脱离本发明技术方案的精神和范围，其均应涵盖在本发明的权利要求范围当中。< / script>

Claims

1. A method for intelligent webpage content crawling based on data analysis, characterized in that: Includes the following steps: Collect initial HTML source code and multimodal web page data, define the region to be processed, convert it into the text density of the region, perform recursive termination condition judgment on the region to be processed, identify candidate content blocks, calculate the Shannon entropy of the candidate content blocks, and generate importance scores by combining the text density. A global vocabulary is constructed based on candidate content blocks and the inverse document frequency of terms is calculated. A topic specificity factor is defined, and the importance score is multiplied by the topic specificity factor to obtain a comprehensive priority. Candidate content blocks with a comprehensive priority higher than the average are selected as target crawling blocks, and crawling rules are generated for each target crawling block. The generated crawling rules are used to extract content from the initial HTML source code. If the extraction fails, a rollback process is initiated.

2. The intelligent webpage content crawling method based on data analysis as described in claim 1, characterized in that: The defined region to be processed is converted into the text density of the region. A recursive termination condition is applied to the region to be processed to identify candidate content blocks and calculate their Shannon entropy. An importance score is then generated by combining the text density, including: Define the region corresponding to the root node of the DOM tree as the initial region to be processed. Traverse all text nodes in the region to be processed, accumulate the number of visible characters in the text content to obtain the total number of plain text characters in the region. Calculate the overall bounding rectangle based on all visual bounding boxes of the region to be processed to obtain the visual bounding box of the region, and convert it into the text density of the region. The region to be processed is recursively terminated. If any recursive termination condition is met, the region to be processed is marked as a candidate content block. The recursion termination condition includes that the DOM node corresponding to the area to be processed is a text node, and that all element nodes in the DOM subtree corresponding to the area to be processed are leaf nodes with no visible child elements. If the region to be processed does not meet the recursion termination condition, then obtain all direct child nodes in the DOM tree, define each child node as a sub-region, calculate the text density of each sub-region, set the region threshold, traverse all sub-regions, and perform separation judgment on the sub-regions. The separation judgment includes separating the sub-region from the parent region if the text density of the sub-region is higher than the region threshold, marking the sub-region as a candidate content block, and taking all direct sub-regions that are not marked as candidate content blocks as new regions to be processed, and returning to the step "judging the recursive termination condition of the region to be processed" to judge the recursive termination condition of the new region to be processed. If none of the sub-regions meet the separation criteria, and the ratio of the text density of all sub-regions to the density of the current region is close to 1, then the entire region to be processed is marked as a candidate content block. Extract the plain text string of each candidate content block, perform standard Chinese word segmentation on the plain text string, remove common stop words, obtain the term sequence of the candidate content block, count the occurrence of each non-repeating term in the sequence, convert it into term probability, and calculate the Shannon entropy of the candidate content block. The importance index is obtained by multiplying the text density by the Shannon entropy. All candidate content blocks are traversed to find the maximum value of the importance index. Normalization is then performed on each candidate content block to obtain the final importance score.

3. The intelligent webpage content crawling method based on data analysis as described in claim 2, characterized in that: The process of constructing a global vocabulary based on candidate content blocks and calculating inverse document frequencies of terms, and defining topic-specific factors, includes: Each candidate content block is treated as an independent document, and the term lists of all candidate content blocks are merged to form a global vocabulary. Based on the global vocabulary, a TF-IDF vector is constructed for each candidate content block. The TF-IDF vectors of all candidate content blocks are summed and averaged to obtain the average topic vector. The cosine similarity between the TF-IDF vector of each candidate content block and the average topic vector is calculated, and a topic specificity factor is defined.

4. The intelligent webpage content crawling method based on data analysis as described in claim 3, characterized in that: The importance score is multiplied by a topic specificity factor to obtain a comprehensive priority. Candidate content blocks with a comprehensive priority higher than the average are selected as target crawling blocks, and crawling rules are generated for each target crawling block, including: The importance score is multiplied by the topic specificity factor to obtain the overall priority. The arithmetic mean of the overall priorities of all candidate content blocks is calculated. Candidate content blocks with an overall priority higher than the average priority are selected to form the target crawling block set. Sort the target blocks in the target grabbing block set according to the order in which the corresponding DOM nodes are accessed during the preorder traversal of the DOM tree, to obtain an ordered list. For each target block in the ordered list, start from the DOM root node and traverse upwards along the parent node chain until the first ancestor element node with an id attribute is found. Use this node as the positioning anchor point and generate a relative XPath path from this anchor point to the root node of the target block, which is used as the preferred positioning path for the target block. If no element with an id attribute is found after traversing to the document root node, a complete absolute XPath path from the document root to the DOM root node is generated as a backup positioning path. The generated two paths, combined priority, and term list are encapsulated into crawling rules, and arranged according to the order of the corresponding target blocks in the DOM tree to form the final ordered crawling rule list.

5. The intelligent webpage content crawling method based on data analysis as described in claim 4, characterized in that: The process of extracting content from the initial HTML source code using the generated crawling rules, and then proceeding to a fallback process if extraction fails, includes: For each rule in the rule list, the initial HTML source code is used as the parsing basis. An XPath query engine is used, and the preferred location path in the rule is used as the input query expression. The XPath query is executed. If the query successfully matches a DOM node, the text content of all text child nodes of the first matching node is extracted, concatenated and basically cleaned to obtain the text content of this crawl. If no node is matched in the query, the execution of the main rule is determined to have failed, the fetched text is set to an empty string, and the rollback process is immediately initiated. When a rollback is triggered, the reconstruction strategy is executed sequentially.

6. The intelligent webpage content crawling method based on data analysis as described in claim 5, characterized in that: The reconstruction strategy includes: First-level fallback: Use the backup positioning path stored in the same rule to re-execute XPath query and text extraction on the initial HTML; Second-level rollback: If the first level fails, abandon the current block, record that the block is not crawlable in this round of crawling, and do not produce any text content.

7. The intelligent webpage content crawling method based on data analysis as described in claim 1, characterized in that: The collection of initial HTML source code and multimodal webpage data includes: Access the target URL through a headless browser, capture the initial HTML source code returned by the server during page loading, and collect multimodal web page data through the headless browser's API. The multimodal web page data includes a DOM tree and a visual bounding box corresponding to each visible element.

8. A webpage content intelligent crawling system based on data analysis, used to implement the webpage content intelligent crawling method based on data analysis as described in any one of claims 1 to 7, characterized in that: include: The multimodal data acquisition module is used to load the target webpage through the headless browser and determine that rendering is complete, while simultaneously acquiring multimodal webpage data; The segmentation and importance assessment module is used to calculate text density by combining the number of plain text characters and visual area with DOM subtrees as the analysis unit. It also performs adaptive candidate content block segmentation of web page regions based on recursion and statistical thresholds, performs word segmentation and entropy calculation on the segmented candidate content blocks, and integrates Shannon entropy with text density to obtain a content importance score that reflects visual compactness and semantic information content. The analysis and target content filtering module is used to construct the topic vector of candidate content blocks, calculate the difference between them and the overall topic of the webpage, and filter out high-priority target crawling candidate content blocks by integrating importance scores. The crawling rules and content extraction module is used to generate XPath crawling rules based on the DOM structure and perform rule matching and text extraction on the initial HTML source code.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the intelligent web page content crawling method based on data analysis as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the intelligent web page content crawling method based on data analysis as described in any one of claims 1 to 7.

Images