Internet webpage content feature extraction method based on artificial intelligence

Abstract

The invention discloses an Internet webpage content feature extraction method based on artificial intelligence, which comprises the following steps: S1, acquiring a webpage HTML source file and a rendering image, and preprocessing the webpage HTML source file and the rendering image; s2, performing semantic classification on DOM nodes, encoding the DOM nodes into three types of identifiers, and constructing a node label sequence; s3, performing time sequence synchronization on the node attribute vector, the node tag sequence and the visual area set, and performing block-level slicing; s4, inputting the block-level slices into a gated recursive attention network, extracting multi-modal joint representation, and executing attention aggregation; s5, performing structure alignment on the feature fusion sequence, calculating a cross-node consistency distance, and screening a target slice set with high structure cohesion; and S6, mapping the target slice set to a content feature space, and generating a content feature tag set. According to the method, the structural accuracy, the semantic integrity and the multi-modal fusion precision of webpage content extraction are improved.

Description

Technical Field

[0001] This invention relates to the field of web page content analysis and information extraction technology, and in particular to an artificial intelligence-based method for extracting features from internet web page content. Background Technology

[0002] In the context of the rapid development of the internet, web pages have become the primary carrier of information dissemination and interaction. The automatic extraction and structured representation of web page content are of significant value for applications such as search engine optimization, public opinion analysis, intelligent recommendation, and content supervision. Current mainstream web page content extraction methods are mostly based on HTML structure analysis, keyword rule matching, or template extraction strategies, typically relying on fixed tag rules or manually defined feature templates to identify the main content areas of a web page. However, with the increasing diversity of web page design styles, the highly dynamic nature of content structures, and the complexity of advertising components and interaction logic, traditional rule-driven or template-matching methods have significant limitations in terms of the generalization ability of web page content structure and the accuracy of information discrimination.

[0003] In existing technologies, some methods introduce deep learning models for feature extraction and classification of web page content, such as using convolutional neural networks to extract image features or recurrent neural networks to process text information. While these methods improve feature representation capabilities to some extent, they often neglect the hierarchical organization of the web page's DOM structure and fail to fully capture the contextual dependencies between nodes. Furthermore, most methods struggle to achieve cross-modal information fusion, leading to the separation of image and text heterogeneous information processing and limiting the accuracy of overall semantic expression. In addition, during the feature selection stage, there is a lack of a comprehensive discrimination mechanism for web page content across structural levels, semantic tags, and visual positions, which can easily generate invalid redundant information or misclassify non-essential content.

[0004] To address the aforementioned issues, there is currently a lack of content feature extraction mechanisms that can integrate webpage DOM structure information, semantic tag expression, and visual region perception capabilities. Furthermore, existing methods generally lack the ability to perform fine-grained alignment modeling of the correlation strength between structural paths, making it impossible to achieve a high-precision and universal webpage content feature extraction and tag generation process.

[0005] Therefore, how to provide a method for extracting features from Internet web page content based on artificial intelligence is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose an artificial intelligence-based method for extracting features from internet web page content. This invention fully integrates structured DOM parsing, semantic tag encoding, visual region modeling, and cross-modal feature fusion mechanisms. It describes in detail how to extract multimodal joint representations between web page nodes through a gated recursive attention network, and achieve high-cohesion target slice selection and content tag generation based on a structural path alignment strategy. It has the advantages of high structural recognition accuracy, accurate semantic extraction, strong adaptability, and high controllability of tag output.

[0007] According to an embodiment of the present invention, an artificial intelligence-based method for extracting features from internet webpage content includes the following steps:

[0008] S1. Obtain the HTML source file and rendered image of the Internet webpage and perform preprocessing;

[0009] S2. Semantically classify the DOM node set, encode text nodes, media nodes and interactive nodes into three types of semantic identifiers, and construct a node label sequence aligned with the structure path matrix;

[0010] S3. Synchronize the node attribute vector, node label sequence and visual region set in time to construct a cross-modal node sequence set, and use a window sliding mechanism to slice the node sequence into blocks.

[0011] S4. Input the block-level slices into the gated recurrent attention network, extract the joint representation of node attributes, semantic labels and visual features in sequence, and perform attention-guided aggregation in combination with the structure path matrix to generate a feature fusion sequence.

[0012] S5. Perform structural alignment operation on the feature fusion sequence, use the cross-path-based sparse alignment algorithm to calculate the cross-node consistency distance, and filter the target slice set with structural cohesion higher than the threshold.

[0013] S6. Map the target slice set to a fixed-dimensional content feature space, and generate a set of web page content feature tags with semantic tags, location indexes and hierarchical structure based on node indexes and structural path numbers.

[0014] Optionally, the HTML source file preprocessing refers to: using a depth-first traversal-based DOM parsing method, sequentially visiting each child node starting from the root node, extracting the attribute information of each node, including tag type, text content, inline styles, class name, ID attribute, event binding attribute, and custom attribute fields, constructing a structural path vector based on the DOM hierarchy and parent-child relationship of the node, encoding the attribute information of all nodes into a multi-dimensional numerical vector set, and recording the hierarchy index and parent node index of each node in the DOM tree, generating a node attribute vector set and a structural path matrix.

[0015] Optionally, the rendering image preprocessing refers to: performing pixel-level region division on the rendering image using an image segmentation algorithm, identifying the text and image content region, function button region, input form region, and multimedia display region based on the web page rendering layout, extracting the boundary coordinates, width and height dimensions, and region classification labels of each visual region, matching the center position of each visual region with the rendering position information of each DOM node in the HTML structure path matrix, determining the DOM node index corresponding to each visual region through position overlap and hierarchy consistency calculations, establishing the index correspondence between the visual region set and the structure path matrix, and generating a visual region set for cross-modal alignment.

[0016] Optionally, the gated recursive attention network consists of a gating unit, a recursive aggregation unit, and an attention fusion unit. The gating unit receives the attribute vector, label vector, and visual vector of each node in the block-level slice and dynamically adjusts the information channels based on the node type. The recursive aggregation unit uses residual connections and a gated loop mechanism to expand the time steps and fuse the context state. The attention fusion unit performs multi-head attention operations on the state vector output at each time step and guides the attention distribution update in conjunction with the structure path matrix, outputting a feature fusion sequence with enhanced inter-node dependencies.

[0017] Optionally, S2 specifically includes:

[0018] S21. Traverse the DOM node set, initially group the nodes according to the node tag type, extract nodes with text information as text nodes, extract nodes containing image, audio or video URL fields as media nodes, and extract nodes with user interaction attributes as interactive nodes.

[0019] S22. Assign preset semantic identifiers to the text nodes, media nodes and interaction nodes respectively, wherein the text nodes are encoded as first-type semantic identifiers, the media nodes are encoded as second-type semantic identifiers, and the interaction nodes are encoded as third-type semantic identifiers, and embed the semantic identifier of each node as an independent label into the attribute vector of the corresponding node.

[0020] S23. Combining the structural path matrix, based on the order and hierarchical position of the node index in the DOM structure, perform structural alignment processing on the semantic identifier of each node, and output the node label sequence.

[0021] Optionally, S3 specifically includes:

[0022] S31. Align the set of node attribute vectors, the sequence of node labels, and the set of visual regions according to the node index order in the structure path matrix. Combine the attribute vectors, semantic labels, and visual regions corresponding to the same node index into a unified node feature unit, arrange them according to the traversal order of the DOM structure, and construct a cross-modal node sequence.

[0023] S32. Perform time synchronization processing on cross-modal node sequences. Based on the node's level depth in the DOM tree, the spatial layout order in the page, and the visibility state, adjust the node order and tag state to generate a standardized node sequence that maintains cross-modal semantic consistency.

[0024] S33. Apply a fixed-length window sliding mechanism to the standardized node sequence, set the sliding step size and window size parameters, extract continuous node segments in sequence, and construct a block-level slice sequence with overlapping relationship for feature fusion operation.

[0025] Optionally, S4 specifically includes:

[0026] S41. Input the block-level slice sequence one by one into the gated recurrent attention network. Each node in the block-level slice contains an attribute vector, semantic label and visual features. The gated recurrent attention network receives the multimodal input of each node and maintains the temporal dependency structure between nodes.

[0027] S42. At each time step, the fusion channel weights of the attribute vector, semantic label vector and visual feature vector are dynamically adjusted using a gating mechanism to weight and combine the multimodal inputs of the node, and the context states of the node before and after are recursively aggregated to form an intermediate fusion vector with contextual dependencies.

[0028] S43. Combining the node hierarchy relationships and parent-child connection paths recorded in the structural path matrix, perform structure-guided multi-head attention operations and output feature fusion sequences through attention aggregation mechanisms.

[0029] Optionally, S5 specifically includes:

[0030] S51. Combine the feature fusion sequences pairwise to construct a block-level slice pairing set. Using a sparse alignment algorithm based on cross paths, and under the constraint of the structural path matrix, calculate the cross-node consistency distance D between the i-th and j-th block-level slices. ij ;

[0031] S52, The cross-node consistency distance D ij As input, a cohesion scoring function for all block-level slices in the structural consistency dimension is constructed to obtain the structural cohesion score of the i-th block-level slice. The calculation formula is as follows:

[0032]

[0033] Among them, S i This represents the structural cohesion score of the i-th block-level slice, where m is the total number of block-level slices, Z is the normalization factor, and κ(D) represents the structural cohesion score of the i-th block-level slice. ij ,θ) represents the distance threshold mapping function, when D ij The value is 1 when θ < 0, otherwise it is 0. γ is the feature distance attenuation coefficient, used to adjust the influence intensity of the average feature difference. i u j Let u be the average vector of the node features in the i-th and j-th block-level slices, respectively. i -u j ||1 represents the norm distance between average features, ψ(p i ,p j ) is the structural path similarity function, which measures the overall similarity between two slices on the structural path, and exp(·) represents the natural exponential function with base e;

[0034] S53. Score the structural cohesion of each block-level slice. i Slices with scores higher than a preset threshold are selected to form a target slice set.

[0035] Optionally, the cross-node consistency distance D ij The calculation formula is:

[0036]

[0037] Among them, D ij Let represent the consistency distance between the i-th and j-th block-level slices, and n represent the number of nodes in each block-level slice. Let W1 and W2 represent the feature fusion vectors of the k-th node in the i-th and j-th block-level slices, respectively. Let b represent the bias term, σ(·) represent the Sigmoid activation function, tanh(·) represent the hyperbolic tangent function, and ⊙ represent the Hadamard element-wise product. For path alignment kernel function, representing node and The structural path difference degree is a non-negative real number. The hierarchical difference normalization function represents the node. and The difference in hierarchical indexes in the DOM tree, ‖·‖2 represents the 2-norm operator.

[0038] Optionally, the sparse alignment algorithm based on cross paths refers to constructing a sparse connection graph by utilizing the cross-overlap relationship of path indexes between nodes and hierarchical differences under the constraints of the structural path matrix, and performing local matching operations on the sparse connection graph to achieve structural consistency alignment between block-level slices.

[0039] The beneficial effects of this invention are:

[0040] First, this invention constructs a cross-modal node sequence by integrating DOM node attributes, semantic tags, and visual region information, and uses a window sliding mechanism for block-level slicing, which effectively improves the ability to synchronously express content in complex web page structures and multi-source heterogeneous data environments.

[0041] Secondly, a gated recursive attention network is used to perform multimodal joint modeling on block-level slices, and attention-guided aggregation is performed in combination with the structural path matrix, which enhances the ability to model the contextual association between nodes. While ensuring the integrity of semantic expression, the hierarchical features and layout information of the web page structure are taken into account. A consistency distance function and structural cohesion scoring mechanism are constructed through a sparse alignment algorithm based on structural path, which effectively achieves fine screening of target content and accurate alignment of structure.

[0042] Finally, based on the content vector mapping and tag generation mechanism of the unified feature space, semantic tags, location indexes and structural hierarchy are integrated into standardized output results. This not only improves the accuracy and generalization ability of content extraction, but also provides a clear and semantically complete content feature expression for downstream web page analysis, recommendation systems and information retrieval tasks, realizing the goals of intelligent, automated and highly robust web page content feature extraction process. Attached Figure Description

[0043] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0044] Figure 1 This is a flowchart of an artificial intelligence-based method for extracting features from internet webpage content, as proposed in this invention.

[0045] Figure 2 This is a flowchart illustrating the webpage content feature extraction method based on artificial intelligence proposed in this invention.

[0046] Figure 3 This is a flowchart illustrating the feature fusion and structural alignment of an artificial intelligence-based method for extracting internet webpage content features proposed in this invention. Detailed Implementation

[0047] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0048] refer to Figure 1-3A method for extracting features from internet webpage content based on artificial intelligence includes the following steps:

[0049] S1. Obtain the HTML source file and rendered image of the Internet webpage and perform preprocessing;

[0050] S2. Semantically classify the DOM node set, encode text nodes, media nodes and interactive nodes into three types of semantic identifiers, and construct a node label sequence aligned with the structure path matrix;

[0051] S3. Synchronize the node attribute vector, node label sequence and visual region set in time to construct a cross-modal node sequence set, and use a window sliding mechanism to slice the node sequence into blocks.

[0052] S4. Input the block-level slices into the gated recurrent attention network, extract the joint representation of node attributes, semantic labels and visual features in sequence, and perform attention-guided aggregation in combination with the structure path matrix to generate a feature fusion sequence.

[0053] S5. Perform structural alignment operation on the feature fusion sequence, use the cross-path-based sparse alignment algorithm to calculate the cross-node consistency distance, and filter the target slice set with structural cohesion higher than the threshold.

[0054] S6. Map the target slice set to a fixed-dimensional content feature space, and generate a set of web page content feature tags with semantic tags, location indexes and hierarchical structure based on node indexes and structural path numbers.

[0055] This invention constructs a complete web page content extraction process, integrating HTML structure, visual images, and multimodal features, which enables automatic identification, semantic aggregation, and structured output of web page content, significantly improving the accuracy and versatility of content extraction.

[0056] In this embodiment, the HTML source file preprocessing refers to: using a depth-first traversal-based DOM parsing method, sequentially visiting each child node starting from the root node, extracting the attribute information of each node, including tag type, text content, inline styles, class name, ID attribute, event binding attribute, and custom attribute fields, constructing a structural path vector based on the DOM hierarchy and parent-child relationship of the node, encoding the attribute information of all nodes into a multi-dimensional numerical vector set, and recording the hierarchy index and parent node index of each node in the DOM tree, generating a node attribute vector set and a structural path matrix.

[0057] This invention employs a depth-first traversal-based DOM parsing method to comprehensively extract node attribute information and construct a structural path matrix, providing a precise and unified structural foundation for subsequent structure-aware modeling and semantic alignment.

[0058] In this embodiment, the rendering image preprocessing refers to: performing pixel-level region division on the rendering image using an image segmentation algorithm, identifying the text and image content area, function button area, input form area, and multimedia display area based on the web page rendering layout, extracting the boundary coordinates, width and height dimensions, and region classification labels of each visual region, and matching the center position of each visual region with the rendering position information of each DOM node in the HTML structure path matrix. Through position overlap and hierarchy consistency calculations, the DOM node index corresponding to each visual region is determined, establishing the index correspondence between the visual region set and the structure path matrix, and generating a visual region set for cross-modal alignment.

[0059] This invention achieves effective fusion of visual information and structural semantic information of web pages by performing pixel-level region division on rendered images and establishing an index correspondence between visual regions and structural paths, thereby improving the accuracy of multimodal feature synchronization.

[0060] In this embodiment, the gated recursive attention network consists of a gating unit, a recursive aggregation unit, and an attention fusion unit. The gating unit receives the attribute vector, label vector, and visual vector of each node in the block-level slice and dynamically adjusts the information channels based on the node type. The recursive aggregation unit uses residual connections and a gated loop mechanism to expand the time steps and fuse the context state. The attention fusion unit performs multi-head attention operations on the state vector output at each time step and guides the attention distribution update in conjunction with the structure path matrix, outputting a feature fusion sequence with enhanced inter-node dependencies.

[0061] This invention designs a gated recurrent attention network to achieve dynamic fusion and temporal modeling of attribute vectors, label vectors, and visual vectors, thereby enhancing the ability to model contextual dependencies between nodes and improving the expressive power and robustness of feature extraction.

[0062] In this embodiment, S2 specifically includes:

[0063] S21. Traverse the DOM node set, initially group the nodes according to the node tag type, extract nodes with text information as text nodes, extract nodes containing image, audio or video URL fields as media nodes, and extract nodes with user interaction attributes as interactive nodes.

[0064] S22. Assign preset semantic identifiers to the text nodes, media nodes and interaction nodes respectively, wherein the text nodes are encoded as first-type semantic identifiers, the media nodes are encoded as second-type semantic identifiers, and the interaction nodes are encoded as third-type semantic identifiers, and embed the semantic identifier of each node as an independent label into the attribute vector of the corresponding node.

[0065] S23. Combining the structural path matrix, based on the order and hierarchical position of the node index in the DOM structure, perform structural alignment processing on the semantic identifier of each node, and output the node label sequence.

[0066] This invention achieves structural alignment and standardization of semantic annotation by classifying DOM node semantics into three categories of identifiers: text, media, and interaction, and aligning them with structural paths to generate node label sequences, thereby enhancing the clarity and consistency of semantic recognition.

[0067] In this embodiment, S3 specifically includes:

[0068] S31. Align the set of node attribute vectors, the sequence of node labels, and the set of visual regions according to the node index order in the structure path matrix. Combine the attribute vectors, semantic labels, and visual regions corresponding to the same node index into a unified node feature unit, arrange them according to the traversal order of the DOM structure, and construct a cross-modal node sequence.

[0069] S32. Perform time synchronization processing on cross-modal node sequences. Based on the node's level depth in the DOM tree, the spatial layout order in the page, and the visibility state, adjust the node order and tag state to generate a standardized node sequence that maintains cross-modal semantic consistency.

[0070] S33. Apply a fixed-length window sliding mechanism to the standardized node sequence, set the sliding step size and window size parameters, extract continuous node segments in sequence, and construct a block-level slice sequence with overlapping relationship for feature fusion operation.

[0071] This invention achieves a feature fragment generation method that is temporally synchronized, semantically consistent, and structurally continuous by constructing cross-modal node sequences and using a block-level sliding slicing strategy, providing high-quality input for subsequent deep feature aggregation.

[0072] In this embodiment, S4 specifically includes:

[0073] S41. Input the block-level slice sequence one by one into the gated recurrent attention network. Each node in the block-level slice contains an attribute vector, semantic label and visual features. The gated recurrent attention network receives the multimodal input of each node and maintains the temporal dependency structure between nodes.

[0074] S42. At each time step, the fusion channel weights of the attribute vector, semantic label vector and visual feature vector are dynamically adjusted using a gating mechanism to weight and combine the multimodal inputs of the node, and the context states of the node before and after are recursively aggregated to form an intermediate fusion vector with contextual dependencies.

[0075] S43. Combining the node hierarchy relationships and parent-child connection paths recorded in the structural path matrix, perform structure-guided multi-head attention operations and output feature fusion sequences through attention aggregation mechanisms.

[0076] This invention enhances the joint semantic and structural expression capabilities between nodes through recursive aggregation and structure-guided attention mechanisms, improves the fusion depth of multimodal features, and effectively supports simultaneous modeling while preserving structure and contextual dependencies.

[0077] In this embodiment, S5 specifically includes:

[0078] S51. Combine the feature fusion sequences pairwise to construct a block-level slice pairing set. Using a sparse alignment algorithm based on cross paths, and under the constraint of the structural path matrix, calculate the cross-node consistency distance D between the i-th and j-th block-level slices. ij ;

[0079] S52, The cross-node consistency distance D ij As input, a cohesion scoring function for all block-level slices in the structural consistency dimension is constructed to obtain the structural cohesion score of the i-th block-level slice. The calculation formula is as follows:

[0080]

[0081] Among them, S i This represents the structural cohesion score of the i-th block-level slice, where m is the total number of block-level slices, Z is the normalization factor, and κ(D) represents the structural cohesion score of the i-th block-level slice. ij ,θ) represents the distance threshold mapping function, when D ij The value is 1 when θ < 0, otherwise it is 0. γ is the feature distance attenuation coefficient, used to adjust the influence intensity of the average feature difference. i u j Let u be the average vector of the node features in the i-th and j-th block-level slices, respectively. i -u j ||1 represents the norm distance between average features, ψ(p i ,p j ) is the structural path similarity function, which measures the overall similarity between two slices on the structural path, and exp(·) represents the natural exponential function with base e;

[0082] S53. Score the structural cohesion of each block-level slice. i Slices with scores higher than a preset threshold are selected to form a target slice set.

[0083] This invention achieves fine feature filtering based on structural relationships by calculating the cross-node consistency distance between block-level slices and introducing a structural cohesion scoring function, ensuring that the output content has structural integrity and information density.

[0084] In this embodiment, the cross-node consistency distance D ij The calculation formula is:

[0085]

[0086] Among them, D ij Let represent the consistency distance between the i-th and j-th block-level slices, and n represent the number of nodes in each block-level slice. Let W1 and W2 represent the feature fusion vectors of the k-th node in the i-th and j-th block-level slices, respectively. Let b represent the bias term, σ(·) represent the Sigmoid activation function, tanh(·) represent the hyperbolic tangent function, and ⊙ represent the Hadamard element-wise product. For path alignment kernel function, representing node and The structural path difference degree is a non-negative real number. The hierarchical difference normalization function represents the node. and The difference in hierarchical indexes in the DOM tree, ‖·‖2 represents the 2-norm operator.

[0087] This invention introduces a nonlinear cross-computation mechanism and a path-level weight function, enabling the consistency distance to simultaneously possess the ability to discriminate differences in both structural and semantic dimensions, thereby enhancing the network's sensitivity and adaptability to structural differences.

[0088] In this embodiment, the sparse alignment algorithm based on cross paths refers to constructing a sparse connection graph by utilizing the cross-overlap relationship of path indexes between nodes and hierarchical differences under the constraints of the structural path matrix, and performing local matching operations on the sparse connection graph to achieve structural consistency alignment between block-level slices.

[0089] This invention achieves an efficient alignment mechanism under structural hierarchy constraints by constructing a sparse connection graph and performing local matching operations based on structural path intersections, thereby improving the accuracy of structure perception and the ability to process complex web page structures.

[0090] Example 1:

[0091] To verify the feasibility of this invention in practice, it was applied to the webpage content extraction process of an information aggregation platform. This platform needs to process a large amount of diverse and complex webpage data every day, covering different types such as news, Q&A, products, comments, and forums. The pages often contain advertisements, invalid scripts, nested tags, and dynamically loaded content. Traditional extraction methods based on rule matching or fixed templates have problems such as poor generalization ability, serious noise interference, and unstable tag output, making it difficult to meet the needs of subsequent content tag generation, sentiment analysis, and recommendation engine calls.

[0092] In practical applications, the HTML source file of the webpage and its corresponding rendered image are first imported into the content processing engine. The system automatically performs DOM parsing and constructs a structural path matrix, extracting information such as the text content, styles, attributes, and interaction parameters of the nodes, and establishing a spatial index relationship between the visual region set and the DOM structure. Based on this, a semantic classification strategy is used to divide the nodes into text nodes, media nodes, and interactive nodes, and a semantic tag sequence is generated by combining the structural path. Subsequently, the system temporally aligns the node attribute vectors, tag vectors, and visual regions, and constructs a block-level slice sequence with context dependencies through a window sliding mechanism.

[0093] Each slice is fed into a gate-controlled recursive attention network for processing, achieving multimodal joint modeling and structure-guided attention distribution, ultimately forming a feature fusion sequence that integrates semantic, structural, and visual features. A cross-path sparse alignment algorithm is used to calculate cross-node consistency distance under the constraint of the structural path matrix, constructing a cohesion scoring function and selecting a set of slices with compact structure and dense semantics. Finally, the target slices are mapped to a unified content feature space, outputting web page content feature labels containing semantic tags, location indices, and structural hierarchies, which are used to drive downstream search recommendation modules and content classification engines.

[0094] During application, comparative tests were conducted with traditional rule-based matching models, pure text extraction models (BERT-Text-Only), and structured convolutional network models (GCN-Baseline). Evaluation was performed on four dimensions: feature extraction accuracy, structural consistency score, semantic label accuracy, and processing time. The total test sample consisted of 12,000 web page instances, covering four content types: news, Q&A, product details, and social commenting pages. The test results are shown in the table below:

[0095] Table 1 Comparison of the effects of different methods on web page content feature extraction tasks.

[0096]

[0097] As can be seen from the test data, the present invention significantly outperforms existing methods in terms of feature extraction accuracy, with an improvement of more than 10 percentage points; the structural consistency score reaches 0.86, effectively improving the accuracy of structural hierarchy alignment; the semantic label accuracy reaches 89.6%, solving the problems of semantic drift and misjudgment in traditional methods; and the processing time is controlled at 58.3 milliseconds per page, taking into account real-time requirements while ensuring effectiveness, making it suitable for large-scale deployment applications.

[0098] In summary, this invention demonstrates excellent performance in terms of structural recognition accuracy, semantic feature extraction precision, and processing efficiency. It effectively solves the problems of poor generalization, structural mismatch, and unstable tags that exist in traditional web page content feature extraction methods, and verifies its feasibility and engineering practical value in complex web page scenarios.

[0099] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for extracting features from internet webpage content based on artificial intelligence, characterized in that, Includes the following steps: S1. Obtain the HTML source file and rendered image of the Internet webpage and perform preprocessing; S2. Semantically classify the DOM node set, encode text nodes, media nodes and interactive nodes into three types of semantic identifiers, and construct a node label sequence aligned with the structure path matrix; S3. Synchronize the node attribute vector, node label sequence and visual region set in time to construct a cross-modal node sequence set, and use a window sliding mechanism to slice the node sequence into blocks. S4. Input the block-level slices into the gated recurrent attention network, extract the joint representation of node attributes, semantic labels and visual features in sequence, and perform attention-guided aggregation in combination with the structure path matrix to generate a feature fusion sequence. S5. Perform structural alignment operation on the feature fusion sequence, use the cross-path-based sparse alignment algorithm to calculate the cross-node consistency distance, and filter the target slice set with structural cohesion higher than the threshold. S6. Map the target slice set to a fixed-dimensional content feature space, and generate a set of web page content feature tags with semantic tags, location indexes and hierarchical structure based on node indexes and structural path numbers.

2. The method for extracting features of Internet webpage content based on artificial intelligence according to claim 1, characterized in that, The HTML source file preprocessing refers to: using a depth-first traversal-based DOM parsing method, visiting each child node sequentially from the root node, extracting the attribute information of each node, including tag type, text content, inline styles, class name, ID attribute, event binding attribute, and custom attribute fields, constructing a structural path vector based on the DOM hierarchy and parent-child relationship of the node, encoding the attribute information of all nodes into a multi-dimensional numerical vector set, and recording the hierarchy index and parent node index of each node in the DOM tree, generating a set of node attribute vectors and a structural path matrix.

3. The method for extracting Internet webpage content features based on artificial intelligence according to claim 1, characterized in that, The rendering image preprocessing refers to: using an image segmentation algorithm to perform pixel-level region division on the rendering image, identifying the text and image content area, function button area, input form area, and multimedia display area based on the web page rendering layout, extracting the boundary coordinates, width and height dimensions, and region classification labels of each visual region, and matching the center position of each visual region with the rendering position information of each DOM node in the HTML structure path matrix. Through position overlap and hierarchy consistency calculations, the DOM node index corresponding to each visual region is determined, establishing the index correspondence between the visual region set and the structure path matrix, and generating a visual region set for cross-modal alignment.

4. The method for extracting Internet webpage content features based on artificial intelligence according to claim 1, characterized in that, The gated recursive attention network consists of a gating unit, a recursive aggregation unit, and an attention fusion unit. The gating unit receives the attribute vector, label vector, and visual vector of each node in the block-level slice and dynamically adjusts the information channels based on the node type. The recursive aggregation unit uses residual connections and a gated loop mechanism to expand the time steps and fuse the context state. The attention fusion unit performs multi-head attention operations on the state vector output at each time step and guides the attention distribution update in conjunction with the structure path matrix, outputting a feature fusion sequence with enhanced inter-node dependencies.

5. The method for extracting Internet webpage content features based on artificial intelligence according to claim 1, characterized in that, S2 specifically includes: S21. Traverse the DOM node set, initially group the nodes according to the node tag type, extract nodes with text information as text nodes, extract nodes containing image, audio or video URL fields as media nodes, and extract nodes with user interaction attributes as interactive nodes. S22. Assign preset semantic identifiers to the text nodes, media nodes and interaction nodes respectively, wherein the text nodes are encoded as first-type semantic identifiers, the media nodes are encoded as second-type semantic identifiers, and the interaction nodes are encoded as third-type semantic identifiers, and embed the semantic identifier of each node as an independent label into the attribute vector of the corresponding node. S23. Combining the structural path matrix, based on the order and hierarchical position of the node index in the DOM structure, perform structural alignment processing on the semantic identifier of each node, and output the node label sequence.

6. The method for extracting Internet webpage content features based on artificial intelligence according to claim 1, characterized in that, S3 specifically includes: S31. Align the set of node attribute vectors, the sequence of node labels, and the set of visual regions according to the node index order in the structure path matrix. Combine the attribute vectors, semantic labels, and visual regions corresponding to the same node index into a unified node feature unit, arrange them according to the traversal order of the DOM structure, and construct a cross-modal node sequence. S32. Perform time synchronization processing on cross-modal node sequences. Based on the node's level depth in the DOM tree, the spatial layout order in the page, and the visibility state, adjust the node order and tag state to generate a standardized node sequence that maintains cross-modal semantic consistency. S33. Apply a fixed-length window sliding mechanism to the standardized node sequence, set the sliding step size and window size parameters, extract continuous node segments in sequence, and construct a block-level slice sequence with overlapping relationship for feature fusion operation.

7. The method for extracting Internet webpage content features based on artificial intelligence according to claim 1, characterized in that, S4 specifically includes: S41. Input the block-level slice sequence one by one into the gated recurrent attention network. Each node in the block-level slice contains an attribute vector, semantic label and visual features. The gated recurrent attention network receives the multimodal input of each node and maintains the temporal dependency structure between nodes. S42. At each time step, the fusion channel weights of the attribute vector, semantic label vector and visual feature vector are dynamically adjusted using a gating mechanism to weight and combine the multimodal inputs of the node, and the context states of the node before and after are recursively aggregated to form an intermediate fusion vector with contextual dependencies. S43. Combining the node hierarchy relationships and parent-child connection paths recorded in the structural path matrix, perform structure-guided multi-head attention operations and output feature fusion sequences through attention aggregation mechanisms.

8. The method for extracting features of Internet webpage content based on artificial intelligence according to claim 1, characterized in that, S5 specifically includes: S51. Combine the feature fusion sequences pairwise to construct a block-level slice pairing set. Using a sparse alignment algorithm based on cross paths, and under the constraint of the structural path matrix, calculate the cross-node consistency distance D between the i-th and j-th block-level slices. ij ; S52, The cross-node consistency distance D ij As input, a cohesion scoring function for all block-level slices in the structural consistency dimension is constructed to obtain the structural cohesion score of the i-th block-level slice. The calculation formula is as follows: Among them, S i This represents the structural cohesion score of the i-th block-level slice, where m is the total number of block-level slices, Z is the normalization factor, and κ(D) represents the structural cohesion score of the i-th block-level slice. ij ,θ) represents the distance threshold mapping function, when D ij The value is 1 when θ < 0, otherwise it is 0. γ is the feature distance attenuation coefficient, used to adjust the influence intensity of the average feature difference. i u j Let u be the average vector of the node features in the i-th and j-th block-level slices, respectively. i -u j ||1 represents the norm distance between average features, ψ(p i ,p j ) is the structural path similarity function, which measures the overall similarity between two slices on the structural path, and exp(·) represents the natural exponential function with base e; S53. Score the structural cohesion of each block-level slice. i Slices with scores higher than a preset threshold are selected to form a target slice set.

9. The method for extracting Internet webpage content features based on artificial intelligence according to claim 8, characterized in that, The cross-node consistency distance D ij The calculation formula is: Among them, D ij Let represent the consistency distance between the i-th and j-th block-level slices, and n represent the number of nodes in each block-level slice. Let W1 and W2 represent the feature fusion vectors of the k-th node in the i-th and j-th block-level slices, respectively. Let b represent the bias term, σ(·) represent the Sigmoid activation function, tanh(·) represent the hyperbolic tangent function, and ⊙ represent the Hadamard element-wise product. For path alignment kernel function, representing node and The structural path difference degree is a non-negative real number. The hierarchical difference normalization function represents the node. and The difference in hierarchical indexes in the DOM tree, ‖·‖2 represents the 2-norm operator.

10. The method for extracting Internet webpage content features based on artificial intelligence according to claim 8, characterized in that, The sparse alignment algorithm based on cross paths refers to constructing a sparse connection graph by utilizing the cross-overlap relationship of path indexes between nodes and hierarchical differences under the constraints of the structural path matrix, and performing local matching operations on the sparse connection graph to achieve structural consistency alignment between block-level slices.

Images