Universal two-stage webpage information extraction method and system based on multi-dimensional features

Abstract

The invention provides a universal two-stage webpage information extraction method and system based on multi-dimensional features, in the first stage, text features, XPath features, position features and node label features are obtained by analyzing text content and HTML structure content of a target webpage, and multi-dimensional node representation is learned through a bidirectional long-short-term memory model; node-level information extraction and webpage abstract extraction are realized; and in the second stage, vector representation is carried out on the webpage abstract based on a language model of a Transform architecture, and entity information identification in the node is realized through text range prediction. By designing the two-stage webpage information extraction process, the problem that in the prior art, only one type of information can be extracted is solved, and the technical effect of synchronous extraction of node-level and entity-level information is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of web page information extraction, and in particular to a universal two-stage web page information extraction method and system based on multi-dimensional features. Background Art

[0002] In today's digital age, the Internet has become an important platform for information dissemination in various fields. There are hundreds of millions of web pages on the Internet, and users need to obtain effective information from a variety of web pages to improve the efficiency of information transmission. For example, a product website needs to display the name, price, and merchant information of the product on the search page, and the web page content needs to be parsed and extracted in advance. In addition, web workers need to collect a large number of related web pages and extract various information from them to support various downstream applications. For example, knowledge graph construction, web page summary generation, personalized user recommendations, etc.

[0003] Traditional web page information extraction methods can no longer meet the growing needs of current users. Early web page information extraction was mainly based on Hypertext Markup Language (HTML) tag rule matching. For example, by parsing the HTML structure information of the target web page, <title>、< / title> <h1>、< / h1> <h2>Tags such as are used as regular matching basis to obtain information such as the title and body of the web page. This type of method can obtain accurate extraction results in a single domain name web page, but it cannot adapt to the hundreds of millions of domain name types today. In recent years, methods based on machine learning have emerged, which are mainly divided into two technical routes. The first type is to represent and model the text nodes in the web page, classify the attributes of the text nodes through multi-classification, and predict their categories. However, this type of method can only extract the entire node-level content, and cannot effectively extract the entity information inside the node. The second method is to serialize the HTML text, treat it as plain text information, and serialize and represent it through the encoder-decoder structure of the Transformer, and then extract text-level information. This method learns the global information of the web page through the attention mechanism, but because it ignores the HTML structure information of the web page, it still cannot adapt to the web page information extraction task based on the web page. Therefore, a unified web page information extraction paradigm is still in urgent need of being proposed. Summary of the invention

[0004] In view of the defects in the prior art, the object of the present invention is to provide a universal two-stage web page information extraction method and system based on multi-dimensional features.

[0005] The general two-stage web page information extraction method based on multi-dimensional features provided by the present invention includes:

[0006] Step S1: cutting and obtaining the structural relationship between text nodes through the HTML text of the target web page;

[0007] Step S2: Obtain three dimensions of discrete information features by parsing the HTML text of the web page, namely XPath features, position features and node label features;

[0008] Step S3: Using the CNN-BiLSTM model, obtain text features with different granularities and dependencies, and fuse the text features with discrete information features into a multi-dimensional node representation vector;

[0009] Step S4: predict the target web page node attribute information through a neural network multi-classifier to complete the first stage of web page node level information extraction;

[0010] Step S5: vectorize and represent the web page summary based on Transformer and pre-trained large language model;

[0011] Step S6: Perform serialized text range prediction on the representation vector of the summary part to achieve two-stage entity-level web page information extraction.

[0012] Preferably, the step S1 comprises:

[0013] Step S101: for the target web page structure, in combination with HTML basic coding rules, retain the basic HTML element tags, remove the format and style tags, and reconstruct the cropped HTML text;

[0014] Step S102: marking the structural relationships of the text nodes, including parent-child relationships and sibling relationships;

[0015] The step S2 comprises:

[0016] Step S201: parse the text XPath features, determine the vector dimension according to the node depth of the target web page, and implement XPath information representation through one-hot encoding. The representation vector is recorded as e XPATH , dimension is denoted as d XPATH ;

[0017] Step S202: Analyze the web page position feature and use the depth-first search algorithm to obtain the position number pos of the target node in the web page x , and calculate the overall relative position information by the following formula;

[0018]

[0019] Obtain dimension d through random encoding strategy POS The relative position representation vector e POS ;

[0020] Step S203: Parse the HTML tags of the web page and obtain the vector e through one-hot encoding TAG , dimension is denoted as d TAG ;

[0021] Step S204: concatenate the three types of discrete information representations to obtain a discrete information representation vector e d =[e XPATH , e POS , e TAG ], the vector dimension is d d =d XPATH +d POS +d TAG .

[0022] Preferably, step S3 comprises:

[0023] Step S301: Split the text information in the node into word level and character level, and represent them as X1=[w1, w2, ..., w L1 ] and X2=[c1,c2,...,c L2 ], the word-level information is initially represented by the pre-trained Word2Vec model, denoted as g w ;

[0024] Step S302: For character-level information, the character-level information is initially encoded using a one-hot method, and its final representation is obtained through a multi-layer convolutional neural network, denoted as h w ;

[0025] Step S303: concatenate the word-level and character-level representations to obtain a comprehensive text representation f w =[g w ;h w ] and encode it through the bidirectional LSTM model to obtain d w dimensional text representation vector e x ;

[0026] Step S304: Set the parent node e F 、child node e C 、Brother node e B The representation vectors are spliced ​​to obtain the final 4D w dimensional text representation vector e s =[e x ;e F ;e C ;e B ].

[0027] Preferably, the step S4 comprises:

[0028] Step S401: Concatenate the discrete information representation vector and the text representation vector into a multi-dimensional text feature vector e through a fully connected layer n , e n =[e s ;e d ], whose dimension is d n , d n =d s +d d ;

[0029] Step S402: define the attribute category to be extracted as A = [a0, a1, ...a M-1 , a M , a M+1 ], which includes M definition types and a summary a for two-stage information extraction M And other types of M+1 ;

[0030] Step S403: After the text feature vector and the attribute representation are concatenated, they are passed through the MLP layer, and the text attribute type is predicted through the softmax layer:

[0031]

[0032] During the training of the multi-attribute classification model, the cross-upper loss function is used for optimization:

[0033]

[0034] in, Represents the final vector representation of the node to be classified; Represents attribute a i Text vector representation of h i represents the output after passing through the MLP layer; MLP() represents a feedforward neural network used to calculate attribute probabilities; p i Indicates that the text node corresponds to attribute a i The predicted probability of; j is the variable in the summation process; represents the final predicted label of the node; p i Indicates that the text node corresponds to attribute a i The predicted probability of m，n Indicates that the nth node corresponds to the true label of the mth attribute; P m,n It represents the predicted probability of the nth base point corresponding to the mth attribute; m and n represent the nth node and the mth attribute respectively;

[0035] The step S5 comprises:

[0036] Step S501: Calculate the target webpage summary x obtained in step S4. desc , define the target attribute of the second stage extraction as s, and obtain the intermediate layer representation through the encoder part in the Transformer architecture as:

[0037] H = Encoder(s1, ..., s |s| , x1, ..., x |x| )

[0038] Among them, s |s| Represents the vector representation of the last token of attribute s; x |x| Indicates the target page summary part x desc The vector representation of the last token;

[0039] Step S502: Obtain the representation vector y of each token using the autoregressive method through the decoder part of the Transformer architecture i for:

[0040]

[0041] in, Decoder status.

[0042] Preferably, step S6 comprises:

[0043] Based on the fine-tuned large language model, the target attribute range is predicted for the text representation vector to obtain the target range [y bos ;y eos ], which is the target value of the second-stage web page information extraction. The specific fine-tuning method is achieved by optimizing the following loss function;

[0044]

[0045] Among them, x represents the input text vector, s represents the structural hint of information extraction, and θ e ,θ d Represent the encoder and decoder parameters respectively; represents the loss function used for fine-tuning; represents the fine-tuning dataset; logp(y|x,s; θ e ,θ d ) represents the conditional likelihood probability of predicting the label y under the current text and structure prompts.

[0046] The general two-stage web page information extraction system based on multi-dimensional features provided by the present invention includes:

[0047] Module M1: Cut and obtain the structural relationship between text nodes through the HTML text of the target web page;

[0048] Module M2: Obtain three dimensions of discrete information features by parsing the HTML text of the web page, namely XPath features, position features and node label features;

[0049] Module M3: Use the CNN-BiLSTM model to obtain text features with different granularities and dependencies, and fuse text features with discrete information features into a multi-dimensional node representation vector;

[0050] Module M4: Use neural network multi-classifiers to predict the target web page node attribute information and complete the first stage of web page node level information extraction;

[0051] Module M5: Vectorize the web page summary based on Transformer and pre-trained large language model;

[0052] Module M6: Serialize the text range prediction of the abstract representation vector to achieve two-stage entity-level web page information extraction.

[0053] Preferably, the module M1 comprises:

[0054] Module M101: Based on the target web page structure, combined with HTML basic coding rules, retain the basic HTML element tags, remove format and style tags, and reconstruct the cropped HTML text;

[0055] Module M102: Structural relationship annotation of text nodes, including parent-child relationship and sibling relationship;

[0056] The module M2 comprises:

[0057] Module M201: Parse the XPath features of the text and determine the vector dimension according to the node depth of the target web page. Use one-hot encoding to represent the XPath information. The representation vector is recorded as e XPATH , dimension is denoted as d XPATH ;

[0058] Module M202: Analyze the position features of the web page and use the depth-first search algorithm to obtain the position number pos of the target node in the web page x , and calculate the overall relative position information by the following formula;

[0059]

[0060] Obtain dimension d through random encoding strategy POS The relative position representation vector e POS ;

[0061] Module M203: Parse HTML tags on web pages and obtain vector e through one-hot encoding TAG , dimension is denoted as d TAG ;

[0062] Module M204: Concatenate the three types of discrete information representations to obtain the discrete information representation vector e d =[e XPATH , e POS , e TAG ], the vector dimension is d d =d XPATH +d PPS +d TAG .

[0063] Preferably, the module M3 comprises:

[0064] Module M301: Split the text information in the node into word level and character level, represented as X1 = [w1, w2, ..., w L1 ] and X2=[c1,c2,...,c L2 ], the word-level information is initially represented by the pre-trained Word2Vec model, denoted as g w ;

[0065] Module M302: For character-level information, the initial encoding is performed using a one-hot method, and its final representation is obtained through a multi-layer convolutional neural network, denoted as h w ;

[0066] Module M303: Concatenate word-level and character-level representations to obtain comprehensive text representations w =[g w ;h w ] and encode it through the bidirectional LSTM model to obtain d w dimensional text representation vector e x ;

[0067] Module M304: Set the parent node e F 、child node e C 、Brother node e B The representation vectors are spliced ​​to obtain the final 4D w dimensional text representation vector e s =[e x ;e F ;e C ;e B ].

[0068] Preferably, the module M4 comprises:

[0069] Module M401: Concatenate the discrete information representation vector and the text representation vector into a multi-dimensional text feature vector e through a fully connected layer n , e n =[e s ;e d ], whose dimension is d n , d n =d s +d d ;

[0070] Module M402: Define the attribute category to be extracted as A = [a0, a1, ...a M-1 , a M , a M+1 ], which includes M definition types and a summary a for two-stage information extraction M And other types of M+1 ;

[0071] Module M403: After the text feature vector and attribute representation are concatenated, they are passed through the MLP layer, and the text attribute type is predicted through the softmax layer:

[0072]

[0073] During the training of the multi-attribute classification model, the cross-upper loss function is used for optimization:

[0074]

[0075] in, Represents the final vector representation of the node to be classified; Represents attribute a i Text vector representation of h i represents the output after passing through the MLP layer; MLP() represents a feedforward neural network used to calculate attribute probabilities; p i Indicates that the text node corresponds to attribute a i The predicted probability of; j is the variable in the summation process; represents the final predicted label of the node; p i Indicates that the text node corresponds to attribute a i The predicted probability of m,n Indicates that the nth node corresponds to the true label of the mth attribute; p m,n It represents the predicted probability of the nth base point corresponding to the mth attribute; m and n represent the nth node and the mth attribute respectively;

[0076] The module M5 comprises:

[0077] Module M501: extract the target web page summary x obtained in module M4 desc , define the target attribute of the second stage extraction as s, and obtain the intermediate layer representation through the encoder part in the Transformer architecture as:

[0078] H = Encoder(s1, ..., s |s| , x1, ..., x |x| )

[0079] Among them, s |s| Represents the vector representation of the last token of attribute s; x |x| Indicates the target page summary part x desc The vector representation of the last token;

[0080] Module M502: Through the decoder part of the Transformer architecture, use autoregression to obtain the representation vector y of each token i for:

[0081]

[0082] in, Decoder status.

[0083] Preferably, the module M6 comprises:

[0084] Based on the fine-tuned large language model, the target attribute range is predicted for the text representation vector to obtain the target range [y bos ;y eos ], which is the target value of the second-stage web page information extraction. The specific fine-tuning method is achieved by optimizing the following loss function;

[0085]

[0086] Among them, x represents the input text vector, s represents the structural hint of information extraction, and θ e ,θ d Represent the encoder and decoder parameters respectively; represents the loss function used for fine-tuning; represents the fine-tuning dataset; logp(y|x, s; θ e ,θ d ) represents the conditional likelihood probability of predicting the label y under the current text and structure prompts.

[0087] Compared with the prior art, the present invention has the following beneficial effects:

[0088] (1) The present invention solves the problems of incomplete web page information utilization and poor node relevance in the past by integrating text information and web page structure, and realizes accurate web page node classification by combining text and rendering information;

[0089] (2) The present invention solves the problems of poor generalization and high labeling requirements in previous web page information extraction technologies by combining a pre-trained language model based on the Transformer architecture with a training paradigm of small sample fine-tuning, and realizes universal web page information extraction in multiple fields;

[0090] (3) The present invention solves the problem that the previous technology can only extract a single category of information by designing a two-stage web page information extraction process, and achieves the technical effect of synchronously extracting node-level and entity-level information. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0092] Figure 1 is a flow chart of the method of the present invention;

[0093] Figure 2 This is a flow chart of the first stage web page node attribute prediction algorithm of the present invention. DETAILED DESCRIPTION

[0094] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0095] Example 1

[0096] The present invention proposes a general two-stage web page information extraction method based on multi-dimensional features. The specific process is as follows: Figure 1 As shown in the figure. The first stage uses web page node classification technology to perform multi-attribute prediction by integrating and representing the text features, XPath features, location features, and label features of web page nodes. The second stage uses Transformer-based range prediction technology to serialize and represent web page summary information, thereby extracting entity-level information. The specific steps include:

[0097] Step S1: cutting and obtaining the structural relationship between text nodes through the HTML text of the target webpage, wherein the relationship includes the parent-child node relationship and the brother node relationship;

[0098] Step S2: Obtaining three dimensions of discrete information features by parsing the HTML text of the web page, wherein the information features include XPath features, position features, and node label features;

[0099] Step S3: Using the CNN-BiLSTM model, obtain text features with different granularities and dependencies, and fuse the text features with discrete information features into a multi-dimensional node representation vector;

[0100] Step S4: predict the target web page node attribute information through a neural network multi-classifier to complete the first stage of web page node level information extraction;

[0101] Step S5: vectorize and represent the web page summary based on Transformer and the pre-trained large language model T5 model;

[0102] Step S6: Perform serialized text range prediction on the representation vector of the summary part to achieve two-stage entity-level web page information extraction.

[0103] In step S1, the HTML code of the target web page is parsed, the style tags used to render the web page are removed, and the relationship between the text nodes of the web page is marked. The main contents are as follows:

[0104] Step S101: Prune and reconstruct the HTML text. Based on the target web page structure, the basic HTML coding rules are combined to retain the basic HTML element tags, such as, and remove the format and style tags, such as <strong>、 <front>, and reconstruct the trimmed HTML text;

[0105] Step S102: marking structural relationships of text nodes, including parent-child relationships and sibling relationships.

[0106] In step S2, independent features of web page nodes are obtained through multiple dimensions to perform vector modeling. The main contents are as follows:

[0107] Step S201: parse the text XPath features, such as / html / head / title, and determine the vector dimension according to the node depth of the target web page, and implement XPath information representation through one-hot encoding. The representation vector is recorded as e XPATH , dimension is denoted as d XPATH ;

[0108] Step S202: Analyze the web page position feature and use the depth-first search algorithm to obtain the position number pos of the target node in the web page x , and calculate the overall relative position information by the following formula:

[0109]

[0110] Obtain dimension d through random encoding strategy POS The relative position representation vector e POS ;

[0111] Step S203: Parse the HTML tags of the web page, for example, the <title> tag often represents the overall title of the web page. Obtain the vector e through one-hot encoding TAG , dimension is denoted as d TAG ;

[0112] Step S204: concatenate the three types of discrete information representations to obtain a discrete information representation vector e d =[e XPATH , e POS , e TAG ], the vector dimension is d d =d XPATH +d POS +d TAG .

[0113] In step S3, the semantic features are divided into character level and word level, and the text information modeling is performed through the CNN-BiLSTM model. The main contents are as follows:

[0114] Step S301: Split the text information in the node into word level and character level, and represent them as X1=[w1, w2, ..., w L1 ] and X2=[c1,c2,...,c L2 ] The word-level information is initially represented by the pre-trained Word2Vec model, denoted as g w ;

[0115] Step S302: For character-level information, the character-level information is initially encoded using a one-hot method, and its final representation is obtained through a multi-layer convolutional neural network, denoted as h w ;

[0116] Step S303: concatenate the word-level and character-level representations to obtain a comprehensive text representation f w =[g w ;h w ] and encode it through the bidirectional LSTM model to obtain d w dimensional text representation vector

[0117] Step S304: The parent node (e F ), child node (e c ), brother node (e B ) to obtain the final 4D w dimensional text representation vector e s =[e x ;e F ;e C ;e B ].

[0118] In step S4, attribute prediction is performed on web page nodes through a multi-classifier based on machine learning, such as Figure 2 The main contents are as follows:

[0119] Step S401: Concatenate the discrete information representation vector and the text representation vector into a multi-dimensional text feature vector e through a fully connected layer n , e n =[e s ;e d ], whose dimension is d n , d n =d s +d d ;

[0120] Step S402: define the attribute category to be extracted as A = [a0, a1, ...a M-1 , a M , a M+1 ], which includes M definition types and a summary a for two-stage information extraction M And other types of M+1 ;

[0121] Step S403: After the text feature vector and the attribute representation are concatenated, they are passed through the MLP layer, and the text attribute type is predicted through the softmax layer. Specifically,

[0122]

[0123] Additionally, during the training of the multi-attribute classification model, the following cross-over loss function is used for optimization:

[0124]

[0125] In step S5, the webpage summary is serialized into text representation through the Transformer encoding-decoding paradigm. The main contents are as follows:

[0126] Step S501: Calculate the target webpage summary x obtained in step S4. desc , define the target attribute of the second stage extraction as s, and obtain the intermediate layer representation through the encoder part in the Transformer architecture as:

[0127] H = Encoder(s1, ..., s |s| , x1, ..., x |x| ).

[0128] Step S502: Obtain the representation vector y of each token using the autoregressive method through the decoder part of the Transformer architecture i for:

[0129]

[0130] in, Decoder status.

[0131] In step S6, the text range prediction technology is used to perform two-stage entity-level information extraction on the web page summary content. The main contents are as follows:

[0132] Based on the fine-tuned large language model T5, the target attribute range is predicted for the text representation vector to obtain the target range [y bos ;y eos ], which is the target value of the second-stage web page information extraction. The specific fine-tuning method is achieved by optimizing the following loss function;

[0133]

[0134] Among them, x represents the input text vector, s represents the structural hint of information extraction, and θ e ,θ d Represent the encoder and decoder parameters respectively.

[0135] Example 2

[0136] The present invention also provides a universal two-stage web page information extraction system based on multi-dimensional features. The universal two-stage web page information extraction system based on multi-dimensional features can be realized by executing the process steps of the universal two-stage web page information extraction method based on multi-dimensional features, that is, technical personnel in this field can understand the universal two-stage web page information extraction method based on multi-dimensional features as a preferred implementation of the universal two-stage web page information extraction system based on multi-dimensional features.

[0137] The general two-stage web page information extraction system based on multi-dimensional features provided by the present invention includes:

[0138] Module M1: Cut and obtain the structural relationship between text nodes through the HTML text of the target web page;

[0139] Module M2: Obtain three dimensions of discrete information features by parsing the HTML text of the web page, namely XPath features, position features and node label features;

[0140] Module M3: Use the CNN-BiLSTM model to obtain text features with different granularities and dependencies, and fuse text features with discrete information features into a multi-dimensional node representation vector;

[0141] Module M4: Use neural network multi-classifiers to predict the target web page node attribute information and complete the first stage of web page node level information extraction;

[0142] Module M5: Vectorize the web page summary based on Transformer and pre-trained large language model;

[0143] Module M6: Serialize the text range prediction of the abstract representation vector to achieve two-stage entity-level web page information extraction.

[0144] The module M1 comprises:

[0145] Module M101: Based on the target web page structure, combined with HTML basic coding rules, retain the basic HTML element tags, remove format and style tags, and reconstruct the cropped HTML text;

[0146] Module M102: Structural relationship annotation of text nodes, including parent-child relationship and sibling relationship;

[0147] The module M2 comprises:

[0148] Module M201: Parse the XPath features of the text and determine the vector dimension according to the node depth of the target web page. Use one-hot encoding to represent the XPath information. The representation vector is recorded as e XPATH , dimension is denoted as d XPATH ;

[0149] Module M202: Analyze the position features of the web page and use the depth-first search algorithm to obtain the position number pos of the target node in the web page x , and calculate the overall relative position information by the following formula;

[0150]

[0151] Obtain dimension d through random encoding strategy POS The relative position representation vector e POS ;

[0152] Module M203: Parse HTML tags on web pages and obtain vector e through one-hot encoding TAG , dimension is denoted as d TAG ;

[0153] Module M204: Concatenate the three types of discrete information representations to obtain the discrete information representation vector e d =[e XPATH , e POS , e TAG ], the vector dimension is d d =d XPATH +d POS +d TAG .

[0154] The module M3 comprises:

[0155] Module M301: Split the text information in the node into word level and character level, represented as X1 = [w1, w2, ..., w L1 ] and X2=[c1,c2,...,c L2 ], the word-level information is initially represented by the pre-trained Word2Vec model, denoted as g w ;

[0156] Module M302: For character-level information, the initial encoding is performed using a one-hot method, and its final representation is obtained through a multi-layer convolutional neural network, denoted as h w ;

[0157] Module M303: Concatenate word-level and character-level representations to obtain comprehensive text representations w =[g w ;h w ] and encode it through the bidirectional LSTM model to obtain d w dimensional text representation vector e x ;

[0158] Module M304: Set the parent node e F 、child node e C 、Brother node e B The representation vectors are spliced ​​to obtain the final 4D w dimensional text representation vector e s =[e x ;e F ;e C ;e B ].

[0159] The module M4 comprises:

[0160] Module M401: Concatenate the discrete information representation vector and the text representation vector into a multi-dimensional text feature vector e through a fully connected layer n , e n =[e s ;e d ], whose dimension is d n , d n =d s +d d ;

[0161] Module M402: Define the attribute category to be extracted as A = [a0, a1, ...a M-1 , a M , a M+1 ], which includes M definition types and a summary a for two-stage information extraction M And other types of M+1 ;

[0162] Module M403: After the text feature vector and attribute representation are concatenated, they are passed through the MLP layer, and the text attribute type is predicted through the softmax layer:

[0163]

[0164] During the training of the multi-attribute classification model, the cross-upper loss function is used for optimization:

[0165]

[0166] in, Represents the final vector representation of the node to be classified; Represents attribute a i Text vector representation of h i represents the output after passing through the MLP layer; MLP() represents a feedforward neural network used to calculate attribute probabilities; p i Indicates that the text node corresponds to attribute a i The predicted probability of; j is the variable in the summation process; represents the final predicted label of the node; p i Indicates that the text node corresponds to attribute a i The predicted probability of m,n Indicates that the nth node corresponds to the true label of the mth attribute; P m,n It represents the predicted probability of the nth base point corresponding to the mth attribute; m and n represent the nth node and the mth attribute respectively;

[0167] The module M5 comprises:

[0168] Module M501: extract the target web page summary x obtained in module M4 desc , define the target attribute of the second stage extraction as s, and obtain the intermediate layer representation through the encoder part in the Transformer architecture as:

[0169] H = Encoder(s1, ..., s |s| , x1, ..., x |x| )

[0170] Among them, s |s| Represents the vector representation of the last token of attribute s; x |x| Indicates the target page summary part x desc The vector representation of the last token;

[0171] Module M502: Through the decoder part of the Transformer architecture, use autoregression to obtain the representation vector y of each token i for:

[0172]

[0173] in, Decoder status.

[0174] The module M6 comprises:

[0175] Based on the fine-tuned large language model, the target attribute range is predicted for the text representation vector to obtain the target range [y bos ;y eos ], which is the target value of the second-stage web page information extraction. The specific fine-tuning method is achieved by optimizing the following loss function;

[0176]

[0177] Among them, x represents the input text vector, s represents the structural hint of information extraction, and θ e ,θ d Represent the encoder and decoder parameters respectively; represents the loss function used for fine-tuning; represents the fine-tuning dataset; logp(y|x, s; θ e ,θ d ) represents the conditional likelihood probability of predicting the label y under the current text and structure prompts.

[0178] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0179] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.< / front> < / strong> < / h2>

Claims

1. A general two-stage web page information extraction method based on multi-dimensional features, characterized in that: include: Step S1: cutting and obtaining the structural relationship between text nodes through the HTML text of the target web page; Step S2: Obtain three dimensions of discrete information features by parsing the HTML text of the web page, namely XPath features, position features and node label features; Step S3: Using the CNN-BiLSTM model, obtain text features with different granularities and dependencies, and fuse the text features with discrete information features into a multi-dimensional node representation vector; Step S4: predict the target web page node attribute information through a neural network multi-classifier to complete the first stage of web page node level information extraction; Step S5: vectorize and represent the web page summary based on Transformer and pre-trained large language model; Step S6: Perform serialized text range prediction on the representation vector of the summary part to achieve two-stage entity-level web page information extraction.

2. The general two-stage web page information extraction method based on multi-dimensional features according to claim 1 is characterized in that: The step S1 comprises: Step S101: for the target web page structure, in combination with HTML basic coding rules, retain the basic HTML element tags, remove the format and style tags, and reconstruct the cropped HTML text; Step S102: marking the structural relationships of the text nodes, including parent-child relationships and sibling relationships; The step S2 comprises: Step S201: parse the text XPath features, determine the vector dimension according to the node depth of the target web page, and implement XPath information representation through one-hot encoding. The representation vector is recorded as e XPATH , dimension is denoted as d XPATH ; Step S202: Analyze the web page position feature and use the depth-first search algorithm to obtain the position number pos of the target node in the web page x , and calculate the overall relative position information by the following formula; Obtain dimension d through random encoding strategy POS The relative position representation vector e POS ; Step S203: Parse the HTML tags of the web page and obtain the vector e through one-hot encoding TAG , dimension is denoted as d TAG ; Step S204: concatenate the three types of discrete information representations to obtain a discrete information representation vector e d =[e XPATH , e POS , e TAG ], the vector dimension is d d =d XPATH +d PPs +d TAG .

3. The general two-stage web page information extraction method based on multi-dimensional features according to claim 2 is characterized in that: The step S3 comprises: Step S301: Split the text information in the node at word level and character level, and represent them as x1=[w1, w2, ..., w L1 ] and X2=[c1,c2,...,c L2 ], the word-level information is initially represented by the pre-trained Word2Vec model, denoted as g w ; Step S302: For character-level information, the character-level information is initially encoded using a one-hot method, and its final representation is obtained through a multi-layer convolutional neural network, denoted as h w ; Step S303: concatenate the word-level and character-level representations to obtain a comprehensive text representation f w =[g w ;h w ] and encode it through the bidirectional LSTM model to obtain d w dimensional text representation vector e x ; Step S304: Set the parent node e F 、child node e C 、Brother node e B The representation vectors are spliced ​​to obtain the final 4D w dimensional text representation vector e s =[e x ;e F ;e c ;e B ].

4. The general two-stage web page information extraction method based on multi-dimensional features according to claim 3 is characterized in that: The step S4 comprises: Step S401: Concatenate the discrete information representation vector and the text representation vector into a multi-dimensional text feature vector e through a fully connected layer n , e n =[e s ;e d ], whose dimension is d n , d n =d s +d d ; Step S402: define the attribute category to be extracted as A = [a0, a1, ...a M-1 , a M , a M+1 ], which includes M definition types and a summary a for two-stage information extraction M And other types of M+1 ; Step S403: After the text feature vector and the attribute representation are concatenated, they are passed through the MLP layer, and the text attribute type is predicted through the softmax layer: During the training of the multi-attribute classification model, the cross-upper loss function is used for optimization: in, Represents the final vector representation of the node to be classified; Represents attribute a i Text vector representation of h i represents the output after passing through the MLP layer; MLP() represents a feedforward neural network used to calculate attribute probabilities; p i Indicates that the text node corresponds to attribute a i The predicted probability of ; j is the variable in the summation process; represents the final predicted label of the node; p i Indicates that the text node corresponds to attribute a i The predicted probability of m，n Indicates that the nth node corresponds to the true label of the mth attribute; p m,n It represents the predicted probability of the nth base point corresponding to the mth attribute; m and n represent the nth node and the mth attribute respectively; The step S5 comprises: Step S501: Calculate the target webpage summary x obtained in step S4. desc , define the target attribute of the second stage extraction as s, and obtain the intermediate layer representation through the encoder part in the Transformer architecture as: H=Encoder(s1,...,s |s| ,x1,...,x |x| ) Among them, s |s| Represents the vector representation of the last token of attribute s; x |x| Indicates the target page summary part x desc The vector representation of the last token; Step S502: Obtain the representation vector y of each token using the autoregressive method through the decoder part of the Transformer architecture i for: in, Decoder status.

5. The general two-stage web page information extraction method based on multi-dimensional features according to claim 4 is characterized in that: The step S6 comprises: Based on the fine-tuned large language model, the target attribute range is predicted for the text representation vector to obtain the target range [y bos ;y eos ], which is the target value of the second-stage web page information extraction. The specific fine-tuning method is achieved by optimizing the following loss function; Among them, x represents the input text vector, s represents the structural hint of information extraction, and θ e ,θ d Represent the encoder and decoder parameters respectively; represents the loss function used for fine-tuning; represents the fine-tuning dataset; log p(y|x, s; θ e ,θ d ) represents the conditional likelihood probability of predicting the label y under the current text and structure prompts.

6. A universal two-stage web page information extraction system based on multi-dimensional features, characterized in that: include: Module M1: Cut and obtain the structural relationship between text nodes through the HTML text of the target web page; Module M2: Obtain three dimensions of discrete information features by parsing the HTML text of the web page, namely XPath features, position features and node label features; Module M3: Use the CNN-BiLSTM model to obtain text features with different granularities and dependencies, and fuse text features with discrete information features into a multi-dimensional node representation vector; Module M4: Use neural network multi-classifiers to predict the target web page node attribute information and complete the first stage of web page node level information extraction; Module M5: Vectorize the web page summary based on Transformer and pre-trained large language model; Module M6: Serialize the text range prediction of the abstract representation vector to achieve two-stage entity-level web page information extraction.

7. The universal two-stage web page information extraction system based on multi-dimensional features according to claim 6 is characterized in that: The module M1 comprises: Module M101: Based on the target web page structure, combined with HTML basic coding rules, retain the basic HTML element tags, remove format and style tags, and reconstruct the cropped HTML text; Module M102: Structural relationship annotation of text nodes, including parent-child relationship and sibling relationship; The module M2 comprises: Module M201: Parse the XPath features of the text and determine the vector dimension according to the node depth of the target web page. Use one-hot encoding to represent the XPath information. The representation vector is recorded as e XPATH , dimension is denoted as d XPATH ; Module M202: Analyze the position features of the web page and use the depth-first search algorithm to obtain the position number pos of the target node in the web page x , and calculate the overall relative position information by the following formula; Obtain dimension d through random encoding strategy POS The relative position representation vector e POS ; Module M203: Parse HTML tags on web pages and obtain vector e through one-hot encoding TAG , dimension is denoted as d TAG ; Module M204: Concatenate the three types of discrete information representations to obtain the discrete information representation vector e d =[e XPATH , e POs , e TAG ], the vector dimension is d d =d XPATH +d POs +d TAG .

8. The universal two-stage web page information extraction system based on multi-dimensional features according to claim 7 is characterized in that: The module M3 comprises: Module M301: Split the text information in the node into word level and character level, represented as X1 = [w1, w2, ..., w L1 ] and X2=[c1,c2,...,c L2 ], the word-level information is initially represented by the pre-trained Word2Vec model, denoted as g w ; Module M302: For character-level information, the initial encoding is performed using a one-hot method, and its final representation is obtained through a multi-layer convolutional neural network, denoted as h w ; Module M303: Concatenate word-level and character-level representations to obtain comprehensive text representations w =[g w ;h w ] and encode it through the bidirectional LSTM model to obtain d w dimensional text representation vector e x ; Module M304: Set the parent node e F 、child node e C 、Brother node e B The representation vectors are spliced ​​to obtain the final 4D w dimensional text representation vector e s =[e x ;e F ;e C ;e B ].

9. The universal two-stage web page information extraction system based on multi-dimensional features according to claim 8, characterized in that: The module M4 comprises: Module M401: Concatenate the discrete information representation vector and the text representation vector into a multi-dimensional text feature vector e through a fully connected layer n , e n =[e s ;e d ], whose dimension is d n , d n =d s +d d ; Module M402: Define the attribute category to be extracted as A = [a0, a1, ...a M-1 , a M , a M+1 ], which includes M definition types and a summary a for two-stage information extraction M And other types of M+1 ; Module M403: After the text feature vector and attribute representation are concatenated, they are passed through the MLP layer, and the text attribute type is predicted through the softmax layer: During the training of the multi-attribute classification model, the cross-upper loss function is used for optimization: in, Represents the final vector representation of the node to be classified; Represents attribute a i Text vector representation of h i represents the output after passing through the MLP layer; MLP() represents a feedforward neural network used to calculate attribute probabilities; p i Indicates that the text node corresponds to attribute a i The predicted probability of ; j is the variable in the summation process; represents the final predicted label of the node; p i Indicates that the text node corresponds to attribute a i The predicted probability of m，n Indicates that the nth node corresponds to the true label of the mth attribute; p m,n It represents the predicted probability of the nth base point corresponding to the mth attribute; m and n represent the nth node and the mth attribute respectively; The module M5 comprises: Module M501: extract the target web page summary x obtained in module M4 desc , define the target attribute of the second stage extraction as s, and obtain the intermediate layer representation through the encoder part in the Transformer architecture as: H=Encoder(s1,...,s |s| ,x1,...,x |x| ) Among them, s |s| Represents the vector representation of the last token of attribute s; x |x| Indicates the target page summary part x desc The vector representation of the last token; Module M502: Through the decoder part of the Transformer architecture, use autoregression to obtain the representation vector y of each token i for: in, Decoder status.

10. The universal two-stage web page information extraction system based on multi-dimensional features according to claim 9, characterized in that: The module M6 comprises: Based on the fine-tuned large language model, the target attribute range is predicted for the text representation vector to obtain the target range [y bos ;y eos ], which is the target value of the second-stage web page information extraction. The specific fine-tuning method is achieved by optimizing the following loss function; Among them, x represents the input text vector, s represents the structural hint of information extraction, and θ e ,θ d Represent the encoder and decoder parameters respectively; represents the loss function used for fine-tuning; represents the fine-tuning dataset; log p(y|x, s; θ e ,θ d ) represents the conditional likelihood probability of predicting the label y under the current text and structure prompts.

Images