General two-stage web information extraction method and system based on multi-dimensional features

Abstract

The application provides a general two-stage web page information extraction method and system based on multi-dimensional features, in the first stage, text features, XPath features, position features and node label features are obtained by analyzing the text content and HTML structure content of a target web page, multi-dimensional node representation is learned through a bidirectional long short-term memory model, and node-level information extraction and web page abstract extraction are realized; in the second stage, a language model based on a Transformer architecture is used for vector representation of the web page abstract, and entity information identification within a node is realized through text range prediction. Through the design of the two-stage web page information extraction process, the application solves the problem that the previous technology can only be used for single category information extraction, and realizes the technical effect of synchronous extraction of node-level and entity-level information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of web information extraction, and in particular to a general two-stage web information extraction method and system based on multi-dimensional features. BACKGROUND

[0002] In today's digital age, the Internet has become an important platform for information dissemination in various fields. There are hundreds of millions of web pages on the Internet, and users need to obtain effective information from the rich web page forms to improve information transmission efficiency. For example, a commodity website needs to display the name, price, and merchant information of a commodity on the search page, and needs to extract the web page content in advance. In addition, web workers need to collect a large number of related web pages and extract various information therefrom to support various downstream applications. For example, knowledge graph construction, web abstract generation, personalized user recommendation, etc.

[0003] Traditional web information extraction methods have been unable to meet the growing needs of users. Early web information extraction is mainly based on HyperText Markup Language (HTML) tag rule matching, for example, by parsing the HTML structure information of the target web page, and extracting the information of the target web page according to the HTML tag rules. <title>、< / title> <h1>、< / h1> <h2>The equal tags are taken as the regular matching basis to obtain the title, the text and other information in the webpage. This kind of method can obtain accurate extraction results in the single domain name webpage, but cannot adapt to the hundreds of millions of domain name types at present. In recent years, methods based on machine learning and the like appear, mainly divided into two technical routes, the first kind is to model the text nodes in the webpage, and the class of the text nodes is classified through multi-classification to predict the class. However, this kind of way can only extract the whole block node level content, and cannot effectively extract the entity information in the node. The second kind of method is to serialize the HTML text, and regard it as pure text information, and then the encoder-decoder structure of the Transformer is used for serializing the representation, and then the text level information extraction is carried out. This kind of way learns the global information of the webpage through the attention mechanism, but still cannot adapt to the webpage information extraction task based on the webpage because the HTML structure information of the webpage is ignored. Therefore, a unified webpage information extraction paradigm is still urgently needed to be put forward. SUMMARY

[0004] In view of the defects in the prior art, the purpose of the present application is to provide a universal two-stage webpage information extraction method and system based on multi-dimensional features.

[0005] The universal two-stage webpage information extraction method based on multi-dimensional features provided by the present application comprises:

[0006] Step S1: the structure relationship between the text nodes is clipped and obtained through the HTML text of the target webpage;

[0007] Step S2: three-dimensional discrete information features are obtained by analyzing the HTML text of the webpage, which are XPath features, position features and node label features;

[0008] Step S3: the CNN-BiLSTM model is used to obtain text features with different granularities and dependency relationships, and the text features and the discrete information features are fused into a multi-dimensional node representation vector;

[0009] Step S4: the neural network multi-classifier is used to predict the attribute information of the nodes of the target webpage, and the first-stage webpage node level information extraction is completed;

[0010] Step S5: the Transformer and the pre-training large language model are used to vectorize the representation of the abstract part of the webpage;

[0011] Step S6: the abstract part representation vector is used for sequence text range prediction to realize the second-stage entity level webpage information extraction.

[0012] Preferably, the step S1 comprises:

[0013] Step S101: for the target webpage structure, basic HTML element tags are retained in combination with HTML basic coding rules, format and style tags are removed, and the cropped HTML text is reconstructed;

[0014] Step S102: text nodes are marked for structural relationship, including parent-child relationship and sibling relationship;

[0015] The step S2 comprises:

[0016] Step S201: parse the text XPath feature, determine the vector dimension according to the node depth of the target webpage, realize the XPath information representation through one-hot coding, and represent the vector as e XPATH , and the dimension is d XPATH ;

[0017] Step S202: parse the webpage location feature, use the depth-first search algorithm to obtain the position sequence pos x of the target node in the webpage, and calculate the overall relative position information through the following formula;

[0018]

[0019] The relative position representation vector e POS with the dimension d POS is obtained through the random coding strategy;

[0020] Step S203: parse the webpage HTML tag, obtain the vector e TAG through one-hot coding, and the dimension is d TAG ;

[0021] Step S204: splice the three types of discrete information representation to obtain the discrete information representation vector e d =[e XPATH , e POS , e TAG ], and the vector dimension is d d =d XPATH +d POS +d TAG .

[0022] Preferably, the step S3 comprises:

[0023] Step S301: split the text information in the node at the word level and the character level, respectively represented as X1=[w1, w2,..., w L1 ] and X2=[c1, c2,..., c L2 ], and the word-level information is initially represented through a pre-trained Word2Vec model, represented as g w ;

[0024] Step S302: For character-level information, initial encoding is performed by one-hot method, and its final representation is obtained by a multi-layer convolutional neural network, denoted as h w ;

[0025] Step S303: The word-level and character-level representations are obtained by splicing to obtain comprehensive text representation f w = [g w ; h w ], and are encoded by a bidirectional LSTM model to obtain a d w -dimensional text representation vector e x ;

[0026] Step S304: The representation vectors of the parent node e F , the child node e C , and the sibling node e B are spliced to obtain a final 4d w -dimensional text representation vector e s = [e x ; e F ; e C ; e B ].

[0027] Preferably, the step S4 comprises:

[0028] Step S401: The discrete information representation vector and the text representation vector are spliced into a multi-dimensional text feature vector e n by a fully connected layer, e n = [e s ; e d ], with a dimension of d n , d n = d s + d d ;

[0029] Step S402: Define the attribute class to be extracted as A = [a0, a1,... a M-1 , a M , a M+1 ], which includes M defined types, as well as summary a M and other types a M+1 for two-stage information extraction;

[0030] Step S403: After splicing the text feature vector and the attribute representation, pass through an MLP layer, and predict the text attribute type by a softmax layer:

[0031]

[0032] In the multi-attribute classification model training process, a cross-entropy loss function is used for optimization:

[0033]

[0034] wherein, represents the final vector representation of the node to be classified; represents the text vector representation of attribute a i h i represents the output after passing through the MLP layer; MLP() represents a feedforward neural network used to calculate attribute probability; p i represents the predicted probability of the text node corresponding to attribute a i ; j is a variable in the summation process; represents the final predicted label of the node; p i represents the predicted probability of the text node corresponding to attribute a i ; y m，n represents the true label of the nth node corresponding to the mth attribute; P m,n represents the predicted probability of the nth node corresponding to the mth attribute; m, n represent the nth node and the mth attribute, respectively;

[0035] The step S5 comprises:

[0036] Step S501: For the target web abstract part x desc obtained in step S4, define the two-stage extraction target attribute as s, and obtain the intermediate layer representation through the encoder part in the Transformer architecture as:

[0037] H = Encoder (s1,..., s |s| , x1,..., x |x| )

[0038] wherein, s |s| represents the vector representation of the last token of attribute s; x |x| represents the vector representation of the last token of the target web abstract part x desc ;

[0039] Step S502: Through the decoder part in the Transformer architecture, the token-by-token representation vector y i is obtained in an autoregressive manner as:

[0040]

[0041] wherein, is the decoder state.

[0042] Preferably, the step S6 comprises:

[0043] The fine-tuning-based large language model performs target attribute range prediction on the text representation vector to obtain a target range [y bos ; y eos ], that is, a two-stage web information extraction target value, and the specific fine-tuning method is implemented by optimizing the following loss function.

[0044]

[0045] wherein x represents the input text vector, s represents the structure prompt of information extraction, and θ e , θ d represent the encoder and decoder parameters, respectively. represents the loss function used for fine-tuning. represents the fine-tuning data set; and logp(y|x, s; θ e , θ d ) represents the conditional likelihood probability that the predicted label is y under the current text and structure prompt.

[0046] The application provides a general two-stage web information extraction system based on multi-dimensional features, which comprises:

[0047] Module M1: The structural relationship between text nodes is obtained by clipping and acquiring the HTML text of the target web page.

[0048] Module M2: Three-dimensional discrete information features, namely XPath features, position features and node label features, are obtained by analyzing the HTML text of the web page.

[0049] Module M3: A CNN-BiLSTM model is used to obtain text features of different granularities with dependency relationships, and the text features and the discrete information features are fused into a multi-dimensional node representation vector.

[0050] Module M4: The target web node attribute information is predicted by a neural network multi-classifier to complete the first-stage web node-level information extraction.

[0051] Module M5: The summary part of the web page is vectorized based on the Transformer and the pre-trained large language model.

[0052] Module M6: The sequence text range prediction is performed on the summary part representation vector to realize the second-stage entity-level web information extraction.

[0053] Preferably, the module M1 comprises:

[0054] Module M101: For the target web structure, the basic HTML element tags are retained, the format and style tags are removed, and the clipped HTML text is reconstructed in combination with the HTML basic coding rules.

[0055] Module M102: structural relationship annotation of text nodes, including parent-child relationship and sibling relationship;

[0056] The module M2 comprises:

[0057] Module M201: parse text XPath features, and determine vector dimension according to node depth of the target webpage, realize XPath information representation through one-hot coding, and the representation vector is denoted as e XPATH , and the dimension is denoted as d XPATH ;

[0058] Module M202: parse webpage position features, use a depth-first search algorithm to obtain a position sequence number pos x of the target node in the webpage, and calculate the overall relative position information through the following formula;

[0059]

[0060] A relative position representation vector e POS with a dimension of d POS is obtained through a random coding strategy;

[0061] Module M203: parse webpage HTML tags, obtain a vector e TAG through one-hot coding, and the dimension is denoted as d TAG ;

[0062] Module M204: splice three types of discrete information representation, obtain a discrete information representation vector e d =[e XPATH , e POS , e TAG ], and the vector dimension is d d =d XPATH +d PPS +d TAG .

[0063] Preferably, the module M3 comprises:

[0064] Module M301: word-level and character-level splitting of the text information in the node, respectively represented as X1=[w1, w2,..., w L1 ] and X2=[c1, c2,..., c L2 ], initial representation of the word-level information is obtained through a pre-trained Word2Vec model, and is denoted as g w ;

[0065] Module M302: for the character-level information, initial coding is performed through one-hot, and the final representation thereof is obtained through a multi-layer convolutional neural network, and is denoted as h w ;

[0066] Module M303: obtain comprehensive text representation f by splicing word-level and character-level representations w = [g w ; h w ], and encode them through a bidirectional LSTM model to obtain d w -dimensional text representation vector e x ;

[0067] Module M304: splice the representation vectors of parent node e F , child node e C , and sibling node e B to obtain the final 4d w -dimensional text representation vector e s = [e x ; e F ; e C ; e B ].

[0068] Preferably, the module M4 comprises:

[0069] Module M401: splice the discrete information representation vector and the text representation vector through a fully connected layer to obtain a multi-dimensional text feature vector e n , e n = [e s ; e d ], with a dimension of d n , d n = d s + d d ;

[0070] Module M402: define the attribute class to be extracted as A = [a0, a1,... a M-1 , a M , a M+1 ], which includes M defined types, as well as summary a M and other types a M+1 ;

[0071] Module M403: splice the text feature vector and the attribute representation, then pass them through an MLP layer, and predict the text attribute type through a softmax layer:

[0072]

[0073] In the training process of the multi-attribute classification model, a cross-entropy loss function is used for optimization:

[0074]

[0075] in, Represents the final vector representation of the node to be classified; Represents attribute a i Text vector representation; h i Represents the output after passing through the MLP layer; MLP() represents the feedforward neural network used to calculate attribute probability; p i Indicates that the text node corresponds to attribute a i The predicted probability of ; j is the variable in the summation process; represents the final predicted label of the node; p i Indicates that the text node corresponds to attribute a i The predicted probability of y m,n Indicates that the nth node corresponds to the true label of the mth attribute; p m,n Indicates the predicted probability of the nth base point corresponding to the mth attribute; m, n represent the nth node and the mth attribute respectively;

[0076] The module M5 includes:

[0077] Module M501: The target web page summary part x obtained in module M4 desc , define the target attribute of the second stage extraction as s, and obtain the intermediate layer representation through the encoder part of the Transformer architecture as:

[0078] H=Encoder(s1,...,s |s| , x1, ..., x |x| )

[0079] Among them, s |s| Represents the vector representation of the last token of attribute s; x |x| Indicates the target page summary part x desc The vector representation of the last token;

[0080] Module M502: Obtain the token-by-token representation vector y using autoregressive method through the decoder part of the Transformer architecture i for:

[0081]

[0082] in, Decoder status.

[0083] Preferably, the module M6 includes:

[0084] Based on the fine-tuned large language model, the target attribute range is predicted for the text representation vector to obtain the target range [y bos ;y eos , i.e. the two-stage web page information extraction target value, and the specific fine-tuning method is realized by optimizing the following loss function:

[0085]

[0086] wherein x represents the input text vector, s represents the structure prompt of information extraction, and theta e , theta d respectively represent the encoder and decoder parameters; represents the loss function used for fine-tuning; represents the fine-tuning data set; logp(y|x, s; theta e , theta d ) represents the conditional likelihood probability that the predicted label is y under the current text and structure prompt.

[0087] Compared with the prior art, the present application has the following beneficial effects:

[0088] (1) The present application solves the problems of incomplete utilization of web page information and poor node correlation in the past by using the fusion representation technology of text information and web page structure, and realizes accurate web page node classification combining text and rendering information;

[0089] (2) The present application solves the problems of poor generalization and high labeling requirements in the past web page information extraction technology by using the training paradigm of pre-training language model based on Transformer architecture combined with small sample fine-tuning, and realizes multi-domain universal web page information extraction;

[0090] (3) The present application solves the problem that the past technology can only extract single category information by designing a two-stage web page information extraction process, and realizes the technical effect of synchronous extraction of node-level and entity-level information. BRIEF DESCRIPTION OF DRAWINGS

[0091] Other features, objects and advantages of the present application will become more apparent through reading the following detailed description of the non-limiting embodiments with reference to the accompanying drawings:

[0092] Figure 1 is the flow chart of the method of the present application;

[0093] Figure 2 is the flow chart of the first-stage web page node attribute prediction algorithm of the present application. DETAILED DESCRIPTION

[0094] The application will be described in detail below with specific examples. The following examples will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the application. These are within the scope of protection of the application.

[0095] Example 1

[0096] The application proposes a general two-stage web page information extraction method based on multi-dimensional features, and the specific process is as shown in Figure 1 The first stage uses web node classification technology to perform multi-attribute prediction by fusing text features, XPath features, position features, and label features of web nodes. The second stage uses a range prediction technology based on Transformer to serialize the web abstract information, thereby extracting entity-level information. Specifically, the following steps are included:

[0097] Step S1: The structure relationship between text nodes is clipped and obtained through the HTML text of the target web page, including parent-child node relationship and sibling node relationship;

[0098] Step S2: Three-dimensional discrete information features are obtained by analyzing the web HTML text, including XPath features, position features, and node label features;

[0099] Step S3: Use the CNN-BiLSTM model to obtain text features with different granularities and dependency relationships, and fuse the text features and discrete information features into a multi-dimensional node representation vector;

[0100] Step S4: Perform target web node attribute information prediction through a neural network multi-classifier to complete the first-stage web node-level information extraction;

[0101] Step S5: Vectorize the web abstract part based on Transformer and the pre-trained large language model T5 model;

[0102] Step S6: Perform sequence text range prediction on the abstract part representation vector to realize the second-stage entity-level web page information extraction.

[0103] In step S1, the HTML code of the target web page is parsed, the style tags used for rendering the web page are removed, and the relationship between the text nodes of the web page is labeled, and the main content is as follows:

[0104] Step S101: Prune and reconstruct the HTML text. For the structure of the target web page, basic HTML element tags such as,, and are retained in combination with the HTML basic coding rules, and format and style tags such as <strong>、 <front>and reconstruct the cropped HTML text;

[0105] Step S102: structure relationship annotation is performed on the text node, including parent-child relationship and sibling relationship.

[0106] In step S2, independent features of the webpage node are acquired through multiple dimensions, vector modeling is performed, and the main content is as follows:

[0107] Step S201: parse the text XPath feature, such as / html / head / title, determine the vector dimension according to the node depth of the target webpage, realize the XPath information representation through one-hot encoding, and the representation vector is denoted as e XPATH , and the dimension is denoted as d XPATH ;

[0108] Step S202: parse the webpage position feature, use the depth-first search algorithm to obtain the position sequence number pos x of the target node in the webpage, and calculate the overall relative position information through the following formula,

[0109]

[0110] The relative position representation vector e POS with a dimension of d POS is obtained through a random encoding strategy;

[0111] Step S203: parse the webpage HTML tag, for example, the <title> tag usually represents the title of the webpage. The vector e TAG is obtained through one-hot encoding, and the dimension is denoted as d TAG ;

[0112] Step S204: splice the three types of discrete information representation to obtain the discrete information representation vector e d =[e XPATH , e POS , e TAG ], and the vector dimension is d d =d XPATH +d POS +d TAG .

[0113] In step S3, the semantic features are divided into word level and character level, and the text information modeling is performed through the CNN-BiLSTM model, and the main content is as follows:

[0114] Step S301: the node text information is divided into word level and character level, and is represented as X1=[w1, w2,..., w L1 ] and X2=[c1, c2,..., c L2 ]. The word-level information is initially represented by a pre-trained Word2Vec model, denoted as g w ;

[0115] Step S302: For character-level information, initial encoding is performed by one-hot method, and its final representation is obtained by a multi-layer convolutional neural network, denoted as h w ;

[0116] Step S303: The word-level and character-level representations are obtained by concatenation to obtain comprehensive text representation f w = [g w ; h w ], and are encoded by a bidirectional LSTM model to obtain a d w -dimensional text representation vector

[0117] Step S304: The representation vectors of parent nodes (e F ), child nodes (e c ), and sibling nodes (e B ) are concatenated to obtain a final 4d w -dimensional text representation vector e s = [e x ; e F ; e C ; e B ].

[0118] In step S4, a multi-classifier based on machine learning is used to predict the attributes of the web nodes, as shown in FIG. 4, and the main content is as follows: Figure 2

[0119] Step S401: The discrete information representation vector and the text representation vector are concatenated by a fully connected layer into a multi-dimensional text feature vector e n , e n = [e s ; e d ], with a dimension of d n , d n = d s + d d ;

[0120] Step S402: Define the attribute categories to be extracted as A = [a0, a1,... a M-1 , a M , a M+1 ], which includes M defined types, as well as a summary a M and other types a M+1 ;

[0121] ​Step S403: The text feature vector is spliced with the attribute representation and then passed through the MLP layer, and the attribute type of the text is predicted through the softmax layer. Specifically,

[0122]

[0123] Additionally, in the multi-attribute classification model training process, the following cross-entropy loss function is used for optimization:

[0124]

[0125] In step S5, the summary part of the web page is serialized as text representation through the encoding-decoding paradigm of the Transformer. The main content is as follows:

[0126] Step S501: The target web page summary part x desc obtained in step S4 is input into the Transformer architecture to obtain the intermediate layer representation s

[0127] H = Encoder (s1,..., s |s| , x1,..., x |x| ).

[0128] Step S502: Through the decoder part of the Transformer architecture, the token-by-token representation vector y i is obtained using the autoregressive method as follows:

[0129]

[0130] where, is the decoder state.

[0131] In step S6, the text range prediction technology is used to perform two-stage entity-level information extraction on the web page summary content. The main content is as follows:

[0132] Based on the fine-tuned large language model T5, the target attribute range prediction is performed on the text representation vector to obtain the target range [y bos ; y eos ], which is the two-stage web information extraction target value. The fine-tuning method is realized by optimizing the following loss function:

[0133]

[0134] where x represents the input text vector, s represents the structure prompt for information extraction, and θ e , θ d represent the encoder and decoder parameters, respectively.

[0135] Embodiment 2

[0136] The application also provides a general two-stage web page information extraction system based on multi-dimensional features, which can be realized by performing the process steps of the general two-stage web page information extraction method based on multi-dimensional features, that is, the general two-stage web page information extraction method based on multi-dimensional features can be understood by those skilled in the art as the preferred embodiment of the general two-stage web page information extraction system based on multi-dimensional features.

[0137] The general two-stage web page information extraction system based on multi-dimensional features provided by the application comprises:

[0138] Module M1: The structure relationship between text nodes is cut and obtained through the HTML text of the target web page.

[0139] Module M2: Three-dimensional discrete information features are obtained by analyzing the web page HTML text, which are XPath features, position features and node label features.

[0140] Module M3: A CNN-BiLSTM model is used to obtain text features of different granularities with dependency relationship, and the text features and the discrete information features are fused into a multi-dimensional node representation vector.

[0141] Module M4: The target web page node attribute information is predicted by a neural network multi-classifier, and the first-stage web page node-level information extraction is completed.

[0142] Module M5: The web page abstract part is vectorized based on the Transformer and the pre-trained large language model.

[0143] Module M6: The abstract part representation vector is subjected to sequence text range prediction, and the second-stage entity-level web page information extraction is realized.

[0144] The module M1 comprises:

[0145] Module M101: For the target web page structure, the basic HTML element tags are retained in combination with the HTML basic coding rules, the format and style tags are removed, and the cut HTML text is reconstructed.

[0146] Module M102: The text nodes are subjected to structure relationship annotation, including parent-child relationship and sibling relationship.

[0147] The module M2 comprises:

[0148] Module M201: parse the text XPath feature, and determine the vector dimension according to the node depth of the target webpage, realize the XPath information representation through one-hot encoding, and the representation vector is denoted as e XPATH , and the dimension is denoted as d XPATH ;

[0149] Module M202: parse the webpage location feature, use the depth-first search algorithm to obtain the position sequence number pos x of the target node in the webpage, and calculate the overall relative position information through the following formula;

[0150]

[0151] The relative position representation vector e POS with a dimension of d POS is obtained through a random encoding strategy;

[0152] Module M203: parse the webpage HTML tag, obtain the vector e TAG through one-hot encoding, and the dimension is denoted as d TAG ;

[0153] Module M204: splice the three types of discrete information representation to obtain the discrete information representation vector e d =[e XPATH , e POS , e TAG ], and the vector dimension is d d =d XPATH +d POS +d TAG .

[0154] The module M3 comprises:

[0155] Module M301: word-level and character-level splitting of the text information in the node, respectively represented as X1=[w1, w2,..., w L1 ] and X2=[c1, c2,..., c L2 ], the word-level information is initially represented through a pre-trained Word2Vec model, denoted as g w ;

[0156] Module M302: for the character-level information, initially encode through one-hot, and obtain the final representation through a multi-layer convolutional neural network, denoted as h w ;

[0157] Module M303: obtain the comprehensive text representation f w =[g w ; h w ] and encoded by a bidirectional LSTM model to obtain d w dimensional text feature vector e x ;

[0158] Module M304: concatenate the feature vectors of the parent node e F , the child node e C , and the sibling node e B to obtain a final 4d w dimensional text feature vector e s = [e x ; e F ; e C ; e B ].

[0159] The module M4 includes:

[0160] Module M401: concatenate the discrete information feature vector and the text feature vector through a fully connected layer to obtain a multi-dimensional text feature vector e n , e n = [e s ; e d ], with a dimension of d n , d n = d s + d d ;

[0161] Module M402: define the attribute class to be extracted as A = [a0, a1,... a M-1 , a M , a M+1 ], which includes M defined types, an abstract a M for two-stage information extraction, and other types a M+1 ;

[0162] Module M403: concatenate the text feature vector and the attribute representation, then pass it through an MLP layer, and predict the text attribute type through a softmax layer:

[0163]

[0164] In the training process of the multi-attribute classification model, a cross-entropy loss function is used for optimization:

[0165]

[0166] wherein, represents the final vector representation of the node to be classified; represents the text vector representation of the attribute a i ; h i represents the output after passing through the MLP layer; MLP() represents a feedforward neural network for computing attribute probabilities; p i represents the predicted probability of the text node corresponding to attribute a i ; j is a variable in the summation process; represents the final predicted label of the node; p i represents the predicted probability of the text node corresponding to attribute a i ; y m,n represents the true label of the nth node corresponding to the mth attribute; P m,n represents the predicted probability of the nth node corresponding to the mth attribute; m, n represent the nth node and the mth attribute, respectively;

[0167] The module M5 includes:

[0168] Module M501: The target web abstract part x desc acquired in module M4 is defined as s, and the intermediate layer representation is obtained through the encoder part in the Transformer architecture:

[0169] H = Encoder (s1,..., s |s| , x1,..., x |x| )

[0170] where s |s| represents the vector representation of the last token of attribute s; x |x| represents the vector representation of the last token of target web abstract part x desc ;

[0171] Module M502: The token-by-token representation vector y i is obtained through the decoder part in the Transformer architecture using the autoregressive method:

[0172]

[0173] where, is the decoder state.

[0174] The module M6 includes:

[0175] Based on the fine-tuned large language model, the text representation vector is used for target attribute range prediction to obtain the target range [y bos ; y eos ], which is the target value of two-stage web information extraction. The specific fine-tuning method is realized by optimizing the following loss function:

[0176]

[0177] wherein x represents the input text vector, s represents the structure prompt of information extraction, and θ e , θ d respectively represent the encoder and decoder parameters; represents the loss function used for fine-tuning; represents the fine-tuning data set; logp(y|x, s; θ e , θ d ) represents the conditional likelihood probability of predicting the label y under the current text and structure prompt.

[0178] Those skilled in the art know that, in addition to implementing the system, device and each module thereof provided by the present application in the form of pure computer readable program code, the same program can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. by logically programming the method steps. Therefore, the system, device and each module thereof provided by the present application can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing methods and structures within hardware components.

[0179] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. The embodiments of the present application and the features in the embodiments can be arbitrarily combined with each other without conflict.< / front> < / strong> < / h2>

Claims

1. A general two-stage web page information extraction method based on multi-dimensional features, characterized in that: include: Step S1: cutting and obtaining the structural relationship between text nodes through the HTML text of the target web page; Step S2: Obtain three dimensions of discrete information features by parsing the HTML text of the web page, namely XPath features, position features and node label features; Step S3: Using the CNN-BiLSTM model, we obtain text features of different granularities and dependencies, and fuse the text features with discrete information features into a multi-dimensional node representation vector. Step S4: Using a neural network multi-classifier, predict the target web page node attribute information to complete the first stage of web page node-level information extraction; Step S5: Vectorize the web page summary based on Transformer and pre-trained large language model; Step S6: perform serialized text range prediction on the representation vector of the summary part to achieve two-stage entity-level web page information extraction; The step S1 comprises: Step S101: Based on the target web page structure, in combination with basic HTML coding rules, retain the basic HTML element tags, remove format and style tags, and reconstruct the cropped HTML text; Step S102: marking the structural relationships of text nodes, including parent-child relationships and sibling relationships; The step S2 comprises: Step S201: Parse the XPath features of the text and determine the vector dimension according to the node depth of the target web page. Use one-hot encoding to represent the XPath information. The representation vector is recorded as e XPATH , dimension is denoted as d XPATH ; Step S202: Analyze the position feature of the web page and use the depth-first search algorithm to obtain the position number pos of the target node in the web page x , and calculate the overall relative position information by the following formula; Obtain dimension d through random encoding strategy POS The relative position representation vector e POS ; Step S203: Parse the HTML tags of the web page and obtain the vector e by one-hot encoding TAG , dimension is denoted as d TAG ; Step S204: concatenate the three types of discrete information representations to obtain a discrete information representation vector e d =[e XPATH ,e POS ,e TAG ], the vector dimension is d d =d XPATH +d POS +d TAG ; The step S3 comprises: Step S301: Split the text information in the node into word level and character level, and represent them as X1=[w1,w2,…,w L1 ] and X2=[c1,c2,…,c L2 ], the word-level information is initially represented by the pre-trained Word2Vec model, denoted as g w ; Step S302: For character-level information, perform initial encoding using a one-hot method and obtain its final representation through a multi-layer convolutional neural network, denoted as h w ; Step S303: Concatenate word-level and character-level representations to obtain a comprehensive text representation f w =[g w ;h w ] and encode it through the bidirectional LSTM model to obtain d w dimensional text representation vector e x ; Step S304: The parent node e F , child node e C 、Brother node e B The representation vectors are spliced ​​to obtain the final 4d w dimensional text representation vector e s =[e x ;e F ;e C ;e B ]; The step S4 comprises: Step S401: Concatenate the discrete information representation vector and the text representation vector through a fully connected layer into a multi-dimensional text feature vector e n , e n =[e s ;e d ], whose dimension is d n , d n =d s +d d ; Step S402: define the attribute category to be extracted as A=[a0, a1, ...a M-1 ,a M ,a M+1 ], which includes M definition types and a summary a for two-stage information extraction M and other types of M+1 ; Step S403: The text feature vector and the attribute representation are concatenated and passed through the MLP layer, and the text attribute type is predicted through the softmax layer: During the training of the multi-attribute classification model, the cross entropy loss function is used for optimization: in, Represents the final vector representation of the node to be classified; Represents attribute a i Text vector representation; h i Represents the output after passing through the MLP layer; MLP() represents the feedforward neural network used to calculate attribute probability; p i Indicates that the text node corresponds to attribute a i The predicted probability of ; j is the variable in the summation process; Represents the final predicted label of the node; y m,n Indicates that the nth node corresponds to the true label of the mth attribute; p m,n Indicates the predicted probability of the nth base point corresponding to the mth attribute; m, n represent the nth node and the mth attribute respectively; The step S5 comprises: Step S501: The target webpage summary part x obtained in step S4 is desc , define the target attribute of the second stage extraction as s, and obtain the intermediate layer representation through the encoder part of the Transformer architecture as: H=Encoder(s1,…,s |s| ,x1,…,x |x| ) Among them, s |s| Represents the vector representation of the last token of attribute s; x |x| Indicates the target page summary part x desc The vector representation of the last token; Step S502: Obtain the token-by-token representation vector y using the autoregressive method through the decoder part of the Transformer architecture i for: in, is the decoder status; The step S6 comprises: Based on the fine-tuned large language model, the target attribute range is predicted for the text representation vector to obtain the target range [y bos ;y eos ], which is the target value of the second-stage web page information extraction. The specific fine-tuning method is achieved by optimizing the following loss function; Among them, x represents the input text vector, s represents the structural prompt of information extraction, and θ e ,θ d Represent the encoder and decoder parameters respectively; represents the loss function used for fine-tuning; represents the fine-tuning dataset; logp(y|x,s; θ e ,θ d ) represents the conditional likelihood probability of predicting the label y under the current text and structural prompts.

2. A general two-stage web page information extraction system based on multi-dimensional features, characterized by: include: Module M1: Clip and obtain the structural relationship between text nodes through the HTML text of the target web page; Module M2: Obtain three dimensions of discrete information features by parsing the HTML text of the web page, namely XPath features, position features and node label features; Module M3: Use the CNN-BiLSTM model to obtain text features of different granularities and dependencies, and fuse text features with discrete information features into a multi-dimensional node representation vector; Module M4: Use neural network multi-classifiers to predict the attribute information of target web page nodes and complete the first stage of web page node-level information extraction; Module M5: Vectorized representation of web page summaries based on Transformer and pre-trained large language models; Module M6: Serializes text range prediction for the representation vector of the summary to achieve two-stage entity-level web page information extraction; The module M1 includes: Module M101: Based on the target web page structure, combine the basic HTML coding rules, retain the basic HTML element tags, remove the format and style tags, and reconstruct the cropped HTML text; Module M102: Mark the structural relationships of text nodes, including parent-child and sibling relationships; The module M2 includes: Module M201: Parse the XPath features of the text and determine the vector dimension based on the node depth of the target web page. Use one-hot encoding to represent the XPath information. The representation vector is recorded as e XPATH , dimension is denoted as d XPATH ; Module M202: Analyze the position features of the web page and use the depth-first search algorithm to obtain the position number pos of the target node in the web page x , and calculate the overall relative position information by the following formula; Obtain dimension d through random encoding strategy POS The relative position representation vector e POS ; Module M203: Parse HTML tags on web pages and obtain vector e through one-hot encoding TAG , dimension is denoted as d TAG ; Module M204: Concatenate the three types of discrete information representations to obtain the discrete information representation vector e d =[e XPATH ,e POS ,e TAG ], the vector dimension is d d =d XPATH +d POS +d TAG ; The module M3 includes: Module M301: Split the text information in the node at word level and character level, and represent them as X1=[w1,w2,…,w L1 ] and X2=[c1,c2,…,c L2 ], the word-level information is initially represented by the pre-trained Word2Vec model, denoted as g w ; Module M302: For character-level information, the initial encoding is performed using a one-hot method, and its final representation is obtained through a multi-layer convolutional neural network, denoted as h w ; Module M303: Concatenate word-level and character-level representations to obtain comprehensive text representations f w =[g w ;h w ] and encode it through the bidirectional LSTM model to obtain d w dimensional text representation vector e x ; Module M304: parent node e F , child node e C 、Brother node e B The representation vectors are spliced ​​to obtain the final 4d w dimensional text representation vector e s =[e x ;e F ;e C ;e B ]; The module M4 includes: Module M401: Concatenate the discrete information representation vector and the text representation vector into a multi-dimensional text feature vector e through a fully connected layer n , e n =[e s ;e d ], whose dimension is d n , d n =d s +d d ; Module M402: Define the attribute category to be extracted as A = [a0, a1, ...a M-1 ,a M ,a M+1 ], which includes M definition types and a summary a for two-stage information extraction M and other types of M+1 ; Module M403: After concatenating the text feature vector and the attribute representation, it passes through the MLP layer and predicts the text attribute type through the softmax layer: During the training of the multi-attribute classification model, the cross entropy loss function is used for optimization: in, Represents the final vector representation of the node to be classified; Represents attribute a i Text vector representation; h i Represents the output after passing through the MLP layer; MLP() represents the feedforward neural network used to calculate attribute probability; p i Indicates that the text node corresponds to attribute a i The predicted probability of ; j is the variable in the summation process; Represents the final predicted label of the node; y m,n Indicates that the nth node corresponds to the true label of the mth attribute; p m,n Indicates the predicted probability of the nth base point corresponding to the mth attribute; m, n represent the nth node and the mth attribute respectively; The module M5 includes: Module M501: The target web page summary part x obtained in module M4 desc , define the target attribute of the second stage extraction as s, and obtain the intermediate layer representation through the encoder part of the Transformer architecture as: H=Encoder(s1,…,s |s| ,x1,…,x |x| ) Among them, s |s| Represents the vector representation of the last token of attribute s; x |x| Indicates the target page summary part x desc The vector representation of the last token; Module M502: Obtain the token-by-token representation vector y using autoregressive method through the decoder part of the Transformer architecture i for: in, is the decoder status; The module M6 includes: Based on the fine-tuned large language model, the target attribute range is predicted for the text representation vector to obtain the target range [y bos ;y eos ], which is the target value of the second-stage web page information extraction. The specific fine-tuning method is achieved by optimizing the following loss function; Among them, x represents the input text vector, s represents the structural prompt of information extraction, and θ e ,θ d Represent the encoder and decoder parameters respectively; represents the loss function used for fine-tuning; represents the fine-tuning dataset; logp(y|x,s; θ e ,θ d ) represents the conditional likelihood probability of predicting the label y under the current text and structural prompts.

Images