A method and apparatus for extracting text of interest in a web page

Abstract

The application provides a method and device for extracting text of interest in a webpage. The method can accurately extract text of interest corresponding to a to-be-matched term from a webpage region based on various statistical features and a semantic matching model of the webpage, and is suitable for any structure and template of the webpage. The method comprises: obtaining a DOM tree of a to-be-processed webpage; determining a decision rule according to the content of the to-be-matched term, wherein the decision rule is used to define conditions that need to be met by statistical features of a candidate DOM block corresponding to the to-be-matched term, and the DOM block is a subtree of the DOM tree; screening candidate DOM blocks that meet the conditions in the decision rule according to the statistical features of each DOM block containing text; determining a target DOM block that is semantically matched with the to-be-matched term from at least one candidate DOM block by using a semantic matching model, wherein the semantic matching model is based on a Transformer structure; and extracting text of interest corresponding to the to-be-matched term in a webpage region corresponding to the target DOM block.

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, and in particular to a method and apparatus for extracting text of interest from a webpage. Background Technology

[0002] Web pages contain a wealth of information, such as main text, advertising text, copyright notices, and comments. On bidding and tendering web pages, users are typically only interested in a portion of the main text, such as the publisher's name, publication date, project name, and project type. The "project name" and "publication date" can be considered terms, while the detailed text corresponding to the "project name" (e.g., "Sports XXXX") and the detailed text corresponding to the "publication date" (e.g., "Year XX Month XX Day") can be considered text of interest.

[0003] Currently, there is a lack of solutions for extracting interesting text from web pages. Summary of the Invention

[0004] This application provides a method and apparatus for extracting text of interest from a webpage. Based on various statistical features and semantic matching models of the webpage, it can accurately extract the text of interest corresponding to the terms to be matched from the webpage area, and is applicable to webpages of any structure and template.

[0005] Firstly, a method for extracting text of interest from a webpage is provided, including:

[0006] Obtain the DOM tree of the webpage to be processed;

[0007] The decision rules are determined based on the content of the terms to be matched. The decision rules are used to specify the conditions that the statistical characteristics of the candidate DOM blocks corresponding to the terms to be matched must meet. The DOM blocks are subtrees of the DOM tree.

[0008] Based on the statistical characteristics of each DOM block containing text, at least one candidate DOM block that meets the conditions in the decision rules is selected.

[0009] A semantic matching model is used to determine the target DOM block that semantically matches the term to be matched from at least one candidate DOM block. The semantic matching model is based on the Transformer structure.

[0010] Extract the text of interest corresponding to the terms to be matched from the web page area corresponding to the target DOM block.

[0011] In one feasible design, based on the statistical characteristics of each DOM block containing text, at least one candidate DOM block that satisfies the conditions in the decision rule is selected, including:

[0012] Perform a depth-first traversal of the DOM tree to determine the statistical characteristics of each DOM block containing text. The statistical characteristics include one or more of the following features: non-linked text length, punctuation length in non-linked tags, punctuation density in non-linked text, linked text length, linked text density, and non-linked text density.

[0013] Based on the conditions in the decision rules, at least one candidate DOM block that meets the conditions is selected from each DOM block containing text.

[0014] In a feasible design, the semantic matching model includes a first encoding layer based on the Transformer structure, a second encoding layer based on the Transformer structure, an attention layer, a first pooling layer, a second pooling layer, a feature fusion layer, and an output layer;

[0015] The process involves using a semantic matching model to determine the target DOM block that semantically matches the term to be matched from at least one candidate DOM block, including:

[0016] Iterate through at least one candidate DOM block, and during the iteration of each candidate DOM block:

[0017] Input the term to be matched into the first coding layer to obtain the first coding vector;

[0018] Input the text corresponding to the current candidate DOM block into the second encoding layer to obtain the second encoding vector;

[0019] An attention layer is used to perform attention alignment on the first and second encoding vectors based on an attention mechanism, resulting in a first attention-weighted vector and a second attention-weighted vector, respectively.

[0020] The first attention weighted vector is pooled using the first pooling layer to obtain the first pooling vector;

[0021] The second attention weighted vector is pooled using the second pooling layer to obtain the second pooling vector;

[0022] The first pooling vector and the second pooling vector are fused using a feature fusion layer to obtain a feature fusion vector;

[0023] Input the feature fusion vector into the output layer to obtain the classification label. The classification label is used to identify whether the text corresponding to the current candidate DOM block matches the term to be matched.

[0024] In a feasible design, obtaining the DOM tree of the webpage to be processed includes:

[0025] The source code of the webpage to be processed is obtained through a web crawler program after being rendered by the browser.

[0026] Construct the DOM tree of the webpage to be processed based on its source code.

[0027] In a feasible design, before performing a depth-first traversal of the DOM tree to determine the statistical characteristics of each DOM block containing text, the method also includes:

[0028] Use regular expressions to delete the DOM nodes and their descendant nodes that correspond to noise information in the DOM tree.

[0029] In a feasible design, before performing a depth-first traversal of the DOM tree to determine the statistical characteristics of each DOM block containing text, the method also includes:

[0030] Merge DOM nodes that are adjacent in the DOM tree and have the same or similar labels.

[0031] In one feasible design, a feature fusion layer is used to fuse the similarity of the first pooling vector and the second pooling vector to obtain a feature fusion vector, including:

[0032] The feature fusion layer performs a subtraction operation on the first pooling vector and the second pooling vector to obtain the difference feature vector;

[0033] The first pooling vector and the second pooling vector are multiplied by a feature fusion layer to obtain the interactive feature vector.

[0034] The first pooling vector, the second pooling vector, the difference feature vector, and the interaction feature vector are concatenated to obtain the feature fusion vector.

[0035] In one feasible design, a first pooling layer is used to pool the first attention-weighted vector to obtain a first pooling vector, which includes:

[0036] The first attention weighted vector is averaged using the first pooling layer to obtain the first pooling vector;

[0037] And / or,

[0038] The second attention weighted vector is pooled using a second pooling layer to obtain a second pooled vector, which includes:

[0039] The second attention weighted vector is averaged using a second pooling layer to obtain the second pooling vector.

[0040] In a feasible design, the method also includes:

[0041] Obtain the training dataset, which includes different terms, text that matches the semantics of each term, and corresponding classification labels;

[0042] Based on minimizing the cross-entropy loss function, the semantic matching model is trained using the training dataset to obtain a well-trained semantic matching model.

[0043] Secondly, an apparatus for extracting text of interest from a webpage is provided, comprising:

[0044] The DOM tree retrieval module is used to retrieve the DOM tree of the webpage to be processed.

[0045] The decision rule determination module is used to determine the decision rules based on the content of the terms to be matched. The decision rules are used to specify the conditions that the statistical characteristics of the candidate DOM blocks corresponding to the terms to be matched must meet. The DOM blocks are subtrees of the DOM tree.

[0046] The candidate DOM block determination module is used to filter out at least one candidate DOM block that meets the conditions in the decision rules based on the statistical characteristics of each DOM block containing text.

[0047] The target DOM block determination module uses a semantic matching model to determine the target DOM block that semantically matches the term to be matched from at least one candidate DOM block. The semantic matching model is based on the Transformer structure.

[0048] The Interest Text Extraction module is used to extract the text of interest corresponding to the terms to be matched from the web page area corresponding to the target DOM block.

[0049] The text of interest on a webpage is usually what most users need, such as the publisher's name, publication time, project name, and detailed text corresponding to the project type. Through analysis of different types of webpages, this application found that these texts of interest have two basic characteristics: 1) The text of interest is presented in a single form on the webpage, mostly in long sentences, and contains a small number of hyperlinked texts; 2) The text of interest is mostly in a continuous and dense structural layout on the webpage.

[0050] Based on the aforementioned two fundamental characteristics, this application summarizes the statistical features of text of interest and predefines corresponding decision rules according to the content of different terms, specifying the conditions that the statistical features of candidate DOM blocks corresponding to different terms must meet. This application, using the DOM tree of a webpage, can systematically and comprehensively calculate the statistical features of each DOM block containing text. After determining the decision rules based on the content of the term to be matched, at least one candidate DOM block whose statistical features meet the conditions of the decision rules is selected, initially obtaining relatively high-quality candidate DOM blocks. To further improve the accuracy of extracting text of interest, this application continues to employ a semantic matching model to determine the target DOM block that semantically matches the term to be matched from at least one candidate DOM block, enabling precise extraction of the text of interest corresponding to the term to be matched from the webpage area.

[0051] This application employs a combined approach of initial screening using statistical indicators and secondary screening using text semantic matching algorithms to obtain target DOM blocks. Compared to using only one approach, this significantly improves the effectiveness of extracting text of interest from web pages. Furthermore, this approach is applicable to web pages of any structure and template, automatically and accurately acquiring text of interest, achieving automated and precise data collection without the need for manual customization of the data collection program. Attached Figure Description

[0052] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is an exemplary flowchart illustrating the extraction of text of interest from a webpage, provided in an exemplary embodiment of this application.

[0054] Figure 2 This is a schematic diagram of an example semantic matching model structure provided by an exemplary embodiment of this application;

[0055] Figure 3 This is a schematic diagram of an exemplary device for extracting text of interest from a webpage, provided in an exemplary embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0057] Currently, web pages contain a wealth of information, including not only text of interest to users but also a significant amount of noise. This application aims to solve the problem of how to extract various types of interesting text from web pages using a general-purpose program, without requiring additional development for different websites.

[0058] To address the aforementioned problems, this application provides a method for extracting text of interest from web pages, such as... Figure 1 As shown, the method includes:

[0059] S110, obtain the DOM tree of the webpage to be processed.

[0060] Current web pages are created using Hyper Text Markup Language (HTML) source code. HTML source code consists of a series of tags that unify the format of documents on the web, connecting scattered internet resources into a logical whole. HTML tags are keywords enclosed in angle brackets, such as `<html>`. HTML tags usually appear in pairs, such as `` and ``.<h1>

[0061] Every webpage has a basic set of structural tags (also called skeleton tags), and page content is written on these basic tags. For example, the `<html>` tag is used to define HTML tags; it's the largest tag on the page and is called the root tag. The `<head>` tag is used to define the document's header. <title>< / title> The title is used to define the document, giving the page its own title. The body is used to define the document's body, and the page's content is basically placed in the body.

[0062] It's important to note that every webpage can be parsed into a DOM tree, where all tags are nodes, and text and images are leaf nodes. The DOM tree can be obtained from externally input DOM tree data, or it can be obtained locally by processing the webpage itself.

[0063] In a feasible design, the DOM tree of the webpage to be processed is obtained in the following way:

[0064] The source code of the webpage to be processed is obtained through a web crawler program after being rendered by the browser.

[0065] Construct the DOM tree of the webpage to be processed based on its source code.

[0066] Since many websites do not store their data or other content in their source code, browser rendering is required before obtaining the actual webpage source code. To ensure the integrity of the obtained DOM tree, the webpage to be processed needs to be rendered by the browser first.

[0067] S120, determine the decision rule based on the content of the term to be matched.

[0068] The decision rules specify the statistical characteristics that the candidate DOM blocks corresponding to the terms to be matched must meet. A DOM block is a subtree of the DOM tree.

[0069] Suppose DB(v) is a subtree rooted at node v, where v is a non-text node, meaning the tag corresponding to v defines non-text content. This tag can be a `<body>` tag that includes the main content of the webpage. The main content includes the webpage's text, as well as images, videos, audio, etc., interspersed within the text. The subtree DB(v) is called a DOM block. It can be seen that DB(v) includes the leaf nodes of the subtree, and text and images are possible contents contained in the leaf nodes. Therefore, DB(v) contains the text content and / or tags of the corresponding webpage area.

[0070] To improve subsequent computational efficiency, DB(v) containing only tags can be removed, and the remaining DB(v) can be defined as DOM blocks. That is, a DOM block is a DB(v) containing text content.

[0071] Container tags such as ``, ``, ``, ``, ``, ``, and `` often contain text content. Through extensive research, the applicant discovered that the nodes corresponding to the main text are generally distributed among the leaf nodes of the DOM tree. Therefore, a depth-first traversal can be used to traverse the DOM tree to analyze the text content in the leaf nodes sequentially, identifying the subtrees corresponding to the leaf nodes and their descendant nodes as DOM blocks.

[0072] For example, in order to prevent the webpage areas corresponding to multiple DOM blocks from being duplicated and to further improve computational efficiency, a termination condition is set to terminate the process of defining DOM blocks.

[0073] For example, there is also a space between and . If tags like `` are used, then the subtree corresponding to the leaf node of `` will be determined as the DOM block, and... Although the corresponding subtree also satisfies the definition of a DOM block, its corresponding subtree is no longer defined as a DOM block. The termination condition is: if the tag corresponding to the leaf node of the DOM block is a preset tag, then the descendant nodes of that leaf node are no longer defined as DOM blocks. The preset tag can be set according to actual needs, such as ``, etc.

[0074] As can be seen, the termination condition is used to prevent the webpage area corresponding to multiple DOM blocks from being repeated. The specific content can be set according to actual needs, and this application does not limit it.

[0075] For example, the decision rule is determined based on the content of the term to be matched in the following way:

[0076] Obtain the first association relationship, which is used to associate different terms with corresponding decision rules;

[0077] The content of the term to be matched is matched with the terms in the first association relationship, and the decision rules associated with the matched terms are extracted.

[0078] For example, statistical characteristics include one or more of the following characteristics:

[0079] Non-link text length, punctuation length in non-link tags, punctuation density in non-link text, link text length, link text density, and non-link text density.

[0080] The length of non-linked text (also known as the number of non-linked text characters) refers to the number of text characters within non-linked tags. Non-linked tags refer to tags other than those used to define hyperlinked text (i.e., hyperlink tags), and the characters include letters, punctuation, and other symbols. It should be understood that since the main text is the portion of the text set on a webpage, the number of non-linked text characters within the main text is relatively large. The text within non-linked tags can be called non-linked text.

[0081] Link text length (also known as hyperlink text character count): The number of text characters in the hyperlink tag. Hyperlinks are a common form of noise on web pages, such as advertisements and related links; hyperlink text generally appears less frequently in the main text. The text within the hyperlink tag can be called hyperlink text.

[0082] Punctuation length in non-link tags: The number of punctuation marks in the text corresponding to non-link tags in the webpage source code. It should be understood that punctuation is generally concentrated in the body text area, while the number of punctuation marks in non-body text areas, such as advertising areas, is relatively small.

[0083] The non-linked text punctuation density is the density of punctuation marks in the text of non-linked tags within a DOM block. It is calculated using the following formula (1):

[0084]

[0085] PL(t) is the number of punctuation marks in non-linked tags in each DOM block, and TL(t) is the number of text characters in non-linked tags in each DOM block (i.e., the number of non-linked text characters). The "TL(t)+1" in the denominator is to prevent the calculation error caused by the denominator being 0. t represents the current DOM block, which can be the identifier of the current DOM block.

[0086] Link text density is the ratio of the number of hyperlink text characters in a DOM block to the total number of text characters in the DOM block. It is calculated using the following formula (2):

[0087]

[0088] LTL(t) is the number of hyperlink text characters in the current DOM block, and TTL(t) is the total number of text characters in the current DOM block (total text includes hyperlink text and non-link text). Since the number of hyperlink text characters is usually a small proportion of the total text characters, TTL(t) can be equal to TL(t) to save computation time. t represents the current DOM block, which can be the identifier of the DOM block.

[0089] The non-linked text density is the ratio of the number of non-linked text characters in the DOM block to the number of text characters in the webpage to be processed. The non-linked text density TD(t) is calculated using the following formula (3):

[0090]

[0091] TL(t) is the number of text characters in the non-linked tags of the current DOM block (i.e., the number of non-linked text characters), where t represents the current DOM block, i is used to number all DOM blocks including the current DOM block, N represents the number of DOM blocks of the web page to be processed, and TL(i) is the number of text characters in the non-linked tags of the i-th DOM block.

[0092] Based on the above statistical characteristics, the decision rules are illustrated below with examples:

[0093] (1) The term to be matched is "author". The decision rule for the term to be matched is: when the statistical features of the DOM block simultaneously satisfy TL(t)<=10, PL(t)<=1 and LPD(t)==0, the DOM block is a candidate DOM block. Among them, "TL(t)<=10, PL(t)<=1 and LPD(t)==0" are the conditions in the decision rule.

[0094] (2) The term to be matched is "publication time". The decision rule for the term to be matched is: when the statistical features of the DOM block simultaneously satisfy TL(t)<=20 and LPD(t)==0, the DOM block is a candidate DOM block. Among them, "TL(t)<=20 and LPD(t)==0" are the conditions in the decision rule.

[0095] It should be understood that the conditions in the decision rules are limitations on the range of values ​​for various statistical characteristics of the DOM block.

[0096] As can be seen, the content of the terms to be matched and the associated decision rules in the decision rules can be set according to actual needs, and this application does not impose any restrictions on this.

[0097] S130, based on the statistical characteristics of each DOM block containing text, select at least one candidate DOM block that meets the conditions in the decision rules.

[0098] In one feasible design, this is achieved by filtering at least one candidate DOM block that satisfies the conditions in the decision rules based on the statistical characteristics of each DOM block containing text, including:

[0099] Perform a depth-first traversal of the DOM tree to determine the statistical characteristics of each DOM block containing text. The statistical characteristics include one or more of the following features: non-linked text length, punctuation length in non-linked tags, punctuation density in non-linked text, linked text length, linked text density, and non-linked text density.

[0100] Based on the conditions in the decision rules, at least one candidate DOM block that meets the conditions is selected from each DOM block containing text.

[0101] In the example above, a depth-first traversal based on the DOM tree structure can comprehensively traverse every DOM block. Based on the statistical characteristics of candidate DOM blocks that must be satisfied in the decision rules, at least one candidate DOM block can be initially selected, facilitating further precise selection of target DOM blocks.

[0102] In a feasible design, before performing a depth-first traversal of the DOM tree to determine the statistical characteristics of each DOM block containing text, the method also includes:

[0103] Use regular expressions to delete the DOM nodes and their descendant nodes that correspond to noise information in the DOM tree.

[0104] In the example above, deleting nodes in the DOM tree that correspond to noise information such as HTML comments and scripts, which are relatively easy to summarize using regular expressions, can improve the efficiency of subsequent calculation of the statistical features of DOM blocks.

[0105] In a feasible design, before performing a depth-first traversal of the DOM tree to determine the statistical characteristics of each DOM block containing text, the method also includes:

[0106] Merge DOM nodes that are adjacent in the DOM tree and have the same or similar labels.

[0107] Here, tag similarity refers to the similarity between tags of multiple DOM nodes being greater than a pre-set threshold. The method for calculating the similarity between tags is similar to the method for calculating the similarity between strings, and this application does not limit it in this way.

[0108] In the example above, if there are multiple identical tags on a webpage, the corresponding nodes can be merged in the DOM tree, which can reduce the number of DOM blocks used to calculate statistical features and improve computational efficiency.

[0109] S140, using a semantic matching model, determine the target DOM block that semantically matches the term to be matched from at least one candidate DOM block.

[0110] The semantic matching model is based on the Transformer structure.

[0111] For example, such as Figure 2 As shown, the semantic matching model includes a first encoding layer based on the Transformer structure, a second encoding layer based on the Transformer structure, an attention layer, a first pooling layer, a second pooling layer, a feature fusion layer, and an output layer.

[0112] The first and second coding layers are both bidirectional encoder representations from transformers (BERT) coding layers based on the Transformer architecture, consisting of multiple layers of Transformers.

[0113] In a feasible design, Figure 2 Based on the structure shown, the following method is used to determine the target DOM block that semantically matches the term to be matched from at least one candidate DOM block:

[0114] Iterate through at least one candidate DOM block, and during the iteration of each candidate DOM block:

[0115] Input the term to be matched into the first coding layer to obtain the first coding vector;

[0116] Input the text corresponding to the current candidate DOM block into the second encoding layer to obtain the second encoding vector;

[0117] An attention layer is used to perform attention alignment on the first and second encoding vectors based on an attention mechanism, resulting in a first attention-weighted vector and a second attention-weighted vector, respectively.

[0118] The first attention weighted vector is pooled using the first pooling layer to obtain the first pooling vector;

[0119] The second attention weighted vector is pooled using the second pooling layer to obtain the second pooling vector;

[0120] The first pooling vector and the second pooling vector are fused using a feature fusion layer to obtain a feature fusion vector;

[0121] Input the feature fusion vector into the output layer to obtain the classification label. The classification label is used to identify whether the text corresponding to the current candidate DOM block matches the term to be matched.

[0122] In this context, the "lexical units" in the terms to be matched are also called "tokens." In large language models, tokens are the basic building blocks of text processing. They represent a discrete element in the text, which can be a word, character, subword, or character. The role of tokens is to break down text into the smallest processable units for subsequent text analysis and applications.

[0123] For example, the output layer includes a forward feedback layer and a normalized exponential function (softmax) layer.

[0124] In the example above, the first and second encoding layers simultaneously encode the text corresponding to the term to be matched and the current candidate DOM block, obtaining encoded vectors. Attention alignment of the encoded vectors of the two sentences using an attention mechanism accurately calculates the similarity information between them (i.e., the first attention-weighted vector and the second attention-weighted vector). Two pooling layers simultaneously pool the two attention-weighted vectors. Similarity fusion of the two pooled vectors yields a feature fusion vector that includes the similarity features between the two sentences. The output layer calculates the similarity features in the feature fusion vector to accurately determine whether the two sentences match.

[0125] In a feasible design, this is achieved by using a feature fusion layer to fuse the similarity of the first pooling vector and the second pooling vector to obtain the feature fusion vector:

[0126] The feature fusion layer performs a subtraction operation on the first pooling vector and the second pooling vector to obtain the difference feature vector;

[0127] The first pooling vector and the second pooling vector are multiplied by a feature fusion layer to obtain the interactive feature vector.

[0128] The first pooling vector, the second pooling vector, the difference feature vector, and the interaction feature vector are concatenated to obtain the feature fusion vector.

[0129] The example above obtains the difference features between the two sentences by subtracting the first and second pooling vectors, and obtains the interaction features by multiplying the first and second pooling vectors. This allows the concatenated feature fusion vector to include both the difference and interaction features of the two sentences, enriching the similarity information in the feature fusion vector and improving the accuracy of subsequent calculations on whether the two sentences match.

[0130] In a feasible design, this is achieved by pooling the first attention-weighted vector using a first pooling layer to obtain the first pooling vector:

[0131] The first attention weighted vector is averaged using the first pooling layer to obtain the first pooling vector;

[0132] And / or,

[0133] This is achieved by using a second pooling layer to pool the second attention weighted vector, resulting in the second pooling vector:

[0134] The second attention weighted vector is averaged using a second pooling layer to obtain the second pooling vector.

[0135] The example above reduces the size of the feature map and lowers the model's complexity by using average pooling on the two attention weighted vectors. Reducing the size of the feature map significantly reduces the computational cost of the next layer, thus improving the model's computational efficiency.

[0136] Based on the above examples, the following example illustrates the model calculation process.

[0137] First, assume the term to be matched is (x1, x2, ..., x n ), where x i Let i represent a token in the sentence, where 1 ≤ i ≤ n. The output vector after inputting into the first encoding layer is E, where E = Bert_encoder([x...). cls ],x1,x2,…,x n ,[x sep ]).

[0138] Bert_encoder represents the encoding function of the Bert model, which consists of multiple layers of Transformers. [x cls The symbol ] represents the beginning character of a sentence; the output vector corresponding to this symbol will serve as the semantic representation of the entire sentence. [x sep The symbol ] represents the boundary of the sentence recognized by the model. That is, the first encoding vector is [x]. cls The corresponding output vector H a The second encoding vector H b The generation method is the same as that of the first encoding vector, and will not be described in detail here.

[0139] Then, for H a and H b Attention alignment between sentence vectors is performed, specifically including:

[0140] (1) Calculate the similarity matrix, referring to the following formula (4):

[0141]

[0142] This represents the encoded vector corresponding to the i-th token of the term to be matched, where T represents the transpose. e represents the encoded vector corresponding to the j-th token in the sentence of the DOM block. ij This represents the similarity between the i-th token of the term to be matched and the j-th token in the sentence of the DOM block.

[0143] (2) Calculate the similarity information between sentences to achieve attention alignment, referring to the following formulas (5) and (6):

[0144]

[0145]

[0146] Among them, l a Indicates the number of tokens in the term to be matched, l b This indicates the number of tokens for sentences within a DOM block. This represents the vector of the i-th token in the term to be matched after attention alignment. This represents the vector of the j-th token in the sentence of the DOM block after attention alignment.

[0147] It can be seen that formulas (5) and (6) implement the use of attention mechanism to re-encode the tokens in the two sentences to obtain the sum, and each token is subject to attention weighting.

[0148] Next, average pooling is performed on the output results after attention alignment, referring to the following formulas (7) and (8):

[0149]

[0150]

[0151] Here, mean_pooling indicates average pooling, u represents the output of the first pooling layer, and v represents the output of the second pooling layer.

[0152] Finally, a feature fusion layer is used to perform similarity fusion to obtain the feature fusion vector R. Then, the output layer is used to calculate the classification label y used to identify the matching result, referring to the following formulas (9) and (10):

[0153] R = [u,v,uv,u*v] Formula (9);

[0154] y=softmax(FFN(R)) formula (10);

[0155] Here, uv represents the subtraction operation between vectors, aiming to obtain differential features; u*v represents the vector-matrix multiplication operation, aiming to obtain interaction features. The results of the subtraction and multiplication operations are concatenated with the original vectors to obtain the feature fusion vector R. FFN represents the feedforward layer, FFN = f(W*R + B), where f is the function of the feedforward layer, and W and B are the parameters of the feedforward layer. Softmax represents the normalization process, and is the function implemented by the softmax layer.

[0156] In a feasible design, the method also includes:

[0157] Obtain the training dataset, which includes different terms, text that matches the semantics of each term, and corresponding classification labels;

[0158] Based on minimizing the cross-entropy loss function, the semantic matching model is trained using the training dataset to obtain a well-trained semantic matching model.

[0159] For example, the cross-entropy loss function is minimized as shown in the following formula (11):

[0160]

[0161] Where loss represents the loss, N represents the number of samples, and y i p represents the actual category label. i This indicates the predicted category label.

[0162] In the example above, when training the semantic matching model using the training dataset, minimizing the cross-entropy loss function can make the model's predictions more consistent with reality, thus enabling better model training.

[0163] S150: Extract the text of interest corresponding to the term to be matched from the webpage area corresponding to the target DOM block.

[0164] The text of interest on a webpage is usually what most users need, such as the publisher's name, publication time, project name, and detailed text corresponding to the project type. Through analysis of different types of webpages, this application found that these texts of interest have two basic characteristics: 1) The text of interest is presented in a single form on the webpage, mostly in long sentences, and contains a small number of hyperlinked texts; 2) The text of interest is mostly in a continuous and dense structural layout on the webpage.

[0165] Based on the aforementioned two fundamental characteristics, this application summarizes the statistical features of text of interest and predefines corresponding decision rules according to the content of different terms, specifying the conditions that the statistical features of candidate DOM blocks corresponding to different terms must meet. This application, using the DOM tree of a webpage, can systematically and comprehensively calculate the statistical features of each DOM block containing text. After determining the decision rules based on the content of the term to be matched, at least one candidate DOM block whose statistical features meet the conditions of the decision rules is selected, initially obtaining relatively high-quality candidate DOM blocks. To further improve the accuracy of extracting text of interest, this application continues to employ a semantic matching model to determine the target DOM block that semantically matches the term to be matched from at least one candidate DOM block, enabling precise extraction of the text of interest corresponding to the term to be matched from the webpage area.

[0166] This application employs a combined approach of initial screening using statistical indicators and secondary screening using text semantic matching algorithms to obtain target DOM blocks. Compared to using only one approach, this significantly improves the effectiveness of extracting text of interest from web pages. Furthermore, this approach is applicable to web pages of any structure and template, automatically and accurately acquiring text of interest, achieving automated and precise data collection without the need for manual customization of the data collection program.

[0167] like Figure 3 As shown, this application also provides an apparatus for extracting text of interest from a webpage, comprising:

[0168] The DOM tree retrieval module is used to retrieve the DOM tree of the webpage to be processed.

[0169] The decision rule determination module is used to determine the decision rules based on the content of the terms to be matched. The decision rules are used to specify the conditions that the statistical characteristics of the candidate DOM blocks corresponding to the terms to be matched must meet. The DOM blocks are subtrees of the DOM tree.

[0170] The candidate DOM block determination module is used to filter out at least one candidate DOM block that meets the conditions in the decision rules based on the statistical characteristics of each DOM block containing text.

[0171] The target DOM block determination module uses a semantic matching model to determine the target DOM block that semantically matches the term to be matched from at least one candidate DOM block. The semantic matching model is based on the Transformer structure.

[0172] The Interest Text Extraction module is used to extract the text of interest corresponding to the terms to be matched from the web page area corresponding to the target DOM block.

[0173] In a feasible design, the candidate DOM block determination module is implemented by filtering at least one candidate DOM block that meets the conditions in the decision rules based on the statistical characteristics of each DOM block containing text:

[0174] Perform a depth-first traversal of the DOM tree to determine the statistical characteristics of each DOM block containing text. The statistical characteristics include one or more of the following features: non-linked text length, punctuation length in non-linked tags, punctuation density in non-linked text, linked text length, linked text density, and non-linked text density.

[0175] Based on the conditions in the decision rules, at least one candidate DOM block that meets the conditions is selected from each DOM block containing text.

[0176] In a feasible design, the semantic matching model includes a first encoding layer based on the Transformer structure, a second encoding layer based on the Transformer structure, an attention layer, a first pooling layer, a second pooling layer, a feature fusion layer, and an output layer;

[0177] The target DOM block determination module is implemented by using a semantic matching model to determine the target DOM block that semantically matches the term to be matched from at least one candidate DOM block:

[0178] Iterate through at least one candidate DOM block, and during the iteration of each candidate DOM block:

[0179] Input the term to be matched into the first coding layer to obtain the first coding vector;

[0180] Input the text corresponding to the current candidate DOM block into the second encoding layer to obtain the second encoding vector;

[0181] An attention layer is used to perform attention alignment on the first and second encoding vectors based on an attention mechanism, resulting in a first attention-weighted vector and a second attention-weighted vector, respectively.

[0182] The first attention weighted vector is pooled using the first pooling layer to obtain the first pooling vector;

[0183] The second attention weighted vector is pooled using the second pooling layer to obtain the second pooling vector;

[0184] The first pooling vector and the second pooling vector are fused using a feature fusion layer to obtain a feature fusion vector;

[0185] Input the feature fusion vector into the output layer to obtain the classification label. The classification label is used to identify whether the text corresponding to the current candidate DOM block matches the term to be matched.

[0186] In a feasible design, the DOM tree retrieval module retrieves the DOM tree of the webpage to be processed in the following way:

[0187] The source code of the webpage to be processed is obtained through a web crawler program after being rendered by the browser.

[0188] Construct the DOM tree of the webpage to be processed based on its source code.

[0189] In one feasible design, the device also includes a filtering module for using regular expressions to remove DOM nodes and their descendant nodes that correspond to noise information in the DOM tree before performing a depth-first traversal of the DOM tree to determine the statistical characteristics of each DOM block containing text.

[0190] In one feasible design, the device also includes a simplification module for merging DOM nodes that are adjacent in position and have the same or similar labels in the DOM tree before performing a depth-first traversal of the DOM tree to determine the statistical characteristics of each DOM block containing text.

[0191] In a feasible design, the target DOM block determination module is implemented by using a feature fusion layer to fuse the first pooling vector and the second pooling vector based on their similarity, resulting in a feature fusion vector:

[0192] The feature fusion layer performs a subtraction operation on the first pooling vector and the second pooling vector to obtain the difference feature vector;

[0193] The first pooling vector and the second pooling vector are multiplied by a feature fusion layer to obtain the interactive feature vector.

[0194] The first pooling vector, the second pooling vector, the difference feature vector, and the interaction feature vector are concatenated to obtain the feature fusion vector.

[0195] In a feasible design, the DOM block determination module is implemented by pooling the first attention-weighted vector using the first pooling layer to obtain the first pooling vector:

[0196] The first attention weighted vector is averaged using the first pooling layer to obtain the first pooling vector;

[0197] And / or, the DOM block determination module is implemented by pooling the second attention weighted vector using a second pooling layer to obtain the second pooling vector:

[0198] The second attention weighted vector is averaged using a second pooling layer to obtain the second pooling vector.

[0199] In one feasible design, the device also includes a model training module for training the semantic matching model in the following manner:

[0200] Obtain the training dataset, which includes different terms, text that matches the semantics of each term, and corresponding classification labels;

[0201] Based on minimizing the cross-entropy loss function, the semantic matching model is trained using the training dataset to obtain a well-trained semantic matching model.

[0202] Other implementations and effects of the above-mentioned device can be found in the description of the method embodiment for extracting text of interest from web pages, and will not be repeated here.

[0203] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0204] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0205] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0206] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0207] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0208] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for extracting text of interest from a webpage, characterized in that, include: Obtain the DOM tree of the webpage to be processed; The decision rules are determined based on the content of the term to be matched. The decision rules are used to specify the conditions that the statistical characteristics of the candidate DOM block corresponding to the term to be matched must meet. The DOM block is a subtree of the DOM tree. Based on the statistical characteristics of each DOM block containing text, at least one candidate DOM block that satisfies the conditions in the decision rule is selected, wherein the step of selecting at least one candidate DOM block that satisfies the conditions in the decision rule based on the statistical characteristics of each DOM block containing text includes: A depth-first traversal is performed on the DOM tree to determine the statistical characteristics of each DOM block containing text. The statistical characteristics include one or more of the following features: non-linked text length, punctuation length in non-linked tags, punctuation density in non-linked text, linked text length, linked text density, and non-linked text density. Based on the conditions in the decision rules, at least one candidate DOM block that satisfies the conditions is selected from each DOM block containing text; A semantic matching model is used to determine a target DOM block that semantically matches the term to be matched from at least one candidate DOM block. The semantic matching model is based on a Transformer structure and includes a first encoding layer based on a Transformer structure, a second encoding layer based on a Transformer structure, an attention layer, a first pooling layer, a second pooling layer, a feature fusion layer, and an output layer. The step of using a semantic matching model to determine the target DOM block that semantically matches the term to be matched from at least one candidate DOM block includes: Traverse at least one of the candidate DOM blocks, and during the traversal of each candidate DOM block: The term to be matched is input into the first coding layer to obtain the first coding vector; Input the text corresponding to the current candidate DOM block into the second encoding layer to obtain the second encoding vector; The attention layer is used to perform attention alignment on the first encoding vector and the second encoding vector based on the attention mechanism, to obtain the first attention weighted vector and the second attention weighted vector, respectively. The first attention weighted vector is pooled using the first pooling layer to obtain the first pooling vector; The second attention weighted vector is pooled using the second pooling layer to obtain the second pooling vector; The feature fusion layer is used to perform similarity fusion on the first pooling vector and the second pooling vector to obtain a feature fusion vector; The feature fusion vector is input into the output layer to obtain a classification label, which is used to identify whether the text corresponding to the current candidate DOM block matches the term to be matched. Extract the text of interest corresponding to the term to be matched from the webpage area corresponding to the target DOM block.

2. The method according to claim 1, characterized in that, The process of obtaining the DOM tree of the webpage to be processed includes: The source code of the webpage to be processed is obtained through a web crawler program after being rendered by the browser. Based on the source code of the webpage to be processed, construct the DOM tree of the webpage to be processed.

3. The method according to claim 1, characterized in that, Before performing a depth-first traversal of the DOM tree to determine the statistical characteristics of each DOM block containing text, the method further includes: Regular expressions are used to delete the DOM nodes and their descendant nodes that correspond to the noise information in the DOM tree.

4. The method according to claim 1, characterized in that, Before performing a depth-first traversal of the DOM tree to determine the statistical characteristics of each DOM block containing text, the method further includes: Merge DOM nodes that are adjacent in the DOM tree and have the same or similar labels.

5. The method according to claim 1, characterized in that, The step of using the feature fusion layer to perform similarity fusion on the first pooling vector and the second pooling vector to obtain a feature fusion vector includes: The feature fusion layer is used to subtract the first pooling vector and the second pooling vector to obtain the difference feature vector; The feature fusion layer is used to perform matrix multiplication on the first pooling vector and the second pooling vector to obtain an interactive feature vector; The first pooling vector, the second pooling vector, the difference feature vector, and the interaction feature vector are concatenated to obtain the feature fusion vector.

6. The method according to claim 1, characterized in that, The step of pooling the first attention weighted vector using the first pooling layer to obtain the first pooled vector includes: The first attention weighted vector is averaged using the first pooling layer to obtain the first pooling vector; And / or, The second attention weighted vector is pooled using the second pooling layer to obtain a second pooling vector, including: The second attention weighted vector is averaged using the second pooling layer to obtain the second pooling vector.

7. The method according to claim 1, characterized in that, The method further includes: Obtain a training dataset, which includes different terms, text that matches the semantics of each term, and corresponding classification labels; The semantic matching model is trained using the training dataset based on minimizing the cross-entropy loss function, resulting in the trained semantic matching model.

8. An apparatus for extracting text of interest from a webpage, characterized in that, include: The DOM tree retrieval module is used to retrieve the DOM tree of the webpage to be processed. The decision rule determination module is used to determine decision rules based on the content of the term to be matched. The decision rules are used to specify the conditions that the statistical features of the candidate DOM block corresponding to the term to be matched must meet, and the DOM block is a subtree of the DOM tree. The candidate DOM block determination module is used to filter at least one candidate DOM block that satisfies the conditions in the decision rule based on the statistical characteristics of each DOM block containing text. The step of filtering at least one candidate DOM block that satisfies the conditions in the decision rule based on the statistical characteristics of each DOM block containing text includes: A depth-first traversal is performed on the DOM tree to determine the statistical characteristics of each DOM block containing text. The statistical characteristics include one or more of the following features: non-linked text length, punctuation length in non-linked tags, punctuation density in non-linked text, linked text length, linked text density, and non-linked text density. Based on the conditions in the decision rules, at least one candidate DOM block that satisfies the conditions is selected from each DOM block containing text; The target DOM block determination module uses a semantic matching model to determine the target DOM block that semantically matches the term to be matched from at least one of the candidate DOM blocks. The semantic matching model is based on the Transformer structure and includes a first encoding layer based on the Transformer structure, a second encoding layer based on the Transformer structure, an attention layer, a first pooling layer, a second pooling layer, a feature fusion layer, and an output layer. The step of using a semantic matching model to determine the target DOM block that semantically matches the term to be matched from at least one candidate DOM block includes: Traverse at least one of the candidate DOM blocks, and during the traversal of each candidate DOM block: The term to be matched is input into the first coding layer to obtain the first coding vector; Input the text corresponding to the current candidate DOM block into the second encoding layer to obtain the second encoding vector; The attention layer is used to perform attention alignment on the first encoding vector and the second encoding vector based on the attention mechanism, to obtain the first attention weighted vector and the second attention weighted vector, respectively. The first attention weighted vector is pooled using the first pooling layer to obtain the first pooling vector; The second attention weighted vector is pooled using the second pooling layer to obtain the second pooling vector; The feature fusion layer is used to perform similarity fusion on the first pooling vector and the second pooling vector to obtain a feature fusion vector; The feature fusion vector is input into the output layer to obtain a classification label, which is used to identify whether the text corresponding to the current candidate DOM block matches the term to be matched. The Interest Text Extraction Module is used to extract the Interest Text corresponding to the term to be matched from the webpage area corresponding to the target DOM block.

Images