News webpage text extraction method and system based on semantic similarity

Abstract

The invention discloses a news webpage text extraction method and system based on semantic similarity, and belongs to the technical field of data processing. The method comprises the following steps: aiming at a target news webpage, extracting a news title of the target news webpage based on semantic similarity, and screening out a most relevant paragraph of the news title; based on the most relevant paragraph, defining a text range corresponding to the news title; and in the text range, extracting the news webpage text. According to the method, the title is used as a clue, text semantics are used as a search tool, and then the text range is defined, so that the correctness of text extraction is enhanced, and the integrity is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and more specifically, to a method and system for extracting news webpage text based on semantic similarity. Background Art

[0002] At present, the methods for extracting web page text mainly include the following categories: a) DOM tree-based method: By building a DOM tree and traversing the tree, various web page noises are identified and removed, and the remaining content is used as the extraction result.

[0003] b) Tag density-based method: The difference in HTML tag density is used to distinguish between text and noise.

[0004] c) Machine learning-based method: linearize and reconstruct the web page source code, and obtain the context paragraphs of the web page text through text classification and clustering.

[0005] d) Text density-based method: This method first removes the HTML tags of the web page, leaving the pure text content, and then constructs the text density function of the web page. On this basis, a threshold is set and the lines or blocks with text density higher than the threshold are regarded as the main text.

[0006] In the label density-based method, the accuracy of text extraction is highly dependent on the determination of the threshold, and the error rate is high in actual use. The clustering-based method is easily affected by noise data, and often contains irrelevant text or misses text paragraphs. The text density-based method has high requirements for web page layout, that is, this method assumes that the text density of the text is the highest in the entire web page, but this assumption is too idealistic. For example, for a web page containing a large amount of comments, this method often extracts the comments as the text. Summary of the invention

[0007] In view of the above problems, the present invention proposes a news webpage text extraction method based on semantic similarity, comprising: For a target news webpage, extract the news title of the target news webpage based on semantic similarity, and filter out the most relevant paragraphs of the news title; Based on the most relevant paragraph, defining the text range corresponding to the news title; Extracting the news webpage text within the text range; The step of extracting the news title of the target news webpage based on semantic similarity includes: Extract multiple types of tags from the target news webpage and determine the common prefix length of the text content of each type of tag; If the common prefix length of at least two types of label text content is greater than 3, the common prefix of the text content with the common prefix length greater than 3 is extracted as the news title; If the common prefix length of at least two types of label text content is not greater than 3, determine the tree subnode of the web document tree of the target news web page, and based on the tree subnode of the web document tree, filter out text of a preset length as the target text, vectorize the target text, and calculate the semantic similarity of the vectorized target texts in pairs, and select the target text with the largest total similarity score as the news title.

[0008] Optional, multi-class labels, including: title tag, h1 tag and h2 tag.

[0009] Optionally, the preset length is less than 30 characters.

[0010] Optionally, the most relevant paragraphs of the news title are filtered out, including: Based on the news title, the tree child nodes of the web page document are taken as potential text paragraphs, and the semantic similarity between the news title and the potential text paragraphs is calculated in turn. Based on the semantic similarity, the most relevant paragraphs are screened out.

[0011] Optionally, the most relevant paragraphs are the top 5 paragraphs with the highest semantic similarity between the latent text paragraph and the news title.

[0012] Optionally, based on the most relevant paragraph, defining a text range corresponding to the news title includes: Determine the potential paragraph node of the most relevant paragraph, and search for the common ancestor node closest to the potential paragraph node on the web page document tree, and use the common ancestor node as the potential news text root tag. After all searches are completed, count the potential news text root tags, and use the potential news text root tag that appears the most times as the text root tag, and use the text root tag to define the text range.

[0013] Optionally, within the text range, extracting the news webpage text includes: The subnodes of the text root tag are traversed in sequence on the web page document tree, the text or image content in the subnodes is extracted and parsed, and the parsed text or image content is spliced ​​in sequence to obtain the news web page text.

[0014] On the other hand, the present invention also proposes a news webpage text extraction system based on semantic similarity, comprising: A news title extraction unit is used to extract the news title of the target news webpage based on semantic similarity and filter out the most relevant paragraphs of the news title; A text defining unit, used to define a text range corresponding to the news title based on the most relevant paragraph; A text extraction unit, used to extract the text of the news web page within the text range; The step of extracting the news title of the target news webpage based on semantic similarity includes: Extract multiple types of tags from the target news webpage and determine the common prefix length of the text content of each type of tag; If the common prefix length of at least two types of label text content is greater than 3, the common prefix of the text content with the common prefix length greater than 3 is extracted as the news title; If the common prefix length of at least two types of label text content is not greater than 3, determine the tree subnode of the web document tree of the target news web page, and based on the tree subnode of the web document tree, filter out text of a preset length as the target text, vectorize the target text, and calculate the semantic similarity of the vectorized target texts in pairs, and select the target text with the largest total similarity score as the news title.

[0015] In yet another aspect, the present invention further provides a computing device, comprising: one or more processors; a processor for executing one or more programs; When the one or more programs are executed by the one or more processors, the above-described method is implemented.

[0016] In yet another aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed, the method described above is implemented.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a method for extracting the text of a news webpage based on semantic similarity, comprising: extracting the news title of the target news webpage based on semantic similarity, and screening out the most relevant paragraphs of the news title; defining the text range corresponding to the news title based on the most relevant paragraphs; and extracting the text of the news webpage within the text range. The present invention uses the title as a clue, uses text semantics as a search tool, and then defines the text range, thereby strengthening the correctness of text extraction and ensuring completeness. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is the principle diagram of the method of the present invention; Figure 3 It is a structural diagram of the system of the present invention. DETAILED DESCRIPTION

[0019] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0020] Embodiment 1: The present invention proposes a news webpage text extraction method based on semantic similarity. Figure 1 As shown, including: Step 1: for a target news webpage, extract the news title of the target news webpage based on semantic similarity, and filter out the most relevant paragraphs of the news title; Step 2: Based on the most relevant paragraph, define the text range corresponding to the news title; Step 3: extract the news webpage text within the text range.

[0021] Among them, for the target news webpage, the news title of the target news webpage is extracted based on semantic similarity, including: Extract multiple types of tags from the target news webpage and determine the common prefix length of the text content of each type of tag; If the common prefix length of at least two types of label text content is greater than 3, the common prefix of the text content with the common prefix length greater than 3 is extracted as the news title; If the common prefix length of at least two types of label text content is not greater than 3, determine the tree subnode of the web document tree of the target news web page, and based on the tree subnode of the web document tree, filter out text of a preset length as the target text, vectorize the target text, and calculate the semantic similarity of the vectorized target texts in pairs, and select the target text with the largest total similarity score as the news title.

[0022] Among them, multiple categories of labels, including: title tag, h1 tag and h2 tag.

[0023] Among them, the preset length is less than 30 characters.

[0024] The most relevant paragraphs of the news title are screened out, including: Based on the news title, the tree child nodes of the web page document are taken as potential text paragraphs, and the semantic similarity between the news title and the potential text paragraphs is calculated in turn. Based on the semantic similarity, the most relevant paragraphs are screened out.

[0025] Among them, the most relevant paragraphs are the latent text paragraphs and the news headlines, and the top 5 paragraphs with the highest semantic similarity.

[0026] Wherein, based on the most relevant paragraph, the text range corresponding to the news title is defined, including: Determine the potential paragraph node of the most relevant paragraph, and search for the common ancestor node closest to the potential paragraph node on the web page document tree, and use the common ancestor node as the potential news text root tag. After all searches are completed, count the potential news text root tags, and use the potential news text root tag that appears the most times as the text root tag, and use the text root tag to define the text range.

[0027] Wherein, within the text range, the news webpage text is extracted, including: The subnodes of the text root tag are traversed in sequence on the web page document tree, the text or image content in the subnodes is extracted and parsed, and the parsed text or image content is spliced ​​in sequence to obtain the news web page text.

[0028] The present invention is further described below with reference to specific cases: The specific case process principle is as follows Figure 2 As shown in the figure, it mainly involves news title recognition, topK paragraph label positioning, text scope definition, text extraction and other stages. The specific implementation plan for each stage is as follows: News headline recognition: When the length of the common prefix of the text content of at least two tags, title tag, h1 tag, and h2 tag, is greater than 3, the common prefix is ​​considered to be a news title. If the above conditions are not met, the tree child nodes of the web document are extracted, and the text with a length of less than 30 words is screened and vectorized. The semantic similarity is calculated for each of them, and the one with the largest sum of similarity scores is selected as the news title.

[0029] topK paragraph tag positioning: Using the obtained title, the leaf nodes in the document tree are taken as potential text paragraphs, the semantic similarity of the title and the potential text paragraphs is calculated in turn, the top 5 most relevant paragraphs are selected, and their node positions are recorded.

[0030] Scope of the text: For the top 5 potential paragraph nodes found, search for their nearest common ancestor nodes in pairs on the document tree as potential news text root tags. After all searches are completed, count the potential text root tags and use the root tag with the most occurrences as the final text root tag.

[0031] Text extraction: Traverse the subnodes of the text tag in the document tree, extract and parse the text or image content in the subnodes, and finally splice the traversed and extracted content in order to get the final news text.

[0032] Embodiment 2: The present invention also proposes a news webpage text extraction system 200 based on semantic similarity, such as Figure 3 As shown, including: A news title extraction unit 201 is used to extract the news title of the target news webpage based on semantic similarity and filter out the most relevant paragraphs of the news title; A text defining unit 202, configured to define a text range corresponding to the news title based on the most relevant paragraph; A text extraction unit 203, used to extract the text of the news web page within the text range; The step of extracting the news title of the target news webpage based on semantic similarity includes: Extract multiple types of tags from the target news webpage and determine the common prefix length of the text content of each type of tag; If the common prefix length of at least two types of label text content is greater than 3, the common prefix of the text content with the common prefix length greater than 3 is extracted as the news title; If the common prefix length of at least two types of label text content is not greater than 3, determine the tree subnode of the web document tree of the target news web page, and based on the tree subnode of the web document tree, filter out text of a preset length as the target text, vectorize the target text, and calculate the semantic similarity of the vectorized target texts in pairs, and select the target text with the largest total similarity score as the news title.

[0033] The present invention takes the title as a clue, uses text semantics as a search tool, and then defines the scope of the text, which not only strengthens the correctness of text extraction but also ensures completeness.

[0034] Embodiment 3: Based on the same inventive concept, the present invention also provides a computer device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding functions, so as to implement the steps of the method in the above embodiment.

[0035] Embodiment 4: Based on the same inventive concept, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It can be understood that the computer-readable storage medium here can include both built-in storage media in a computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the steps of the method in the above embodiment.

[0036] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A news webpage text extraction method based on semantic similarity, characterized in that: include: For a target news webpage, extract the news title of the target news webpage based on semantic similarity, and filter out the most relevant paragraphs of the news title; Based on the most relevant paragraph, defining the text range corresponding to the news title; Extracting the news webpage text within the text range; The step of extracting the news title of the target news webpage based on semantic similarity includes: Extract multiple types of tags from the target news webpage and determine the common prefix length of the text content of each type of tag; If the common prefix length of at least two types of label text content is greater than 3, the common prefix of the text content with the common prefix length greater than 3 is extracted as the news title; If the common prefix length of at least two types of label text content is not greater than 3, determine the tree subnode of the web document tree of the target news web page, and based on the tree subnode of the web document tree, filter out text of a preset length as the target text, vectorize the target text, and calculate the semantic similarity of the vectorized target texts in pairs, and select the target text with the largest total similarity score as the news title.

2. The news webpage text extraction method according to claim 1, characterized in that: The multiple types of labels include: title tag, h1 tag and h2 tag.

3. The news webpage text extraction method according to claim 1, characterized in that: The preset length is less than 30 characters.

4. The news webpage text extraction method according to claim 1, characterized in that: The filtering out the most relevant paragraphs of the news title includes: Based on the news title, the tree child nodes of the web page document are taken as potential text paragraphs, and the semantic similarity between the news title and the potential text paragraphs is calculated in turn. Based on the semantic similarity, the most relevant paragraphs are screened out.

5. The news webpage text extraction method according to claim 1, characterized in that: The most relevant paragraphs are the potential text paragraphs and the news headlines, and the top five paragraphs with the highest semantic similarity.

6. The news webpage text extraction method according to claim 1, characterized in that: Defining the text range corresponding to the news title based on the most relevant paragraph includes: Determine the potential paragraph node of the most relevant paragraph, and search for the common ancestor node closest to the potential paragraph node on the web page document tree, and use the common ancestor node as the potential news text root tag. After all searches are completed, count the potential news text root tags, and use the potential news text root tag that appears the most times as the text root tag, and use the text root tag to define the text range.

7. The news webpage text extraction method according to claim 1, characterized in that: The extracting of the news webpage text within the text range includes: The subnodes of the text root tag are traversed in sequence on the web page document tree, the text or image content in the subnodes is extracted and parsed, and the parsed text or image content is spliced ​​in sequence to obtain the news web page text.

8. A news webpage text extraction system based on semantic similarity, characterized in that: include: A news title extraction unit is used to extract the news title of the target news webpage based on semantic similarity and filter out the most relevant paragraphs of the news title; A text defining unit, used to define a text range corresponding to the news title based on the most relevant paragraph; A text extraction unit, used to extract the text of the news web page within the text range; The step of extracting the news title of the target news webpage based on semantic similarity includes: Extract multiple types of tags from the target news webpage and determine the common prefix length of the text content of each type of tag; If the common prefix length of at least two types of label text content is greater than 3, the common prefix of the text content with the common prefix length greater than 3 is extracted as the news title; If the common prefix length of at least two types of label text content is not greater than 3, determine the tree subnode of the web document tree of the target news web page, and based on the tree subnode of the web document tree, filter out text of a preset length as the target text, vectorize the target text, and calculate the semantic similarity of the vectorized target texts in pairs, and select the target text with the largest total similarity score as the news title.

9. A computer device, characterized in that: include: one or more processors; a processor for executing one or more programs; When the one or more programs are executed by the one or more processors, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images