An abstract extraction method combining a page analysis rule and NLP text vectorization

Abstract

The invention discloses an abstract extraction method combining a page analysis rule and NLP text vectorization. The abstract extraction method comprises the following steps: S1, extracting text datain an html format in a'body 'label of text data of a webpage by using a Readability packet; S2, obtaining the text length of the text corpus, and eliminating unqualified text corpus; S3, judging whether the number of sentences of the text corpus is greater than a threshold value or not; S4, judging whether paragraph subtitle phrases can be obtained or not; S5, defining regular matching keywords, and removing the texts matched with the regular matching keywords to obtain filtered text corpora; S6, judging the compliance of the language segments; And S7, training a Word2Vec model, splitting thetext corpus into sentences, splitting the sentences into words, performing vectorization operation, solving sentence similarity by using EMD, giving weights based on the sentence similarity by using aTextRank algorithm, and determining the sentence with the highest weight as a text abstract sentence. According to the method, relatively core sentences can be obtained for long blogs and news articles, so that the subjects can be quickly known.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to an abstract extraction method combining page parsing rules and NLP text vectorization. Background technique [0002] The importance of text summarization can be reflected in life. In this era of information and data explosion, more and more information makes it difficult for people to receive in a short time. Filter out cumbersome text information, and use a few simple sentences to The way to express the core information is particularly important. The most common ones are news, Weibo, etc. that we often come into contact with every day. In terms of technical application, the obtained summary information can be used for NLP tasks such as classification and theme analysis. At present, the abstract extraction uses the textrank+word2vec model to extract the core sentences of the entire text. However, for long articles, the word2vec model is used to divide the tex...

AI Technical Summary

Problems solved by technology

At present, the abstract extraction uses the textrank+word2vec model to extract the core sentences of the entire text. However, for long articles, the word2vec model is used to divide the text into sentences, and then split the sentences into words, and then vectorize the words and The process of calculating the distance will have a high computational complexity, and textrank is based on the weight given by the similarity of sentences. According to the actual extraction results, especially for texts like news, the content and paragraph shapes are different. , many interfering sentences will affect the extraction results

Images