A method for identifying the similarity of a large amount of web text information based on word net

Abstract

The invention discloses a method for identifying the similarity of a large amount of Web text information based on a word net, comprising the following steps: (1) constructing a word net; (2) identifying the similarity of text information of a new Web page, comprising the following steps: Extract text information to form a new document, and extract feature words f from the new document 1 , f 2 ,...,f m ; Solve the set of similar words of each feature word f; Solve the set of similar documents of each feature word f; Determine the similar documents of the new document and calculate the similarity value of the documents in the similar document set; Filter the documents in the similar document set to get the final (3) update the word net of the new Web page according to the method of step (1). This method can be used to discover information plagiarism or information imitation and tampering, and can be used to discover hidden correlations between different fields, eliminate duplicate web pages, reduce the burden on search engines, and optimize storage and index structures.

Description

technical field [0001] The invention relates to a method for identifying the similarity of Internet text information, in particular to a method for identifying the similarity of a large amount of Web text information based on word nets. Background technique [0002] The transformation of Internet technology not only transmits information and knowledge, but also provides a platform for offline Internet users to release information and communicate with each other, and introduces ordinary users to participate in the rapid growth of a large amount of online information, making the Internet an important part of the information resource library. one. [0003] In order to cope with the rapid growth of Internet information, many research projects focus on how to effectively organize these large amounts of information, so that end users can quickly and accurately obtain the information they need, and reduce the cost of organizing information. Web information in the Internet is displ...

AI Technical Summary

Problems solved by technology

[0006] The above method determines the similarity of the text according to the literal content of the text. The method can return the basic results required by the query request or can compare the similarity of the text from the literal content. Certain ambiguity, that is, the target result that should be returned by the query request is not clear, so that the content of the input query request is not targeted, so that the returned query result may not be the result expected by the user; (2) cannot identify two Although the literal form of the content of the document is very different, the information or meaning contained in it is almost the same. It is just described with different words from different angles, or even a synonymous paraphrase of the same problem.

Havelivala et al. pointed out that the method of identifying relevant webpages based on the webpage link relationship does not have a good effect when the number of links is small. He proposed a combination of anchor text and anchor window methods to make up for the small number of webpage links. Method is easily influenced by the number of links between pages, the type or quality of pages

Images