Unstructured text similarity judgment method and system

Abstract

The invention belongs to the technical field of data processing, and particularly relates to an unstructured text similarity judgment method and system. The invention provides the unstructured text similarity judgment method and system. The specific implementation steps of the scheme are as follows: 1, inputting unstructured data; wherein the unstructured data may be a web page or word document orthe like from a web crawler; 2, extracting text, namely extracting text information from the unstructured data, wherein an apache tika (apache organization open source text extraction component) is adopted to extract text content. The method can be compatible with extraction of text content in various formats, such as excel, pdf, xml, json, markdown and the like, and the extracted txt file is finally output in the step.

Description

technical field [0001] The invention belongs to the technical field of data processing, and in particular relates to a method and system for judging the similarity of unstructured texts. Background technique [0002] Unstructured data refers to the data whose data structure is irregular or incomplete, without a predefined data model, and which is inconvenient to be represented by two-dimensional logical tables of the database. Including web page text, office documents in all formats, text, pictures, XML, HTML, various reports, images and audio / video information, etc. [0003] Text similarity measurement refers to the measurement of the similarity between two texts, which has a wide range of applications in many fields. For example, in information retrieval, similarity can be used to identify similar words and improve the recall rate. In the automatic question answering scenario, the similarity can be used to calculate the matching degree between the user's question sentenc...

AI Technical Summary

Problems solved by technology

At the same time, some improvements have been made to the final similarity score, but this part of the improvement will bring a significant increase in time complexity

[0009] This scheme is relatively rough and has obvious shortcomings: First, it does not consider the semantics of documents, ignores the contextual relationship of words, and ignores the positional relationship of words, and only judges the similarity from the level of string comparison

[0012] The disadvantage of this algorithm applied to the calculation of text similarity is that it uses the product of TF and IDF as the value measure of the feature space coordinate system, and uses IDF to complete the weighted adjustment of the word frequency TF. The purpose of adjusting the weight is to highlight important words. suppress secondary words

However, when IDF calculates the weight of feature items, it takes the total number of documents in the document set as the benchmark. When the number of documents of various types in the document set is unbalanced, for example, when a certain type of document is relatively small, IDF basically has no inhibitory effect.

Therefore, TFIDF fails to compromise the results of TF and IDF. The weight of this feature item will blindly depend on the document frequency TF, which will eventually lead to a high weight of word frequency or even play a leading role.

At this time, there are obviously major flaws in the measurement of the importance of words, and the number of occurrences of important words may not be many

At the same time, the most important thing is that this algorithm cannot reflect the position information and context information of words, and ignores the semantics of the text. The words that appear in the front position and the words that appear in the back position are both considered to be of the same importance.

[0013] minHash is an implementation of Locality-Sensitive Hashing (LSH), which is used to estimate the similarity of two sets, but compared with other algorithms, such as simhash, it lacks the description of the similarity (simhash uses sea distance to measure similarity

Images