A method and system for automatic comparison, analysis and processing of ultra-large-scale data

Abstract

The present invention relates to the technical field of data comparison and analysis, and proposes a method and system for automatic comparison and analysis of ultra-large-scale data, including inputting a target document, classifying the target document, obtaining a target document library, and obtaining a target virtual vector value of the target document library; calculating the vector of the target document and each document in the target document library; calculating the vector difference D1 from the target vector to the target virtual vector value, and calculating the vector difference D2 from the vector of each document in the target document library to the target virtual vector value; and selecting a document whose vector difference D2 is close to the vector difference D1 as the first approximate document of the target document. Data indexing is performed using a vector indexing method, and its indexing speed is orders of magnitude higher than that of traditional text indexing, and the retrieval results are more conducive to subsequent data mining using statistical formulas.

Description

Technical Field

[0001] The present invention relates to the technical field of comparison and analysis, and in particular to a method and system for automatic comparison and analysis processing of ultra-large-scale data. Background Art

[0002] The amount of literature data that can be compared with existing patents is at the trillion level, which is ultra-large-scale data. The current system's query on patents is mainly based on traditional text keyword combination indexing technology, combined with data mining algorithms such as syntactic analysis and semantic understanding technology. When it comes to the highly professional text semantic understanding of patents or specific disciplines, the number of various applications is still scarce and there are technical defects. In addition, the current natural language understanding results and emerging artificial intelligence technologies are limited by the size of data and the complexity of algorithms, and have not been applied in large-scale patent search systems.

[0003] Therefore, there is an urgent need for a technology that can quickly query and compare patents, use cutting-edge artificial intelligence technology in ultra-large-scale data scenarios, and quickly complete patent retrieval so that patent examiners can make professional and quick judgments, thereby improving the efficiency of patent review and management. Summary of the invention

[0004] The present invention proposes a method and system for automatic comparison, analysis and processing of ultra-large-scale data, innovates the data mining algorithm, and attempts to design a system architecture suitable for ultra-large-scale data scenarios based on this, which can handle the comparison and review of patents at high speed.

[0005] The technical solution of the present invention is as follows:

[0006] In a first aspect, a method for automatic comparison, analysis and processing of ultra-large-scale data comprises the following steps:

[0007] Obtain multiple document libraries, and calculate a virtual vector value of each document library;

[0008] Input the target document, classify the target document, obtain the document library corresponding to the target document as the target document library; the virtual vector value of the target document library is used as the target virtual vector value;

[0009] Calculate the vector of each document in the target document library, wherein the target document library includes the target document and a comparison document group, wherein the comparison document group is the documents in the target document library other than the target document;

[0010] Calculate the vector difference D1 from the vector of the target document to the target virtual vector value, and calculate the vector difference D2 from the vector of each document in the comparison document group to the target virtual vector value;

[0011] Calculate the distance between the vector difference D2 and the vector difference D1, and select the document corresponding to the vector difference D2 whose distance is less than a set threshold as the first approximate document of the target document;

[0012] Input the first approximate literature into the molecular sieve model and set the coarse particle screening condition a;

[0013] Output possible data set R1 and impossible data set R2;

[0014] Input the first approximate document in the possible data set R1 into the molecular sieve model and set the fine particle screening condition b;

[0015] Output possible data set update R1 and impossible data set R3;

[0016] Merge the impossible data set R2 and the impossible data set R3 as the backup data set R4, and use the updated possible data set R1 as the result set R;

[0017] Data analysis is performed on the result set R. If similar documents are found, the task is terminated. If no similar documents are found, the search continues in the backup data set R4 until similar documents are found.

[0018] In a second aspect, a system for automatically comparing, analyzing and processing ultra-large-scale data includes:

[0019] A first calculation module is used to obtain multiple document libraries and calculate the virtual vector value of each document library;

[0020] The first processing module is used to input the target document, classify the target document, obtain the document library corresponding to the target document as the target document library, and use the virtual vector value of the target document library as the target virtual vector value;

[0021] A second calculation module is used to calculate the vector of each document in the target document library, wherein the target document library includes the target document and a comparison document group, wherein the comparison document group is the documents in the target document library other than the target document;

[0022] A third calculation module is used to calculate the vector difference D1 from the vector of the target document to the target virtual vector value, and calculate the vector difference D2 from the vector of each document in the comparison document group to the target virtual vector value;

[0023] The first selection module is used to calculate the distance between the vector difference D2 and the vector difference D1, and select the document corresponding to the vector difference D2 whose distance is less than a set threshold as the first approximate document of the target document.

[0024] In a third aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for automated comparison, analysis and processing of ultra-large-scale data.

[0025] The working principle and beneficial effects of the present invention are:

[0026] 1. The present invention calculates the virtual vector value of the document library and obtains the document closest to the target document according to the distance between the document vector in the document library and the virtual vector value. The data is indexed using a vector index, and its indexing speed is orders of magnitude higher than that of traditional text indexing, and the retrieval results are more conducive to subsequent data mining using statistical formulas.

[0027] 2. Based on the vector indexing technology, the present invention uses molecular sieve technology to further classify and filter the candidate data according to their quality, thereby limiting the calculation of data with high quality, improving the hit rate, and enhancing the efficiency of patent comparison.

[0028] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0029] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Example 1

[0031] This embodiment proposes a method for automatic comparison, analysis and processing of ultra-large-scale data, including:

[0032] 1. Classify all documents to obtain multiple document libraries, and calculate the virtual vector value of each document library.

[0033] 2. Input the target document, classify the target document, obtain the target document library, and obtain the target virtual vector value of the target document library.

[0034] ① Participle

[0035] Word segmentation technology is the basis of semantic recognition. The so-called word segmentation refers to "breaking up" a sentence into core words and single words and arranging them in order so that the computer can further analyze the meaning of the sentence.

[0036] Common word segmentation tools include Jieba, SnowNLP, THULAC, NLPIR, etc.

[0037] Compared with Jieba, SnowNLP has lower analysis efficiency and runs slowly. THULAC is a word segmentation tool from Tsinghua University, which has the functions of word segmentation and part-of-speech tagging. Its performance is close to Jieba, but its word segmentation accuracy is worse than Jieba. NLPIR has higher analysis efficiency and is easier to install than Jieba. However, Jieba supports a wider range of word segmentation methods.

[0038] Therefore, after comparing and screening a variety of word segmentation models, the system implementation of the present invention finally adopts Jieba for word segmentation.

[0039] The principle of Jieba word segmentation is to use a prefix dictionary to scan the entire word graph, thereby generating a directed acyclic graph (DAG) consisting of all possible generated words for a sentence. Dynamic programming is performed on all possible generation times to find the maximum probability path of the undirected graph, and the maximum segmentation combination based on word frequency can be found. For uncollected words, Jieba uses a hidden Markov model (HMM) based on the word-forming ability of Chinese characters for detection and learning.

[0040] ②Word vector calculation

[0041] Word2Vec is also the name of a third-party library that supports Python. Using this library, you can train a corpus formed by natural language word segmentation to obtain a multi-dimensional word vector model. Using this model, you can vectorize natural language. Since Word2Vec needs to simulate word vectors in the form of multi-dimensional vectors to describe and define each word, it is usually necessary to embed a high-dimensional space with the dimension of the dictionary into a lower-dimensional continuous vector space, so as to map the word to a vector on the real field.

[0042] Since Word2Vec does not run very fast, when performing patent analysis tasks, we can also design a probabilistic neural network of vocabulary based on the idea of ​​word vectors, introduce more professional vocabulary into the model, and use non-professional vocabulary as stop words. This method will make the system run much faster.

[0043] ③ Calculate the vector center point mean sum(M i ) / i, as the virtual vector value of the document library.

[0044] After we have the feature vector of the center point of a specific classification, we can select this feature vector as the reference point and perform vector indexing on all documents. This feature vector is called a virtual vector because there is no vector value corresponding to a specific document. This vector is just an average value of the vectors of all documents in this classification.

[0045] With the classified virtual vector, we can store the vector difference D between the vector of all documents and the virtual vector in the database as a numerical index, which is the vector index we want to use. This index is stored entirely in numbers, so the storage space occupied is much smaller than the text index. This also means that its search performance will be much higher than the text index.

[0046] The so-called vector index is to calculate the vector difference D between the vector of a certain document and the center vector of all classifications. It represents the distance of the document vector from the classification vector in different dimensions.

[0047] 3. Calculate the vector of the target document and each document in the target document library.

[0048] ① Perform word segmentation and use non-professional words as stop words.

[0049] ② Select multiple words with higher probability and weight to calculate word vectors,

[0050] ③ Calculate the sum of the Euclidean distances from the multiple words to the origin through word vectors as the target document target vector and the vector of each document in the target document library.

[0051] 4. Calculate the vector difference D1 from the target vector to the target virtual vector, calculate the vector difference D2 from the vector of each document in the target document library to the target virtual vector value, and select the document whose vector difference D2 is close to the vector difference D1 as the first approximate document of the target document.

[0052] In the above algorithm, we did not introduce word frequency weights. Introducing word frequency weights requires more complex algorithm design and higher computing performance. Practical tests have found that the introduction of word frequency weights does not significantly improve the accuracy of retrieval results.

[0053] For the retrieval scenarios of big data or ultra-large-scale data, the most effective way at present is to use index-based distributed retrieval. However, if the word vector technology is used directly to search the data set, the retrieval results are highly accurate and can meet the needs of classification, query, and comparison to a large extent. However, this query method is not only incredibly time-consuming, but also requires unimaginably large system resources. The performance of distributed retrieval will drop sharply in the combined retrieval of multiple words. When facing the retrieval of document similarity, it is even more stretched because of the complexity of the query. In addition, the query result accuracy of this retrieval is also very low, and it is not completely applicable in scenarios such as patent retrieval that require extremely high professionalism. Obviously, using this technology for query is even less suitable for big data scenarios.

[0054] An effective way to process big data and ultra-large-scale data is to pre-screen the data roughly. Therefore, in order to solve the performance problems caused by big data, we propose a fast text pan-retrieval model combined with similarity calculation and indexing technology to solve the pre-screening of big data and ultra-large-scale data.

[0055] The present invention first pre-screens the ultra-large-scale data through "vector indexing". Different from simple word vectors, the vector value in this embodiment not only represents the words appearing in the document, but also includes the probability of the words appearing.

[0056] The molecular sieve model is a fast filtering model for data result sets. It is a fast classifier formed by combining two sets of binary weak classifiers and is suitable for big data scenarios. When the amount of data that needs to be calculated and verified is large, we often do not verify or filter the data set in one step, but first perform a rough selection to select high-quality data, and then proceed to the next step of fine selection. The molecular sieve model is a model proposed based on this idea.

[0057] The naming of the molecular sieve model borrows the concept from the field of materials science. The so-called molecular sieve is a molecular sieve that distinguishes molecules of different substances by adsorption priority and size.

[0058] 5. Input the first approximate literature into the molecular sieve model, set the coarse particle screening condition a, and output the possible data set R1 and the impossible data set R2;

[0059] 6. Input the first approximate literature in the possible data set R1 into the molecular sieve model, set the fine particle screening condition b, and output the possible data set update R1 and the impossible data set R3;

[0060] 7. Merge the impossible data set R2 and the impossible data set R3 as the backup data set R4, and use the updated possible data set R1 as the result set R;

[0061] 8. Perform precise verification on the result set R. If similar documents are found, the task is terminated. If no similar documents are found, the search continues in the backup data set R4 until similar documents are found.

[0062] When searching and analyzing text, we often need to adopt different data analysis strategies and algorithm designs for different types of validation data sets. For example, if most of the result sets in the validation data set are correct sets, the algorithm can be designed to be more complex. However, if the validation data set contains fewer correct result sets, our algorithm will prioritize performance.

[0063] The significance of the molecular sieve model is that it can classify data sets. According to different screening results, algorithms of different complexity can be adapted for data analysis, thereby further improving the overall computing performance of the system.

[0064] Example 2

[0065] This embodiment also proposes a system for automatic comparison, analysis and processing of ultra-large-scale data, including:

[0066] A first calculation module is used to obtain multiple document libraries and calculate the virtual vector value of each document library;

[0067] The first processing module is used to input the target document, classify the target document, obtain the document library corresponding to the target document as the target document library, and use the virtual vector value of the target document library as the target virtual vector value;

[0068] A second calculation module is used to calculate the vector of each document in the target document library, wherein the target document library includes the target document and a comparison document group, wherein the comparison document group is the documents in the target document library other than the target document;

[0069] A third calculation module is used to calculate the vector difference D1 from the vector of the target document to the target virtual vector value, and calculate the vector difference D2 from the vector of each document in the comparison document group to the target virtual vector value;

[0070] A first selection module is used to calculate the distance between the vector difference D2 and the vector difference D1, and select the document corresponding to the vector difference D2 whose distance is less than a set threshold as the first approximate document of the target document;

[0071] A first input module, used to input the first approximate literature into a molecular sieve model and set a coarse particle screening condition a;

[0072] A first output module, used for outputting a possible data set R1 and an impossible data set R2;

[0073] The second input module is used to input the first approximate document in the possible data set R1 into the molecular sieve model to set the fine particle screening condition b;

[0074] A second output module, used to output a possible data set update R1 and an impossible data set R3;

[0075] The third processing module is used to merge the impossible data set R2 and the impossible data set R3 as a backup data set R4, and use the updated possible data set R1 as a result set R;

[0076] The first search module is used to perform data analysis on the result set R. If similar documents are found, the task is terminated. If no similar documents are found, the search continues in the backup data set R4 until similar documents are found.

[0077] Furthermore, the first calculation module includes:

[0078] The first processing module is used to segment each document in any document library and calculate the frequency of each word in the document;

[0079] The fourth calculation module is used to calculate the word vectors of multiple words with a frequency higher than a set threshold in each document, and calculate the mean of the word vectors to obtain the vector center point Mi, i∈ (1, 2, 3, ...);

[0080] The fifth calculation module is used to calculate the vector center point mean sum(Mi) / i of all documents in the document library as the virtual vector value of the document library.

[0081] Furthermore, the second calculation module includes:

[0082] The second processing module is used to segment any document and use non-professional words as stop words;

[0083] The second selection module is used to select multiple words whose probabilities and weights are higher than a set threshold to calculate word vectors, so as to obtain word vectors of the multiple words;

[0084] The sixth calculation module is used to calculate the sum of the Euclidean distances from the word vectors of the multiple words to the origin as the vector of each document in the target document library.

[0085] Example 3

[0086] This embodiment also proposes a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the method for automatic comparison, analysis and processing of ultra-large-scale data described in Example 1 are implemented.

[0087] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatic comparison, analysis and processing of ultra-large-scale data, characterized in that: The following steps are included: Obtain multiple document libraries, and calculate a virtual vector value of each document library; Input the target document, classify the target document, and obtain the document library corresponding to the target document as the target document library; The virtual vector value of the target document library is used as the target virtual vector value; Calculate the vector of each document in the target document library, wherein the target document library includes the target document and a comparison document group, wherein the comparison document group is the documents in the target document library other than the target document; Calculate the vector difference D1 from the vector of the target document to the target virtual vector value, and calculate the vector difference D2 from the vector of each document in the comparison document group to the target virtual vector value; Calculate the distance between the vector difference D2 and the vector difference D1, and select the document corresponding to the vector difference D2 whose distance is less than a set threshold as the first approximate document of the target document; The step of calculating the virtual vector value of each document library comprises: For any document library, segment each document in the document library and calculate the frequency of each word appearing in the document; Calculate the word vectors of multiple words in each document whose frequency is higher than the set threshold, and calculate the mean of the word vectors to obtain the vector center point Mi, i∈(1,2,3,…); Calculate the vector center point mean sum(Mi) / i of all documents in the document library as the virtual vector value of the document library; The following steps are also included: Input the first approximate literature into the molecular sieve model and set the coarse particle screening condition a; Output possible data set R1 and impossible data set R2; Input the first approximate document in the possible data set R1 into the molecular sieve model and set the fine particle screening condition b; Output possible data set update R1 and impossible data set R3; Merge the impossible data set R2 and the impossible data set R3 as the backup data set R4, and use the updated possible data set R1 as the result set R; Data analysis is performed on the result set R. If similar documents are found, the task is terminated. If no similar documents are found, the search continues in the backup data set R4 until similar documents are found.

2. The method for automatic comparison and analysis of ultra-large-scale data according to claim 1, characterized in that: The step of calculating the vector of each document in the target document library includes: Segment any document and use non-professional words as stop words; Select multiple words whose probabilities and weights are higher than a set threshold to calculate word vectors, and obtain word vectors for the multiple words; The sum of the Euclidean distances from the word vectors of the multiple words to the origin is calculated as the vector of each document in the target document library.

3. A system for automatic comparison, analysis and processing of ultra-large-scale data, characterized in that: include, A first calculation module is used to obtain multiple document libraries and calculate the virtual vector value of each document library; The first processing module is used to input the target document, classify the target document, obtain the document library corresponding to the target document as the target document library, and use the virtual vector value of the target document library as the target virtual vector value; A second calculation module is used to calculate the vector of each document in the target document library, wherein the target document library includes the target document and a comparison document group, wherein the comparison document group is the documents in the target document library other than the target document; A third calculation module is used to calculate the vector difference D1 from the vector of the target document to the target virtual vector value, and calculate the vector difference D2 from the vector of each document in the comparison document group to the target virtual vector value; A first selection module is used to calculate the distance between the vector difference D2 and the vector difference D1, and select the document corresponding to the vector difference D2 whose distance is less than a set threshold as the first approximate document of the target document; The first calculation module includes: The first processing module is used to segment each document in any document library and calculate the frequency of each word in the document; The fourth calculation module is used to calculate the word vectors of multiple words with a frequency higher than a set threshold in each document, and calculate the mean of the word vectors to obtain the vector center point Mi, i∈ (1, 2, 3, ...); The fifth calculation module is used to calculate the vector center point mean sum(Mi) / i of all documents in the document library as the virtual vector value of the document library; Also includes, A first input module, used to input the first approximate literature into a molecular sieve model and set a coarse particle screening condition a; A first output module, used for outputting a possible data set R1 and an impossible data set R2; The second input module is used to input the first approximate document in the possible data set R1 into the molecular sieve model to set the fine particle screening condition b; A second output module, used to output a possible data set update R1 and an impossible data set R3; The third processing module is used to merge the impossible data set R2 and the impossible data set R3 as a backup data set R4, and use the updated possible data set R1 as a result set R; The first search module is used to perform data analysis on the result set R. If similar documents are found, the task is terminated. If no similar documents are found, the search continues in the backup data set R4 until similar documents are found.

4. The ultra-large-scale data automated comparison, analysis and processing system according to claim 3, characterized in that: The second calculation module includes: The second processing module is used to segment any document and use non-professional words as stop words; The second selection module is used to select multiple words whose probabilities and weights are higher than a set threshold to calculate word vectors, so as to obtain word vectors of the multiple words; The sixth calculation module is used to calculate the sum of the Euclidean distances from the word vectors of the multiple words to the origin as the vector of each document in the target document library.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the method for automatic comparison, analysis and processing of ultra-large-scale data as described in any one of claims 1-2 are implemented.