Similarity detection method and system for science and technology project texts

Abstract

The invention belongs to the technical field of computer software data processing and text mining, and particularly relates to a similarity detection method and system for science and technology project texts. The method comprises the following steps: step 1, obtaining a text, and preprocessing the text; 2, judging an application scene of the text, performing feature extraction according to selection of the application scene, then converting the extracted features into character strings with fixed lengths, and finally obtaining fingerprint features of the text through Hash calculation; 3, creating an index instance, adding the data into an index, constructing a Faiiss vector retrieval library, then performing similarity retrieval through an AAN algorithm, and obtaining the similarity between the fingerprint features of the text and the text in the index library; and 4, sorting and filtering the retrieval results, and judging whether text repetition exists or not according to a set threshold value and outputting the text repetition. According to the invention, the accuracy and reliability of the detection result are improved, and the detection efficiency is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer software data processing and text mining, and specifically relates to a similarity detection method and system for scientific and technological project texts. Background Art

[0002] In science and technology project management and academic research, text similarity detection is a crucial task. With the rapid development of science and technology, the number of science and technology project applications and academic papers has increased dramatically, but the problems of plagiarism and duplicate applications have also become increasingly serious. In order to ensure the originality and academic integrity of science and technology projects, it is particularly important to develop an efficient and accurate method for text similarity detection of science and technology projects.

[0003] At present, the existing technologies that are closest to the text similarity detection methods of the claimed scientific and technological projects are mainly based on the following methods: TF-IDF (term frequency-inverse document frequency) method, vector space model (VSM), rule-based text comparison method, machine learning and deep learning methods, etc. These methods have the following problems and shortcomings: (1) Insufficient semantic understanding: Most of the existing technologies are based on word frequency or simple text matching, ignoring the semantic relationship and contextual information between words, resulting in poor results when processing texts containing polysemous words or complex contexts. (2) Low computational efficiency: For long texts or large-scale text data sets, the computational efficiency of the existing technologies is low and it is difficult to meet the real-time requirements. (3) High risk of misjudgment: Due to the complexity of the text and the limitations of the existing technologies, the existing methods may have a certain risk of misjudgment when detecting text similarity. In addition, they also have the following technical difficulties: (1) Data acquisition and processing: Building a large-scale, high-quality text data set is the basis for training machine learning models and verifying the effectiveness of algorithms, but this process is usually time-consuming and labor-intensive. (2) Algorithm optimization: How to improve the accuracy and robustness of the algorithm while ensuring computational efficiency is a huge challenge. (3) Semantic modeling: How to effectively perform semantic modeling on text to capture the semantic relationship and contextual information between words is the key to improving the effect of text similarity detection.

[0004] In summary, although the existing technology has made some progress in the detection of similarity between scientific and technological project texts, there are still many problems and challenges. In order to improve the detection effect and application value, it is necessary to continuously explore new methods and technical means. Summary of the invention

[0005] The purpose of the present invention is to solve the following problems: (1) When processing scientific and technological project texts, the prior art often judges similarity only based on word frequency or simple text matching, which lacks the depth of semantic understanding and ignores the deep semantic relationship and contextual information between words. This results in low accuracy of similarity evaluation when detecting scientific and technological project texts with complex semantics or professional terms. (2) The prior art faces the problem of low computational efficiency when processing large-scale data sets. At the same time, for the ever-increasing amount of new text data, the existing methods may find it difficult to ensure efficient scalability and real-time performance. (3) Due to the complexity and diversity of text content, the prior art is prone to misjudgment (i.e., judging dissimilar texts as similar) or missed judgment (i.e., ignoring similar texts) when detecting text similarity. This seriously affects the accuracy and reliability of the detection results.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a similarity detection method for scientific and technological project texts, comprising the following steps:

[0007] Step 1: Obtain the text and preprocess the text;

[0008] Step 2: Determine the application scenario of the text, select feature extraction based on the application scenario, then convert the extracted features into a string of fixed length, and finally obtain the fingerprint features of the text through hash calculation;

[0009] Step 3: Create an index instance, add data to the index, build a Faiss vector retrieval library, and then perform similarity retrieval using the AAN algorithm to obtain the similarity between the fingerprint features of the text and the text in the index library;

[0010] Step 4: Sort and filter the search results, and determine whether there is text duplication based on the set threshold and output it.

[0011] In step 3, a priority queue is used to manage candidate points during the search process, and specific operations include: building a priority queue, inserting a queue, deleting and updating a queue, and maintaining a queue.

[0012] In step 3, a multi-core CPU or GPU is used to initiate multi-process synchronous execution of the detection algorithm to achieve parallel computing.

[0013] The step 3 also includes the step of adjusting the parameters of the ANN algorithm, wherein the parameters include the number of numbers and the maximum number of points in each subspace.

[0014] In step 2, when the application scenario is fast detection, TF-IDF is used to extract features from the text, and when the application scenario is deep detection, Word2Vec algorithm is used to extract features from the text.

[0015] In step 2, the hash calculation uses the SHA-256 algorithm to generate the fingerprint feature of the text.

[0016] In addition, the present invention also provides a similarity detection system for scientific and technological project texts, comprising:

[0017] Text acquisition module: used to acquire text and pre-process the text;

[0018] Fingerprint feature generation module: used to determine the application scenario of the text, select feature extraction according to the application scenario, then convert the extracted features into a string of fixed length, and finally obtain the fingerprint features of the text through hash calculation;

[0019] Similarity calculation module: used to create an index instance, add data to the index, build a Faiss vector retrieval library, and then perform similarity retrieval through the AAN algorithm to obtain the similarity between the fingerprint features of the text and the text in the index library;

[0020] Output module: Sort and filter the search results, and determine whether there is text duplication based on the set threshold and output it.

[0021] The similarity detection system for scientific and technological project texts is implemented using a master-slave server architecture, in which the master server is responsible for processing external requests and scheduling tasks, and the slave server is responsible for data storage, index construction and comparison calculation.

[0022] The similarity detection system for scientific and technological project texts further includes:

[0023] Data backup module: used to regularly back up data in the relational database to a reliable storage medium to ensure that data is not lost; at the same time, regular snapshots and AOF log records are also taken for cached data in the Redis memory database;

[0024] Data synchronization module: realizes real-time or quasi-real-time data synchronization between relational database and Redis memory database, ensuring the consistency and latest of cached data:

[0025] Data encryption module: used to encrypt, store and transmit sensitive data;

[0026] Resource monitoring and dynamic adjustment module: used to monitor system resource usage and dynamically adjust system configuration based on resource usage.

[0027] The fingerprint feature generation module and the similarity calculation module adopt an asynchronous processing mechanism.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) Improve detection accuracy and efficiency:

[0030] Professional word segmentation engine: Adopt a professional word segmentation engine for scientific and technological projects to ensure the professionalism, accuracy and efficiency of the word segmentation process, thereby improving the quality of text fingerprint feature generation.

[0031] Combination of multiple algorithms: Combining advanced hash algorithms (such as SHA-256) with natural language processing (NLP) technology (such as TF-IDF, Word2Vec), and integrating the advantages of multiple methods, the generated text fingerprint features are more representative and robust.

[0032] Multi-language support: Through multi-language pre-trained language models, effective processing of Chinese, English, and Chinese-English mixed texts is achieved, expanding the scope of application of the system.

[0033] (2) Optimize similarity calculation:

[0034] Cosine similarity and threshold setting: By calculating the cosine similarity of text fingerprint features and setting a reasonable threshold, accurate judgment of text similarity is achieved.

[0035] Approximate nearest neighbor search algorithm (ANN) optimization: The similarity calculation is optimized using algorithms such as Faiss. Through multi-tree strategies, priority queues, parallel computing and parameter tuning, the calculation speed and accuracy of large-scale data sets are significantly improved.

[0036] (3) Flexibility and scalability of system architecture:

[0037] Master-slave server architecture: The master-slave node strategy of the Elasticsearch cluster is adopted to ensure the high availability and load balancing of the system, and improve the stability and reliability of the system.

[0038] Distributed and cluster deployment: Supports multiple operating systems and Docker containerization technology to facilitate rapid deployment and migration. It also supports horizontal expansion to increase system processing capabilities by adding slave servers and database instances to meet growing business needs.

[0039] Read-write separation and caching strategy: The read-write separation strategy is used to improve database access efficiency, and the Redis memory database is used to cache high-frequency query results, reducing database access pressure and further improving the overall performance of the system.

[0040] In summary, the present invention improves the accuracy, computational efficiency and scalability of similarity detection of scientific and technological project texts by introducing more advanced text representation technology (such as pre-trained language models, semantic embedding, etc.), optimizing similarity calculation algorithms (such as similarity metrics based on deep learning), and designing efficient text processing processes. At the same time, through refined parameter tuning and experimental verification, the risks of misjudgment and missed judgment are reduced to ensure the accuracy and reliability of the detection results. Therefore, the present invention shows significant advantages in terms of detection accuracy, computational efficiency, flexibility and scalability of system architecture, and has high practical application value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a flow chart of fingerprint feature generation in a similarity detection method for scientific and technological project texts provided in an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of the system architecture of a similarity detection system for scientific and technological project texts provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the structure of a similarity detection system for scientific and technological project texts provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] Embodiment 1

[0046] Embodiment 1 of the present invention provides a method for detecting similarity of scientific and technological project texts, comprising the following steps:

[0047] Step 1: Get the text and preprocess it.

[0048] Different from the conventional preprocessing which only cleans the text and performs word segmentation after removing HTML tags and stop words, in this embodiment, a professional word segmentation engine for scientific and technological projects is used for preprocessing. In addition to basic text cleaning, the engine also has a word segmentation function for scientific and technological professional words to ensure that the word segmentation is professional, accurate and efficient.

[0049] Step 2: Determine the application scenario of the text, select feature extraction based on the application scenario, then convert the extracted features into a string of fixed length, and finally apply hash calculation to the string to obtain the fingerprint features of the text.

[0050] like Figure 1 As shown, in step 2 of this embodiment, when the application scenario is fast detection, TF-IDF is used to extract features from the text, and when the application scenario is deep detection, the Word2Vec algorithm is used to extract features from the text. TF-IDF is to calculate the TF-IDF value (term frequency-inverse document frequency) of each word. The algorithm calculates the importance of each word or phrase. The Word2Vec algorithm obtains the vector representation of each word and aggregates it into a vector of the text. This embodiment uses the Word2Vec model library generated based on scientific and technological big data training. The vectors generated by this model library are more in line with scientific and technological business.

[0051] In step 2, the hash calculation uses the SHA-256 algorithm to generate the fingerprint feature of the text. Through the SHA-256 algorithm, the fingerprint feature of the text can be generated and output, and the output is the SHA-256 fingerprint of the text, which is subsequently used in the application scenario of similarity detection of scientific and technological project texts.

[0052] Furthermore, in this embodiment, for long texts such as the content of scientific and technological project construction and the description of derived scientific and technological achievements, a strategy of first performing feature extraction and text representation according to paragraph segments, and then merging and processing to generate the final fingerprint features is adopted, which can improve the robustness and recognition of the features.

[0053] In this embodiment, for Chinese, English, and Chinese-English mixed texts, the pre-trained language model Word2Vec that supports multiple languages ​​is used, and text vectorization training is performed on scientific research big data to generate a model library. Then, based on the model library, the project research content text is converted into fingerprint features to ensure that the similarity between texts in different languages ​​can be accurately calculated.

[0054] Step 3: Create an index instance, add data to the index, build a Faiss vector retrieval library, and then perform similarity retrieval through the AAN algorithm (Approximate Nearest Neighbor) to obtain the fingerprint features of the text and the similarity between the text in the index library.

[0055] In this embodiment, during similarity retrieval, text similarity calculation adopts an optimization adjustment strategy for the approximate nearest neighbor search algorithm. The following is a description of the solution process:

[0056] (1) Build an index: Create an index instance and add data to the index. To optimize index adjustment, a multi-tree strategy is used to build multiple index structures (Faiss vector retrieval library uses multiple index types such as Flat, IVF, HNSW, etc.) to increase the diversity and accuracy of the search.

[0057] (2) Perform similarity search: Perform an index search on the normalized sample vector to obtain the most similar vector in the index and its similarity score. The retrieval calculation similarity uses cosine similarity calculation. Using the generated text fingerprint features, the cosine similarity between the vectors is calculated to determine the similarity between the sample and the text in the index. The similarity threshold is set according to actual needs. When the calculated similarity exceeds the threshold, it is considered that the text is likely to be duplicated or copied.

[0058] (3) Result processing: Further process the search results as needed, such as sorting, filtering, etc.

[0059] In step 3, in order to improve the efficiency of text similarity calculation, the optimization measures adopted include:

[0060] (1) Use a priority queue to manage candidate points during the search process to ensure that the candidate points that are most likely to become similar points are processed first. A priority queue is a special queue in which elements are assigned priorities. The order in which elements are removed from the queue no longer follows the traditional first-in-first-out (FIFO) principle, but is determined based on the priority of the element. By defining a suitable priority evaluation function, building an efficient priority queue data structure, and adopting reasonable search strategies and maintenance measures, the performance and accuracy of the search algorithm can be significantly improved. Specific operations include: building a priority queue, inserting a queue, deleting and updating a queue, and maintaining a queue.

[0061] (2) Parallel computing: Use multi-core CPUs or GPUs to initiate multiple processes to synchronously execute the detection algorithm to achieve parallel computing, thereby further improving the search speed.

[0062] (3) Parameter tuning: According to the specific application scenario and data characteristics, adjust the parameters of the approximate nearest neighbor search algorithm (such as the number of trees, the maximum number of points in each subspace, etc.) to achieve optimal search efficiency and accuracy.

[0063] Step 4: Sort and filter the search results, and determine whether there is text duplication based on the set threshold and output it.

[0064] Embodiment 2

[0065] like Figure 2As shown, the second embodiment of the present invention provides a similarity detection system for scientific and technological project texts, which is used to check for duplicate content in scientific and technological project texts. It is designed and developed in the form of a WebApi interface, and the fingerprint features of the text data set and the fingerprint features of the input text are created through the interface. The text similarity is calculated through the fingerprint features of the text to determine whether the text is repeated. The TPTPDS system is divided into two parts: a duplicate checking engine (middleware) and a duplicate checking interface (WEB service). It supports Windos\Linux deployment, distributed + cluster deployment, Msql\Sql Server\Sqlite database, read-write separation, Redis memory database cache, Chinese\English\Chinese-English mixed text duplicate checking, and comes with a disaster recovery backup strategy. Figure 3 As shown, the system of this embodiment specifically includes:

[0066] Text acquisition module: used to acquire text and pre-process the text;

[0067] Fingerprint feature generation module: used to determine the application scenario of the text, select feature extraction according to the application scenario, then convert the extracted features into a string of fixed length, and finally obtain the fingerprint features of the text through hash calculation;

[0068] Similarity calculation module: used to create an index instance, add data to the index, build a Faiss vector retrieval library, and then perform similarity retrieval through the AAN algorithm to obtain the similarity between the fingerprint features of the text and the text in the index library;

[0069] Output module: Sort and filter the search results, and determine whether there is text duplication based on the set threshold and output it.

[0070] like Figure 3 As shown, a similarity detection system for scientific and technological project texts in this embodiment adopts a master-slave node strategy of an Elasticsearch cluster to ensure high availability and load balancing of the system. It is implemented based on a master-slave server architecture. The master server is responsible for processing external requests and scheduling tasks, and the slave server is responsible for data warehousing, index construction, and comparison calculations.

[0071] In addition, this system supports deployment on Windows and Linux systems through distributed and clustered deployment, and uses Docker containerization technology to achieve rapid deployment and migration. It supports horizontal expansion and improves system processing capabilities by adding slave servers and database instances. It supports multiple databases such as MySQL, SQL Server, and SQLite, and adopts a read-write separation strategy to improve database access efficiency, that is, database operations are divided into read operations and write operations, and these operations are performed on different database servers or database instances respectively. These operations can be processed in parallel, thereby improving overall performance. For large-scale data sets, the Redis memory database is introduced to cache high-frequency query results to reduce database access pressure.

[0072] Furthermore, the similarity detection system for scientific and technological project texts of this embodiment further includes:

[0073] Data backup module: used to regularly back up data in the relational database to a reliable storage medium to ensure that data is not lost; at the same time, regular snapshots and AOF log records are also taken for cached data in the Redis memory database;

[0074] Data synchronization module: realizes real-time or quasi-real-time data synchronization between the relational database and the Redis memory database to ensure the consistency and currency of the cached data: when the Redis node fails, the data synchronization module can quickly restore data from the relational database.

[0075] Furthermore, in the similarity detection system for scientific and technological project texts of this embodiment, the fingerprint feature generation module and the similarity calculation module adopt an asynchronous processing mechanism. For tasks that take a long time (such as large text fingerprint generation and large-scale data comparison), the asynchronous processing mechanism can improve the system response speed and user experience.

[0076] Furthermore, the similarity detection system for scientific and technological project texts of this embodiment further includes:

[0077] Data encryption module: used to encrypt, store and transmit sensitive data (such as user information, text content, etc.);

[0078] Resource monitoring and dynamic adjustment module: used to monitor system resource usage (such as CPU, memory, disk I / O, etc.), and dynamically adjust system configuration according to resource usage (such as adding servers, database instances, etc.) to ensure that the system can still run stably under high load.

[0079] In addition, the similarity detection system for scientific and technological project texts of this embodiment also includes a permission control module, which is used to implement fine-grained permission management to ensure that different users can only access data and functions within their authorized scope.

[0080] The present invention has the following innovative features:

[0081] 1. Comprehensive text fingerprint feature generation: This invention combines advanced hash algorithms (such as SHA-256) with natural language processing (NLP) technology (such as TF-IDF, Word2Vec) to innovatively propose a comprehensive text fingerprint feature generation method. This method not only uses the hash algorithm to ensure the unique identification of the text, but also extracts the deep semantic features of the text through NLP technology, so that the generated fingerprint features are both efficient and highly semantically recognizable. In particular, for scientific and technological project texts, the use of a professional scientific and technological word segmentation engine and a Word2Vec model library trained based on scientific and technological big data significantly improves the accuracy and efficiency of feature extraction.

[0082] 2. Text vectorization training model library with multi-language support: Build a multi-language pre-trained language model library that supports Chinese, English, and Chinese-English mixed texts, and conduct targeted training based on scientific research big data to generate a text vectorization model that meets the characteristics of science and technology business. This innovation solves the semantic understanding problem faced by traditional methods when processing multi-language science and technology project texts, and improves the accuracy and robustness of cross-language text similarity detection.

[0083] 3. Optimized text similarity calculation strategy: The text similarity calculation optimization strategy based on the approximate nearest neighbor search algorithm (Faiss) includes multiple technical means such as multi-tree strategy, priority queue, parallel computing and parameter tuning. The combined use of these strategies not only improves the similarity calculation efficiency under large-scale data sets, but also ensures the diversity and accuracy of the search, providing strong technical support for the rapid similarity detection of scientific and technological project texts.

[0084] 4. Efficient and scalable system architecture design: The present invention adopts a master-slave server architecture and a distributed cluster deployment solution, combined with Elasticsearch cluster and Docker containerization technology, to achieve high availability, load balancing and rapid deployment and migration of the system. At the same time, it supports a variety of database systems and adopts a read-write separation strategy and Redis memory database caching technology to further enhance the system's data processing capabilities and response speed. This efficient and scalable system architecture design provides a solid platform support for similarity detection of scientific and technological project texts.

[0085] In summary, the present invention not only solves the key technical problems in text duplication checking of scientific and technological projects, but also provides scientific research management departments and scientific and technological users with an efficient, reliable and easy-to-use text duplication checking solution by optimizing the system architecture, improving performance and scalability, and strengthening security and authority management.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A similarity detection method for scientific and technological project texts, characterized in that: The following steps are involved: Step 1: Obtain the text and preprocess the text; Step 2: Determine the application scenario of the text, select feature extraction based on the application scenario, then convert the extracted features into a string of fixed length, and finally obtain the fingerprint features of the text through hash calculation; Step 3: Create an index instance, add data to the index, build a Faiss vector retrieval library, and then perform similarity retrieval using the AAN algorithm to obtain the similarity between the fingerprint features of the text and the text in the index library; Step 4: Sort and filter the search results, and determine whether there is text duplication based on the set threshold and output it.

2. A similarity detection method for scientific and technological project texts according to claim 1, characterized in that: In step 3, a priority queue is used to manage candidate points during the search process, and specific operations include: building a priority queue, inserting a queue, deleting and updating a queue, and maintaining a queue.

3. A similarity detection method for scientific and technological project texts according to claim 1, characterized in that: In step 3, a multi-core CPU or GPU is used to initiate multi-process synchronous execution of the detection algorithm to achieve parallel computing.

4. The similarity detection method of a scientific and technological project text according to claim 1 is characterized in that: The step 3 also includes the step of adjusting the parameters of the ANN algorithm, wherein the parameters include the number of numbers and the maximum number of points in each subspace.

5. The similarity detection method of a scientific and technological project text according to claim 1 is characterized in that: In step 2, when the application scenario is fast detection, TF-IDF is used to extract features from the text, and when the application scenario is deep detection, Word2Vec algorithm is used to extract features from the text.

6. A similarity detection method for scientific and technological project texts according to claim 1, characterized in that: In step 2, the hash calculation uses the SHA-256 algorithm to generate the fingerprint feature of the text.

7. A similarity detection system for scientific and technological project texts, characterized in that: include: Text acquisition module: used to acquire text and pre-process the text; Fingerprint feature generation module: used to determine the application scenario of the text, select feature extraction according to the application scenario, then convert the extracted features into a string of fixed length, and finally obtain the fingerprint features of the text through hash calculation; Similarity calculation module: used to create an index instance, add data to the index, build a Faiss vector retrieval library, and then perform similarity retrieval through the AAN algorithm to obtain the similarity between the fingerprint features of the text and the text in the index library; Output module: Sort and filter the search results, and determine whether there is text duplication based on the set threshold and output it.

8. A similarity detection system for scientific and technological project texts according to claim 7, characterized in that: It adopts a master-slave server architecture, where the master server is responsible for processing external requests and scheduling tasks, and the slave server is responsible for data storage, index construction, and comparison calculations.

9. A similarity detection system for scientific and technological project texts according to claim 7, characterized in that: Also includes: Data backup module: used to regularly back up data in the relational database to reliable storage media to ensure that data is not lost; At the same time, regular snapshots and AOF logs are also taken for cached data in the Redis memory database; Data synchronization module: realizes real-time or quasi-real-time data synchronization between relational database and Redis memory database, ensuring the consistency and latest of cached data: Data encryption module: used to encrypt, store and transmit sensitive data; Resource monitoring and dynamic adjustment module: used to monitor system resource usage and dynamically adjust system configuration based on resource usage.

10. A similarity detection system for scientific and technological project texts according to claim 7, characterized in that: The fingerprint feature generation module and the similarity calculation module adopt an asynchronous processing mechanism.

Images