Text information compression method and compression device for indexing a distributed database

Abstract

The application discloses a text information compression method and compression device for indexing a distributed database. The text information compression method comprises the following steps: obtaining a text to be processed and preprocessing the text to obtain a pure text in a predetermined format; performing multi-level semantic paragraph division on the pure text in the predetermined format to obtain a corresponding semantic paragraph set; performing fragment division and mapping processing on each semantic paragraph in the semantic paragraph set by considering weight and position to obtain a signature identifier of each semantic paragraph; and performing indexing processing on the signature identifier, combining the signature identifier with identification information of the semantic paragraph, and generating compressed text information for indexing the distributed database. The method can avoid misjudgment caused by changes in the chapter structure, can maintain unbiased estimation of similarity while performing high compression, and can help to identify texts that have been rewritten, partially copied or restructured.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of text processing and information retrieval, and particularly relates to a text information compression method and compression device for indexing a distributed database. BACKGROUND

[0002] With the rapid development of digital technology, text data is growing explosively. In the fields of academic research, enterprise office, and Internet content dissemination, problems such as text plagiarism and repeated content are increasingly prominent. Text duplication detection technology has become a key support for ensuring information originality and compliance.

[0003] Currently, there are various methods in the field of text duplication detection. For example, the publication number CN115759082A provides a text duplication detection method and device based on an improved Simhash algorithm. The Simhash algorithm is used to calculate the similarity of texts. However, the algorithm has poor performance when facing problems such as word replacement, sentence transformation, or rewriting without changing the original meaning. The weight calculation of the word segmentation vector aggregation is usually set to 1 or the number of word segmentation occurrences, which can easily cause information loss, resulting in reduced accuracy of the final text vector. Moreover, the algorithm cannot well represent the word segmentation distribution information and context semantic information. For texts containing the same word segmentation but different meanings, the text vectors generated after adjusting the order of word segmentation are the same, which can easily result in a large number of overlapping text vectors. In addition, the publication number CN118885602A provides a text duplication detection method and system based on the LLM2Vec algorithm. The algorithm uses string matching-based algorithms such as the longest common subsequence algorithm, which is easily disturbed by text rewriting and synonym replacement. The tf-idf-based method cannot capture deep semantic information of texts, resulting in insufficient semantic understanding of text content and limited duplication detection accuracy.

[0004] Therefore, there is an urgent need for a text compression / duplication detection solution that can deeply integrate semantic understanding, efficiently retrieve, and have continuous optimization capability, aiming to solve the above technical defects and achieve efficient, accurate, and dynamically optimized text duplication detection. SUMMARY

[0005] In view of the defects in the above-mentioned prior art, the present application provides a text information compression method and compression device for indexing a distributed database. The method can avoid misjudgment caused by changes in the structure of the text. The method can perform high compression while maintaining unbiased estimation of similarity, which can help identify texts that have been semantically rewritten, partially plagiarized, or structurally reorganized.

[0006] In a first aspect, the present application provides a text information compression method for indexing a distributed database, comprising:

[0007] The method comprises the following steps:

[0008] perform multi-level semantic paragraph division on the pure text in the predetermined format, and give a corresponding semantic paragraph set;

[0009] perform fragment division and mapping processing on each semantic paragraph in the semantic paragraph set, taking into account the weight and position, and give a signature identification of each semantic paragraph;

[0010] perform indexing processing on the signature identification, and generate compressed text information for indexing the distributed database in combination with the signature identification and the identification information of the semantic paragraph.

[0011] Further, the text to be processed is obtained and preprocessed to give a pure text in a predetermined format, including:

[0012] Obtain multi-source text to be processed;

[0013] Perform unified encoding on the multi-source text to be processed to generate an initial text;

[0014] Perform cleaning and denoising processing on the initial text to obtain a pure text in a predetermined format.

[0015] Further, perform multi-level semantic paragraph division on the pure text in the predetermined format, and give a corresponding semantic paragraph set, including:

[0016] Perform punctuation and blank line recognition on the pure text in the predetermined format to determine the position and type of all punctuation in the pure text, and the position of the blank line formed by two or more consecutive line breaks;

[0017] Determine the forced segmentation point based on the position and type of all punctuation in the pure text, and the position of the blank line formed by two or more consecutive line breaks;

[0018] Perform a text cut on the pure text in the predetermined format according to the forced segmentation point to obtain a plurality of basic text segments;

[0019] Perform processing and analysis on the basic text segments through a sliding window to give a cosine similarity matrix;

[0020] Determine the cosine similarity rate of a plurality of continuous sliding windows based on the cosine similarity matrix, and give a semantic break point in combination with a preset amplitude threshold;

[0021] Perform a second text cut on the plurality of basic text segments based on the semantic break point to give an initial semantic paragraph set;

[0022] Filter the initial semantic paragraph set based on a preset filtering strategy to give a final semantic paragraph set.

[0023] Further, the semantic paragraph set is subjected to a segment division and mapping process considering weight and position, and a signature of each semantic paragraph is given, including:

[0024] The semantic paragraph set is subjected to a word segmentation process, and a word sequence of each semantic paragraph is generated;

[0025] The word sequence is subjected to a similarity calculation, and adjacent words with a similarity reaching a predetermined threshold are combined to give a first semantic segment set;

[0026] Based on a sliding window determination condition, each semantic paragraph in the semantic paragraph set is subjected to a segment division to give a second semantic segment set;

[0027] The first semantic segment set and the second semantic segment set are combined and filtered to obtain a final semantic segment set;

[0028] Based on a preset hash function family, each semantic segment in the final semantic segment set is subjected to an identification and multi-fusion weight process to give a signature of each semantic paragraph.

[0029] Further, the word sequence is subjected to a similarity calculation, and adjacent words with a similarity reaching a predetermined threshold are combined to give a first semantic segment set, including:

[0030] Each word sequence is subjected to a vectorization by a pre-trained language model to give a word vector of each word in the word sequence;

[0031] The word vectors of adjacent words in each word sequence are subjected to a similarity calculation to obtain a similarity of adjacent words;

[0032] Adjacent words with a similarity exceeding a predetermined threshold are combined into a merged word;

[0033] The merged word and the words not subjected to merging are respectively integrated as a semantic segment to obtain a first semantic segment set.

[0034] Further, based on a sliding window determination condition, each semantic paragraph in the semantic paragraph set is subjected to a segment division to give a second semantic segment set, including:

[0035] A character sequence of each semantic paragraph is determined;

[0036] Based on all character sequences, an occurrence probability of each character in each character sequence is given;

[0037] Based on the characters possessed by each character sequence and in combination with the corresponding occurrence probability, an information entropy of each semantic paragraph is given;

[0038] Based on the information entropy of each semantic paragraph, a corresponding sliding window size is determined;

[0039] segmenting each semantic paragraph based on a corresponding sliding window size to obtain a second semantic segment set.

[0040] Further, based on a preset hash function family, the semantic segments of each semantic paragraph are identified and subjected to multiple fusion weight processing, to give a signature identification of each semantic paragraph, including:

[0041] The semantic segments of each semantic paragraph are subjected to weight calculation and fusion by multiple weight calculation methods, to give a fusion weight of each semantic segment;

[0042] Each semantic segment is spliced with its position information to give a position-enhanced semantic segment;

[0043] Based on each hash function in the preset hash function family using different seed values, all position-enhanced semantic segments in each semantic paragraph are subjected to hash calculation, to give a hash value of each position-enhanced semantic segment corresponding to different hash functions;

[0044] Based on the seed value of each hash function, the identification of the text to be processed, and the fusion weight of each semantic segment, an adjustment value of each position-enhanced semantic segment corresponding to different hash functions is given;

[0045] The hash value of each position-enhanced semantic segment in the semantic paragraph corresponding to different hash functions is fused with the corresponding adjustment value, to give a weighted hash value of each position-enhanced semantic segment in the semantic paragraph corresponding to different hash functions;

[0046] The minimum weighted hash value of each hash function is selected from all the weighted hash values of the semantic paragraph, to give a signature identification of each semantic paragraph.

[0047] Further, the signature identification is subjected to indexing processing, and the signature identification and the identification information of the semantic paragraph are combined to generate compressed text information for indexing a distributed database, including:

[0048] Based on a preset division strategy, the signature identification is divided into multiple wave bands;

[0049] The corresponding byte sequence of each wave band is calculated, to give a bucket key corresponding to each wave band;

[0050] Taking the bucket key as an index, the signature identification, the text to be processed corresponding to the signature identification, and the identification information of the semantic paragraph corresponding to the signature identification are stored in the index distributed database.

[0051] Further, the method further includes:

[0052] The to-be-checked duplicate text is sequentially subjected to preprocessing, multi-level semantic paragraph division, segment division, mapping processing and indexing processing, and a signature mark and an indexing processing result of the to-be-checked duplicate text are given;

[0053] Based on the indexing processing result of the to-be-checked duplicate text, the corresponding signature mark is queried from the index distributed database;

[0054] Based on the signature mark of the to-be-checked duplicate text and the signature mark of the index distributed database, a text duplicate checking result is given.

[0055] In the second aspect, the application further provides a text information compression device for an index distributed database, which adopts the above-mentioned text information compression method, and the device comprises:

[0056] A preprocessing module is configured to acquire a to-be-processed text and perform preprocessing to give a pure text in a predetermined format;

[0057] A paragraph division module is configured to perform multi-level semantic paragraph division on the pure text in the predetermined format to give a corresponding semantic paragraph set;

[0058] An identification generation module is configured to perform segment division and mapping processing on each semantic paragraph in the semantic paragraph set in consideration of weight and position to give a signature mark of each semantic paragraph;

[0059] An indexing and storage module is configured to perform indexing processing on the signature mark and generate compressed text information for indexing the distributed database in combination with the signature mark and identification information of the semantic paragraph.

[0060] Further, the preprocessing module is configured to:

[0061] acquire a plurality of to-be-processed texts;

[0062] perform unified coding on the plurality of to-be-processed texts to generate an initial text;

[0063] perform cleaning and denoising processing on the initial text to obtain a pure text in a predetermined format.

[0064] Further, the paragraph division module is configured to:

[0065] perform punctuation and blank line recognition on the pure text in the predetermined format to determine the positions and types of all punctuation marks in the pure text and the positions of blank lines formed by two or more consecutive line breaks;

[0066] determine forced segmentation points based on the positions and types of all punctuation marks in the pure text and the positions of blank lines formed by two or more consecutive line breaks;

[0067] perform one-time text cutting on the pure text in the predetermined format according to the forced segmentation points to obtain a plurality of basic text segments;

[0068] The cosine similarity matrix is given by processing and analyzing the base text segments through a sliding window;

[0069] Based on the cosine similarity matrix, the cosine similarity change rates of multiple continuous sliding windows are determined, and the semantic breakpoints are given in combination with a preset amplitude threshold;

[0070] Based on the semantic breakpoints, the multiple base text segments are subjected to secondary text cutting to give an initial semantic paragraph set;

[0071] Based on a preset filtering strategy, the initial semantic paragraph set is filtered to give a final semantic paragraph set.

[0072] Further, the identification generation module is configured to:

[0073] Each semantic paragraph in the semantic paragraph set is subjected to word segmentation processing to generate a word sequence of each semantic paragraph;

[0074] Similarity calculation is performed on each word sequence, and adjacent words with a similarity reaching a predetermined threshold are combined to give a first semantic segment set;

[0075] Based on the sliding window determination condition, each semantic paragraph in the semantic paragraph set is subjected to segment division to give a second semantic segment set;

[0076] The first semantic segment set and the second semantic segment set are subjected to mixed filtering to obtain a final semantic segment set;

[0077] Based on a preset hash function family, each semantic segment in the final semantic segment set is subjected to identification and multiple fusion weight processing to give a signature identification of each semantic paragraph.

[0078] Further, the identification generation module is configured to:

[0079] Each word sequence is subjected to vectorization through a pre-trained language model to give a word vector of each word in the word sequence;

[0080] The word vectors of adjacent words in each word sequence are subjected to similarity calculation to obtain the similarity of adjacent words;

[0081] Adjacent words with a similarity exceeding a predetermined threshold are combined into a merged word;

[0082] The merged word and the words that have not been merged are integrated as a semantic segment respectively to obtain a first semantic segment set.

[0083] Further, the identification generation module is configured to:

[0084] The character sequence of each semantic paragraph is determined;

[0085] give the occurrence probability of each character in each character sequence based on all character sequences;

[0086] give the information entropy of each semantic paragraph based on the characters in each character sequence and the corresponding occurrence probability;

[0087] determine the corresponding sliding window size based on the information entropy of each semantic paragraph;

[0088] perform segment division on each semantic paragraph based on the corresponding sliding window size to obtain a second semantic segment set.

[0089] Further, the identification generation module is further configured to:

[0090] perform weight calculation and fusion on each semantic segment of each semantic paragraph by multiple weight calculation methods respectively to give the fusion weight of each semantic segment;

[0091] splice each semantic segment with its position information to give a position-enhanced semantic segment;

[0092] perform hash calculation on all position-enhanced semantic segments in each semantic paragraph based on each hash function in the preset hash function family using different seed values to give the hash value of each position-enhanced semantic segment corresponding to different hash functions;

[0093] give the adjustment value of each position-enhanced semantic segment corresponding to different hash functions based on the seed value of each hash function, the identification of the text to be processed, and the fusion weight of each semantic segment;

[0094] fuse the hash value of each position-enhanced semantic segment corresponding to different hash functions in the semantic paragraph with the corresponding adjustment value respectively to give the weighted hash value of each position-enhanced semantic segment corresponding to different hash functions in the semantic paragraph;

[0095] select the minimum weighted hash value of each hash function from all weighted hash values of the semantic paragraph to give the signature identification of each semantic paragraph.

[0096] Further, the index and storage module is configured to:

[0097] divide the signature identification into multiple wave bands based on a preset division strategy;

[0098] perform calculation on the corresponding byte sequence of all wave bands respectively to give the bucket key corresponding to each wave band;

[0099] store the signature identification, the identification information of the text to be processed corresponding to the signature identification, and the identification information of the semantic paragraph corresponding to the signature identification to the index distributed database by taking the bucket key as an index.

[0100] The application provides a text information compression method and compression device for indexing a distributed database.

[0101] (1) By semantic segmentation and generating high-distinguishable signature identification, accurate identification of semantic rewriting and partial plagiarism is realized. By dividing the text into paragraph units with complete semantics, misjudgment caused by changes in the chapter structure can be effectively avoided. And through the generated signature identification, the paragraph content can be highly compressed and the similarity unbiased estimation is maintained, which helps to identify texts that have been rewritten, partially plagiarized or structurally reorganized.

[0102] (2) When generating the signature identification, the paragraph-level processing after semantic segmentation can accurately capture local similarity, further improving the identification of texts that have been rewritten, partially plagiarized or structurally reorganized.

[0103] (3) By the fragment division method of semantic unit merging and dynamic K-gram combination, the semantic integrity and local features of semantic fragments are considered; by position enhancement, the text structure information is retained, which helps to restore the paragraph structure and sentence order of the text, and is suitable for downstream tasks such as document layout restoration, multi-document alignment and text duplication detection that require structure information; by the weight setting method of fusing semantic weight, TF-IDF weight and part-of-speech weight, high weight is assigned to the core content, which is preferentially retained during compression and removes redundant information, which can better reflect the essential features of the text than the conventional compression method; the efficient signature generation method of weighted MinHash compresses the text into fixed-length signatures that retain key semantics and structure, which is more suitable for large-scale text compression. BRIEF DESCRIPTION OF DRAWINGS

[0104] Figure 1 A flowchart of a text information compression method for indexing a distributed database is provided for the application;

[0105] Figure 2 A flowchart of a semantic paragraph set is provided for an embodiment of the application;

[0106] Figure 3 A flowchart of a signature identification is provided for an embodiment of the application;

[0107] Figure 4 A schematic diagram of determining the final semantic fragment set is provided for an embodiment of the application;

[0108] Figure 5 A flowchart of identification and multiple fusion weight processing is provided for an embodiment of the application;

[0109] Figure 6 A schematic diagram of determining the signature component is provided for an embodiment of the application;

[0110] Figure 7 A flowchart for generating compressed text information is provided for an embodiment of the present application;

[0111] Figure 8 A flowchart for checking duplication of text information is provided for an embodiment of the present application;

[0112] Figure 9 A schematic diagram of a text information compression device for indexing a distributed database is provided for the present application. DETAILED DESCRIPTION

[0113] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0114] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. "Multiple" generally includes at least two.

[0115] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the product or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such product or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the product or device including the element.

[0116] As Figure 1 shown, the present application provides a text information compression method for indexing a distributed database, which can include:

[0117] Obtaining the text to be processed and pre-processing to give the pure text in a predetermined format;

[0118] Performing multi-level semantic paragraph division on the pure text in the predetermined format to give the corresponding semantic paragraph set;

[0119] Performing fragment division and mapping processing on each semantic paragraph in the semantic paragraph set considering the weight and position to give the signature identification of each semantic paragraph;

[0120] The signature identifier is indexed, and the signature identifier and the semantic paragraph identifier are combined to generate compressed text information for indexing the distributed database.

[0121] In the process of obtaining the to-be-processed text and preprocessing, the pure text in the predetermined format can be obtained, which can include:

[0122] Obtain multiple source to-be-processed texts;

[0123] Uniformly encode the multiple source to-be-processed texts to generate initial texts;

[0124] Clean and denoise the initial texts to obtain pure texts in a predetermined format.

[0125] In actual application scenarios, when preprocessing the to-be-processed text, the original to-be-processed text in different formats can be converted into a unified UTF-8 encoded pure text, and irrelevant noise can be removed, so as to support input of texts / documents in multiple formats such as PDF, Word, HTML, TXT, and Markdown. For PDF and Word format documents, ApacheTika can be used for parsing and extraction, and finally pure text can be obtained; HTML format documents can be parsed by Jsoup to extract valid text within the tag to obtain pure text; TXT and Markdown format documents directly read the original text. When cleaning and denoising, fixed text segments that appear more than a certain number of times (such as headers and footers) can be identified, and noise can be removed in combination with the location characteristics of the text on the page. Based on the multilingual stop word dictionary and regular expressions, filter the words and special symbols without actual semantics to ensure that the subsequent processed text content is pure and complete in semantics.

[0126] In order to divide the preprocessed pure text into semantic paragraphs with complete semantics and filter invalid paragraphs without actual compression meaning, avoid short text fragments, data-intensive segments, and template content from interfering with subsequent fingerprint (signature identifier) generation and similarity calculation, a two-level processing strategy of primary segmentation + advanced segmentation is used for semantic segmentation processing to divide semantic paragraphs. Specifically, as shown in Figure 2 The predetermined format pure text is divided into multiple semantic paragraphs, and the corresponding semantic paragraph set is given, which can include:

[0127] Punctuation and blank line recognition is performed on the predetermined format pure text to determine the position and type of all punctuation in the pure text and the position of the blank line formed by two or more consecutive line breaks;

[0128] Based on the position and type of all punctuation in the pure text and the position of the blank line formed by two or more consecutive line breaks, the forced segmentation point is determined;

[0129] According to the forced segmentation point, the pure text of the predetermined format is subjected to text cutting once to obtain a plurality of basic text segments;

[0130] The plurality of basic text segments are processed and analyzed through a sliding window to give a cosine similarity matrix, including: slidingly selecting sentences for each basic text segment through the sliding window; vectorizing each sentence to give a corresponding semantic vector; processing the semantic vectors based on a cosine similarity algorithm to give semantic similarity between sentences corresponding to adjacent sliding windows; integrating the semantic similarity based on the sliding selection order of the sliding window to give the cosine similarity matrix;

[0131] Based on the cosine similarity matrix, the cosine similarity change rates of the plurality of continuous sliding windows are determined, and in combination with a preset amplitude threshold, semantic breakpoints are given.

[0132] Based on the semantic breakpoints, the plurality of basic text segments are subjected to secondary text cutting to give an initial semantic paragraph set.

[0133] The initial semantic paragraph set is filtered based on a preset filtering strategy to give a final semantic paragraph set. The preset filtering strategy includes: a length filtering strategy, a number proportion strategy, and a template matching strategy.

[0134] In an actual application scenario, the primary segmentation is a basic division stage driven by a rule engine. The rule engine has two types of core rules, namely, punctuation division rules and blank line detection rules. The punctuation division rules cover the logic of forced segmentation points of end-of-sentence punctuation (such as periods, question marks, and exclamation marks). When the end-of-sentence punctuation that meets the forced segmentation logic has a line feed character, the text after the line feed character is divided into a new semantic paragraph. The punctuation division rules also include the division prompt logic of in-sentence punctuation (such as commas and semicolons). The division prompt logic is only used as a potential segmentation reference and does not directly trigger segmentation. The blank line detection rules identify the text interval formed by two or more consecutive line feed characters in the pure text, and determine the interval position as a forced segmentation point. Through the combination of the punctuation division rules and the blank line detection rules, the preprocessed pure text is preliminarily cut into a plurality of basic text segments.

[0135] The advanced segmentation adopts the fine-tuned BERT-TextTiling model to perform semantic optimization on the basic text segments. During the semantic optimization, the sliding window is used as a unit, each sentence in the window is converted into a semantic vector, and then the cosine similarity algorithm is used to calculate the semantic similarity between the sentences to construct the cosine similarity matrix between the sentences in the window. When the cosine similarity of three consecutive sliding windows decreases by more than a preset amplitude threshold, it is determined that the position is a semantic break point, i.e., the semantic correlation of the front and rear text is significantly reduced, and the paragraph needs to be split. Based on these semantic break points, the primary segmentation result is adjusted twice, the semantic coherent basic segments are merged, and the semantic broken segments are split, and finally the initial semantic paragraph set with complete semantics and logical coherence is formed.

[0136] After obtaining the initial semantic paragraph set, it is also necessary to filter the invalid paragraphs in the initial semantic paragraph set. Specifically, a three-level filtering strategy (a preset filtering strategy) is used to screen the initial semantic paragraph set, so that only the effective paragraphs with actual value are retained.

[0137] The first-level filtering strategy is a length filtering strategy. There are short semantic paragraphs in the semantic paragraph set. Such short semantic paragraphs only contain scattered words or short sentences and cannot reflect complete semantics, which can easily lead to noise in subsequent signature identification generation and affect compression efficiency. The length filtering strategy can be used for filtering and removal. Specifically, the total number of characters of each semantic paragraph is counted by character counting, and the semantic paragraph with a character number less than a preset threshold is removed.

[0138] The second-level filtering strategy is a digital proportion filtering strategy. All digital characters in the semantic paragraph are extracted by using a regular expression, and the proportion of the digital characters in the total number of characters of the semantic paragraph is calculated. When the proportion exceeds a preset threshold, it is determined that the paragraph is invalid content dominated by data, and it is removed.

[0139] The third-level filtering strategy is a template matching strategy. Based on a pre-constructed document template library (including templates of common headers and footers, copyright declaration templates, table title templates, reference format templates, etc. in documents), a text similarity algorithm is used to calculate the similarity between the semantic paragraph to be detected and the text in the template library. When the similarity exceeds a preset threshold, it is determined that the semantic paragraph is template content. Such content is highly repetitive in different documents but has no actual originality, and it is removed. After three-level filtering, the remaining semantic paragraphs are effective semantic paragraphs used for subsequent signature identification generation.

[0140] As shown in Figure 3 After obtaining the effective semantic paragraph set, each semantic paragraph in the semantic paragraph set can be subjected to segment division and mapping processing considering the weight and position, and the signature identification of each semantic paragraph is given, including:

[0141] perform word segmentation processing on each semantic paragraph in the semantic paragraph set to generate a word sequence of each semantic paragraph;

[0142] perform similarity calculation on each word sequence, merge adjacent words with a similarity reaching a predetermined threshold, and give a first semantic segment set; when performing similarity calculation, only adjacent words in the word sequence can be calculated;

[0143] perform segment division on each semantic paragraph in the semantic paragraph set based on a sliding window determination condition, and give a second semantic segment set;

[0144] merge and filter the first semantic segment set and the second semantic segment set to obtain a final semantic segment set;

[0145] based on a preset hash function family, identify and perform multiple fusion weight processing on each semantic segment in the final semantic segment set to give a signature identification of each semantic paragraph.

[0146] The word segmentation processing can use a light-weight model such as DistilBERT to perform similarity calculation on each word sequence, merge adjacent words with a similarity reaching a predetermined threshold, and give a first semantic segment set, which can include:

[0147] vectorize each word in the word sequence through a pre-trained language model to give a corresponding word vector; the pre-trained language model can be a FastText model;

[0148] perform similarity calculation on the word vectors of adjacent words in each word sequence to obtain the similarity of adjacent words;

[0149] combine adjacent words with a similarity exceeding a predetermined threshold into a merged word;

[0150] integrate the merged word and the words that have not been merged as a semantic segment to give a first semantic segment set.

[0151] It can be understood that a language model such as FastText can learn the co-occurrence relationship between words and capture the context information of the words through pre-training. For example, if “deep learning” and “model” appear frequently, they are considered to be semantically related words, and their corresponding word vectors will be adjusted to be closer during training, thereby increasing the similarity between them. Merge adjacent words with a similarity exceeding a predetermined threshold as a semantic segment, and all semantic segments and remaining words that have not been merged constitute a first semantic segment set.

[0152] Specifically, perform word segmentation on the semantic paragraph to generate a word sequence represented as:

[0153]

[0154] In the formula, W represents a word sequence, ω i (1≤i≤m) represents the i-th word in the word sequence.

[0155] For each word ω i , a corresponding word vector v is generated using the FastText pre-training model;

[0156] Based on the word vectors of adjacent words, the similarity of adjacent words can be determined, satisfying the following relationship:

[0157]

[0158] In the formula, sim is the similarity, v i is the word vector of the i-th word, v i+1 is the word vector of the i+1-th word, v i and v i+1 represent the word vectors of adjacent words.

[0159] For the determined similarity, when sim is greater than a predetermined threshold, the adjacent words are merged into one semantic unit as a semantic segment, and all semantic segments and the remaining words that are not merged constitute a first semantic segment set.

[0160] Based on the sliding window determination condition, each semantic paragraph in the semantic paragraph set is divided into segments to give a second semantic segment set, which can include:

[0161] Determine the character sequence of each semantic paragraph; wherein the semantic paragraph itself is a character sequence composed of characters;

[0162] Based on all character sequences, give the appearance probability of each character in each character sequence;

[0163] Based on the characters possessed by each character sequence and combined with the corresponding appearance probability, give the information entropy of each semantic paragraph;

[0164] Based on the information entropy of each semantic paragraph, determine the corresponding sliding window size;

[0165] Based on the corresponding sliding window size, divide each semantic paragraph into segments to obtain the second semantic segment set.

[0166] In this example, K-gram sliding window strategy is used to divide each semantic paragraph into segments. The semantic paragraph is regarded as a character sequence T=[c1,c2,...,c n ], where c n represents the n-th character, n is the total number of characters in the semantic paragraph, the sliding window size K is determined according to the information entropy of the semantic paragraph, the window step is 1, and continuous character segments, i.e. K-gram segments, are generated by window sliding, which is specifically represented as {ci c i+1 ...c i+K+1 {|1 ≤ i ≤ n - K + 1}, and all character segments generated by window sliding together constitute the second semantic segment set of this semantic paragraph.

[0167] Among them, the sliding window size K is obtained through information entropy, and the information entropy satisfies the following relationship:

[0168]

[0169] In the formula, H is the information entropy of the semantic paragraph, c represents the characters in the semantic paragraph, chars represents the set of all characters in the semantic paragraph, and p(c) is the probability of the character c appearing in the semantic paragraph.

[0170] Based on information entropy, according to the preset K - value selection strategy, the sliding window size K of the semantic paragraph is dynamically determined, specifically expressed as:

[0171]

[0172] In the formula, H ≥ 4 corresponds to high information entropy, indicating that the semantic paragraph has relatively complex content. When H < 3, it corresponds to low information entropy, indicating that the semantic paragraph has relatively simple content. By dynamically adjusting the size of the sliding window K, it can adapt to semantic paragraphs or text types with different complexities. For example, long terms in technical documents are retained in detail with a small sliding window, and short sentences in prose are resistant to noise with a large sliding window, achieving the optimization of the subsequent signature identification quality and improving the quality and efficiency of text compression.

[0173] Merge the first semantic segment set and the second semantic segment set to obtain a preliminary mixed set; perform filtering processing on the preliminary mixed set to obtain the final semantic segment set. Specifically, the filtering processing of the preliminary mixed set can include complete de - duplication, overlapping processing, and length filtering. Among them, complete de - duplication is to remove completely identical semantic segments and only retain one such semantic segment; overlapping processing is to retain both the semantic segments that partially overlap in the second semantic segment set and the first semantic segment set to capture the details of local segments, so that the final semantic segment set can retain the complete semantics; length filtering is to remove overly short semantic segments (such as segments with length < 2, such as remaining words like "in", "among" with no actual compression value), reducing invalid data and improving the text information compression efficiency. As Figure 4 shown, the finally obtained semantic segment set contains both K - gram segments representing local features and semantic units with complete semantics, comprehensively retaining the multi - dimensional information of the text to be compressed, and can also improve the accuracy of text duplication checking in the text duplication checking scenario.

[0174] As Figure 5As shown, based on the preset hash function family, the semantic segments of each semantic paragraph are identified and processed by multiple fusion weights, and the signature identification of each semantic paragraph is given, which can include:

[0175] Each semantic segment of each semantic paragraph is calculated by multiple weight calculation methods and fused to give the fusion weight of each semantic segment.

[0176] Each semantic segment is spliced with its position information to give a position-enhanced semantic segment.

[0177] Based on each hash function in the preset hash function family using different seed values, all position-enhanced semantic segments in each semantic paragraph are calculated by hash to give the hash value of each position-enhanced semantic segment corresponding to different hash functions.

[0178] Based on the seed value of each hash function, the identification of the text to be processed, and the fusion weight of each semantic segment, the adjustment value of each position-enhanced semantic segment corresponding to different hash functions is given.

[0179] The hash values of each position-enhanced semantic segment in the semantic paragraph corresponding to different hash functions are fused with the corresponding adjustment values to give the weighted hash values of each position-enhanced semantic segment in the semantic paragraph corresponding to different hash functions.

[0180] The minimum weighted hash value of each hash function is selected from all weighted hash values of the semantic paragraph to give the signature identification of each semantic paragraph, which specifically includes:

[0181] The minimum weighted hash value of each hash function is selected from all weighted hash values of the semantic paragraph to determine the signature component of each hash function in the semantic paragraph.

[0182] All signature components in the semantic paragraph are arranged in the order of hash functions to generate the signature identification of the corresponding semantic paragraph.

[0183] In actual application scenarios, signature identification generation is to convert effective semantic paragraphs into fixed-length digital signatures. Through multi-hash function calculation, high compression of text content is realized while preserving the similarity features of the text.

[0184] The TF-IDF weight, part-of-speech weight, semantic weight, and other ways are used to calculate the weight of each semantic segment in each semantic paragraph, and the weights obtained by each method are fused to determine the fusion weight of each semantic segment.

[0185] Specifically, the TF-IDF weight of each semantic segment in the semantic segment set is calculated, which satisfies the following relationship:

[0186]

[0187] wherein, is the TF-IDF weight of the jth semantic segment, is the appearance frequency of the jth semantic segment s j , df(s j ) is the appearance frequency of the text containing the jth semantic segment , and N is the total number of texts, which is the number of texts in the corpus, wherein the corpus can be pre-set according to actual scene requirements. The TF-IDF weight is based on the statistical information of the corpus, and is used to evaluate the importance of the semantic segment in the global context, is the inverse document frequency, which measures the scarcity of the semantic segment.

[0188] For the part-of-speech weight of each semantic segment in the semantic segment set, it can be determined based on a pre-set part-of-speech weight table. First, a dictionary or a part-of-speech tagging tool is used to label each semantic segment with a part of speech, such as using jieba, LTP or spaCy tagging tool, and then according to the labeled part of speech, the pre-set weight table is used to assign a part-of-speech weight to each semantic segment. In this example, the part-of-speech weight of nouns / professional terms is 1.0, the part-of-speech weight of verbs is 0.8, the part-of-speech weight of adjectives is 0.6, and the part-of-speech weight of other words is 0.2.

[0189] The semantic weight refers to: if a semantic segment is a semantic unit (i.e., a semantic segment obtained by merging adjacent words in the semantic segment set), a fixed high weight such as 1.5 is assigned to it; or the TF-IDF weight, the part-of-speech weight of the semantic unit (semantic segment) is multiplied by an enhancement coefficient (such as 2) as the semantic weight. For non-semantic unit semantic segments, a fixed low weight is assigned to them; or the TF-IDF weight, the part-of-speech weight of the semantic segment is multiplied by a penalty coefficient (such as 0.5) as the semantic weight.

[0190] An attention mechanism is used to let the fusion model learn the importance of different weights, and the TF-IDF weight, the part-of-speech weight, and the semantic weight of each semantic segment are fused as the fusion weight. The fusion model is a Query-Key-Value (QKV) model, which calculates the attention score of each weight, then multiplies the attention score by the corresponding weight, and then sums up to complete the weight fusion to obtain the fusion weight of the semantic segment. The attention mechanism is specifically represented as:

[0191]

[0192] In the formula, Q represents the query, indicating the importance of the weights. In this example, it is the mean of the vectorized TF-IDF weights, part-of-speech weights, and semantic weights. K represents the key, which in this example is the concatenated matrix of TF-IDF weights, part-of-speech weights, and semantic weights. V represents the value, which in this example is the concatenated matrix of TF-IDF weights, part-of-speech weights, and semantic weights. softmax is the normalization function.

[0193] The fusion coefficients for different weights can also be dynamically adjusted based on text attributes. For example, different fusion coefficients can be set for TF-IDF weights, part-of-speech weights, and semantic weights based on text types such as academic texts, news texts, and novels; and different fusion coefficients can be set for TF-IDF weights, part-of-speech weights, and semantic weights based on attributes such as computer science, medicine, and finance. This makes the acquisition of fusion weights more closely aligned with text characteristics. For instance, for academic texts, the semantic weight coefficient can be increased to focus on semantic fragments and rare words; for news texts, the TF-IDF weight coefficient can be increased to focus on event keywords and rare words.

[0194] In this method, the hash function family is designed using 128 independent MurmurHash3 hash functions. Each hash function is generated using a different seed value, with the seed value ranging from [1, 2]. 32 -1] can ensure the independence between hash functions;

[0195] During the signature generation phase, for each semantic segment s in the semantic segment set S corresponding to the semantic segment, j ∈S, determine its starting position p j Positional encoding, i.e., character offset encoding within a semantic segment, is then concatenated into the semantic segment to form a position-enhanced semantic segment, satisfying the following relationship:

[0196]

[0197] In the formula, || represents concatenation, encode() is the encoder, and s j For semantic fragments, p j For semantic fragments s j The starting position, Position-enhanced semantic fragments are added. The hash values ​​of all position-enhanced semantic fragments are calculated using 128 hash functions. These fragments contain location information, enhancing the readability and verifiability of the compressed text, facilitating source tracing, and enabling precise location of duplicate content in text plagiarism detection scenarios. This avoids misjudgments in structural recombination and helps detect localized plagiarism.

[0198] According to the seed value of each hash function, the seed value of the random number generator is given, and a uniform random variable on [0, 1] is generated, combined with the fusion weight of each semantic segment, to give the adjustment value of the corresponding position-enhanced semantic segment. Specifically, it is expressed as:

[0199]

[0200] In the formula, represents the adjustment value of the position-enhanced semantic segment , is the fusion weight of the semantic segment corresponding to the position-enhanced semantic segment, is a uniform random variable on [0, 1], generated by the seed value of the random number generator, and the seed value of the random number generator is obtained by the seed value of the hash function and the identification of the text to be processed. Specifically, it is expressed as:

[0201]

[0202] In the formula, seed is the seed value of the random number generator, is the seed value of the i-th hash function, UUID is the identification of the text to be processed, which is the unique ID of the text to be processed, and M is a power of 2, which can be determined according to the actual situation.

[0203] The adjustment value of the position-enhanced semantic segment means that the core semantic segment weight is high, and its hash function value is more likely to become the smallest component of the signature identification; in the text duplication detection scenario, it can improve the recognition of local plagiarism. Specifically, when copying the core semantic segment, the weighted hash value of its high-weight semantic segment will dominate the generation of the signature representation, and the Jaccard similarity calculation in the duplication detection is more accurate, which can avoid missed judgment.

[0204] Fuse the hash value of the position-enhanced semantic segment with the corresponding adjustment value to get the weighted hash value. Specifically, it is expressed as:

[0205]

[0206] In the formula, is the weighted hash value of the position-enhanced semantic segment , is the hash value of the position-enhanced semantic segment , is the seed value of the i-th hash function, is the adjustment value of the position-enhanced semantic segment .

[0207] For each hash function h j , select the minimum value of all position-enhanced semantic segment weighted hash values under this hash function as the signature component, such asFigure 6 All signature components are arranged in the order of the hash function to generate the signature identifier corresponding to the semantic paragraph, which is specifically represented as:

[0208]

[0209] wherein M is the signature identifier of the semantic paragraph, represents the minimum value of the weighted hash values of all position-enhanced semantic segments in the semantic segment set S under the i-th hash function, that is, the signature component, and 128 signature components are arranged in the order of the hash function to form a 128-dimensional fixed-length signature identifier M = [m1, m2, …, m 128 ] where m i represents the i-th signature component and m i = Whas{h i (s)|s∈S}, which significantly compresses the representation dimension when performing text compression and improves the compression efficiency.

[0210] As Figure 7 shown, after obtaining the signature identifier, the signature identifier can be indexed, and the signature identifier and the identification information of the semantic paragraph are combined to generate compressed text information for indexing the distributed database, including:

[0211] Based on the preset division strategy, the signature identifier is divided into multiple wave bands;

[0212] The corresponding byte sequence of each wave band is calculated, and the bucket key corresponding to each wave band is given.

[0213] The signature identifier, the signature identifier corresponding to the text to be processed, and the identification information of the semantic paragraph corresponding to the signature identifier are stored in the index distributed database by taking the bucket key as the index.

[0214] In actual application scenarios, the wave bands are divided by the LSH (Local Sensitivity Hashing) bucketing strategy. The LSH bucketing strategy stores the MinHash signature after classification, which can realize the rapid aggregation of similar signature identifiers, and at the same time adopts a distributed storage architecture to meet the efficient storage and query requirements of large-scale signature data. Specifically, the following steps are included:

[0215] The generated 128-dimensional MinHash signature (i.e., the signature identifier of the semantic paragraph) M = [m1, m2, …, m 128 ] is taken as the processing object, wherein the signature component m j, the 128-dimensional MinHash signature is evenly divided into multiple bands band according to the continuous dimensions, each band contains multiple signature components, for example, the 128-dimensional MinHash signature is evenly divided into 16 bands band according to the continuous dimensions, each band contains 8 signature components, that is, band index band t The corresponding signature component interval is [m 8(t-1)+1 ,m 8(t-1)+2 ,...,m 8t ], band t is the tth band, band t ={m k |k=8(t-1)+1,8(t-1)+2,...,8t}, t is the band index, 1≤t≤16, m k is the kth component in the MinHash signature, 8 is the number of signature components contained in each band, for each band band t , extract the 8 signature components contained to form a subvector, use the MD5 hash algorithm to operate the subvector to generate a bucket key, convert the subvector [m 8(t-1)+1 ,m 8(t-1)+2 ,...,m 8t ] into a byte sequence, each signature component is encoded as a 4-byte unsigned integer, and a 128-bit hash value is obtained as the bucket key BK t corresponding to the band, that is, BK t ==MD5((bytes(m k ))=MD5(bytes(m 8(t-1)+1 )||bytes(m 8(t-1)+2 )||...||bytes(m 8t )), where bytes(m k ) represents the operation of converting the kth signature component into a byte sequence, || represents the splicing operation of the byte sequence, and BK t is the bucket key corresponding to the tth band.

[0216] After determining the bucket key corresponding to each wave band, a collaborative architecture of a real-time cache database and a full-amount storage database is used to store the bucket key, the signature identifier, the signature identifier corresponding to the to-be-processed text, and the identification information of the signature identifier corresponding to the semantic paragraph; the real-time cache database stores the mapping relationship between the bucket key and the MinHash signature, that is, after the bucket key is determined, the corresponding MinHash signature can be read from the real-time cache database, the full-amount storage database stores the bucket key, the MinHash signature, and the mapping relationship therebetween, and also stores full-amount data such as the to-be-processed text corresponding to the signature identifier and the identification information of the semantic paragraph corresponding to the signature identifier. In an actual application scenario, a distributed database architecture in which Redis (Remote Dictionary Server) and HBase (Hadoop Database) are collaborative can be used to store the MinHash signature and the original text and paragraph identification information corresponding thereto in a database indexed by the bucket key, so as to realize efficient data management of the cache and the full-amount storage. Redis, as a cache database, is used to store the mapping relationship between the bucket key and the MinHash signature, each bucket key is taken as a key, and the corresponding MinHash signature M is taken as a value, so as to construct a key-value pair storage. Since Redis is based on memory storage, it can provide a millisecond-level query response speed and can meet the rapid candidate recall requirement in a high-frequency query scenario; HBase, as a distributed column storage database, is used to store full-amount data to ensure long-term reliability and scalability of the data, and the table structure thereof is designed as follows: the row key is set as the bucket key to ensure centralized storage of data with the same bucket key; the column family is divided into three categories, among which, the sig column family is used to store the MinHash signature M (stored in the form of binary or string), the doc column family is used to store the identification information of the text (including a text unique identifier, a text name, an upload time, a file format, and the like), and the para column family is used to store the semantic paragraph identification information (including the starting position and the ending position of the semantic paragraph in the text, a paragraph serial number, and the like); this column family type storage structure supports flexible field extension and facilitates fast query of the complete document and the paragraph information corresponding to the signature according to the bucket key; in the data writing process, the bucket key, the MinHash signature, and the identification information are synchronously written into the Redis cache and the HBase full-amount database to ensure consistency of the cache and the full-amount data; when the cache data in Redis exceeds the preset validity period or storage capacity, the cache is evicted through the LRU (Least Recently Used) strategy, and the evicted data can be re-read from HBase to ensure the availability of the data.

[0217] As shown in Figure 8 After generating the compressed text information for indexing the distributed database, the present application can further include:

[0218] The generated compressed text information is used to process the to-be-duplicated text (for example, the to-be-duplicated text is sequentially preprocessed, multi-level semantic paragraph division, segment division, mapping processing, and indexing processing), and the signature identification and indexing processing result of the to-be-duplicated text are given.

[0219] Based on the indexing processing result of the to-be-duplicated text, the corresponding signature identification is queried from the index distributed database.

[0220] Based on the signature identification of the to-be-duplicated text and the signature identification of the index distributed database, the text duplication result is given.

[0221] In the process of text information duplication checking, the input text to be checked (which can be a single text or a paragraph) is preprocessed, multi-level semantic paragraph division, segment division and mapping processing, and indexing processing are repeatedly performed as in the process of building the distributed database. Specifically, the text to be checked is converted to a standard text by format conversion and noise removal; the semantic segmentation and invalid paragraph filtering are performed to obtain the effective semantic paragraphs of the text to be checked; K-gram processing, hash function calculation and MinHash signature generation are performed on each effective paragraph to obtain the MinHash signature set of the text to be checked; and each MinHash signature is divided into multiple wave segments according to the LSH bucketing strategy to generate the corresponding bucket key set. To improve the recall efficiency of the candidate, the BloomFilter is used to pre-filter the bucket key set of the text to be checked. Specifically, for each bucket key of the text to be checked, the BloomFilter is queried first. If the BloomFilter determines that the bucket key does not exist, the subsequent query of the bucket key is skipped. If the BloomFilter determines that the bucket key exists, the MinHash signature corresponding to the bucket key is queried from the distributed database. In the query process, the Redis cache is queried first. If the MinHash signature corresponding to the bucket key exists in the Redis, the MinHash signature is directly read. If the Redis does not hit (i.e., the bucket key data does not exist in the cache), the HBase full database is queried to read the MinHash signature corresponding to the bucket key. Then, all the MinHash signatures queried from the Redis and the HBase are de-duplicated, i.e., the repeated signatures are removed to avoid repeated calculation in the future, and finally the candidate signature set of the text to be checked is obtained. The Jaccard similarity between the query MinHash signature of the text to be checked and the candidate MinHash signature queried from the index distributed database is calculated, i.e., Jaccard(A,B)=|A∩B| / |A∪B|, where A is the semantic segment set corresponding to the query MinHash signature, and B is the semantic segment set corresponding to the candidate MinHash signature. The similarity value is obtained by calculating the size ratio of the intersection and union of the two semantic segment sets. When the Jaccard similarity between the query MinHash signature of the text to be checked and the candidate MinHash signature is greater than or equal to the first threshold value, the position information of the repeated semantic paragraph is recorded, including the UUID of the text, the starting position of the semantic paragraph, the length of the repetition, etc. When the Jaccard similarity is less than the first threshold value and greater than or equal to the second threshold value, it is determined that the text is a suspected duplicate text, which needs to be reviewed and marked for manual review to determine whether it is a duplicate text. When the Jaccard similarity is less than the second threshold value, it is determined that the text is not a duplicate text. The first threshold value is in the range of 0.8-0.9, and the preferred value is 0.85. The second threshold value is in the range of 0.5-0.7, and the preferred value is 0.6.After the judgment based on the first threshold value and the second threshold value, the proportion of the repeated text and the suspected repeated text in the to-be-checked repeated text can be calculated respectively, and the repeated content is sorted from high to low according to the proportion, and a duplicate checking report containing the similarity value, the repeated position and the proportion information is generated, so as to facilitate subsequent rapid positioning of the repeated content.

[0222] After realizing data storage and text duplicate checking, the duplicate checking result can be written into a transaction database, an analysis database and a search engine index synchronously, multi-dimensional query and analysis are supported, at the same time, the whole process data is recorded through a log collection link, and performance indicators such as false positive rate and false negative rate are monitored in real time, when the performance indicators deviate from the predetermined range, the duplicate checking result is manually audited, the misjudgment cases are collected, and the seed value of the MinHash hash function and the band number of the LSH bucket are adjusted based on the misjudgment cases, that is, when the false positive rate > 5% or the false negative rate > 5%, an alarm is triggered to start an offline training process, the seed value of the MinHash hash function and the LSH bucket parameter are optimized and updated, the band number is increased from 16 to 20, so as to reduce the number of candidate signatures in the bucket, or the band number is reduced from 16 to 12, so as to expand the recall range of the candidate signature, and the updated parameter is deployed to the online system after testing.

[0223] As shown in Figure 9 The present application also provides a text information compression device for indexing a distributed database, which adopts the above-mentioned text information compression method, and the device comprises:

[0224] a preprocessing module for obtaining and preprocessing to-be-processed text to give pure text in a predetermined format;

[0225] a paragraph division module for performing multi-level semantic paragraph division on the pure text in the predetermined format to give a corresponding semantic paragraph set;

[0226] an identification generation module for performing fragment division and mapping processing on each semantic paragraph in the semantic paragraph set considering weight and position to give a signature identification of each semantic paragraph;

[0227] an indexing and storage module for indexing the signature identification and generating compressed text information for indexing the distributed database in combination with the signature identification and the identification information of the semantic paragraph.

[0228] Further, the preprocessing module is used for:

[0229] obtaining multi-source to-be-processed text;

[0230] performing unified coding on the multi-source to-be-processed text to generate initial text;

[0231] performing cleaning and denoising processing on the initial text to obtain pure text in a predetermined format.

[0232] Further, the paragraph division module is configured to:

[0233] punctuation and line space recognition on the plain text in the predetermined format to determine the positions and types of all punctuation in the plain text and the positions of line spaces formed by two or more consecutive line spaces;

[0234] determine the forced segmentation points based on the positions and types of all punctuation in the plain text and the positions of line spaces formed by two or more consecutive line spaces;

[0235] perform a text cutting on the plain text in the predetermined format according to the forced segmentation points to obtain a plurality of basic text segments;

[0236] perform processing and analysis on the basic text segments through a sliding window to give a cosine similarity matrix;

[0237] determine the cosine similarity change rates of a plurality of consecutive sliding windows based on the cosine similarity matrix, and give semantic break points in combination with a preset amplitude threshold;

[0238] perform a secondary text cutting on the plurality of basic text segments based on the semantic break points to give an initial semantic paragraph set;

[0239] filter the initial semantic paragraph set based on a preset filtering strategy to give a final semantic paragraph set.

[0240] Further, the signature generation module is configured to:

[0241] perform word segmentation processing on each semantic paragraph in the semantic paragraph set to generate a word sequence of each semantic paragraph;

[0242] perform similarity calculation on each word sequence, merge adjacent words with a similarity reaching a predetermined threshold, and give a first semantic segment set;

[0243] perform segment division on each semantic paragraph in the semantic paragraph set based on a sliding window determination condition to give a second semantic segment set;

[0244] perform mixed filtering on the first semantic segment set and the second semantic segment set to obtain a final semantic segment set;

[0245] perform signature and multiple fusion weight processing on each semantic segment in the final semantic segment set based on a preset hash function family to give a signature of each semantic paragraph.

[0246] Further, the signature generation module is configured to:

[0247] perform vectorization on each word sequence through a pre-trained language model to give a word vector of each word in the word sequence;

[0248] Similarity calculation is performed on the word vectors of adjacent words in each word sequence to obtain the similarity of the adjacent words.

[0249] Adjacent words with similarity exceeding a predetermined threshold are combined into a merged word.

[0250] The merged word and the word that has not been merged are integrated as a semantic segment respectively to obtain a first set of semantic segments.

[0251] Further, the identification generation module is configured to:

[0252] determine a character sequence of each semantic paragraph;

[0253] based on all character sequences, give the occurrence probability of each character in each character sequence respectively;

[0254] based on the characters possessed by each character sequence and in combination with the corresponding occurrence probability, give the information entropy of each semantic paragraph;

[0255] based on the information entropy of each semantic paragraph, determine the corresponding sliding window size;

[0256] based on the corresponding sliding window size, perform segment division on each semantic paragraph to obtain a second set of semantic segments.

[0257] Further, the identification generation module is configured to:

[0258] perform weight calculation and fusion on the semantic segments of each semantic paragraph respectively through multiple weight calculation methods to give the fusion weight of each semantic segment;

[0259] splice each semantic segment with its position information to give a position-enhanced semantic segment;

[0260] based on each hash function in the preset hash function family using different seed values, perform hash calculation on all position-enhanced semantic segments in each semantic paragraph respectively to give the hash value of each position-enhanced semantic segment corresponding to different hash functions;

[0261] based on the seed value of each hash function, the identification of the text to be processed, and the fusion weight of each semantic segment, give the adjustment value of each position-enhanced semantic segment corresponding to different hash functions;

[0262] fuse the hash value of each position-enhanced semantic segment corresponding to different hash functions in the semantic paragraph with the corresponding adjustment value respectively to give the weighted hash value of each position-enhanced semantic segment corresponding to different hash functions in the semantic paragraph;

[0263] select the minimum weighted hash value of each hash function from all weighted hash values of the semantic paragraph to give the signature identification of each semantic paragraph.

[0264] Further, the index and storage module is used for:

[0265] dividing the signature identifier into multiple wave bands based on a preset division strategy;

[0266] respectively calculating corresponding byte sequences for all wave bands, and giving a corresponding bucket key for each wave band;

[0267] storing the signature identifier, the signature identifier corresponding to the to-be-processed text and the signature identifier corresponding to the semantic paragraph identifier information into an index distributed database by taking the bucket key as an index.

[0268] Although the preferred embodiments of the present application have been described, those skilled in the art who understand the basic inventive concept can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application. Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for compressing text information for indexing a distributed database, characterized in that, include: Get the text to be processed and preprocess it to give plain text in a predefined format; Perform multi-level semantic paragraph division on plain text with a predefined format and provide the corresponding set of semantic paragraphs; Each semantic segment in the semantic segment set is segmented into words to generate a word sequence for each semantic segment; Calculate the similarity of each word sequence, merge adjacent words whose similarity reaches a predetermined threshold, and give the first set of semantic segments; Based on the sliding window determination conditions, each semantic segment in the semantic segment set is divided into segments to give a second semantic segment set, including: determining the corresponding sliding window size based on the information entropy of each semantic segment; and dividing each semantic segment into segments based on the corresponding sliding window size. The first and second semantic fragment sets are merged and filtered to obtain the final semantic fragment set. The semantic segments of each semantic paragraph are weighted and fused using multiple weighting methods to provide a fusion weight for each semantic segment. Each semantic segment is then concatenated with its positional information to provide a position-enhanced semantic segment. Based on each hash function with a different seed value from a pre-defined hash function family, hash calculations are performed on all position-enhanced semantic segments within each semantic paragraph to provide a hash value for each position-enhanced semantic segment corresponding to a different hash function. Based on the seed value of each hash function, the identifier of the text to be processed, and the fusion weight of each semantic segment, adjustment values ​​for each position-enhanced semantic segment corresponding to different hash functions are provided. The hash values ​​of each position-enhanced semantic segment corresponding to different hash functions in the semantic paragraph are fused with their corresponding adjustment values ​​to provide a weighted hash value for each position-enhanced semantic segment corresponding to different hash functions in the semantic paragraph. Finally, the minimum weighted hash value for each hash function is selected from all weighted hash values ​​of the semantic paragraph to provide a signature identifier for each semantic paragraph. The signature identifier is indexed and combined with the identifier information of the semantic paragraph to generate compressed text information for indexing the distributed database.

2. The text information compression method as described in claim 1, characterized in that, Obtain the text to be processed and preprocess it to provide plain text in a predefined format, including: Retrieve text from multiple sources; The text from multiple sources to be processed is uniformly encoded to generate the initial text; The initial text is cleaned and denoised to obtain plain text in a predetermined format.

3. The text information compression method as described in claim 1, characterized in that, Perform multi-level semantic paragraph division on plain text with a predefined format, and provide the corresponding set of semantic paragraphs, including: Punctuation and blank lines are identified in plain text with a predefined format. The position and type of all punctuation marks in the plain text are determined, as well as the position of blank lines formed by two or more consecutive line breaks. The forced breakpoint is determined based on the position and type of all punctuation marks in the plain text, as well as the position of the blank line formed by two or more consecutive newline characters. Based on the forced segmentation points, the plain text in the predefined format is cut into multiple basic text segments. The basic text fragments are processed and analyzed using a sliding window, and a cosine similarity matrix is ​​given. Based on the cosine similarity matrix, the cosine similarity change rate of multiple consecutive sliding windows is determined, and semantic breakpoints are given in combination with a preset amplitude threshold. Based on semantic breakpoints, multiple basic text segments are subjected to secondary text segmentation to give an initial set of semantic paragraphs; The initial set of semantic paragraphs is filtered based on a preset filtering strategy to give the final set of semantic paragraphs.

4. The text information compression method as described in claim 1, characterized in that, For each word sequence, similarity is calculated, and adjacent words with similarity reaching a predetermined threshold are merged to give the first semantic fragment set, including: Each word sequence is vectorized using a pre-trained language model, resulting in a word vector for each word in the sequence. The similarity between adjacent words is calculated by performing word vector similarity calculation on the word vectors of adjacent words in each word sequence. Combine adjacent words with a similarity exceeding a predetermined threshold into a single merged word; The merged words and the unmerged words are treated as semantic segments and then integrated to obtain the first set of semantic segments.

5. The text information compression method as described in claim 4, characterized in that, Based on the sliding window determination conditions, each semantic segment in the semantic segment set is divided into segments, resulting in a second set of semantic segments, including: Determine the character sequence for each semantic segment; Based on all character sequences, give the probability of each character appearing in each character sequence; Based on the characters in each character sequence and their corresponding occurrence probabilities, the information entropy of each semantic segment is given; The size of the corresponding sliding window is determined based on the information entropy of each semantic segment. Each semantic segment is divided into segments based on the corresponding sliding window size to obtain a second set of semantic segments.

6. The text information compression method as described in claim 1, characterized in that, The signature identifier is indexed, and combined with the signature identifier and the identifier information of semantic paragraphs, compressed text information for indexing the distributed database is generated, including: Based on a preset partitioning strategy, the signature identifier is divided into multiple bands; Calculate the corresponding byte sequence for each band and give the bucket key for each band; Using bucket keys as indexes, the signature identifier, the text to be processed corresponding to the signature identifier, and the identifier information of the semantic paragraph corresponding to the signature identifier are stored in the indexed distributed database.

7. The text information compression method as described in claim 1, characterized in that, Also includes: The text to be checked for plagiarism is preprocessed, divided into multi-level semantic paragraphs, fragments, mapped, and indexed in sequence, and the signature identifier and indexing result of the text to be checked for plagiarism are given. Based on the indexing results of the text to be deduplicated, the corresponding signature identifier is queried from the indexed distributed database; Based on the signature identifier of the text to be checked and the signature identifier of the indexed distributed database, the text plagiarism detection results are given.

8. A text information compression device for indexing a distributed database, characterized in that, The apparatus employing the text information compression method as described in any one of claims 1 to 7 includes: The preprocessing module is used to acquire the text to be processed and preprocess it to give plain text in a predetermined format; The paragraph segmentation module is used to perform multi-level semantic paragraph segmentation on plain text with a predefined format and provide the corresponding set of semantic paragraphs; The identifier generation module is used to perform segmentation and mapping processing on each semantic segment in the semantic segment set, taking into account weight and position, and to give a signature identifier for each semantic segment. The indexing and storage module is used to index the signature identifier and, in combination with the signature identifier and the identifier information of the semantic paragraph, generate compressed text information for indexing the distributed database.

Images